I am writing an Azure python function that is triggered and then generates multiple outgoing messages. I need to send EventData (body + properties) messages to an Eventhub. Thus far I have not found any way to do add properties to an outgoing message using the EventHub output bindings. It appears that the output string is put into the "body" property.
One possible solution that I see is to write an EventHubClient into the function, but is that really the most effective way to get properties send with the message? Why would there be output bindings then?
My function.json file is:
{
"type": "eventHub",
"name": "outputHub",
"eventHubName": "test",
"connection": "TestSendConnection",
"direction": "out"
}
Here is my code:
def main(events: func.EventHubEvent,
referenceInput: func.InputStream,
outputHub: func.Out[str]):
logging.info('Send an output event to eventhub')
evt_data_list = []
for k in range(0,10):
evt_data = EventData("Sample Body")
evt_data.properties['EventType'] = "log"
evt_data_list.append(evt_data)
logging.info('Send an output event to eventhub')
import random
outputHub.set("[" + ",".join([str(evt) for evt in evt_data_list]) + "]")
I am monitoring the incoming messages with the Azure Event Hub Explorer and I receive multiple messages, but they arrive in the following format. I need the body and the properties sections to be separate for the external parser.
{
"body": {
"body": "Sample Body",
"properties": {
"EventType": "log"
}
},
"enqueuedTimeUtc": "2020-06-09T17:59:04.803Z",
"offset": "1335734859528",
"sequenceNumber": 4995022
}
Currently I am afraid there is no way to add properties to an outgoing message using the EventHub output bindings.
The workaround is using EventHub SDK inside the function.
Reference:
Microsoft Azure SDK for Event Hubs(Python)
Related
I'm testing the Kafka Trigger and Output Binding in Azure Functions to consume a topic and write the message to another topic, very simple code.
But when I enable the auto-scale feature and the function provision new instances, I'm losing the 'exactly-once' feature and apparently some messages are being delivered to more than one instance.
host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.6.0, 4.0.0)"
}
}
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"type": "kafkaTrigger",
"name": "kafkaTrigger",
"direction": "in",
"brokerList": "%PEP_BEES_KAFKA_BOOTSTRAP%",
"topic": "%PEP_BEES_KAFKA_SOURCE_TOPIC%",
"consumerGroup": "%PEP_BEES_KAFKA_SOURCE_TOPIC_CONSUMER_GROUP%"
},
{
"type": "kafka",
"direction": "out",
"name": "kafkaOutput",
"brokerList": "%PEP_BEES_KAFKA_BOOTSTRAP%",
"topic": "%PEP_BEES_KAFKA_DESTINATION_TOPIC%"
}
]
}
init.py
import logging, json
from azure.functions import KafkaEvent
import azure.functions as func
import typing
def main(kafkaTrigger : typing.List[KafkaEvent], kafkaOutput: func.Out[str]):
message = json.loads(str(kafkaTrigger.get_body().decode('utf-8')))
input_msg = str(message['Value'])
kafkaOutput.set(input_msg)
As you can see in the image below, the 'test-topic-output' (destination) has more messages than the 'test-topic' (source), indicating that sometimes more than one instance is consuming a message:
Message count
If I disable the auto-scaling feature, this behavior does not happen.
I just need that the 'exactly-once' feature works even with the function auto-scale enabled, to have an elastic environment.
EDIT
I just found out some erros during the Kafka Trigger processing:
Kafka Trigger errors
Confluent.Kafka.KafkaException:
at Confluent.Kafka.Impl.SafeKafkaHandle.StoreOffsets (Confluent.Kafka, Version=1.9.0.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e)
at Confluent.Kafka.Consumer2.StoreOffset (Confluent.Kafka, Version=1.9.0.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e) at Microsoft.Azure.WebJobs.Extensions.Kafka.AsyncCommitStrategy2.Commit (Microsoft.Azure.WebJobs.Extensions.Kafka, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: /mnt/vss/_work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Trigger/AsyncCommitStrategy.cs:28)
at Microsoft.Azure.WebJobs.Extensions.Kafka.FunctionExecutorBase`2.Commit (Microsoft.Azure.WebJobs.Extensions.Kafka, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: /mnt/vss/_work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Trigger/FunctionExecutorBase.cs:87)
If I disable the auto-scaling feature, this behavior does not happen
If you scale up, that wouldn't prevent records that were just consumed from existing inputs from being sent to output function.
You'd need a way to catch a consumer group rebalance within the function, and upon that action, then don't send to the output... Given that Azure doesn't expose Kafka native API, then its unlikely that is possible. You also would need to control when the function commits its consumer offsets (assuming PEP_BEES_KAFKA_SOURCE_TOPIC_CONSUMER_GROUP is a constant value); e.g. if it commits before kafkaTrigger.get_body(), then you should get less "extras", but if it commits after kafkaOutput.set(), then you'll have more upon scaling up...
Rather than use Python or serverless functions for replicating topics, you can use MirrorMaker or Kafka Streams instead.
I had noticed similar behavior of the kafka triggers consuming duplicate events when the scaling up the instance count (and surprisingly also during scale down). In batch mode with batch size of 50, its always the same 50 events that get read by 2 instance. I have also noticed this happens only once per instance scale up or (down).
One of the potential solution you can explore is using a warmup trigger that run during scale-out operation.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-warmup?tabs=in-process&pivots=programming-language-python
Hello I'm trying to figure out how to use azure functions binding to extract data from a trigger and use it in an input db for filtering. I'm using service bus message as a trigger. It seems that service bus bindings listed in Azure Functions documentation always return null for me, (even if inside a function this parameter has value set).
That's what i've been trying so far in example below, i want to extract ApplicationProperties.zoneId set inside a message.
function.json
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"queueName": "myqueue",
"connection": "ServiceBusConnection"
},
{
"name": "documents",
"type": "cosmosDB",
"direction": "in",
"databaseName": "test_db",
"collectionName": "items",
"sqlQuery": "SELECT * from c where c.zoneId = {ApplicationProperties.zoneId}",
"connectionStringSetting": "CosmosDBConnection"
}]
for testing i'm sending test message to service bus:
def send_single_message(sender):
message = ServiceBusMessage(
"woah_a_test",
correlation_id="1",
subject="az-fcn",
)
message.application_properties = {"zoneId": 1}
sender.send_messages(message)
print("Sent a single message")
It seems quite strange for me that i could not access in function.json any of the trigger parameters (i've also tried CorrelationId and Subject), but inside of a triggered function those parameters return correct value.
I'm aware that i can bypass this issue by filtering inside a function code instead of using input binding, but I'm just curious why those params does not return expected value there. Is there any way to debug it?
I recently analyzed the Zenly application via mitmproxy, it turned out that it works on gRPC, I was able to extract the payload of the request. I want to emulate this request using Python with my own parameters. For example, there is such a URL:
https://secret.zenly.rpc.com/co.znly.users.services.sessions.SessionsService/SessionCreate, the request to which sends an SMS with a confirmation code. The payload for calling this function is as follows:
{
"PhoneNumber": "secret",
"device": {
"appVersion": "4.63.14",
"type": "ANDROID",
"osVersion": "12",
"model": "secret",
"acceptLanguages": "en-US;q=1.0",
"coreVersion": "1.96.7",
"appBundle": "app.zenly.locator"
},
"deviceOsUuid": "secret",
"carrierInformations": {
"networkOperatorCode": "25001",
"networkOperatorName": "MTS",
"networkCountryIso": "ru",
"simOperatorCode": "25001",
"simOperatorName": "MTS RUS",
"simCountryIso": "ru"
}
}
So, how to send all this to the server correctly, in serialized (?) a format without Proto-files?
p.s.
grpc_requests and grpcurl did NOT fit
You can reference my answer in this question: How to create GRPC client directly from protobuf without compiling it into java code, it should wrok in python also.
I wrote a azure function in python on Service bus topic, it works well but the issue is its not marking message as completed after finish.. the message process processed several time untill the count reached.. need to know how to mark message as complete here is my code.
def main(message: func.ServiceBusMessage):
try:
message_content_type = message.content_type
req_body = message.get_body().decode("utf-8")
logging.info(req_body)
response = obj_engine.extract_payload_message(req_body)
connection_str = os.environ["idpdev_SERVICEBUS"];
ocr_topic_name = os.environ["idpdev_topic"]
servicebus_client = ServiceBusClient.from_connection_string(conn_str=connection_str, logging_enable=True)
with servicebus_client:
sender = servicebus_client.get_topic_sender(topic_name=sender_topic_name)
with sender:
send_message = ServiceBusMessage(json.dumps(response))
sender.send_messages(send_message)
except ValueError:
pass
can someone please help to know what setting i am missing to add.
According to the docs you need to enable auto completion by setting autoComplete to true using the configuration:
Must be true for non-C# functions, which means that the trigger should either automatically call complete after processing, or the function code manually calls complete.
When set to true, the trigger completes the message automatically if the function execution completes successfully, and abandons the message otherwise.
Exceptions in the function results in the runtime calls abandonAsync in the background. If no exception occurs, then completeAsync is called in the background. This property is available only in Azure Functions 2.x and higher.
The configuration can be found in the function.json file, make sure the property is available and set to true, see the example below:
{
"scriptFile": "__init__.py",
"entryPoint": "main",
"bindings": [
{
"name": "message",
"type": "serviceBusTrigger",
"direction": "in",
"topicName": "mytopic",
"subscriptionName": "mysubscription",
"connection": "",
"autoComplete": "true"
}
]
}
I am implementing an azure function in python which is trying to batch process messages from a servicebus queue. I have modified the host.json file as follows:
{
"version": "2.0",
"serviceBus": {
"batchOptions": {
"maxMessageCount": 20,
"operationTimeout": "01:00:00",
"autoComplete": true
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
}
}
However, I am a bit lost as to how the messages will be received in my entry function in __init__.py. Will it just be a list of messages and I can just loop through the messages as follows:
def main(messages: func.ServiceBusMessage):
for msg in messages:
json_obj = json.loads(msg.get_body().decode("utf-8"))
print(json_obj)
Your binding configuration is not right, it is to be done in function.json properly. After your Azure function is configured as a Service Bus trigger, and bound correctly to the queue, then for each message generated, your function will get invoked once. Your function will not get array of messages. See the example code here to get more details.
Instead of a service bus Queue trigger, if you need your function app to process bunch of service bus messages, then you need to connect to the Service Bus Client and then receive the messages, see the code in this page.