Develop Reference

Azure Functions Input Binding - Python

Hello I'm trying to figure out how to use azure functions binding to extract data from a trigger and use it in an input db for filtering. I'm using service bus message as a trigger. It seems that service bus bindings listed in Azure Functions documentation always return null for me, (even if inside a function this parameter has value set).
That's what i've been trying so far in example below, i want to extract ApplicationProperties.zoneId set inside a message.
function.json
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"queueName": "myqueue",
"connection": "ServiceBusConnection"
},
{
"name": "documents",
"type": "cosmosDB",
"direction": "in",
"databaseName": "test_db",
"collectionName": "items",
"sqlQuery": "SELECT * from c where c.zoneId = {ApplicationProperties.zoneId}",
"connectionStringSetting": "CosmosDBConnection"
}]
for testing i'm sending test message to service bus:
def send_single_message(sender):
message = ServiceBusMessage(
"woah_a_test",
correlation_id="1",
subject="az-fcn",
)
message.application_properties = {"zoneId": 1}
sender.send_messages(message)
print("Sent a single message")
It seems quite strange for me that i could not access in function.json any of the trigger parameters (i've also tried CorrelationId and Subject), but inside of a triggered function those parameters return correct value.
I'm aware that i can bypass this issue by filtering inside a function code instead of using input binding, but I'm just curious why those params does not return expected value there. Is there any way to debug it?

Python Azure Function service bus topic complete after process

I wrote a azure function in python on Service bus topic, it works well but the issue is its not marking message as completed after finish.. the message process processed several time untill the count reached.. need to know how to mark message as complete here is my code.
def main(message: func.ServiceBusMessage):
try:
message_content_type = message.content_type
req_body = message.get_body().decode("utf-8")
logging.info(req_body)
response = obj_engine.extract_payload_message(req_body)
connection_str = os.environ["idpdev_SERVICEBUS"];
ocr_topic_name = os.environ["idpdev_topic"]
servicebus_client = ServiceBusClient.from_connection_string(conn_str=connection_str, logging_enable=True)
with servicebus_client:
sender = servicebus_client.get_topic_sender(topic_name=sender_topic_name)
with sender:
send_message = ServiceBusMessage(json.dumps(response))
sender.send_messages(send_message)
except ValueError:
pass
can someone please help to know what setting i am missing to add.

According to the docs you need to enable auto completion by setting autoComplete to true using the configuration:
Must be true for non-C# functions, which means that the trigger should either automatically call complete after processing, or the function code manually calls complete.
When set to true, the trigger completes the message automatically if the function execution completes successfully, and abandons the message otherwise.
Exceptions in the function results in the runtime calls abandonAsync in the background. If no exception occurs, then completeAsync is called in the background. This property is available only in Azure Functions 2.x and higher.
The configuration can be found in the function.json file, make sure the property is available and set to true, see the example below:
{
"scriptFile": "__init__.py",
"entryPoint": "main",
"bindings": [
{
"name": "message",
"type": "serviceBusTrigger",
"direction": "in",
"topicName": "mytopic",
"subscriptionName": "mysubscription",
"connection": "",
"autoComplete": "true"
}
]
}

Timeout errors when testing Azure function app

Using Azure functions in a python runtime env. to take latitude/longitude tuples from a snowflake database and return the respective countries. We also want to convert any non-english country names into English.
We initially found that although the script would show output in the terminal while testing on azure, it would soon return a 503 error (although the script continues to run at this point). If we cancelled the script it would show as a success in the monitor screen of azure portal, however leaving the script to run to completion would result in the script failing. We decided (partially based on this post) this was due to the runtime exceeding the maximum http response time allowed. To combat this we tried a number of solutions.
First we extended the function timeout value in the function.json file:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
},
"functionTimeout": "00:10:00"
}
We then modified our script to use a queue trigger by adding the output
def main(req: func.HttpRequest, msg: func.Out[func.QueueMessage]) ->func.HttpResponse:
to the main .py script. We also then modified the function.json file to
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "queue",
"direction": "out",
"name": "msg",
"queueName": "processing",
"connection": "QueueConnectionString"
}
]
}
and the local.settings.json file to
{
"IsEncrypted": false,
"Values": {
"FUNCTIONS_WORKER_RUNTIME": "python",
"AzureWebJobsStorage": "{AzureWebJobsStorage}",
"QueueConnectionString": "<Connection String for Storage Account>",
"STORAGE_CONTAINER_NAME": "testdocs",
"STORAGE_TABLE_NAME": "status"
}
}
We also then added a check to see if the country name was already in English. The intention here was to cut down on calls to the translate function.
After each of these changes we redeployed to the functions app and tested again. Same result. The function will run, and print output to terminal, however after a few seconds it will show a 503 error and eventually fail.
I can show a code sample but cannot provide the tables unfortunately.
from snowflake import connector
import pandas as pd
import pyarrow
from geopy.geocoders import Nominatim
from deep_translator import GoogleTranslator
from pprint import pprint
import langdetect
import logging
import azure.functions as func
def main(req: func.HttpRequest, msg: func.Out[func.QueueMessage]) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
# Connecting string to Snowflake
conn = connector.connect(
user='<USER>',
password='<PASSWORD>',
account='<ACCOUNT>',
warehouse='<WH>',
database='<DB>',
schema='<SCH>'
)
# Creating objects for Snowflake, Geolocation, Translate
cur = conn.cursor()
geolocator = Nominatim(user_agent="geoapiExercises")
translator = GoogleTranslator(target='en')
# Fetching weblog data to get the current latlong list
fetchsql = "SELECT PAGELATLONG FROM <TABLE_NAME> WHERE PAGELATLONG IS NOT NULL GROUP BY PAGELATLONG;"
logging.info(fetchsql)
cur.execute(fetchsql)
df = pd.DataFrame(cur.fetchall(), columns = ['PageLatLong'])
logging.info('created data frame')
# Creating and Inserting the mapping into final table
for index, row in df.iterrows():
latlong = row['PageLatLong']
location = geolocator.reverse(row['PageLatLong']).raw['address']
logging.info('got addresses')
city = str(location.get('state_district'))
country = str(location.get('country'))
countrycd = str(location.get('country_code'))
logging.info('got countries')
# detect language of country
res = langdetect.detect_langs(country)
lang = str(res[0]).split(':')[0]
conf = str(res[0]).split(':')[0]
if lang != 'en' and conf > 0.99:
country = translator.translate(country)
logging.info('translated non-english country names')
insertstmt = "INSERT INTO <RESULTS_TABLE> VALUES('"+latlong+"','"+city+"','"+country+"','"+countrycd+"')"
logging.info(insertstmt)
try:
cur.execute(insertstmt)
except Exception:
pass
return func.HttpResponse("success")
If anyone had an idea what may be causing this issue I'd appreciate any suggestions.
Thanks.

To resolve timeout errors, you can try following ways:
As suggested by MayankBargali-MSFT , You can try to define the retry policies and for Trigger like HTTP and timer, don't resume on a new instance. This means that the max retry count is a best effort, and in some rare cases an execution could be retried more than the maximum, or for triggers like HTTP and timer be retried less than the maximum. You can navigate to Diagnose and solve problems to see if it helps you to know the root cause of 503 error as there can be multiple reasons for this error
As suggested by ryanchill , 503 issue is the result of high memory consumption which exceeded the limits of the consumption plan. The best resolve for this issue is switching to a dedicated hosting plan which provides more resources. However, if that isn't an option, reducing the amount of data being retrieved should be explored.
References: https://learn.microsoft.com/en-us/answers/questions/539967/azure-function-app-503-service-unavailable-in-code.html , https://learn.microsoft.com/en-us/answers/questions/522216/503-service-unavailable-while-executing-an-azure-f.html , https://learn.microsoft.com/en-us/answers/questions/328952/azure-durable-functions-timeout-error-in-activity.html and https://learn.microsoft.com/en-us/answers/questions/250623/azure-function-not-running-successfully.html

Azure Function Servicebus batch process messages - python

I am implementing an azure function in python which is trying to batch process messages from a servicebus queue. I have modified the host.json file as follows:
{
"version": "2.0",
"serviceBus": {
"batchOptions": {
"maxMessageCount": 20,
"operationTimeout": "01:00:00",
"autoComplete": true
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
}
}
However, I am a bit lost as to how the messages will be received in my entry function in __init__.py. Will it just be a list of messages and I can just loop through the messages as follows:
def main(messages: func.ServiceBusMessage):
for msg in messages:
json_obj = json.loads(msg.get_body().decode("utf-8"))
print(json_obj)

Your binding configuration is not right, it is to be done in function.json properly. After your Azure function is configured as a Service Bus trigger, and bound correctly to the queue, then for each message generated, your function will get invoked once. Your function will not get array of messages. See the example code here to get more details.
Instead of a service bus Queue trigger, if you need your function app to process bunch of service bus messages, then you need to connect to the Service Bus Client and then receive the messages, see the code in this page.

Azure Functions Python | Send EventData messages with properties to Event Hub output

I am writing an Azure python function that is triggered and then generates multiple outgoing messages. I need to send EventData (body + properties) messages to an Eventhub. Thus far I have not found any way to do add properties to an outgoing message using the EventHub output bindings. It appears that the output string is put into the "body" property.
One possible solution that I see is to write an EventHubClient into the function, but is that really the most effective way to get properties send with the message? Why would there be output bindings then?
My function.json file is:
{
"type": "eventHub",
"name": "outputHub",
"eventHubName": "test",
"connection": "TestSendConnection",
"direction": "out"
}
Here is my code:
def main(events: func.EventHubEvent,
referenceInput: func.InputStream,
outputHub: func.Out[str]):
logging.info('Send an output event to eventhub')
evt_data_list = []
for k in range(0,10):
evt_data = EventData("Sample Body")
evt_data.properties['EventType'] = "log"
evt_data_list.append(evt_data)
logging.info('Send an output event to eventhub')
import random
outputHub.set("[" + ",".join([str(evt) for evt in evt_data_list]) + "]")
I am monitoring the incoming messages with the Azure Event Hub Explorer and I receive multiple messages, but they arrive in the following format. I need the body and the properties sections to be separate for the external parser.
{
"body": {
"body": "Sample Body",
"properties": {
"EventType": "log"
}
},
"enqueuedTimeUtc": "2020-06-09T17:59:04.803Z",
"offset": "1335734859528",
"sequenceNumber": 4995022
}

Currently I am afraid there is no way to add properties to an outgoing message using the EventHub output bindings.
The workaround is using EventHub SDK inside the function.
Reference:
Microsoft Azure SDK for Event Hubs(Python)