Currently developing an python backend API that will increment a value in COSMOS DB(azure serverless database) that is currently running NOSQL. Permissions(output/input) bindings is correct. However I'm able to update the cosmos NOSQL document staticky like this:
`
import azure.functions as func
import json
import logging
def main(req: func.HttpRequest, inputDocument: func.DocumentList, outputDocument: func.Out[func.DocumentList]) ->str:
logging.info('Python HTTP trigger function processed a request.')
name = req.route_params.get('name')
for name in inputDocument:
logging.info(name.to_json())
if name:
newdocs = func.DocumentList()
visitors = {"id": "VISITOR", "testpartitionkey": "testpartitionkey", "visitor": 12}
key = "visitor"
if key in visitors:
visitors[key] +=1
newdocs_load_json= json.dumps(visitors)
newdocs.append(func.Document.from_json(newdocs_load_json))
outputDocument.set(newdocs)
return func.HttpResponse(f"Hello, new value is: {name} {newdocs} {newdocs_load_json} . ....")
`
Document is retrieved from a different function reading the specific document, I'm only getting as far as incrementing the correct document only once, but as Python code is static, I need to somewhat fetch the value of the ID and PartitionkeyAPI response that states correct value from Cosmos DB.
After I change python code to following, I'm just getting a default error 500 message. I have been struggeling with this one for a long time, and can't find another way around to increment a value in a JSON document.
Python code I'm trying with:
import azure.functions as func
import json
import logging
def main(req: func.HttpRequest, inputDocument: func.DocumentList, outputDocument: func.Out[func.DocumentList]) ->str:
logging.info('Python HTTP trigger function processed a request.')
name = req.route_params.get('name')
for name in inputDocument:
logging.info(name.to_json())
if name:
newdocs = func.DocumentList()
for user in inputDocument:
user_json = {
"id": "VISITOR",
"testpartitionkey": "testpartitionkey",
"visitor": user['visitor']
}
key = "visitor"
if key in user_json:
user_json[key] +=1
newdocs_load_json= json.dumps(user_json)
newdocs.append(func.Document.from_json(newdocs_load_json))
outputDocument.set(newdocs)
return func.HttpResponse(f"Hello, new value is: {name} {newdocs} {newdocs_load_json} . ....")
When checking logs in Azure function, this is what's it stating, it's not able to find the key I would like to update cannot be founderror message received in azure functions
Any ideas how to approach this?
I did change in the code as it stated below here, but not sure if I'm approaching this correct. In short terms I'm expecting this to update an document in a nosql db.
Related
I'm struggling to find documentation and examples for Python Client for BigQuery Data Transfer Service. A new query string is generated by my application from time to time and I'd like to update the existing scheduled query accordingly. This is the most helpful thing I have found so far, however I am still unsure where to pass my query string. Is this the correct method?
from google.cloud import bigquery_datatransfer_v1
def sample_update_transfer_config():
# Create a client
client = bigquery_datatransfer_v1.DataTransferServiceClient()
# Initialize request argument(s)
transfer_config = bigquery_datatransfer_v1.TransferConfig()
transfer_config.destination_dataset_id = "destination_dataset_id_value"
request = bigquery_datatransfer_v1.UpdateTransferConfigRequest(
transfer_config=transfer_config,
)
# Make the request
response = client.update_transfer_config(request=request)
# Handle the response
print(response)
You may refer to this Update Scheduled Queries for python documentation from BigQuery for the official reference on the usage of Python Client Library in updating scheduled queries.
However, I updated the code for you to update your query string. I added the updated query string in the params and define what attributes of the TransferConfig() will be updated in the update_mask.
See updated code below:
from google.cloud import bigquery_datatransfer
from google.protobuf import field_mask_pb2
transfer_client = bigquery_datatransfer.DataTransferServiceClient()
transfer_config_name = "projects/{your-project-id}/locations/us/transferConfigs/{unique-ID-of-transferconfig}"
new_display_name = "Your Desired Updated Name if Necessary" #--remove if no need to update **scheduled query name**.
query_string_new = """
SELECT
CURRENT_TIMESTAMP() as current_time
"""
new_params={
"query": query_string_new,
"destination_table_name_template": "your_table_{run_date}",
"write_disposition": "WRITE_TRUNCATE",
"partitioning_field": "",
}
transfer_config = bigquery_datatransfer.TransferConfig(name=transfer_config_name,
)
transfer_config.display_name = new_display_name #--remove if no need to update **scheduled query name**.
transfer_config.params = new_params
transfer_config = transfer_client.update_transfer_config(
{
"transfer_config": transfer_config,
"update_mask": field_mask_pb2.FieldMask(paths=["display_name","params"]), #--remove "display_name" from the list if no need to update **scheduled query name**.
}
)
print("Updates are executed successfully")
For you to get the value of your transfer_config_name, you may list all your scheduled queries by following this SO post.
I have a successfully compiled and run a django rest consuming cocktaildb api. On local server when I run http://127.0.0.1:8000/api/ I get
{
"ingredients": "http://127.0.0.1:8000/api/ingredients/",
"drinks": "http://127.0.0.1:8000/api/drinks/",
"feeling-lucky": "http://127.0.0.1:8000/api/feeling-lucky/"
}
But when I go to one of the links mentioned in the json result above, for example:
http://127.0.0.1:8000/api/ingredients/
I get an empty [] with a status 200OK!
I need an endpoint to GET drinks and ingredients before I can destructure to specific details using angular.
I implemented helper folder in the app with the the API function as below:
class TheCoctailDBAPI:
THECOCTAILDB_URL = 'https://www.thecocktaildb.com/api/json/v1/1/'
async def __load_coctails_for_drink(self, drink, session):
for i in range(1, 16):
ingredientKey = 'strIngredient' + str(i)
ingredientName = drink[ingredientKey]
if not ingredientName:
break
if ingredientName not in self.ingredients:
async with session.get(f'{TheCoctailDBAPI.THECOCTAILDB_URL}search.php?i={ingredientName}') \
as response:
result = json.loads(await response.text())
self.ingredients[ingredientName] = result['ingredients'][0]
What was your expected responce?
Add the function that is called by this API as well as the DB settings in the question, so that we can properly help you.
Are you sure that you are connecting and pulling data from a remote location? It looks to me like your local DB is empty, so the API has no data to return.
I'd like to make a python Azure Function App (web API) to process a queue of tasks. I already setup some trigger that call this API whenever a task is inserted into the queue. As this API will process all of the current tasks in the queue, I would like to prevent the API to execute if there is other execution of this API at the time, to avoid processing conflicts.
I think of using a database locking mechanism but it doesn't look so elegant. Is there any singleton design pattern that can used in Python Azure function App for this purpose? Thanks.
I found a way to solve this problem using Azure Durable function. There are 3 types of functions in an Azure Durable Function app: Orchestration Client function, Orchestrator function, Activity functions. I just need to add some checking steps in the Orchestration Client function like the following example:
# This function an HTTP starter function for Durable Functions.
import logging
import azure.functions as func
import azure.durable_functions as df
def is_finished(runtime_status : df.models.OrchestrationRuntimeStatus):
result = False
if runtime_status is None or \
runtime_status in [df.OrchestrationRuntimeStatus.Canceled,
df.OrchestrationRuntimeStatus.Completed,
df.OrchestrationRuntimeStatus.Failed,
df.OrchestrationRuntimeStatus.Terminated]:
result = True
return result
async def main(req: func.HttpRequest, starter: str) -> func.HttpResponse:
client = df.DurableOrchestrationClient(starter)
# general azure function url : http://<APP_NAME>.azurewebsites.net/api/<FUNCTION_NAME>
# function.json -> "route": "orchestrators/{functionName}/{instanceId}"
orchestrator_instance_id = req.route_params['instanceId']
function_name = req.route_params['functionName']
INVENSYNC_ORCHESTRATOR_INSTANCE_ID = '117610EF-BC37-4E31-BFA4-205EBB3CC54E' # just select any key
if orchestrator_instance_id == INVENSYNC_ORCHESTRATOR_INSTANCE_ID:
existing_instance_status = await client.get_status(orchestrator_instance_id)
logging.info(f"InventorySyncHttpStart() - existing_instance_status = '{existing_instance_status}'.")
if existing_instance_status is None or \
is_finished(existing_instance_status.runtime_status):
logging.info(f"InventorySyncHttpStart() - existing_instance_status.runtime_status = '{existing_instance_status.runtime_status}'.")
orchestrator_instance_id = await client.start_new(function_name, orchestrator_instance_id)
logging.info(f"Started orchestration with ID = '{orchestrator_instance_id}'.")
result = client.create_check_status_response(req, orchestrator_instance_id)
else:
result = func.HttpResponse(status_code=409, body=f"An instance with ID '{orchestrator_instance_id}' already exists")
else:
result = func.HttpResponse(status_code=406, body=f"Invalid Instance ID '{orchestrator_instance_id}' in URL")
return result
I am very new to Azure Function Apps and OAuth so please bear with me.
My Setup
I have an Azure Function App with a simple python-function doing nothing else but printing out the request headers:
import logging
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
aadIdToken = req.headers.get('X-MS-TOKEN-AAD-ID-TOKEN')
aadAccessToken = req.headers.get('X-MS-TOKEN-AAD-ACCESS-TOKEN')
principalID = req.headers.get('X-MS-CLIENT-PRINCIPAL-ID')
principalName = req.headers.get('X-MS-CLIENT-PRINCIPAL-NAME')
idProviderId = req.headers.get('X-MS-CLIENT-PRINCIPAL-IDP')
aadRefreshToken = req.headers.get('X-MS-TOKEN-AAD-REFRESH-TOKEN')
clientPrincipal = req.headers.get('X-MS-CLIENT-PRINCIPAL')
result = "\n"
myDict = sorted(dict(req.headers))
for key in myDict:
result += f"{key} = {dict(req.headers)[key]}\n"
return func.HttpResponse(
f"Hello, {name}. How are you ? Doing well ?"\
f"\n\nHere is some data concerning your Client principal:"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-ID: {principalID}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-NAME: {principalName}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-IDP: {idProviderId}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL: {clientPrincipal}"\
f"\n\nHere is some data concerning your AAD-token:"\
f"\nThis is your X-MS-TOKEN-AAD-ID-TOKEN: {aadIdToken}"\
f"\nThis is your X-MS-TOKEN-AAD-ACCESS-TOKEN: {aadAccessToken}"\
f"\nThis is your X-MS-TOKEN-AAD-REFRESH-TOKEN: {aadRefreshToken}"\
f"\n\n\nresult: {result}"\
)
else:
return func.HttpResponse(
"This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
status_code=200
)
I followed this guide to let the user authenticate via EasyAuth before calling the function.
This seems to work fine. When accessing the function via browser I am redirected to sign-in. After successful sign-in I am then redirected again and the HTTP response is printed out in the browser. As I am able to access X-MS-CLIENT-PRINCIPAL-ID and X-MS-CLIENT-PRINCIPAL-NAME I suppose the authentication was successful. However when printing out the whole request header I did not find a X-MS-TOKEN-AAD-REFRESH-TOKEN, X-MS-TOKEN-AAD-ACCESS-TOKEN or X-MS-TOKEN-AAD-ID-TOKEN.
This is the output (output too large; below the output shown in the screenshot I can see the header content):
First half of my output
My question
What I am trying to do now is to access the groups assigned to the logged-in user via the python code of the function to further authorize his request (e.g. "user can only execute the function when group xyz is assigned, else he will be prompted 'not allowed'").
To achieve this I added the "groups"-claim to the Token Configuration of my App Registration.
From what I understand accessing the user groups via a function coded in .NET is easily possible by using the ClaimsPrinciple object (source).
How would I be able to access the user assigned groups via python code?
Is that possible?
Am I understanding something completely wrong?
Followup:
One thing that I do not understand by now, is that I can see an id_token in the callback-http-request of the browser-debuggger when accessing the function via browser for the first time (to trigger sign in):
Browser debugger: id_token in callback-request
When I decrypted that token using jwt.io I was able to see some IDs of assigned user groups which seems to be exactly what I want to access via the python code.
Re-loading the page (I suppose the request then uses the already authenticated browser session) makes the callback disappear.
The header X-MS-CLIENT-PRINCIPAL contains the same claims as the id_token. So if we want to get the group claim, we can base64 decode the header.
For example
My code
import logging
import azure.functions as func
import base64
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
aadAccessToken = req.headers.get('X-MS-TOKEN-AAD-ACCESS-TOKEN')
principalID = req.headers.get('X-MS-CLIENT-PRINCIPAL-ID')
principalName = req.headers.get('X-MS-CLIENT-PRINCIPAL-NAME')
idProviderId = req.headers.get('X-MS-CLIENT-PRINCIPAL-IDP')
aadRefreshToken = req.headers.get('X-MS-TOKEN-AAD-REFRESH-TOKEN')
clientPrincipal = req.headers.get('X-MS-CLIENT-PRINCIPAL')
clientPrincipal= base64.b64decode(clientPrincipal)
result = "\n"
myDict = sorted(dict(req.headers))
for key in myDict:
result += f"{key} = {dict(req.headers)[key]}\n"
return func.HttpResponse(
f"Hello, {name}. How are you ? Doing well ?"\
f"\n\nHere is some data concerning your Client principal:"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-ID: {principalID}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-NAME: {principalName}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-IDP: {idProviderId}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL: {clientPrincipal}"\
f"\n\nHere is some data concerning your AAD-token:"\
f"\nThis is your X-MS-TOKEN-AAD-ID-TOKEN: {aadIdToken}"\
f"\nThis is your X-MS-TOKEN-AAD-ACCESS-TOKEN: {aadAccessToken}"\
f"\nThis is your X-MS-TOKEN-AAD-REFRESH-TOKEN: {aadRefreshToken}"\
f"\n\n\nresult: {result}"\
)
else:
return func.HttpResponse(
"This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
status_code=200
)
I want to create an external function that can be used to upsert rows into MongoDB. I've created the function, tested it locally using Postman and after publishing. I've followed the documentation from https://docs.snowflake.com/en/sql-reference/external-functions-creating-azure-ui.html and at first, I used the javascript function they proposed to test and worked. However, when I run it it python I get an error. This is the code.
import logging
import azure.functions as func
import pymongo
import json
import os
from datetime import datetime
cluster = pymongo.MongoClient(os.environ['MongoDBConnString'])
db = cluster[f"{os.environ['MongoDB']}"]
collection = db[f"{os.environ['MongoDBCollection']}"]
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.")
else:
collection.update_one(
filter={
'_id':req_body['_id']
},
update={
'$set': {'segment_ids': req_body['segment_ids']}
},
upsert=True)
return func.HttpResponse(
json.dumps({"status_code": 200,
"status_message": "Upsert Success",
"Timestamp": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%S"),
"_id": req_body['_id']}),
status_code=200,
mimetype="text/plain"
)
The error states that req_body is referenced before being defined, failing at line '_id':req_body['_id']. In Snowflake I've created an external function called mongoUpsert(body variant) and I am parsing a simple query to test.
select mongoUpsert(object_construct('_id', 'someuuid', 'segment_ids;, array_construct(1,2,3,4)))
From what I can tell, the function is not receiving the body I'm parsing in Snowflake for some reason. I don't know what I am doing wrong. Can anyone help me? Can anyone also explain how Snowflake is sending the parameters (as body, params, headers) and is there a way to specify if I want to parse a body or params?
External functions send and receive data in a particular format. All the parameters are sent in the request body.
https://docs.snowflake.com/en/sql-reference/external-functions-data-format.html
You can checkout snowflake-labs
for external functions samples.
There is one specifically for Azure Python functions that calls the Translator API.
I've started from scratch and stripped the layers one by one in Snowflake. So the Snowflake parameter is parsed to the body of the function but wrapped in an array which is then wrapped in another object called 'data'. Furthermore, it expects the same schema as a response back. So here's below the template to use for Azure Functions when using Python.
import logging
import azure.functions as func
import json
def main(req: func.HttpRequest) -> func.HttpResponse:
# Get body response from Snowflake
req_body = req.get_json()['data'][0][1]
###### Do Something
# Return Response
message = {"Task": "Completed"}
return func.HttpResponse(
json.dumps({'data': [[0, message]]}),
status_code=200)
As an example, I've used a simple JSON object:
{
"_id": "someuuid"
}
And created an external function in Snowflake called testfunc(body variant) and called it using select testfunc(object_construct('_id', 'someuuid')).
If you would log the response (using logging.info(req.get_json())) it would print the following
{
"data":
[
[
0,
{
"_id": "someuuid"
}
]
]
}
So to get the clean input I fed in snowflake I have the line
req_body = req.get_json()['data'][0][1]
However, I kept getting errors on the response until I tried just echoing the input and noticed it returned it without the wrapping. The returned body needs to be a string (hence why using json.dumps()) but it also needs the wrapping. So to print it out, first define a message you want (it may be a calculation of the input or an acknowledgement), then wrap the message in {'data': [[0, message]]} and finally compile it as a string (json.dumps())