I want to create an external function that can be used to upsert rows into MongoDB. I've created the function, tested it locally using Postman and after publishing. I've followed the documentation from https://docs.snowflake.com/en/sql-reference/external-functions-creating-azure-ui.html and at first, I used the javascript function they proposed to test and worked. However, when I run it it python I get an error. This is the code.
import logging
import azure.functions as func
import pymongo
import json
import os
from datetime import datetime
cluster = pymongo.MongoClient(os.environ['MongoDBConnString'])
db = cluster[f"{os.environ['MongoDB']}"]
collection = db[f"{os.environ['MongoDBCollection']}"]
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.")
else:
collection.update_one(
filter={
'_id':req_body['_id']
},
update={
'$set': {'segment_ids': req_body['segment_ids']}
},
upsert=True)
return func.HttpResponse(
json.dumps({"status_code": 200,
"status_message": "Upsert Success",
"Timestamp": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%S"),
"_id": req_body['_id']}),
status_code=200,
mimetype="text/plain"
)
The error states that req_body is referenced before being defined, failing at line '_id':req_body['_id']. In Snowflake I've created an external function called mongoUpsert(body variant) and I am parsing a simple query to test.
select mongoUpsert(object_construct('_id', 'someuuid', 'segment_ids;, array_construct(1,2,3,4)))
From what I can tell, the function is not receiving the body I'm parsing in Snowflake for some reason. I don't know what I am doing wrong. Can anyone help me? Can anyone also explain how Snowflake is sending the parameters (as body, params, headers) and is there a way to specify if I want to parse a body or params?
External functions send and receive data in a particular format. All the parameters are sent in the request body.
https://docs.snowflake.com/en/sql-reference/external-functions-data-format.html
You can checkout snowflake-labs
for external functions samples.
There is one specifically for Azure Python functions that calls the Translator API.
I've started from scratch and stripped the layers one by one in Snowflake. So the Snowflake parameter is parsed to the body of the function but wrapped in an array which is then wrapped in another object called 'data'. Furthermore, it expects the same schema as a response back. So here's below the template to use for Azure Functions when using Python.
import logging
import azure.functions as func
import json
def main(req: func.HttpRequest) -> func.HttpResponse:
# Get body response from Snowflake
req_body = req.get_json()['data'][0][1]
###### Do Something
# Return Response
message = {"Task": "Completed"}
return func.HttpResponse(
json.dumps({'data': [[0, message]]}),
status_code=200)
As an example, I've used a simple JSON object:
{
"_id": "someuuid"
}
And created an external function in Snowflake called testfunc(body variant) and called it using select testfunc(object_construct('_id', 'someuuid')).
If you would log the response (using logging.info(req.get_json())) it would print the following
{
"data":
[
[
0,
{
"_id": "someuuid"
}
]
]
}
So to get the clean input I fed in snowflake I have the line
req_body = req.get_json()['data'][0][1]
However, I kept getting errors on the response until I tried just echoing the input and noticed it returned it without the wrapping. The returned body needs to be a string (hence why using json.dumps()) but it also needs the wrapping. So to print it out, first define a message you want (it may be a calculation of the input or an acknowledgement), then wrap the message in {'data': [[0, message]]} and finally compile it as a string (json.dumps())
Related
Currently developing an python backend API that will increment a value in COSMOS DB(azure serverless database) that is currently running NOSQL. Permissions(output/input) bindings is correct. However I'm able to update the cosmos NOSQL document staticky like this:
`
import azure.functions as func
import json
import logging
def main(req: func.HttpRequest, inputDocument: func.DocumentList, outputDocument: func.Out[func.DocumentList]) ->str:
logging.info('Python HTTP trigger function processed a request.')
name = req.route_params.get('name')
for name in inputDocument:
logging.info(name.to_json())
if name:
newdocs = func.DocumentList()
visitors = {"id": "VISITOR", "testpartitionkey": "testpartitionkey", "visitor": 12}
key = "visitor"
if key in visitors:
visitors[key] +=1
newdocs_load_json= json.dumps(visitors)
newdocs.append(func.Document.from_json(newdocs_load_json))
outputDocument.set(newdocs)
return func.HttpResponse(f"Hello, new value is: {name} {newdocs} {newdocs_load_json} . ....")
`
Document is retrieved from a different function reading the specific document, I'm only getting as far as incrementing the correct document only once, but as Python code is static, I need to somewhat fetch the value of the ID and PartitionkeyAPI response that states correct value from Cosmos DB.
After I change python code to following, I'm just getting a default error 500 message. I have been struggeling with this one for a long time, and can't find another way around to increment a value in a JSON document.
Python code I'm trying with:
import azure.functions as func
import json
import logging
def main(req: func.HttpRequest, inputDocument: func.DocumentList, outputDocument: func.Out[func.DocumentList]) ->str:
logging.info('Python HTTP trigger function processed a request.')
name = req.route_params.get('name')
for name in inputDocument:
logging.info(name.to_json())
if name:
newdocs = func.DocumentList()
for user in inputDocument:
user_json = {
"id": "VISITOR",
"testpartitionkey": "testpartitionkey",
"visitor": user['visitor']
}
key = "visitor"
if key in user_json:
user_json[key] +=1
newdocs_load_json= json.dumps(user_json)
newdocs.append(func.Document.from_json(newdocs_load_json))
outputDocument.set(newdocs)
return func.HttpResponse(f"Hello, new value is: {name} {newdocs} {newdocs_load_json} . ....")
When checking logs in Azure function, this is what's it stating, it's not able to find the key I would like to update cannot be founderror message received in azure functions
Any ideas how to approach this?
I did change in the code as it stated below here, but not sure if I'm approaching this correct. In short terms I'm expecting this to update an document in a nosql db.
I am working with an app that sends data to a server with a POST request,
POST https://www.somedomain.com//sendImage HTTP/2.0
looking like this:
{
"user": {
"consent": true,
"currentNumberIs": 1,
"images": {
"data": "BASE64ENCODED IMAGE",
"docType": "avatar"
},
"totalNumberOfImages": 1
}
}
I want to replace the data part of this Json, but only if the docType is avatar. Trying to use a python script for that, that I found here and edited:
def response(flow: http.HTTPFlow) -> None:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
Launched mitmproxy with -s script.py, but according to the web console, the specific request does not trigger the script at all. Which kinda limits the scope to debug.
Would glady appreciate any help.
As #nneonneo mentioned in the comments, I would first recommend to make extensive use of mitmproxy.ctx.log() to make sure that your event hook is triggered properly. Second, if I understand things correctly, you intend to modify the request and not the response? If you want to modify request contents before they are sent to the server, you need to use the request hook and not the response hook:
def request(flow: http.HTTPFlow) -> None:
# this is executed after we have received the request
# from the client, but before it is sent to the server.
def response(flow: http.HTTPFlow) -> None:
# this is executed after we have sent the request
# to the server and received the response at the proxy.
Finally, you currently read from flow.request.text and then later assign to flow.response.text. I don't know your specific use case, but usually that should be flow.request.text as well.
You're altering the flow variable in a function, but not using the edited flow. If you return the new flow you can then use it and post it.
def response(flow: http.HTTPFlow) -> http.HTTPFlow:
if "somedomain.com" in flow.request.pretty_url:
request_data = json.loads(flow.request.get_text())
if request_data["user"]["images"]["docType"] == "avatar":
data = json.loads(flow.response.get_text())
data["user"]["images"]["data"] = "NEWDATA"
flow.response.text = json.dumps(data)
return flow
I am very new to Azure Function Apps and OAuth so please bear with me.
My Setup
I have an Azure Function App with a simple python-function doing nothing else but printing out the request headers:
import logging
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
aadIdToken = req.headers.get('X-MS-TOKEN-AAD-ID-TOKEN')
aadAccessToken = req.headers.get('X-MS-TOKEN-AAD-ACCESS-TOKEN')
principalID = req.headers.get('X-MS-CLIENT-PRINCIPAL-ID')
principalName = req.headers.get('X-MS-CLIENT-PRINCIPAL-NAME')
idProviderId = req.headers.get('X-MS-CLIENT-PRINCIPAL-IDP')
aadRefreshToken = req.headers.get('X-MS-TOKEN-AAD-REFRESH-TOKEN')
clientPrincipal = req.headers.get('X-MS-CLIENT-PRINCIPAL')
result = "\n"
myDict = sorted(dict(req.headers))
for key in myDict:
result += f"{key} = {dict(req.headers)[key]}\n"
return func.HttpResponse(
f"Hello, {name}. How are you ? Doing well ?"\
f"\n\nHere is some data concerning your Client principal:"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-ID: {principalID}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-NAME: {principalName}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-IDP: {idProviderId}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL: {clientPrincipal}"\
f"\n\nHere is some data concerning your AAD-token:"\
f"\nThis is your X-MS-TOKEN-AAD-ID-TOKEN: {aadIdToken}"\
f"\nThis is your X-MS-TOKEN-AAD-ACCESS-TOKEN: {aadAccessToken}"\
f"\nThis is your X-MS-TOKEN-AAD-REFRESH-TOKEN: {aadRefreshToken}"\
f"\n\n\nresult: {result}"\
)
else:
return func.HttpResponse(
"This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
status_code=200
)
I followed this guide to let the user authenticate via EasyAuth before calling the function.
This seems to work fine. When accessing the function via browser I am redirected to sign-in. After successful sign-in I am then redirected again and the HTTP response is printed out in the browser. As I am able to access X-MS-CLIENT-PRINCIPAL-ID and X-MS-CLIENT-PRINCIPAL-NAME I suppose the authentication was successful. However when printing out the whole request header I did not find a X-MS-TOKEN-AAD-REFRESH-TOKEN, X-MS-TOKEN-AAD-ACCESS-TOKEN or X-MS-TOKEN-AAD-ID-TOKEN.
This is the output (output too large; below the output shown in the screenshot I can see the header content):
First half of my output
My question
What I am trying to do now is to access the groups assigned to the logged-in user via the python code of the function to further authorize his request (e.g. "user can only execute the function when group xyz is assigned, else he will be prompted 'not allowed'").
To achieve this I added the "groups"-claim to the Token Configuration of my App Registration.
From what I understand accessing the user groups via a function coded in .NET is easily possible by using the ClaimsPrinciple object (source).
How would I be able to access the user assigned groups via python code?
Is that possible?
Am I understanding something completely wrong?
Followup:
One thing that I do not understand by now, is that I can see an id_token in the callback-http-request of the browser-debuggger when accessing the function via browser for the first time (to trigger sign in):
Browser debugger: id_token in callback-request
When I decrypted that token using jwt.io I was able to see some IDs of assigned user groups which seems to be exactly what I want to access via the python code.
Re-loading the page (I suppose the request then uses the already authenticated browser session) makes the callback disappear.
The header X-MS-CLIENT-PRINCIPAL contains the same claims as the id_token. So if we want to get the group claim, we can base64 decode the header.
For example
My code
import logging
import azure.functions as func
import base64
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
aadAccessToken = req.headers.get('X-MS-TOKEN-AAD-ACCESS-TOKEN')
principalID = req.headers.get('X-MS-CLIENT-PRINCIPAL-ID')
principalName = req.headers.get('X-MS-CLIENT-PRINCIPAL-NAME')
idProviderId = req.headers.get('X-MS-CLIENT-PRINCIPAL-IDP')
aadRefreshToken = req.headers.get('X-MS-TOKEN-AAD-REFRESH-TOKEN')
clientPrincipal = req.headers.get('X-MS-CLIENT-PRINCIPAL')
clientPrincipal= base64.b64decode(clientPrincipal)
result = "\n"
myDict = sorted(dict(req.headers))
for key in myDict:
result += f"{key} = {dict(req.headers)[key]}\n"
return func.HttpResponse(
f"Hello, {name}. How are you ? Doing well ?"\
f"\n\nHere is some data concerning your Client principal:"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-ID: {principalID}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-NAME: {principalName}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL-IDP: {idProviderId}"\
f"\nThis is your X-MS-CLIENT-PRINCIPAL: {clientPrincipal}"\
f"\n\nHere is some data concerning your AAD-token:"\
f"\nThis is your X-MS-TOKEN-AAD-ID-TOKEN: {aadIdToken}"\
f"\nThis is your X-MS-TOKEN-AAD-ACCESS-TOKEN: {aadAccessToken}"\
f"\nThis is your X-MS-TOKEN-AAD-REFRESH-TOKEN: {aadRefreshToken}"\
f"\n\n\nresult: {result}"\
)
else:
return func.HttpResponse(
"This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
status_code=200
)
I have been working on trying to write an Azure Function that does the following:
A client/browser makes a request to https://foo.azurewebsites.com/$A/$B. An example could look like http://foo.azurewebsites.com/john/30vjs0-s9fjs-fnsne.
$A = one of up to 20 possible dictionary words.
$B = unknown strings, but the strings need to be passed onto the secondary server too.
I would like to redirect that entire HTTP request (GET and POST ideally) to another HTTP server (outside Az).
I understand I need to make separate functions for each one of $A, (each with effectively the same code inside) to redirect it to the other HTTP server. However I can't seem to figure out how to parse the URL correctly in Python within the Az Functions modules. It is dealing with the unknown variables in $B that is causing the problem I believe, as I can't hardcode them as I cannot predict them. Az Functions currently doesn't pass anything that doesn't exactly match the /functionName argument - but in my use case there will always be a $B.
So https://foo.azurewebsites.com/john will be forwarded to my seocondary server, but https://foo.azurewebsites.com/john/test.txt appears to be being dropped by Az.
My Python to forward on GET and POST is here (repeated for each function in the list of $A): I have been experiementing mainly with GET requests.
def main(req: func.HttpRequest) -> func.HttpResponse:
if req.method == "GET":
return get(req)
if req.method == "POST":
return post(req)
def post(req: func.HttpRequest) -> func.HttpResponse:
header_dict = {}
get_url = 'https://secondaryserver.com'
for key, value in dict(req.headers).items():
header_dict.update({key : value})
post_data = req.get_body()
request = urllib.request.Request(get_url, data=post_data, headers=header_dict)
with urllib.request.urlopen(request) as response:
html = response.read()
return func.HttpResponse(html)
def get(req: func.HttpRequest) -> func.HttpResponse:
suffix = req.url
get_url = 'https://secondaryserver.com'
final_url = get_url + suffix
header_dict = {}
print(final_url)
for key, value in dict(req.headers).items():
header_dict.update({key : value})
request = urllib.request.Request(final_url, headers=header_dict)
with urllib.request.urlopen(request) as response:
html = response.read()
return func.HttpResponse(html)
I have also investigated this article from Microsoft, trying "route": {url} in my function.json and then referencing it in the Python init.py but was unsuccessful (and Az reported the trigger URLS as %7BURL%7B
I guess a key question is 'is this actually possible on Az Functions - to pass parameters AFTER the name of a function within an app, and have that able to be referenced in Python code?
Thank you for any help you might be able to provide.
If you want to route some request path to the same function, you can define the route in the function.json. For more details, please refer to here.
For example. I want to route https://{}/$A/$B to the same function. We should define the function.json as below
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": ["get", "post"],
"route": "{url}/{url1}"
},
{
"type": "http",
"direction": "out",
"name": "$return"
}
]
}
Test
My function code
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
return func.HttpResponse(
req.url,
status_code=200
)
Update
If you want to forward the request to another HTTP server, you can refer to the following code
def main(req: func.HttpRequest) -> func.HttpResponse:
if req.method == "GET":
# send you request
if req.method == "POST":
# send you request
To redirect a URL from Azure Functions (python), use the following return statement with a status code of 302 with the new redirection URL stored in RedirectionURL variable
return func.HttpResponse(RedirectionURL,headers={'Location': RedirectionURL},status_code=302)
I have created one HttpTrigger Azure functions using python. But I want to create another one HttpTrigger Azure Functions using python in the same project. In this application I want to execute first HttpTrigger Azure functions after that second HttpTrigger Azure functions execute. How can I implement that?
Because in Python Azure Functions there is no Durable Functions. That's why I am not able to understood how can I execute one after another azure function executes.
Assumed there are two HttpTrigger functions HttpTriggerA and HttpTriggerB, a direct solution in my mind is to request the public url of HttpTriggerA from HttpTriggerB function via a HTTP client request using requests, as the code below in my first version of HttpTriggerB code.
import logging
import azure.functions as func
import requests
from urllib.parse import urlparse
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name') or 'Peter Pan'
codeA = req.params.get('codeA') or ''
o = urlparse(req.url)
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
print("B be invoked.")
resp = requests.get(f"{o.scheme}://{o.netloc}/api/HttpTriggerA?name={name}&code={codeA}")
return func.HttpResponse(f"Hello {name}! from B {resp.status_code == 200 and resp.text or ''}")
else:
return func.HttpResponse(
"Please pass a name on the query string or in the request body",
status_code=400
)
However, it will not work after I test the code above on local or on Azure. It will hang at the code line resp = requests.get(f"{o.scheme}://{o.netloc}/api/HttpTriggerA?name={name}&code={codeA}"), when I access the url http(s)://<the host of local or Azure>:<7071 or 80>/api/HttpTriggerB?name=Peter%20Pan&code=<code for HttpTriggerB>&codeA=<code for HttpTriggerA>. The reason for the hang issue seems to be caused by functions running in singleton or in single thread.
So I switched to the other solution to use Ajax request from the html content of HttpTriggerB. It works as I wish as the figure below.
Here is my code for HttpTriggerB function, the HttpTriggerA function is simply generated by func new.
import logging
import azure.functions as func
#import requests
from urllib.parse import urlparse
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name') or 'Peter Pan'
codeA = req.params.get('codeA') or ''
o = urlparse(req.url)
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
print("B be invoked.")
#resp = requests.get(f"{o.scheme}://{o.netloc}/api/httptriggera?name={name}&code={req.params.get('codeA')}")
html = """
<script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
Hello """+name+"""! from B<br/>
<span id="A"></span>
<script>
$.when(
$.get('"""+o.scheme+'://'+o.netloc+'/api/HttpTriggerA?name='+name+'&code='+codeA+"""'),
$.ready
).done(function( data ) {
$( "#A" ).html( data[0] );
});
</script>
"""
return func.HttpResponse(html, mimetype="text/html")
else:
return func.HttpResponse(
"Please pass a name on the query string or in the request body",
status_code=400
)
Hope it helps.