I'm using a queue trigger to pass in some data about a job that I want to run with Azure Functions(I'm using python). Part of the data is the name of a file that I want to pull from blob storage. Because of this, declaring a file path/name in an input binding doesn't seem like the right direction, since the function won't have the file name until it gets the queue trigger.
One approach I've tried is to use the azure-storage sdk, but I'm unsure of how to handle authentication from within the Azure Function.
Is there another way to approach this?
In Function.json, The blob input binding can refer to properties from the queue payload. The queue payload needs to be a JSON object
Since this is function.json, it works for all languages.
See official docs at https://learn.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings
For example, in you function.json,
{
"name": "imageSmall",
"type": "blob",
"path": "container/{filename}",
}
And if your queue message payload is:
{
"filename" : "myfilename"
}
Then the {filename} token in the blob's path expression will get substituted.
Typically, you store connection strings / account keys in App Settings of the Function App, and then read them by accessing environment variables. I haven't used python in Azure, but I believe that looks like
connection = open(os.environ['ConnectionString']).read()
I've found one example of python function which does what you ask for: queue trigger + blob operation.
Storing secrets can (also) be done using App Settings.
In Azure, go to your Azure Functions App Service, Then click "Application Settings". Then, scroll down to the "App Settings" list. This list consists of Key-Value pairs. Add your key, for example MY_CON_STR and the actual connection string as the value.
Don't forget to click save at this point
Now, in your application (your Function for this example), you can load the stored value using its key. For example, in python, you can use:
os.environ['MY_CON_STR']
Note that since the setting isn't saved locally, you have to execute it from within Azure. Unfortunately, Azure Functions applications do not contain a web.config file.
Related
I am sending some events to mixpanel from within a cloud function, using the Python SDK. I don't want the users' location to be set to the location of the cloud server. I have read the Mixpanel article referencing this, but the documentation only shows how to ignore IP for a people_set call, using the meta argument. I assumed the same logic would translate to the track call, as it also has the meta argument in its documentation.
After testing, the people_set call is not taking the server location, but the track call is still taking the server location. Does anyone have any ideas why this might be, or how to correctly go about this task for a track() call? Below are the code snippets for the two calls:
mp_eu.people_set(user_id, user_data,
meta={'$ignore_time': True, '$ip': 0})
mp_eu.track(user_id, 'event_name', event_data,
meta={'$ignore_time': True, '$ip': 0})
You should add "ip" to "properties".
properties["ip"] = ip
mp_eu.track(user_id, 'event_name', properties)
check this.
https://help.mixpanel.com/hc/en-us/articles/115004499343
I'm building a flask server in python with Cloud Run, for a chatbot to call.
Sometimes if user wants to do something with the chatbot, the bot need ask the user to login to a 3rd party server before doing the things.
I have two routes:
Route 1 is "/login", it returns a simple iframe which will open a login page in a 3rd party server, generate a "session_id", and save some info I already get to a global variable dict called "runtimes" with the "session_id" as key, so that I can use it later when visitor successfully logged in.
Route 2 is "/callback/<session_id>". After user successfully login to its account, the 3rd party server will call this route with a token in url parameters. Then I will use the "session_id" to read the saved info from "runtimes", and do later things.
It works well in my local machine. But in Google Cloud Run, because it support multiple instances, sometimes it will trigger a new instance when server calls "callback", so it cannot get the "runtime" because they are in different instances.
I know that I can save the runtimes dict to a database to solve this problem, but it looks too overkill...Just not seem right.
Is there any easy way that I can make the "runtimes" be shared between instances?
The solution here is to use a central point of storage: database, memorystore, firestore,... something out of Cloud Run itself.
You can also try the Cloud Run execution runtime v2 that allow you to mount a network disk, such as Cloud Storage or Filestore. You can imagine to store the session data in a file which has the name of the session ID.
Note: On Cloud Run side, something is cooking, but it's not 100% safe, it will be a best effort. A database backup will be required even with that new feature
I am trying to deploy a rest api in cloud run where one endpoint launches an async job. The job is defined inside a function in the code.
It seems one way to do it is to use Cloud Task, but this would mean to make a self-call to another endpoint of the deployed api. Specifically, to create an auxiliary endpoint that contains the job code (e.g. /run-my-function) and another one to set the queue to cloud task that launches the /run-my-function?
Is this the right way to do it or I have misunderstand something? In case it's the right way how to specify the url of the /run-my-function endpoint without explicitly hard-code the cloud run deployed uRL name?
The code for the endpoint that launches the endpoint with the run-my-function code would be:
from google.cloud import tasks_v2
client = tasks_v2.CloudTasksClient()
project = 'myproject'
queue = 'myqueue'
location = 'mylocation'
url = 'https://cloudrunservice-abcdefg-ca.b.run.app/run-my-function'
service_account_email = '12345#cloudbuild.gserviceaccount.com'
parent = client.queue_path(project, location, queue)
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
'url': url,
"oidc_token": {"service_account_email": service_account_email},
}
}
response = client.create_task(parent=parent, task=task)
However, this requires to hard-code the service name https://cloudrunservice-abcdefg-ca.b.run.app and to define an auxiliary endpoint /run-my-function that can be called via http
In your code you are able to get the Cloud Run URL without hardcoding it or setting it in an environment variable.
You can have a look to a previous article that I wrote, in the gracefull termison part. I provide a working code in Go, not so difficult to re-implement in Python.
Here the principle:
Get the Region and the project Number from the Metadata server. Keep in mind that Cloud Run has specific metadata like the region
Get the K_SERVICE env var (it's a standard Cloud Run env var)
Perform a call to the Cloud Run Rest API to get the service detail and customize the request with the data got previously
Extract the status.url JSON entry from the response.
Now you have it!
Let me know if you have difficulties to achieve that. I'm not good at Python, but I will be able to write that piece of code!
I've stored some records on my datastore console. Now I wanna manipulate them to optimize Big Data analytics.
How can I write a Python cloud routine to make some transformations on my data? Can I trigger it with Datastore events?
Thanks a lot.
I have coded a little bit myself. You can find my code in GitHub.
What it is doing:
It is an HTTP Cloud Function
Establishing connection to Google Cloud Datastore through client()
Updates a value of specific entry in the entity using ID number and entry's column name
How to use:
Test this Cloud Function and get the idea of how it is working. Then manipulate according to your needs. I have tested this my self and it is working.
Create an HTTP trigger Python Cloud Function.
Set the name to updateDatastore
Copy and paste the code from GitHub.
Add this line google-cloud-datastore to the requirements.txt file.
In main code assign ENTITY_KIND your entity's kind value
In main code assign ENTITY_KEY your entity's key value
When clicked on HTTP trigger URL, after your Cloud Function's execution current time will be written in the column.
I have an app that is meant to integrate with third-party apps. These apps should be able to trigger a function when data changes.
The way I was envisioning this, I would use a node function to safely prepare data for the third parties, and get the url to call from the app's configuration on firestore. I would call that url from the node function, and wait for it to return, updating results as necessary (actually, triggering a push notification). -- these third-party functions would tend to be python functions, so my demo should be in python.
I have the initial node function and firestore setup so that I am currently triggering a ECONNREFUSED -- because I don't know how to set up the third-party function.
Let's say this is the function I need to trigger:
def hello_world(request):
request_json = request.get_json()
if request_json and 'name' in request_json:
name = request_json['name']
else:
name = 'World'
return 'Hello, {}!\n'.format(name)
Do I need to set up a separate gcloud account to host this function, or can I include it in my firestore functions? If so, how do I deploy this to firestore? Typically with my node functions, I am running firebase deploy and it automagically finds my functions from my index.js file.
If you're asking whether Cloud Functions that are triggered by Cloud Firestore can co-exist in a project with Cloud Functions that are triggered by HTTP(S) requests, then the answer is "yes they can". There is no need to set up a separate (Firebase or Cloud) project for each function type.
However: when you deploy your Cloud Functions through the Firebase CLI with firebase deploy, it will remove any functions that it finds in the project, that are not in the code. If you have functions both in Python and in Node.js, there is never a single codebase that contains both, so a blanket deploy would always delete some of your functions. So in that case, you should use the granular deploy option of the Firebase CLI.