I am trying to build an application using Google Cloud Platform AutoML using Python. My overall code flow looks like this:
User Interacts--> data sent to PubSub--> callback invokes my AutoML--> Result
The snippet that calls pubsub looks like this:
blob=blob+bytes(doc_type,'utf-8')
publisher.publish(topic,blob)
future=subscriber.subscribe(subscription,callback=callback)
#flash("The object is "+future,'info')
try:
future.result()
except Exception as ex:
subscriber.close()
In PubSub callback:
def callback(message):
new_message=message.data
display_name,score=predict_value(new_message,"modelID","projectid",'us-central1')
message.ack()
And my predict_value gets the model_id, project id and compute region and performs the prediction.
When I directly call predict_value without using PubSub it is working fine. If I do like this, I am getting the below error:
google.api_core.exceptions.PermissionDenied: 403 Permission 'automl.models.predict' denied on resource 'projects/projectID/locations/us-central1/models/' (or it may not exist).
Please help me to resolve the issue
Thank you so much for all your responses. I have just fixed the issue using the below snippet example
def receive_messages_synchronously(project, subscription_name):
"""Pulling messages synchronously."""
# [START pubsub_subscriber_sync_pull]
# project = "Your Google Cloud Project ID"
# subscription_name = "Your Pubsub subscription name"
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
# Builds a pull request with a specific number of messages to return.
# `return_immediately` is set to False so that the system waits (for a
# bounded amount of time) until at lease one message is available.
response = subscriber.pull(
subscription_path,
max_messages=3,
return_immediately=False)
ack_ids = []
for received_message in response.received_messages:
print("Received: {}".format(received_message.message.data))
ack_ids.append(received_message.ack_id)
# Acknowledges the received messages so they will not be sent again.
subscriber.acknowledge(subscription_path, ack_ids)
# [END pubsub_subscriber_sync_pull]
The reason being the subscription that is created uses the pull request. I guess the callback method concept used is mainly for "push" which may be the reason because I didnt give the endpoint and token to publish the message. Hope what I am guessing is correct. Let me know your views as well.
This is likely due to one of two factors:
invalid credentials being used when sending the request to the AutoML API - it is very likely that pubsub executes in other context and can't get the default credentials
invalid model resource name (make sure it is correct) - it should be something like: "projects/12423534/locations/us-central1/models/23432423"
Related
newbie here. I'm working on the Stripe payment method using flask and it all works well on my local machine but when I deploy my code on the server and listen to webhook events in the stripe dashboard, I get this error"No signatures found matching the expected signature for payload". Already tried so many solutions but nothing worked. Any help will be appreciated.
def webhook_received(self, user_id):
payload = request.data
endpoint_secret = 'my_secret_key'
sig_header = request.headers.get('stripe-signature')
try:
event = stripe.Webhook.construct_event(
json.loads(payload), sig_header, endpoint_secret
)
data = event['data']
except Exception as e:
return str(e)
event_type = event['type']
if event_type == 'checkout.session.completed':
self.handle_checkout_session(data, user_id)
elif event_type == 'invoice.paid':
pass
Okay I think I see the problem but I'll try to cover both potential issues.
(Most Likely): Stripe requires the raw, unmodified request body to form the webhook signature. In your try: block you are using json.loads(payload) which converts it to a Python dict object. Try using the raw payload data instead.
If the problem only occurs when you deploy your code to a remote server then the most likely problem is with the endpoint_secret value. I would add some logging in your webhook_received() function to log the value after it's loaded and make sure the value matches the webhook signing secret you can view in your Stripe dashboard.
Lastly, it's important to return proper responses to avoid webhook delivery retries. I know Flask does some stuff implicitly (a pet peeve of mine) but I'm not seeing a 200 or 500 response being returned here. You'll want to make sure you respond appropriately to avoid headaches later. You can check the best practices here. There's also a handy webhook builder here so you can check your implementation against Stripe's Flask code.
I am working with multiple applications that communicate asynchronously using Kafka. These applications are managed by several departments and contract testing is appropriate to ensure that the messages used during communication follow the expected schema and will evolve according to the contract specification.
It sounded like the pact library for python is a good fit because it helps creating contract tests for HTTP and message integrations.
What I wanted to do is to send an HTTP request and to listen from the appropriate and dedicated Kafka topic immediately after. But it seems that the test is forcing me specify an HTTP code even if what I am expecting is a message from a queue without an HTTP status code. Furthermore, it seems that the HTTP request is being sent before the consumer is listening. Here is some sample code.
from pact.consumer import Consumer as p_Consumer
from pact.provider import Provider as p_Provider
from confluent_kafka import Consumer as k_Consumer
pact = p_Consumer('Consumer').has_pact_with(p_Provider('Provider'))
pact.start_service()
atexit.register(pact.stop_service)
config = {'bootstrap.servers':'server', 'group.id':0, 'auto.offset.reset':'latest'}
consumer = k_consumer(config)
consumer.subscribe(['usertopic'])
def user():
while True:
msg = consumer.poll(timeout=1)
if msg is None:
continue
else:
return msg.value().decode()
class ConstractTesting(unittest.TestCase):
expected = {
'username': 'UserA',
'id':123,
'groups':['Editors']
}
pact.given('UserA exists and is not an administrator')
.upon_receiving('a request for UserA')
.with_request(method='GET',path='/user/')
.will_respond_with(200, body=expected)
with pact:
result = user()
self.assertEqual(result,expected)
How would I carry out contract testing in Python using Kafka? It feels like I am going through a lot of hoops to carry out this test.
With Pact message it's a different API you write tests against. You don't use the standard HTTP one, in fact the transport itself is ignored altogether and it's just the payload - the message - we're interested in capturing and verifying. This allows us to test any queue without having to build specific interfaces for each
See this example: https://github.com/pact-foundation/pact-python/blob/02643d4fb89ff7baad63e6436f6a929256c6bf12/examples/message/tests/consumer/test_message_consumer.py#L65
You can read more about message pact testing here: https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
And finally here are some Kafka examples for other languages that may be helpful: https://docs.pactflow.io/docs/examples/kafka/js/consumer
I'm trying to create a python script that continuously reads mail from a service account in my organization. I'm attempting to use the Microsoft Graph API, but the more I read, the more confused I get. I have registered an app in Azure Portal and have my client id, client secret, etc, then it's my understanding you have to use those, call the API that requires you to paste a url into your browser to log in to consent access, and that provides a token that only lasts an hour? How can I do this programmatically?
I guess my question is, has anyone had any luck doing this with the graph api? How can I do this without having to do the browser handshake every hour? I would like to be able to just run this script and let it run without worrying about needing to refresh a token ever so often. Am I just dumb, or is this way too complicated lol. Any python examples on how people are authenticating to the graph api and staying authenticated would be greatly appreciated!
I was just working on something similar today. (Microsoft recently deprecated basic authentication for exchange, and I can no longer send mail using a simple username/password from a web application I support.)
Using the microsoft msal python library https://github.com/AzureAD/microsoft-authentication-library-for-python, and the example in sample/device_flow_sample.py, I was able to build a user-based login that retrieves an access token and refresh token in order to stay logged in (using "device flow authentication"). The msal library handles storing and reloading the token cache, as well as refreshing the token whenever necessary.
Below is the code for logging in the first time
#see https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/dev/sample/device_flow_sample.py
import sys
import json
import logging
import os
import atexit
import requests
import msal
# logging
logging.basicConfig(level=logging.DEBUG) # Enable DEBUG log for entire script
logging.getLogger("msal").setLevel(logging.INFO) # Optionally disable MSAL DEBUG logs
# config
config = dict(
authority = "https://login.microsoftonline.com/common",
client_id = 'YOUR CLIENT ID',
scope = ["User.Read"],
username = 'user#domain',
cache_file = 'token.cache',
endpoint = 'https://graph.microsoft.com/v1.0/me'
)
# cache
cache = msal.SerializableTokenCache()
if os.path.exists(config["cache_file"]):
cache.deserialize(open(config["cache_file"], "r").read())
atexit.register(lambda:
open(config["cache_file"], "w").write(cache.serialize())
if cache.has_state_changed else None)
# app
app = msal.PublicClientApplication(
config["client_id"], authority=config["authority"],
token_cache=cache)
# exists?
result = None
accounts = app.get_accounts()
if accounts:
logging.info("found accounts in the app")
for a in accounts:
print(a)
if a["username"] == config["username"]:
result = app.acquire_token_silent(config["scope"], account=a)
break
else:
logging.info("no accounts in the app")
# initiate
if result:
logging.info("found a token in the cache")
else:
logging.info("No suitable token exists in cache. Let's get a new one from AAD.")
flow = app.initiate_device_flow(scopes=config["scope"])
if "user_code" not in flow:
raise ValueError(
"Fail to create device flow. Err: %s" % json.dumps(flow, indent=4))
print(flow["message"])
sys.stdout.flush() # Some terminal needs this to ensure the message is shown
# Ideally you should wait here, in order to save some unnecessary polling
input("Press Enter after signing in from another device to proceed, CTRL+C to abort.")
result = app.acquire_token_by_device_flow(flow) # By default it will block
# You can follow this instruction to shorten the block time
# https://msal-python.readthedocs.io/en/latest/#msal.PublicClientApplication.acquire_token_by_device_flow
# or you may even turn off the blocking behavior,
# and then keep calling acquire_token_by_device_flow(flow) in your own customized loop.
if result and "access_token" in result:
# Calling graph using the access token
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
print("Graph API call result: %s" % json.dumps(graph_data, indent=2))
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
You'll need to fix up the config, and update the scope for the appropriate privileges.
All the magic is in here:
result = app.acquire_token_silent(config["scope"], account=a)
and putting the Authorization access_token in the requests headers:
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
As long as you call acquire_token_silent before you invoke any graph APIs, the tokens will stay up to date. The refresh token is good for 90 days or something, and automatically updates. Once you login, the tokens will be updated and stored in the cache (and persisted to a file), and will stay alive more-or-less indefinitely (there are some things that can invalidate it on the server side).
Unfortunately, I'm still having problems because it's an unverified multi-tenant application. I successfully added the user as a guest in my tenant, and the login works, but as soon as I try to get more interesting privileges in scope, the user can't log in - I'll either have to get my mpn verified, or get my client's 3rd party IT guys admin to grant permission for this app in their tenant. If I had admin privileges for their tenant, I'd probably be looking at the daemon authentication method instead of user-based.
(to be clear, the code above is the msal example almost verbatim, with config and persistence tweaks)
I have a situation where I am trying to create two Cloud Functions namely CF1 & CF2 and I have one Cloud Scheduler. Both cloud functions are having authenticated invocation enabled. My flow is Cloud Scheduler will trigger CF1. On completion of CF1, the CF1 will trigger CF2 as a http call. I have referred Cannot invoke Google Cloud Function from GCP Scheduler to access authenticated CF1 from Cloud Scheduler and able to access CF1. But I am getting problem when accessing CF2 from CF1. The CF1 does not trigger CF2 and also not giving any error message. Do we need to follow any other technique when accessing authenticated Cloud Function from another authenticated Cloud Function.
CF1 code:
import json
import logging
from requests_futures.sessions import FuturesSession
def main(request):
# To read parameter values from request (url arguments or Json body).
raw_request_data = request.data
string_request_data = raw_request_data.decode("utf-8")
request_json: dict = json.loads(string_request_data)
request_args = request.args
if request_json and 'cf2_endpoint' in request_json:
cf2_endpoint = request_json['cf2_endpoint']
elif request_args and 'cf2_endpoint' in request_args:
cf2_endpoint = request_args['cf2_endpoint']
else:
cf2_endpoint = 'Invalid endpoint for CF2'
logger = logging.getLogger('test')
try:
session = FuturesSession()
session.get("{}".format(cf2_endpoint))
logger.info("First cloud function executed successfully.")
except RuntimeError:
logger.error("Exception occurred {}".format(RuntimeError))
CF2 code:
import logging
def main(request):
logger = logging.getLogger('test')
logger.info("second cloud function executed successfully.")
Current output logs:
First cloud function executed successfully.
Expected output logs:
First cloud function executed successfully.
second cloud function executed successfully.
Note: Same flow is working if I use unauthenticated access to the both cloud functions.
Two things are happening here:
You're not using request-futures entirely correctly. Since the request is made asynchronously, you need to block on the result before the function implicitly returns, otherwise it might return before your HTTP request completes (although it probably is in this example):
session = FuturesSession()
future = session.get("{}".format(cf2_endpoint))
resp = future.result() # Block on the request completing
The request you're making to the second function is not actually an authenticated request. Outbound requests from a Cloud Function are not authenticated by default. If you looked at what the actual response is above, you would see:
>>> resp.status_code
403
>>> resp.content
b'\n<html><head>\n<meta http-equiv="content-type" content="text/html;charset=utf-8">\n<title>403 Forbidden</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Forbidden</h1>\n<h2>Your client does not have permission to get URL <code>/function_two</code> from this server.</h2>\n<h2></h2>\n</body></html>\n'
You could jump through a lot of hoops to properly authenticate this request, as detailed in the docs: https://cloud.google.com/functions/docs/securing/authenticating#function-to-function
However, a better alternative would be to make your second function a "background" function and invoke it via a PubSub message published from the first function instead:
from google.cloud import pubsub
publisher = pubsub.PublisherClient()
topic_name = 'projects/{project_id}/topics/{topic}'.format(
project_id=<your project id>,
topic='MY_TOPIC_NAME', # Set this to something appropriate.
)
def function_one(request):
message = b'My first message!'
publisher.publish(topic_name, message)
def function_two(event, context):
message = event['data'].decode('utf-8')
print(message)
As long as your functions have the permissions to publish PubSub messages, this avoids the need to add authorization to the HTTP requests, and also ensures at-least-once delivery.
Google Cloud Function provide REST API interface what include call method that can be used in another Cloud Function HTTP invokation.
Although the documentation mention using Google-provided client libraries there is still non one for Cloud Function on Python.
And instead you need to use general Google API Client Libraries. [This is the python one].3
Probably, the main difficulties while using this approach is an understanding of authentification process.
Generally you need provide two things to build a client service:
credentials ans scopes.
The simpliest way to get credentials is relay on Application Default Credentials (ADC) library. The rigth documentation about that are:
https://cloud.google.com/docs/authentication/production
https://github.com/googleapis/google-api-python-client/blob/master/docs/auth.md
The place where to get scopes is the each REST API function documentation page.
Like, OAuth scope: https://www.googleapis.com/auth/cloud-platform
The complete code example of calling 'hello-world' clound fucntion is below.
Before run:
Create default Cloud Function on GCP in your project.
Keep and notice the default service account to use
Keep the default body.
Notice the project_id, function name, location where you deploy function.
If you will call function outside Cloud Function environment (locally for instance) setup the environment variable GOOGLE_APPLICATION_CREDENTIALS according the doc mentioned above
If you will call actualy from another Cloud Function you don't need to configure credentials at all.
from googleapiclient.discovery import build
from googleapiclient.discovery_cache.base import Cache
import google.auth
import pprint as pp
def get_cloud_function_api_service():
class MemoryCache(Cache):
_CACHE = {}
def get(self, url):
return MemoryCache._CACHE.get(url)
def set(self, url, content):
MemoryCache._CACHE[url] = content
scopes = ['https://www.googleapis.com/auth/cloud-platform']
# If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set,
# ADC uses the service account file that the variable points to.
#
# If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set,
# ADC uses the default service account that Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run,
# and Cloud Functions provide
#
# see more on https://cloud.google.com/docs/authentication/production
credentials, project_id = google.auth.default(scopes)
service = build('cloudfunctions', 'v1', credentials=credentials, cache=MemoryCache())
return service
google_api_service = get_cloud_function_api_service()
name = 'projects/{project_id}/locations/us-central1/functions/function-1'
body = {
'data': '{ "message": "It is awesome, you are develop on Stack Overflow language!"}' # json passed as a string
}
result_call = google_api_service.projects().locations().functions().call(name=name, body=body).execute()
pp.pprint(result_call)
# expected out out is:
# {'executionId': '3h4c8cb1kwe2', 'result': 'It is awesome, you are develop on Stack Overflow language!'}
I am working with slack command (python code is running behind this), it works fine, but this gives error
This slash command experienced a problem: 'Timeout was reached' (error detail provided only to team owning command).
How to avoid this ?
According to the Slack slash command documentation, you need to respond within 3000ms (three seconds). If your command takes longer then you get the Timeout was reached error. Your code obviously won't stop running, but the user won't get any response to their command.
Three seconds is fine for a quick thing where your command has instant access to data, but might not be long enough if you're calling out to external APIs or doing something complicated. If you do need to take longer, then see the Delayed responses and multiple responses section of the documentation:
Validate the request is okay.
Return a 200 response immediately, maybe something along the lines of {'text': 'ok, got that'}
Go and perform the actual action you want to do.
In the original request, you get passed a unique response_url parameter. Make a POST request to that URL with your follow-up message:
Content-type needs to be application/json
With the body as a JSON-encoded message: {'text': 'all done :)'}
you can return ephemeral or in-channel responses, and add attachments the same as the immediate approach
According to the docs, "you can respond to a user commands up to 5 times within 30 minutes of the user's invocation".
After dealing with this issue myself and having my Flask app hosted on Heroku I found that the simplest solution was to use threading. I followed the example from here:
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xi-email-support
from threading import Thread
def backgroundworker(somedata,response_url):
# your task
payload = {"text":"your task is complete",
"username": "bot"}
requests.post(response_url,data=json.dumps(payload))
#app.route('/appmethodaddress',methods=['POST','GET'])
def receptionist():
response_url = request.form.get("response_url")
somedata = {}
thr = Thread(target=backgroundworker, args=[somedata,response_url])
thr.start()
return jsonify(message= "working on your request")
All the slow heavy work is performed by the backgroundworker() function. My slack command points to https://myappaddress.com/appmethodaddress where the receptionist() function takes the response_url of the received Slack message and passes it alongside any other optional data to the backgroundworker(). As the process is now split it simply returns the "working on your request" message to your Slack channel pretty much instantly and upon completion backgroundworker() sends the second message "your task is complete".
I too was facing this error frequently:
"Darn – that slash command didn't work (error message: Timeout was reached). Manage the command at slash-command"
I was writing a Slack slash-command "bot" on AWS Lambda that sometimes needed to perform slow operations (invoking other external APIs etc). The Lambda function would take greater than 3 seconds in some cases causing the Timeout was reached error from Slack.
I found #rcoup's excellent answer here and applied it in the context of AWS Lambda. The error doesn't appear any more.
I did this with two separate Lambda functions. One is a "dispatcher" or "receptionist" that greets the incoming Slack slash command with a "200 OK" and returns the simple "Ok, got that" type of message to the user. The other is the actual "worker" Lambda function that starts the long-ish operation asynchronously and posts the result of that operation to the Slack response_url later.
This is the dispatcher/receptionist Lambda function:
def lambda_handler(event, context):
req_body = event['body']
try:
retval = {}
# the param_map contains the 'response_url' that the worker will need to post back to later
param_map = _formparams_to_dict(req_body)
# command_list is a sequence of strings in the slash command such as "slashcommand weather pune"
command_list = param_map['text'].split('+')
# publish SNS message to delegate the actual work to worker lambda function
message = {
"param_map": param_map,
"command_list": command_list
}
sns_response = sns_client.publish(
TopicArn=MY_SNS_TOPIC_ARN,
Message=json.dumps({'default': json.dumps(message)}),
MessageStructure='json'
)
retval['text'] = "Ok, working on your slash command ..."
except Exception as e:
retval['text'] = '[ERROR] {}'.format(str(e))
return retval
def _formparams_to_dict(req_body):
""" Converts the incoming form_params from Slack into a dictionary. """
retval = {}
for val in req_body.split('&'):
k, v = val.split('=')
retval[k] = v
return retval
As you can see from the above, I didn't invoke the worker Lambda Function directly from the dispatcher (though this is possible). I chose to use AWS SNS to publish a message that the worker receives and processes.
Based on this StackOverflow answer, this is the better approach as it's non-blocking (asynchronous) and scalable. Also it was easier to use SNS to decouple the two functions in the context of AWS Lambda, direct invocation is trickier for this use-case.
Finally, here's how I consume the SNS event in my worker Lambda Function:
def lambda_handler(event, context):
message = json.loads(event['Records'][0]['Sns']['Message'])
param_map = message['param_map']
response_url = param_map['response_url']
command_list = message['command_list']
main_command = command_list[0].lower()
# process the command as you need to and finally post results to `response_url`