I am working with slack command (python code is running behind this), it works fine, but this gives error
This slash command experienced a problem: 'Timeout was reached' (error detail provided only to team owning command).
How to avoid this ?
According to the Slack slash command documentation, you need to respond within 3000ms (three seconds). If your command takes longer then you get the Timeout was reached error. Your code obviously won't stop running, but the user won't get any response to their command.
Three seconds is fine for a quick thing where your command has instant access to data, but might not be long enough if you're calling out to external APIs or doing something complicated. If you do need to take longer, then see the Delayed responses and multiple responses section of the documentation:
Validate the request is okay.
Return a 200 response immediately, maybe something along the lines of {'text': 'ok, got that'}
Go and perform the actual action you want to do.
In the original request, you get passed a unique response_url parameter. Make a POST request to that URL with your follow-up message:
Content-type needs to be application/json
With the body as a JSON-encoded message: {'text': 'all done :)'}
you can return ephemeral or in-channel responses, and add attachments the same as the immediate approach
According to the docs, "you can respond to a user commands up to 5 times within 30 minutes of the user's invocation".
After dealing with this issue myself and having my Flask app hosted on Heroku I found that the simplest solution was to use threading. I followed the example from here:
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xi-email-support
from threading import Thread
def backgroundworker(somedata,response_url):
# your task
payload = {"text":"your task is complete",
"username": "bot"}
requests.post(response_url,data=json.dumps(payload))
#app.route('/appmethodaddress',methods=['POST','GET'])
def receptionist():
response_url = request.form.get("response_url")
somedata = {}
thr = Thread(target=backgroundworker, args=[somedata,response_url])
thr.start()
return jsonify(message= "working on your request")
All the slow heavy work is performed by the backgroundworker() function. My slack command points to https://myappaddress.com/appmethodaddress where the receptionist() function takes the response_url of the received Slack message and passes it alongside any other optional data to the backgroundworker(). As the process is now split it simply returns the "working on your request" message to your Slack channel pretty much instantly and upon completion backgroundworker() sends the second message "your task is complete".
I too was facing this error frequently:
"Darn – that slash command didn't work (error message: Timeout was reached). Manage the command at slash-command"
I was writing a Slack slash-command "bot" on AWS Lambda that sometimes needed to perform slow operations (invoking other external APIs etc). The Lambda function would take greater than 3 seconds in some cases causing the Timeout was reached error from Slack.
I found #rcoup's excellent answer here and applied it in the context of AWS Lambda. The error doesn't appear any more.
I did this with two separate Lambda functions. One is a "dispatcher" or "receptionist" that greets the incoming Slack slash command with a "200 OK" and returns the simple "Ok, got that" type of message to the user. The other is the actual "worker" Lambda function that starts the long-ish operation asynchronously and posts the result of that operation to the Slack response_url later.
This is the dispatcher/receptionist Lambda function:
def lambda_handler(event, context):
req_body = event['body']
try:
retval = {}
# the param_map contains the 'response_url' that the worker will need to post back to later
param_map = _formparams_to_dict(req_body)
# command_list is a sequence of strings in the slash command such as "slashcommand weather pune"
command_list = param_map['text'].split('+')
# publish SNS message to delegate the actual work to worker lambda function
message = {
"param_map": param_map,
"command_list": command_list
}
sns_response = sns_client.publish(
TopicArn=MY_SNS_TOPIC_ARN,
Message=json.dumps({'default': json.dumps(message)}),
MessageStructure='json'
)
retval['text'] = "Ok, working on your slash command ..."
except Exception as e:
retval['text'] = '[ERROR] {}'.format(str(e))
return retval
def _formparams_to_dict(req_body):
""" Converts the incoming form_params from Slack into a dictionary. """
retval = {}
for val in req_body.split('&'):
k, v = val.split('=')
retval[k] = v
return retval
As you can see from the above, I didn't invoke the worker Lambda Function directly from the dispatcher (though this is possible). I chose to use AWS SNS to publish a message that the worker receives and processes.
Based on this StackOverflow answer, this is the better approach as it's non-blocking (asynchronous) and scalable. Also it was easier to use SNS to decouple the two functions in the context of AWS Lambda, direct invocation is trickier for this use-case.
Finally, here's how I consume the SNS event in my worker Lambda Function:
def lambda_handler(event, context):
message = json.loads(event['Records'][0]['Sns']['Message'])
param_map = message['param_map']
response_url = param_map['response_url']
command_list = message['command_list']
main_command = command_list[0].lower()
# process the command as you need to and finally post results to `response_url`
Related
I am trying to figure out how I can determine if a lambda function is being run for the third time. When I schedule a lambda function to run in cloudwatch the lambda function will run as expected. Sometimes the function fails on the first try. Lambda by default will run the function three times. Usually, on the second attempt, the function has no problem and runs as expected.
The problem is we have another Lambda function that fires when there is a failure. This function will send a slack message on failure. Because these functions recover on the second attempt so ofter I really want to find a way to know if this is the third attempt or not. I know I can do something like adding a record to Redis then check Redis whenever you run to see which attempt number this is but I'd rather not go that route.
Thanks
'''
Follow these steps to configure the webhook in Slack:
1. Navigate to https://<your-team-domain>.slack.com/services/new
2. Search for and select "Incoming WebHooks".
3. Choose the default channel where messages will be sent and click "Add Incoming WebHooks Integration".
4. Copy the webhook URL from the setup instructions and use it in the next section.
To encrypt your secrets use the following steps:
1. Create or use an existing KMS Key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html
2. Expand "Encryption configuration" and click the "Enable helpers for encryption in transit" checkbox
3. Paste <SLACK_CHANNEL> into the slackChannel environment variable
Note: The Slack channel does not contain private info, so do NOT click encrypt
4. Paste <SLACK_HOOK_URL> into the kmsEncryptedHookUrl environment variable and click "Encrypt"
Note: You must exclude the protocol from the URL (e.g. "hooks.slack.com/services/abc123").
5. Give your function's role permission for the `kms:Decrypt` action using the provided policy template
'''
import boto3
import json
import logging
import os
from base64 import b64decode
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
# The base-64 encoded, encrypted key (CiphertextBlob) stored in the kmsEncryptedHookUrl environment variable
ENCRYPTED_HOOK_URL = os.environ['kmsEncryptedHookUrl']
# The Slack channel to send a message to stored in the slackChannel environment variable
SLACK_CHANNEL = os.environ['slackChannel']
HOOK_URL = "https://" + boto3.client('kms').decrypt(
CiphertextBlob=b64decode(ENCRYPTED_HOOK_URL),
EncryptionContext={'LambdaFunctionName': os.environ['AWS_LAMBDA_FUNCTION_NAME']}
)['Plaintext'].decode('utf-8')
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info("Event: " + str(event))
message = json.loads(event['Records'][0]['Sns']['Message'])
logger.info("Message: " + str(message))
alarm_name = message['AlarmName']
#old_state = message['OldStateValue']
new_state = message['NewStateValue']
reason = message['NewStateReason']
slack_message = {
'channel': SLACK_CHANNEL,
'text': "%s state is now %s: %s" % (alarm_name, new_state, reason)
}
req = Request(HOOK_URL, json.dumps(slack_message).encode('utf-8'))
try:
response = urlopen(req)
response.read()
logger.info("Message posted to %s", slack_message['channel'])
except HTTPError as e:
logger.error("Request failed: %d %s", e.code, e.reason)
except URLError as e:
logger.error("Server connection failed: %s", e.reason)
Coming back to this old question to give an answer after gaining a bunch of experience. You just have to set the number of retries the lambda should do. The lambda isn’t considered “failed” until it fails after you’ve used up your retries. That was the part I didn’t understand. So after it truly fails you can send a message to sqs or a number of services. You just have to set the destination on failure to whatever you want.
Hope this helps someone else who didnt understand this.
I have AWS API gateway setup for a public endpoint with no auth. It connects to a websocket that triggers a Lambda.
I was creating connections with Python's websocket-client lib at https://pypi.org/project/websocket_client/.
I noticed that connections would fail ~10% of the time, and get worse as I increased load. I can't find anywhere that would be throttling me seeing as my general API Gateway settings say Your current account level throttling rate is 10000 requests per second with a burst of 5000 requests.. That’s beside the point that just 2-3 requests per second would trigger issue fairly often.
Meanwhile the failure response would be like {u'message': u'Forbidden', u'connectionId': u'Z2Jp-dR5vHcCJkg=', u'requestId': u'Z2JqAEJRvHcFzvg='}
I went into my CloudWatch log insights and searched for the connection ID and request ID. The log group for the API gateway would find no results with either ID. Yet a search on my Lambda that fires on websocket connect, would have a log with that connection ID. The log showed everything running as expected on our side. The lambda simply runs a MySQL query that fires.
Why would I get a response of forbidden, despite the lambda working as expected?
The existing question over at getting message: forbidden reply from AWS API gateway, seems to address if it's ALWAYS returning forbidden for some private endpoints. Nothing lined up with my use case.
UPDATE
I think this may be related to locust.io, or python, which I'm using to connect every second. I installed https://www.npmjs.com/package/wscat on my machine and am connecting and closing as fast as possible repeatedly. I am not getting a Forbidden message. It's just extra confusing since I'm not sure how the way I connect would randomly spit back a Forbidden message some of the time.
class SocketClient(object):
def __init__(self, host):
self.host = host
self.session_id = uuid4().hex
def connect(self):
self.ws = websocket.WebSocket()
self.ws.settimeout(10)
self.ws.connect(self.host)
events.quitting += self.on_close
data = self.attach_session({})
return data
def attach_session(self, payload):
message_id = uuid4().hex
start_time = time.time()
e = None
try:
print("Sending payload {}".format(payload))
data = self.send_with_response(payload)
assert data['mykey']
except AssertionError as exp:
e = exp
except Exception as exp:
e = exp
self.ws.close()
self.connect()
elapsed = int((time.time() - start_time) * 1000)
if e:
events.request_failure.fire(request_type='sockjs', name='send',
response_time=elapsed, exception=e)
else:
events.request_success.fire(request_type='sockjs', name='send',
response_time=elapsed,
response_length=0)
return data
def send_with_response(self, payload):
json_data = json.dumps(payload)
g = gevent.spawn(self.ws.send, json_data)
g.get(block=True, timeout=2)
g = gevent.spawn(self.ws.recv)
result = g.get(block=True, timeout=10)
json_data = json.loads(result)
return json_data
def on_close(self):
self.ws.close()
class ActionsTaskSet(TaskSet):
#task
def streams(self):
response = self.client.connect()
logger.info("Connect Response: {}".format(response))
class WSUser(Locust):
task_set = ActionsTaskSet
min_wait = 1000
max_wait = 3000
def __init__(self, *args, **kwargs):
super(WSUser, self).__init__(*args, **kwargs)
self.client = SocketClient('wss://mydomain.amazonaws.com/endpoint')
Update 2
I have enabled access logs, the one type of log that wasn't there before. I can now see that my lambdas are always getting a 200 with no issue. The 403 is coming from some MESSAGE eventType that doesn't hit an actual routeKey. Not sure where it comes from, but pretty sure finding that answer will solve this.
I was also able to confirm there are no ENI issues.
You might be running into some VPC-related limits. See https://winterwindsoftware.com/scaling-lambdas-inside-vpc/. Sounds like you might be running out of ENIs. You could try moving the function to a different VPC. How long does each invocation of the lambda run for? And what language is you lambda written in?
The payload in my example is empty. The API is configured to use $request.body.action to know the routeKey. Connecting makes the default $connect route work.
Adding a proper action in my body made the 403s go away. This is the solution. I was essentially getting 200 responses from the act of connecting and disconnecting, but was getting the 403 whenever my message without a payload went through.
So we have a single thread flask server running where we receive requests from a python app client. In this flask server we use rabbitMQ with pika library to distribute messages to other clients.
What is happening is that in the get function the program is crashing with the error:
pika.exceptions.ConnectionClosed: (505, 'UNEXPECTED_FRAME - expected
content header for class 60, got non content header frame instead')
I've searched a lot of topics about this in stack overflow and others but they all address problems with multi threading which is not the case. Flask should only serve with one thread unless it is called in app.run(threaded=yes).
The program normally crashes when multiple messages are sent in a short interval (e.g. 5 per second) and it's also important to note that messages are being received every second with a request to this function:
#app.route('/api/users/getMessages', methods=['POST'])
def get_Messages():
data = json.loads(request.data)
token = data['token']
payload = jwt.decode(token, 'SECRET', algorithms=['HS256'])
istid = payload['istid']
print('istid: '+istid)
messages = []
queue = channel.queue_declare(queue=istid)
for i in range(queue.method.message_count):
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
if method_frame:
#print(method_frame, header_frame, body)
messages.append(body)
else:
print('No message returned')
res = {'messages':messages, 'error':0}
return jsonify(res)
In this code it crashes normally in the line:
queue = channel.queue_declare(queue=istid)
But we also tried to change the code to use a while instead of a for where it ends when the body is None and it crashes in the line:
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
in that case.
Also important, the crashes are random and it can work a few times and then randomly crashes after a get request while messages are being sent. If anyone knows anything related to this we would appreciate any help.
Another note, we thought about using basic_consume with callback instead of basic_get but we didn't find a way in which this would work since we have to send the messages back and have several user making requests to this same function.
EDIT #1:
In the rabbitMQ docs rabbitmq if you search for the function "def basic_get" you will notice there are some TODO comments and also a reference to this
Due to implementation details, this cannot be called a second time
until the callback is executed.
So I suspected that this could be what was happening but even if it is I don't know how could it be solved.
For anyone interested in the solution, as it is in the other comments, the program was not thread safe since flask as of version 1.0 uses threaded = True as default.
The solution is either:
1) running flask with app.run(threaded = False)
2) Making the program thread safe by implementing locks whenever accessing the channel /connection with pika.
I am trying to build an application using Google Cloud Platform AutoML using Python. My overall code flow looks like this:
User Interacts--> data sent to PubSub--> callback invokes my AutoML--> Result
The snippet that calls pubsub looks like this:
blob=blob+bytes(doc_type,'utf-8')
publisher.publish(topic,blob)
future=subscriber.subscribe(subscription,callback=callback)
#flash("The object is "+future,'info')
try:
future.result()
except Exception as ex:
subscriber.close()
In PubSub callback:
def callback(message):
new_message=message.data
display_name,score=predict_value(new_message,"modelID","projectid",'us-central1')
message.ack()
And my predict_value gets the model_id, project id and compute region and performs the prediction.
When I directly call predict_value without using PubSub it is working fine. If I do like this, I am getting the below error:
google.api_core.exceptions.PermissionDenied: 403 Permission 'automl.models.predict' denied on resource 'projects/projectID/locations/us-central1/models/' (or it may not exist).
Please help me to resolve the issue
Thank you so much for all your responses. I have just fixed the issue using the below snippet example
def receive_messages_synchronously(project, subscription_name):
"""Pulling messages synchronously."""
# [START pubsub_subscriber_sync_pull]
# project = "Your Google Cloud Project ID"
# subscription_name = "Your Pubsub subscription name"
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
# Builds a pull request with a specific number of messages to return.
# `return_immediately` is set to False so that the system waits (for a
# bounded amount of time) until at lease one message is available.
response = subscriber.pull(
subscription_path,
max_messages=3,
return_immediately=False)
ack_ids = []
for received_message in response.received_messages:
print("Received: {}".format(received_message.message.data))
ack_ids.append(received_message.ack_id)
# Acknowledges the received messages so they will not be sent again.
subscriber.acknowledge(subscription_path, ack_ids)
# [END pubsub_subscriber_sync_pull]
The reason being the subscription that is created uses the pull request. I guess the callback method concept used is mainly for "push" which may be the reason because I didnt give the endpoint and token to publish the message. Hope what I am guessing is correct. Let me know your views as well.
This is likely due to one of two factors:
invalid credentials being used when sending the request to the AutoML API - it is very likely that pubsub executes in other context and can't get the default credentials
invalid model resource name (make sure it is correct) - it should be something like: "projects/12423534/locations/us-central1/models/23432423"
Suppose I have two scripts - one is for cloudfront(cf),and another elb.I have some assigned value in json inside a queue(Rabbitmq), called request queue.when I run my elb scripts it should get only the msg with file_type=elb
{'file_type':'elb','elb_name':'name1','customer_id':'1','s3_bucket':'elb-logs-bucket'}
when I run my cf scripts it should get only the msg with file_type=cf
{'file_type':'cf','cf_name':'name1','customer_id':'1','s3_bucket':'cf-logs-bucket'}
My below code works only for the first msg I published to the queue, not type.Anyone can provide the idea what can I do to get file_type=cf msg from cf scripts and vice-versa..I am using default exchange
import json
import pika
import logging
def get_rmq_request_queue(channel,connection,req_queue_name):
payload_dict={}
try:
queue_message=channel.basic_get(queue=req_queue_name,no_ack=False)
payload=queue_message[2]
print payload
print payload
payload_dict=json.loads(payload)
return payload_dict
except Exception as e:
payload_dict={}
payload_dict['error']=str(e)
return payload_dict
main():
res_req_que=get_rmq_request_queue(channel,connection,req_queue_name)
if 'error' not in res_req_que:
s3_bucket=res_req_que['s3_bucket']
customer_id=res_req_que['customer_id']
file_type=res_req_que['file_type']
cf_name=res_req_que['cf_name']
I might have many messages in a queue, the challenge is I need to get the recent message with file type...
One of the points of RabbitMQ is that you can define queues to bind to exchanges with particular routing keys. You should use the type parameter as part of your routing key, then bind your queue so that it listens to the type you want.
See for example part 4 of the RabbitMQ tutorial.