I'm using python and when I read things from SQS they don't get removed properly and they get read again a few seconds later.
I use this code snippet to post a job
#!/usr/bin/python
#simple code snippet that posts a message to SQS
import boto
from boto.sqs.message import Message
conn = boto.connect_sqs()
q = conn.create_queue('myqueue')
newJob.set_body("Message")
q.write(newJob)
Here is the code that digests messages
#!/usr/bin/python
#simple code snippet that reads a message from SQS
import boto
from boto.sqs.message import Message
conn = boto.connect_sqs()
q = conn.get_queue('myqueue')
while True:
try:
print q.read(10).get_body()
except:
pass
After running both scripts the second one will print out 'Message' every 15 seconds or so.
I'm really confused as to why it's not being removed AND why it comes back only after 15 seconds.
(It also doesn't show up in the AWS console, and messages I send from there are never processed)
Disclaimer: I never used boto.sqs before and this answer is just based on reading its doc.
According to below description, it looks like a message after being read 1) is NOT automatically deleted , and 2) becomes invisible for a specific time period.
There is an optional parameter to create_queue called visibility_timeout. This basically controls how long a message will remain invisible to other queue readers once it has been read (see SQS documentation for more detailed explanation). If this is not explicitly specified the queue will be created with whatever default value SQS provides (currently 30 seconds).
update: Amazon SQS Doc confirms my above understanding.
...Therefore, Amazon SQS does not delete the message, and instead, your consuming component must delete the message from the queue after receiving and processing it.
...Therefore, Amazon SQS blocks them with a visibility timeout, which is a period of time during which Amazon SQS prevents other consuming components from receiving and processing that message...
So I guess you need to explicitly delete the message after you have successfully read it from the queue (if you don't want another client consume it).
Related
There is a receive function at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.receive_message to get SQS message,
Is there a function that I can just take a Peek at the SQS messages, without actually receiving it. Because If I receive the messages, it will be deleted from the queue. But I want the messages to be stay in the queue after peeking.
you can check
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
aws sqs sdk's (and client libraries written on top of them) by default they dont delete messages. but they have 'Visibility Timeout' which is 30 seconds by default. That means after you read the message , it wont be visible to other consumers for 30 seconds. It is up to a client to delete it within that time frame so that no one else will get that message ever.
So You can reduce that visility time out to something really small like 1 second. So you can download the message and within 1 second it will be available to other consumers . you can even set it to 0 so everyone can read the message at any point.
But this still means you will receieve the message. SQS is pretty simple queue system. you might want to check other queue system like Kafka or different way to design your system like using Notification services such as SNS
I have an architecture which looks like that:
As soon as a message is sent to a SQS queue, an ECS task picks this message and process it.
Which means that if X messages are sent into the queue, X ECS task will be spun up in parallel. An ECS task is only able to fetch one message (per my code above)
The ECS task uses a dockerized Python container, and uses boto3 SQS client to retrieve and parse the SQS message:
sqs_response = get_sqs_task_data('<sqs_queue_url>')
sqs_message = parse_sqs_message(sqs_response)
while sqs_message is not None:
# Process it
# Delete if from the queue
# Get next message in queue
sqs_response = get_sqs_task_data('<sqs_queue_url>')
sqs_message = parse_sqs_message(sqs_response)
def get_sqs_task_data(queue_url):
client = boto3.client('sqs')
response = client.receive_message(
QueueUrl=queue_url,
MaxNumberOfMessages=1
)
return response
def parse_sqs_message(response_sqs_message):
if 'Messages' not in response_sqs_message:
logging.info('No messages found in queue')
return None
# ... parse it and return a dict
return {
data_1 = ...,
data_2 = ...
}
All in all, pretty straightforward.
In get_sqs_data(), I explicitely specify that I want to retrieve only one message (because 1 ECS task has to process only one message).
In parse_sqs_message(), I test if there are some messages left in the queue with
if 'Messages' not in response_sqs_message:
logging.info('No messages found in queue')
return None
When there is only one message in the queue (meaning one ECS task has been triggered), everything is working fine. The ECS task is able to pick the message, process it and delete it.
However, when the queue is populated with X messages (X > 1) at the same time, X ECS task are triggered, but only ECS task is able to fetch one of the message and process it.
All the others ECS tasks will exit with No messages found in queue, although there are X - 1 messages left to be processed.
Why is that? Why are the others task not able to pick the messages left to be picked?
If that matters, the VisibilityTimeout of SQS is set to 30mins.
Any help would greatly be appreciated!
Feel free to ask for more precision if you want so.
I forgot to give an answer to that question.
The problem was the fact the the SQS was setup as a FIFO queue.
A FIFO Queue only allows one consumer at a time (to preserve the order of the message). Changing it to a normal (standard) queue fixed this issue.
I'm not sure to understand how the tasks are triggered from SQS, but from what I understand in the SQS SDK documentation, this might happen if the number of messages is small when using short polling. From the get_sqs_task_data definition, I see that your are using short polling.
Short poll is the default behavior where a weighted random set of machines
is sampled on a ReceiveMessage call. Thus, only the messages on the
sampled machines are returned. If the number of messages in the queue
is small (fewer than 1,000), you most likely get fewer messages than you requested
per ReceiveMessage call.
If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response.
If this happens, repeat the request.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.receive_message
You might want to try to use Long polling with a value superior to the visibility timeout
I hope it helps
Scenario :
create a lambda and it will be triggered whenever a message comes to SQS(let's assume SQS-A). The lambda (written in python)is responsible for sending the incoming payload to the another endpoint.
The problem is, whenever the target endpoint or server is down, I was trying to place it into the another SQS (let's assume SQS-B), if other exceptions comes than placing it into Deal Letter Queue.
Here I want to two things.
If ConnectionError (it is the python exception says which says endpoint is down) comes I want to stop the the SQS-A(there is no point to run the lambda as the target server is down).
(or)
As whenever I get this error I am sending it to the SQS-B, I want the SQS-B to be triggered for when the first request comes and it should check if still there is a connection error, it has to trigger after 10 minutes, and again check, if exception persists trigger after 30 minutes, like this
I want to increment the time up to 4 hours and after that check/trigger the lambda every 4 hours. If there is no exception then it should read all the messages in the SQS-B.
Help me how to achieve any one of the approach or recommend any other better approach
You are creating a complex architecture due to a simple problem (the target not being available). Try not to over-complicate things.
I would recommend:
Have the originating system send the message to an Amazon SNS topic
The topic triggers a Lambda function
If it successfully processes the message, no further action required
If the remote endpoint is not available, put the message into an Amazon SQS queue for later processing
Use Amazon CloudWatch Events to trigger a Lambda function every n minutes that grabs any messages in the queue and tries to send them again. If the remote endpoint is still down, it exits and the process will be attempted again n minutes later.
Probably worth also sending an email to an Admin if a message gets older than a few hours.
If you must send the original message to an SQS queue, then you could do as you described... send to Queue-A first, which triggers a Lambda function. If the endpoint is down, Lambda sends the message to Queue-B for later processing. However, only process from Queue-B every n minutes (rather than trying to make each individual message have its own delay timer).
I have a some EC2 servers pulling work off of a SQS queue. Occasionally, they encounter a situation where the can't finish the job. I have the process email me of the condition. As it stands now, the message stays "in flight" until it times out. I would like for the process to immediately release it back to the queue after the email is sent. But, I'm not sure how to accomplish this. Is there a way? If so, can you please point me to the call or post a code snippet.
I'm using Python 2.7.3 and Boto 2.5.2.
If you have read a message and decide, for whatever reason, that you do not want to process it and would rather make it immediately available to other readers of the queue, you can simply set that message's visibility timeout to zero using the change_visibility method of the Message object in boto. See The SQS Developer's Guide for details.
I'm following this tutorial: http://boto.s3.amazonaws.com/sqs_tut.html
When there's something in the queue, how do I assign one of my 20 workers to process it?
I'm using Python.
Unfortunately, SQS lacks some of the semantics we've often come to expect in queues. There's no notification or any sort of blocking "get" call.
Amazon's related SNS/Simple Notification Service may be useful to you in this effort. When you've added work to the queue, you can send out a notification to subscribed workers.
See also:
http://aws.amazon.com/sns/
Best practices for using Amazon SQS - Polling the queue
This is (now) possible with Long polling on a SQS queue.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Query_QueryReceiveMessage.html
Long poll support (integer from 1 to 20) - the duration (in seconds) that the ReceiveMessage action call will wait until a message is in the queue to include in the response, as opposed to returning an empty response if a message is not yet available.
If you do not specify WaitTimeSeconds in the request, the queue attribute ReceiveMessageWaitTimeSeconds is used to determine how long to wait.
Type: Integer from 0 to 20 (seconds)
Default: The ReceiveMessageWaitTimeSeconds of the queue.
Further to point out a problem with SQS - You must poll for new notifications, and there is no guarantee that on any particular poll you will receive an event that exists in the queue (this is due to the redundancy of their architecture). This means you need to consider the possibility that your polling didn't return a message that existed (which for me meant I needed to increase the polling rate).
All in all I found too many limitations with SQS (as I've found with some other AWS tools such as SimpleDB). But that's just my injected opinion.
Actual if you dont require a low latency, you can try this:
Create an cloudwatch alarm on your queue, like messages visible or messages received > 0.
As an action you will send a message to an sns topic, which then can send the message to your workers via an http/s endpoint.
normally this kind of approach is used for autoscaling.
There is now an JMS wrapper for SQS from Amazon that will let you create listeners that are automatically triggered when a new message is available.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/jmsclient.html#jmsclient-gsg