I am trying to build a system where I can send messages to diffferent users based on their subscription to certain events. Basically I have an api which gives me live stream events. Some of the users will be subscribed to those events. My task is to send message to those users whenever such an event occurs. I am trying to design the system in Python.
Currently I have the following questions.
How yo continously poll for events from a live stream api in Python.
How to find out which users are subscribed to that particular event. (Redis or Mysql)
How to send notification to all the users of a particular event. (Pub/sub)
I am thinking of using Amazon SNS. But not quite sure about the overall architecture.
RabbitMQ is lightweight and easy to deploy on premise and in the
cloud. It supports multiple messaging protocols. RabbitMQ can be
deployed in distributed and federated configurations to meet
high-scale, high-availability requirements.
Just small example:
Producer sends messages to the "hello" queue. The consumer receives messages from that queue. This will create a Queue (hello) with a message on the RabbitMQ cluster.
#!/usr/bin/env python
import pika
RABBITMQ_USERNAME = 'ansible'
RABBITMQ_PASSWORD = 'ansible'
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='example.eu-central-1.elb.amazonaws.com',
heartbeat_interval=25,
credentials=pika.PlainCredentials(RABBITMQ_USERNAME,RABBITMQ_PASSWORD)))
channel = connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',
routing_key='hello',
body='Hello World!')
print(" [x] Sent 'Hello World!'")
connection.close()
Receive a message from a named queue:
#!/usr/bin/env python
import pika
RABBITMQ_USERNAME = 'ansible'
RABBITMQ_PASSWORD = 'ansible'
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='example.elb.amazonaws.com',
heartbeat_interval=25,
credentials=pika.PlainCredentials(RABBITMQ_USERNAME,RABBITMQ_PASSWORD)))
channel = connection.channel()
channel.queue_declare(queue='hello')
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
channel.basic_consume(callback,
queue='hello',
no_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
Related
I have a virtual assistant which receives messages and send it to a event broker (e.g rabbitmq).
An event broker allows me to connect my running assistant to other services that process the data coming in from conversations.
Example:
If I have a RabbitMQ server running, as well as another application that consumes the events, then this consumer needs to implement Pika's start_consuming() method with a callback action. Here's a simple example:
import json
import pika
def _callback(self, ch, method, properties, body):
# Do something useful with your incoming message body here, e.g.
# saving it to a database
print('Received event {}'.format(json.loads(body)))
if __name__ == '__main__':
# RabbitMQ credentials with username and password
credentials = pika.PlainCredentials('username', 'password')
# Pika connection to the RabbitMQ host - typically 'rabbit' in a
# docker environment, or 'localhost' in a local environment
connection = pika.BlockingConnection(
pika.ConnectionParameters('rabbit', credentials=credentials))
# start consumption of channel
channel = connection.channel()
channel.basic_consume(_callback,
queue='events',
no_ack=True)
channel.start_consuming()
What is the correct way to use fastapi with Pika to consume these live messages and saving it to a database ?
Do I need a websocket route ?
Using the pika client, I want to display all the messages currently in the queue, without consuming them. Just to know how busy is the queue and display the jobs.
So far, I can only read one message as it arrives:
channel.queue_declare(queue='queue1', durable=True)
channel.basic_consume(on_message, queue='queue1')
channel.start_consuming()
def on_message(channel, method, properties, message):
channel.basic_ack(delivery_tag=method.delivery_tag)
print("Message: %s", message)
How can I read the whole queue?
To read messages "without consuming them", don't acknowledge delivery of the message. In your case above, get rid of
channel.basic_ack(delivery_tag=method.delivery_tag)
or set auto_ack to False:
def callback(ch, method, properties, body):
print(body)
channel.basic_consume(queue='your_queue', on_message_callback=callback, auto_ack=False)
The messages will be read and marked as unacked in rabbitMQ but will still be available in the queue.
I am trying to reliably send a message from a publisher to multiple consumers using RabbitMQ topic exchange.
I have configured durable queues (one per consumer) and I am sending persistent messages delivery_mode=2. I am also setting the channel in confim_delivery mode, and have added mandatory=True flag to publish.
Right now the service is pretty reliable, but messages get lost to one of the consumers if it stays down during a broker restart followed by a
message publication.
It seems that broker can recover queues and messages on restart, but it doesn't seem to keep the binding between consumers and queues. So messages only reach one of the consumers and get lost for the one that is down.
Note: Messages do reach the queue and the consumer if the broker doesn't suffer a restart during the time a consumer is down. They accumulate properly on the queue and they are delivered to the consumer when it is up again.
Edit - adding consumers code:
import pika
class Consumer(object):
def __init__(self, queue_name):
self.queue_name = queue_name
def consume(self):
credentials = pika.PlainCredentials(
username='myuser', password='mypassword')
connection = pika.BlockingConnection(
pika.ConnectionParameters(host='myhost', credentials=credentials))
channel = connection.channel()
channel.exchange_declare(exchange='myexchange', exchange_type='topic')
channel.queue_declare(queue=self.queue_name, durable=True)
channel.queue_bind(
exchange='myexchange', queue=self.queue_name, routing_key='my.route')
channel.basic_consume(
consumer_callback=self.message_received, queue=self.queue_name)
channel.start_consuming()
def message_received(self, channel, basic_deliver, properties, body):
print(f'Message received: {body}')
channel.basic_ack(delivery_tag=basic_deliver.delivery_tag)
You can assume each consumer server does something similar to:
c = Consumer('myuniquequeue') # each consumer has a permanent queue name
c.consume()
Edit - adding publisher code:
def publish(message):
credentials = pika.PlainCredentials(
username='myuser', password='mypassword')
connection = pika.BlockingConnection(
pika.ConnectionParameters(host='myhost', credentials=credentials))
channel = connection.channel()
channel.exchange_declare(exchange='myexchange', exchange_type='topic')
channel.confirm_delivery()
success = channel.basic_publish(
exchange='myexchange',
routing_key='my.route',
body=message,
properties=pika.BasicProperties(
delivery_mode=2, # make message persistent
),
mandatory=True
)
if success:
print("Message sent")
else:
print("Could not send message")
# Save for sending later
It is worth saying that I am handling the error case on my own, and it is not the part I would like to improve. When my messages get lost to some of the consumers the flow goes through the success section
Use basic.ack(delivery_tag=basic_deliver.delivery_tag) in your consumer callback method. This acknowledgement tells whether the consumer has received a message and processed it or not. If it's a negative acknowledgement, the message will be requeued.
Edit #1
In order to receive messages during broker crash, the broker needs to be distributed. It is a concept called Mirrored Queues in RabbitMQ. Mirrored Queues lets your queues to be replicated across the nodes in your cluster. If one of the nodes containing the queue goes down, the other node containing the queue will act as your broker.
For complete understanding refer this Mirrored Queues
I have implemented RabbitMQ in my servers. So basically what it does is that the main server passes messages to the worker server.
The problem that I am facing is that all the message that I pass is not received by the server.
i.e if i send 10 messages only 4 of them are received.
Any idea where am I going wrong.
Receiving code
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
channel.basic_consume(callback,
queue='hello',
no_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
Publishing code
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
message = ' '.join(sys.argv[1:]) or "Hello World!"
channel.basic_publish(exchange='',
routing_key='task_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # make message persistent
))
print(" [x] Sent %r" % message)
connection.close()
Assuming that you are publishing to the same queue (as the examples you posted shows otherwise). I would recommend that you enable the confirm delivery flag. This will ensure that your message gets delivered, and if not it will either throw an exception, or publish will return False.
channel = connection.channel()
channel.confirm_delivery()
published = channel.basic_publish(...)
if not published:
raise Exception("Unable to publish message!")
It might also be worth to install the management plugin for RabbitMQ and inspect the queue before you start consuming messages. This way you can verify that the messages got published, and later consumed.
How do you ensure that messages get delivered with Pika? By default it will not provide you with an error if the message was not delivered succesfully.
In this example several messages can be sent before pika acknowledges that the connection was down.
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
for index in xrange(10):
channel.basic_publish(exchange='', routing_key='hello',
body='Hello World #%s!' % index)
print('Total Messages Sent: %s' % x)
connection.close()
When using Pika the channel.confirm_delivery() flag needs to be set before you start publishing messages. This is important so that Pika will confirm that each message has been sent successfully before sending the next message. This will however increase the time it takes to send messages to RabbitMQ, as delivery needs to be confirmed before the program can proceed with the next message.
channel.confirm_delivery()
try:
for index in xrange(10):
channel.basic_publish(exchange='', routing_key='hello',
body='Hello World #%s!' % index)
print('Total Messages Sent: %s' % x)
except pika.exceptions.ConnectionClosed as exc:
print('Error. Connection closed, and the message was never delivered.')
basic_publish will return a Boolean depending if the message was sent or not. But, it is important to catch potential exceptions in case the connection is closed during transfer and handle it appropriately. As in those cases the exception will interrupt the flow of the program.
after trying myself and failing to receive other than ack,
i decided to implement a direct reply to the sender.
i followed the example given here