I'm a newby to all things mqtt and as a first exercise I wanted to create a “mailbox” service through a persistent mqtt session. The incentive is a low power ESP8266 device that sleeps most of the time and periodically wakes up and checks if there are any pending commands for it.
I tried implementing this through a sender and receiver on my Linux host with python and paho mqtt. Mosquitto is running in the background as the broker.
First here is the "mbox" sender, which sends another message every time Enter is pressed.
import paho.mqtt.client as mqtt
broker_address='127.0.0.1'
client = mqtt.Client('MBoxClient')
client.connect(broker_address)
counter = 1
while True:
print('Press Enter to send msg #'+str(counter)+': ', end='')
if input().startswith('q'):
break
client.publish("mbox/mail","Hello "+str(counter), qos=1)
counter += 1
client.disconnect()
print('done!')
And here is my mbox receiver:
import paho.mqtt.client as mqtt
import time
def on_message(client, userdata, message):
print("message:", message.topic + ': ' + str(message.payload.decode("utf-8")))
print('I\'m listening for mbox messages!')
broker_address="127.0.0.1"
client_name='mbox'
is_first=True
while 1:
client = mqtt.Client(client_name, clean_session=is_first)
is_first=False
print("polling")
client.on_message=on_message
client.connect(broker_address)
client.subscribe('mbox/#',qos=1)
client.loop_start()
time.sleep(0.1) # How long should this time be?
client.loop_stop()
# client.loop(0.1) # why doesn't this do the same action as the previous three lines?
client.disconnect()
time.sleep(5)
Even though this works, I feel that my solution is very hackish. client.loop_start() and client.loop_stop() creates another thread. But when I tried doing client.loop(0.1) instead it didn't work.
So my questions are:
Is there a direct way of polling for a message, instead of the indirect method of using loop_start();…;loop_stop()?
If using loop_start();time.sleep(t);loop_end() is idiomatic, how do I know how long time to sleep for?
Why doesn’t the receiver work when I do loop(0.1); instead of loop_start(); sleep(0.1); loop_stop()`? What is the difference?
Is the receiver guaranteed to receive all the messages?
Is there a better way to implementing this pattern?
Questions answered in order.
No, polling totally defeats the point of a pub/sub protocol like MQTT
You should really be calling client.loop() in a loop, it defaults to only handling 1 packet in the timeout period supplied. QOS 1 needs multiple packets to complete delivery.
calling client.loop(0.1) is going to block for 0.1 seconds waiting for an incoming message, then return. if a message arrives after that 0.1 seconds it's going to sit in the OS TCP/IP stack until you call client.loop() again. If you are not calling it on a regular interval then the broker is going to boot the client because the KeepAlive test will fail. The client loop also handles sending all the subscribe messages.
Assuming the messages are published at QOS > 0 and you have subscribed at QOS > 0 and the client id is kept the same and clean session is false the broker should deliver and messages published while the subscriber is offline
As previously mentioned you need to call client.loop() multiple times per message, as it is you are only calling it once per wake up period. Starting the background thread will handle all the required messages for the length of time you let it run for.
Related
I am trying to send messages from one python script to another using MQTT. One script is a publisher. The second script is a subscriber. I send messages every 0.1 second.
Publisher:
client = mqtt.Client('DataReaderPub')
client.connect('127.0.0.1', 1883, 60)
print("MQTT parameters set.")
# Read from all files
count = 0
for i in range(1,51):
payload = "Hello world" + str(count)
client.publish(testtopic, payload, int(publisherqos))
client.loop()
count = count+1
print(count, ' msg sent: ', payload)
sleep(0.1)
Subscriber:
subclient = mqtt.Client("DynamicDetectorSub")
subclient.on_message = on_message
subclient.connect('127.0.0.1')
subclient.subscribe(testtopic, int(subscriberqos))
subclient.loop_forever()
mosquitto broker version - 3.1
mosquitto.conf has max inflight messages set to 0, persistence true.
publisher QOS = 2
subscriber QOS = 2
topic = 'test' in both scripts
When I run subscriber and publisher in the same script, the messages are sent and received as expected. But when they are in separate scripts, I do not receive all the messages and sometimes no messages. I run subscriber first and then publisher. I have tried subscriber with loop.start() and loop.stop() with waiting for few minutes.
I am unable to debug this problem. Any pointers would be great!
EDIT:
I included client.loop() after publish. -> Same output as before
When I printed out statements in 'on_connect' and 'on_disconnect', I noticed that client mqtt connection gets established and disconnects almost immediately. This happens every second. I even got this message once -
[WinError 10053] An established connection was aborted by the software in your host machine
Keep Alive = 60
Is there any other parameter I should look at?
You need to call the network loop function in the publisher as well so the client actually gets some time to do the IO (And the dual handshake for the QOS2).
Add client.loop() after the call to client.publish() in the client:
import paho.mqtt.client as mqtt
import time
client = mqtt.Client('DataReaderPub')
client.connect('127.0.0.1', 1883, 60)
print("MQTT parameters set.")
# Read from all files
count = 0
for i in range(1,51):
payload = "Hello world" + str(count)
client.publish("test", payload, 2)
client.loop()
count = count+1
print(count, ' msg sent: ', payload)
time.sleep(0.1)
Subscriber code:
import paho.mqtt.client as mqtt
def on_message(client, userdata, msg):
print(msg.topic + " " + str(msg.payload))
subclient = mqtt.Client("DynamicDetectorSub")
subclient.on_message = on_message
subclient.connect('127.0.0.1')
subclient.subscribe("test", 2)
subclient.loop_forever()
When I ran your code, the subscriber was often missing the last packet. I was not otherwise able to reproduce the problems you described.
If I rewrite the publisher like this instead...
from time import sleep
import paho.mqtt.client as mqtt
client = mqtt.Client('DataReaderPub')
client.connect('127.0.0.1', 1883, 60)
print("MQTT parameters set.")
client.loop_start()
# Read from all files
count = 0
for i in range(1,51):
payload = "Hello world" + str(count)
client.publish('test', payload, 2)
count = count+1
print(count, ' msg sent: ', payload)
sleep(0.1)
client.loop_stop()
client.disconnect()
...then I no longer see the dropped packet. I'm using the start_loop/stop_loop methods here, which run the mqtt loop asynchronously. I'm not sure exactly what was causing your dropped packet, but I suspect that the final message was still in the publisher's send queue when the code exits.
It turned out to be a silly bug.
As hardillb suggested I looked at the broker logs. It showed that the subscriber client was already connected.
I am using Pycharm after a really really long time. So I had accidentally ran publisher and subscriber so many times that they were running in parallel in the output console. No wonder they got disconnected since the client IDs were the same. Sorry for the trouble. BTW client.loop() after publish is not needed. Thanks hardillb.
I am implementing a program that listen to a specific topic and react to it when a new message is published by my ESP8266. When a new message is received from ESP8266, my program will trigger the callback and perform a set of tasks. I am publishing two messages in my callback function back to the topic that the Arduino is listening. However, the messages are published only after the function exits.
Thank you for all your time in advance.
I have tried to use loop(1) with a timeout of 1 second inside the callback function. The program will publish the message immediately, but it seems to stuck in the loop. Will someone be able to give me some pointers how can I execute each publish function immediately in my callback function, instead of when the whole callback completes and return to the main loop_forever()?
import paho.mqtt.client as mqtt
import subprocess
import time
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("ESP8266")
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
client.publish("cooking", '4')
client.loop(1)
print("Busy status published back to ESP8266")
time.sleep(5)
print("Starting playback.")
client.publish("cooking", '3')
client.loop(1)
print("Free status published published back to ESP8266")
time.sleep(5)
print("End of playback.")
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("192.168.1.9", 1883, 60)
#client.loop_start()
# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()
You can't do this, you are already in the message handling loop (that's what called the on_message function) at the point you call publish. This will queue the outgoing messages to be handled by the next iteration of the loop, that's why they are sent once on_message returns.
It hangs when you call the loop method because the loop is already running.
You should not be making blocking (sleep) calls in the on_message callback anyway, if you need to do thing that take time, start up a second thread to do these. By doing this you free up the network loop to handle the outgoing publishes as soon as they are made.
I am implementing a program that listen to a specific topic and react to it when a new message is published by my ESP8266. When a new message is received from ESP8266, my program will trigger the callback and perform a set of tasks. I am publishing two messages in my callback function back to the topic that the Arduino is listening. However, the messages are published only after the function exits.
Thank you for all your time in advance.
I have tried to use loop(1) with a timeout of 1 second inside the callback function. The program will publish the message immediately, but it seems to stuck in the loop. Will someone be able to give me some pointers how can I execute each publish function immediately in my callback function, instead of when the whole callback completes and return to the main loop_forever()?
import paho.mqtt.client as mqtt
import subprocess
import time
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("ESP8266")
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
client.publish("cooking", '4')
client.loop(1)
print("Busy status published back to ESP8266")
time.sleep(5)
print("Starting playback.")
client.publish("cooking", '3')
client.loop(1)
print("Free status published published back to ESP8266")
time.sleep(5)
print("End of playback.")
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("192.168.1.9", 1883, 60)
#client.loop_start()
# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()
You can't do this, you are already in the message handling loop (that's what called the on_message function) at the point you call publish. This will queue the outgoing messages to be handled by the next iteration of the loop, that's why they are sent once on_message returns.
It hangs when you call the loop method because the loop is already running.
You should not be making blocking (sleep) calls in the on_message callback anyway, if you need to do thing that take time, start up a second thread to do these. By doing this you free up the network loop to handle the outgoing publishes as soon as they are made.
I have this code that should run indefinitely, however, it doesn't. It keeps on stopping every few hours from the client's side (stop publishing, the loop keeps on running, but nothing is received at the broker), and the only thing that can be done is to rerun it again.
I was advised here to increase the number of max_packets for the loop function, but it's not working and the client stops publishing randomly without continuing. What should be done? I tried the values of 1, 3, 5, 50 and a 1000 but no use.
Code:
client = mqtt.Client()
client.connect(address, 1883, 60)
while True:
data = getdata()
client.publish("$ahmed/",data,0)
client.loop(timeout=1.0, max_packets = 1) # what should be the parameters here so it doesn't stop publishing?
time.sleep(0.2)
In addition to applications messages which are published/subscribed, MQTT also have internal keepalive to avoid problem of half open TCP connections(1). And it is the responsibility of client to make sure keepalives are sent. As per specification, the broker will disconnect clients which doesn't send keepalives in one and half times of keepalive time interval( in absence of other messages).
In addition to sending messages, the loop()* functions also maintains this keepalive traffic flow between broker and client.
A random try: Try using loop_start() once instead of calling loop() in while loop. E.g.
client = mqtt.Client()
client.connect(address)
#runs a thread in background to call loop function internally.
#In addition, this also reconnects to broker on a lost connection.
client.loop_start()
while True:
data = getdata()
client.publish("$ahmed",data)
client.loop_stop()
Just a random guess... has the client disconnected?
In your code you are not handling any callback like on_disconnect(client, userdata, rc) which is called when the client disconnects from the broker.
def on_disconnect_handler(client, userdata, rc):
if rc != 0:
print("Unexpected disconnection.")
client.on_disconnect = on_disconnect_handler
You are also not checking loop() return value:
Returns MQTT_ERR_SUCCESS on success.
Returns >0 on error.
You should do something like
while True:
rc = client.loop(timeout=1.0)
if rc:
# handle loop error here
Just make the client connect every time the loops is through. I have tested it and connecting to the brokers doesn't any significant extra latency on the flow. Since I have to rerun the program to make work it again, I may as well reconnect the client in the loop, so I don't have to do it myself. This is the rawest idea I could come up with that seems to be working without any problems.
client = mqtt.Client()
client.connect(address, 1883, 60)
while True:
client.connect(address, 1883, 60) # just let it reconnect every time it loops ;)!
data = getdata()
client.publish("$ahmed/",data,0)
client.loop(timeout=1.0, max_packets = 1)
time.sleep(0.2)
I just got started with ZMQ. I am designing an app whose workflow is:
one of many clients (who have random PULL addresses) PUSH a request to a server at 5555
the server is forever waiting for client PUSHes. When one comes, a worker process is spawned for that particular request. Yes, worker processes can exist concurrently.
When that process completes it's task, it PUSHes the result to the client.
I assume that the PUSH/PULL architecture is suited for this. Please correct me on this.
But how do I handle these scenarios?
the client_receiver.recv() will wait for an infinite time when server fails to respond.
the client may send request, but it will fail immediately after, hence a worker process will remain stuck at server_sender.send() forever.
So how do I setup something like a timeout in the PUSH/PULL model?
EDIT: Thanks user938949's suggestions, I got a working answer and I am sharing it for posterity.
If you are using zeromq >= 3.0, then you can set the RCVTIMEO socket option:
client_receiver.RCVTIMEO = 1000 # in milliseconds
But in general, you can use pollers:
poller = zmq.Poller()
poller.register(client_receiver, zmq.POLLIN) # POLLIN for recv, POLLOUT for send
And poller.poll() takes a timeout:
evts = poller.poll(1000) # wait *up to* one second for a message to arrive.
evts will be an empty list if there is nothing to receive.
You can poll with zmq.POLLOUT, to check if a send will succeed.
Or, to handle the case of a peer that might have failed, a:
worker.send(msg, zmq.NOBLOCK)
might suffice, which will always return immediately - raising a ZMQError(zmq.EAGAIN) if the send could not complete.
This was a quick hack I made after I referred user938949's answer and http://taotetek.wordpress.com/2011/02/02/python-multiprocessing-with-zeromq/ . If you do better, please post your answer, I will recommend your answer.
For those wanting lasting solutions on reliability, refer http://zguide.zeromq.org/page:all#toc64
Version 3.0 of zeromq (beta ATM) supports timeout in ZMQ_RCVTIMEO and ZMQ_SNDTIMEO. http://api.zeromq.org/3-0:zmq-setsockopt
Server
The zmq.NOBLOCK ensures that when a client does not exist, the send() does not block.
import time
import zmq
context = zmq.Context()
ventilator_send = context.socket(zmq.PUSH)
ventilator_send.bind("tcp://127.0.0.1:5557")
i=0
while True:
i=i+1
time.sleep(0.5)
print ">>sending message ",i
try:
ventilator_send.send(repr(i),zmq.NOBLOCK)
print " succeed"
except:
print " failed"
Client
The poller object can listen in on many recieving sockets (see the "Python Multiprocessing with ZeroMQ" linked above. I linked it only on work_receiver. In the infinite loop, the client polls with an interval of 1000ms. The socks object returns empty if no message has been recieved in that time.
import time
import zmq
context = zmq.Context()
work_receiver = context.socket(zmq.PULL)
work_receiver.connect("tcp://127.0.0.1:5557")
poller = zmq.Poller()
poller.register(work_receiver, zmq.POLLIN)
# Loop and accept messages from both channels, acting accordingly
while True:
socks = dict(poller.poll(1000))
if socks:
if socks.get(work_receiver) == zmq.POLLIN:
print "got message ",work_receiver.recv(zmq.NOBLOCK)
else:
print "error: message timeout"
The send wont block if you use ZMQ_NOBLOCK, but if you try closing the socket and context, this step would block the program from exiting..
The reason is that the socket waits for any peer so that the outgoing messages are ensured to get queued.. To close the socket immediately and flush the outgoing messages from the buffer, use ZMQ_LINGER and set it to 0..
If you're only waiting for one socket, rather than create a Poller, you can do this:
if work_receiver.poll(1000, zmq.POLLIN):
print "got message ",work_receiver.recv(zmq.NOBLOCK)
else:
print "error: message timeout"
You can use this if your timeout changes depending on the situation, instead of setting work_receiver.RCVTIMEO.