Develop Reference

How to accumulate messages as mqtt client for 1 second, then save it to a file

my problem is as follows:
I wrote a program that subscribes to a topic, where 2 dictionaries with one key respectively arrive more times a second. On every message they change their value.
I save those dictionaries in a big buffer-dictionary called "Status". What I need is to save a "snapshot" of Status every second into a file.
I tried time.sleep(1) but it drifts. And I don't know how to handle the problem with a schedule due to the already existing client-loop...
I'm pretty new to python and mqtt and would appreciate your help
My code:
import paho.mqtt.client as mqtt
import time
import json
Status = {}
#create client instance
client = mqtt.Client(client_id=None, clean_session=True, transport="tcp")
#connect to broker
client.connect("my_broker", 1883)
#use subscribe() to subscribe to a topic and receive messages
client.subscribe("topic/#", qos=0)
def test1_callback(client, userdata, msg):
msg_dict = json.loads((msg.payload))
Status.update(msg_dict)
client.message_callback_add("topic/test1", test1_callback)
while True:
client.loop_start()
time.sleep(1)
client.loop_stop()
with open('Data.txt', 'a+') as file:
t = time.localtime()
Status["time"]= time.strftime("%H:%M:%S", t)
file.write(str(Status["time"]) + " ")
file.write(str(Status["key1"]) + " ")
file.write(str(Status["key2"]) + " ")
client.loop_start()

Instead of manually stopping the networking thread I would prefer using a timer which fires every second. In addition it might be a good idea to lock the data when storing it to a file - otherwise there might occur an update in between:
# ...
import threading
def test1_callback(client, userdata, msg):
msg_dict = json.loads((msg.payload))
lock.acquire()
Status.update(msg_dict)
lock.release()
def timer_event():
lock.acquire()
# save to file here
lock.release()
# restart timer
threading.Timer(1, timer_event).start()
Status = {}
lock = threading.Lock()
# client initialization
# ...
client.loop_start()
threading.Timer(1, timer_event).start()
while True:
pass
But this won't prevent your stored value to drift away because the topic is apparently published too frequently so your subscriber (or even the broker) is not able to handle a message fast enough.
So you might want to reduce the interval in which this topic is published. Also notice that you subscribed to a multi-level topic - even if the topics besides "topic/test1" are not handled in your code they still cause load for the broker and the subscribing client

How to implement a "mailbox" service through mqtt?

I'm a newby to all things mqtt and as a first exercise I wanted to create a “mailbox” service through a persistent mqtt session. The incentive is a low power ESP8266 device that sleeps most of the time and periodically wakes up and checks if there are any pending commands for it.
I tried implementing this through a sender and receiver on my Linux host with python and paho mqtt. Mosquitto is running in the background as the broker.
First here is the "mbox" sender, which sends another message every time Enter is pressed.
import paho.mqtt.client as mqtt
broker_address='127.0.0.1'
client = mqtt.Client('MBoxClient')
client.connect(broker_address)
counter = 1
while True:
print('Press Enter to send msg #'+str(counter)+': ', end='')
if input().startswith('q'):
break
client.publish("mbox/mail","Hello "+str(counter), qos=1)
counter += 1
client.disconnect()
print('done!')
And here is my mbox receiver:
import paho.mqtt.client as mqtt
import time
def on_message(client, userdata, message):
print("message:", message.topic + ': ' + str(message.payload.decode("utf-8")))
print('I\'m listening for mbox messages!')
broker_address="127.0.0.1"
client_name='mbox'
is_first=True
while 1:
client = mqtt.Client(client_name, clean_session=is_first)
is_first=False
print("polling")
client.on_message=on_message
client.connect(broker_address)
client.subscribe('mbox/#',qos=1)
client.loop_start()
time.sleep(0.1) # How long should this time be?
client.loop_stop()
# client.loop(0.1) # why doesn't this do the same action as the previous three lines?
client.disconnect()
time.sleep(5)
Even though this works, I feel that my solution is very hackish. client.loop_start() and client.loop_stop() creates another thread. But when I tried doing client.loop(0.1) instead it didn't work.
So my questions are:
Is there a direct way of polling for a message, instead of the indirect method of using loop_start();…;loop_stop()?
If using loop_start();time.sleep(t);loop_end() is idiomatic, how do I know how long time to sleep for?
Why doesn’t the receiver work when I do loop(0.1); instead of loop_start(); sleep(0.1); loop_stop()`? What is the difference?
Is the receiver guaranteed to receive all the messages?
Is there a better way to implementing this pattern?

Questions answered in order.
No, polling totally defeats the point of a pub/sub protocol like MQTT
You should really be calling client.loop() in a loop, it defaults to only handling 1 packet in the timeout period supplied. QOS 1 needs multiple packets to complete delivery.
calling client.loop(0.1) is going to block for 0.1 seconds waiting for an incoming message, then return. if a message arrives after that 0.1 seconds it's going to sit in the OS TCP/IP stack until you call client.loop() again. If you are not calling it on a regular interval then the broker is going to boot the client because the KeepAlive test will fail. The client loop also handles sending all the subscribe messages.
Assuming the messages are published at QOS > 0 and you have subscribed at QOS > 0 and the client id is kept the same and clean session is false the broker should deliver and messages published while the subscriber is offline
As previously mentioned you need to call client.loop() multiple times per message, as it is you are only calling it once per wake up period. Starting the background thread will handle all the required messages for the length of time you let it run for.

How to publish and subscribe the latest message correctly using pyzmq?

I have a process A that publishes a message constantly and processes B and C subscribe to the topic and get the latest message published by the publisher in process A.
So, I set zmq.CONFLATE to both publisher and subscriber. However, I found that one subscriber was not able to receive messages.
def publisher(sleep_time=1.0, port="5556"):
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.setsockopt(zmq.CONFLATE, 1)
socket.bind("tcp://*:%s" % port)
print ("Running publisher on port: ", port)
while True:
localtime = time.asctime( time.localtime(time.time()))
string = "Message published time: {}".format(localtime)
socket.send_string("{}".format(string))
time.sleep(sleep_time)
def subscriber(name="sub", sleep_time=1, ports="5556"):
print ("Subscriber Name: {}, Sleep Time: {}, Port: {}".format(name, sleep_time, ports))
context = zmq.Context()
print ("Connecting to publisher with ports %s" % ports)
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.CONFLATE, 1)
socket.setsockopt_string(zmq.SUBSCRIBE, "")
socket.connect ("tcp://localhost:%s" % ports)
while True:
message = socket.recv()
localtime = time.asctime( time.localtime(time.time()))
print ("\nSubscriber [{}]\n[RECV]: {} at [TIME]: {}".format(name, message, localtime))
time.sleep(sleep_time)
if __name__ == "__main__":
Process(target=publisher).start()
Process(target=subscriber, args=("SUB1", 1.2, )).start()
Process(target=subscriber, args=("SUB2", 1.1, )).start()
I tried to unset the socket.setsockopt(zmq.CONFLATE, 1) in the publisher, and that seemed to solve the problem. Both subscribers in processes B and C could receive messages and the messages seemed to be the latest ones.
I'm trying to find out why setting the publisher with CONFLATE caused the problem I had. I could not find information about it. Does anyone know what causes this behavior?
Also, I want to know, in the situation of one publisher to multiple subscribers, what is the correct code setup, so that subscriber can always get the latest messages?

It's most likely a timing issue, the ZMQ_CONFLATE socket option limits the inbound and outbound queue to 1 message.
The way PUB/SUB works is the subscriber sends a subscription message to the publisher when you set the ZMQ_SUBSCRIBE option. If you start both subscribers at the same time then its possible that one of the subscription messages that arrived on the publisher queue will be discarded.
Try adding a sleep between the starting each subscriber.
From the zeromq docs
If set, a socket shall keep only one message in its inbound/outbound
queue, this message being the last message received/the last message
to be sent. Ignores ZMQ_RCVHWM and ZMQ_SNDHWM options. Does not
support multi-part messages, in particular, only one part of it is
kept in the socket internal queue.
I am not saying this is the solution to you problem, but if that is the case we may need to post a change to libzmq to make the conflate options more granular so you can choose if conflate should be applied to inbound or outbound queues.

There is a manner to get "Last message only" option in ZMQ Subscribe socket (using CONFLATE option).
You need it on the subscriber side.
Here is an example:
import zmq
port = "5556"
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.SUBSCRIBE, '')
socket.setsockopt(zmq.CONFLATE, 1) # last msg only.
socket.connect("tcp://localhost:%s" % port) # must be placed after above options.
while True:
data = socket.recv()
print data
On the other word, I removed any buffered queue in subscriber code.
[In Additional]:
With the zmq.SNDBUF and zmq.RCVBUF options we could set a limit on ZMQ buffer size. (More complete and an example)

Publish and subscribe both ways using MQTT Python

I currently have a Python program written on the Raspberry Pi 3 to read in humidity and temperature sensor data and publish this data to a topic. I can then receive this data using my laptop. Here is my code for reading sensor data and publishing it to a topic from my Raspberry Pi:
import RPi.GPIO as GPIO
import time
import json
import Adafruit_DHT as dht
import math
import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt
# Creating the JSON Objects
dht22 = {}
arduino = {}
dht22Temp = []
dht22Hum = []
arduinoLED = []
dht22['temperature'] = dht22Temp
dht22['humidity'] = dht22Hum
dht22['sensor'] = 'DHT22'
arduino['blink'] = arduinoLED
arduino['actuator'] = 'arduinoLED'
# Timing constants
E_PULSE = 0.0005
E_DELAY = 0.0005
def main():
# Main program block
while True:
h, t = dht.read_retry(dht.DHT22, 17) //Reading humidity and temp data from GPIO17
t = round(t,2)
h = round(h,2)
if t > 25:
if len(arduinoLED) == 3:
arduinoLED.pop(0)
arduinoLED.append("true")
else:
arduinoLED.append("true")
else:
if len(arduinoLED) == 3:
arduinoLED.pop(0)
arduinoLED.append("false")
else:
arduinoLED.append("false")
if len(dht22Temp) == 3:
dht22Temp.pop(0)
dht22Temp.append(t)
else:
dht22Temp.append(t)
if len(dht22Hum) == 3:
dht22Hum.pop(0)
dht22Hum.append(h)
else:
dht22Hum.append(h)
# lm35dzTemp.append(tempc)
# Publishing sensor information by JSON converting object to a string
publish.single("topic/sensorTemperature", json.dumps(dht22), hostname = "test.mosquitto.org")
publish.single("topic/sensorTemperature", json.dumps(arduino), hostname = "test.mosquitto.org")
# Printing JSON objects
print(dht22)
print(arduino)
time.sleep(2)
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
pass
finally:
GPIO.cleanup()
Here is my code for subscribing and receiving data from my laptop:
import paho.mqtt.client as mqtt
import json
# This is the Subscriber
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
client.subscribe("topic/sensorTemperature")
def on_message(client, userdata, msg):
print(json.loads(msg.payload)) #converting the string back to a JSON object
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("test.mosquitto.org", 1883, 60)
client.loop_forever()
What I want to do is now publish something from my laptop (perhaps in the same code as the subscriber, or in a separate file that will just publish a message to the same topic - "topic/sensorTemperature"). But my question is: how do I also publish and subscribe to messages on my Raspberry Pi (in my first code that I published)? Since I am publishing messages in an infinite loop to my laptop, I will also need an infinite loop to subscribe to the same (or different topic) to receive messages. How do you run two of these loops at once? Will I need two different threads?
Thank you.

As suggested by Sergey you can use loop_start to create a separate thread for receiving messages.
Here is how your main function will look like:
def main():
# Create a new client for receiving messages
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.subscribe(topic)
client.connect(mqttserver)
client.loop_start()
while True:
#code for publishing
pass

The easiest way is to start another Python process (similar to your laptop's script) on Raspberry in parallel, handling messages received from laptop.
But if you want to implement everything in one script, you can extend your second code fragment (processing messages) with implementation of first fragment (publishing sensors data).
Of course, you can't use loop_forever() in this case. When you call loop_forever(), it will never return until client calls disconnect(), so you can't process received messages (main thread is blocked). Paho client also has routines loop() and loop_start()/loop_stop() to control over network loop.
Take a look on them:
1) Function loop() can take timeout as an argument. It will block until new message arrives or time is out. In first case - preform the processing of received message and calculate time until the next publish. Pass this time as parameter to loop() again. In second case, just publish data and call loop() with time until next publish (2 seconds in your example).
2) loop_start()/loop_stop() starts and stops background thread doing job of sending and receiving(and processing) data for you. Create client, register on_message() callback, connect/subscribe, and call loop_start() to start this thread. Main thread is free for you now - use it with logic of first fragment (loop with 2 seconds sleep).

Simply put your code from subscribing script into publishing script before while True: and replace loop_forever() with loop_start(). Use loop_stop() when you script is exitting before GPIO.cleanup().

Python WebSocket Client connecting but not sending messages

While using websocket client to send test messages to a django server, I cannot get a script to work which can both send and receive messages.
The following python script is what I have attempted:
import websocket
import threading
import json
from time import sleep
# handle message event
def on_message(ws, message):
print("message recieved: %s" % message)
# handle close event
def on_close(ws):
print("channel closed")
# execute as main script
if __name__ == "__main__":
websocket.enableTrace(True)
# new app object connecting to headstation
ws = websocket.WebSocketApp("ws://192.168.0.106:8000/?testI123", on_message = on_message, on_close = on_close)
# run in a new thread - kill if script ends
ws_listener = threading.Thread(target=ws.run_forever())
ws_listener.daemon = True
# start second thread
ws_listener.start()
# attempt connection 5 times
timeout = 5
while not ws.sock.connected and timeout:
sleep(1)
timeout -= 1
# error on timeout
if (timeout == 0):
print("Connection to server timed out")
print("test 1")
# periodically send test message to server
message_num = 0
while ws.sock.connected:
# send node id and message
message = 'hello %d'%message_num
ws.send(message)
sleep(1)
message_num += 1
This connections successfully, indicted by the server, and receives messages sent from the server, but does not send anything.
Periodically, something like this is displayed on the terminal:
send: b'\x8a\x84\xe2\xe9\xa8\xe2\x8f\xdc\xe2\x84'
If I simply use
ws = websocket.WebSocket()
ws.connect(url)
ws.send("hello")
then this works perfectly. Suggesting it is something wrong with my little python script displayed above.

Found the problem, stupid mistake of course:
ws_listener = threading.Thread(target=ws.run_forever())
should be:
ws_listener = threading.Thread(target=ws.run_forever)
without parentheses.
First one passes result of ws.run_forever to the target, second one sets ws.run_forever as the target, which was the intended outcome.