I currently have a Python program written on the Raspberry Pi 3 to read in humidity and temperature sensor data and publish this data to a topic. I can then receive this data using my laptop. Here is my code for reading sensor data and publishing it to a topic from my Raspberry Pi:
import RPi.GPIO as GPIO
import time
import json
import Adafruit_DHT as dht
import math
import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt
# Creating the JSON Objects
dht22 = {}
arduino = {}
dht22Temp = []
dht22Hum = []
arduinoLED = []
dht22['temperature'] = dht22Temp
dht22['humidity'] = dht22Hum
dht22['sensor'] = 'DHT22'
arduino['blink'] = arduinoLED
arduino['actuator'] = 'arduinoLED'
# Timing constants
E_PULSE = 0.0005
E_DELAY = 0.0005
def main():
# Main program block
while True:
h, t = dht.read_retry(dht.DHT22, 17) //Reading humidity and temp data from GPIO17
t = round(t,2)
h = round(h,2)
if t > 25:
if len(arduinoLED) == 3:
arduinoLED.pop(0)
arduinoLED.append("true")
else:
arduinoLED.append("true")
else:
if len(arduinoLED) == 3:
arduinoLED.pop(0)
arduinoLED.append("false")
else:
arduinoLED.append("false")
if len(dht22Temp) == 3:
dht22Temp.pop(0)
dht22Temp.append(t)
else:
dht22Temp.append(t)
if len(dht22Hum) == 3:
dht22Hum.pop(0)
dht22Hum.append(h)
else:
dht22Hum.append(h)
# lm35dzTemp.append(tempc)
# Publishing sensor information by JSON converting object to a string
publish.single("topic/sensorTemperature", json.dumps(dht22), hostname = "test.mosquitto.org")
publish.single("topic/sensorTemperature", json.dumps(arduino), hostname = "test.mosquitto.org")
# Printing JSON objects
print(dht22)
print(arduino)
time.sleep(2)
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
pass
finally:
GPIO.cleanup()
Here is my code for subscribing and receiving data from my laptop:
import paho.mqtt.client as mqtt
import json
# This is the Subscriber
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
client.subscribe("topic/sensorTemperature")
def on_message(client, userdata, msg):
print(json.loads(msg.payload)) #converting the string back to a JSON object
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("test.mosquitto.org", 1883, 60)
client.loop_forever()
What I want to do is now publish something from my laptop (perhaps in the same code as the subscriber, or in a separate file that will just publish a message to the same topic - "topic/sensorTemperature"). But my question is: how do I also publish and subscribe to messages on my Raspberry Pi (in my first code that I published)? Since I am publishing messages in an infinite loop to my laptop, I will also need an infinite loop to subscribe to the same (or different topic) to receive messages. How do you run two of these loops at once? Will I need two different threads?
Thank you.
As suggested by Sergey you can use loop_start to create a separate thread for receiving messages.
Here is how your main function will look like:
def main():
# Create a new client for receiving messages
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.subscribe(topic)
client.connect(mqttserver)
client.loop_start()
while True:
#code for publishing
pass
The easiest way is to start another Python process (similar to your laptop's script) on Raspberry in parallel, handling messages received from laptop.
But if you want to implement everything in one script, you can extend your second code fragment (processing messages) with implementation of first fragment (publishing sensors data).
Of course, you can't use loop_forever() in this case. When you call loop_forever(), it will never return until client calls disconnect(), so you can't process received messages (main thread is blocked). Paho client also has routines loop() and loop_start()/loop_stop() to control over network loop.
Take a look on them:
1) Function loop() can take timeout as an argument. It will block until new message arrives or time is out. In first case - preform the processing of received message and calculate time until the next publish. Pass this time as parameter to loop() again. In second case, just publish data and call loop() with time until next publish (2 seconds in your example).
2) loop_start()/loop_stop() starts and stops background thread doing job of sending and receiving(and processing) data for you. Create client, register on_message() callback, connect/subscribe, and call loop_start() to start this thread. Main thread is free for you now - use it with logic of first fragment (loop with 2 seconds sleep).
Simply put your code from subscribing script into publishing script before while True: and replace loop_forever() with loop_start(). Use loop_stop() when you script is exitting before GPIO.cleanup().
Related
my problem is as follows:
I wrote a program that subscribes to a topic, where 2 dictionaries with one key respectively arrive more times a second. On every message they change their value.
I save those dictionaries in a big buffer-dictionary called "Status". What I need is to save a "snapshot" of Status every second into a file.
I tried time.sleep(1) but it drifts. And I don't know how to handle the problem with a schedule due to the already existing client-loop...
I'm pretty new to python and mqtt and would appreciate your help
My code:
import paho.mqtt.client as mqtt
import time
import json
Status = {}
#create client instance
client = mqtt.Client(client_id=None, clean_session=True, transport="tcp")
#connect to broker
client.connect("my_broker", 1883)
#use subscribe() to subscribe to a topic and receive messages
client.subscribe("topic/#", qos=0)
def test1_callback(client, userdata, msg):
msg_dict = json.loads((msg.payload))
Status.update(msg_dict)
client.message_callback_add("topic/test1", test1_callback)
while True:
client.loop_start()
time.sleep(1)
client.loop_stop()
with open('Data.txt', 'a+') as file:
t = time.localtime()
Status["time"]= time.strftime("%H:%M:%S", t)
file.write(str(Status["time"]) + " ")
file.write(str(Status["key1"]) + " ")
file.write(str(Status["key2"]) + " ")
client.loop_start()
Instead of manually stopping the networking thread I would prefer using a timer which fires every second. In addition it might be a good idea to lock the data when storing it to a file - otherwise there might occur an update in between:
# ...
import threading
def test1_callback(client, userdata, msg):
msg_dict = json.loads((msg.payload))
lock.acquire()
Status.update(msg_dict)
lock.release()
def timer_event():
lock.acquire()
# save to file here
lock.release()
# restart timer
threading.Timer(1, timer_event).start()
Status = {}
lock = threading.Lock()
# client initialization
# ...
client.loop_start()
threading.Timer(1, timer_event).start()
while True:
pass
But this won't prevent your stored value to drift away because the topic is apparently published too frequently so your subscriber (or even the broker) is not able to handle a message fast enough.
So you might want to reduce the interval in which this topic is published. Also notice that you subscribed to a multi-level topic - even if the topics besides "topic/test1" are not handled in your code they still cause load for the broker and the subscribing client
The example program is below. It is whittled down to the smallest pieces that will show the problem which is a very long "Pause" after publishing a topic and message. When run, the program will create a little window with two buttons. One just prints something, the other, (FUNC) publishes a message. I can see that the topic / message gets to the broker, and since I am subscribed I can see the message come back. The program "freezes" for about 6 seconds, then data starts coming in again.
I have another RPI running the broker and yet more RPIs running on the same network (Ethernet with Fixed IPs). You will see in my on_message function I am also reading data from other RPIs so that I can see the pace of the information flowing. Those other RPIs are sending packets at 1 second intervals.
I have run this program on an RPI4 with Buster and on an RPI3 with Stretch, the behaviour is the same. In the actual program I am creating (which is about 6000 lines of code across all modules), instead of having just a 6 second "Freeze", the program Freezes and never starts running again.
import tkinter as tk
from tkinter import *
import tkinter.filedialog
from tkinter import ttk
from time import sleep
################################################################################
### START OF MQTT
import paho.mqtt.client as paho
broker = "192.168.2.170"
port = 1883
Mq_C = paho.Client("RzBrain")
Mq_C.connect(broker,port)
Mq_C.loop_start()
def on_log(client, userdata, level, buf):
print("Log: ",buf)
def on_publish(client, userdata, result):
print("Data published \n")
pass
def on_message(client,userdata, message):
Topic = message.topic
Raw_Msg = message
try:
Msg = message.payload.decode("utf-8")
except:
print("Msg error on MQTT Read")
if Topic == "RSV/TEMP":
print(Msg)
elif Topic == "RSV/HUMIDITY":
print(Msg)
elif Topic == "RSV/PRESSURE":
print(Msg)
elif Topic == "RSV/PHRASE":
print("\n",Msg)
Mq_C.on_publish = on_publish
Mq_C.on_message = on_message
Mq_C.subscribe("RSV/TEMP") #This data is being published on my network from another RPI
Mq_C.subscribe("RSV/HUMIDITY") #This data is being published on my network from another RPI
Mq_C.subscribe("RSV/PRESSURE") #This data is being published on my network from another RPI
Mq_C.subscribe("RSV/PHRASE") # This is the return trip of a published topic / message
def Call_The_Function():
print("Function called")
Pay_Load = "Wuz Up"
Mq_C.publish("RSV/PHRASE",Pay_Load)
print("Message sent and moving on \n")
def Call_The_Print():
print("Print Function called")
print("Print moving on \n")
WinRoot = tk.Tk()
WinRoot.geometry('300x200+0+0')
WinRoot.title("TEST")
canvas = tk.Canvas(WinRoot, width=300, height=200, bg="gray30")
canvas.place(x=0,y=0)
MQTT_B = tk.Button(WinRoot,text="FUNC",command=Call_The_Function,bg="gray30",fg="red",height=2, width=7,highlightbackground="gray60",relief="raised",borderwidth=3)
MQTT_B.place(x=170,y=100)
Print_B = tk.Button(WinRoot,text="Print",command=Call_The_Print,bg="gray30",fg="red",height=2, width=7,highlightbackground="gray60",relief="raised",borderwidth=3)
Print_B.place(x=50,y=100)
WinRoot.mainloop()
I am trying to send messages from one python script to another using MQTT. One script is a publisher. The second script is a subscriber. I send messages every 0.1 second.
Publisher:
client = mqtt.Client('DataReaderPub')
client.connect('127.0.0.1', 1883, 60)
print("MQTT parameters set.")
# Read from all files
count = 0
for i in range(1,51):
payload = "Hello world" + str(count)
client.publish(testtopic, payload, int(publisherqos))
client.loop()
count = count+1
print(count, ' msg sent: ', payload)
sleep(0.1)
Subscriber:
subclient = mqtt.Client("DynamicDetectorSub")
subclient.on_message = on_message
subclient.connect('127.0.0.1')
subclient.subscribe(testtopic, int(subscriberqos))
subclient.loop_forever()
mosquitto broker version - 3.1
mosquitto.conf has max inflight messages set to 0, persistence true.
publisher QOS = 2
subscriber QOS = 2
topic = 'test' in both scripts
When I run subscriber and publisher in the same script, the messages are sent and received as expected. But when they are in separate scripts, I do not receive all the messages and sometimes no messages. I run subscriber first and then publisher. I have tried subscriber with loop.start() and loop.stop() with waiting for few minutes.
I am unable to debug this problem. Any pointers would be great!
EDIT:
I included client.loop() after publish. -> Same output as before
When I printed out statements in 'on_connect' and 'on_disconnect', I noticed that client mqtt connection gets established and disconnects almost immediately. This happens every second. I even got this message once -
[WinError 10053] An established connection was aborted by the software in your host machine
Keep Alive = 60
Is there any other parameter I should look at?
You need to call the network loop function in the publisher as well so the client actually gets some time to do the IO (And the dual handshake for the QOS2).
Add client.loop() after the call to client.publish() in the client:
import paho.mqtt.client as mqtt
import time
client = mqtt.Client('DataReaderPub')
client.connect('127.0.0.1', 1883, 60)
print("MQTT parameters set.")
# Read from all files
count = 0
for i in range(1,51):
payload = "Hello world" + str(count)
client.publish("test", payload, 2)
client.loop()
count = count+1
print(count, ' msg sent: ', payload)
time.sleep(0.1)
Subscriber code:
import paho.mqtt.client as mqtt
def on_message(client, userdata, msg):
print(msg.topic + " " + str(msg.payload))
subclient = mqtt.Client("DynamicDetectorSub")
subclient.on_message = on_message
subclient.connect('127.0.0.1')
subclient.subscribe("test", 2)
subclient.loop_forever()
When I ran your code, the subscriber was often missing the last packet. I was not otherwise able to reproduce the problems you described.
If I rewrite the publisher like this instead...
from time import sleep
import paho.mqtt.client as mqtt
client = mqtt.Client('DataReaderPub')
client.connect('127.0.0.1', 1883, 60)
print("MQTT parameters set.")
client.loop_start()
# Read from all files
count = 0
for i in range(1,51):
payload = "Hello world" + str(count)
client.publish('test', payload, 2)
count = count+1
print(count, ' msg sent: ', payload)
sleep(0.1)
client.loop_stop()
client.disconnect()
...then I no longer see the dropped packet. I'm using the start_loop/stop_loop methods here, which run the mqtt loop asynchronously. I'm not sure exactly what was causing your dropped packet, but I suspect that the final message was still in the publisher's send queue when the code exits.
It turned out to be a silly bug.
As hardillb suggested I looked at the broker logs. It showed that the subscriber client was already connected.
I am using Pycharm after a really really long time. So I had accidentally ran publisher and subscriber so many times that they were running in parallel in the output console. No wonder they got disconnected since the client IDs were the same. Sorry for the trouble. BTW client.loop() after publish is not needed. Thanks hardillb.
I have a Python script wherein I'm connected to a MQTT server. I'm expecting to get a message through the topic where I'm subscribed to, but should I not receive the message, I want to terminate the script entirely.
The script I'm working with looks as follows:
#!/usr/bin/python
import sys
import json
import paho.mqtt.client as mqtt
def on_message(client, userdata, msg):
if msg.topic == "discovery":
data = json.loads(msg.payload)
serial = data['serial']
print "test successful!"
sys.exit(0)
def on_connect(client, userdata, flags, rc):
client.subscribe([("discovery", 2)])
client = mqtt.Client()
try:
client.connect('localhost', 4444)
except:
print "ERROR: Could not connect to MQTT
client.on_connect = on_connect
client.on_message = on_message
client.loop_forever()
I have tried using a while True statement to loop and figure the time passing in between starting the script and it getting the message, but it seemed to (obviously) not escape the loop even while it gets the message through.
Is there a way that I can say how long it be connected for, and when it exceeds that time, just terminate the script entirely?
Or perhaps, is there a way (as I tried before) to make a loop but also consider the message coming through while in the loop?
Thanks for your suggestions!
Try something like this
It should wait for about 5 seconds for an incoming message then quit. You can adjust the wait time by changing the value waitTime just before the while loop
I have used the version of the mqtt network loop function that only runs for a short time and put it in a while loop. The loop also checks elapsed time and disconnects the client cleanly before bailing out of the loop. I also added a clean client exit for when a message is received.
#!/usr/bin/python
import sys
import json
import paho.mqtt.client as mqtt
import time
def on_message(client, userdata, msg):
if msg.topic == "discovery":
data = json.loads(msg.payload)
serial = data['serial']
print "test successful!"
client.disconnect()
sys.exit(0)
def on_connect(client, userdata, flags, rc):
client.subscribe([("discovery", 2)])
client = mqtt.Client()
try:
client.connect('localhost', 4444)
except:
print "ERROR: Could not connect to MQTT"
client.on_connect = on_connect
client.on_message = on_message
startTime = time.time()
waitTime = 5
while True:
client.loop()
elapsedTime = time.time() - startTime
if elapsedTime > waitTime:
client.disconnect()
break
I have this code that should run indefinitely, however, it doesn't. It keeps on stopping every few hours from the client's side (stop publishing, the loop keeps on running, but nothing is received at the broker), and the only thing that can be done is to rerun it again.
I was advised here to increase the number of max_packets for the loop function, but it's not working and the client stops publishing randomly without continuing. What should be done? I tried the values of 1, 3, 5, 50 and a 1000 but no use.
Code:
client = mqtt.Client()
client.connect(address, 1883, 60)
while True:
data = getdata()
client.publish("$ahmed/",data,0)
client.loop(timeout=1.0, max_packets = 1) # what should be the parameters here so it doesn't stop publishing?
time.sleep(0.2)
In addition to applications messages which are published/subscribed, MQTT also have internal keepalive to avoid problem of half open TCP connections(1). And it is the responsibility of client to make sure keepalives are sent. As per specification, the broker will disconnect clients which doesn't send keepalives in one and half times of keepalive time interval( in absence of other messages).
In addition to sending messages, the loop()* functions also maintains this keepalive traffic flow between broker and client.
A random try: Try using loop_start() once instead of calling loop() in while loop. E.g.
client = mqtt.Client()
client.connect(address)
#runs a thread in background to call loop function internally.
#In addition, this also reconnects to broker on a lost connection.
client.loop_start()
while True:
data = getdata()
client.publish("$ahmed",data)
client.loop_stop()
Just a random guess... has the client disconnected?
In your code you are not handling any callback like on_disconnect(client, userdata, rc) which is called when the client disconnects from the broker.
def on_disconnect_handler(client, userdata, rc):
if rc != 0:
print("Unexpected disconnection.")
client.on_disconnect = on_disconnect_handler
You are also not checking loop() return value:
Returns MQTT_ERR_SUCCESS on success.
Returns >0 on error.
You should do something like
while True:
rc = client.loop(timeout=1.0)
if rc:
# handle loop error here
Just make the client connect every time the loops is through. I have tested it and connecting to the brokers doesn't any significant extra latency on the flow. Since I have to rerun the program to make work it again, I may as well reconnect the client in the loop, so I don't have to do it myself. This is the rawest idea I could come up with that seems to be working without any problems.
client = mqtt.Client()
client.connect(address, 1883, 60)
while True:
client.connect(address, 1883, 60) # just let it reconnect every time it loops ;)!
data = getdata()
client.publish("$ahmed/",data,0)
client.loop(timeout=1.0, max_packets = 1)
time.sleep(0.2)