I am new to using MQTT. I have set up an Arduino publishing MQTT messages to ActiveMQ. I also have a Python script subscribed to the same topic (using paho) which gets the data from the broker and inserts it into a database.
The problem is if this Python script is down or loses connection for any reason I lose all messages being published while its down. How can I ensure all the data is inserted into the database? I see ActiveMQ has some sort of storage is it possible to retrieve historic data from it?
If you want to ensure that your subscription persists and receives messages even if the subscriber gets disconnected then you need to set CleanSession=false on your MQTT client (assuming you're using MQTT 3.x) and use the same client ID when you reconnect.
Also, if you want messages to survive a broker restart or crash you need to send them with QoS 1.
I think the PubSubClient Arduino library only publishes with fire and forget (QoS 0). I've instead used the stomp.py library to create a durable consumer subscribed to the same topic. If there's no connection on Arduino It's saved to the SD card and if there is connection but no client at the time the messages are stored by ActiveMQ until the client is active.
This solved my problem but I'm still testing the durability
Related
I am seeing behavior in our stomp.py 7.0 client (listener only) where after some idle time of not receiving messages the ActiveMQ 5.15.9 broker seems to drop the client (i.e. looking at the ActiveMQ management console shows zero consumers). The odd thing is the on_disconnected handler of the client never gets called and we have a health check on the client service that checks the connection is_connected(), however it still returns true.
Based on my understanding/research (please correct if any of this is false) this is due to the broker trying to clean-up resources it perceives as inactive. Also based on my research "heartbeating" can be used to avoid this perception on the broker.
I know how to send the heartbeat header from the client and how to check the response from the server/broker (as far as what it expects) in on_connected but my question is how do I send the actual heartbeat from the client to the server/broker? Do I need to send a message on the queue I am listening to? If so how do I send a "heartbeat message" and not have to adjust message handling code in my listeners? Do I send it without a body? Also does the broker need to be configured to accept heartbeats? and If it is not configured would declaring and sending them from the client still result in the broker disconnecting the client?
Heart-beating is part of STOMP 1.2 so as long as your client supports STOMP 1.2 you should be able to configure heart-beating when the connection is established. Also, if your broker supports STOMP 1.2 it should accept the heart-beat header and adjust its behavior accordingly. You shouldn't have to send your own heart-beats. In the absence of any "normal" STOMP frames the client itself should send an EOL as described in the specification.
If your client doesn't support STOMP 1.2 then you should upgrade to a client that does. The STOMP 1.2 spec was released in October 2012, almost a decade ago now so there's been plenty of time to implement support.
I'm developing an application with Paho MQTT and QOS = 1. I know for sure that if the network is not available for a certain period of time the client puts every message in a queue and sends them when the connection is established again.
What I want to understand it's if the queue is saved even when the application itself gets restarted (for instance a machine reboot). If not, how do you suggest to implement it?
Thanks for your help.
How come that even when only one instance of Redis connection created, every time I call publish or subscribe on that instance, it counts it like another client. So when I connect to redis using python
import redis
redis_server = redis.Redis()
it does not recognize it as new client. Only when I call one of these
redis_server.publish("channel", message)
redis_server.subscribe("channel")
I can see that there are 2 clients connected. Are the pub/sub clients treated seperately in redis? Why not registering connected client when the new connection is open?
By default redis-py gives you get a connection pool with only a maximum number of connections. On the first command you issue a real connection will be made and you'll see it appear in the CLIENT LIST on the server.
Whenever any client library for Redis issues a subscribe command, that entire connection is occupied by this, so redis-py is probably creating a separate connection dedicated to this.
This should explain why you see no clients connected, then 2. It's not necessarily 1 connection for every command issued as the connections in the pool will be reused.
I was able to set up a simple socket server and client connection between two devices, with the ability to send and receive values. My issue is with setting up the remote server to accept two clients from the same device, and differentiate the data being received by them.
Specifically, each client will be running a similar code to accept encoder/decoder values from their respective motor. My main program, attached to the server, needs to use the data from each client separately, in order to carry out the appropriate calculations. How do I differentiate the incoming signals coming from both clients?
When the communication isn't heavy between clients and server, one way to do this is to have clients do a handshake to server and have the server enumerate clients and send back id's for communication.
Then the client sends it's id along with any communication it has with server in order for the server to identify it. At least that is what I did.
I'm using a pyzmq pub/sub socket for a server to advertise notifications to client subscribers. It works nicely but I have a question:
Is there any way to use the same socket to send information back to the server? Or do I need a separate socket for that?
Use case: I just want to allow the server to see who's actively subscribing to notifications, so I was hoping I could allow clients to send back periodic "heartbeat" messages. I have a use case where if no clients are listening, I want the server to spawn one. (This is a multiprocess system that uses localhost only.)
You need a separate socket. From the ZMQ guide (http://zguide.zeromq.org/page:all#Pros-and-Cons-of-Pub-Sub):
Killing back-chatter is essential to real scalability. With pub-sub, it's how the pattern can map cleanly to the PGM multicast protocol, which is handled by the network switch. In other words, subscribers don't connect to the publisher at all, they connect to a multicast group on the switch, to which the publisher sends its messages.
In order for this to work, the PUB socket will not send back data to the subscribers (at least not in a way visible to the user. The heartbeating problem was discussed in-depth in the guide: http://zguide.zeromq.org/page:all#The-Asynchronous-Client-Server-Pattern
Also, check out the 7/MDP and 18/MDP protocols (http://rfc.zeromq.org/spec:7 -- this is also discussed in the guide) if you want to keep track of clients.