I am quite new to MQTT and brokers, but I am having an issue with VerneMQ not sending offline messages to clients. Here is my setup. I have a backend written in Python which is using the Paho Eclipse MQTT library's single() method to send messages to a connected client. The client, a virtual machine on my development station, has a client which is written in go-lang, using paho.mqtt.golang to connect to the broker and subscribe.
The call to single() on the backend looks like this:
def send_message(device_id, payload):
token = get_jwt('my_token').decode()
mqtt.single(
f'commands/{device_id}',
payload=payload,
qos=2,
hostname=MESSAGING_HOST,
port=8080,
client_id='client_id',
auth={'username': 'username', 'password': f'Bearer {token}'},
transport='websockets'
)
On the client, the session is established with the following options:
func startListenerRun(cmd *cobra.Command, args []string) {
//mqtt.DEBUG = log.New(os.Stdout, "", 0)
mqtt.ERROR = log.New(os.Stdout, "", 0)
opts := mqtt.NewClientOptions().AddBroker(utils.GetMessagingHost()).SetClientID(utils.GetClientId())
opts.SetKeepAlive(20 * time.Second)
opts.SetDefaultPublishHandler(f)
opts.SetPingTimeout(5 * time.Second)
opts.SetCredentialsProvider(credentialsProvider)
opts.SetConnectRetry(false)
opts.SetAutoReconnect(true)
opts.willQos=2
opts.SetCleanSession(false)
I am not showing all the code, but hopefully enough to illustrate how the session is being set up.
I am running VerneMQ as a docker container. We are using the following environment variables to change configuration defaults in the Dockerfile:
ENV DOCKER_VERNEMQ_PLUGINS.vmq_diversity on
ENV DOCKER_VERNEMQ_VMQ_DIVERSITY.myscript1.file /etc/vernemq/authentication.lua
ENV DOCKER_VERNEMQ_VMQ_ACL.acl_file /etc/vernemq/vmq.acl
ENV DOCKER_VERNEMQ_PLUGINS.vmq_acl on
ENV DOCKER_VERNEMQ_RETRY_INTERVAL=3000
As long as the client has an active connection to the broker, the server's published messages arrive seamlessly. However, if I manually close the client's connection to the broker, and then publish a message on the backend to that client, when the client's connection reopens, the message is not resent by the broker. As I said, I am new to MQTT, so I may need to configure additional options, but so far I've yet to determine which. Can anyone shed any light on what might be happening on my setup that would cause offline messages to not be sent? Thanks for any information.
As thrashed out in the comments
Messages will only be queued for an offline client that has subscribed at greater than QOS 0
More details can be found here
You need to make QOS to 1 or 2 depending on your requirement and also you can use --retain flag which is quite useful. retain flag will make sure that last message will be delivered irrespective of any failure. You can know the last status of device. Check this http://www.steves-internet-guide.com/mqtt-retained-messages-example/
Related
I am trying to test a Mosquitto MQTT server. My plan is to hopefully get it to ingest some notifications from our monitoring system in my work IT dept. Seems perfect for the task. I was able to install Mosquitto on a Ubuntu 20.04 server, and testing with the mosquitto_sub and mosquitto_pub clients worked right away (see attached screenshot). I also was able to Google around and set up some Python script using paho mqtt module for subscribe and publish. I set up the subscribe script as a service and it runs fine. I can receive published messages from both mosquitto_pub and the "publish" Python script...
Now where the issues come up is when I try publishing to the Mosquitto server from a remote computer within my office. For the life of me I cannot determine what the issue is. :(
I am able to telnet to the Mosquitto server on port 1883 so it does not seem to be a firewall issue on the Mosquitto server... Whenever I try to use the publish Python script from the remote computer, it executes but it does not work -- no messages are received at the subscriber running on the server. One odd thing is that the "on_connect" portion on the script does not work when the script is run on the remote computer; no "Connected to MQTT Broker!" message is printed when the script is run remotely. (it does work when it's run locally on the Mosquitto server).
I am attaching a screenshot of my /etc/mosquitto/conf.d/default.conf file. I have tried all sorts of combinations on the config and each time things work when run on the server itself but not remotely. I tried putting the IP address for the server on the config as "listener 1883 10.x.x.x". (Note for this post using "10.x.x.x" instead of my real IP address.) The last thing I tried was putting the adapter name, and still does not work. As you can see I have also defined authentication (usernames, passwords) for a pubuser and subuser. The account info does work with the mosquitto_sub and mosquitto_pub clients, and when used in Python scripts (when script are run on Mosquitto server) so I don't think it's an account info issue. All signs are pointing to either some misconfiguration I have in Mosquitto, or some firewall policy in my work location. Please help!
Please share any tips or possible fixes for me. I might be doing something wrong in my config; please do let me know if you see something amiss. I am a newbie with Mosquitto and MQTT but would love for it to work, kind of ruins the point if I cannot publish to the server form a remote computer though :)..
Below is example Python code I am using on the remote computer to publish (works run locally on the Mosquitto server). I redacted the IP address of the server with 10.x.x.x).
When I run the script from the remote computer I just get "Failed to send message to topic python/mqtt".
:
import random
import time
from paho.mqtt import client as mqtt_client
broker = '10.x.x.x'
port = 1883
topic = "python/mqtt"
# generate client ID with pub prefix randomly
#client_id = f'python-mqtt-{random.randint(0, 1000)}'
#Set the static client id
client_id = 'python-mqtt01'
username = 'pubuser'
password = 'something'
def connect_mqtt():
def on_connect(client, userdata, flags, rc):
if rc == 0:
print("Connected to MQTT Broker!")
else:
print("Failed to connect, return code %d\n", rc)
client = mqtt_client.Client(client_id)
client.username_pw_set(username, password)
client.on_connect = on_connect
client.connect(broker, port)
return client
def publish(client):
msg_count = 0
while True:
time.sleep(1)
msg = f"messages: {msg_count}"
result = client.publish(topic, msg)
# result: [0, 1]
status = result[0]
if status == 0:
print(f"Send `{msg}` to topic `{topic}`")
else:
print(f"Failed to send message to topic {topic}")
msg_count += 1
def run():
client = connect_mqtt()
client.loop_start()
publish(client)
if __name__ == '__main__':
run()
Thanks for reading! Hopefully someone out there has encountered the same issue and fixed it!
P.S. - forgot to mention I also disabled the firewall on the Ubuntu server until I get things working (ufw disable).
Example of my default.cfg (before) removing adapter and adding log_type all:
listener 1883
allow_anonymous false
bind_interface ens33
I wrote server in python and now I would like to configure apache web server to support websockets.
My server returns information when a client sends queries to these addresses:
def make_app():
return tornado.web.Application([
(r"/playgame", EmptyGame),
(r"/playgame/", EmptyGame),
(r"/playgame/(.*)", PlayerGameWebsocket)
])
How to configure the server to support regular user traffic but also to enable websockets when the client establishes such a connection?
I user apache2.4 server.
Ok, it turned out that the solution is trivial. If someone ever looked for an answer, just add a simple redirection to the application in the virtual host configuration which listens on localhost:
ProxyPassMatch "/playgame/(.*)" "ws://127.0.0.1:8888/playgame/$1"
ProxyPassReverse "/playgame/(.*)" "ws://127.0.0.1:8888/playgame/$1"
Thanks to such syntax, we can even pass additional data, e.g. "/playgame/123".
We connect from the client without specifying the port:
var adr = "ws://serverip/playgame/" + gameid;
var ws = new WebSocket(adr);
I am trying to create and node.js api to which users can subscribe to get event notifications?
I created the below API and was able to call the API using python ,however its not clear to me how can folks subscribe to it?
How can folks subscribe to this API to get notification of New root build released?what do I need to change?
node.js API
app.get("/api/root_event_notification", (req, res, next) => {
console.log(req.query.params)
var events = require('events');
var eventEmitter = new events.EventEmitter();
//Create an event handler:
var myEventHandler = function () {
console.log('new_root_announced!');
res.status(200).json({
message: "New root build released!",
posts: req.query.params
});
}
import requests
python call
input_json = {'BATS':'678910','root_version':'12A12'}
url = 'http://localhost:3000/api/root_event_notification?params=%s'%input_json
response = requests.get(url)
print response.text
OUTPUT:-
{"message":"New root build released!","posts":"{'root_version': '12A12', 'BATS': '678910'}"}
You can't just postpone sending an http response for an arbitrary amount of time. Both client and server (and sometimes the hosting provider's infrastructure) will timeout the http request after some number of minutes. There are various tricks to try to keep the http connection alive, but all have limitations.
Using web technologies, the usual options for get clients getting updated server data:
http polling (client regularly polls the server). There's also a long polling adaptation version of this that attempts to improve efficiency a bit.
Websocket. Clients makes a websocket connection to the server which is a lasting, persistent connection. Then either client or server can send data/events of this connection at any time, allowing the server to efficiently send notifications to the client at any time.
Server Sent Events (SSE). This is a newer http technology that allows one-way notification from server to client using some modified http technology.
Since a server cannot typically connect directly to a client due to firewall and public IP address issues, the usual mechanism for a server to notify a client is to use either a persistent webSocket connection from client to server over which either side can then send webSocket packets or use the newer SSE (server sent events) which allows some server events to be sent to a client over a long lasting connection.
The client can also "poll" the server repeatedly, but this is not really an event notification system (and not particularly efficient or timely) as much as it is some state that the client can check.
Following is a Python based RESTful library client (recommended by HP https://developer.hpe.com/platform/ilo-restful-api/home) that uses Redfish REST API (https://github.com/HewlettPackard/python-ilorest-library) to connect to the remote HPE iLO5 server of ProLiant DL360 Gen10 based hardware
#! /usr/bin/python
import redfish
iLO_host = "https://xx.xx.xx.xx"
username = "admin"
password = "xxxxxx"
# Create a REST object
REST_OBJ = redfish.redfish_client(base_url=iLO_host,username=username, password=password, default_prefix='/redfish/v1')
# Login into the server and create a session
REST_OBJ.login(auth="session")
# HTTP GET request
response = REST_OBJ.get("/redfish/v1/systems/1", None)
print response
REST_OBJ.logout()
I am getting RetriesExhaustedError when creating REST object. However, I can successfully do SSH to the server from the VM (RHEL7.4) where I am running this script. The authentication details are given correctly. I verified that the Web Server is enabled (both port 443 and 80) in the iLO Security - Access settings. Also, in my VM box the Firewalld service has been stopped and IPTables is flushed. But still connection could not be established. What other possibilities I can try yet?
I found the root cause. The issue is with SSL Certificate verification being done by the Python code.
This can be turned off by setting the environment variable PYTHONHTTPSVERIFY=0 before running the code solved the problem.
This is a very old topic, but perhaps for other people that have a similar issue when accessing the iLO in any way, and not just over Python:
You most likely need to update the firmware in your server, so that the TLS is updated. You will most likely need to use an old browser to do this, as modern versions of Mozilla/Chrome will not work with old TLS. I have had luck with Konqueror.
I have a question regarding rabbitmq. I am trying to set up a messaging system based off the queue name and so far everything has been good with localhost. As soon as I set some credentials for local connection, I get a timeout error. (I have lengthened the timeout time as well.) I have also gave the user administrative privileges and the guest account administrative privileges. I get this error when ran with both the consume and produce script. Port 5672 is open as well. This is all being done on a ubuntu 14.04 LTS machine and python 2.7.14. Guest and my other rabbit user are both allowed to use the default vhost too.
import pika, json
credentials = pika.credentials.PlainCredentials('guest', 'guest')
connection = pika.BlockingConnection(pika.ConnectionParameters('<ip here>',
5672, '/', credentials))
channel = connection.channel()
result = channel.queue_declare(queue='hello', durable=True)
def callback(ch, method, properties, body):
print "localhost received %r" % json.loads(body)
channel.basic_consume(callback,
queue='hello')
print 'localhost waiting for messages. To exit press CTRL+C'
channel.start_consuming()
channel.close()
Here is my error message too. Just a timeout method which would make me think that the connection is failing and this is a network problem but when I replace my ip in the credentials with 'localhost', everything works fine. Any ideas?
pika.exceptions.ConnectionClosed: Connection to <ip here>:5672 failed: timeout
The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
You are probably running into multiple issues.
First of all, the guest user can only connect via localhost by default. This document goes into detail. FWIW that document is the first hit when site:rabbitmq.com localhost guest are used as search terms on google.
Second, timeout means that the TCP connection can't be established. Please see this document for a step-by-step guide to diagnosis.