I'm trying to use paho.mqtt for python (project pages) and all works nice. The only problem I have is I would find it very useful to find out who had sent the message. I looked up the source code but could not quite get my head around if the client variable passed within on_message is the client I use to connect to or details of the client who published the message (I'm guessing it's the first option).
So the question is - is it possible to find out who (the user name) had sent the message?
The MQTT protocol was designed to be as light weight as possible, this means that the message header contains the absolute bare minimum to deliver a message to a specific topic. There is no room in the header for anything else.
MQTT is also a Pub/Sub protocol, one of the key features of this type of protocol is to decouple the publisher from the subscriber as much as possible. This means that the publisher shouldn't care how many subscribers there are and subscribers shouldn't care where the information comes from as long as it is to a topic it's interested in.
If you want any more information other than the message topic then you have to add it to the payload yourself.
Related
Is there an elegant way to pull one message off the broker without having to:
subscribe
create an on_message()
receive the message
unsubscribe
I ask because we are using a json message which has multiple fields. When new data comes in I want to ONLY update that particular field in the json message but not remove the rest of the data. Since we have a TON of these json topics, we don't really want to keep all of them in program memory (also in case the program has to be relaunched). On top of that, this program could be running for months without supervision.
So ideally, I'd like to post the json message to an ID'd topic with the retain flag set to True. Then when new data comes in for that ID, I do a pull of the info on that topic, update that particular field in the json message and repost to the same topic.
I can post example code but I'm hoping there is a simple function that I am unaware of.
Thanks, in advance, for any suggestions.
The Paho Python client comes with a set of help classes that do this single shot type of pattern for you.
Doc here
e.g. the following connects to a broker, subscribes to a topic and returns on receipt of the first message on that topic.
import paho.mqtt.subscribe as subscribe
msg = subscribe.simple("paho/test/simple", hostname="mqtt.eclipse.org")
print("%s %s" % (msg.topic, msg.payload))
And the matching publish call:
import paho.mqtt.publish as publish
publish.single("paho/test/single", "payload", hostname="mqtt.eclipse.org")
I don't think that is possible. You say "When new data comes in..." That's exacty why you need to subscribe and use the callback function. That's basically a "pull when something is actually there".
Just to get an idea of how it should work: you are sending that json message via MQTT, right? And you are re-sending it when it changes?
But you don't have to keep them all in the RAM. You could use a retained message in combination with a fixed topic (not ID'ed) and send the ID in the message.
If you use retained messages with ID'ed topics, that might fill the memory.
What does the ID stand for? A uniqie number? Something like a timestamp? A hash? The sender?
I think you can solve that problem by clearly separating your things, e.g. say in data and message, where data is something you maintain in Python (e.g. a database or something in RAM) and message is something that you acually send / receive via MQTT.
Then you can add / send / update data depending on what is received in MQTT and you don't have to send / update the complete set.
I am writing a simple SOAP client application in Python.
WSDL file can be found here: https://clients.nationalmailing.com.au/ServiceTest/OrderService.svc?wsdl
Unfortunately the server declared usage of wsHttpBinding in its WSDL file and I had to learn how many troubles it brings to not-.NET developers.
I have working C# code (and it is pretty simple there) and used Fiddler to capture the traffic and analyze messages. Now I know the structure to follow. Client sends 2 subsequental messages.
I managed to create and send first request and receive a response from the server. BUT second request is a way more complex. I have found a library signxml which helped me to create <Signature> structure with all the fields that should present (as per captured traffic).
But the server continues to answer with "Error 500: An error occurred when verifying security for the message."
I realized that in the first message I put just random values for the following structure:
<s:Body>
<trust:RequestSecurityToken xmlns:trust="http://docs.oasis-open.org/ws-sx/ws-trust/200512">
<trust:TokenType>http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512/sct</trust:TokenType>
<trust:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType>
<trust:Entropy>
<trust:BinarySecret
u:Id="uuid-0649fd7a-9ae2-4f9f-964c-e3aa5d68e8cd-1"
Type="http://docs.oasis-open.org/ws-sx/ws-trust/200512/Nonce">h/MaeQVSL5Br30Hnt/SAl274flYfZVZyx2Fri9zNuEY=</trust:BinarySecret>
</trust:Entropy>
<trust:KeySize>256</trust:KeySize>
</trust:RequestSecurityToken>
</s:Body>
The value of BinarySecret is just a random string encoded with Base64. I think this should be an issue on this stage. I also do not use the same parameters from server's response.
Could anyone explain how should I use Entropy.BinarySecret - should it take part in the calculations of Signature and how it is used?
Answering my own question. Yes, the issue was in improper usage of Entropy parameter.
To sign the message you need to generate a key, it consists of two parts (client entropy and server's entropy). They get combined with P_SHA1 algorithm into a key.
To anyone who find this post in the future: for Python have a look on signxml library and section 4 of ws-trust spec.
Problem: My use case is I want to receive messages from Google Cloud Pub/Sub - one message at a time using the Python Api. All the current examples mention using Async/callback option for pulling the messages from a Pub/Sub subscription. The problem with that approach is I need to keep the thread alive.
Is it possible to just receive 1 message and close the connection i.e. is there a feature where I can just set a parameter (something like a max_messages) to 1 so that once it receives 1 message the thread terminates?
The documentation here doesn't list anything for Python Synchronous pull which seem to have num_of_messages option for other languages like Java.
See the following example in this link:
from google.cloud import pubsub_v1
client = pubsub_v1.SubscriberClient()
subscription = client.subscription_path('[PROJECT]', '[SUBSCRIPTION]')
max_messages = 1
response = client.pull(subscription, max_messages)
print(response)
I've tried myself and using that you get one message at a time.
If you get some error try updating pubsub library to the last version:
pip install --upgrade google-cloud-pubsub
In docs here you have more info about the pull method used in the code snippet:
The Pull method relies on a request/response model:
The application sends a request for messages. The server replies with
zero or more messages and closes the connection.
As per the official documentation here:
...you can achieve exactly once processing of Pub/Sub message streams,
as PubsubIO de-duplicates messages based on custom message identifiers
or identifiers assigned by Pub/Sub.
So you should be able to use record IDs, i.e. identifiers for you messages, to allow for exactly-once processing across the boundary between Dataflow and other systems. To use record IDs, you invoke idLabel when constructing PubsubIO.Read or PubsubIO.Write transforms, passing a string value of your choice. In java this would be:
public PubsubIO.Read.Bound<T> idLabel(String idLabel)
This returns a transform that's like this one but that reads unique message IDs from the given message attribute.
If a message is sent to the topic in an mqtt broker, I want to know the topic by Python. In order to use client.subscribe (), I have to manually enter a topic, so I need to know the topic before client.subscribe() dynamically. Is there a way to know what broker topics are?
If you send "EVERY" message to broker with retain message = True, then you can:
Connects to server with subscribe '#'
Check all the retain message and their topic (then you can got all topics)
Unsubscribe '#'
Subscribe the topic you want
This solution subscribe twice, may not fit your original requirement (only subscribe once), but can do what you want
I am working on an application that will use mqtt. I will be using the python library. I have been leaning towards using mosquitto but can find no way of programmatically setting access control limits for it. The application I'm writing needs to be able to differentiate between users, and only allow them to subscribe to certain topics. The current solution looks like this is done from a config file. Is there a scalable solution to access control limits in mosquitto? If not, do you know of a mqtt broker in which this exists?
Even if this might not concern you anymore, others could find it useful. I am following here mosquitto's man page.
There are two configuration files, a general one, say mosquitto.conf, and an ACL (Access Control List) one, say acl.conf.
mosquitto.conf enables the acl.conf file for access control:
acl_file acl.conf
acl.conf defines the access control behavior:
# users can anonymously publish to the topic 'in'
topic write in
# users can subscribe topics named 'out/%u', where %u is the user's name
pattern read out/%u
# an admin may subscribe to 'in'
# and publish to all subtopics of 'out/' (note the +)
user adminWithSecretName
topic read in
topic write out/+
We execute mosquitto -c mosquitto.conf to run mosquitto with the configuration file.
In this case, a dynamic authentication mechanism can be established by using randomly generated user names.
Example: Alice wants to subscribe so that she can read here private messages. She sends her credentials in combination with a nonceN1 to in. Furthermore, she also subscribes the topic out/N1, using N1 as user name. The pattern read out/%u allows that.
A third-party server application, connected as adminWithSecretName and subscribed to the topic in receives Alice' message. It verifies its authenticity and then generates a new nonce N2 and publishes it to out/N1 where Alice has subscribed to.
From now on -- at least for this session -- out/N2 is the regular topic where Alice respectively here devices will receive messages. Therefore, Alice unsubscribes and disconnects form out/N1 and subscribes to out/N2. The third-party server application publishes all new messages, belonging to Alice, to the topic out/N2.
Further Considerations: One may also want to reflect on other aspects of security such as TLS and/or per-message encryption. The configuration discussed here would, depending on the grade of targeted security/privacy, probably also require TLS.
On the other hand, this could be obsolete if the messages are encrypted separately. One, say Eve, could intercept (even subscribe!) the messages if she had access to the cable/WiFi stream, as she would see the secret user name as plain text. But: when one already has access to the data stream, he/she can intercept the bytes anyway. They are encrypted either way, using TLS or per-message encryption. Also, traffic analysis may be applied to both approaches.
I would suggest to use either TLS or per-message encryption. Both should, correctly implemented and applied, lead to comparable security.
You could write a plugin to handle this for you. See http://mosquitto.org/2013/07/authentication-plugins/ for some examples.
You may find more answers if you ask on the mosquitto mailing list.
If you are familiar with Java you should try the HiveMQ MQTT broker: http://www.hivemq.com.
There is an open PluginSDK, which enables you to write any kind of extensions to the broker.
You can implement the authentication or authorization method that fits your use case best, for example from database, file...
The authorization based on topic is a common use case and there is an example in the HiveMQ Plugin Guide.
As entry point into HiveMQ plugin development see the Get started with Plugins page: http://www.hivemq.com/documentations/getting-started-plugins/
Disclosure: I'm one of the developers of HiveMQ.