If a message is sent to the topic in an mqtt broker, I want to know the topic by Python. In order to use client.subscribe (), I have to manually enter a topic, so I need to know the topic before client.subscribe() dynamically. Is there a way to know what broker topics are?
If you send "EVERY" message to broker with retain message = True, then you can:
Connects to server with subscribe '#'
Check all the retain message and their topic (then you can got all topics)
Unsubscribe '#'
Subscribe the topic you want
This solution subscribe twice, may not fit your original requirement (only subscribe once), but can do what you want
Related
I am new to the kafka world and trying to do the following for a kafka consumer in python
get a list of all kafka topics.
get a list of topics a consumer has subscribed to.
subscribe to new topics (that have not yet been subscribed to).
Note: I am ok to use either confluent-kafka / kafka-python library to achieve this.
Any help would be appreciated.
if you created your Consumer with kafka-python
from kafka import KafkaConsumer
consumer = KafkaConsumer(
bootstrap_servers = 'hostname:port',
)
you can review the list of topics available with
consumer.topics()
when you subscribe to topics, you can review the consumer subscriptions with
consumer.subscription()
You can do one minus the other to find the topics you still need to subscribe to and then you can do so with
consumer.subscribe(topics=[list_of_topic_names])
I would like to publish a set (~100) of messages at once to an MQTT broker. To do so I will use publish.multiple().
The messages are of the form topic/one, topic/two, ... with each a payload. The order for these messages does not matter.
I need, however, to send first a message to topic with an empty payload in order to delete the existing messages. I would like to place it first in the list of messages to be sent.
Is there a guarantee that it will be processed first?
Looking at the paho publish code, it invokes Collections.popleft() so if you pass a tuple that has the no-op payload first, I believe it should work.
[This is not a duplicate question because even though similar questions available on stack overflow but no answer is working in my case. Hence, I am asking my doubt here.]
I want to fetch list of available Kafka topics and check whether a particular topic exist or not. I wanna do this without using consumer and any shell command. I am looking for the solutions via libraries itself. So, to fetch list of kafka topic I did following -
import kafka
client = kafka.KafkaClient(bootstrap_servers='localhost:9092')
topicList = client.topic_partitions
But, here I am getting error as -
'KafkaClient' object has no attribute 'topic_partitions'
It would be a great help if anyone can tell me how to fix the error or suggest me any other solution. Thanks in advance.
You can’t do this without creating a KafkaConsumer client. You get topic partitions in kafka only after they are assigned to a particular consumer.
You would have to use the KafkaConsumer client to retrieve the topics.
Other way to do this is with confluent-kafka consumer client or the admin client, though the latter retuns metadata about the cluster including the topic paritions. With admin client, you can do something like this:
from confluent_kafka.admin import AdminClient
admin_client = AdminClient({'bootstrap.servers': 'localhost:9002'})
admin_client.list_topics().topics
gives
{'topic1': TopicMetadata(topic1, $N partitions),}
Is there an elegant way to pull one message off the broker without having to:
subscribe
create an on_message()
receive the message
unsubscribe
I ask because we are using a json message which has multiple fields. When new data comes in I want to ONLY update that particular field in the json message but not remove the rest of the data. Since we have a TON of these json topics, we don't really want to keep all of them in program memory (also in case the program has to be relaunched). On top of that, this program could be running for months without supervision.
So ideally, I'd like to post the json message to an ID'd topic with the retain flag set to True. Then when new data comes in for that ID, I do a pull of the info on that topic, update that particular field in the json message and repost to the same topic.
I can post example code but I'm hoping there is a simple function that I am unaware of.
Thanks, in advance, for any suggestions.
The Paho Python client comes with a set of help classes that do this single shot type of pattern for you.
Doc here
e.g. the following connects to a broker, subscribes to a topic and returns on receipt of the first message on that topic.
import paho.mqtt.subscribe as subscribe
msg = subscribe.simple("paho/test/simple", hostname="mqtt.eclipse.org")
print("%s %s" % (msg.topic, msg.payload))
And the matching publish call:
import paho.mqtt.publish as publish
publish.single("paho/test/single", "payload", hostname="mqtt.eclipse.org")
I don't think that is possible. You say "When new data comes in..." That's exacty why you need to subscribe and use the callback function. That's basically a "pull when something is actually there".
Just to get an idea of how it should work: you are sending that json message via MQTT, right? And you are re-sending it when it changes?
But you don't have to keep them all in the RAM. You could use a retained message in combination with a fixed topic (not ID'ed) and send the ID in the message.
If you use retained messages with ID'ed topics, that might fill the memory.
What does the ID stand for? A uniqie number? Something like a timestamp? A hash? The sender?
I think you can solve that problem by clearly separating your things, e.g. say in data and message, where data is something you maintain in Python (e.g. a database or something in RAM) and message is something that you acually send / receive via MQTT.
Then you can add / send / update data depending on what is received in MQTT and you don't have to send / update the complete set.
I'm trying to use paho.mqtt for python (project pages) and all works nice. The only problem I have is I would find it very useful to find out who had sent the message. I looked up the source code but could not quite get my head around if the client variable passed within on_message is the client I use to connect to or details of the client who published the message (I'm guessing it's the first option).
So the question is - is it possible to find out who (the user name) had sent the message?
The MQTT protocol was designed to be as light weight as possible, this means that the message header contains the absolute bare minimum to deliver a message to a specific topic. There is no room in the header for anything else.
MQTT is also a Pub/Sub protocol, one of the key features of this type of protocol is to decouple the publisher from the subscriber as much as possible. This means that the publisher shouldn't care how many subscribers there are and subscribers shouldn't care where the information comes from as long as it is to a topic it's interested in.
If you want any more information other than the message topic then you have to add it to the payload yourself.