I am following this tutorial and I already have my code publishing messages to /devices/sm1/events topic, in which sm1 is my device id.
I would like to know how to subscribe to this topic since the tutorial says to use /devices/sm1/config but I am getting empty messages. I already tried use the same "path" used in publishing (/devices/sm1/events), but it also did not work.
It is strange that the name I gave to the topic was sm1 and the topic associated to my device is on GoogleIoT console is exhibited as projects/myprojectname/topics/sm1. So, besides to discover how to subscribe to mentioned topic, I appreciate also any explanation related to the correct way of using pub/sub topics in GoogleIoT (the documentation is not so clear).
This is my subscribe.py:
mqtt_url = "mqtt.googleapis.com"
mqtt_port = 8883
topic = "/devices/sm1/config"
def on_connect(client, userdata, flags, response_code):
print("Connected with status: {0}".format(response_code))
client.subscribe(topic, 1)
def on_message(client, userdata, msg):
print("Topic: {0} -- Payload: {1}".format(msg.topic, msg.payload))
if __name__ == "__main__":
client = mqtt.Client("projects/{}/locations/{}/registries/{}/devices/{}".format(
project_id,
cloud_region,
registry_id,
device_id))
client.username_pw_set(username='unused',
password=jwt_maker.create_jwt(project_id,
private_key,
algorithm="RS256"))
client.tls_set(root_ca,
certfile = public_crt,
keyfile = private_key,
cert_reqs = ssl.CERT_REQUIRED,
tls_version = ssl.PROTOCOL_TLSv1_2,
ciphers = None)
client.on_connect = on_connect
client.on_message = on_message
print "Connecting to Google IoT Broker..."
client.connect(mqtt_url, mqtt_port, keepalive=60)
client.loop_forever()
My output:
Connected with status: 0
Topic: /devices/sm1/config -- Payload:
Topic: /devices/sm1/config -- Payload:
After reading the discussion in the comments section in the answer by #GabeWeiss, I think there is a bit of confusion of what you want to achieve and how (or what for) are you trying to use Pub/Sub.
Given that I think the issue here is more conceptual, let me first refer you to a generic documentation page about Cloud IoT Core key concepts, where you will actually find some information regarding the relationship between Cloud IoT Core and Pub/Sub. In summary, device telemetry data is published into a Cloud IoT Core topic, which later, through the Data Broker, is published into a Cloud Pub/Sub topic, which you can use externally for other purposes: triggering Cloud Functions, analyzing the stream of data with Dataflow, etc.
Now, if you follow the Quickstart guide of Cloud IoT Core, you will see how, at a given point, you create a device registry which is bound to a Pub/Sub topic where device telemetry events are published. If instead of writing to the default Pub/Sub topic you wish to write to multiple topics, you can follow the explanations under this other section on the documentation.
Finally, getting to the issue of subscribing to topics (Cloud IoT Core topics, and not Pub/Sub topics, as only the former are relevant for the devices), you can subscribe to MQTT topics with the following command, where (as an example), the device is subscribing to the config topic, where configuration updates are published:
# This is the topic that the device will receive configuration updates on.
mqtt_config_topic = '/devices/{}/config'.format(device_id)
# Subscribe to the config topic.
client.subscribe(mqtt_config_topic, qos=1)
Then, with the on_message() function you can can process messages published on the topics where you are actually subscribed. Note that the payload has to be parsed as string (str(message.payload)).
def on_message(unused_client, unused_userdata, message):
payload = str(message.payload)
print('Received message \'{}\' on topic \'{}\' with Qos {}'.format(
payload, message.topic, str(message.qos)))
Then, in your question you stated that you first subscribed to /devices/{device-id}/config, but this might not be what you want, as this is the topic were configuration updates are published (i.e. not where telemetry events are published). Then, I understand that you should subscribe to /devices/{device-id}/events which is the actual MQTT topic where telemetry metrics are published. If that does not work, there might be another issue related, so make sure that you are parsing the message variable correctly, and maybe try to use Pub/Sub with the default topic created with the registry in order to check whether telemetry metrics are being properly published or not.
Edit: Clarifying based on comment below...
There's two GCP components in play here. There's the MQTT topic (which is the /events topic), which is used by the device to talk to IoT Core. Then there's the projects/myprojectname/topics/sm1 which isn't in IoT Core, it's in Pub/Sub. When you send messages to the /events MQTT topic, IoT Core brokers the payloads from your device that was sent to the /events MQTT topic through to the Pub/Sub topic that was created and attached to the IoT Core registry where your device was registered.
To see those messages, you have to create a subscription in Pub/Sub on the topic projects/myprojectname/topics/sm1. If you go to the console, and Pub/Sub->topics. Click the three dots next to the topic and select "New subscription". Once you have the subscription, you can send some data from your device, then on commandline you can run (assuming you have the gcloud tools installed):
gcloud beta pubsub subscriptions pull --max-messages=3 <subscription_id>
To do anything with the messages, you can script subscribing to the Pub/Sub topic (check out the Pub/Sub APIs) to trigger on messages being added to the topic.
Original message:
Are you sending a config message to the device? The confusion might be that the MQTT topics are one-directional.
So: 1) the /events topic is for device->IoT Core. 2) the /config topic is for IoT Core Admin SDK->device
In another script somewhere, or from the IoT Core UI interface you need to send a configuration message to see the on_message fire properly.
In the IoT Core UI (on console.cloud.google.com) you can drill down to an individual device you have registered, and at the top of the screen, click on "Update Config". A popup window will come up that lets you send a text or a base64 encoded message to that device and it will come in on the /config topic.
I had to surrender to Google Pub/Sub library in order to receive notifications related to my specified topic.
My publish code (only the important parts):
mqtt_url = "mqtt.googleapis.com"
mqtt_port = 8883
mqtt_topic = "/devices/sm1/events"
client = mqtt.Client("projects/{}/locations/{}/registries/{}/devices/{}".format(
project_id,
cloud_region,
registry_id,
device_id), protocol=mqtt.MQTTv311)
client.username_pw_set(username='unused',
password=jwt_maker.create_jwt(project_id,
private_key,
algorithm="RS256"))
client.tls_set(root_ca,
certfile = public_crt,
keyfile = private_key,
cert_reqs = ssl.CERT_REQUIRED,
tls_version = ssl.PROTOCOL_TLSv1_2,
ciphers = None)
client.connect(mqtt_url, mqtt_port, keepalive=60)
res = client.publish(mqtt_topic, some_data, qos=1)
In the Google Cloud Platform portal, I had to create a subscription, in the Pub/Sub section, assigning it to my created topic, which was already my default topic (probably linked to /events topic). The created subscription has the following format:
projects/project_name/subscriptions/subscription_name
My subscribe code, using the Google Pub/Sub library, since it is not possible to use the MQTT protocol:
from google.cloud import pubsub
def callback(message):
print(message.data)
message.ack()
project_id = "project_name"
subscription_name = "sm1"
subscriber = pubsub.SubscriberClient()
subscription_name = 'projects/{}/subscriptions/{}'.format(project_id, subscription_name)
subscription = subscriber.subscribe(subscription_name)
future = subscription.open(callback)
try:
future.result()
except Exception as ex:
subscription.close()
raise
I hope this can help anyone. More details can be found here(github).
Related
I am able to send messages and reported-properties from iot hub to a simulated device through azure-iot-sdk-python.
Now i wanna get acknowledgments (success,expired,rejected,purjed,DeliveryCountexceeded) for messages sent to the device/module from IoT Hub
ServiceClient.GetFeedbackReceiver method is available for .Net but i am not able to find a python sdk for getting message delivery feedback.
below is code used for sending c2d message
from azure.iot.hub import IoTHubRegistryManager
CONNECTION_STRING = ""
DEVICE_ID = ""
registry_manager = IoTHubRegistryManager(CONNECTION_STRING)
data = "Hello.. message from cloud"
props={}
registry_manager.send_c2d_message(DEVICE_ID, data, properties=props)
i want to get a feedback when the message sending is failed.. please suggest me a solution.
Thanks in advance.. :)
ServiceClient.GetFeedbackReceiver method is available for .Net but i am not able to find a python sdk for getting message delivery feedback.
You can try receive_feedback_notification() as available on cloud_to_device_messages_operations.py
def receive_feedback_notification(self, custom_headers=None, raw=False, **operation_config):
Gets the feedback for cloud-to-device messages.
See https://learn.microsoft.com/azure/iot-hub/iot-hub-devguide-messaging for
more information. This capability is only available in the standard tier IoT Hub. For more information, see Choose the right IoT Hubtier.
:param dict custom_headers: headers that will be added to the request
:param bool raw: returns the direct response alongside the
deserialized response
:param operation_config: :ref:`Operation
configuration overrides<msrest:optionsforoperations>`.
:return: None
or ClientRawResponse if raw=true
:rtype: None or
~msrest.pipeline.ClientRawResponse
:raises:
:class:`HttpOperationError<msrest.exceptions.HttpOperationError>`
#Construct URL
url = self.receive_feedback_notification.metadata["url"]
If you want to test feedback notification, refer to test_iothub_http_runtime_manager.py
I am trying to get the number of messages still in the ServiceBus queue in Python.
I am able to receive messages but I can only find documentation to get the message count via the portal, CLI, or powershell, but not for the python SDK. Is there such a function or property?
from azure.servicebus import ServiceBusClient
with ServiceBusClient.from_connection_string(CONNECTION_STRING) as client:
with client.get_queue_receiver(QUEUE_NAME, max_wait_time=30) as receiver:
msgs = receiver.receive_messages(2)
for msg in msgs:
receiver.complete_message(msg)
number_of_retrieved_messages = len(msgs)
number_of_messages_still_in_queue = ?
Looking through the code of the Python SDK (the Azure Service Bus client library for Python), it appears that there's a ServiceBusAdministrationClient that has a get_queue_runtime_properties method.
queue_runtime_properties = await servicebus_mgmt_client.get_queue_runtime_properties(QUEUE_NAME)`
print("Message Count:", queue_runtime_properties.total_message_count)
Example found in async_samples/mgmt_queue_async.py
Implementation found in management/_management_client.py
I am working with multiple applications that communicate asynchronously using Kafka. These applications are managed by several departments and contract testing is appropriate to ensure that the messages used during communication follow the expected schema and will evolve according to the contract specification.
It sounded like the pact library for python is a good fit because it helps creating contract tests for HTTP and message integrations.
What I wanted to do is to send an HTTP request and to listen from the appropriate and dedicated Kafka topic immediately after. But it seems that the test is forcing me specify an HTTP code even if what I am expecting is a message from a queue without an HTTP status code. Furthermore, it seems that the HTTP request is being sent before the consumer is listening. Here is some sample code.
from pact.consumer import Consumer as p_Consumer
from pact.provider import Provider as p_Provider
from confluent_kafka import Consumer as k_Consumer
pact = p_Consumer('Consumer').has_pact_with(p_Provider('Provider'))
pact.start_service()
atexit.register(pact.stop_service)
config = {'bootstrap.servers':'server', 'group.id':0, 'auto.offset.reset':'latest'}
consumer = k_consumer(config)
consumer.subscribe(['usertopic'])
def user():
while True:
msg = consumer.poll(timeout=1)
if msg is None:
continue
else:
return msg.value().decode()
class ConstractTesting(unittest.TestCase):
expected = {
'username': 'UserA',
'id':123,
'groups':['Editors']
}
pact.given('UserA exists and is not an administrator')
.upon_receiving('a request for UserA')
.with_request(method='GET',path='/user/')
.will_respond_with(200, body=expected)
with pact:
result = user()
self.assertEqual(result,expected)
How would I carry out contract testing in Python using Kafka? It feels like I am going through a lot of hoops to carry out this test.
With Pact message it's a different API you write tests against. You don't use the standard HTTP one, in fact the transport itself is ignored altogether and it's just the payload - the message - we're interested in capturing and verifying. This allows us to test any queue without having to build specific interfaces for each
See this example: https://github.com/pact-foundation/pact-python/blob/02643d4fb89ff7baad63e6436f6a929256c6bf12/examples/message/tests/consumer/test_message_consumer.py#L65
You can read more about message pact testing here: https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
And finally here are some Kafka examples for other languages that may be helpful: https://docs.pactflow.io/docs/examples/kafka/js/consumer
I am trying to build an application using Google Cloud Platform AutoML using Python. My overall code flow looks like this:
User Interacts--> data sent to PubSub--> callback invokes my AutoML--> Result
The snippet that calls pubsub looks like this:
blob=blob+bytes(doc_type,'utf-8')
publisher.publish(topic,blob)
future=subscriber.subscribe(subscription,callback=callback)
#flash("The object is "+future,'info')
try:
future.result()
except Exception as ex:
subscriber.close()
In PubSub callback:
def callback(message):
new_message=message.data
display_name,score=predict_value(new_message,"modelID","projectid",'us-central1')
message.ack()
And my predict_value gets the model_id, project id and compute region and performs the prediction.
When I directly call predict_value without using PubSub it is working fine. If I do like this, I am getting the below error:
google.api_core.exceptions.PermissionDenied: 403 Permission 'automl.models.predict' denied on resource 'projects/projectID/locations/us-central1/models/' (or it may not exist).
Please help me to resolve the issue
Thank you so much for all your responses. I have just fixed the issue using the below snippet example
def receive_messages_synchronously(project, subscription_name):
"""Pulling messages synchronously."""
# [START pubsub_subscriber_sync_pull]
# project = "Your Google Cloud Project ID"
# subscription_name = "Your Pubsub subscription name"
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
# Builds a pull request with a specific number of messages to return.
# `return_immediately` is set to False so that the system waits (for a
# bounded amount of time) until at lease one message is available.
response = subscriber.pull(
subscription_path,
max_messages=3,
return_immediately=False)
ack_ids = []
for received_message in response.received_messages:
print("Received: {}".format(received_message.message.data))
ack_ids.append(received_message.ack_id)
# Acknowledges the received messages so they will not be sent again.
subscriber.acknowledge(subscription_path, ack_ids)
# [END pubsub_subscriber_sync_pull]
The reason being the subscription that is created uses the pull request. I guess the callback method concept used is mainly for "push" which may be the reason because I didnt give the endpoint and token to publish the message. Hope what I am guessing is correct. Let me know your views as well.
This is likely due to one of two factors:
invalid credentials being used when sending the request to the AutoML API - it is very likely that pubsub executes in other context and can't get the default credentials
invalid model resource name (make sure it is correct) - it should be something like: "projects/12423534/locations/us-central1/models/23432423"
I have the following scenario I would like to implement:
User surfs to our website
User enters a bitcoin address.
A websocket is created to the server, passing the address.
The server registers a callback with Blocktrail
When the callback is triggered (a payment was seen by Blocktrail) we send a message back to the browser.
The page the user is browsing is updated to show the message recieved
I'm using webhooks from the Blocktrail API to "listen" to an event, being the reception of coins on an address.
Now, when the event happens, the API does a POST to my URL. This should send a message to the browser that is connected to my server with socket.io (such as 'payment seen on blockchain')
So the question is,
How can I send a message from a route to a socket using flask-socketio
Pseudo code:
#app.route('/callback/<address>')
def callback(id):
socketio.send('payment seen on blockchain')
#socketio.on('address',address)
def socketlisten(address):
registerCallback(address)
I'm going to describe how to solve this using Flask-SocketIO beta version 1.0b1. You can also do this with the 0.6 release, but it is a bit more complicated, the 1.0 release makes addressing individual clients easier.
Each client of a socket connection gets assigned a session id that uniquely identifies it, the so called sid. Within a socket function handler, you can access it as request.sid. Also, upon connection, each client is assigned to a private room, named with the session id.
I assume the metadata that you receive with the callback allows you to identify the user. What you need is to obtain the sid of that user. Once you have it, you can send your alert to the corresponding room.
Example (with some hand-waving regarding how you attach a sid to an address):
#app.route('/callback/<address>')
def callback(address):
sid = get_sid_from_address(address)
socketio.send('payment seen on blockchain', room=sid)
#socketio.on('address')
def socketlisten(address):
associate_address_with_sid(address, request.sid)