I'd like to setup a stream of data from Azure to my local instance.
The challenge is options like Azure functions don't allow streaming of data.
I'm looking at Azure SignalR, or the Azure bot framework but am not having any luck so far. I think server sent events are the way to go, but don't know where to start with Azure
I want to use Python, and have data streamed, for example, I'd like to stream the following sentence one word at a time.:
words = 'hello how are you today?'
From my knowledge, you can use the Azure Event Hubs. You can use the Event Hubs client library for Python to consume the datas from Azure Event Hubs.
pip install azure-eventhub
Example code:
from azure.eventhub import EventHubProducerClient, EventData
CONNECTION_STR "{Event Hubs Connection String}"
EVENTHUB_NAME = "{Event Hub name}"
producer = EventHubProducerClient.from_connection_string(CONNECTION_STR, eventhub_name=EVENTHUB_NAME)
for word in words.split():
event_data_batch = producer.create_batch()
event_data = EventData(word)
try:
event_data_batch.add(event_data)
except ValueError:
producer.send_batch(event_data_batch)
event_data_batch = producer.create_batch()
event_data_batch(event_data_batch)
producer.send_batch(event_data_batch)
On local instance, here is the code:
from azure.eventhub import EventHubConsumerClient
CONNECTION_STR = "{Event Hubs connection string}"
EVENTHUB_NAME = "{Event Hub name}"
CONSUMER_GROUP = "$default"
consumer = EventHubConsumerClient.from_connection_string(CONNECTION_STR, consumer_group=CONSUMER_GROUP, eventhub_name=EVENTHUB_NAME)
def on_event(partition_context, event):
print(event,body_as_str())
with consumer:
consumer.receive(on_event)
I am able to send messages and reported-properties from iot hub to a simulated device through azure-iot-sdk-python.
Now i wanna get acknowledgments (success,expired,rejected,purjed,DeliveryCountexceeded) for messages sent to the device/module from IoT Hub
ServiceClient.GetFeedbackReceiver method is available for .Net but i am not able to find a python sdk for getting message delivery feedback.
below is code used for sending c2d message
from azure.iot.hub import IoTHubRegistryManager
CONNECTION_STRING = ""
DEVICE_ID = ""
registry_manager = IoTHubRegistryManager(CONNECTION_STRING)
data = "Hello.. message from cloud"
props={}
registry_manager.send_c2d_message(DEVICE_ID, data, properties=props)
i want to get a feedback when the message sending is failed.. please suggest me a solution.
Thanks in advance.. :)
ServiceClient.GetFeedbackReceiver method is available for .Net but i am not able to find a python sdk for getting message delivery feedback.
You can try receive_feedback_notification() as available on cloud_to_device_messages_operations.py
def receive_feedback_notification(self, custom_headers=None, raw=False, **operation_config):
Gets the feedback for cloud-to-device messages.
See https://learn.microsoft.com/azure/iot-hub/iot-hub-devguide-messaging for
more information. This capability is only available in the standard tier IoT Hub. For more information, see Choose the right IoT Hubtier.
:param dict custom_headers: headers that will be added to the request
:param bool raw: returns the direct response alongside the
deserialized response
:param operation_config: :ref:`Operation
configuration overrides<msrest:optionsforoperations>`.
:return: None
or ClientRawResponse if raw=true
:rtype: None or
~msrest.pipeline.ClientRawResponse
:raises:
:class:`HttpOperationError<msrest.exceptions.HttpOperationError>`
#Construct URL
url = self.receive_feedback_notification.metadata["url"]
If you want to test feedback notification, refer to test_iothub_http_runtime_manager.py
I am working with multiple applications that communicate asynchronously using Kafka. These applications are managed by several departments and contract testing is appropriate to ensure that the messages used during communication follow the expected schema and will evolve according to the contract specification.
It sounded like the pact library for python is a good fit because it helps creating contract tests for HTTP and message integrations.
What I wanted to do is to send an HTTP request and to listen from the appropriate and dedicated Kafka topic immediately after. But it seems that the test is forcing me specify an HTTP code even if what I am expecting is a message from a queue without an HTTP status code. Furthermore, it seems that the HTTP request is being sent before the consumer is listening. Here is some sample code.
from pact.consumer import Consumer as p_Consumer
from pact.provider import Provider as p_Provider
from confluent_kafka import Consumer as k_Consumer
pact = p_Consumer('Consumer').has_pact_with(p_Provider('Provider'))
pact.start_service()
atexit.register(pact.stop_service)
config = {'bootstrap.servers':'server', 'group.id':0, 'auto.offset.reset':'latest'}
consumer = k_consumer(config)
consumer.subscribe(['usertopic'])
def user():
while True:
msg = consumer.poll(timeout=1)
if msg is None:
continue
else:
return msg.value().decode()
class ConstractTesting(unittest.TestCase):
expected = {
'username': 'UserA',
'id':123,
'groups':['Editors']
}
pact.given('UserA exists and is not an administrator')
.upon_receiving('a request for UserA')
.with_request(method='GET',path='/user/')
.will_respond_with(200, body=expected)
with pact:
result = user()
self.assertEqual(result,expected)
How would I carry out contract testing in Python using Kafka? It feels like I am going through a lot of hoops to carry out this test.
With Pact message it's a different API you write tests against. You don't use the standard HTTP one, in fact the transport itself is ignored altogether and it's just the payload - the message - we're interested in capturing and verifying. This allows us to test any queue without having to build specific interfaces for each
See this example: https://github.com/pact-foundation/pact-python/blob/02643d4fb89ff7baad63e6436f6a929256c6bf12/examples/message/tests/consumer/test_message_consumer.py#L65
You can read more about message pact testing here: https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact
And finally here are some Kafka examples for other languages that may be helpful: https://docs.pactflow.io/docs/examples/kafka/js/consumer
Utilizing Twilio, upon receiving any SMS I'm seeking to trigger a Python function that reads the contents of the message, then conditionally performs an action.
I'm referencing from Twilio docs right now (.py):
# Download the helper library from https://www.twilio.com/docs/python/install
from twilio.rest import Client
# Your Account Sid and Auth Token from twilio.com/console
account_sid = 'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth_token = 'your_auth_token'
client = Client(account_sid, auth_token)
message = client.messages('MMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX').fetch()
print(message.to)
Does anyone know how to automatically set up a trigger OnReceive?
The example you show fetches a known message (a message that has already been received and you know its ID).
In order to have a script that is triggered by incoming message you need to setup a webhook and you need to have an endpoint (server) where you can pick up a trigger sent to you by twilio (eg at www.yourdomain.com/sms).
You might not be able to do it just from within a raspbery pi connected to the internet. You need to setup a trigger somewhere on a server, store the incoming messages somehow and then one solution would be to keep polling the server from raspberry pi, fetching new messages and clearing the list.
I am following this tutorial and I already have my code publishing messages to /devices/sm1/events topic, in which sm1 is my device id.
I would like to know how to subscribe to this topic since the tutorial says to use /devices/sm1/config but I am getting empty messages. I already tried use the same "path" used in publishing (/devices/sm1/events), but it also did not work.
It is strange that the name I gave to the topic was sm1 and the topic associated to my device is on GoogleIoT console is exhibited as projects/myprojectname/topics/sm1. So, besides to discover how to subscribe to mentioned topic, I appreciate also any explanation related to the correct way of using pub/sub topics in GoogleIoT (the documentation is not so clear).
This is my subscribe.py:
mqtt_url = "mqtt.googleapis.com"
mqtt_port = 8883
topic = "/devices/sm1/config"
def on_connect(client, userdata, flags, response_code):
print("Connected with status: {0}".format(response_code))
client.subscribe(topic, 1)
def on_message(client, userdata, msg):
print("Topic: {0} -- Payload: {1}".format(msg.topic, msg.payload))
if __name__ == "__main__":
client = mqtt.Client("projects/{}/locations/{}/registries/{}/devices/{}".format(
project_id,
cloud_region,
registry_id,
device_id))
client.username_pw_set(username='unused',
password=jwt_maker.create_jwt(project_id,
private_key,
algorithm="RS256"))
client.tls_set(root_ca,
certfile = public_crt,
keyfile = private_key,
cert_reqs = ssl.CERT_REQUIRED,
tls_version = ssl.PROTOCOL_TLSv1_2,
ciphers = None)
client.on_connect = on_connect
client.on_message = on_message
print "Connecting to Google IoT Broker..."
client.connect(mqtt_url, mqtt_port, keepalive=60)
client.loop_forever()
My output:
Connected with status: 0
Topic: /devices/sm1/config -- Payload:
Topic: /devices/sm1/config -- Payload:
After reading the discussion in the comments section in the answer by #GabeWeiss, I think there is a bit of confusion of what you want to achieve and how (or what for) are you trying to use Pub/Sub.
Given that I think the issue here is more conceptual, let me first refer you to a generic documentation page about Cloud IoT Core key concepts, where you will actually find some information regarding the relationship between Cloud IoT Core and Pub/Sub. In summary, device telemetry data is published into a Cloud IoT Core topic, which later, through the Data Broker, is published into a Cloud Pub/Sub topic, which you can use externally for other purposes: triggering Cloud Functions, analyzing the stream of data with Dataflow, etc.
Now, if you follow the Quickstart guide of Cloud IoT Core, you will see how, at a given point, you create a device registry which is bound to a Pub/Sub topic where device telemetry events are published. If instead of writing to the default Pub/Sub topic you wish to write to multiple topics, you can follow the explanations under this other section on the documentation.
Finally, getting to the issue of subscribing to topics (Cloud IoT Core topics, and not Pub/Sub topics, as only the former are relevant for the devices), you can subscribe to MQTT topics with the following command, where (as an example), the device is subscribing to the config topic, where configuration updates are published:
# This is the topic that the device will receive configuration updates on.
mqtt_config_topic = '/devices/{}/config'.format(device_id)
# Subscribe to the config topic.
client.subscribe(mqtt_config_topic, qos=1)
Then, with the on_message() function you can can process messages published on the topics where you are actually subscribed. Note that the payload has to be parsed as string (str(message.payload)).
def on_message(unused_client, unused_userdata, message):
payload = str(message.payload)
print('Received message \'{}\' on topic \'{}\' with Qos {}'.format(
payload, message.topic, str(message.qos)))
Then, in your question you stated that you first subscribed to /devices/{device-id}/config, but this might not be what you want, as this is the topic were configuration updates are published (i.e. not where telemetry events are published). Then, I understand that you should subscribe to /devices/{device-id}/events which is the actual MQTT topic where telemetry metrics are published. If that does not work, there might be another issue related, so make sure that you are parsing the message variable correctly, and maybe try to use Pub/Sub with the default topic created with the registry in order to check whether telemetry metrics are being properly published or not.
Edit: Clarifying based on comment below...
There's two GCP components in play here. There's the MQTT topic (which is the /events topic), which is used by the device to talk to IoT Core. Then there's the projects/myprojectname/topics/sm1 which isn't in IoT Core, it's in Pub/Sub. When you send messages to the /events MQTT topic, IoT Core brokers the payloads from your device that was sent to the /events MQTT topic through to the Pub/Sub topic that was created and attached to the IoT Core registry where your device was registered.
To see those messages, you have to create a subscription in Pub/Sub on the topic projects/myprojectname/topics/sm1. If you go to the console, and Pub/Sub->topics. Click the three dots next to the topic and select "New subscription". Once you have the subscription, you can send some data from your device, then on commandline you can run (assuming you have the gcloud tools installed):
gcloud beta pubsub subscriptions pull --max-messages=3 <subscription_id>
To do anything with the messages, you can script subscribing to the Pub/Sub topic (check out the Pub/Sub APIs) to trigger on messages being added to the topic.
Original message:
Are you sending a config message to the device? The confusion might be that the MQTT topics are one-directional.
So: 1) the /events topic is for device->IoT Core. 2) the /config topic is for IoT Core Admin SDK->device
In another script somewhere, or from the IoT Core UI interface you need to send a configuration message to see the on_message fire properly.
In the IoT Core UI (on console.cloud.google.com) you can drill down to an individual device you have registered, and at the top of the screen, click on "Update Config". A popup window will come up that lets you send a text or a base64 encoded message to that device and it will come in on the /config topic.
I had to surrender to Google Pub/Sub library in order to receive notifications related to my specified topic.
My publish code (only the important parts):
mqtt_url = "mqtt.googleapis.com"
mqtt_port = 8883
mqtt_topic = "/devices/sm1/events"
client = mqtt.Client("projects/{}/locations/{}/registries/{}/devices/{}".format(
project_id,
cloud_region,
registry_id,
device_id), protocol=mqtt.MQTTv311)
client.username_pw_set(username='unused',
password=jwt_maker.create_jwt(project_id,
private_key,
algorithm="RS256"))
client.tls_set(root_ca,
certfile = public_crt,
keyfile = private_key,
cert_reqs = ssl.CERT_REQUIRED,
tls_version = ssl.PROTOCOL_TLSv1_2,
ciphers = None)
client.connect(mqtt_url, mqtt_port, keepalive=60)
res = client.publish(mqtt_topic, some_data, qos=1)
In the Google Cloud Platform portal, I had to create a subscription, in the Pub/Sub section, assigning it to my created topic, which was already my default topic (probably linked to /events topic). The created subscription has the following format:
projects/project_name/subscriptions/subscription_name
My subscribe code, using the Google Pub/Sub library, since it is not possible to use the MQTT protocol:
from google.cloud import pubsub
def callback(message):
print(message.data)
message.ack()
project_id = "project_name"
subscription_name = "sm1"
subscriber = pubsub.SubscriberClient()
subscription_name = 'projects/{}/subscriptions/{}'.format(project_id, subscription_name)
subscription = subscriber.subscribe(subscription_name)
future = subscription.open(callback)
try:
future.result()
except Exception as ex:
subscription.close()
raise
I hope this can help anyone. More details can be found here(github).