I am trying to get the number of messages still in the ServiceBus queue in Python.
I am able to receive messages but I can only find documentation to get the message count via the portal, CLI, or powershell, but not for the python SDK. Is there such a function or property?
from azure.servicebus import ServiceBusClient
with ServiceBusClient.from_connection_string(CONNECTION_STRING) as client:
with client.get_queue_receiver(QUEUE_NAME, max_wait_time=30) as receiver:
msgs = receiver.receive_messages(2)
for msg in msgs:
receiver.complete_message(msg)
number_of_retrieved_messages = len(msgs)
number_of_messages_still_in_queue = ?
Looking through the code of the Python SDK (the Azure Service Bus client library for Python), it appears that there's a ServiceBusAdministrationClient that has a get_queue_runtime_properties method.
queue_runtime_properties = await servicebus_mgmt_client.get_queue_runtime_properties(QUEUE_NAME)`
print("Message Count:", queue_runtime_properties.total_message_count)
Example found in async_samples/mgmt_queue_async.py
Implementation found in management/_management_client.py
I have a dashboard with data that I would like updated in real-time from a subscribed websocket
channel. I am currently using Flask and get updated data with request and response cycles.
I am investigating Flask-SocketIO, but can't seem to figure out how to connect a websocket
feed from Alpaca (the datasource) and get that data to my dashboard webclient.
Would the data flow from 1) Alapca => Flask => webclient? or 2) Alpaca => webclient => Flask => webclient?
I would prefer case (1) because I want to do some processing and also because I have just started dabbling with javascript for the first time for this project (but whatever works determine the choice).
What packages or libraries would you use to accomplish this? How would you do it? (I'm on a windows pc)
I currently have a file that runs the flask app on one port and another file that connects to the
data source that streams the data on another port. How do you connect the two? Can both Flask-SocketIO and Alpaca listen/stream data on the same port? I looked around and can't find any examples that help. Can it be done?
This doesn't seem kosher, but could something like the following be done?
# Alpaca websocket connection: data inflow
#conn.on(r'^T.*$')
async def on_data(conn, channel, data):
data_dict = data._raw
print('data:', data_dict)
# FastAPI websocket creation: pass-through/ data outflow
#app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
while True:
payload = data_dict
await websocket.send_json(payload)
conn.run(['T.AAPL']])
Can something like this be accomplished using Flask-SocketIO?
References:
Connecting to Alpaca websocket
Creating websocket using FastAPI
I am attempting to get some sample JSON data to show up in Azure Time Series Insights (TSI) using Python so I can visualize the data within their exploratory browser.
I have gone through the necessary prerequisites with regards to Azure EventHubs and Time Series Insights setup. These include:
Create a resource group in the Azure portal
Create an event hub namespace within the resource group
Create an event hub entity within that event hub namespace
Create a consumer group within the event hub entity
I set up an Azure TSI environment within the resource group.
And finally I added an Event Source to the Azure TSI environment using the different created sources as the details (resource group, event hub namespace, event hub name, etc.)
Beyond that I tested successfully sending event messages to my event hub (but not into the TSI environment) using python by following this documentation: https://learn.microsoft.com/en-us/azure/event-hubs/get-started-python-send-v2 and using this code (albeit con_str filled in and eventhub_name filled in:
import asyncio
from azure.eventhub.aio import EventHubProducerClient
from azure.eventhub import EventData
async def run():
# Create a producer client to send messages to the event hub.
# Specify a connection string to your event hubs namespace and
# the event hub name.
producer = EventHubProducerClient.from_connection_string(conn_str="EVENT HUBS NAMESPACE - CONNECTION STRING", eventhub_name="EVENT HUB NAME")
async with producer:
# Create a batch.
event_data_batch = await producer.create_batch()
# Add events to the batch.
event_data_batch.add(EventData('First event '))
event_data_batch.add(EventData('Second event'))
event_data_batch.add(EventData('Third event'))
# Send the batch of events to the event hub.
await producer.send_batch(event_data_batch)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
I also tested successfully sending into my TSI environment Microsoft's Windmill Simulator data by following this documentation: https://learn.microsoft.com/en-us/azure/time-series-insights/time-series-insights-send-events
I am now at a loss on how to actually get sample JSON data into the Azure TSI environment using Python.
Any help would be appreciated. Thank you!
As the article mentions here, you will have to send the body of the EventData with the JSON formatted String.
Sharing you the modified snippet of the your above code :
import asyncio
import nest_asyncio
nest_asyncio.apply()
from azure.eventhub.aio import EventHubProducerClient
from azure.eventhub import EventData
import json
async def run():
# Create a producer client to send messages to the event hub.
# Specify a connection string to your event hubs namespace and
# the event hub name.
producer = EventHubProducerClient.from_connection_string("<>", eventhub_name="<>")
async with producer:
# Create a batch.
event_data_batch = await producer.create_batch()
# Add events to the batch.
#Method 1 - You provide a JSON string
body1 = '{"id":"device2","timestamp":"2016-01-17T01:17:00Z"}'
event_data_batch.add(EventData(body1))
#Method 2 - You get the JSON Object and convert to string
json_obj = {"id":"device3","timestamp":"2016-01-18T01:17:00Z"}
body2= json.dumps(json_obj)
event_data_batch.add(EventData(body2))
#This just sending the string which will not be captured by TSI
event_data_batch.add(EventData('Third event'))
# Send the batch of events to the event hub.
await producer.send_batch(event_data_batch)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
Output :
I am using eventhub for ingesting a lot of events. I have multiple consumers which are running behing a scaling group reading these events from the eventhub which has multiple partitions. I was going through the Azure SDK in python and was confused as to what to use. There is eventhubconsumerclient, eventprocessorHost ....
I would like to use a library where my multiple consumer can connect using the consumer group, the partitions are assigned dynamically to each consumer and checkpointing is made in the storage account, just like how I used kafka.
Update:
For production usage, I suggest you should use the stable version of event hub sdk. You can use eph, sample code is here.
I can use the pre-release eventhub 5.0.0b6 to use consumer group as well as set checkpoint.
But the strange thing is that, in blob storage, I can see 2 folders created for the eventhub: checkpoint and ownership folder. Inside the folders, there're blob created for the partitions, but blob is empty. More stranger thing is that, even the blob is empty, every time I read from eventhub, it always read the latest data(means that it never reads the data has been read already in the same consumer group).
You need to install azure-eventhub 5.0.0b6 and use pip install --pre azure-eventhub-checkpointstoreblob to install azure-eventhub-checkpointstoreblob. For blob storage, you should install the latest version 12.1.0 of azure-storage-blob.
I follow this sample. In this sample, it uses event hub level connection string(NOT event hub namespace level connection string). You need to create an event hub level connection string by nav to azure portal -> your eventhub namespace -> your event hub instance -> Shared access policies -> click "Add" -> then specify a policy name, and select permission. If you just want to receive data, you can only select the Listen permission. The screenshot as below:
After the policy created, you can copy the connection string as per screenshot below:
Then you can follow this code below:
import os
from azure.eventhub import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblob import BlobCheckpointStore
CONNECTION_STR = 'Endpoint=sb://ivanehubns.servicebus.windows.net/;SharedAccessKeyName=saspolicy;SharedAccessKey=xxx;EntityPath=myeventhub'
STORAGE_CONNECTION_STR = 'DefaultEndpointsProtocol=https;AccountName=xx;AccountKey=xxx;EndpointSuffix=core.windows.net'
def on_event(partition_context, event):
# do something with event
print(event)
print('on event')
partition_context.update_checkpoint(event)
if __name__ == '__main__':
#the "a22" is the blob container name
checkpoint_store = BlobCheckpointStore.from_connection_string(STORAGE_CONNECTION_STR, "a22")
#the "$default" is the consumer group
client = EventHubConsumerClient.from_connection_string(
CONNECTION_STR, "$default", checkpoint_store=checkpoint_store)
try:
print('ok')
client.receive(on_event)
except KeyboardInterrupt:
client.close()
The test result:
azure-eventhub v5 has been GAed in 2020 Jan, and the latest version is v5.2.0
It's available on pypi: https://pypi.org/project/azure-eventhub/
Please follow the migration guide from v1 to v5 to migrate your program.
For receiving with checkpoint, please follow the sample code:
import os
import logging
from azure.eventhub import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblob import BlobCheckpointStore
CONNECTION_STR = os.environ["EVENT_HUB_CONN_STR"]
EVENTHUB_NAME = os.environ['EVENT_HUB_NAME']
STORAGE_CONNECTION_STR = os.environ["AZURE_STORAGE_CONN_STR"]
BLOB_CONTAINER_NAME = "your-blob-container-name" # Please make sure the blob container resource exists.
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
def on_event_batch(partition_context, event_batch):
log.info("Partition {}, Received count: {}".format(partition_context.partition_id, len(event_batch)))
# put your code here
partition_context.update_checkpoint()
def receive_batch():
checkpoint_store = BlobCheckpointStore.from_connection_string(STORAGE_CONNECTION_STR, BLOB_CONTAINER_NAME)
client = EventHubConsumerClient.from_connection_string(
CONNECTION_STR,
consumer_group="$Default",
eventhub_name=EVENTHUB_NAME,
checkpoint_store=checkpoint_store,
)
with client:
client.receive_batch(
on_event_batch=on_event_batch,
max_batch_size=100,
starting_position="-1", # "-1" is from the beginning of the partition.
)
if __name__ == '__main__':
receive_batch()
One more thing worth to note is that in V5, we use the metadata of blob to store checkpoint and ownership information instead of storing them as the content of a blob in v1. So it's expected that the content of a blob is empty when using the v5 sdk.
I am following this tutorial and I already have my code publishing messages to /devices/sm1/events topic, in which sm1 is my device id.
I would like to know how to subscribe to this topic since the tutorial says to use /devices/sm1/config but I am getting empty messages. I already tried use the same "path" used in publishing (/devices/sm1/events), but it also did not work.
It is strange that the name I gave to the topic was sm1 and the topic associated to my device is on GoogleIoT console is exhibited as projects/myprojectname/topics/sm1. So, besides to discover how to subscribe to mentioned topic, I appreciate also any explanation related to the correct way of using pub/sub topics in GoogleIoT (the documentation is not so clear).
This is my subscribe.py:
mqtt_url = "mqtt.googleapis.com"
mqtt_port = 8883
topic = "/devices/sm1/config"
def on_connect(client, userdata, flags, response_code):
print("Connected with status: {0}".format(response_code))
client.subscribe(topic, 1)
def on_message(client, userdata, msg):
print("Topic: {0} -- Payload: {1}".format(msg.topic, msg.payload))
if __name__ == "__main__":
client = mqtt.Client("projects/{}/locations/{}/registries/{}/devices/{}".format(
project_id,
cloud_region,
registry_id,
device_id))
client.username_pw_set(username='unused',
password=jwt_maker.create_jwt(project_id,
private_key,
algorithm="RS256"))
client.tls_set(root_ca,
certfile = public_crt,
keyfile = private_key,
cert_reqs = ssl.CERT_REQUIRED,
tls_version = ssl.PROTOCOL_TLSv1_2,
ciphers = None)
client.on_connect = on_connect
client.on_message = on_message
print "Connecting to Google IoT Broker..."
client.connect(mqtt_url, mqtt_port, keepalive=60)
client.loop_forever()
My output:
Connected with status: 0
Topic: /devices/sm1/config -- Payload:
Topic: /devices/sm1/config -- Payload:
After reading the discussion in the comments section in the answer by #GabeWeiss, I think there is a bit of confusion of what you want to achieve and how (or what for) are you trying to use Pub/Sub.
Given that I think the issue here is more conceptual, let me first refer you to a generic documentation page about Cloud IoT Core key concepts, where you will actually find some information regarding the relationship between Cloud IoT Core and Pub/Sub. In summary, device telemetry data is published into a Cloud IoT Core topic, which later, through the Data Broker, is published into a Cloud Pub/Sub topic, which you can use externally for other purposes: triggering Cloud Functions, analyzing the stream of data with Dataflow, etc.
Now, if you follow the Quickstart guide of Cloud IoT Core, you will see how, at a given point, you create a device registry which is bound to a Pub/Sub topic where device telemetry events are published. If instead of writing to the default Pub/Sub topic you wish to write to multiple topics, you can follow the explanations under this other section on the documentation.
Finally, getting to the issue of subscribing to topics (Cloud IoT Core topics, and not Pub/Sub topics, as only the former are relevant for the devices), you can subscribe to MQTT topics with the following command, where (as an example), the device is subscribing to the config topic, where configuration updates are published:
# This is the topic that the device will receive configuration updates on.
mqtt_config_topic = '/devices/{}/config'.format(device_id)
# Subscribe to the config topic.
client.subscribe(mqtt_config_topic, qos=1)
Then, with the on_message() function you can can process messages published on the topics where you are actually subscribed. Note that the payload has to be parsed as string (str(message.payload)).
def on_message(unused_client, unused_userdata, message):
payload = str(message.payload)
print('Received message \'{}\' on topic \'{}\' with Qos {}'.format(
payload, message.topic, str(message.qos)))
Then, in your question you stated that you first subscribed to /devices/{device-id}/config, but this might not be what you want, as this is the topic were configuration updates are published (i.e. not where telemetry events are published). Then, I understand that you should subscribe to /devices/{device-id}/events which is the actual MQTT topic where telemetry metrics are published. If that does not work, there might be another issue related, so make sure that you are parsing the message variable correctly, and maybe try to use Pub/Sub with the default topic created with the registry in order to check whether telemetry metrics are being properly published or not.
Edit: Clarifying based on comment below...
There's two GCP components in play here. There's the MQTT topic (which is the /events topic), which is used by the device to talk to IoT Core. Then there's the projects/myprojectname/topics/sm1 which isn't in IoT Core, it's in Pub/Sub. When you send messages to the /events MQTT topic, IoT Core brokers the payloads from your device that was sent to the /events MQTT topic through to the Pub/Sub topic that was created and attached to the IoT Core registry where your device was registered.
To see those messages, you have to create a subscription in Pub/Sub on the topic projects/myprojectname/topics/sm1. If you go to the console, and Pub/Sub->topics. Click the three dots next to the topic and select "New subscription". Once you have the subscription, you can send some data from your device, then on commandline you can run (assuming you have the gcloud tools installed):
gcloud beta pubsub subscriptions pull --max-messages=3 <subscription_id>
To do anything with the messages, you can script subscribing to the Pub/Sub topic (check out the Pub/Sub APIs) to trigger on messages being added to the topic.
Original message:
Are you sending a config message to the device? The confusion might be that the MQTT topics are one-directional.
So: 1) the /events topic is for device->IoT Core. 2) the /config topic is for IoT Core Admin SDK->device
In another script somewhere, or from the IoT Core UI interface you need to send a configuration message to see the on_message fire properly.
In the IoT Core UI (on console.cloud.google.com) you can drill down to an individual device you have registered, and at the top of the screen, click on "Update Config". A popup window will come up that lets you send a text or a base64 encoded message to that device and it will come in on the /config topic.
I had to surrender to Google Pub/Sub library in order to receive notifications related to my specified topic.
My publish code (only the important parts):
mqtt_url = "mqtt.googleapis.com"
mqtt_port = 8883
mqtt_topic = "/devices/sm1/events"
client = mqtt.Client("projects/{}/locations/{}/registries/{}/devices/{}".format(
project_id,
cloud_region,
registry_id,
device_id), protocol=mqtt.MQTTv311)
client.username_pw_set(username='unused',
password=jwt_maker.create_jwt(project_id,
private_key,
algorithm="RS256"))
client.tls_set(root_ca,
certfile = public_crt,
keyfile = private_key,
cert_reqs = ssl.CERT_REQUIRED,
tls_version = ssl.PROTOCOL_TLSv1_2,
ciphers = None)
client.connect(mqtt_url, mqtt_port, keepalive=60)
res = client.publish(mqtt_topic, some_data, qos=1)
In the Google Cloud Platform portal, I had to create a subscription, in the Pub/Sub section, assigning it to my created topic, which was already my default topic (probably linked to /events topic). The created subscription has the following format:
projects/project_name/subscriptions/subscription_name
My subscribe code, using the Google Pub/Sub library, since it is not possible to use the MQTT protocol:
from google.cloud import pubsub
def callback(message):
print(message.data)
message.ack()
project_id = "project_name"
subscription_name = "sm1"
subscriber = pubsub.SubscriberClient()
subscription_name = 'projects/{}/subscriptions/{}'.format(project_id, subscription_name)
subscription = subscriber.subscribe(subscription_name)
future = subscription.open(callback)
try:
future.result()
except Exception as ex:
subscription.close()
raise
I hope this can help anyone. More details can be found here(github).