Develop Reference

How do i send data to specific partition (let's say partition 0) in event hub

i have created 2 partitions in event hub
following the same code to send data to event hub
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/eventhub/azure-eventhub/README.md#publish-events-to-an-event-hub
below code i tried i was able to send data but was not able to send it to specific partition
def submit_images(jsondata):
connection_str = ['****************************']
eventhub_name = ['*********']
client = EventHubProducerClient.from_connection_string(connection_str, eventhub_name=eventhub_name)
event_data_batch = client.create_batch(partition_id=0)
event_data_batch.add(EventData(jsondata))
with client:
client.send_batch(event_data_batch)
return jsondata

the partition_id kwarg takes in a string value, so changing this line to event_data_batch = client.create_batch(partition_id='0') should send the data to the intended partition.
Another way to do that is to pass in a kwarg in to send_batch like this
client.send_batch(event_data_batch, partition_id='1').
Here is a sample that has more examples

Azure Function Python - serviceBusTrigger - How to deadletter a message

I have a plain simple Python function which should dead-letter a message if it does not match few constraint. Actually I'm raising an exception and everything works fine (I mean the message is being dead-lettered), but I would like to understand if there is a "clean" way to dead-letter the message without raising an exception.
async def function_handler(message: func.ServiceBusMessage, starter: str):
for msg in [message]:
client = df.DurableOrchestrationClient(starter)
message_body = msg.get_body().decode("utf-8")
msg = json.loads(message_body)
if 'valid' in msg:
instance_id = await client.start_new('orchestrator', None, json.dumps(message_body))
else:
raise Exception(f'not found valid {msg["id"]}')
This is part of host.json, this should indicate I'm working with version 2.0 of Azure Functions
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
},
Suggestions are welcome

At time of writing, in Python it is not possible interactively send a message in dead-letter.
I found out that autocomplete=false is only supported for C#.
This basically means that the only way to dead letter a message is raise an exception, just like I was doing in my code.
Thanks to #GauravMantri to pointing me the right way (i.e. have a look at how to use the autocomplete configuration parameter).

Azure Service Bus Queue has this Max Delivery Count property that you can make use of. Considering you only want to process a message exactly once and then deadletter the message in case Function is unable to process, what you can do is set the max delivery count to 1. That way the message will be automatically deadlettered after 1st delivery.
By default, Function runtime tries to auto-complete the message if there is no exception in processing the message. You do not want Function runtime to do that. For that what you would need to do is set auto complete setting to false. However if the message is processed successfully, you would want to delete that message thus you will need to call auto complete manually if the message processing is successful.
Something like:
if 'valid' in msg:
instance_id = await client.start_new('orchestrator', None, json.dumps(message_body))
//auto complete the message here...
else:
//do nothing and the message will be dead-lettered

Get Latest Message for a Confluent Kafka Topic in Python

Here's what I've tried so far:
from confluent_kafka import Consumer
c = Consumer({... several security/server settings skipped...
'auto.offset.reset': 'beginning',
'group.id': 'my-group'})
c.subscribe(['my.topic'])
msg = poll(30.0) # msg is of None type.
msg almost always ends up being None though. I think the issue might be that 'my-group' has already consumed all the messages for 'my.topic'... but I don't care whether a message has already been consumed or not - I still need the latest message. Specifically, I need the timestamp from that latest message.
I tried a bit more, and from this it looks like there are probably 25 messages in the topic, but I have no idea how to get at them:
a = c.assignment()
print(a) # Outputs [TopicPartition{topic=my.topic,partition=0,offset=-1001,error=None}]
offsets = c.get_watermark_offsets(a[0])
print(offsets) # Outputs: (25, 25)
If there are no messages because the topic has never had anything written to it at all, how can I determine that? And if that's the case, how can I determine how long the topic has existed for? I'm looking to write a script that automatically deletes any topics that haven't been written to in the past X days (14 initially - will probably tweak it over time.)

I run into the same issue, and no example on this. In my case there is one partition, and I need to read the last message, to know the some info from that message to setup the consumer/producer component I have.
Logic is that start Consumer, subscribe to topic, poll for message -> this triggers on_assign, where the rewinding happens, by assigning the modified partitions back. After on_assign finishes, the poll for msg continues and reads the last message from topic.
settings = {
"bootstrap.servers": "my.kafka.server",
"group.id": "my-work-group",
"client.id": "my-work-client-1",
"enable.auto.commit": False,
"session.timeout.ms": 6000,
"default.topic.config": {"auto.offset.reset": "largest"},
}
consumer = Consumer(settings)
def on_assign(a_consumer, partitions):
# get offset tuple from the first partition
last_offset = a_consumer.get_watermark_offsets(partitions[0])
# position [1] being the last index
partitions[0].offset = last_offset[1] - 1
consumer.assign(partitions)
consumer.subscribe(["test-topic"], on_assign=on_assign)
msg = consumer.poll(6.0)
Now msg is having the last message inside.

If anyone still needs an example for case with multiple partitions; this is how I did it:
from confluent_kafka import OFFSET_END, Consumer
settings = {
'bootstrap.servers': "my.kafka.server",
'group.id': "my-work-group",
'auto.offset.reset': "latest"
}
def on_assign(consumer, partitions):
for partition in partitions:
partition.offset = OFFSET_END
consumer.assign(partitions)
consumer = Consumer(settings)
consumer.subscribe(["test-topic"], on_assign=on_assign)
msg = consumer.poll(1.0)

Django Channels - Receive JSON objects

for a WebRTC project, I have to build up a signaling mechanism. I use Django Channels for that job. As you might know in WebRTC, there are Session Description objects that will be passed back and forth as "offer"/"answer" and also there are ICE candidate objects which will also be passed between two (or more) clients.
Playing with Django Channels a little bit, I have wrote the following consumer:
'''
A basic consumer that accepts WebSocket connections on the path
/ws/chat/ROOM_NAME/ that takes any message it receives on the
WebSocket and echos it back to the same WebSocket.
we want to have multiple instances of SignallingConsumer in the same room communicate
with each other. To do that we will have each SignallingConsumer add its channel
to a group whose name is based on the room name. That will allow SignallingConsumers
to transmit messages to all other SignallingConsumers in the same room.
'''
class SignallingConsumer(WebsocketConsumer):
def connect(self):
print("connect() is called.")
'''
Obtains the 'room_name' parameter from the URL route in chat/routing.py
that opened the WebSocket connection to the consumer.
Every consumer has a scope that contains information about its connection,
including in particular any positional or keyword arguments from the URL route
and the currently authenticated user if any.
'''
self.room_name = self.scope['url_route']['kwargs']['room_name']
'''
Constructs a Channels group name directly from the user-specified room name,
without any quoting or escaping.
Group names may only contain letters, digits, hyphens, and periods.
Therefore this example code will fail on room names that have other characters.
'''
self.room_group_name = 'chat_%s' % self.room_name
'''
Join room group by adding the channel name to the group.
The async_to_sync(…) wrapper is required because ChatConsumer is a synchronous
WebsocketConsumer but it is calling an asynchronous channel layer method.
(All channel layer methods are asynchronous.)
Group names are restricted to ASCII alphanumerics, hyphens, and periods only.
Since this code constructs a group name directly from the room name, it will
fail if the room name contains any characters that aren’t valid in a group
name.
'''
async_to_sync(self.channel_layer.group_add)(
self.room_group_name,
self.channel_name
)
'''
Accepts the WebSocket connection.
If you do not call accept() within the connect() method then the
connection will be rejected and closed. You might want to reject a
connection for example because the requesting user is not authorized
to perform the requested action.
It is recommended that accept() be called as the last action
in connect() if you choose to accept the connection.
'''
self.accept()
def disconnect(self, close_code):
print("disconnect() is called.")
# Leave room group
async_to_sync(self.channel_layer.group_discard)(
self.room_group_name,
self.channel_name
)
# Receive message from WebSocket
# text_data is of type String
def receive(self, text_data):
print("Received data: " + text_data)
# json.loads(): takes in a String and returns a JSON object
# text_data_json is of type JSONObject
text_data_json = json.loads(text_data)
async_to_sync(self.channel_layer.group_send)(
self.room_group_name,
{
'type': 'offer_answer_message',
'data': text_data_json
}
)
def offer_answer_message(self, event):
# we access the JSONObject via event['data'] and
# store it into data variable
data = event['data']
# json.dumps(): takes in a JSONObject and returns a String
self.send(json.dumps(data))
The comments are basically for myself. I have tweaked the code from the official site of Django Channels a little bit for my purpose. For my question, the receive() method is very important. The questions regarding this are the following ones:
So, what the consumer should do is to pass the offer (or answer)
from client A ( or client B ) to the other remote peer. Does this
work here ?
Also, the ICE candidates which contain the network information about a peer
should also be passed to the remote peer. Does this work here ?
The receive() method takes a String as parameter. For that reason, I have to convert my JSON objects to Strings on the client side before sending them. I don't like that solution. What I would rather do is passing the JSON objects directly to the receive() method on the server side. How I would do it ?

Yes, you can receive a offer or answer using this receive method say one is a host other is peer, you have already made a group... so just forward offer/answer accordingly to required consumer, keep track what type of message it is i.e. its a offer / answer.
Similarly when ever you set the remote description or the local description on client side a event onicecandidate must be handled by sending the candidate information to other client just like offer and answer..
PC.onicecandidate = (event)=>{
if(iceCand){
await Socket.send(JSON.stringify({
"who" : "user1",
"for" : "user2",
"candidate" : event.candidate
}));
}
};
You need to get it via JSON ... otherwise you get a no data for the candidate key in dict given in code above.. but for AsyncConsumer or SyncConsumer the receive method takes in a dict as far as I know, dict like given below:
event = {"type" : "websocket.receive", "text" : data}

Amazon SQS message parsing using python + boto

I am using boto library to read messages from SQS queue. My messages have text like this:
{ Command:XXXXXXXXXXX Key:XXXXXXX Input:XXXXXX} . Boto sends with base64 encoded and also reads it, so that if I read the message body then the text is there.
But how can I read the message like
Command = input['Command']
Key = input_message['Key'].split(',')
so that I can use those values for further processing...
I am quite new to Python also

Ok, you seem to have the input in some kind of a format - is it anything standardised? If not, you would need to parse the contents of your message and get the individual keys.
What I have been doing before in my projects was using JSON to facilitate data exchange between platforms.
If you do not have a luxury to edit your incoming data, you would need to do something like this (very naiive example):
input = "{ Command:XXXXXXXXXXX Key:XXXXXXX Input:XXXXXX }"
data = filter(lambda x: ":" in x, input.split())
message_dict = dict()
for item in data:
key, val = item.split(":")
message_dict[key] = val

Consider using good old fashioned JSON to easily send and receive dictionaries acrost the wire.
This test function verifies that the data format is very clear with JSON:
test_sqs.py
import json
import boto3
from moto import mock_sqs
#mock_sqs
def test_sqs():
sqs = boto3.resource('sqs', 'us-east-1')
queue = sqs.create_queue(QueueName='votes')
queue.send_message(MessageBody=json.dumps(
{'Command': 'drink', 'Key': 'beer', 'Input': 'tasty'}))
messages = queue.receive_messages()
assert len(messages) == 1
assert messages[0].body == (
'{"Input": "tasty", "Command": "drink", "Key": "beer"}')