I have a plain simple Python function which should dead-letter a message if it does not match few constraint. Actually I'm raising an exception and everything works fine (I mean the message is being dead-lettered), but I would like to understand if there is a "clean" way to dead-letter the message without raising an exception.
async def function_handler(message: func.ServiceBusMessage, starter: str):
for msg in [message]:
client = df.DurableOrchestrationClient(starter)
message_body = msg.get_body().decode("utf-8")
msg = json.loads(message_body)
if 'valid' in msg:
instance_id = await client.start_new('orchestrator', None, json.dumps(message_body))
else:
raise Exception(f'not found valid {msg["id"]}')
This is part of host.json, this should indicate I'm working with version 2.0 of Azure Functions
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
},
Suggestions are welcome
At time of writing, in Python it is not possible interactively send a message in dead-letter.
I found out that autocomplete=false is only supported for C#.
This basically means that the only way to dead letter a message is raise an exception, just like I was doing in my code.
Thanks to #GauravMantri to pointing me the right way (i.e. have a look at how to use the autocomplete configuration parameter).
Azure Service Bus Queue has this Max Delivery Count property that you can make use of. Considering you only want to process a message exactly once and then deadletter the message in case Function is unable to process, what you can do is set the max delivery count to 1. That way the message will be automatically deadlettered after 1st delivery.
By default, Function runtime tries to auto-complete the message if there is no exception in processing the message. You do not want Function runtime to do that. For that what you would need to do is set auto complete setting to false. However if the message is processed successfully, you would want to delete that message thus you will need to call auto complete manually if the message processing is successful.
Something like:
if 'valid' in msg:
instance_id = await client.start_new('orchestrator', None, json.dumps(message_body))
//auto complete the message here...
else:
//do nothing and the message will be dead-lettered
Related
I'm trying to understand if grpc server using streams is able to wait for all client messages to be read in prior to sending responses.
I have a trivial application where I send in several numbers I'd like to add and return.
I've set up a basic proto file to test this:
syntax = "proto3";
message CalculateRequest{
int64 x = 1;
int64 y = 2;
};
message CalculateReply{
int64 result = 1;
}
service Svc {
rpc CalculateStream (stream CalculateRequest) returns (stream CalculateReply);
}
On my server-side I have implemented the following code which returns the answer message as the message is received:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
resultToOutput = request.x + request.y
yield contracts_pb2.CalculateReply(result=resultToOutput)
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
contracts_pb2_grpc.add_SvcServicer_to_server(
CalculatorServicer(), server)
server.add_insecure_port('localhost:9000')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
print( "We're up")
logging.basicConfig()
serve()
I'd like to tweak this to first read in all the numbers and then send these out at a later stage - something like the following:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
listToReturn = []
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
listToReturn.append (request.x + request.y)
# ...
# do some other stuff first before returning
for item in listToReturn:
yield contracts_pb2.CalculateReply(result=resultToOutput)
Currently, my implementation to write out later doesn't work as the code at the bottom is never reached. Is this by design that the connection seems to "close" before reaching there?
The grpc.io website suggests that this should be possible with BiDirectional streaming:
for example, the server could wait to receive all the client messages before writing its responses, or it could alternately read a message then write a message, or some other combination of reads and writes.
Thanks in advance for any help :)
The issue here is the definition of "all client messages." At the transport level, the server has no way of knowing whether the client has finished independent of the client closing its connection.
You need to add some indication of the client's having finished sending requests to the protocol. Either add a bool field to the existing CalculateRequest or add a top-level oneof with one of the options being something like a StopSendingRequests
Here's what I've tried so far:
from confluent_kafka import Consumer
c = Consumer({... several security/server settings skipped...
'auto.offset.reset': 'beginning',
'group.id': 'my-group'})
c.subscribe(['my.topic'])
msg = poll(30.0) # msg is of None type.
msg almost always ends up being None though. I think the issue might be that 'my-group' has already consumed all the messages for 'my.topic'... but I don't care whether a message has already been consumed or not - I still need the latest message. Specifically, I need the timestamp from that latest message.
I tried a bit more, and from this it looks like there are probably 25 messages in the topic, but I have no idea how to get at them:
a = c.assignment()
print(a) # Outputs [TopicPartition{topic=my.topic,partition=0,offset=-1001,error=None}]
offsets = c.get_watermark_offsets(a[0])
print(offsets) # Outputs: (25, 25)
If there are no messages because the topic has never had anything written to it at all, how can I determine that? And if that's the case, how can I determine how long the topic has existed for? I'm looking to write a script that automatically deletes any topics that haven't been written to in the past X days (14 initially - will probably tweak it over time.)
I run into the same issue, and no example on this. In my case there is one partition, and I need to read the last message, to know the some info from that message to setup the consumer/producer component I have.
Logic is that start Consumer, subscribe to topic, poll for message -> this triggers on_assign, where the rewinding happens, by assigning the modified partitions back. After on_assign finishes, the poll for msg continues and reads the last message from topic.
settings = {
"bootstrap.servers": "my.kafka.server",
"group.id": "my-work-group",
"client.id": "my-work-client-1",
"enable.auto.commit": False,
"session.timeout.ms": 6000,
"default.topic.config": {"auto.offset.reset": "largest"},
}
consumer = Consumer(settings)
def on_assign(a_consumer, partitions):
# get offset tuple from the first partition
last_offset = a_consumer.get_watermark_offsets(partitions[0])
# position [1] being the last index
partitions[0].offset = last_offset[1] - 1
consumer.assign(partitions)
consumer.subscribe(["test-topic"], on_assign=on_assign)
msg = consumer.poll(6.0)
Now msg is having the last message inside.
If anyone still needs an example for case with multiple partitions; this is how I did it:
from confluent_kafka import OFFSET_END, Consumer
settings = {
'bootstrap.servers': "my.kafka.server",
'group.id': "my-work-group",
'auto.offset.reset': "latest"
}
def on_assign(consumer, partitions):
for partition in partitions:
partition.offset = OFFSET_END
consumer.assign(partitions)
consumer = Consumer(settings)
consumer.subscribe(["test-topic"], on_assign=on_assign)
msg = consumer.poll(1.0)
I have created a proxy between QGC(Ground Control Station) and vehicle in Python. Here is the code:
gcs_conn = mavutil.mavlink_connection('tcpin:localhost:15795')
gcs_conn.wait_heartbeat()
print("Heartbeat from system (system %u component %u)" %(gcs_conn.target_system, gcs_conn.target_system))
vehicle = mavutil.mavlink_connection('tcp:localhost:5760')
vehicle.wait_heartbeat() # recieving heartbeat from the vehicle
print("Heartbeat from system (system %u component %u)" %(vehicle.target_system, vehicle.target_system))
while True:
gcs_msg = gcs_conn.recv_match()
if gcs_msg == None:
pass
else:
vehicle.mav.send(gcs_msg)
print(gcs_msg)
vcl_msg = vehicle.recv_match()
if vcl_msg == None:
pass
else:
gcs_conn.mav.send(vcl_msg)
print(vcl_msg)
I need to receive the messages from the QGC and then forward them to the vehicle and also receive the messages from the vehicle and forward them to the QGC.
When I run the code I get this error.
is there any one who can help me?
If you print your message before sending you'll notice it always fails when you try to send a BAD_DATA message type.
So this should fix it (same for vcl_msg):
if gcs_msg and gcs_msg.get_type() != 'BAD_DATA':
vehicle.mav.send(gcs_msg)
PD: I noticed that you don't specify tcp as input or output, it defaults to input. Than means both connections are inputs. I recommend setting up the GCS connection as output:
gcs_conn = mavutil.mavlink_connection('tcp:localhost:15795', input=False)
https://mavlink.io/en/mavgen_python/#connection_string
For forwarding MAVLink successfully a few things need to happen. I'm assuming you need a usable connection to a GCS, like QGroundControl or MissionPlanner. I use QGC, and my design has basic testing with it.
Note that this is written with Python3. This snippet is not tested, but I have a (much more complex) version tested and working.
from pymavlink import mavutil
import time
# PyMAVLink has an issue that received messages which contain strings
# cannot be resent, because they become Python strings (not bytestrings)
# This converts those messages so your code doesn't crash when
# you try to send the message again.
def fixMAVLinkMessageForForward(msg):
msg_type = msg.get_type()
if msg_type in ('PARAM_VALUE', 'PARAM_REQUEST_READ', 'PARAM_SET'):
if type(msg.param_id) == str:
msg.param_id = msg.param_id.encode()
elif msg_type == 'STATUSTEXT':
if type(msg.text) == str:
msg.text = msg.text.encode()
return msg
# Modified from the snippet in your question
# UDP will work just as well or better
gcs_conn = mavutil.mavlink_connection('tcp:localhost:15795', input=False)
gcs_conn.wait_heartbeat()
print(f'Heartbeat from system (system {gcs_conn.target_system} component {gcs_conn.target_system})')
vehicle = mavutil.mavlink_connection('tcp:localhost:5760')
vehicle.wait_heartbeat()
print(f'Heartbeat from system (system {vehicle.target_system} component {vehicle.target_system})')
while True:
# Don't block for a GCS message - we have messages
# from the vehicle to get too
gcs_msg = gcs_conn.recv_match(blocking=False)
if gcs_msg is None:
pass
elif gcs_msg.get_type() != 'BAD_DATA':
# We now have a message we want to forward. Now we need to
# make it safe to send
gcs_msg = fixMAVLinkMessageForForward(gcs_msg)
# Finally, in order to forward this, we actually need to
# hack PyMAVLink so the message has the right source
# information attached.
vehicle.mav.srcSystem = gcs_msg.get_srcSystem()
vehicle.mav.srcComponent = gcs_msg.get_srcComponent()
# Only now is it safe to send the message
vehicle.mav.send(gcs_msg)
print(gcs_msg)
vcl_msg = vehicle.recv_match(blocking=False)
if vcl_msg is None:
pass
elif vcl_msg.get_type() != 'BAD_DATA':
# We now have a message we want to forward. Now we need to
# make it safe to send
vcl_msg = fixMAVLinkMessageForForward(vcl_msg)
# Finally, in order to forward this, we actually need to
# hack PyMAVLink so the message has the right source
# information attached.
gcs_conn.mav.srcSystem = vcl_msg.get_srcSystem()
gcs_conn.mav.srcComponent = vcl_msg.get_srcComponent()
gcs_conn.mav.send(vcl_msg)
print(vcl_msg)
# Don't abuse the CPU by running the loop at maximum speed
time.sleep(0.001)
Notes
Make sure your loop isn't being blocked
The loop must quickly check if a message is available from one connection or the other, instead of waiting for a message to be available from a single connection. Otherwise a message on the other connection will not go through until the blocking connection has a message.
Check message validity
Check that you actually got a valid message, as opposed to a BAD_DATA message. Attempting to send BAD_DATA will crash
Make sure the recipient gets the correct information about the sender
By default PyMAVLink, when sending a message, will encode YOUR system and component IDs (usually left at zero), instead of the IDs from the message. A GCS receiving this may be confused (ie, QGC) and not properly connect to the vehicle (despite showing the messages in MAVLink inspector).
This is fixed by hacking PyMAVLink such that your system and component IDs match the forwarded message. This can be revered after the message is sent if necessary. See the example to see how I did it.
Loop update rate
It's important that the update rate is fast enough to handle high traffic conditions (especially, say, for downloading params), but it shouldn't peg out the CPU either. I find that a 1000hz update rate works well enough.
I wrote a Telegram bot in Python, which is running on my Raspberry Pi (Raspbian). I eventually see an error after the bot has been running for a few hours.
Before I post the complete code, could someone please help me understand the error? I would like to run the bot indefinitely, or at least for multiple days, before I need to restart it.
The error is as follows:
Traceback (most recent call last):
File "/home/pi/Schreibtisch/StreamrPreisBot/telepot/loop.py", line 37, in run_forever
self._handle(msg)
File "/home/pi/Schreibtisch/StreamrPreisBot/streamrinfobot.py", line 32, in handle
command = msg['text']
KeyError: 'text'
Edit:
Following code is used:
def handle(msg):
chat_id = msg['chat']['id']
command = msg['text']
Might this code solve the problem?
def handle(msg):
chat_id = msg['chat']['id']
command = msg.get('text')
Error says there is no text key inside msg dict. Maybe it's some special telegram message that has no text or there is a bug in you code that delete text key in some cases. You could use
command = msg.get('text')
To get None when there is no text. Or
command = msg.get('text', '')
To get empty string (i.e. '') when there is no text.
You could also check that there is a text inside msg or not with in operator:
if 'text' not in msg:
logger.error('bad message received!')
return
If you want to your service to be always up you should add some mechanism for automatic restart.
like in Python to restart after any error:
while True:
try:
logger.info("Starting bot")
run_bot()
except Exception:
logger.exception("Something bad happened. Restarting")
I also suggest to log errors in a file or services such as Sentry to investigate why there is no text afterwards.
A KeyError is raised when a value is requested from a dict but the key does not exist in the dictionary.
So, in your case, the msg dictionary does not have the key text.
You should inspect your code to ensure that the msg dictionary contains a value associated with the key text. Or, if you expect that msg will sometimes not contain the key text, you should instead access the dictionary with the get method, which never raises a KeyError. See the docs for more information.
I'm using ROS in my project and I need to send one message from time to time. I have this function:
void RosNetwork::sendMessage(string msg, string channel) {
_mtx.lock();
ros::Publisher chatter_pub = _n.advertise<std_msgs::String>(channel.c_str(),10);
ros::Rate loop_rate(10);
std_msgs::String msgToSend;
msgToSend.data = msg.c_str();
chatter_pub.publish(msgToSend);
loop_rate.sleep();
cout << "Message Sent" << endl;
_mtx.unlock();
}
And I have this in python:
def callbackFirst(data):
#rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)
print("Received message from first filter")
def callbackSecond(data):
#rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)
print("Received message from second filter")
def listener():
rospy.Subscriber("FirstTaskFilter", String, callbackFirst)
print("subscribed to FirstTaskFilter")
rospy.Subscriber("SecondTaskFilter", String, callbackSecond)
print("subscribed to SecondTaskFilter")
rospy.spin()
The listener is a thread in python.
I get to the function sendMessage (I see in the terminal "Message Sent" a lot of times) but I don't see that the python script receives the message.
Update: I tested the python callback with rostopic pub /FirstTaskFilter std_msgs/String "test" and this works perfectly.
Any thought?
You are re-advertising the publisher every time and then you are immediately using it to publish something.
This is problematic as it needs some time for the subscribers to subscribe to newly emerging publishers. If you are publishing messages before the subscriber has finished with this, these messages will not arrive.
To avoid this problem, do not advertise a new publisher every time, but do it only once in the constructor of your class and store the publisher in a member variable. Your code could look something like this:
RosNetwork() {
_chatter_pub = _n.advertise<std_msgs::String>(channel.c_str(),10);
ros::Duration(1).sleep(); // optional, to make sure no message gets lost
}
void RosNetwork::sendMessage(string msg, string channel) {
...
_chatter_pub.publish(msgToSend);
...
}
The one-second-sleep after advertise makes sure that all existing subscribers can subscribe before you start publishing messages. This is only necessary, if it is important that not a single message gets lost. In most practical cases it can be omitted.
The proper solution to your problem is to use pub.getNumSubscribers() and wait until that is > 0. Then publish.