I am using websocket-client for a websocket server and it appears that there is a race condition occurring in the on_message function below:
def on_message(self, ws, message):
"""
Called automatically by the `websocket.WebSocketApp`
Args:
message: message received by the websocket
"""
try:
message = json.loads(message)
# True when the message received is the Websocket connection confirmation
if 'result' in message.keys():
if type(message['result']) is str:
self.subscription_id = message['result']
return
msg_id = message['params']['result']['id']
# Do some stuff
self.last_msg_id = msg_id
except Exception as e:
print(f'Error processing message: {e}')
The issue I am having seems to be that last_msg_id is not being updated by each message in the order they are received. Since I expect a message to be received every few seconds, I was wondering if this function is executed in a new thread each time, added to some sort of task queue, or maybe something else entirely?
How might I be able to ensure that last_msg_id is always updated in the order that the messages come in? I thought about using a lock/mutex to ensure mutual exclusion in the body of the function.
Related
I am using a third party API that provides websocket functionality for continuous stream of messages.
I want to use variables that are defined out of the on_message() function. The structure of the code is as below:
soc = WebSocket()
Live_Data = {}
Prev_Data = {}
def on_message(ws, message):
if <check condition about Prev_Data>:
<Do something on Prev_Data and Live_Data>
def on_open(ws):
print("on open")
def on_error(ws, error):
print(error)
def on_close(ws):
print("Close")
# Assign the callbacks.
soc._on_open = on_open
soc._on_message = on_message
soc._on_error = on_error
soc._on_close = on_close
soc.connect()
But when I run this, it throws error that
Prev_data is referenced before it is assigned
I think this is because the on_message() method is asynchronous and the next on_message() method tries to access the Prev_Data before the first on_message() has finished writing it.
So what is the mechanism to synchronously access the Prev_Data and some other such variables here?
P.S: When I don't use the Prev_Data at all, the code runs fine.
I need to consume JSON messages from Rabbitmq and perform long-running tasks on each of these Jsons. I am using a job-queue mechanism with maxsize 1 and the channel pre-fetch count as 1 so that my code has to handle one message from Rabbitmq at a time. This part seems to be working. The code seems to fail when in the same thread from which the pika connection was made and then the long-running task is run, when the ack callback is called, the code is failing in an internal library throwing up error:
required argument is not an integer.
Have discovered that the error occurs when encode function in an internal library is called. It throws an error when it appends '>Q' string with the delivery_tag. Not sure what this string is though. The following is the piece of code where it fails:
def encode(self):
pieces = list()
pieces.append(struct.pack('>Q', self.delivery_tag))
My ack method and the code block where the callback is defined are as follows:
def __ack_message(self, delivery_tag, ack):
print("Inside the __ack_message function")
if self.channel.is_open:
try:
if ack:
self.channel.basic_ack(delivery_tag)
else:
self.channel.basic_nack(delivery_tag, requeue=False)
except Exception as e:
self.channel.basic_nack(delivery_tag, requeue=False)
print("Failure inside __ack_message consumer.py report: {}".format(e), "error")
pass
else:
pass
def do_work(self, ch, delivery_tag, body):
thread_id = threading.get_ident()
print(f'Thread id:{thread_id} Delivery tag: {delivery_tag} Message body: {body}')
# long-running work begins here
print(f"I am inside the message_consume")
if body != b'':
message = json.loads(body)
self.slave_object.push_to_job_queue(self.connection, message)
cb = functools.partial(self.__ack_message, ch, delivery_tag)
ch.connection.add_callback_threadsafe(cb)
def on_message(self, ch, method_frame, _header_frame, body, args):
print("Inside on_message function")
thrds = args
delivery_tag = method_frame.delivery_tag
t = threading.Thread(target=self.do_work, args=(ch, delivery_tag, body))
t.start()
thrds.append(t)
Trying to receive the messages from Pub/Sub SUbscription using StreamingPull but the callback function failed with timeout error concurrent.futures._base.TimeoutError .
Error:
streaming_pull_future.result(timeout=timeout)
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 447, in result
raise TimeoutError()"
def callback(message):
message.ack()
message_dict = {}
my_message = message.data.decode('utf8')
message_dict = eval(my_message)
data = json.dumps(message_dict)
data_json = json.loads(data)
# iterating over the json object and capturing & updating the json file
for i in pipeLine:
if i['moduleName'] == 'module1':
i['processingHistory'][0]['firstArg'] ['subArg'] = data_json[0]
i['processingHistory'][1]['firstArg'] ['anotherArg'] = data_json[1]
subscriber_r = pubsub.SubscriberClient()
subscription_path = subscriber_r.subscription_path(project_id, cvd_subscription_id)
streaming_pull_future = subscriber_r.subscribe(subscription_path, callback=callback)
print(f"Listening for messages on {subscription_path}..\n")
# Wrap subscriber in a 'with' block to automatically call close() when done.
with subscriber_r:
try:
streaming_pull_future.result(timeout=timeout)
except TimeoutError:
streaming_pull_future.cancel()
streaming_pull_future.result()
How to handle this error & fix it.
You are probably passing a value into timeout. When the timeout is set, it indicates how long you want the subscriber to receive messages. The call to streaming_pull_future.result(timeout=timeout) will error out after the timeout passes. If you want the subscriber to receive messages indefinitely, don't set the timeout in that statement (or set it to None).
im trying to build a simple server-client chatroom with python socket.
i have the following code:
def handle_connection(client):
while(1):
try:
message = receive_message()
broadcast(message["data"])
except: # for now i don't mind which exception
print("client disconnected")
def receive_message(client_socket):
try:
message_header = client_socket.recv(HEADER)
if len(message_header) == 0:
return False
message_length = int(message_header.decode("utf-8"))
message = client_socket.recv(message_length).decode("utf-8")
return {"header": message_header, "data": message}
except: # most likely will trigger when a client disconnects
return False
where receive_message() calls inside of it to client.recv(HEADER) and returns either False when there is no message, or {"header": msg_header, "data": msg} when everything is ok.
my question is: if client.recv() fails inside of receive_message() due to the client CLI closing, will it raise the exception and print "client disconnected", or not?
i did come up with the following solution i think works:
i defined a function called handle_disconnection() that handles all the content inside of the except in the code above.
def handle_connection(client_socket):
while 1:
try:
message = receive_message()
if not message:
handle_disconnection(client_socket)
break
broadcast(message["data"])
except: # client disconnected
handle_disconnection(client_socket)
break
is this a valid and/or right programming approach to the problem?
if this approach is wrong, how can i handle it correctly?
If client.recv() will raise an exception you will handle it inside of receive_message() and handle_connection() will not receive the exception.
I suggest you to identify situations when you want to control the flow with exceptions or with if-else. I think receive_message() should return a value of message or throw ConnectionError when there are connection issues. In case when there are no messages from the socket you can return None or raise NoMessagesAvailableError.
There is also a rule that tells you should catch specified exceptions, not all of them. Your code will print client disconnected when you are out of memory.
I have successfully created a method that pulls the messages from an SQS queue using long polling, that looks like this:
def dequeue_message(self, callback):
result = self.queue.receive_messages(MaxNumberOfMessages=1)
if len(result) != 0:
body = result[0].body
try:
callback(body)
result.delete()
except Exception as e:
print("message not dequeued because an error occurred"
"when running callback: " + str(e))
But I could not find a way to stop the poll without killing the running Python process (or, obviously, wait for the timeout). What can I do?
You can simply do this using a flag. Assume that a button should interrupt the polling process. You update a flag as soon as the button is pressed. Then when the polling returns a set of messages, you check the flag, and ignore processing the messages. Don't worry, the messages will still be in the queue. see: link
Amazon SQS doesn't automatically delete a message after receiving it
for you, in case you don't successfully receive the message (for
example, the consumers can fail or lose connectivity). To delete a
message, you must send a separate request which acknowledges that you
no longer need the message because you've successfully received and
processed it.
SAMPLE CODE:
# This is the flag to check for interruptions
is_interrupted = False
# This function will set the flag if an interruption is occurred
def interrupt_polling():
is_interrupted = True
def dequeue_message(self, callback):
result = self.queue.receive_messages(MaxNumberOfMessages=1)
# this is the check that will bypass the Polling process
# Handle this logic as required
if is_interrupted:
# assuming that this needs to be reset to False
is_interrupted = False
return
if len(result) != 0:
body = result[0].body
try:
callback(body)
result.delete()
except Exception as e:
print("message not dequeued because an error occurred"
"when running callback: " + str(e))
Hope this helps.