I want to implement paho-mqtt in which it should process the incoming messages asynchronously.
I have implement gmqtt with asyncio which runs perfectly fine, but as far as I understand paho-mqtt is better to use rather than gmqtt (Link : https://www.emqx.io/blog/comparision-of-python-mqtt-client).
gmqtt w/ asyncio :
def assign_callbacks_to_client(self, client):
""" Helper function which sets up client's callbacks. """
client.on_connect = self.on_connect
client.on_message = self.on_message
client.on_disconnect = self.on_disconnect
client.on_subscribe = self.on_subscribe
async def subscriber(self, mqtt_name):
""" Connect to mqtt-broker. """
sub_client = MQTTClient(mqtt_name)
self.assign_callbacks_to_client(sub_client)
logging.info("connecting")
await sub_client.connect(host=config.MQTT_HOST, port=int(config.MQTT_PORT))
return sub_client
could you please let me know, how to implement paho with asyncio library?
Will using loop_start asynchronously, as I understand everytime it executes it starts a new thread in background.
loop_start() only create a single background thread that all the callbacks will be run on, you should not be doing long running tasks directly in these callbacks as they will block all other actions of the client.
If you want to process incoming messages without blocking then you will need to implement your own thread pool and just use the on_message() callback to push messages to this thread pool.
Related
I have a TCP server running and have a handler function which needs to take the contents of the request, add it to an asyncio queue and reply with an OK status.
On the background I have an async coroutine running that detects when a new item is added and performs some processing.
How do I put items in the asyncio queue from the handler function, which isn't and can't be an async coroutine?
I am running a DICOM server pynetdicom which listens on port 104 for incoming TCP requests (DICOM C-STORE specifically).
I need to save the contents of the request to a queue and return a a 0x0000 response so that the listener is available to the network.
This is modeled by a producer-consumer pattern.
I have tried to define a consumer co-routine consume_dicom() that is currently stuck in await queue.get() since I can't properly define the producer.
The producer needs to simply invoke queue.put(produce_item) but this happens inside a handle_store(event) function which is not part of the event_loop but is called every time a request is received by the server.
import asyncio
from pynetdicom import (
AE, evt,
StoragePresentationContexts
)
class PacsServer():
def __init__(self, par, listen=True):
# Initialize other stuff...
# Initialize DICOM server
ae = AE(ae_title='DICOM-NODE')
ae.supported_contexts = StoragePresentationContexts
# When a C-STORE request comes, it will be passed to self.handle_store
handlers = [(evt.EVT_C_STORE, self.handle_store)]
# Define queue
loop = asyncio.get_event_loop()
self.queue = asyncio.Queue(loop=loop)
# Define consumer
loop.create_task(self.consume_dicom(self.queue))
# Start server in the background with specified handlers
self.scp = ae.start_server(('', 104), block=False, evt_handlers=handlers)
# Start async loop
self.loop.run_forever()
def handle_store(self, event):
# Request handling
ds = event.dataset
# Here I want to add to the queue but this is not an async method
await queue.put(ds)
return 0x0000
async def consume_dicom(self, queue):
while True:
print(f"AWAITING FROM QUEUE")
ds = await queue.get()
do_some_processing(ds)
I would like to find a way to add items to the queue and return the OK status in the handle_store() function.
Since handle_store is running in a different thread, it needs to tell the event loop to enqueue the item. This is done with call_soon_threadsafe:
self.loop.call_soon_threadsafe(queue.put_nowait, ds)
Note that you need to call queue.put_nowait instead of queue.put because the former is a function rather than a coroutine. The function will always succeed for unbounded queues (the default), otherwise it will raise an exception if the queue is full.
Sorry for the long post but I've been poking at this for over a week so I've tried a lot of different stuff. I know Python well enough but I don't have any experience with asyncio or non-blocking functions in Python.
I'm writing an API library/module/package/whatever for a web service that requires a websocket connection. There are many incoming messages to act on, and some control-related (web app level, not websocket control messages) that I need to send on occasion. I can easily receive messages over the connection and act on them. I can send messages, but only in response to received messages because the receive loop is always blocking waiting for messages. I don't want to wait for an incoming messages to process an outgoing one so the script doesn't have to hang on input until a new messages is received. In my struggles to get two-way communication working as desired I discovered I need to use something like Twisted, Tornado, or asyncio but so far every implementation I've tried has failed. Note that the sending has to happen over the same connection. Opening a short-lived connection outside of the receive loop will not work. Here's what I've done so far:
The first iteration of the websocket code was using the websocket-client package. It was very close to the example from the docs:
import websocket
try:
import thread
except ImportError:
import _thread as thread
import time
def on_message(ws, message):
# Send message frames to respective functions
# for sorting, objectification, and processing
def on_error(ws, error):
print(error)
def on_close(ws):
print("### closed ###")
def on_open(ws):
def run(*args):
# Send initial frames required for server to send the desired frames
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp(buildWebsocketURL()),
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()
This blocks any further execution outside of the loop. I tried learning up on the _thread module but I couldn't find any indication that I could "communicate" with the websocket thread from outside. I tried setting up a pub/sub listener function that would forward data to ws.send() from another sender function but it didn't work. No errors or anything, just no indication of any sent messages.
Next I tried the Websockets module. This one seems to be built from the ground up to utilize asyncio. Again, I got a client build that would send initial messages and act on received messages but the progress stopped there:
async def wsconnection():
async with websockets.connect(getWebsocketURL()) as websocket:
while True:
message = await websocket.recv()
if message == '{"type":"broadcaster.ready"}':
subscriptions = getSubscriptions() # Get subscriptions from ident data
logging.info('Sending bookmarks to server as subscription keys')
subscriptionupdate = '{{"type": "subscribe","subscription_keys": ["{0}"],"subscription_scope": "update"}}'.format(
'","'.join(subscriptions))
subscriptioncontent = '{{"subscription_keys": ["{0}"],"subscription_scope": "content","type": "subscribe"}}'.format(
'","'.join(subscriptions))
logging.debug(subscriptioncontent)
await websocket.send(subscriptionupdate)
await websocket.send(subscriptioncontent)
await websocket.send(
'{"type":"message_lobby.read","lobby_id":"1","message_id:"16256829"}')
sortframe(message)
asyncio.get_event_loop().run_until_complete(wsconnection())
I tried the aforementioned pub/sub listener applied here to no avail. Upon reading the docs for this module more thoroughly I tried getting the websocket protocol object (that contains the send() and recv() methods) outside of the loop then creating two coroutines(?), one listening for incoming messages and one listening for and sending outgoing messages. So far I've been completely unable to get the websocket protocol object without running the async with websockets.connect(getWebsocketURL()) as websocket: line within the scope of the wsconnection() function. I tried using websocket = websockets.client.connect() which according to the docs I thought would set the protocol object I need but it doesn't. All of the examples I can find don't seem to reveal any apparent way to structure the websockets sender and receiver in the way I require without extensive knowledge of asyncio.
I also poked around with autobahn with similar code structures as above using both asyncio and Twisted but I came up with all the same problems as above.
So far the closest I've gotten was with the Websockets package above. The docs have an example snippet for a send/recv connection but I can't really read what's going on there as it's all very specific to asyncio. I'm really having trouble wrapping my head around asyncio in general and I think a big problem is it seems to have very rapidly evolved recently so there is a ton of very version-specific information floating around that conflicts. Not good for learning, unfortunately. ~~~~This is what I tried using that example and it connects, receives initial messages, then the connection is lost/closed:
async def producer(message):
print('Sending message')
async def consumer_handler(websocket, path):
while True:
message = await websocket.recv()
await print(message)
await pub.sendMessage('sender', message)
async def producer_handler(websocket, path):
while True:
message = await producer()
await websocket.send(message)
async def wsconnect():
async with websockets.connect(getWebsocketURL()) as websocket:
path = "443"
async def handler(websocket, path):
consumer_task = asyncio.ensure_future(
consumer_handler(websocket, path))
producer_task = asyncio.ensure_future(
producer_handler(websocket, path))
done, pending = await asyncio.wait(
[consumer_task, producer_task],
return_when=asyncio.FIRST_COMPLETED,
)
for task in pending:
task.cancel()
pub.subscribe(producer, 'sender')
asyncio.get_event_loop().run_until_complete(wsconnect())
So how do I structure this code to get sending and receiving over the same websocket connection? I also have various API calls to make in the same script while the websocket connection is open which further complicates things.
I'm using Python 3.6.6 and this script is intended to be imported as a module into other scripts so the websocket functionality will need to be wrapped up in a function or class for external calls.
I am in the exact same situation as u. I know that this is a very inelegant solution
because it still isn't full-duplex but i can't seem to find any example on the internet or stackoverflow involving asyncio and the websockets module which i used.
I don't think i completely understand your websockets example (is that server-side or client-side code?) but i'm going to explain my situation and "solution" and maybe that would be usable for you too.
So i have a server main function that has a websocket listening for messages in a loop with recv(). When i send "start" it will start a function that will send data every second to the javascript client in the browser. But while the function is sending data i sometimes want to pause or stop the stream of data from my client be sending a stop message. The problem is that when i use recv() while the data sending has already begun the server stops sending data and only waits for a message. I tried threads,multiprocessing and some other stuff but eventually i came to the hopefully temporarily solution of sending a "pong" message to the server immediately after the client receives a piece of data so that the server continues sending data at the next loop iteration or stop sending data if the "pong" message is "stop" instead for example but yeah this is not real duplex just fast half-duplex...
code on my python "server"
async def start_server(self,websocket,webserver_path):
self.websocket = websocket
self.webserver_path = webserver_path
while True:
command = await self.websocket.recv()
print("received command")
if command == "start":
await self.analyze()
asyncio.sleep(1)
in my analyze function:
for i,row in enumerate(data)
await self.websocket.send(json.dumps(row))
msg = await self.websocket.recv()
if msg == "stop":
self.stopFlag = True
return
await asyncio.sleep(1)
main
start_server = websockets.serve(t.start_server, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
code on the javascript client
var ws = new WebSocket("ws://127.0.0.1:5678/");
ws.onmessage = function (event) {
var datapoint = JSON.parse(event.data);
console.log(counter);
counter++;
data.push(datapoint);
if (data.length > 40){
var element = data.shift();
render(data);
}
ws.send("pong");//sending dummy message to let server continue
};
I know it is not THE solution and i hope somebody else provides a better one but since i have the same or very similar problem and there are no other answers i decided to post and i hope it helps.
I'm using a python websockets library in order to create a websocket server. My goals is to export a synchronous API, as it's going to be used outside of python.
As such, I need, at least at the beginning, start() and stop() methods. So it seems right to create a websocket server class for that.
Main issue it that the way to create (and start) a server through the library is by awaiting, and so these methods should encouraged to be async, which I try to avoid.
The following code work perfectly when I run the main() function.
When runnig server = WebsocketServer(); server.start(1234) through ipython shell I can't seem to connect through a client code. What am I missing?
class WebsocketServer():
def __init__(self):
self._server = None
def start(self, port):
start_server = websockets.serve(self._handle_client,
host='localhost',
port=port)
self._server = asyncio.get_event_loop().run_until_complete(start_server)
def stop(self):
if self._server:
self._server.close()
self._server = None
async def _handle_client(self, client, path):
async for buffer in client:
await self._handle_buffer(client, buffer)
async def _handle_buffer(self, client, buffer):
print(buffer)
await client.send(buffer)
def main():
server = WebsocketServer()
server.start(1234)
asyncio.get_event_loop().run_forever()
The synchronous interface into the IO loop is via tasks. Scheduling methods return a future that can be synchronously waited on if needed. The run an event loop section of the docs features a combo for synchronous shutdown in the bottom.
When running inside an iPython shell, one option is to spawn a daemon background thread for the IO loop and register an atexit callback to synchronously shutdown the IO loop when the shell exits.
Another option is to "borrow" shell's thread once in a while for the IO tasks (only works for short tasks, of cause) using the UI Event Loop integration point described here describing how to borrow the shell thread for the IO.
You are missing the last line from you main function.
asyncio.get_event_loop().run_forever()
Nothing happens when the loop is not running. So, the server won't be running unless you run the loop.
I am writing a Python 3.5 program which handles some signals and serves this data to a small amount of websocket clients.
I want the websocket server and the signal handling to happen in the same program, therefore I am using threading.
The problem is I don't know how to send data from the worker thread to the client.
The Websocket server is implemented with a simple library called "websockets". The server is set up and clients can connect and talk to the server within the "new websocket client has connected" handler.
The server is set up with the help of an event loop:
start_server = websockets.serve(newWsHandler, host, port)
loop = asyncio.get_event_loop()
loop.run_until_complete(start_server)
loop.run_forever()
Because I want my program to do signal handling too, and loop.run_forever() is a blocking call, I create an endless worker thread before I start my server. This works as expected.
When the worker thread detects a signal change, it has to alert the connected websocket clients. But a simple client.send() does not work. Putting await in front of it does not work either (since that only works within coroutines, I think). I tried making a separate "async def" function and adding it to the event loop, but it gets a bit complicated because it's not on the same thread.
So the main question is: what is the best way send something to a websocket client from a worker thread? I don't receive anything in response.
EDIT:
It will probably help if I add some mock code.
def signalHandler():
#check signals
...
if alert:
connections[0].send("Alert") #NEED HELP HERE
async def newWsHandler(websocket, path):
connections.append(websocket)
while True:
#keep the connection open until the client disconnects
msg = await websocket.recv()
#top level
connections = []
...
start_server = websockets.serve(newWsHandler, host, port)
signalThread = Thread(target = signalHandler)
signalThread.setDaemon(True)
signalThread.start()
loop = asyncio.get_event_loop()
loop.run_until_complete(start_server)
loop.run_forever()
I have a small asynchronous server implemented using bottle and gevent.wsgi. There is a routine used to implement long poll that looks pretty much like the "Event Callbacks" example in the bottle documentation:
def worker(body):
msg = msgbus.recv()
body.put(msg)
body.put(StopIteration)
#route('/poll')
def poll():
body = gevent.queue.Queue()
worker = gevent.spawn(worker, body)
return body
Here, msgbus is a ZMQ sub socket.
This all works fine, but if a client breaks the connection while
worker is blocked on msgbus.recv(), that greenlet task will hang
around "forever" (well, until a message is received), and will only
find out about the disconnected client when it attempts to send a
response.
I can use msgbus.poll(timeout=something) if I don't want to block
forever waiting for ipc messages, but I still can't detect a client
disconnect.
What I want to do is get something like a reference to the client
socket so that I can use it in some kind of select or poll loop,
or get some sort of asynchronous notification inside my greenlet, but
I'm not sure how to accomplish either of these things with these
frameworks (bottle and gevent).
Is there a way to get notified of client disconnects?
Aha! The wsgi.input variable, at least under gevent.wsgi, has an rfile member that is a file-like object. This doesn't appear to be required by the WSGI spec, so it might not work with other servers.
With this I was able to modify my code to look something like:
def worker(body, rfile):
poll = zmq.Poller()
poll.register(msgbus)
poll.register(rfile, zmq.POLLIN)
while True:
events = dict(poll.poll())
if rfile.fileno() in events:
# client disconnect!
break
if msgbus in events:
msg = msgbus.recv()
body.put(msg)
break
body.put(StopIteration)
#route('/poll')
def poll():
rfile = bottle.request.environ['wsgi.input'].rfile
body = gevent.queue.Queue()
worker = gevent.spawn(worker, body, rfile)
return body
And this works great...
...except on OpenShift, where you will have to use the
alternate frontend on port 8000 with websockets support.