Trying to use pyserial with asyncio on a windows machine.
Inspired by https://stackoverflow.com/a/27927704/1629704 my code is constantly watching a serial port for incoming data.
# This coroutine is added as a task to the event loop.
#asyncio.coroutine
def get_from_serial_port(self):
while 1:
serial_data = yield from self.get_byte_async()
<doing other stuff with serial_data>
# The method which gets executed in the executor
def get_byte(self):
data = self.s.read(1)
time.sleep(0.5)
tst = self.s.read(self.s.inWaiting())
data += tst
return data
# Runs blocking function in executor, yielding the result
#asyncio.coroutine
def get_byte_async(self):
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
res = yield from self.loop.run_in_executor(executor, self.get_byte)
return res
After serial data has been returned. the coroutine get_byte_async is called inside the while loop creating a new executor. I always learned creating a new thread is expensive so I feel I should take another approach, but I am not sure how to do that.
I've been reading this article https://hackernoon.com/threaded-asynchronous-magic-and-how-to-wield-it-bba9ed602c32#.964j4a5s7
And I guess I need to do the reading of the serial port in another thread. But how to get the serial data back to the "main" loop ?
You can either use the default executor and lock the access to get_byte with an asyncio lock:
async def get_byte_async(self):
async with self.lock:
return await self.loop.run_in_executor(None, self.get_byte)
Or simply create your own executor once:
async def get_byte_async(self):
if self.executor is None:
self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
return await self.loop.run_in_executor(self.executor, self.get_byte)
Related
I have a Python asyncio script that needs to run a long running task in a thread. During the operation of the thread, it needs to make network connections to another server. Is there any problem calling network/socket write functions in a thread as opposed to doing it in the main thread?
I know that in the Tiwsted library for example, one must always do network operations in the main thread. Are there any such limitations in asyncio? And if so, how does one get around this problem.
Here's my sample code:
import asyncio
import threading
#
# global servers dict keeps track of connected instances of each protocol
#
servers={}
class SomeOtherServer(asyncio.Protocol):
def __init__(self):
self.transport = None
def connection_made(self,transport):
self.transport=transport
servers["SomeOtherServer"] = self
def connection_lost(self):
self.transport=None
class MyServer(asyncio.Protocol):
def __init__(self):
self.transport = None
def connection_made(self,transport);
self.transport=transport
servers["MyServer"] = self
def connection_lost(self):
self.transport=None
def long_running_task(self,data):
# some long running operations here, then write data to other server
# other_server is also an instance of some sort of asyncio.Protocol
# is it ok to call this like this, even though this method is running in a thread?
other_server = servers["SomeOtherServer"]
other_server.transport.write(data)
def data_received(self,data):
task_thread = threading.Thread(target=self.long_running_task,args=[data])
task_thread.start()
async def main():
global loop
loop = asyncio.get_running_loop()
other_server_obj = await loop.create_server(lambda: SomeOtherServer(),"localhost",9001)
my_server_obj = await loop.create_server(lambda: MyServer(),"localhost",9002)
async with other_server_obj, my_server_obj:
while True:
await asyncio.sleep(3600)
asyncio.run(main())
Note that data_received will set up and call long_running_task in a thread, and long running_task makes a network connection to another server, and does so in the task thread, not the main thread. Is this ok or is there some other way this must be done?
I have a bog-standard synchronous python program that needs to be able to read data from websockets and update the GUI with the data. However, asyncio creep is constantly tripping me up.
How do I make a module that:
accepts multiple subscriptions to multiple sources
sends an update to the requester whenever there's data
opens exactly one websocket connection per URL
resets the websocket if it closes
Here's what I have already, but it's failing at many points:
run_forever() means that the loop gets stuck before the subscription completes and then handle() is stuck in the falsey while loop
it does not seem to want to restart sockets when they're down because a websockets object does not have a connected property (websocket without an s does, but I'm not clear on the differences and can't find info online either)
I'm absolutely not sure if my approach is remotely correct.
Been fighting with this for weeks. Would appreciate some pointers.
class WSClient():
subscriptions = set()
connections = {}
started = False
def __init__(self):
self.loop = asyncio.get_event_loop()
def start(self):
self.started = True
self.loop.run_until_complete(self.handle())
self.loop.run_until_forever() # problematic, because it does not allow new subscribe() events
async def handle(self):
while len(self.connections) > 0:
# listen to every websocket
futures = [self.listen(self.connections[url]) for url in self.connections]
done, pending = await asyncio.wait(futures)
# the following is apparently necessary to avoid warnings
# about non-retrieved exceptions etc
try:
data, ws = done.pop().result()
except Exception as e:
print("OTHER EXCEPTION", e)
for task in pending:
task.cancel()
async def listen(self, ws):
try:
async for data in ws:
data = json.loads(data)
# call the subscriber (listener) back when there's data
[s.listener._handle_result(data) for s in self.subscriptions if s.ws == ws]
except Exception as e:
print('ERROR LISTENING; RESTARTING SOCKET', e)
await asyncio.sleep(2)
self.restart_socket(ws)
def subscribe(self, subscription):
task = self.loop.create_task(self._subscribe(subscription))
asyncio.gather(task)
if not self.started:
self.start()
async def _subscribe(self, subscription):
try:
ws = self.connections.get(subscription.url, await websockets.connect(subscription.url))
await ws.send(json.dumps(subscription.sub_msg))
subscription.ws = ws
self.connections[subscription.url] = ws
self.subscriptions.add(subscription)
except Exception as e:
print("ERROR SUBSCRIBING; RETRYING", e)
await asyncio.sleep(2)
self.subscribe(subscription)
def restart_socket(self, ws):
for s in self.subscriptions:
if s.ws == ws and not s.ws.connected:
print(s)
del self.connections[s.url]
self.subscribe(s)
I have a bog-standard synchronous python program that needs to be able to read data from websockets and update the GUI with the data. However, asyncio creep is constantly tripping me up.
As you mentioned GUI, then it is probably not a "bog-standard synchronous python program". Usually a GUI program has a non-blocking event-driven main thread, which allows concurrent user behaviors and callbacks. That is very much similar to asyncio, and it is usually a common way for asyncio to work together with GUIs to use GUI-specific event loop to replace default event loop in asyncio, so that your asyncio coroutines just run in GUI event loop and you can avoid calling run_forever() blocking everything.
An alternative way is to run asyncio event loop in a separate thread, so that your program could at the same time wait for websocket data and wait for user clicks. I've rewritten your code as follows:
import asyncio
import threading
import websockets
import json
class WSClient(threading.Thread):
def __init__(self):
super().__init__()
self._loop = None
self._tasks = {}
self._stop_event = None
def run(self):
self._loop = asyncio.new_event_loop()
self._stop_event = asyncio.Event(loop=self._loop)
try:
self._loop.run_until_complete(self._stop_event.wait())
self._loop.run_until_complete(self._clean())
finally:
self._loop.close()
def stop(self):
self._loop.call_soon_threadsafe(self._stop_event.set)
def subscribe(self, url, sub_msg, callback):
def _subscribe():
if url not in self._tasks:
task = self._loop.create_task(
self._listen(url, sub_msg, callback))
self._tasks[url] = task
self._loop.call_soon_threadsafe(_subscribe)
def unsubscribe(self, url):
def _unsubscribe():
task = self._tasks.pop(url, None)
if task is not None:
task.cancel()
self._loop.call_soon_threadsafe(_unsubscribe)
async def _listen(self, url, sub_msg, callback):
try:
while not self._stop_event.is_set():
try:
ws = await websockets.connect(url, loop=self._loop)
await ws.send(json.dumps(sub_msg))
async for data in ws:
data = json.loads(data)
# NOTE: please make sure that `callback` won't block,
# and it is allowed to update GUI from threads.
# If not, you'll need to find a way to call it from
# main/GUI thread (similar to `call_soon_threadsafe`)
callback(data)
except Exception as e:
print('ERROR; RESTARTING SOCKET IN 2 SECONDS', e)
await asyncio.sleep(2, loop=self._loop)
finally:
self._tasks.pop(url, None)
async def _clean(self):
for task in self._tasks.values():
task.cancel()
await asyncio.gather(*self._tasks.values(), loop=self._loop)
You can try tornado and autobahn-twisted for websockets.
#asyncio.coroutine
def listener():
while True:
message = yield from websocket.recieve_message()
if message:
yield from handle(message)
loop = asyncio.get_event_loop()
loop.run_until_complete(listener())
Let's say i'm using websockets with asyncio. That means I recieve messages from websockets. And when I recieve a message, I want to handle the message but I'm loosing all the async thing with my code. Because the yield from handle(message) is definetly blocking... How could I find a way to make it non-blocking ? Like, handle multiple messages in the same time. Not having to wait the message to be handled before I can handle another message.
Thanks.
If you don't care about the return value from handle message, you can simply create a new Task for it, which will run in the event loop alongside your websocket reader. Here is a simple example:
#asyncio.coroutine
def listener():
while True:
message = yield from websocket.recieve_message()
if message:
asyncio.ensure_future(handle(message))
ensure_future will create a task and attach it to the default event loop. Since the loop is already running, it will get processed alongside your websocket reader in parallel. In fact, if it is a slow-running I/O blocked task (like sending an email), you could easily have a few dozen handle(message) tasks running at once. They are created dynamically when needed, and destroyed when finished (with much lower overhead than spawning threads).
If you want a bit more control, you could simply write to an asyncio.Queue in the reader and have a task pool of a fixed size that can consume the queue, a typical pattern in multi-threaded or multi-process programming.
#asyncio.coroutine
def consumer(queue):
while True:
message = yield from queue.get()
yield from handle(message)
#asyncio.coroutine
def listener(queue):
for i in range(5):
asyncio.ensure_future(consumer(queue))
while True:
message = yield from websocket.recieve_message()
if message:
yield from q.put(message)
q = asyncio.Queue()
loop = asyncio.get_event_loop()
loop.run_until_complete(listener(q))
I need at times more than one asyncio coroutine, where the routines then would be nested : coroutine B running in coroutine A, C in B and so on. The problem is with stopping a given loop. For example, using loop.stop() in the last top loop such as loop 'C' kills actually all asyncio coroutines - and not just this loop 'C'. I suspect that stop() actually kills coroutine A, and by doing so it annihilates all other dependent routines. The nested routines are call_soon_threadsafe, and all routines begin as run_forever.
I tried using specific loop names, or 'return', or 'break' (in the while loop inside the coroutine) but nothing exits the loop - except stop() which then kills non-specifically all loops at once.
My problem I described here is actually related to an earlier question of mine...
python daemon server crashes during HTML popup overlay callback using asyncio websocket coroutines
...which I thought I had solved - till running into this loop.stop() problem.
below now my example code for Python 3.4.3 where I try to stop() the coroutine_overlay_websocket_server loop as soon as it is done with the websocket job. As said, my code at the current state breaks all running loops. Thereafter the fmDaemon recreates a new asyncio loop that knows nothing of what was computed before :
import webbrowser
import websockets
import asyncio
class fmDaemon( Daemon):
# Daemon - see : http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
# Daemon - see : http://www.jejik.com/files/examples/daemon3x.py
def __init__( self, me):
self.me = me
def run( self):
while True:
#asyncio.coroutine
def coroutine_daemon_websocket_server( websocket, path):
msg = yield from websocket.recv()
if msg != None:
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
self.me = Function1( self.me, msg)
loop = asyncio.get_event_loop()
loop.run_until_complete( websockets.serve( coroutine_daemon_websocket_server, self.me.IP, self.me.PORT))
loop.run_forever()
def Function1( me, msg):
# doing some stuff :
# creating HTML file,
# loading HTML webpage with a webbrowser call,
# awaiting HTML button press signal via websocket protocol :
#asyncio.coroutine
def coroutine_overlay_websocket_server( websocket, path):
while True:
msg = yield from websocket.recv()
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
if msg == 'my_expected_string':
me.flags['myStr'] = msg
break
loop.call_soon_threadsafe( loop.stop)
loop = asyncio.get_event_loop()
loop.call_soon_threadsafe( asyncio.async, websockets.serve( coroutine_overlay_websocket_server, me.IP, me.PORT_overlay))
loop.run_forever()
loop.call_soon_threadsafe( loop.close)
# program should continue here...
My two questions : 1) Is there a way to exit a given coroutine without killing a coroutine lower down? 2) Or, alternatively, do you know of a method for reading websocket calls that does not make use of asyncio ?
I'm still a little confused by what you're trying to do, but there's definitely no need to try to nest event loops - your program is single-threaded, so when you call asyncio.get_event_loop() multiple times, you're always going to get the same event loop back. So you're really not creating two different loops in your example; both fmDaemon.run and Function1 use the same one. That's why stopping loop inside Function1 also kills the coroutine you launched inside run.
That said, there's no reason to try to create two different event loops to begin with. Function1 is being called from a coroutine, and wants to call other coroutines, so why not make it a coroutine, too? Then you can just call yield from websockets.serve(...) directly, and use an asyncio.Event to wait for coroutine_overlay_websocket_server to complete:
import webbrowser
import websockets
import asyncio
class fmDaemon( Daemon):
def __init__( self, me):
self.me = me
def run( self):
#asyncio.coroutine
def coroutine_daemon_websocket_server(websocket, path):
msg = yield from websocket.recv()
if msg != None:
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
self.me = Function1( self.me, msg)
loop = asyncio.get_event_loop()
loop.run_until_complete(websockets.serve(coroutine_daemon_websocket_server,
self.me.IP,
self.me.PORT))
loop.run_forever()
#asyncio.coroutine
def Function1(me, msg):
#asyncio.coroutine
def coroutine_overlay_websocket_server(websocket, path):
while True:
msg = yield from websocket.recv()
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
if msg == 'my_expected_string':
me.flags['myStr'] = msg
break
event.set() # Tell the outer function it can exit.
event = asyncio.Event()
yield from websockets.serve(coroutine_overlay_websocket_server, me.IP, me.PORT_overlay))
yield from event.wait() # This will block until event.set() is called.
I want to run my code asnychronously. What should I decorate with #asyncio.coroutine and what should I call with yield from for async operations?
In my case, I have some example code without decorator. (Simple chat bot look-like IRC)
import asyncio
class ChatBot:
def __init__(self, loop):
conn = asyncio.open_connection(HOST, PORT, loop=loop)
self.reader, self.writer = yield from conn
def send(self, msg):
self.writer.write(msg)
def read(self):
msg = yield from self.reader.readline()
return msg
def run(self):
while True:
msg = self.read()
self.parse(msg)
def parse(self, msg):
if msg.startswith('PING'):
self.some_work(msg)
elif msg.startswith('ERROR'):
self.some_error()
else:
self.server_log(msg)
def some_work(self, msg):
# some work. It can call asynchronous function like I/O or long operation. It can use self.send().
def some_error(self, msg):
# some work. It can call asynchronous function like I/O or long operation. It can use self.send().
def server_log(self, msg):
# some work. It can call asynchronous function like I/O or long operation. It can use self.send().
loop = asyncio.get_event_loop()
bot = ChatBot(loop)
loop.run_until_complete(???)
loop.close()
I think ??? is bot.run() and ChatBot.run must be decorated with #asyncio.coroutine. Then, how about other methods? I can't understand when use #asyncio.coroutine decorator and call method with yield from or asyncio.async. (I already read PEP-3156 for understanding asnycio. But I can't understand fully.)
When to use the #asyncio.coroutine decorator
If you have a function that needs to use yield from to call a coroutine, you should decorate it with asyncio.coroutine. Also note that coroutines are often (not always) "viral". As soon as you add yield from to a function it becomes a coroutine, and additionally any function that calls that coroutine usually (though not always) needs to be come a coroutine, too.
When to use asyncio.async
Why are coroutines not always viral? Because you actually don't always need to use yield from to call a coroutine. You only need to use yield from if you want to call a coroutine and wait for it to finish. If you just want to kick off a coroutine in the background, you can just do this:
asyncio.async(coroutine())
This will schedule coroutine to run as soon as control returns to the event loop; it won't wait for coroutine to finish before moving on to the next line. An ordinary function can use this to schedule a coroutine to run without also having to become a coroutine itself.
You can also use this approach to run multiple coroutines concurrently. So, imagine you have these two coroutines:
#asyncio.coroutine
def coro1():
yield from asyncio.sleep(1)
print("coro1")
#asyncio.coroutine
def coro2():
yield from asyncio.sleep(2)
print("coro2")
If you had this:
#asyncio.coroutine
def main():
yield from coro1()
yield from coro2()
yield from asyncio.sleep(5)
asyncio.get_event_loop().run_until_complete(main())
After 1 second, "coro1" would be printed. Then, after two more seconds (so three seconds total), "coro2" would be printed, and five seconds later the program would exit, making for 8 seconds of total runtime. Alternatively, if you used asyncio.async:
#asyncio.coroutine
def main():
asyncio.async(coro1())
asyncio.async(coro2())
yield from asyncio.sleep(5)
asyncio.get_event_loop().run_until_complete(main())
This will print "coro1" after one second, "coro2" one second later, and the program would exit 3 seconds later, for a total of 5 seconds of runtime.
How does this affect your code?
So following those rules, your code needs to look like this:
import asyncio
class ChatBot:
def __init__(self, reader, writer):
# __init__ shouldn't be a coroutine, otherwise you won't be able
# to instantiate ChatBot properly. So I've removed the code that
# used yield from, and moved it outside of __init__.
#conn = asyncio.open_connection(HOST, PORT, loop=loop)
#self.reader, self.writer = yield from conn
self.reader, self.writer = reader, writer
def send(self, msg):
# writer.write is not a coroutine, so you
# don't use 'yield from', and send itself doesn't
# need to be a coroutine.
self.writer.write(msg)
#asyncio.coroutine
def read(self):
msg = yield from self.reader.readline()
return msg
#asyncio.coroutine
def run(self):
while True:
msg = yield from self.read()
yield from self.parse(msg)
#asyncio.coroutine
def parse(self, msg):
if msg.startswith('PING'):
yield from self.some_work(msg)
elif msg.startswith('ERROR'):
yield from self.some_error()
else:
yield from self.server_log(msg)
#asyncio.coroutine
def some_work(self, msg):
# some work. It can call asynchronous function like I/O or long operation. It can use self.send().
#asyncio.coroutine
def some_error(self, msg):
# some work. It can call asynchronous function like I/O or long operation. It can use self.send().
#asyncio.coroutine
def server_log(self, msg):
# some work. It can call asynchronous function like I/O or long operation. It can use self.send()
#asyncio.coroutine
def main(host, port):
reader, writer = yield from asyncio.open_connection(HOST, PORT, loop=loop)
bot = ChatBot(reader, writer)
yield from bot.run()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
One other thing to keep in mind - adding yield from in front of a function doesn't magically make that call non-blocking. Neither does adding the #asyncio.coroutine decorator. Functions are only non-blocking if they're actually directly or indirectly calling native asyncio coroutines, which use non-blocking I/O and are integrated with the asyncio event loop. You mentioned making REST API calls, for example. In order for those REST API calls to not block the event loop, you'd need to use the aiohttp library, or asyncio.open_connection. Using something like requests or urllib will block the loop, because they're not integrated with `asyncio.
You should decorate everything that uses yield from, because the decorator asyncio.coroutine will take your function as a generator and do all the callback/async job while you're yielding.
In your case, run needs to be rewritten as the following:
#asyncio.coroutine
def run(self):
while True:
msg = yield from self.read()
yield from self.parse(msg)
Then, read and parse must be coroutine as well.
You should read about how asynchronous work before use it, it will help you a lot.
#asyncio.coroutine
def read(self):
msg = yield from self.reader.readline()
return msg
#asyncio.coroutine
def run(loop):
while True:
msg = yield from read()
yield from parse(msg)
loop = asyncio.get_event_loop()
loop.run_until_complete(run(loop))
loop.close()