Since examples of asynchronous programming are still fairly sparse, I am trying to come up with my own using the new async def. I figured the most basic example I could make would be an event loop which listens to the user inputs and prints them back.
I want an event loop which listens to the keys the user presses and prints them every second. I'm trying to do so by having one task that prints from a queue of keys and a second task that listens to pressed keys and adds them to the queue.
I am missing a good way to asynchronously listen to key strokes. Here is what I have for now.
import asyncio
KEY_QUEUE = []
async def printer():
while True:
await asyncio.sleep(1)
print('In the last second you pressed:', *KEY_QUEUE)
KEY_QUEUE.clear()
async def listener():
while True:
... # await a key to be pressed and add it to KEY_QUEUE
loop = asyncio.get_event_loop()
loop.create_task(printer())
loop.create_task(listener())
loop.run_forever()
Expected output would look like this
In the last second you pressed: h e l l o
In the last second you pressed: w o r
In the last second you pressed: l d
In the last second you pressed:
In the last second you pressed: i t w o
In the last second you pressed: r k s
I doubt there is an awaitable coroutine such as asyncio.await_pressed_key, how would we proceed to create one?
Any other approach is welcome as well as my goal is not to make this specific example work, but rather to generate meaningful asynchronous programmign examples.
Assuming that you want to read characters from a TTY (terminal device on Unix-like system), accessing characters as they are typed requires:
the TTY to be in "raw mode", so that the system returns characters without waiting for a newline to be typed by the user;
the standard input, which are blocking, to be wrapped into an asyncio non-blocking stream.
The former is provided by the tty module that comes with the Python standard library, and the latter by the aioconsole third-party library.
With those two, your example could look like this:
import asyncio, aioconsole, sys, tty
async def main():
loop = asyncio.get_event_loop()
typed = []
p = loop.create_task(printer(typed))
await listener(typed)
p.cancel()
async def listener(typed):
tty.setraw(sys.stdin.fileno())
stdin, _ = await aioconsole.stream.get_standard_streams()
while True:
ch = await stdin.read(1)
if ch == b'\x03': # ctrl-c
break
typed.append(ch)
tty.setcbreak(sys.stdin.fileno())
async def printer(typed):
while True:
await asyncio.sleep(1)
print('In the last second you pressed', typed, end='\r\n')
del typed[:]
asyncio.get_event_loop().run_until_complete(main())
This example will only work on Unix-like systems.
Related
A program I am developing has a long running process in another thread. I would like to interrupt that thread in the event something goes awry.
Of other SO posts I've seen like this, they use syntax similar to this:
while True:
if condition_here:
break
else:
await asyncio.sleep(1)
which does work in catching KeyboardInterrupts. However, I'm not a big fan of using while loops like this and would like to avoid this if at all possible.
For some example code, here is what I currently have (which does not catch the interrupts until after the thread is done):
import asyncio
import time
from threading import Thread
def some_long_process():
time.sleep(60)
async def main():
thread = Thread(target=some_long_process)
thread.start()
# Doesn't work
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, thread.join)
# await asyncio.wait([loop.run_in_executor(None, thread.join)])
# await asyncio.wait_for(loop.run_in_executor(None, thread.join), None)
# await asyncio.gather(asyncio.to_thread(thread.join))
# Works
# while thread.is_alive():
# await asyncio.sleep(1)
if __name__ == '__main__':
asyncio.run(main())
I'm also open to suggestions to reconsider my entire approach to the way this is designed if this isn't possible. Thanks for your time.
I'm coding a telegram userbot (with telethon) which sends a message,every 60 seconds, to some chats.
I'm using 2 threads but I get the following errors: "RuntimeWarning: coroutine 'sender' was never awaited" and "no running event loop".
My code:
....
async def sender():
for chat in chats :
try:
if chat.megagroup == True:
await client.send_message(chat, messaggio)
except:
await client.send_message(myID, 'error')
schedule.every(60).seconds.do(asyncio.create_task(sender()))
...
class checker1(Thread):
def run(self):
while True:
schedule.run_pending()
time.sleep(1)
class checker2(Thread):
def run(self):
while True:
client.add_event_handler(handler)
client.run_until_disconnected()
checker2().start()
checker1().start()
I searched for a solution but I didn't find anything...
You should avoid using threads with asyncio unless you know what you're doing. The code can be rewritten using asyncio as follows, since most of the time you don't actually need threads:
import asyncio
async def sender():
for chat in chats :
try:
if chat.megagroup == True:
await client.send_message(chat, messaggio)
except:
await client.send_message(myID, 'error')
async def checker1():
while True:
await sender()
await asyncio.sleep(60) # every 60s
async def main():
await asyncio.create_task(checker1()) # background task
await client.run_until_disconnected()
client.loop.run_until_complete(main())
This code is not perfect (you should properly cancel and wait checker1 at the end of the program), but it should work.
As a side note, you don't need client.run_until_disconnected(). The call simply blocks (runs) until the client is disconnected. If you can keep the program running differently, as long as asyncio runs, the client will work.
Another thing: bare except: are a very bad idea, and will probably cause issues with exception. At least, replace it with except Exception.
There are a few problems with your code. asyncio is complaining about "no running event loop" because your program never starts the event loop anywhere, and tasks can't be scheduled without an event loop running. See Asyncio in corroutine RuntimeError: no running event loop. In order to start the event loop, you can use asyncio.run_until_complete() if you have a main coroutine for your program, or you can use asyncio.get_event_loop().run_forever() to run the event loop forever.
The second problem is the incorrect usage of schedule.every(60).seconds.do(), which is hidden by the first error. schedule expects a function to be passed in, not an awaitable (which is what asyncio.create_task(sender()) returns). This normally would have caused a TypeError, but the create_task() without a running event loop raised an exception first, so this exception was never raised. You'll need to define a function and then pass it to schedule, like this:
def start_sender():
asyncio.create_task(sender())
schedule.every(60).seconds.do(start_sender)
This should work as long as the event loop is started somewhere else in your program.
Can somebody provide a sample of code which listen to keypress in nonblocking manner with asynio and put the keycode in console on every click?
It's not a question about some graphical toolkit
So the link provided by Andrea Corbellini is a clever and thorough solution to the problem, but also quite complicated. If all you want to do is prompt your user to enter some input (or simulate raw_input), I prefer to use the much simpler solution:
import sys
import functools
import asyncio as aio
class Prompt:
def __init__(self, loop=None):
self.loop = loop or aio.get_event_loop()
self.q = aio.Queue()
self.loop.add_reader(sys.stdin, self.got_input)
def got_input(self):
aio.ensure_future(self.q.put(sys.stdin.readline()), loop=self.loop)
async def __call__(self, msg, end='\n', flush=False):
print(msg, end=end, flush=flush)
return (await self.q.get()).rstrip('\n')
prompt = Prompt()
raw_input = functools.partial(prompt, end='', flush=True)
async def main():
# wait for user to press enter
await prompt("press enter to continue")
# simulate raw_input
print(await raw_input('enter something:'))
loop = aio.get_event_loop()
loop.run_until_complete(main())
loop.close()
EDIT: I removed the loop parameter form Queue as it is removed in 3.10.
Also, these days I use structured concurrency (trio), and if anyone is curious this is pretty easy to do in trio:
import trio, sys
async def main():
async with trio.lowlevel.FdStream(sys.stdin.fileno()) as stdin:
async for line in stdin:
if line.startswith(b'q'):
break
print(line)
trio.run(main)
I wrote something similar as part of a package called aioconsole.
It provides a coroutine called get_standard_streams that returns two asyncio streams corresponding to stdin and stdout.
Here's an example:
import asyncio
import aioconsole
async def echo():
stdin, stdout = await aioconsole.get_standard_streams()
async for line in stdin:
stdout.write(line)
loop = asyncio.get_event_loop()
loop.run_until_complete(echo())
It also includes an asynchronous equivalent to input:
something = await aioconsole.ainput('Entrer something: ')
It should work for both file and non-file streams. See the implementation here.
Reading lines
The high-level pure-asyncio way to do this is as follows.
import asyncio
import sys
async def main():
# Create a StreamReader with the default buffer limit of 64 KiB.
reader = asyncio.StreamReader()
pipe = sys.stdin
loop = asyncio.get_event_loop()
await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), pipe)
async for line in reader:
print(f'Got: {line.decode()!r}')
asyncio.run(main())
The async for line in reader loop can be written more explicitly, e.g. if you want to print a prompt or catch exceptions inside the loop:
while True:
print('Prompt: ', end='', flush=True)
try:
line = await reader.readline()
if not line:
break
except ValueError:
print('Line length went over StreamReader buffer limit.')
else:
print(f'Got: {line.decode()!r}')
An empty line (not '\n' but an actually empty string '') means end-of-file. Note that it is possible for await reader.readline() to return '' right after reader.at_eof() returned False. See Python asyncio: StreamReader for details.
Here readline() is asynchronously gathering a line of input. That is, the event loop can run while the reader waits for more characters. In contrast, in the other answers, the event loop could block: it could detect that some input is available, enter the function calling sys.stdin.readline(), and then block on it until an endline becomes available (blocking any other tasks from entering the loop). Of course this isn't a problem in most cases, as the endline becomes available together with (in case of line buffering, which is the default) or very soon after (in other cases, assuming reasonably short lines) any initial characters of a line.
Reading character by character
You can also read individual bytes with await reader.readexactly(1) to read byte-per-byte, when reading from a pipe. When reading key-presses from a terminal, it needs to be set up properly, see Key Listeners in python? for more. On UNIX:
import asyncio
import contextlib
import sys
import termios
#contextlib.contextmanager
def raw_mode(file):
old_attrs = termios.tcgetattr(file.fileno())
new_attrs = old_attrs[:]
new_attrs[3] = new_attrs[3] & ~(termios.ECHO | termios.ICANON)
try:
termios.tcsetattr(file.fileno(), termios.TCSADRAIN, new_attrs)
yield
finally:
termios.tcsetattr(file.fileno(), termios.TCSADRAIN, old_attrs)
async def main():
with raw_mode(sys.stdin):
reader = asyncio.StreamReader()
loop = asyncio.get_event_loop()
await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), sys.stdin)
while not reader.at_eof():
ch = await reader.read(1)
# '' means EOF, chr(4) means EOT (sent by CTRL+D on UNIX terminals)
if not ch or ord(ch) <= 4:
break
print(f'Got: {ch!r}')
asyncio.run(main())
Note this is not really one character or one key at a time: if the user presses a key combination that gives a multi-byte character, like ALT+E, nothing will happen on pressing ALT and two bytes will be sent by the terminal on pressing E, which will result in two iterations of the loop. But it's good enough for ASCII characters like letters and ESC.
If you need actual key presses like ALT, I suppose the only way is to use a suitable library and make it work with asyncio by calling it in a separate thread, like here. In fact the library+thread approach is probably simpler in other cases as well.
Under the hood
If you want finer control you can implement your own protocol in place of StreamReaderProtocol: a class implementing any number of functions of asyncio.Protocol. Minimal example:
class MyReadProtocol(asyncio.Protocol):
def __init__(self, reader: asyncio.StreamReader):
self.reader = reader
def connection_made(self, pipe_transport):
self.reader.set_transport(pipe_transport)
def data_received(self, data: bytes):
self.reader.feed_data(data)
def connection_lost(self, exc):
if exc is None:
self.reader.feed_eof()
else:
self.reader.set_exception(exc)
You could replace the StreamReader with your own buffering mechanism. After you call connect_read_pipe(lambda: MyReadProtocol(reader), pipe), there will be exactly one call to connection_made, then arbitrary many calls to data_received (with data depending on terminal and python buffering options), then eventually exactly one call to connection_lost (on end-of-file or on error). In case you ever need them, connect_read_pipe returns a tuple (transport, protocol), where protocol is an instance of MyReadProtocol (created by the protocol factory, which in our case is a trivial lambda), while transport is an instance of asyncio.ReadTransport (specifically some private implementation like _UnixReadPipeTransport on UNIX).
But in the end this is all boilerplate that eventually relies on loop.add_reader (unrelated to StreamReader).
For Windows you might need to choose the ProactorEventLoop (the default since Python 3.8), see Python asyncio: Platform Support.
An alternative to using queues would be to make the command line an asyn generator, and process the commands as they come in, like so:
import asyncio
import sys
class UserInterface(object):
def __init__(self, task, loop):
self.task = task
self.loop = loop
def get_ui(self):
return asyncio.ensure_future(self._ui_task())
async def _ui_cmd(self):
while True:
cmd = sys.stdin.readline()
cmd = cmd.strip()
if cmd == 'exit':
self.loop.stop()
return
yield cmd
async def _ui_task(self):
async for cmd in self._ui_cmd():
if cmd == 'stop_t':
self.task.stop()
elif cmd == 'start_t':
self.task.start()
Python 3.10 update to the solution provided by bj0:
class Prompt:
def __init__(self):
self.loop = asyncio.get_running_loop()
self.q = asyncio.Queue()
self.loop.add_reader(sys.stdin, self.got_input)
def got_input(self):
asyncio.ensure_future(self.q.put(sys.stdin.readline()), loop=self.loop)
async def __call__(self, msg, end='\n', flush=False):
print(msg, end=end, flush=flush)
# https://docs.python.org/3/library/asyncio-task.html#coroutine
task = asyncio.create_task(self.q.get())
return (await task).rstrip('\n')
I tested it on a websocket client, inside an async function that would get stuck waiting for input, so I replaced s = input("insert string") with s = await prompt("insert string") and now ping-ponging works even while the program is waiting for user input, the connection does not stop anymore and the issue "timed out waiting for keepalive pong" is solved.
#asyncio.coroutine
def listener():
while True:
message = yield from websocket.recieve_message()
if message:
yield from handle(message)
loop = asyncio.get_event_loop()
loop.run_until_complete(listener())
Let's say i'm using websockets with asyncio. That means I recieve messages from websockets. And when I recieve a message, I want to handle the message but I'm loosing all the async thing with my code. Because the yield from handle(message) is definetly blocking... How could I find a way to make it non-blocking ? Like, handle multiple messages in the same time. Not having to wait the message to be handled before I can handle another message.
Thanks.
If you don't care about the return value from handle message, you can simply create a new Task for it, which will run in the event loop alongside your websocket reader. Here is a simple example:
#asyncio.coroutine
def listener():
while True:
message = yield from websocket.recieve_message()
if message:
asyncio.ensure_future(handle(message))
ensure_future will create a task and attach it to the default event loop. Since the loop is already running, it will get processed alongside your websocket reader in parallel. In fact, if it is a slow-running I/O blocked task (like sending an email), you could easily have a few dozen handle(message) tasks running at once. They are created dynamically when needed, and destroyed when finished (with much lower overhead than spawning threads).
If you want a bit more control, you could simply write to an asyncio.Queue in the reader and have a task pool of a fixed size that can consume the queue, a typical pattern in multi-threaded or multi-process programming.
#asyncio.coroutine
def consumer(queue):
while True:
message = yield from queue.get()
yield from handle(message)
#asyncio.coroutine
def listener(queue):
for i in range(5):
asyncio.ensure_future(consumer(queue))
while True:
message = yield from websocket.recieve_message()
if message:
yield from q.put(message)
q = asyncio.Queue()
loop = asyncio.get_event_loop()
loop.run_until_complete(listener(q))
I need at times more than one asyncio coroutine, where the routines then would be nested : coroutine B running in coroutine A, C in B and so on. The problem is with stopping a given loop. For example, using loop.stop() in the last top loop such as loop 'C' kills actually all asyncio coroutines - and not just this loop 'C'. I suspect that stop() actually kills coroutine A, and by doing so it annihilates all other dependent routines. The nested routines are call_soon_threadsafe, and all routines begin as run_forever.
I tried using specific loop names, or 'return', or 'break' (in the while loop inside the coroutine) but nothing exits the loop - except stop() which then kills non-specifically all loops at once.
My problem I described here is actually related to an earlier question of mine...
python daemon server crashes during HTML popup overlay callback using asyncio websocket coroutines
...which I thought I had solved - till running into this loop.stop() problem.
below now my example code for Python 3.4.3 where I try to stop() the coroutine_overlay_websocket_server loop as soon as it is done with the websocket job. As said, my code at the current state breaks all running loops. Thereafter the fmDaemon recreates a new asyncio loop that knows nothing of what was computed before :
import webbrowser
import websockets
import asyncio
class fmDaemon( Daemon):
# Daemon - see : http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
# Daemon - see : http://www.jejik.com/files/examples/daemon3x.py
def __init__( self, me):
self.me = me
def run( self):
while True:
#asyncio.coroutine
def coroutine_daemon_websocket_server( websocket, path):
msg = yield from websocket.recv()
if msg != None:
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
self.me = Function1( self.me, msg)
loop = asyncio.get_event_loop()
loop.run_until_complete( websockets.serve( coroutine_daemon_websocket_server, self.me.IP, self.me.PORT))
loop.run_forever()
def Function1( me, msg):
# doing some stuff :
# creating HTML file,
# loading HTML webpage with a webbrowser call,
# awaiting HTML button press signal via websocket protocol :
#asyncio.coroutine
def coroutine_overlay_websocket_server( websocket, path):
while True:
msg = yield from websocket.recv()
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
if msg == 'my_expected_string':
me.flags['myStr'] = msg
break
loop.call_soon_threadsafe( loop.stop)
loop = asyncio.get_event_loop()
loop.call_soon_threadsafe( asyncio.async, websockets.serve( coroutine_overlay_websocket_server, me.IP, me.PORT_overlay))
loop.run_forever()
loop.call_soon_threadsafe( loop.close)
# program should continue here...
My two questions : 1) Is there a way to exit a given coroutine without killing a coroutine lower down? 2) Or, alternatively, do you know of a method for reading websocket calls that does not make use of asyncio ?
I'm still a little confused by what you're trying to do, but there's definitely no need to try to nest event loops - your program is single-threaded, so when you call asyncio.get_event_loop() multiple times, you're always going to get the same event loop back. So you're really not creating two different loops in your example; both fmDaemon.run and Function1 use the same one. That's why stopping loop inside Function1 also kills the coroutine you launched inside run.
That said, there's no reason to try to create two different event loops to begin with. Function1 is being called from a coroutine, and wants to call other coroutines, so why not make it a coroutine, too? Then you can just call yield from websockets.serve(...) directly, and use an asyncio.Event to wait for coroutine_overlay_websocket_server to complete:
import webbrowser
import websockets
import asyncio
class fmDaemon( Daemon):
def __init__( self, me):
self.me = me
def run( self):
#asyncio.coroutine
def coroutine_daemon_websocket_server(websocket, path):
msg = yield from websocket.recv()
if msg != None:
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
self.me = Function1( self.me, msg)
loop = asyncio.get_event_loop()
loop.run_until_complete(websockets.serve(coroutine_daemon_websocket_server,
self.me.IP,
self.me.PORT))
loop.run_forever()
#asyncio.coroutine
def Function1(me, msg):
#asyncio.coroutine
def coroutine_overlay_websocket_server(websocket, path):
while True:
msg = yield from websocket.recv()
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
if msg == 'my_expected_string':
me.flags['myStr'] = msg
break
event.set() # Tell the outer function it can exit.
event = asyncio.Event()
yield from websockets.serve(coroutine_overlay_websocket_server, me.IP, me.PORT_overlay))
yield from event.wait() # This will block until event.set() is called.