Can somebody provide a sample of code which listen to keypress in nonblocking manner with asynio and put the keycode in console on every click?
It's not a question about some graphical toolkit
So the link provided by Andrea Corbellini is a clever and thorough solution to the problem, but also quite complicated. If all you want to do is prompt your user to enter some input (or simulate raw_input), I prefer to use the much simpler solution:
import sys
import functools
import asyncio as aio
class Prompt:
def __init__(self, loop=None):
self.loop = loop or aio.get_event_loop()
self.q = aio.Queue()
self.loop.add_reader(sys.stdin, self.got_input)
def got_input(self):
aio.ensure_future(self.q.put(sys.stdin.readline()), loop=self.loop)
async def __call__(self, msg, end='\n', flush=False):
print(msg, end=end, flush=flush)
return (await self.q.get()).rstrip('\n')
prompt = Prompt()
raw_input = functools.partial(prompt, end='', flush=True)
async def main():
# wait for user to press enter
await prompt("press enter to continue")
# simulate raw_input
print(await raw_input('enter something:'))
loop = aio.get_event_loop()
loop.run_until_complete(main())
loop.close()
EDIT: I removed the loop parameter form Queue as it is removed in 3.10.
Also, these days I use structured concurrency (trio), and if anyone is curious this is pretty easy to do in trio:
import trio, sys
async def main():
async with trio.lowlevel.FdStream(sys.stdin.fileno()) as stdin:
async for line in stdin:
if line.startswith(b'q'):
break
print(line)
trio.run(main)
I wrote something similar as part of a package called aioconsole.
It provides a coroutine called get_standard_streams that returns two asyncio streams corresponding to stdin and stdout.
Here's an example:
import asyncio
import aioconsole
async def echo():
stdin, stdout = await aioconsole.get_standard_streams()
async for line in stdin:
stdout.write(line)
loop = asyncio.get_event_loop()
loop.run_until_complete(echo())
It also includes an asynchronous equivalent to input:
something = await aioconsole.ainput('Entrer something: ')
It should work for both file and non-file streams. See the implementation here.
Reading lines
The high-level pure-asyncio way to do this is as follows.
import asyncio
import sys
async def main():
# Create a StreamReader with the default buffer limit of 64 KiB.
reader = asyncio.StreamReader()
pipe = sys.stdin
loop = asyncio.get_event_loop()
await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), pipe)
async for line in reader:
print(f'Got: {line.decode()!r}')
asyncio.run(main())
The async for line in reader loop can be written more explicitly, e.g. if you want to print a prompt or catch exceptions inside the loop:
while True:
print('Prompt: ', end='', flush=True)
try:
line = await reader.readline()
if not line:
break
except ValueError:
print('Line length went over StreamReader buffer limit.')
else:
print(f'Got: {line.decode()!r}')
An empty line (not '\n' but an actually empty string '') means end-of-file. Note that it is possible for await reader.readline() to return '' right after reader.at_eof() returned False. See Python asyncio: StreamReader for details.
Here readline() is asynchronously gathering a line of input. That is, the event loop can run while the reader waits for more characters. In contrast, in the other answers, the event loop could block: it could detect that some input is available, enter the function calling sys.stdin.readline(), and then block on it until an endline becomes available (blocking any other tasks from entering the loop). Of course this isn't a problem in most cases, as the endline becomes available together with (in case of line buffering, which is the default) or very soon after (in other cases, assuming reasonably short lines) any initial characters of a line.
Reading character by character
You can also read individual bytes with await reader.readexactly(1) to read byte-per-byte, when reading from a pipe. When reading key-presses from a terminal, it needs to be set up properly, see Key Listeners in python? for more. On UNIX:
import asyncio
import contextlib
import sys
import termios
#contextlib.contextmanager
def raw_mode(file):
old_attrs = termios.tcgetattr(file.fileno())
new_attrs = old_attrs[:]
new_attrs[3] = new_attrs[3] & ~(termios.ECHO | termios.ICANON)
try:
termios.tcsetattr(file.fileno(), termios.TCSADRAIN, new_attrs)
yield
finally:
termios.tcsetattr(file.fileno(), termios.TCSADRAIN, old_attrs)
async def main():
with raw_mode(sys.stdin):
reader = asyncio.StreamReader()
loop = asyncio.get_event_loop()
await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), sys.stdin)
while not reader.at_eof():
ch = await reader.read(1)
# '' means EOF, chr(4) means EOT (sent by CTRL+D on UNIX terminals)
if not ch or ord(ch) <= 4:
break
print(f'Got: {ch!r}')
asyncio.run(main())
Note this is not really one character or one key at a time: if the user presses a key combination that gives a multi-byte character, like ALT+E, nothing will happen on pressing ALT and two bytes will be sent by the terminal on pressing E, which will result in two iterations of the loop. But it's good enough for ASCII characters like letters and ESC.
If you need actual key presses like ALT, I suppose the only way is to use a suitable library and make it work with asyncio by calling it in a separate thread, like here. In fact the library+thread approach is probably simpler in other cases as well.
Under the hood
If you want finer control you can implement your own protocol in place of StreamReaderProtocol: a class implementing any number of functions of asyncio.Protocol. Minimal example:
class MyReadProtocol(asyncio.Protocol):
def __init__(self, reader: asyncio.StreamReader):
self.reader = reader
def connection_made(self, pipe_transport):
self.reader.set_transport(pipe_transport)
def data_received(self, data: bytes):
self.reader.feed_data(data)
def connection_lost(self, exc):
if exc is None:
self.reader.feed_eof()
else:
self.reader.set_exception(exc)
You could replace the StreamReader with your own buffering mechanism. After you call connect_read_pipe(lambda: MyReadProtocol(reader), pipe), there will be exactly one call to connection_made, then arbitrary many calls to data_received (with data depending on terminal and python buffering options), then eventually exactly one call to connection_lost (on end-of-file or on error). In case you ever need them, connect_read_pipe returns a tuple (transport, protocol), where protocol is an instance of MyReadProtocol (created by the protocol factory, which in our case is a trivial lambda), while transport is an instance of asyncio.ReadTransport (specifically some private implementation like _UnixReadPipeTransport on UNIX).
But in the end this is all boilerplate that eventually relies on loop.add_reader (unrelated to StreamReader).
For Windows you might need to choose the ProactorEventLoop (the default since Python 3.8), see Python asyncio: Platform Support.
An alternative to using queues would be to make the command line an asyn generator, and process the commands as they come in, like so:
import asyncio
import sys
class UserInterface(object):
def __init__(self, task, loop):
self.task = task
self.loop = loop
def get_ui(self):
return asyncio.ensure_future(self._ui_task())
async def _ui_cmd(self):
while True:
cmd = sys.stdin.readline()
cmd = cmd.strip()
if cmd == 'exit':
self.loop.stop()
return
yield cmd
async def _ui_task(self):
async for cmd in self._ui_cmd():
if cmd == 'stop_t':
self.task.stop()
elif cmd == 'start_t':
self.task.start()
Python 3.10 update to the solution provided by bj0:
class Prompt:
def __init__(self):
self.loop = asyncio.get_running_loop()
self.q = asyncio.Queue()
self.loop.add_reader(sys.stdin, self.got_input)
def got_input(self):
asyncio.ensure_future(self.q.put(sys.stdin.readline()), loop=self.loop)
async def __call__(self, msg, end='\n', flush=False):
print(msg, end=end, flush=flush)
# https://docs.python.org/3/library/asyncio-task.html#coroutine
task = asyncio.create_task(self.q.get())
return (await task).rstrip('\n')
I tested it on a websocket client, inside an async function that would get stuck waiting for input, so I replaced s = input("insert string") with s = await prompt("insert string") and now ping-ponging works even while the program is waiting for user input, the connection does not stop anymore and the issue "timed out waiting for keepalive pong" is solved.
Related
I'm reading from a named pipe in a blocking manner.
I want my python script to react to SIGTERM signals.
This is what i've got so far:
#!/usr/bin/python3
import signal
def handler_stop_signals(signum, frame):
global fifo
fifo.close()
exit
signal.signal(signal.SIGTERM, handler_stop_signals)
fifo = open("someNamedPipe", "r")
while True:
for line in fifo:
doSomething
fifo.close()
exit
When the script receives a SIGTERM signal, it closes the pipe as expected but raises a RuntimeError.
RuntimeError: reentrant call inside <_io.BufferedReader name='someNamedPipe'>
Is there another way to get out of the foor loop and close the fifo gently?
TL;DR You need a non-blocking read in order to be able to control termination; asyncio using aiofiles is probably the most elegant solution, but they all have their quirks.
Sample Producer
I'm going to start with how one would go about writing a well-behaved producer of data to a named pipe, because it's an easier vehicle to introduce some APIs.
import os
from threading import Event
class Producer:
def __init__(self, path):
self.path = path
self.event = Event()
def start(self):
os.mkfifo(self.path)
try:
print('Waiting for a listener...')
with open(self.path, 'w') as fifo:
fifo.write('Starting the convoluted clock...\n')
fifo.flush()
while not self.event.wait(timeout=1):
print('Writing a line...')
fifo.write(str(datetime.now()) + '\n')
fifo.flush()
fifo.write('The convoluted clock has finished.\n')
fifo.flush()
print('Finished.')
finally:
os.unlink(self.path)
def stop(self, *args, **kwargs):
self.event.set()
producer = Producer('/tmp/someNamedPipe')
signal.signal(signal.SIGINT, producer.stop)
signal.signal(signal.SIGTERM, producer.stop)
producer.start()
This writes the current date out to the named pipe as a string once a second. SIGINT and SIGTERM will both shut the pipe down gracefully, writing The convoluted clock has finished. as the last line to the pipe before closing down. It uses a threading.Event to communicate between the stop method (which will be run on a background thread) and start (which waits for at most one second before advancing to the next iteration of the loop). self.event.wait(timeout=1) immediately returns True if the signal is set, or False after waiting at most one second without the signal being set.
Sample (Buggy) Consumer
It would be tempting to use a similar technique to write the consumer:
import signal, os
from threading import Event
class BuggyConsumer:
def __init__(self, path):
self.path = path
self.event = Event()
def start(self):
with open(self.path, 'r') as fifo:
# we'll be a bit more aggressive on checking for termination
# because we could have new data for us at any moment!
while not self.event.wait(0.1):
print('Got from the producer:', fifo.readline())
print('The consumer was nicely stopped.')
# technically the pipe gets closed AFTER this print statement
# because we're using a with block
finally:
fifo.close()
def stop(self, *args, **kwargs):
self.event.set()
consumer = BuggyConsumer('/tmp/someNamedPipe')
signal.signal(signal.SIGINT, consumer.stop)
signal.signal(signal.SIGTERM, consumer.stop)
consumer.start()
Unfortunately this won't work great in practice because open() opens files in blocking mode. This means read() calls block the calling thread, which essentially prevent "nice" aborts unless you check in-between read calls. Concretely, if the producer stops producing but kept the pipe open, the consumer would sit forever at fifo.readline() and would never get around to checking the signal for "nice" termination.
Sample (Less Buggy) Consumer
This example avoids the problem of a misbehaving producer trapping the consumer in a blocking read call, but it's considerably more complicated and forces you to use lower-level APIs that are not nearly as friendly:
import signal, os
from threading import Event
class ComplicatedConsumer:
def __init__(self, path):
self.path = path
self.event = Event()
def start(self):
# Open a file descriptor in a non-blocking way.
fifo = os.open(self.path, os.O_RDONLY | os.O_NONBLOCK)
try:
while not self.event.wait(0.1):
try:
# This is FAR from a comprehensive implementation.
# We're making some pretty yucky assumptions.
line = os.read(fifo, 1000).decode('utf8')
if line:
print('Got from the producer:', line)
else:
print('EOF from the producer.')
break
except BlockingIOError:
# the call to os.read would have blocked (meaning we're
# caught up)
pass
print('The consumer was nicely stopped.')
finally:
os.close(fifo)
def stop(self, *args, **kwargs):
self.event.set()
A proper implementation would be FAR more complicated, because this code naively assumes that:
each read() call from the pipe is a single, complete "message"; this is the worst assumption. You could speed up the producer and see that this less buggy consumer starts reading multiple "lines" as a single line.
a line never spans more than 1000 bytes; a more advanced implementation would need to buffer "partial" messages, look for newlines, and split accordingly
In all but the most simplistic and slow-moving use cases (like, say, a once-a-second ticking clock), this implementation would need a TON of work in order to be practically useful.
Sample Consumer (asyncio)
The challenge in writing this properly is that there are multiple unpredictable sources of events (signals, incoming data from a pipe). asyncio allows you to express your code as coroutines, and they can be suspended and resumed when Python feels like it, but with you specifying the rules.
import asyncio
import aiofiles
class AsyncConsumer:
def __init__(self, path):
loop = asyncio.get_event_loop()
self.path = path
self.fifo_closed = loop.create_future()
self.fifo = None
async def start(self):
import aiofiles
self.fifo = await aiofiles.open(self.path, 'r')
done, pending = await asyncio.wait(
[self._read_lines(), self.fifo_closed],
return_when=asyncio.FIRST_COMPLETED)
print('The consumer is going to be nicely stopped...')
await self.fifo.close()
print('The consumer was nicely stopped.')
async def _read_lines(self):
try:
async for line in self.fifo:
print('Got from the producer:', line)
print('EOF from the producer.')
except ValueError:
# aiofiles raises a `ValueError` when the underlying file is closed
# from underneath it
pass
def stop(self, *args, **kwargs):
if self.fifo is not None:
print('we got the message')
self.fifo_closed.set_result(None)
loop = asyncio.get_event_loop()
consumer = AsyncConsumer('/tmp/someNamedPipe')
loop.add_signal_handler(signal.SIGINT, consumer.stop)
loop.add_signal_handler(signal.SIGTERM, consumer.stop)
loop.run_until_complete(consumer.start())
The async start() method kicks off two threads streams things of work: one which reads lines one-by-one as they come in, and the other which essentially hangs until a signal is received. It proceeds when EITHER of these two things finishes.
Unfortunately I noticed that ultimately aiofiles relies on a blocking implementation under the hood, because the await self.fifo.close() method still hangs if a read() is in progress. But at least there is a spot to place your code.
Wrapping it up
Ultimately there isn't a super great out-of-the-box solution to solving your problem, but hopefully one of these variations can help you solve your problem.
I'm new in python and I have to create a program that keeps in linstening from web socket and pipe so I need two asynchronous functions.
Each of these functions call other method in different thrad that elaborate the content of received json.
i.e. I receive a message on socket thread, I get the message and throw a new thread to elaborate the message.
This is the actual code:
import asyncio
import sys
import json
import websockets
# Keep listening from web socket and pipe
async def socket_receiver():
"""Listening from web socket"""
file_socket = open(r"SocketReceived.txt", "w")
header = {"Authorization": r"Basic XXXXXXXXXXXXXX="}
async with websockets.connect(
'wss://XXXXXXXXX', extra_headers=header) as web_socket:
print("SOCKET receiving:")
greeting = await web_socket.recv()
json_message = json.loads(greeting)
file_socket.write(json_message)
print(json_message)
file_socket.close()
async def pipe_receiver():
"""Listening from pipe"""
file_pipe = open(r"ipeReceived.txt", "w")
while True:
print("PIPE receiving:")
line = sys.stdin.readline()
if not line:
break
jsonObj = json.loads(line);
file_pipe.write(jsonObj['prova'] + '\n')
# jsonValue = json.dump(str(line), file);
sys.stdout.flush()
file_pipe.close()
asyncio.get_event_loop().run_until_complete(socket_receiver())
asyncio.get_event_loop().run_until_complete(pipe_receiver())
run_until_complete method keep forever in my case (it waits the end of function), so only the socket starts.
How can I start both? Thanks
asyncio.gather does the trick, the only point is that both functions should share the same event loop, and both should be fully asynchronous.
asyncio.get_event_loop().run_until_complete(
asyncio.gather( socket_receiver(),pipe_receiver()))
From a quick reading of pipe_receiver, you will hang your event loop in sys.stdin.readline call, please consider using aioconsole to asynchronously handle the input.
Since examples of asynchronous programming are still fairly sparse, I am trying to come up with my own using the new async def. I figured the most basic example I could make would be an event loop which listens to the user inputs and prints them back.
I want an event loop which listens to the keys the user presses and prints them every second. I'm trying to do so by having one task that prints from a queue of keys and a second task that listens to pressed keys and adds them to the queue.
I am missing a good way to asynchronously listen to key strokes. Here is what I have for now.
import asyncio
KEY_QUEUE = []
async def printer():
while True:
await asyncio.sleep(1)
print('In the last second you pressed:', *KEY_QUEUE)
KEY_QUEUE.clear()
async def listener():
while True:
... # await a key to be pressed and add it to KEY_QUEUE
loop = asyncio.get_event_loop()
loop.create_task(printer())
loop.create_task(listener())
loop.run_forever()
Expected output would look like this
In the last second you pressed: h e l l o
In the last second you pressed: w o r
In the last second you pressed: l d
In the last second you pressed:
In the last second you pressed: i t w o
In the last second you pressed: r k s
I doubt there is an awaitable coroutine such as asyncio.await_pressed_key, how would we proceed to create one?
Any other approach is welcome as well as my goal is not to make this specific example work, but rather to generate meaningful asynchronous programmign examples.
Assuming that you want to read characters from a TTY (terminal device on Unix-like system), accessing characters as they are typed requires:
the TTY to be in "raw mode", so that the system returns characters without waiting for a newline to be typed by the user;
the standard input, which are blocking, to be wrapped into an asyncio non-blocking stream.
The former is provided by the tty module that comes with the Python standard library, and the latter by the aioconsole third-party library.
With those two, your example could look like this:
import asyncio, aioconsole, sys, tty
async def main():
loop = asyncio.get_event_loop()
typed = []
p = loop.create_task(printer(typed))
await listener(typed)
p.cancel()
async def listener(typed):
tty.setraw(sys.stdin.fileno())
stdin, _ = await aioconsole.stream.get_standard_streams()
while True:
ch = await stdin.read(1)
if ch == b'\x03': # ctrl-c
break
typed.append(ch)
tty.setcbreak(sys.stdin.fileno())
async def printer(typed):
while True:
await asyncio.sleep(1)
print('In the last second you pressed', typed, end='\r\n')
del typed[:]
asyncio.get_event_loop().run_until_complete(main())
This example will only work on Unix-like systems.
I'm trying to get Python's curses and python-slackclient to work together, but both of them utilize a while loop. I'm new to asynchronous programming and so am having trouble figuring out the best solution to the problem.
I need both of these while loops to run simultaneously, and be able to share information with each other. When slackclient receives data, it needs to be displayed on the screen. When a user enters text, it needs to be sent out over slackclient.
I've been able to write a test run of slackclient that uses asyncio to listen on stdin, and write out what it hears from stdin. However I'm stuck here and don't know how to proceed. What might my next step be?
Here's the slackclient/asyncio test code:
import asyncio
import time
import sys
from slackclient import SlackClient
class SlackCLI(object):
def __init__(self):
self.token = 'xoxp-TOKEN-HIDDEN'
self.client = SlackClient(self.token)
self.channel = '#general'
def callback(self):
text = sys.stdin.readline()
print('[*] sending: {} to channel {}'.format(text, self.channel))
print(self.client.api_call('chat.postMessage', channel='#general',
text='{}'.format(text),
username='pybot123', icon_emoji=':robot_face:'))
#asyncio.coroutine
def run(self):
if self.client.rtm_connect():
while True:
last = self.client.rtm_read()
if last:
try:
text = last[0]['text']
chan = last[0]['channel']
print('[!] {}: {}'.format(chan, text))
except:
pass
yield from asyncio.sleep(1)
else:
print('connection failed')
def main(self):
loop = asyncio.get_event_loop()
loop.add_reader(sys.stdin, self.callback)
loop.run_until_complete(self.run())
a = SlackCLI()
a.main()
I need at times more than one asyncio coroutine, where the routines then would be nested : coroutine B running in coroutine A, C in B and so on. The problem is with stopping a given loop. For example, using loop.stop() in the last top loop such as loop 'C' kills actually all asyncio coroutines - and not just this loop 'C'. I suspect that stop() actually kills coroutine A, and by doing so it annihilates all other dependent routines. The nested routines are call_soon_threadsafe, and all routines begin as run_forever.
I tried using specific loop names, or 'return', or 'break' (in the while loop inside the coroutine) but nothing exits the loop - except stop() which then kills non-specifically all loops at once.
My problem I described here is actually related to an earlier question of mine...
python daemon server crashes during HTML popup overlay callback using asyncio websocket coroutines
...which I thought I had solved - till running into this loop.stop() problem.
below now my example code for Python 3.4.3 where I try to stop() the coroutine_overlay_websocket_server loop as soon as it is done with the websocket job. As said, my code at the current state breaks all running loops. Thereafter the fmDaemon recreates a new asyncio loop that knows nothing of what was computed before :
import webbrowser
import websockets
import asyncio
class fmDaemon( Daemon):
# Daemon - see : http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
# Daemon - see : http://www.jejik.com/files/examples/daemon3x.py
def __init__( self, me):
self.me = me
def run( self):
while True:
#asyncio.coroutine
def coroutine_daemon_websocket_server( websocket, path):
msg = yield from websocket.recv()
if msg != None:
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
self.me = Function1( self.me, msg)
loop = asyncio.get_event_loop()
loop.run_until_complete( websockets.serve( coroutine_daemon_websocket_server, self.me.IP, self.me.PORT))
loop.run_forever()
def Function1( me, msg):
# doing some stuff :
# creating HTML file,
# loading HTML webpage with a webbrowser call,
# awaiting HTML button press signal via websocket protocol :
#asyncio.coroutine
def coroutine_overlay_websocket_server( websocket, path):
while True:
msg = yield from websocket.recv()
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
if msg == 'my_expected_string':
me.flags['myStr'] = msg
break
loop.call_soon_threadsafe( loop.stop)
loop = asyncio.get_event_loop()
loop.call_soon_threadsafe( asyncio.async, websockets.serve( coroutine_overlay_websocket_server, me.IP, me.PORT_overlay))
loop.run_forever()
loop.call_soon_threadsafe( loop.close)
# program should continue here...
My two questions : 1) Is there a way to exit a given coroutine without killing a coroutine lower down? 2) Or, alternatively, do you know of a method for reading websocket calls that does not make use of asyncio ?
I'm still a little confused by what you're trying to do, but there's definitely no need to try to nest event loops - your program is single-threaded, so when you call asyncio.get_event_loop() multiple times, you're always going to get the same event loop back. So you're really not creating two different loops in your example; both fmDaemon.run and Function1 use the same one. That's why stopping loop inside Function1 also kills the coroutine you launched inside run.
That said, there's no reason to try to create two different event loops to begin with. Function1 is being called from a coroutine, and wants to call other coroutines, so why not make it a coroutine, too? Then you can just call yield from websockets.serve(...) directly, and use an asyncio.Event to wait for coroutine_overlay_websocket_server to complete:
import webbrowser
import websockets
import asyncio
class fmDaemon( Daemon):
def __init__( self, me):
self.me = me
def run( self):
#asyncio.coroutine
def coroutine_daemon_websocket_server(websocket, path):
msg = yield from websocket.recv()
if msg != None:
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
self.me = Function1( self.me, msg)
loop = asyncio.get_event_loop()
loop.run_until_complete(websockets.serve(coroutine_daemon_websocket_server,
self.me.IP,
self.me.PORT))
loop.run_forever()
#asyncio.coroutine
def Function1(me, msg):
#asyncio.coroutine
def coroutine_overlay_websocket_server(websocket, path):
while True:
msg = yield from websocket.recv()
msg_out = "{}".format( msg)
yield from websocket.send( msg_out)
if msg == 'my_expected_string':
me.flags['myStr'] = msg
break
event.set() # Tell the outer function it can exit.
event = asyncio.Event()
yield from websockets.serve(coroutine_overlay_websocket_server, me.IP, me.PORT_overlay))
yield from event.wait() # This will block until event.set() is called.