Develop Reference

Twisted Python IRC bot - how to execute a function asynchronously so that it doesn't block the bot?

I'm trying to write an IRC bot that continues to work normally while it executes a long (10+ seconds) function.
I started by writing the bot using socket. When I called a 'blocking' function (computation that takes few seconds to execute), the bot naturally stopped responding and did not record any messages sent in chat while the function was computing.
I did some googling and saw a lot of people recommend using Twisted.
I implemented basic IRC bot, heavily based on some examples:
# twisted imports
from twisted.words.protocols import irc
from twisted.internet import reactor, protocol
from twisted.python import log
# system imports
import time, sys, datetime
def a_long_function():
time.sleep(180)
print("finished")
class BotMain(irc.IRCClient):
nickname = "testIRC_bot"
def connectionMade(self):
irc.IRCClient.connectionMade(self)
def connectionLost(self, reason):
irc.IRCClient.connectionLost(self, reason)
# callbacks for events
def signedOn(self):
"""Signed to server"""
self.join(self.factory.channel)
def joined(self, channel):
"""Joined channel"""
def privmsg(self, user, channel, msg):
"""Received message"""
user = user.split('!', 1)[0]
if 'test' in msg.lower():
print("timeout started")
a_long_function()
msg = "test finished"
self.msg(channel, msg)
if 'ping' in msg.lower():
self.msg(channel, "pong")
print("pong")
class BotMainFactory(protocol.ClientFactory):
"""A factory for BotMains """
protocol = BotMain
def __init__(self, channel, filename):
self.channel = channel
self.filename = filename
def clientConnectionLost(self, connector, reason):
"""Try to reconnect on connection lost"""
connector.connect()
def clientConnectionFailed(self, connector, reason):
print ("connection failed:", reason)
reactor.stop()
if __name__ == '__main__':
log.startLogging(sys.stdout)
f = BotMainFactory("#test", "log.txt")
reactor.connectTCP("irc.freenode.net", 6667, f)
reactor.run()
This approach is definitely better than my earlier socket implementation, because now the bot still receives the messages sent while it executes a_long_function().
However, it only 'sees' these messages after the function is complete. This means that when I was logging the messages to txt file, all messages received when a_long_function() was executing receive the same timestamp of when the function has finished - and not when they were actually sent in the chatroom.
Also, the bot still isn't able to send any messages while its executing the long function.
Could someone point me in the right direction of how I should go about changing the code so that this long function can be executed asynchronously, so that the bot can still log and reply to messages as it's executing?
Thanks in advance.
Edit:
I came across this answer, which gave me an idea that I could add deferLater calls into my a_long_function to split it into smaller chunks (that say take 1s to execute), and have the bot resume normal operation in between to reply to and log any messages that were sent to the IRC channel in mean time. Or perhaps add a timer that counts how long a_long_function has been running for, and if its longer than a threshold, it would call a deferLater to let the bot catch up on the buffered messages.
This does seem like a bit of hack thought - is there a more elegant solution?

No, there is not really a more elegant solution. Unless you want to use threading, which might look more elegant but could easily lead to an unstable program. If you can avoid it, go with the deferral solution.

To asynchronously call a function, you should use the asyncio package along with async/await, or coroutines. Keep in mind that calling async/await is a v3 implementation, not v2.
Using async/await:
#!/usr/bin/env python3
# countasync.py
import asyncio
async def count():
print("One")
await asyncio.sleep(1)
print("Two")
async def main():
await asyncio.gather(count(), count(), count())
if __name__ == "__main__":
import time
s = time.perf_counter()
asyncio.run(main())
elapsed = time.perf_counter() - s
print(f"{__file__} executed in {elapsed:0.2f} seconds.")
There is a really good tutorial you can read here that goes over using asyncio, in depth.
Hope of help!

Wrapping a Queue in Future

I am writing a Tornado webserver in Python 3.7 to display the status of processes run by the multiprocessing library.
The following code works, but I'd like to be able to do it using Tornado's built-in library instead of hacking in the threading library. I haven't figured out how to do it without blocking Tornado during queue.get. I think the correct solution is to wrap the get calls in some sort of future. I've tried for hours, but haven't figured out how to do this.
Inside of my multiprocessing script:
class ProcessToMonitor(multiprocessing.Process)
def __init__(self):
multiprocessing.Process.__init__(self)
self.queue = multiprocessing.Queue()
def run():
while True:
# do stuff
self.queue.put(value)
Then, in my Tornado script
class MyWebSocket(tornado.websocket.WebSocketHandler):
connections = set()
def open(self):
self.connections.add(self)
def close(self):
self.connections.remove(self)
#classmethod
def emit(self, message):
[client.write_message(message) for client in self.connections]
def worker():
ptm = ProcessToMonitor()
ptm.start()
while True:
message = ptm.queue.get()
MyWebSocket.emit(message)
if __name__ == '__main__':
app = tornado.web.Application([
(r'/', MainHandler), # Not shown
(r'/websocket', MyWebSocket)
])
app.listen(8888)
threading.Thread(target=worker)
ioloop = tornado.ioloop.IOLoop.current()
ioloop.start()

queue.get isn't a blocking function, it just waits until there's an item in the queue in case the queue is empty. I can see from your code that queue.get fits perfectly for you use case inside a while loop.
I think you're probably using it incorrectly. You'll have to make the worker function a coroutine (async/await syntax):
async def worker():
...
while True:
message = await queue.get()
...
However, if you don't want to wait for an item and would like to proceed immediately, its alternative is queue.get_nowait.
One thing to note here is thatqueue.get_nowait will raise an exception called QueueEmpty if the queue is empty. So, you'll need to handle that exception.
Example:
while True:
try:
message = queue.get_nowait()
except QueueEmpty:
# wait for some time before
# next iteration
# otherwise this loop will
# keep running for no reason
MyWebSocket.emit(message)
As you can see, you'll have to use pause the while loop for some time if the queue is empty to prevent it from overwhelming the system.
So why not use queue.get in the first place?

Python: Websockets in Synchronous Program

I have a bog-standard synchronous python program that needs to be able to read data from websockets and update the GUI with the data. However, asyncio creep is constantly tripping me up.
How do I make a module that:
accepts multiple subscriptions to multiple sources
sends an update to the requester whenever there's data
opens exactly one websocket connection per URL
resets the websocket if it closes
Here's what I have already, but it's failing at many points:
run_forever() means that the loop gets stuck before the subscription completes and then handle() is stuck in the falsey while loop
it does not seem to want to restart sockets when they're down because a websockets object does not have a connected property (websocket without an s does, but I'm not clear on the differences and can't find info online either)
I'm absolutely not sure if my approach is remotely correct.
Been fighting with this for weeks. Would appreciate some pointers.
class WSClient():
subscriptions = set()
connections = {}
started = False
def __init__(self):
self.loop = asyncio.get_event_loop()
def start(self):
self.started = True
self.loop.run_until_complete(self.handle())
self.loop.run_until_forever() # problematic, because it does not allow new subscribe() events
async def handle(self):
while len(self.connections) > 0:
# listen to every websocket
futures = [self.listen(self.connections[url]) for url in self.connections]
done, pending = await asyncio.wait(futures)
# the following is apparently necessary to avoid warnings
# about non-retrieved exceptions etc
try:
data, ws = done.pop().result()
except Exception as e:
print("OTHER EXCEPTION", e)
for task in pending:
task.cancel()
async def listen(self, ws):
try:
async for data in ws:
data = json.loads(data)
# call the subscriber (listener) back when there's data
[s.listener._handle_result(data) for s in self.subscriptions if s.ws == ws]
except Exception as e:
print('ERROR LISTENING; RESTARTING SOCKET', e)
await asyncio.sleep(2)
self.restart_socket(ws)
def subscribe(self, subscription):
task = self.loop.create_task(self._subscribe(subscription))
asyncio.gather(task)
if not self.started:
self.start()
async def _subscribe(self, subscription):
try:
ws = self.connections.get(subscription.url, await websockets.connect(subscription.url))
await ws.send(json.dumps(subscription.sub_msg))
subscription.ws = ws
self.connections[subscription.url] = ws
self.subscriptions.add(subscription)
except Exception as e:
print("ERROR SUBSCRIBING; RETRYING", e)
await asyncio.sleep(2)
self.subscribe(subscription)
def restart_socket(self, ws):
for s in self.subscriptions:
if s.ws == ws and not s.ws.connected:
print(s)
del self.connections[s.url]
self.subscribe(s)

I have a bog-standard synchronous python program that needs to be able to read data from websockets and update the GUI with the data. However, asyncio creep is constantly tripping me up.
As you mentioned GUI, then it is probably not a "bog-standard synchronous python program". Usually a GUI program has a non-blocking event-driven main thread, which allows concurrent user behaviors and callbacks. That is very much similar to asyncio, and it is usually a common way for asyncio to work together with GUIs to use GUI-specific event loop to replace default event loop in asyncio, so that your asyncio coroutines just run in GUI event loop and you can avoid calling run_forever() blocking everything.
An alternative way is to run asyncio event loop in a separate thread, so that your program could at the same time wait for websocket data and wait for user clicks. I've rewritten your code as follows:
import asyncio
import threading
import websockets
import json
class WSClient(threading.Thread):
def __init__(self):
super().__init__()
self._loop = None
self._tasks = {}
self._stop_event = None
def run(self):
self._loop = asyncio.new_event_loop()
self._stop_event = asyncio.Event(loop=self._loop)
try:
self._loop.run_until_complete(self._stop_event.wait())
self._loop.run_until_complete(self._clean())
finally:
self._loop.close()
def stop(self):
self._loop.call_soon_threadsafe(self._stop_event.set)
def subscribe(self, url, sub_msg, callback):
def _subscribe():
if url not in self._tasks:
task = self._loop.create_task(
self._listen(url, sub_msg, callback))
self._tasks[url] = task
self._loop.call_soon_threadsafe(_subscribe)
def unsubscribe(self, url):
def _unsubscribe():
task = self._tasks.pop(url, None)
if task is not None:
task.cancel()
self._loop.call_soon_threadsafe(_unsubscribe)
async def _listen(self, url, sub_msg, callback):
try:
while not self._stop_event.is_set():
try:
ws = await websockets.connect(url, loop=self._loop)
await ws.send(json.dumps(sub_msg))
async for data in ws:
data = json.loads(data)
# NOTE: please make sure that `callback` won't block,
# and it is allowed to update GUI from threads.
# If not, you'll need to find a way to call it from
# main/GUI thread (similar to `call_soon_threadsafe`)
callback(data)
except Exception as e:
print('ERROR; RESTARTING SOCKET IN 2 SECONDS', e)
await asyncio.sleep(2, loop=self._loop)
finally:
self._tasks.pop(url, None)
async def _clean(self):
for task in self._tasks.values():
task.cancel()
await asyncio.gather(*self._tasks.values(), loop=self._loop)

You can try tornado and autobahn-twisted for websockets.

Python: Continuously running a section of a script

I have written a python script that idles waiting for gmail to push a notification indicating that an email has been received. Then the contents are parsed and a database is searched, then database data is emailed back to the original sender.
Currently once an email is received, the script cannot process another email until the script has emailed back. I am wondering if there is a way for the script to be continually listening for an email. As at this point in time if two emails are received at similar times the second one will not be processed.
I think multi-threading might be a solution but I am not sure. Possibly create a new thread for the processEmail.py section of code below?
Sorry if I have explained badly, I am struggling to explain this adequately, feel free to ask for more information.
EDIT: Instead of down voting me could you help me by commenting telling me what more information I should include?
EDIT 2: Here is a code example, I am trying to have the ability to still listen for an email while an email is being processed in processEmail.py
import imaplib2
import time
import subprocess
from threading import *
from subprocess import call
import processEmail
class Idler(object):
def __init__(self, conn):
self.thread = Thread(target=self.idle)
self.M = conn
self.event = Event()
def start(self):
self.thread.start()
def stop(self):
self.event.set()
def join(self):
self.thread.join()
def idle(self):
while True:
if self.event.isSet():
return
self.needsync = False
def callback(args):
if not self.event.isSet():
self.needsync = True
self.event.set()
self.M.idle(callback=callback)
self.event.wait()
if self.needsync:
self.event.clear()
self.dosync()
def dosync(self):
print "An email has been received, please wait...\n"
self.execute()
def execute(self):
processEmail.main()
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login("email_address","email_pass")
M.select("Folder")
idler = Idler(M)
idler.start()
x = False
while not x: time.sleep(0.1)

You can directly inherit from threading.Thread and override its run method:
class SomeTask(threading.Thread):
def run(self):
# Will be executed in separate thread
Start the thread via its start method.
Communication between threads should be handled via queues.

Listen to keypress with asyncio

Can somebody provide a sample of code which listen to keypress in nonblocking manner with asynio and put the keycode in console on every click?
It's not a question about some graphical toolkit

So the link provided by Andrea Corbellini is a clever and thorough solution to the problem, but also quite complicated. If all you want to do is prompt your user to enter some input (or simulate raw_input), I prefer to use the much simpler solution:
import sys
import functools
import asyncio as aio
class Prompt:
def __init__(self, loop=None):
self.loop = loop or aio.get_event_loop()
self.q = aio.Queue()
self.loop.add_reader(sys.stdin, self.got_input)
def got_input(self):
aio.ensure_future(self.q.put(sys.stdin.readline()), loop=self.loop)
async def __call__(self, msg, end='\n', flush=False):
print(msg, end=end, flush=flush)
return (await self.q.get()).rstrip('\n')
prompt = Prompt()
raw_input = functools.partial(prompt, end='', flush=True)
async def main():
# wait for user to press enter
await prompt("press enter to continue")
# simulate raw_input
print(await raw_input('enter something:'))
loop = aio.get_event_loop()
loop.run_until_complete(main())
loop.close()
EDIT: I removed the loop parameter form Queue as it is removed in 3.10.
Also, these days I use structured concurrency (trio), and if anyone is curious this is pretty easy to do in trio:
import trio, sys
async def main():
async with trio.lowlevel.FdStream(sys.stdin.fileno()) as stdin:
async for line in stdin:
if line.startswith(b'q'):
break
print(line)
trio.run(main)

I wrote something similar as part of a package called aioconsole.
It provides a coroutine called get_standard_streams that returns two asyncio streams corresponding to stdin and stdout.
Here's an example:
import asyncio
import aioconsole
async def echo():
stdin, stdout = await aioconsole.get_standard_streams()
async for line in stdin:
stdout.write(line)
loop = asyncio.get_event_loop()
loop.run_until_complete(echo())
It also includes an asynchronous equivalent to input:
something = await aioconsole.ainput('Entrer something: ')
It should work for both file and non-file streams. See the implementation here.

Reading lines
The high-level pure-asyncio way to do this is as follows.
import asyncio
import sys
async def main():
# Create a StreamReader with the default buffer limit of 64 KiB.
reader = asyncio.StreamReader()
pipe = sys.stdin
loop = asyncio.get_event_loop()
await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), pipe)
async for line in reader:
print(f'Got: {line.decode()!r}')
asyncio.run(main())
The async for line in reader loop can be written more explicitly, e.g. if you want to print a prompt or catch exceptions inside the loop:
while True:
print('Prompt: ', end='', flush=True)
try:
line = await reader.readline()
if not line:
break
except ValueError:
print('Line length went over StreamReader buffer limit.')
else:
print(f'Got: {line.decode()!r}')
An empty line (not '\n' but an actually empty string '') means end-of-file. Note that it is possible for await reader.readline() to return '' right after reader.at_eof() returned False. See Python asyncio: StreamReader for details.
Here readline() is asynchronously gathering a line of input. That is, the event loop can run while the reader waits for more characters. In contrast, in the other answers, the event loop could block: it could detect that some input is available, enter the function calling sys.stdin.readline(), and then block on it until an endline becomes available (blocking any other tasks from entering the loop). Of course this isn't a problem in most cases, as the endline becomes available together with (in case of line buffering, which is the default) or very soon after (in other cases, assuming reasonably short lines) any initial characters of a line.
Reading character by character
You can also read individual bytes with await reader.readexactly(1) to read byte-per-byte, when reading from a pipe. When reading key-presses from a terminal, it needs to be set up properly, see Key Listeners in python? for more. On UNIX:
import asyncio
import contextlib
import sys
import termios
#contextlib.contextmanager
def raw_mode(file):
old_attrs = termios.tcgetattr(file.fileno())
new_attrs = old_attrs[:]
new_attrs[3] = new_attrs[3] & ~(termios.ECHO | termios.ICANON)
try:
termios.tcsetattr(file.fileno(), termios.TCSADRAIN, new_attrs)
yield
finally:
termios.tcsetattr(file.fileno(), termios.TCSADRAIN, old_attrs)
async def main():
with raw_mode(sys.stdin):
reader = asyncio.StreamReader()
loop = asyncio.get_event_loop()
await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), sys.stdin)
while not reader.at_eof():
ch = await reader.read(1)
# '' means EOF, chr(4) means EOT (sent by CTRL+D on UNIX terminals)
if not ch or ord(ch) <= 4:
break
print(f'Got: {ch!r}')
asyncio.run(main())
Note this is not really one character or one key at a time: if the user presses a key combination that gives a multi-byte character, like ALT+E, nothing will happen on pressing ALT and two bytes will be sent by the terminal on pressing E, which will result in two iterations of the loop. But it's good enough for ASCII characters like letters and ESC.
If you need actual key presses like ALT, I suppose the only way is to use a suitable library and make it work with asyncio by calling it in a separate thread, like here. In fact the library+thread approach is probably simpler in other cases as well.
Under the hood
If you want finer control you can implement your own protocol in place of StreamReaderProtocol: a class implementing any number of functions of asyncio.Protocol. Minimal example:
class MyReadProtocol(asyncio.Protocol):
def __init__(self, reader: asyncio.StreamReader):
self.reader = reader
def connection_made(self, pipe_transport):
self.reader.set_transport(pipe_transport)
def data_received(self, data: bytes):
self.reader.feed_data(data)
def connection_lost(self, exc):
if exc is None:
self.reader.feed_eof()
else:
self.reader.set_exception(exc)
You could replace the StreamReader with your own buffering mechanism. After you call connect_read_pipe(lambda: MyReadProtocol(reader), pipe), there will be exactly one call to connection_made, then arbitrary many calls to data_received (with data depending on terminal and python buffering options), then eventually exactly one call to connection_lost (on end-of-file or on error). In case you ever need them, connect_read_pipe returns a tuple (transport, protocol), where protocol is an instance of MyReadProtocol (created by the protocol factory, which in our case is a trivial lambda), while transport is an instance of asyncio.ReadTransport (specifically some private implementation like _UnixReadPipeTransport on UNIX).
But in the end this is all boilerplate that eventually relies on loop.add_reader (unrelated to StreamReader).
For Windows you might need to choose the ProactorEventLoop (the default since Python 3.8), see Python asyncio: Platform Support.

An alternative to using queues would be to make the command line an asyn generator, and process the commands as they come in, like so:
import asyncio
import sys
class UserInterface(object):
def __init__(self, task, loop):
self.task = task
self.loop = loop
def get_ui(self):
return asyncio.ensure_future(self._ui_task())
async def _ui_cmd(self):
while True:
cmd = sys.stdin.readline()
cmd = cmd.strip()
if cmd == 'exit':
self.loop.stop()
return
yield cmd
async def _ui_task(self):
async for cmd in self._ui_cmd():
if cmd == 'stop_t':
self.task.stop()
elif cmd == 'start_t':
self.task.start()

Python 3.10 update to the solution provided by bj0:
class Prompt:
def __init__(self):
self.loop = asyncio.get_running_loop()
self.q = asyncio.Queue()
self.loop.add_reader(sys.stdin, self.got_input)
def got_input(self):
asyncio.ensure_future(self.q.put(sys.stdin.readline()), loop=self.loop)
async def __call__(self, msg, end='\n', flush=False):
print(msg, end=end, flush=flush)
# https://docs.python.org/3/library/asyncio-task.html#coroutine
task = asyncio.create_task(self.q.get())
return (await task).rstrip('\n')
I tested it on a websocket client, inside an async function that would get stuck waiting for input, so I replaced s = input("insert string") with s = await prompt("insert string") and now ping-ponging works even while the program is waiting for user input, the connection does not stop anymore and the issue "timed out waiting for keepalive pong" is solved.