I experimenting with the new asyncio features in Python 3.9, and have the following code:
import asyncio
async def inc(start):
i = start
while True:
print(i)
i += 2
return None
async def main():
x = asyncio.gather(
asyncio.to_thread(inc, 0),
asyncio.to_thread(inc, 1)
)
try:
await asyncio.wait_for(x, timeout=5.0)
except asyncio.TimeoutError:
print("timeout!")
asyncio.run(main())
My expectation is that the program will print numbers starting from 0 and terminate after 5 seconds. However, when I execute the program here, I see no output and the following warning:
/usr/lib/python3.9/asyncio/events.py:80: RuntimeWarning: coroutine 'inc' was never awaited
self._context.run(self._callback, *self._args)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.9/concurrent/futures/thread.py:85: RuntimeWarning: coroutine 'inc' was never awaited
del work_item
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
The asyncio.to_thread converts a regular function to a coroutine.
All you need is to change the async def inc to def inc. You can do this, because there is no await, it is not a real coroutine.
However, you cannot kill a thread. The timeout will cause to stop waiting, not to stop computing numbers.
UPDATE: In detail, the asyncio.gather awaits the coroutines until the timeout. When cancelled by the timeout, the gather cancels all still running coroutines a and waits for their termination. Cancelling a coroutine means to deliver an exception at the nearest await statement. In this case, there is no await and the coroutines will never receive the cancellation exception. The program hangs at that point.
Related
I'm trying to modify the graceful shutdown example from RogueLynn to cancel running processes that were spawned by the tasks.
Below is a minimal example to demonstrate the issue I'm facing. With this example, I get a warning message that the callback function isn't awaited and when I do try to terminate the script, the asyncio.gather call doesn't seem to complete. Any idea how to resolve this such that the shutdown callback executes completely?
import asyncio
import functools
import signal
async def run_process(time):
try:
print(f"Starting to sleep for {time} seconds")
await asyncio.sleep(time)
print(f"Completed sleep of {time} seconds")
except asyncio.CancelledError:
print("Received cancellation terminating process")
raise
async def main():
tasks = [run_process(10), run_process(5), run_process(2)]
for future in asyncio.as_completed(tasks):
try:
await future
except Exception as e:
print(f"Caught exception: {e}")
async def shutdown(signal, loop):
# Cancel running tasks on keyboard interrupt
print(f"Running shutdown")
tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]
[task.cancel() for task in tasks]
await asyncio.gather(*tasks, return_exceptions=True)
print("Finished waiting for cancelled tasks")
loop.stop()
try:
loop = asyncio.get_event_loop()
signals = (signal.SIGINT,)
for sig in signals:
loop.add_signal_handler(sig, functools.partial(asyncio.create_task, shutdown(sig, loop)))
loop.run_until_complete(main())
finally:
loop.close()
Output when run to completion:
Starting to sleep for 2 seconds
Starting to sleep for 10 seconds
Starting to sleep for 5 seconds
Completed sleep of 2 seconds
Completed sleep of 5 seconds
Completed sleep of 10 seconds
/home/git/envs/lib/python3.8/asyncio/unix_events.py:140: RuntimeWarning: coroutine 'shutdown' was never awaited
del self._signal_handlers[sig]
And output when script is interrupted:
Starting to sleep for 2 seconds
Starting to sleep for 10 seconds
Starting to sleep for 5 seconds
Completed sleep of 2 seconds
^CRunning shutdown
Received cancellation terminating process
Received cancellation terminating process
Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<shutdown() running at ./test.py:54> wait_for=<_GatheringFuture finished result=[CancelledError(), CancelledError(), CancelledError()]>>
Traceback (most recent call last):
File "./test.py", line 65, in <module>
loop.run_until_complete(main())
File "/home/git/envs/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
asyncio.exceptions.CancelledError
The CancelledError you see results from your as_completed for loop.
If you just want to fix it, you could add an exception handling for it, e.g.
[...]
try:
await future
except Exception as e:
print(f"Caught exception: {e}")
except asyncio.CancelledError:
print("task was cancelled")
[...]
Note that, you will find a warning telling you that Task was destroyed but it is pending! for your shutdown task, which you could just ignore. I guess it is because you stop the loop within a task.
Still, I would like to point at the difference between co-routines, tasks, and futures, see https://docs.python.org/3/library/asyncio-task.html.
What you call tasks and future are co-routines.
For the type of problem that you are trying to solve, I would advise to have a look at Asynchronous Context Managers. Graceful shutdown sounds to me like you want to close some database connections or dump some process variables... Here, you could have a look at https://docs.python.org/3/library/contextlib.html#contextlib.asynccontextmanager
However, if things become more complex, you may want to write such a signal handler which adds its own task to the loop. In this case, I would advise to create the relevant tasks explicitly with asyncio.create_task(coro, name="my-task-name") so you can select exactly the tasks you want to cancel first by name, e.g.
tasks = [
task for task in asyncio.all_tasks()
if task.get_name().startswith("my-task")
]
Otherwise, you may accidentally cancel a cleanup-task.
Looking at the asyncio docs, I came across this example
async def main():
# Create a "cancel_me" Task
task = asyncio.create_task(cancel_me())
# Wait for 1 second
await asyncio.sleep(1)
task.cancel()
try:
await task
except asyncio.CancelledError:
print("main(): cancel_me is cancelled now")
asyncio.run(main())
After task.cancel(), what is the purpose of doing await task? Is this to wait for the future to be finished if it was ever shielded from cancellation?
In other words, why not:
async def main():
# Create a "cancel_me" Task
task = asyncio.create_task(cancel_me())
# Wait for 1 second
await asyncio.sleep(1)
task.cancel()
asyncio.run(main())
From the documentation of cancel() (under asyncio.Task):
This arranges for a CancelledError exception to be thrown into the wrapped coroutine on the next cycle of the event loop.
The coroutine then has a chance to clean up or even deny the request by suppressing the exception with a try … … except CancelledError … finally block.
When asyncio.CancelledError is thrown into cancel_me(), execution resumes in the except asyncio.CancelledError block. For the snippet provided with the documentation, it does not in fact make any difference whether cancel_me() is awaited or not after cancelling, because the exception handling block executes synchronously.
On the other hand, if the exception handling block did perform asynchronous operations, the difference would become visible:
async def cancel_me():
print('cancel_me(): before sleep')
try:
# Wait for 1 hour
await asyncio.sleep(3600)
except asyncio.CancelledError:
await asyncio.sleep(1)
print('cancel_me(): cancel sleep, this never gets printed')
raise
finally:
print('cancel_me(): after sleep')
async def main():
# Create a "cancel_me" Task
task = asyncio.create_task(cancel_me())
# Wait for 1 second
await asyncio.sleep(1)
task.cancel()
print("main(): cancel_me is cancelled now")
asyncio.run(main())
# Expected output:
#
# cancel_me(): before sleep
# main(): cancel_me is cancelled now
# cancel_me(): after sleep
The last, surprising print takes place because of the following:
main() returns after its last print
asyncio.run() tries to cancel all pending tasks
cancel_me(), albeit already cancelled, is still pending, awaiting on the exception block sleep
the finally clause in cancel_me() is executed and the even loop terminates
Also worth noting: given that asyncio.run() throws a CancelledError into all the tasks that are still pending, if cancel_me() had not been cancelled already, the except asyncio.CancelledError block would execute in its entirety.
Here is the code, I thought the program will crash at once because of the uncaught exception. However it waited 10s when the main task coro2 completes.
import asyncio
#asyncio.coroutine
def coro1():
print("coro1 primed")
yield
raise Exception("abc")
#asyncio.coroutine
def coro2(loop):
try:
print("coro2 primed")
ts = [asyncio.Task(coro1(),loop=loop) for _ in range(2)]
res = yield from asyncio.sleep(10)
print(res)
except Exception as e:
print(e)
raise
loop= asyncio.get_event_loop()
loop.run_until_complete(coro2(loop))
I think this is a serious problems because in more complicated programs, this makes the process stuck forever, instead of crashing with exception information.
Besides, I set a breakpoint in the except block in source code of run_until_complete but it's not triggered. I am interested in which piece of code handled that exception in python asyncio.
First, there is no reason to use generator-based coroutines in Python with the async/await syntax available for many years, and the coroutine decorator now deprecated and scheduled for removal. Also, you don't need to pass the event loop down to each coroutine, you can always use asyncio.get_event_loop() to obtain it when you need it. But these are unrelated to your question.
The except block in coro2 didn't trigger because the exception raised in coro1 didn't propagate to coro2. This is because you explicitly ran coro1 as a task, which executed it in the background, and didn't await it. You should always ensure that your tasks are awaited and then exceptions won't pass unnoticed; doing this systematically is sometimes referred to as structured concurrency.
The correct way to write the above would be something like:
async def coro1():
print("coro1 primed")
await asyncio.sleep(0) # yield to the event loop
raise Exception("abc")
async def coro2():
try:
print("coro2 primed")
ts = [asyncio.create_task(coro1()) for _ in range(2)]
await asyncio.sleep(10)
# ensure we pick up results of the tasks that we've started
for t in ts:
await t
print(res)
except Exception as e:
print(e)
raise
asyncio.run(coro2())
Note that this will run sleep() to completion and only then propagate the exceptions raised by the background tasks. If you wanted to propagate immediately, you could use asyncio.gather(), in which case you wouldn't have to bother with explicitly creating tasks in the first place:
async def coro2():
try:
print("coro2 primed")
res, *ignored = await asyncio.gather(
asyncio.sleep(10),
*[(coro1()) for _ in range(2)]
)
print(res)
except Exception as e:
print(e)
raise
I am interested in which piece of code handled that exception in python asyncio.
An exception raised by a coroutine which is not handled is caught by asyncio and stored in the task object. This allows you to await the task or (if you know it's completed) obtain its result using the result() method, either of which will propagate (re-raise) the exception. Since your code never accessed the task's result, the exception instance remained forgotten inside the task object. Python goes so far to notice this and print a "Task exception was never retrieved" warning when the task object is destroyed along with a traceback, but this warning is provided on a best-effort basis, usually comes too late, and should not be relied upon.
I am using python 3.8 with this code
async def main():
pass
async def build():
pass
asyncio.create_task(build())
loop = asyncio.get_event_loop()
asyncio.create_task(main())
pending = asyncio.all_tasks()
loop.run_until_complete(asyncio.gather(*pending))
and get the following error
sys:1: RuntimeWarning: coroutine 'build' was never awaited
What am I missing here? shouldn't run until complete wait for all the tasks to finish?
As the loop is not running when creating the task, asyncio is unable to attach tasks to it.
This can be fixed by replacing asyncio.create_task() with loop.create_task().
The full error can be retrieved using asyncio.gather(..., return_exceptions=True) so gather() will raise a RuntimeError: no running event loop.
I have an asyncio.Condition named cond. I wish to wait on it, but only for so long before giving up. As asyncio.Condition.wait does not take a timeout, this cannot be done directly. The docs state that asyncio.wait_for should be used to wrap and provide a timeout instead:
The asyncio.wait_for() function can be used to cancel a task after a timeout.
Thus, we arrive at the following solution:
async def coro():
print("Taking lock...")
async with cond:
print("Lock acquired.")
print("Waiting!")
await asyncio.wait_for(cond.wait(), timeout=999)
print("Was notified!")
print("Lock released.")
Now assume that coro itself is cancelled five seconds after being run. This throws CancelledError in the wait_for, which cancels the cond.wait before re-raising the error. The error then propagates to coro, which due to the async with block, implicitly attempts to release the lock in cond. However, the lock is not currently held; cond.wait has been cancelled but hasn't had a chance to process that cancellation and re-acquire the lock. Thus, we get an ugly exception like the following:
Taking lock...
Lock acquired.
Waiting!
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<coro() done, defined at [REDACTED]> exception=RuntimeError('Lock is not acquired.',)>
Traceback (most recent call last):
[REDACTED], in coro
await asyncio.wait_for(cond.wait(), timeout=999)
[REDACTED], in wait_for
yield from waiter
concurrent.futures._base.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
[REDACTED], in coro
print("Was notified!")
[REDACTED], in coro
res = func(*args, **kw)
[REDACTED], in __aexit__
self.release()
[REDACTED], in release
raise RuntimeError('Lock is not acquired.')
RuntimeError: Lock is not acquired.
In other words, whilst handling the CancelledError, coro raised a RuntimeError from trying to release a lock that wasn't held. The reason the stacktrace shows the print("Was notified!") line is because that is the last line of the offending async with block.
This doesn't feel like something I can fix; I'm starting to suspect it's a bug in the library itself. However, I can't think of any way to avoid the issue or create a workaround, so any ideas would be appreciated.
Whilst writing this question and investigating further, I stumbled into similar problems on the Python bug tracker, ended up inspecting the asyncio source code, and determined that this is, in fact, a bug in asyncio itself.
I've submitted it to the issue tracker here for those who have the same problem, and answered my own question with a workaround that I have created.
EDIT: As requested by ParkerD, here is the full runnable example that produces the above issue:
EDIT 2: updated example to use the new asyncio.run and asyncio.create_task features from Python 3.7+
import asyncio
async def coro():
cond = asyncio.Condition()
print("Taking lock...")
async with cond:
print("Lock acquired.")
print("Waiting!")
await asyncio.wait_for(cond.wait(), timeout=999)
print("Was notified!")
print("Lock released.")
async def cancel_after_5(c):
task = asyncio.create_task(c)
await asyncio.sleep(5)
task.cancel()
await asyncio.wait([task])
asyncio.run(cancel_after_5(coro()))
As stated at the end of the question, I've determined that the issue is actually a bug in the library. I'll reiterate that the issue tracker for that bug is here, and present my workaround.
The following function is based on wait_for itself (source here), and is a version of it specialised for waiting on conditions, with the added guarantee that cancelling it is safe.
Calling wait_on_condition_with_timeout(cond, timeout) is roughly equivalent to asyncio.wait_for(cond.wait(), timeout).
async def wait_on_condition_with_timeout(condition: asyncio.Condition, timeout: float) -> bool:
loop = asyncio.get_event_loop()
# Create a future that will be triggered by either completion or timeout.
waiter = loop.create_future()
# Callback to trigger the future. The varargs are there to consume and void any arguments passed.
# This allows the same callback to be used in loop.call_later and wait_task.add_done_callback,
# which automatically passes the finished future in.
def release_waiter(*_):
if not waiter.done():
waiter.set_result(None)
# Set up the timeout
timeout_handle = loop.call_later(timeout, release_waiter)
# Launch the wait task
wait_task = loop.create_task(condition.wait())
wait_task.add_done_callback(release_waiter)
try:
await waiter # Returns on wait complete or timeout
if wait_task.done():
return True
else:
raise asyncio.TimeoutError()
except (asyncio.TimeoutError, asyncio.CancelledError):
# If timeout or cancellation occur, clean up, cancel the wait, let it handle the cancellation,
# then re-raise.
wait_task.remove_done_callback(release_waiter)
wait_task.cancel()
await asyncio.wait([wait_task])
raise
finally:
timeout_handle.cancel()
The crucial part is that if a timeout or cancellation occurs, the method waits for the condition to re-acquire the lock before re-raising the exception:
except (asyncio.TimeoutError, asyncio.CancelledError):
# If timeout or cancellation occur, clean up, cancel the wait, let it handle the cancellation,
# then re-raise.
wait_task.remove_done_callback(release_waiter)
wait_task.cancel()
await asyncio.wait([wait_task]) # This line is missing from the real wait_for
raise
I've tested this on Python 3.6.9 and it works perfectly. The same bug exists in 3.7 and 3.8 too, so I imagine it is also useful for those versions. If you want to know when the bug will be fixed, check the issue tracker above. If you want a version for things other than Conditions, it should be trivial to change the parameter and create_task line.