I have some code which is part of a larger application. It sets up a future like so:
self.__task = asyncio.ensure_future(self.__func())
My understanding is that I never need to explicitly await that task, that will be done automatically by the run loop. But at some point in working with the application I get the following exception:
RuntimeWarning: coroutine 'MyClass.__func' was never awaited
How can that happen? Does it mean that the run loop exited before it could run the task?
Related
I am trying to build a number of Machine Learning models on a single dataset. The output of all models is then to be used in further steps. I would like the training of the models, to happen simultaneously to save time and manual labour.
I am completely new to asynchronous processing, and that has manifested itself in my code below not working. I get the error:
sys:1 RuntimeWarning: coroutine 'level1models' was never awaited
This appears to be a fairly common issue when await isn't used, but wherever I place this command the error persists, and answers I find online do not seem to address functions that return values.
To provide a reproducible example I have altered my code while keeping the structure identical to the original.
from time import sleep
nrs_list = [1, 2, 3, 4, 5]
def subtract(n):
return n - 1
async def subtract_nrs(nrs):
# Train selected ML models
numbers = {nr: subtract(nr) for nr in nrs}
sleep(50)
# Loop to check if all models are trained
while True:
print([i for i in numbers.values()])
if [i for i in numbers.values()] != [None for _ in range(len(numbers))]:
break
sleep(5)
return numbers
r = subtract_nrs(nrs_list)
print(r)
<coroutine object subtract_nrs at 0x000002A413A4C4C0>
sys:1: RuntimeWarning: coroutine 'subtract_nrs' was never awaited
Anytime you create a coroutine (here when you call subtract_nrs) but don't await it, asyncio will emit the warning you received [0]. The wait you avoid this is by awaiting the coroutine, either via
await subtract_nrs(nrs_list)
or by using asyncio.gather [1], which itself must be awaited
await asyncio.gather(subtract_nrs(nrs_list)
Note that here there's no value in using asyncio.gather. That would only come if you needed to wait for multiple coroutines at once.
Based on your code, you seem to be using subtract_nrs as the entry point to your program. await can't be used outside of an async def, so you need another way to wait for it. For that, you'll typically want to use asyncio.run [2]. This will handle creating, running, and closing the event loop along with waiting for your coroutine.
asyncio.run(subtract_nrs(nrs_list))
Now that we've covered all that, asyncio won't actually help you achieve your goal of simultaneous execution. asyncio never does things simultaneously; it does things concurrently [3]. While one task is waiting for I/O to complete, asyncio's event loop allows another to execute. While you've stated that this is a simplified version of your actual code, the code you've provided isn't I/O-bound; it's CPU-bound. This kind of code doesn't work well with asyncio. To use your CPU-bound code and achieve something more akin to simultaneous execution, you should use processes. not asyncio. The best way to do this is with ProcessPoolExecutor from concurrent.futures [4].
I am running asyncio.run(func) in separate callback threads, and am receiving this error:
RuntimeError: Task <Task pending coro=<StreamReader.read() running at /usr/lib/python3.7/asyncio/streams.py:640> cb=[_release_waiter(<Future pendi...ab3a77710>()]>)() at /usr/lib/python3.7/asyncio/tasks.py:392]> got Future <Future pending> attached to a different loop
If I catch the exception, and proceed, everything is working fine.
One thing to note is, that my threads are started from an async function, but the callbacks themselves are synchronous. So the structure is something like: async-->sync threads-->async.run
Unfortunately, I cannot change the behavior of callbacks, as they are part of a 3rd party library (boto3) which is why I am resorting to async.run which works totally fine, but causes these Future warnings.
I'm writing a library using asyncio. Under some circumstances it detects a ciritical error and cannot continue. It is not a bug, the problem has an external cause.
A regular library code would raise an exception. The program will terminate by default, but the caller also has a chance to catch the exception and perform a cleanup or some kind of reset. That is what I want, but unfortunately exceptions do not work that way in asyncio. More about that:
https://docs.python.org/3/library/asyncio-dev.html#detect-never-retrieved-exceptions
What is a reasonable way to notify the async library user about the problem? Probably some callback activated when the error occurs. The user may do necessary cleanup and then exit in the callback.
But what should be the default action? Cancel the current task? Stop the entire event loop? Calling sys.exit?
In general, there should be no need for error-specific callbacks. Asyncio fully supports propagating exceptions across await boundaries inside coroutines, as well as across calls like run_until_complete where sync and async code meet. When someone awaits your coroutine, you can just raise an exception in the usual way.
One pitfall is with the coroutines that run as "background tasks". When such coroutines fail, potentially rendering the library unusable, no one will get notified automatically. This is a known deficiency in asyncio (see here for a detailed discussion), which is currently being addressed. In the meantime, you can achieve equivalent functionality with code like this:
class Library:
async def work_forever(self):
loop = asyncio.get_event_loop()
self._exit_future = loop.create_future()
await self._exit_future
async def stop_working(self):
self._cleanup()
self._exit_future.set_result(None)
async def _failed(self):
self._cleanup()
self._exit_future.set_exception(YourExceptionType())
def _cleanup(self):
# cancel the worker tasks
...
work_forever is analogous to serve_forever, a call that can be awaited by a main() coroutine, or even directly passed to asyncio.run(). In this design the library may detect an erroneous state and propagate the exception, or the main program (presumably through a separately spawned coroutine) can request it to exit cleanly.
Given the following program:
import asyncio
async def coro():
future = asyncio.Future()
future.cancel()
print(future) # <Future cancelled>
await future # no output
loop = asyncio.get_event_loop()
loop.create_task(coro())
loop.run_forever()
Why is the CancelledError thrown by await future not shown? Explicitly wrapping await future in try/except shows that it occurs. Other unhandled errors are shown with Task exception was never retrieved, like this:
async def coro2():
raise Exception('foo')
loop.create_task(coro2())
Why is this not the case for awaiting cancelled Futures?
Additional question: What happens internally if a coroutine awaits a cancelled Future? Does it wait forever? Do I have to do any "cleanup" work?
Why is the CancelledError thrown by await future not shown?
The exception is not shown because you're never actually retrieving the result of coro. If you retrieved it in any way, e.g. by calling result() method on the task or just by awaiting it, you would get the expected error at the point where you retrieve it. The easiest way to observe the resulting traceback is by changing run_forever() to run_until_complete(coro()).
What happens internally if a coroutine awaits a cancelled Future? Does it wait forever? Do I have to do any "cleanup" work?
It doesn't wait forever, it receives the CancelledError at the point of await. You have discovered this already by adding a try/except around await future. The cleanup you need to do is the same like for any other exception - either nothing at all, or using with and finally to make sure that the resources you acquired get released in case of an exit.
Other unhandled errors are shown with Task exception was never retrieved [...] Why is this not the case for awaiting cancelled Futures?
Because Future.cancel intentionally disables the logging of traceback. This is to avoid spewing the output with "exception was never retrieved" whenever a future is canceled. Since CancelledError is normally injected from the outside and can happen at (almost) any moment, very little value is derived from retrieving it.
If it sounds strange to show the traceback in case of one exception but not in case of another, be aware that the tracebacks of failed tasks are displayed on a best-effort basis to begin with. Tasks created with create_task and not awaited effectively run "in the background", much like a thread that is not join()ed. But unlike threads, coroutines have the concept of a "result", either an object returned from the coroutine or an exception raised by it. The return value of the coroutine is provided by the result of its task. When the coroutine exits with an exception, the result encapsulates the exception, to be automatically raised when the result is retrieved. This is why Python cannot immediately print the traceback like it does when a thread terminates due to an unhandled exception - it has to wait for someone to actually retrieve the result. It is only when a Future whose result holds an exception is about to get garbage-collected that Python can tell that the result would never be retrieved. It then displays the warning and the traceback to avoid the exception passing silently.
If I run this on the python3 interpreter:
import asyncio
#asyncio.coroutine
def wait(n):
asyncio.sleep(n)
loop = asyncio.get_event_loop()
fut = asyncio.async(wait(10))
fut.add_done_callback(lambda x: print('Done'))
asyncio.Task.all_tasks()
I get the following result:
{<Task pending coro=<coro() running at /usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/coroutines.py:139> cb=[<lambda>() at <ipython-input-5-c72c2da2ffa4>:1]>}
Now if I run fut.cancel() I get True returned. But typing fut returns a representation of the task stating it is cancelling:
<Task cancelling coro=<coro() running at /usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/coroutines.py:139> cb=[<lambda>() at <ipython-input-5-c72c2da2ffa4>:1]>
And the task never actually cancels (fut.cancelled() never returns True)
Why won't it cancel?
Calling task.cancel() only schedules the task to be cancelled on the next run of the event loop; it doesn't immediately cancel the task, or even guarantee that the task will be actually be cancelled when the event loop runs its next iteration. This is all described in the documentation:
cancel()
Request that this task cancel itself.
This arranges for a CancelledError to be thrown into the wrapped
coroutine on the next cycle through the event loop. The coroutine then
has a chance to clean up or even deny the request using
try/except/finally.
Unlike Future.cancel(), this does not guarantee that the task will be
cancelled: the exception might be caught and acted upon, delaying
cancellation of the task or preventing cancellation completely. The
task may also return a value or raise a different exception.
Immediately after this method is called, cancelled() will not return
True (unless the task was already cancelled). A task will be marked as
cancelled when the wrapped coroutine terminates with a CancelledError
exception (even if cancel() was not called).
In your case, you're never actually starting the event loop, so the task never gets cancelled. You would need to call loop.run_until_complete(fut) (or loop.run_forever(), though that's not really the best choice for this particular case) for the task to actually end up getting cancelled.
Also, for what it's worth, it's usually easier to test asyncio code using actual scripts, rather than the interpreter, since it tends to get tedious to have to constantly rewrite coroutines and start/stop the event loop.
With asyncio testing in the interpreter is tricky, because python needs to keep the event loop constantly polling its tasks.
So a few pieces of advice to test asyncio are:
Write and run scripts instead of using the interactive interpreter
Add a loop.run_forever() at the end of the script so all tasks get executed.
An alternative is to run loop.run_until_complete(coro()) for each task you want to run.
Have yield from in front of asyncio.sleep(n) so it can actually be run. The current code returns a generator and does nothing.