I'm watching import asyncio: Learn Python's AsyncIO #3 - Using Coroutines. The instructor gave the following example:
import asyncio
import datetime
async def keep_printing(name):
while True:
print(name, end=" ")
print(datetime.datetime.now())
await asyncio.sleep(0.5)
async def main():
group_task = asyncio.gather(
keep_printing("First"),
keep_printing("Second"),
keep_printing("Third")
)
try:
await asyncio.wait_for(group_task, 3)
except asyncio.TimeoutError:
print("Time's up!")
if __name__ == "__main__":
asyncio.run(main())
The output had an exception:
First 2020-08-11 14:53:12.079830
Second 2020-08-11 14:53:12.079830
Third 2020-08-11 14:53:12.080828
First 2020-08-11 14:53:12.580865
Second 2020-08-11 14:53:12.580865
Third 2020-08-11 14:53:12.581901
First 2020-08-11 14:53:13.081979
Second 2020-08-11 14:53:13.082408
Third 2020-08-11 14:53:13.082408
First 2020-08-11 14:53:13.583497
Second 2020-08-11 14:53:13.583935
Third 2020-08-11 14:53:13.584946
First 2020-08-11 14:53:14.079666
Second 2020-08-11 14:53:14.081169
Third 2020-08-11 14:53:14.115689
First 2020-08-11 14:53:14.570694
Second 2020-08-11 14:53:14.571668
Third 2020-08-11 14:53:14.635769
First 2020-08-11 14:53:15.074124
Second 2020-08-11 14:53:15.074900
Time's up!
_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError()>
concurrent.futures._base.CancelledError
The instructor tried to the handle the CancelledError by adding a try/except in keep_printing:
async def keep_printing(name):
while True:
print(name, end=" ")
print(datetime.datetime.now())
try:
await asyncio.sleep(0.5)
except asyncio.CancelledError:
print(name, "was cancelled!")
break
However, the same exception still occurred:
# keep printing datetimes
...
First was cancelled!
Second was cancelled!
Third was cancelled!
Time's up!
_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError()>
concurrent.futures._base.CancelledError
The instructor then just proceeded to other topics and never came back to this example to show how to fix it. Fortunately, through experimentation, I discovered that we could fix it by adding another try/except under the except asyncio.TimeoutError: in the main async function:
async def main():
group_task = asyncio.gather(
keep_printing("First"),
keep_printing("Second"),
keep_printing("Third")
)
try:
await asyncio.wait_for(group_task, 3)
except asyncio.TimeoutError:
print("Time's up!")
try:
await group_task
except asyncio.CancelledError:
print("Main was cancelled!")
The final output was:
# keep printing datetimes
...
First was cancelled!
Second was cancelled!
Third was cancelled!
Time's up!
Main was cancelled!
In fact, with this edition of main, we don't even need the try...except asyncio.CancelledError in keep_printing. It would still work fine.
Why was that? Why did catching CancelledError in main work but not in keep_printing? The way that the video instructor dealt with this exception only made me more confused. He didn't need to change any code of keep_printing in the first place!
Let's find out what's going on:
This code schedules three coroutines to be executed and return Future object group_task (instance of internal class _GatheringFuture) aggregating results.
group_task = asyncio.gather(
keep_printing("First"),
keep_printing("Second"),
keep_printing("Third")
)
This code waits for future to complete with a timeout. And if a timeout occurs, it cancels the future and raises asyncio.TimeoutError.
try:
await asyncio.wait_for(group_task, 3)
except asyncio.TimeoutError:
print("Time's up!")
Timeout occurs. Let's look inside the asyncio library task.py. wait_for does the following:
timeout_handle = loop.call_later(timeout, _release_waiter, waiter)
...
await waiter
...
await _cancel_and_wait(fut, loop=loop) # _GatheringFuture.cancel() inside
raise exceptions.TimeoutError()
When we do _GatheringFuture.cancel(), if any child tasks were actually cancelled CancelledError is propagated
class _GatheringFuture(futures.Future):
...
def cancel(self):
...
for child in self._children:
if child.cancel():
ret = True
if ret:
# If any child tasks were actually cancelled, we should
# propagate the cancellation request regardless of
# *return_exceptions* argument. See issue 32684.
self._cancel_requested = True
return ret
And later
...
if outer._cancel_requested:
# If gather is being cancelled we must propagate the
# cancellation regardless of *return_exceptions* argument.
# See issue 32684.
outer.set_exception(exceptions.CancelledError())
else:
outer.set_result(results)
Thus, it is more correct to extract the result or exception from gathering future
async def main():
group_task = asyncio.gather(
keep_printing("First"),
keep_printing("Second"),
keep_printing("Third")
)
try:
await asyncio.wait_for(group_task, 3)
except asyncio.TimeoutError:
print("Time's up!")
try:
result = await group_task
except asyncio.CancelledError:
print("Gather was cancelled")
I think you need to put await before asyncio.gather.
So this call taken from your code:
group_task = asyncio.gather(
keep_printing("First"),
keep_printing("Second"),
keep_printing("Third")
)
Needs to be changed into:
group_task = await asyncio.gather(
keep_printing("First"),
keep_printing("Second"),
keep_printing("Third")
)
Not sure why, I'm still learning this stuff.
When aw is cancelled due to a timeout, wait_for waits for aw to be cancelled. You get the timeout error if handle CancelledError into your coroutine. This changed in version 3.7.
Example
import asyncio
import datetime
async def keep_printing(name):
print(datetime.datetime.now())
try:
await asyncio.sleep(3600)
except asyncio.exceptions.CancelledError:
print("done")
async def main():
try:
await asyncio.wait_for(keep_printing("First"), timeout=3)
except asyncio.exceptions.TimeoutError:
print("timeouted")
if __name__ == "__main__":
asyncio.run(main())
The gather method used for retrieving results from Task or Future, you have an infinite loop and never return any result. If any Task or Future from the aws sequence is cancelled (what's happening with the wait_for), it is treated as if it raised CancelledError – the gather() call is not cancelled in this case. This is to prevent the cancellation of one submitted Task/Future to cause other Tasks/Futures to be cancelled.
For the protecting gather method, you can cover it to the shield.
import asyncio
import datetime
async def keep_printing(name):
while True:
print(name, datetime.datetime.now())
try:
await asyncio.sleep(0.5)
except asyncio.exceptions.CancelledError:
print(f"canceled {name}")
return None
async def main():
group_task = asyncio.shield(asyncio.gather(
keep_printing("First"),
keep_printing("Second"),
keep_printing("Third"))
)
try:
await asyncio.wait_for(group_task, 3)
except asyncio.exceptions.TimeoutError:
print("Done")
if __name__ == "__main__":
asyncio.run(main())
Related
The following code works, but fails to exit upon CTRL+c
Can anyone explain why, and how to fix?
It's asyncio code, but I'm using trio-asyncio (forward planning) to allow me to also use trio code.
#!.venv/bin/python
import asyncio
import trio
import trio_asyncio
from binance import AsyncClient, BinanceSocketManager
as_trio = trio_asyncio.aio_as_trio
as_aio = trio_asyncio.trio_as_aio
from contextlib import asynccontextmanager
#asynccontextmanager
async def AClient():
client = await AsyncClient.create()
try:
yield client
except KeyboardInterrupt:
print('ctrl+c')
exit(0)
finally:
await client.close_connection()
async def kline_listener(client):
print(1)
bm = BinanceSocketManager(client)
async with bm.kline_socket(symbol='BNBBTC') as stream:
while True:
try:
res = await stream.recv()
print(res)
except KeyboardInterrupt:
break
#as_trio
async def aio_main():
async with AClient() as client:
exchange_info = await client.get_exchange_info()
tickers = await client.get_all_tickers()
print(client, exchange_info.keys(), tickers[:2])
kline = asyncio.create_task(kline_listener(client))
await kline
if __name__ == "__main__":
trio_asyncio.run(aio_main)
I've placed except KeyboardInterrupt at 2 places, though it appears to do nothing.
Hi I have to process several objects queued 5 at time.
I have a queue of 5 items.
Sometimes process fails and an exception occurs:
async def worker(nam):
while True:
queue_item = await queue.get()
Worker starts the process loop and tries to process items
try:
loop = asyncio.get_event_loop()
task = loop.create_task(download(queue_item, path))
download_result = await asyncio.wait_for(task, timeout=timeout)
except asyncio.TimeoutError:
unfortunately the process timed out.
Can I add like this ?
except asyncio.TimeoutError:
await queue.put(queue_item)
I want to process again that item on next round
Thank
Yes, you can re-queue an object at the end of the queue for processing. A simple
example based on your code:
import asyncio
from random import randrange
async def download(item):
print("Process item", item)
if randrange(4) == 1: # simulate occasional event
await asyncio.sleep(100) # trigger timeout error
async def worker(queue):
while True:
queue_item = await queue.get()
try:
result = await asyncio.wait_for(download(queue_item), timeout=1)
except asyncio.TimeoutError:
print("Timeout for ", queue_item)
await queue.put(queue_item)
queue.task_done()
async def main():
q = asyncio.Queue()
asyncio.create_task(worker(q))
for i in range(5): # put 5 items to process
await q.put(i)
await q.join()
asyncio.run(main())
Process item 0
Timeout for 0
Process item 1
Process item 2
Process item 3
Timeout for 3
Process item 4
Process item 0
Process item 3
I can't figure how to stop loop after one task is finished. In sample when WsServe count to 5 I expect loop to close. But instead stop I got RuntimeError: Cannot close a running event loop
#!/usr/bin/env python
import asyncio
async def rxer():
i=0
while True:
i+=1
print ('Rxer ',i)
await asyncio.sleep(1)
async def WsServe():
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
loop.stop()
loop.close()
loop=asyncio.get_event_loop()
loop.create_task(rxer())
loop.run_until_complete(WsServe())
loop.run_forever()
The error comes from calling loop.close() from inside the loop. You don't need to bother with loop.close(), loop.stop() is quite sufficient to stop the loop. loop.close() is only relevant when you want to ensure that all the resources internally acquired by the loop are released. It is not needed when your process is about to exit anyway, and removing the call to loop.close() indeed eliminates the error.
But also, loop.stop() is incompatible with run_until_complete(). It happens to work in this code because the coroutine returns immediately after calling loop.stop(); if you added e.g. an await asyncio.sleep(1) after loop.stop(), you'd again get a (different) RuntimeError.
To avoid such issues, I suggest that you migrate to the newer asyncio.run API and avoid both run_until_complete and stop. Instead, you can just use an event to terminate the main function, and the loop with it:
# rxer() defined as before
async def WsServe(stop_event):
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
stop_event.set()
await asyncio.sleep(1)
async def main():
asyncio.get_event_loop().create_task(rxer())
stop_event = asyncio.Event()
asyncio.get_event_loop().create_task(WsServe(stop_event))
await stop_event.wait()
asyncio.run(main())
# python 3.6 and older:
#asyncio.get_event_loop().run_until_complete(main())
Check commented lines of your implementation as below:
import asyncio
async def rxer():
i=0
while True:
i+=1
print ('Rxer ',i)
await asyncio.sleep(1)
async def WsServe():
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
#loop.stop()
#loop.close()
loop=asyncio.get_event_loop()
loop.create_task(rxer())
loop.run_until_complete(WsServe())
#loop.run_forever()
And here is the output:
Rxer 1
WsServe 0
Rxer 2
WsServe 1
Rxer 3
WsServe 2
Rxer 4
WsServe 3
Rxer 5
WsServe 4
Rxer 6
Finish
This is a test script I created to better understand task cancellation -
import asyncio
import random
import signal
import traceback
async def shutdown(signame, loop):
print("Shutting down")
tasks = [task for task in asyncio.Task.all_tasks()]
for task in tasks:
task.cancel()
try:
await task
except asyncio.CancelledError:
print("Task cancelled: %s", task)
loop.stop()
async def another():
await asyncio.sleep(2)
async def some_other_process():
await asyncio.sleep(5)
return "Me"
async def process(job, loop, i):
print(i)
task = loop.create_task(some_other_process())
value = await task
if i < 1:
another_task = loop.create_task(another())
await another_task
# await some_other_process()
def pull(loop):
i = 0
while True:
job = f"random-integer-{random.randint(0, 100)}"
try:
loop.run_until_complete(process(job, loop, i))
i += 1
except asyncio.CancelledError as e:
print("Task cancelled")
break
except Exception:
print(traceback.format_exc())
# asyncio.get_event_loop().stop()
def main():
try:
loop = asyncio.get_event_loop()
for signame in ['SIGINT']:
loop.add_signal_handler(
getattr(signal, signame),
lambda: asyncio.ensure_future(shutdown(signame, loop))
)
try:
pull(loop)
except Exception:
print(traceback.format_exc())
finally:
loop.close()
finally:
print("Done")
if __name__ == "__main__":
main()
And I can not understand why I see -
Task was destroyed but it is pending!
task: <Task cancelling coro=<shutdown() done, defined at test.py:6>>
loop.add_signal_handler(
getattr(signal, signame),
lambda: asyncio.ensure_future(shutdown(signame, loop))
)
Here using asyncio.ensure_future you create task for shutdown coroutine, but you don't await anywhere for this task to be finished. Later when you close event loop it warns you this task is pending.
Upd:
If you want to do some clenup, the best place for it is right before loop.close() regardless of reason your script ended (signal, exception, etc.)
Try to alter your code this way:
# ...
async def shutdown(loop): # remove `signal` arg
# ...
def main():
try:
loop = asyncio.get_event_loop()
try:
pull(loop)
except Exception:
print(traceback.format_exc())
finally:
loop.run_until_complete(shutdown(loop)) # just run until shutdown is done
loop.close()
finally:
print("Done")
# ...
Upd2:
In case you still want signal handler, you probably want to do something like this:
from functools import partial
loop.add_signal_handler(
getattr(signal, signame),
partial(cb, signame, loop)
)
def cb(signame, loop):
loop.stop()
loop.run_until_complete(shutdown(signame, loop))
Just started experimenting with asynch which looks really cool. I'm trying to use futures with an asynch coroutine that runs forever but I get this error:
Task exception was never retrieved
future: <Task finished coro=<slow_operation() done, defined at ./asynchio-test3.py:5> exception=InvalidStateError("FINISHED: <Future finished result='This is the future!'>",)>
This is my code which runs as expected if I remove the 3 lines related to futures:
import asyncio
#asyncio.coroutine
def slow_operation():
yield from asyncio.sleep(1)
print ("This is the task!")
future.set_result('This is the future!')
asyncio.async(slow_operation())
def got_result(future):
print(future.result())
loop = asyncio.get_event_loop()
future = asyncio.Future()
future.add_done_callback(got_result)
asyncio.async(slow_operation())
try:
loop.run_forever()
finally:
loop.close()
slow_operator is called indefinitely, calling set_result for the same future object multiple times; which is not possbile.
>>> import asyncio
>>> future = asyncio.Future()
>>> future.set_result('result')
>>> future.set_result('result')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\asyncio\futures.py", line 329, in set_result
raise InvalidStateError('{}: {!r}'.format(self._state, self))
asyncio.futures.InvalidStateError: FINISHED: <Future finished result='result'>
Create new future for each slow_operator call. For example:
#asyncio.coroutine
def slow_operation(future):
yield from asyncio.sleep(1)
print ("This is the task!")
future.set_result('This is the future!')
asyncio.async(slow_operation(new_future()))
def got_result(future):
print(future.result())
def new_future():
future = asyncio.Future()
future.add_done_callback(got_result)
return future
loop = asyncio.get_event_loop()
asyncio.async(slow_operation(new_future()))
try:
loop.run_forever()
finally:
loop.close()
BTW, you can use new syntax (async, await) if you're using Python 3.5+:
async def slow_operation(future):
await asyncio.sleep(1)
print ("This is the task!")
future.set_result('This is the future!')
asyncio.ensure_future(slow_operation(new_future()))
Following #falsetru answer this is a complete program that has 3 asynch coroutines each with their own got_result function. I'm using v3.4 so thats why I don't use the new syntax. As an interesting side effect the output clearly demonstrates the single threaded nature of coroutines. I hope its useful as a template for someone:
import asyncio
#asyncio.coroutine
def task1(future):
yield from asyncio.sleep(1)
print ("This is operation#1")
future.set_result('This is the result of operation #1!')
asyncio.async(task1(new_future(got_result1)))
def got_result1(future):
print(future.result())
#asyncio.coroutine
def task2(future):
yield from asyncio.sleep(1)
print ("This is operation#2")
future.set_result('This is the result of operation #2!')
asyncio.async(task2(new_future(got_result2)))
def got_result2(future):
print(future.result())
#asyncio.coroutine
def task3(future):
yield from asyncio.sleep(1)
print ("This is operation#3")
future.set_result('This is the result of operation #3!')
asyncio.async(task3(new_future(got_result3)))
def got_result3(future):
print(future.result())
def new_future(callback):
future = asyncio.Future()
future.add_done_callback(callback)
return future
tasks = [task1(new_future(got_result1)),
task2(new_future(got_result2)),
task3(new_future(got_result3))]
loop = asyncio.get_event_loop()
for task in tasks:
asyncio.async(task)
try:
loop.run_forever()
finally:
loop.close()