I have several http requests to fire simultaneously. I am trying to use async for to do this.
import asyncio
async def ticker(delay, to):
for i in range(to):
yield i
print(i)
await asyncio.sleep(delay) # instead of aiohttp request
print(i, ' fin')
async def main():
async for x in ticker(1,2):
pass
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
I keep getting subsequent calls with the following:
0
0 fin
1
1 fin
Instead I need the output as shown below:
0
1
0 fin
1 fin
Could you please advise me on how to do this?
The problem is that async for is exactly what you don't need.
async for is designed to iterate while waiting for a task to complete between each iteration; you want to iterate (starting requests) without waiting for the previous task(s) to finish.
You'll want something like
async def do_request():
await asyncio.sleep(1)
async def main():
await asyncio.gather(*[
do_request() for i in range(10)
])
Comment with a follow-up if that doesn't answer your question.
Related
I was experimenting with some basic constructs in python asyncio. I came across the following scenarios:
Snippet 1
import asyncio
async def A():
await asyncio.sleep(10)
print("A...")
async def B():
await asyncio.sleep(15)
print("B...")
async def main():
t = asyncio.create_task(A())
n = asyncio.create_task(B())
await n
print("CCDD...")
await t
asyncio.run(main())
Snippet 2
import asyncio
async def A():
await asyncio.sleep(10)
print("A...")
async def B():
await asyncio.sleep(15)
print("B...")
async def main():
t = asyncio.create_task(A())
n = asyncio.create_task(B())
await n
await t
print("CCDD...")
asyncio.run(main())
Snippet 3
import asyncio
async def A():
await asyncio.sleep(10)
print("A...")
async def B():
await asyncio.sleep(15)
print("B...")
async def main():
t = asyncio.create_task(A())
n = asyncio.create_task(B())
print("CCDD...")
await n
await t
asyncio.run(main())
I find it difficult to understand how the output produced by the first two snippets above is the same, but the output produced by the last snippet is different than the first two?
Output of snippets 1, 2
A...
B...
CCDD...
Output of snippet 3
CCDD...
A...
B...
It's all a matter of thinking about the sequencing. First off, n/15/B is always the 15-second task and t/10/A is the 10-second one. For all snippets, that means A will be printed before B, as you start them at roughly the same time.
In snippet 1, you start them both then wait for the 15-second task, meaning that both will finish before main prints CCDD (waiting for the 10-second task after that but it's already finished). Hence you see A B CCDD.
In snippet 2, you wait for the both the 15-second and 10-second task to finish before main prints CCDD, resulting in A B CCDD.
In snippet 3, you start both tasks then immediately print CCDD before waiting for them both. This gives you CCDD A B.
I get this error:
D:\pythonstuff\demo.py:28: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
await asyncio.wait([
Waited 1 second!
Waited 5 second!
Time passed: 0hour:0min:5sec
Process finished with exit code 0
When I run the code:
import asyncio
import time
class class1():
async def function_inside_class(self):
await asyncio.sleep(1)
print("Waited 1 second!")
async def function_inside_class2(self):
await asyncio.sleep(5)
print("Waited 5 second!")
def tic():
global _start_time
_start_time = time.time()
def tac():
t_sec = round(time.time() - _start_time)
(t_min, t_sec) = divmod(t_sec,60)
(t_hour,t_min) = divmod(t_min,60)
print('Time passed: {}hour:{}min:{}sec'.format(t_hour,t_min,t_sec))
object = class1()
async def main():
tic()
await asyncio.wait([
object.function_inside_class(),
object.function_inside_class2()
])
tac()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
Are there any good alternatives to asyncio.wait? I don't want a warning in console every time I launch my application.
Edit: I don't want to just hide the error, that's bad practice, and I'm looking for other ways to do the same or a similar thing, not another async library to restore the old functionality.
You can just call it this way as it recommends in the docs here
Example from the docs:
async def foo():
return 42
task = asyncio.create_task(foo())
done, pending = await asyncio.wait({task})
So your code would become:
await asyncio.wait([
asyncio.create_task(object.function_inside_class()),
asyncio.create_task(object.function_inside_class2())
])
I can't figure how to stop loop after one task is finished. In sample when WsServe count to 5 I expect loop to close. But instead stop I got RuntimeError: Cannot close a running event loop
#!/usr/bin/env python
import asyncio
async def rxer():
i=0
while True:
i+=1
print ('Rxer ',i)
await asyncio.sleep(1)
async def WsServe():
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
loop.stop()
loop.close()
loop=asyncio.get_event_loop()
loop.create_task(rxer())
loop.run_until_complete(WsServe())
loop.run_forever()
The error comes from calling loop.close() from inside the loop. You don't need to bother with loop.close(), loop.stop() is quite sufficient to stop the loop. loop.close() is only relevant when you want to ensure that all the resources internally acquired by the loop are released. It is not needed when your process is about to exit anyway, and removing the call to loop.close() indeed eliminates the error.
But also, loop.stop() is incompatible with run_until_complete(). It happens to work in this code because the coroutine returns immediately after calling loop.stop(); if you added e.g. an await asyncio.sleep(1) after loop.stop(), you'd again get a (different) RuntimeError.
To avoid such issues, I suggest that you migrate to the newer asyncio.run API and avoid both run_until_complete and stop. Instead, you can just use an event to terminate the main function, and the loop with it:
# rxer() defined as before
async def WsServe(stop_event):
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
stop_event.set()
await asyncio.sleep(1)
async def main():
asyncio.get_event_loop().create_task(rxer())
stop_event = asyncio.Event()
asyncio.get_event_loop().create_task(WsServe(stop_event))
await stop_event.wait()
asyncio.run(main())
# python 3.6 and older:
#asyncio.get_event_loop().run_until_complete(main())
Check commented lines of your implementation as below:
import asyncio
async def rxer():
i=0
while True:
i+=1
print ('Rxer ',i)
await asyncio.sleep(1)
async def WsServe():
for i in range(5):
print ('WsServe',i)
await asyncio.sleep(1)
print ('Finish')
#loop.stop()
#loop.close()
loop=asyncio.get_event_loop()
loop.create_task(rxer())
loop.run_until_complete(WsServe())
#loop.run_forever()
And here is the output:
Rxer 1
WsServe 0
Rxer 2
WsServe 1
Rxer 3
WsServe 2
Rxer 4
WsServe 3
Rxer 5
WsServe 4
Rxer 6
Finish
I've been using asyncio for a bit but I'm still fairly unfamiliar with it. My current issue is that while trying to wait for a response from a function with asyncio, the waiting (while loop) blocks the function from happening. Here is the code that sums up the problem:
import asyncio
response = 0
async def handle(x):
await asyncio.sleep(0.1)
return x
async def run():
global response
for number in range(1, 21):
response = await handle(number)
print(response)
if response == 10:
await wait_for_next(response)
async def wait_for_next(x):
while response == x:
print('waiting',response,x)
await asyncio.sleep(0.5)
print('done')
tasks = [run()]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
wait_for_next is supposed to wait for the next response, but the while loop blocks the run() function. How could I stop this happening? Should I be using loop.run_in_executor, and if so, how?
(There were a couple of other examples of this I could find, but they were very specific and I didn't understand if our problems/solutions would be the same.)
As already noted, loop stuck because await wait_for_next(response) blocks execution flow until this coroutine wouldn't be finished.
If you want some of your coroutines to be started without blocking execution flow you can start it as asyncio.Task (more about tasks) using ensure_future function:
import asyncio
response = 0
async def handle(x):
await asyncio.sleep(0.1)
return x
async def run():
global response
for number in range(1, 21):
response = await handle(number)
print(response)
if response == 10:
# run wait_for_next "in background" instead of blocking flow:
asyncio.ensure_future(wait_for_next(response))
async def wait_for_next(x):
while response == x:
print('waiting',response,x)
await asyncio.sleep(0.5)
print('done')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
Output:
1
2
3
4
5
6
7
8
9
10
waiting 10 10
11
12
13
14
done
15
16
17
18
19
20
I have 2 functions: The first one, def_a, is an asynchronous function and the second one is def_b which is a regular function and called with the result of def_a as a callback with the add_done_callback function.
My code looks like this:
import asyncio
def def_b(result):
next_number = result.result()
# some work on the next_number
print(next_number + 1)
async def def_a(number):
await some_async_work(number)
return number + 1
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(def_a(1))
task.add_done_callback(def_b)
response = loop.run_until_complete(task)
loop.close()
And it's work perfectly.
The problem began when also the second function, def_b, became asynchronous. Now it looks like this:
async def def_b(result):
next_number = result.result()
# some asynchronous work on the next_number
print(next_number + 1)
But now I can not provide it to the add_done_callback function, because it's not a regular function.
My question is- Is it possible and how can I provide def_b to the add_done_callback function if def_b is asynchronous?
add_done_callback is considered a "low level" interface. When working with coroutines, you can chain them in many ways, for example:
import asyncio
async def my_callback(result):
print("my_callback got:", result)
return "My return value is ignored"
async def coro(number):
await asyncio.sleep(number)
return number + 1
async def add_success_callback(fut, callback):
result = await fut
await callback(result)
return result
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(coro(1))
task = add_success_callback(task, my_callback)
response = loop.run_until_complete(task)
print("response:", response)
loop.close()
Keep in mind add_done_callback will still call the callback if your future raises an exception (but calling result.result() will raise it).
This only works for one future job, if you have multiple async jobs, they will blocks each other, a better way is using asyncio.as_completed() to iterate future list:
import asyncio
async def __after_done_callback(future_result):
# await for something...
pass
async def __future_job(number):
await some_async_work(number)
return number + 1
loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(__future_job(x)) for x in range(100)] # create 100 future jobs
for f in asyncio.as_completed(tasks):
result = await f
await __after_done_callback(result)
loop.close()
You can try the aiodag library. It's a very lightweight wrapper around asyncio that abstracts away some of the async plumbing that you usually have to think about. From this example you won't be able to tell that things are running asynchronously since it's just 1 task that depends on another, but it is all running async.
import asyncio
from aiodag import task
#task
async def def_b(result):
# some asynchronous work on the next_number
print(result + 1)
#task
async def def_a(number):
await asyncio.sleep(number)
return number + 1
async def main():
a = def_a(1)
b = def_b(a) # this makes task b depend on task a
return await b
loop = asyncio.get_event_loop()
asyncio.set_event_loop(loop)
response = loop.run_until_complete(main())