Having read the documents and watched a number of videos, i am testing asyncio as an alternative to threading.
The docs are here:
https://docs.python.org/3/library/asyncio.html
I have constructed the following code with the expectation that it would produce the following.
before the sleep
hello
world
But in fact is produces this (world comes before hello):
before the sleep
world
hello
Here is the code:
import asyncio
import time
def main():
''' main entry point for the program '''
# create the event loop and add to the loop
# or run directly.
asyncio.run(main_async())
return
async def main_async():
''' the main async function '''
await foo()
await bar()
return
async def foo():
print('before the sleep')
await asyncio.sleep(2)
# time.sleep(0)
print('world')
return
async def bar():
print('hello')
await asyncio.sleep(0)
return
if __name__=='__main__':
''' This is executed when run from the command line '''
main()
The main() function calls the async main_async() function which in turn calls both the foo and bar async functions and both of those run the await asyncio.sleep(x) command.
So my question is: why is the hello world comming in the wrong (unexpected) order given that i was expecting world to be printed approximately 2 seconds after hello ?
You awaited foo() immediately, so bar() was never scheduled until foo() had run to completion; the execution of main_async will never do things after an await until the await has completed. If you want to schedule them both and let them interleave, replace:
await foo()
await bar()
with something like:
await asyncio.gather(foo(), bar())
which will convert both awaitables to tasks, scheduling both on the running asyncio event loop, then wait for both tasks to run to completion. With both scheduled at once, when one blocks on an await (and only await-based blocks, because only await yields control back to the event loop), the other will be allowed to run (and control can only return to the other task when the now running task awaits or finishes).
Basically, you have to remember that asyncio is cooperative multitasking. If you're only executing one task, and that task performs an await, there is nothing else to schedule, so nothing else runs until that await completes. If you block by any means other than an await, you still hold the event loop, and nothing else will get a chance to run, even if it's ready to go. So to gain any benefit from asyncio you need to be careful to:
Ensure other tasks are launched in time to occupy the event loop while the original task(s) are blocking on await.
Ensure you only block via await, so you don't monopolize the event loop unnecessarily.
Related
I need to have two function in my Python 3.11 code.
One function must be sync, it retrive some data from a local machine so i need to wait to finish.
Another function must be async, it get the data from the first function and send to the server. Since i don't know how many time can be (5 seconds to 30 seconds) this function must doesn't interrupt the first one
Pratically, the second function start always when the first finish but the first always start and don't care about the second one. This code run H24
My attempt:
import time
import asyncio
async def task1():
print("Recover data... waiting")
time.sleep(3)
print("End data recover")
return "slow"
async def task2(p):
print("I'm so" + p)
time.sleep(10)
print("END--->")
async def main():
while True:
print("create task1 and wait to finish")
x = await task1()
print("create task2 and not wait to finishing")
asyncio.create_task(task2(x))
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.create_task(main())
loop.run_forever()
I dont' need to use asyncio like a requirement, i just want to meet the goal without run out all the memory of the machine. Thanks
basically, "yes". Like in: that is the way you do it. If you need some task to complete before goind on with code in a given place, the thing to do is to wait for that data - if it is in an async func, then you use the await keyword, just as depicted in the code above.
If there are other tasks in parallel to calling main, then it would be nice if the code in task1 would return execution to the asyncio.loop while it waits for its result - That code could run in another thread, with the use of await asyncio.run_in_executor(sync_call, args) or simply await asyncio.sleep(...)`
I have a asyncio running loop, and from the coroutine I'm calling a sync function, is there any way we can call and get result from an async function in a sync function
tried below code, it is not working
want to print output of hel() in i() without changing i() to async function
is it possible, if yes how?
import asyncio
async def hel():
return 4
def i():
loop = asyncio.get_running_loop()
x = asyncio.run_coroutine_threadsafe(hel(), loop) ## need to change
y = x.result() ## this lines
print(y)
async def h():
i()
asyncio.run(h())
This is one of the most commonly asked type of question here. The tools to do this are in the standard library and require only a few lines of setup code. However, the result is not 100% robust and needs to be used with care. This is probably why it's not already a high-level function.
The basic problem with running an async function from a sync function is that async functions contain await expressions. Await expressions pause the execution of the current task and allow the event loop to run other tasks. Therefore async functions (coroutines) have special properties that allow them to yield control and resume again where they left off. Sync functions cannot do this. So when your sync function calls an async function and that function encounters an await expression, what is supposed to happen? The sync function has no ability to yield and resume.
A simple solution is to run the async function in another thread, with its own event loop. The calling thread blocks until the result is available. The async function behaves like a normal function, returning a value. The downside is that the async function now runs in another thread, which can cause all the well-known problems that come with threaded programming. For many cases this may not be an issue.
This can be set up as follows. This is a complete script that can be imported anywhere in an application. The test code that runs in the if __name__ == "__main__" block is almost the same as the code in the original question.
The thread is lazily initialized so it doesn't get created until it's used. It's a daemon thread so it will not keep your program from exiting.
The solution doesn't care if there is a running event loop in the main thread.
import asyncio
import threading
_loop = asyncio.new_event_loop()
_thr = threading.Thread(target=_loop.run_forever, name="Async Runner",
daemon=True)
# This will block the calling thread until the coroutine is finished.
# Any exception that occurs in the coroutine is raised in the caller
def run_async(coro): # coro is a couroutine, see example
if not _thr.is_alive():
_thr.start()
future = asyncio.run_coroutine_threadsafe(coro, _loop)
return future.result()
if __name__ == "__main__":
async def hel():
await asyncio.sleep(0.1)
print("Running in thread", threading.current_thread())
return 4
def i():
y = run_async(hel())
print("Answer", y, threading.current_thread())
async def h():
i()
asyncio.run(h())
Output:
Running in thread <Thread(Async Runner, started daemon 28816)>
Answer 4 <_MainThread(MainThread, started 22100)>
In order to call an async function from a sync method, you need to use asyncio.run, however this should be the single entry point of an async program so asyncio makes sure that you don't do this more than once per program, so you can't do that.
That being said, this project https://github.com/erdewit/nest_asyncio patches the asyncio event loop to do that, so after using it you should be able to just call asyncio.run in your sync function.
I have an application which already runs infinitely with asyncio event loop run forever and also I need to run a specific function every 10 seconds.
def do_something():
pass
a = asyncio.get_event_loop()
a.run_forever()
I would like to call the function do_something every 10 seconds. How to achieve this without replacing asynctio event loop with while loop ?
Edited:
I can achieve this with the below code
def do_something():
pass
while True:
time.sleep(10)
do_something()
But I dont want to use while loop to run infinitely in my application instead I would like to go with asyncio run_forever(). So how to call the same function every 10 seconds with asyncio ? is there any scheduler like which will not block my ongoing work ?
asyncio does not ship with a builtin scheduler, but it is easy enough to build your own. Simply combine a while loop with asyncio.sleep to run code every few seconds.
async def every(__seconds: float, func, *args, **kwargs):
while True:
func(*args, **kwargs)
await asyncio.sleep(__seconds)
a = asyncio.get_event_loop()
a.create_task(every(1, print, "Hello World"))
...
a.run_forever()
Note that the design has to be slightly different if func is itself a coroutine or a long-running subroutine. In the former case use await func(...) and in the latter case use asyncio's thread capabilities.
You can achieve it with
async def do_something():
while True:
await asyncio.wait(10)
...rest of code...
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(do_something())
loop.run_forever()
In python, there are 3 main types awaitable objects: coroutines, Tasks, and Futures.
I can await a coroutine, and also a tasks.
Awaiting a coroutine
import asyncio
async def nested():
return 42
async def main():
print(await nested()) # will print "42".
asyncio.run(main())
Awaiting a task
import asyncio
async def nested():
return 42
async def main():
task = asyncio.create_task(nested())
await task
asyncio.run(main())
What is the value of wrapping the coroutine in a task in the first place? It looks like they do the same thing.
When would I need to use a task vs a coroutine?
Coroutine is just a function that runs in the context of current awaitable. It can yield execution to the event loop on behalf of the caller (the one who calls await). Think of a function that is allowed to pause it's thread. You can call one coroutine from another, but they still share the same thread.
Task, on other hand, immediately posts a separate job to an event loop. The task itself is a handle to that job. You may await a task, but it can run on itself just fine in "parallel" — in single threaded context this means that task can run while other josb are yielding (e.g. waiting for the I/O). Task may complete even before you call await.
Example without tasks:
job_1 = sleep(5)
job_2 = sleep(2)
# will sleep for 5 seconds
await job_1
# will sleep for another 2 seconds
await job_2
Example with tasks:
job_1 = sleep(5)
job_2 = asyncio.create_task(sleep(2))
# will sleep for 5 seconds
await job_1
# by this time, job_2 is complete
# because previous job has yielded at some point, allowing other jobs to run
# thus await takes no time
await job_2
In this case there's no real difference: by awaiting the coroutine it's going to get scheduled as part of the task it's part of. However that means it's driven by its parent.
By wrapping a coroutine in a task, it gets independently scheduled on the event loop, meaning it is not driven by the containing task anymore (it has its own lifecycle) and it can be interacted with more richly (e.g. cancelled or have callbacks added to it).
Think "function" versus "thread", really. A coroutine is just a function which can be suspended (if it awaits stuff), but it still only exists within the lexical and dynamic context of its caller. A task is freed from that context, it makes the wrapped coroutine live its own life in the same way a thread makes the wrapped function (target) live its own life.
Creating a Task schedules the passed coroutine to be run on an event loop. You can use the Task to cancel the underlying coroutine.
In the following code:
import asyncio
async def task_func():
print('in task_func')
return 'the result'
async def main(loop):
print('creating task')
task = loop.create_task(task_func())
print('waiting for {!r}'.format(task))
await asyncio.sleep(2)
return_value = await task
print('task completed {!r}'.format(task))
print('return value: {!r}'.format(return_value))
event_loop = asyncio.new_event_loop()
try:
event_loop.run_until_complete(main(event_loop))
finally:
event_loop.close()
When I execute the code, the result is the following:
creating task
waiting for `<Task pending coro=<task_func() running at <ipython-input-29-797f29858344>:1>>`
in task_func
task completed `<Task finished coro=<task_func() done, defined at <ipython-input-29-797f29858344>:1> result='the result'>`
return value: 'the result'
But I don't understand when the code you set at loop.create_task(task_func()) is executed. Specifically, I assumed when you add a task to the event loop, it is executed soon, so I thought in task_func is printed before waiting for <Task....
Then I found it is always executed after the waiting for <Task..., so I added await asyncio.sleep(2), but only found that the in task_func is printed before the finish of 2 seconds.
I also added task_func_2() which is practically the same as task_func() and create its task below task = loop.create_task(task_func()) but do NOT add return_value_2 = await task2, so the await does not execute the task (otherwise the task_func_2() is never executed).
So now I got confuesed. When is the task is executed after it is added to the event loop in loop.create_task()?
Specifically, I assumed when you add a task to the event loop, it is executed soon, so I thought in task_func is printed before waiting for <Task....
"Executed soon" doesn't mean executed right away. Instead, you can think of it as "executed the first chance we get," we being the event loop. Since print immediately follows the call to create_task, at that point the event loop hasn't yet had a chance to run at all. To give event loop a chance to run, you must return to the event loop, either by returning from the current coroutine, or by awaiting something that blocks.
When you await a blocking coroutine such as asyncio.sleep(), the coroutine will temporarily suspend itself and relinquish control to the event loop. The event loop will look at what else there is to do before the sleep elapses and will find the tasks scheduled using create_task in its run queue. This is why task_func and task_func_2 are executed when the main coroutine awaits the sleep - but not before that, and regardless of whether you await them in particular or something else that blocks.
awaiting a coroutine such as task_func means requesting its result then and there, and being prepared to wait for it if the coroutine suspends. (Waiting on something that suspended automatically defers execution to the event loop, allowing other coroutines to make progress.) Although the implementation differs, an await is conceptually similar to joining a thread.