I'm trying to create a function performing some asynchronous operations using asyncio, users of this function should not need to know that asyncio is involved under the hood.
I'm having a very hard time understanding how this shall be done with the asyncio API as most functions seem to operate under some global loop-variable accessed with get_event_loop and calls to this are affected by the global state inside this loop.
I have four examples here where two (foo1 and foo3) seem to be reasonable use cases but they all show very strange behaviors:
async def bar(loop):
# Disregard how simple this is, it's just for example
s = await asyncio.create_subprocess_exec("ls", loop=loop)
def foo1():
# Example1: Just use get_event_loop
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000))
# On exit this is written to stderr:
# Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
# Traceback (most recent call last):
# File "/usr/lib/python3.5/asyncio/base_events.py", line 510, in __del__
# File "/usr/lib/python3.5/asyncio/unix_events.py", line 65, in close
# File "/usr/lib/python3.5/asyncio/unix_events.py", line 146, in remove_signal_handler
# File "/usr/lib/python3.5/signal.py", line 47, in signal
# TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object
def foo2():
# Example2: Use get_event_loop and close it when done
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000)) # RuntimeError: Event loop is closed --- if foo2() is called twice
loop.close()
def foo3():
# Example3: Always use new_event_loop
loop = asyncio.new_event_loop()
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000)) #RuntimeError: Cannot add child handler, the child watcher does not have a loop attached
loop.close()
def foo4():
# Example4: Same as foo3 but also set_event_loop to the newly created one
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop) # Polutes global event loop, callers of foo4 does not expect this.
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000)) # OK
loop.close()
None of these functions work and i don't see any other obvious way to do it, how is asyncio supposed to be used? It's seems like it's only designed to be used under the assumption that the entry point of the application is the only place where you can create and close the global loop. Do i have to fiddle around with event loop policies?
foo3 seems like the correct solution but i get an error even though i explicitly pass along loop, because deep down inside create_subprocess_exec it is using the current policy to get a new loop which is None, is this a bug in asyncio subprocess?
I'm using Python 3.5.3 on Ubuntu.
foo1 error happens because you didn't close event loop, see this issue.
foo2 because you can't reuse closed event loop.
foo3 because you didn't set new event loop as global.
foo4 is almost what you want, all you left to do is store old event loop and set it back as global after bar executed:
import asyncio
async def bar():
# After you set new event loop global,
# there's no need to pass loop as param to bar or anywhere else.
process = await asyncio.create_subprocess_exec("ls")
await process.communicate()
def sync_exec(coro): # foo5
old_loop = asyncio.get_event_loop()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(coro)
finally:
loop.close()
asyncio.set_event_loop(old_loop)
sync_exec(asyncio.wait_for(bar(), 1000))
One more important thing: it's not clear why you want to hide using of asyncio behind some sync functions, but usually it's bad idea. Whole thing about one global event loop is to allow user to run different concurrent jobs in this single event loop. You're trying to take away this possibility. I think you should reconsider this decision.
Upgrade to Python 3.6, then foo1() will work, without the need to explicitly close the default event loop.
Not the answer i was hoping for as we only use 3.5 :(
Related
I have a asyncio running loop, and from the coroutine I'm calling a sync function, is there any way we can call and get result from an async function in a sync function
tried below code, it is not working
want to print output of hel() in i() without changing i() to async function
is it possible, if yes how?
import asyncio
async def hel():
return 4
def i():
loop = asyncio.get_running_loop()
x = asyncio.run_coroutine_threadsafe(hel(), loop) ## need to change
y = x.result() ## this lines
print(y)
async def h():
i()
asyncio.run(h())
This is one of the most commonly asked type of question here. The tools to do this are in the standard library and require only a few lines of setup code. However, the result is not 100% robust and needs to be used with care. This is probably why it's not already a high-level function.
The basic problem with running an async function from a sync function is that async functions contain await expressions. Await expressions pause the execution of the current task and allow the event loop to run other tasks. Therefore async functions (coroutines) have special properties that allow them to yield control and resume again where they left off. Sync functions cannot do this. So when your sync function calls an async function and that function encounters an await expression, what is supposed to happen? The sync function has no ability to yield and resume.
A simple solution is to run the async function in another thread, with its own event loop. The calling thread blocks until the result is available. The async function behaves like a normal function, returning a value. The downside is that the async function now runs in another thread, which can cause all the well-known problems that come with threaded programming. For many cases this may not be an issue.
This can be set up as follows. This is a complete script that can be imported anywhere in an application. The test code that runs in the if __name__ == "__main__" block is almost the same as the code in the original question.
The thread is lazily initialized so it doesn't get created until it's used. It's a daemon thread so it will not keep your program from exiting.
The solution doesn't care if there is a running event loop in the main thread.
import asyncio
import threading
_loop = asyncio.new_event_loop()
_thr = threading.Thread(target=_loop.run_forever, name="Async Runner",
daemon=True)
# This will block the calling thread until the coroutine is finished.
# Any exception that occurs in the coroutine is raised in the caller
def run_async(coro): # coro is a couroutine, see example
if not _thr.is_alive():
_thr.start()
future = asyncio.run_coroutine_threadsafe(coro, _loop)
return future.result()
if __name__ == "__main__":
async def hel():
await asyncio.sleep(0.1)
print("Running in thread", threading.current_thread())
return 4
def i():
y = run_async(hel())
print("Answer", y, threading.current_thread())
async def h():
i()
asyncio.run(h())
Output:
Running in thread <Thread(Async Runner, started daemon 28816)>
Answer 4 <_MainThread(MainThread, started 22100)>
In order to call an async function from a sync method, you need to use asyncio.run, however this should be the single entry point of an async program so asyncio makes sure that you don't do this more than once per program, so you can't do that.
That being said, this project https://github.com/erdewit/nest_asyncio patches the asyncio event loop to do that, so after using it you should be able to just call asyncio.run in your sync function.
My question is more or less like this, which is really an X-Y problem leading back to this. This is, however, not a duplicate, because my use case is slightly different and the linked threads don't answer my question.
I am porting a set of synchronous programs from Java to Python. These programs interact with an asynchronous library. In Java, I could block and wait for this library's asynchronous functions to return a value and then do things with that value.
Here's a code sample to illustrate the problem.
def do_work_sync_1(arg1, arg2, arg3):
# won't even run because await has to be called from an async function
value = await do_work_async(arg1, arg2, arg3)
def do_work_sync_2(arg1, arg2, arg3):
# throws "Loop already running" error because the async library referenced in do_work_async is already using my event loop
event_loop = asyncio.get_event_loop()
event_loop.run_until_complete(do_work_async(arg1, arg2, arg3))
def do_work_sync_3(arg1, arg2, arg3):
# throws "got Future attached to a different loop" because the do_work_async refers back to the asynchronous library, which is stubbornly attached to my main loop
thread_pool = ThreadPoolExecutor()
future = thread_pool.submit(asyncio.run, do_work_async(arg1, arg2, arg3)
result = future.result()
def do_work_sync_4(arg1, arg2, arg3):
# just hangs forever
event_loop = asyncio.get_event_loop()
future = asyncio.run_coroutine_threadsafe(do_work_async(arg1, arg2, arg3), event_loop)
return_value = future.result()
async def do_work_async(arg1, arg2, arg3):
value_1 = await async_lib.do_something(arg1)
value_2 = await async_lib.do_something_else(arg2, arg3)
return value_1 + value_2
Python appears to be trying very hard to keep me from blocking anything, anywhere. await can only be used from async def functions, which in their turn must be awaited. There doesn't seem to be a built-in way to keep async def/await from spreading through my code like a virus.
Tasks and Futures don't have any built-in blocking or wait_until_complete mechanisms unless I want to loop on Task.done(), which seems really bad.
I tried asyncio.get_event_loop().run_until_complete(), but that produces an error: This event loop is already running. Apparently I'm not supposed to do that for anything except main().
The second linked question above suggests using a separate thread and wrapping the async function in that. I tested this with a few simple functions and it seems to work as a general concept. The problem here is that my asynchronous library keeps a reference to the main thread's event loop and throws an error when I try to refer to it from the new thread: got Future <Future pending> attached to a different loop.
I considered moving all references to the asynchronous library into a separate thread, but I realized that I still can't block in the new thread, and I'd have to create a third thread for blocking calls, which would bring me back to the Future attached to a different loop error.
I'm pretty much out of ideas here. Is there a way to block and wait for an async function to return, or am I really being forced to convert my entire program to async/await? (If it's the latter, an explanation would be nice. I don't get it.)
It took me some time, but finally I've found the actual question 😇
Is there a way to block and wait for an async function to return, or am I really being forced to convert my entire program to async/await?
There is a high-level function asyncio.run(). It does three things:
create new event loop
run your async function in that event loop
wait for any unfinished tasks and close the loop
Its source code is here: https://github.com/python/cpython/blob/3221a63c69268a9362802371a616f49d522a5c4f/Lib/asyncio/runners.py#L8 You see it uses loop.run_until_complete(main) under the hood.
If you are writing completely asynchronous code, you are supposed to call asyncio.run() somewhere at the end of your main() function, I guess. But that doesn't have to be the case. You can run it wherever you want, as many times you want. Caveats:
in given thread, at one time, there can be only one running event loop
do not run it from async def function, because, obviously, you have already one event loop running, so you can just call that function using await instead
Example:
import asyncio
async def something_async():
print('something_async start')
await asyncio.sleep(1)
print('something_async done')
for i in range(3):
asyncio.run(something_async())
You can have multiple threads with their own event loop:
import asyncio
import threading
async def something_async():
print('something_async start in thread:', threading.current_thread())
await asyncio.sleep(1)
print('something_async done in thread:', threading.current_thread())
def main():
t1 = threading.Thread(target=asyncio.run, args=(something_async(), ))
t2 = threading.Thread(target=asyncio.run, args=(something_async(), ))
t1.start()
t2.start()
t1.join()
t2.join()
if __name__ == '__main__':
main()
If you encounter this error: Future attached to a different loop That may mean two tings:
you are using resources tied to another event loop than you are running right now
you have created some resource before starting an event loop - it uses a "default event loop" in that case - but when you run asyncio.run(), you start a different loop. I've encountered this before: asyncio.Semaphore RuntimeError: Task got Future attached to a different loop
You need to use Python version at least 3.5.3 - explanation here.
I am creating a cryptocurrency exchange API client using Python3.5 and Tkinter. I have several displays that I want to update asynchronously every 10 seconds. I am able to update the displays every 10 seconds using Tk.after() like in this example
def updateLoans():
offers = dd.loanOffers()
demands = dd.loanDemands()
w.LoanOfferView.delete(1.0, END)
w.LoanDemandView.delete(1.0, END)
w.LoanOfferView.insert(END, offers)
w.LoanDemandView.insert(END, demands)
print('loans refreshed')
root.after(10000, updateLoans)
In order for the after method to continue to update continuously every 10 seconds the function updateLoans() needs to be passed as a callable into after() inside of the function.
Now the part that is stumping me, when I make this function asynchronous with python's new async and await keywords
async def updateLoans():
offers = await dd.loanOffers()
demands = await dd.loanDemands()
w.LoanOfferView.delete(1.0, END)
w.LoanDemandView.delete(1.0, END)
w.LoanOfferView.insert(END, offers)
w.LoanDemandView.insert(END, demands)
print('loans refreshed')
root.after(10000, updateLoans)
The problem here is that I can not await a callable inside of the parameters for the after method. So I get a runtime warning. RuntimeWarning: coroutine 'updateLoans' was never awaited.
My initial function call IS placed inside of an event loop.
loop = asyncio.get_event_loop()
loop.run_until_complete(updateLoans())
loop.close()
The display populates just fine initially but never updates.
How can I use Tk.after to continuously update a tkinter display asynchronously?
tk.after accepts a normal function, not a coroutine. To run the coroutine to completion, you can use run_until_complete, just as you did the first time:
loop = asyncio.get_event_loop()
root.after(10000, lambda: loop.run_until_complete(updateLoans()))
Also, don't call loop.close(), since you'll need the loop again.
The above quick fix will work fine for many use cases. The fact is, however, that it will render the GUI completely unresponsive if updateLoans() takes a long time due to slow network or a problem with the remote service. A good GUI app will want to avoid this.
While Tkinter and asyncio cannot share an event loop yet, it is perfectly possible to run the asyncio event loop in a separate thread. The main thread then runs the GUI, while a dedicated asyncio thread runs all asyncio coroutines. When the event loop needs to notify the GUI to refresh something, it can use a queue as shown here. On the other hand, if the GUI needs to tell the event loop to do something, it can call call_soon_threadsafe or run_coroutine_threadsafe.
Example code (untested):
gui_queue = queue.Queue()
async def updateLoans():
while True:
offers = await dd.loanOffers()
demands = await dd.loanDemands()
print('loans obtained')
gui_queue.put(lambda: updateLoansGui(offers, demands))
await asyncio.sleep(10)
def updateLoansGui(offers, demands):
w.LoanOfferView.delete(1.0, END)
w.LoanDemandView.delete(1.0, END)
w.LoanOfferView.insert(END, offers)
w.LoanDemandView.insert(END, demands)
print('loans GUI refreshed')
# http://effbot.org/zone/tkinter-threads.htm
def periodicGuiUpdate():
while True:
try:
fn = gui_queue.get_nowait()
except queue.Empty:
break
fn()
root.after(100, periodicGuiUpdate)
# Run the asyncio event loop in a worker thread.
def start_loop():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.create_task(updateLoans())
loop.run_forever()
threading.Thread(target=start_loop).start()
# Run the GUI main loop in the main thread.
periodicGuiUpdate()
root.mainloop()
# To stop the event loop, call loop.call_soon_threadsafe(loop.stop).
# To start a coroutine from the GUI, call asyncio.run_coroutine_threadsafe.
I have an app which adds coroutines to an already-running event loop.
The arguments for these coroutines depend on I/O and are not available when I initially start the event loop - with loop.run_forever(), so I add the tasks later. To demonstrate the phenomenon, here is some example code:
import asyncio
from threading import Thread
from time import sleep
loop = asyncio.new_event_loop()
def foo():
loop.run_forever()
async def bar(s):
while True:
await asyncio.sleep(1)
print("s")
#loop.create_task(bar("A task created before thread created & before loop started"))
t = Thread(target=foo)
t.start()
sleep(1)
loop.create_task(bar("secondary task"))
The strange behaviour is that everything works as expected when there is at least one task in the loop when invoking loop.run_forever(). i.e. when the commented line is not commented out.
But when it is commented out, as shown above, nothing is printed and it appears I am unable to add a task to the event_loop. Should I avoid invoking run_forever() without adding a single task? I don't see why this should be a problem. Adding tasks to an event_loop after it is running is standard, why should the empty case be an issue?
Adding tasks to an event_loop after it is running is standard, why should the empty case be an issue?
Because you're supposed to add tasks from the thread running the event loop. In general one should not mix threads and asyncio, except through APIs designed for that purpose, such as loop.run_in_executor.
If you understand this and still have good reason to add tasks from a separate thread, use asyncio.run_coroutine_threadsafe. Change loop.create_task(bar(...)) to:
asyncio.run_coroutine_threadsafe(bar("in loop"), loop=loop)
run_coroutine_threadsafe accesses the event loop in a thread-safe manner, and also ensures that the event loop wakes up to notice the new task, even if it otherwise has nothing to do and is just waiting for IO/timeouts.
Adding another task beforehand only appeared to work because bar happens to be an infinite coroutine that makes the event loop wake up every second. Once the event loop wakes up for any reason, it executes all runnable tasks regardless of which thread added them. It would be a really bad idea to rely on this, though, because loop.create_task is not thread-safe, so there could be any number of race conditions if it executed in parallel with a running event loop.
Because loop.create_task is not thread safe, and if you set loop._debug = True, you should see the error as
Traceback (most recent call last):
File "test.py", line 23, in <module>
loop.create_task(bar("secondary task"))
File "/Users/soulomoon/.pyenv/versions/3.6.3/lib/python3.6/asyncio/base_events.py", line 284, in create_task
task = tasks.Task(coro, loop=self)
File "/Users/soulomoon/.pyenv/versions/3.6.3/lib/python3.6/asyncio/base_events.py", line 576, in call_soon
self._check_thread()
File "/Users/soulomoon/.pyenv/versions/3.6.3/lib/python3.6/asyncio/base_events.py", line 615, in _check_thread
"Non-thread-safe operation invoked on an event loop other "
RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one
Hello I fairly new to Python and I am trying to convert an existing application I have on Flask into Quart (https://gitlab.com/pgjones/quart) which is supposed to be built on top of asyncio, so I can use Goblin OGM to interact with JanusGraph or TinkerPop. According to the examples I found on Goblin I need to obtain an event loop to run the commands asynchronously.
>>> import asyncio
>>> from goblin import Goblin
>>> loop = asyncio.get_event_loop()
>>> app = loop.run_until_complete(
... Goblin.open(loop))
>>> app.register(Person, Knows)
However I can't find a way to obtain the event loop from Quart even though it is build on top of asyncio.
Does anyone know how I can get that ? Any help will be highly appreciated.
TL;DR To obtain the event loop, call asyncio.get_event_loop().
In an asyncio-based application, the event loop is typically not owned by Quart or any other protocol/application level component, it is provided by asyncio or possibly an accelerator like uvloop. The event loop is obtained by calling asyncio.get_event_loop(), and sometimes set with asyncio.set_event_loop().
This is what quart's app.run() uses to run the application, which means it works with the default event loop created by asyncio for the main thread. In your case you could simply call quart's run() after registering Goblin:
loop = asyncio.get_event_loop()
goblin_app = loop.run_until_complete(Goblin.open(loop))
goblin_app.register(Person, Knows)
quart_app = Quart(...)
# ... #app.route, etc
# now they both run in the same event loop
quart_app.run()
The above should answer the question in the practical sense. But that approach wouldn't work if more than one component insisted on having their own run() method that spins the event loop - since app.run() doesn't return, you can only invoke one such function in a thread.
If you look more closely, though, that is not really the case with quart either. While Quart examples do use app.run() to serve the application, if you take a look at the implementation of app.run(), you will see that it calls the convenience function run_app(), which trivially creates a server and spins up the main loop forever:
def run_app(...):
loop = asyncio.get_event_loop()
# ...
create_server = loop.create_server(
lambda: Server(app, loop, ...), host, port, ...)
server = loop.run_until_complete(create_server)
# ...
loop.run_forever()
If you need to control how the event loop is actually run, you can always do it yourself:
# obtain the event loop from asyncio
loop = asyncio.get_event_loop()
# hook Goblin to the loop
goblin_app = loop.run_until_complete(Goblin.open(loop))
goblin_app.register(Person, Knows)
# hook Quart to the loop
quart_server = loop.run_until_complete(loop.create_server(
lambda: quart.serving.Server(quart_app, loop), host, port))
# actually run the loop (and the program)
try:
loop.run_forever()
except KeyboardInterrupt: # pragma: no cover
pass
finally:
quart_server.close()
loop.run_until_complete(quart_server.wait_closed())
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()