Hello I fairly new to Python and I am trying to convert an existing application I have on Flask into Quart (https://gitlab.com/pgjones/quart) which is supposed to be built on top of asyncio, so I can use Goblin OGM to interact with JanusGraph or TinkerPop. According to the examples I found on Goblin I need to obtain an event loop to run the commands asynchronously.
>>> import asyncio
>>> from goblin import Goblin
>>> loop = asyncio.get_event_loop()
>>> app = loop.run_until_complete(
... Goblin.open(loop))
>>> app.register(Person, Knows)
However I can't find a way to obtain the event loop from Quart even though it is build on top of asyncio.
Does anyone know how I can get that ? Any help will be highly appreciated.
TL;DR To obtain the event loop, call asyncio.get_event_loop().
In an asyncio-based application, the event loop is typically not owned by Quart or any other protocol/application level component, it is provided by asyncio or possibly an accelerator like uvloop. The event loop is obtained by calling asyncio.get_event_loop(), and sometimes set with asyncio.set_event_loop().
This is what quart's app.run() uses to run the application, which means it works with the default event loop created by asyncio for the main thread. In your case you could simply call quart's run() after registering Goblin:
loop = asyncio.get_event_loop()
goblin_app = loop.run_until_complete(Goblin.open(loop))
goblin_app.register(Person, Knows)
quart_app = Quart(...)
# ... #app.route, etc
# now they both run in the same event loop
quart_app.run()
The above should answer the question in the practical sense. But that approach wouldn't work if more than one component insisted on having their own run() method that spins the event loop - since app.run() doesn't return, you can only invoke one such function in a thread.
If you look more closely, though, that is not really the case with quart either. While Quart examples do use app.run() to serve the application, if you take a look at the implementation of app.run(), you will see that it calls the convenience function run_app(), which trivially creates a server and spins up the main loop forever:
def run_app(...):
loop = asyncio.get_event_loop()
# ...
create_server = loop.create_server(
lambda: Server(app, loop, ...), host, port, ...)
server = loop.run_until_complete(create_server)
# ...
loop.run_forever()
If you need to control how the event loop is actually run, you can always do it yourself:
# obtain the event loop from asyncio
loop = asyncio.get_event_loop()
# hook Goblin to the loop
goblin_app = loop.run_until_complete(Goblin.open(loop))
goblin_app.register(Person, Knows)
# hook Quart to the loop
quart_server = loop.run_until_complete(loop.create_server(
lambda: quart.serving.Server(quart_app, loop), host, port))
# actually run the loop (and the program)
try:
loop.run_forever()
except KeyboardInterrupt: # pragma: no cover
pass
finally:
quart_server.close()
loop.run_until_complete(quart_server.wait_closed())
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()
Related
I am learning about asyncion in Python. There is one confusion that I cannot wrap my head around.
Suppose I have 2 Python scripts: dummy1.py and dummy2.py. In the first script, I have my code written like the following:
loop = asyncio.get_event_loop()
loop.create_task(a_task)
The first script will be imported into the second script, and in the second script, I arrange my code like the following:
loop = asyncio.get_event_loop()
loop.run_forever()
Are there 2 different event loop created? Thanks for your time guys!
The main thing going on with your example is that you only ever started one loop. And when you got the event loop the second time, get_event_loop() is still pointing to the same loop. The loop still has a pending task as well that will not start until the event loop is running.
The important thing to note about this, is that you have one thread. That threads execution can create tasks, save event loops to variables etc., but once you start the event loop, that thread is now working on that event loop. The code wont continue past the line where you started the loop until it ends. To have multiple event loops you would essentially need multiple threads.
You can see this playout with this example:
test1.py
import asyncio
async def a_task():
while True:
await asyncio.sleep(5)
print('a_task did something')
loop = asyncio.get_event_loop()
print('Got Event loop in test1.py')
loop.create_task(a_task())
print('Created task in test1.py')
test2.py
import asyncio
import test1
loop = asyncio.get_event_loop()
print('Got event loop in test2.py')
print('Starting Event loop in test2.py')
loop.run_forever()
print('Event loop ended')
Output
Imagine an asynchronous aiohttp web application that is supported by a Postgresql database connected via asyncpg and does no other I/O. How can I have a middle-layer hosting the application logic, that is not async? (I know I can simply make everything async -- but imagine my app to have massive application logic, only bound by database I/O, and I cannot touch everything of it).
Pseudo code:
async def handler(request):
# call into layers over layers of application code, that simply emits SQL
...
def application_logic():
...
# This doesn't work, obviously, as await is a syntax
# error inside synchronous code.
data = await asyncpg_conn.execute("SQL")
...
# What I want is this:
data = asyncpg_facade.execute("SQL")
...
How can a synchronous façade over asyncpg be built, that allows the application logic to make database calls? The recipes floating around like using async.run() or asyncio.run_coroutine_threadsafe() etc. do not work in this case, as we're coming from an already asynchronous context. I'd assume this cannot be impossible, as there already is an event loop that could in principle run the asyncpg coroutine.
Bonus question: what is the design rationale of making await inside sync a syntax error? Wouldn't it be pretty useful to allow await from any context that originated from a coroutine, so we'd have simple means to decompose an application in functional building blocks?
EDIT Extra bonus: beyond Paul's very good answer, that stays inside the "safe zone", I'd be interested in solutions that avoid blocking the main thread (leading to something more gevent-ish). See also my comment on Paul's answer ...
You need to create a secondary thread where you run your async code. You initialize the secondary thread with its own event loop, which runs forever. Execute each async function by calling run_coroutine_threadsafe(), and calling result() on the returned object. That's an instance of concurrent.futures.Future, and its result() method doesn't return until the coroutine's result is ready from the secondary thread.
Your main thread is then, in effect, calling each async function as if it were a sync function. The main thread doesn't proceed until each function call is finished. BTW it doesn't matter if your sync function is actually running in an event loop context or not.
The calls to result() will, of course, block the main thread's event loop. That can't be avoided if you want to get the effect of running an async function from sync code.
Needless to say, this is an ugly thing to do and it's suggestive of the wrong program structure. But you're trying to convert a legacy program, and it may help with that.
import asyncio
import threading
from datetime import datetime
def main():
def thr(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
loop = asyncio.new_event_loop()
t = threading.Thread(target=thr, args=(loop, ), daemon=True)
t.start()
print("Hello", datetime.now())
t1 = asyncio.run_coroutine_threadsafe(f1(1.0), loop).result()
t2 = asyncio.run_coroutine_threadsafe(f1(2.0), loop).result()
print(t1, t2)
if __name__ == "__main__":
main()
>>> Hello 2021-10-26 20:37:00.454577
>>> Hello 1.0 2021-10-26 20:37:01.464127
>>> Hello 2.0 2021-10-26 20:37:03.468691
>>> 1.0 2.0
I am trying to learn to use asyncio in Python to optimize scripts.
My example returns a coroutine was never awaited warning, can you help to understand and find how to solve it?
import time
import datetime
import random
import asyncio
import aiohttp
import requests
def requete_bloquante(num):
print(f'Get {num}')
uid = requests.get("https://httpbin.org/uuid").json()['uuid']
print(f"Res {num}: {uid}")
def faire_toutes_les_requetes():
for x in range(10):
requete_bloquante(x)
print("Bloquant : ")
start = datetime.datetime.now()
faire_toutes_les_requetes()
exec_time = (datetime.datetime.now() - start).seconds
print(f"Pour faire 10 requêtes, ça prend {exec_time}s\n")
async def requete_sans_bloquer(num, session):
print(f'Get {num}')
async with session.get("https://httpbin.org/uuid") as response:
uid = (await response.json()['uuid'])
print(f"Res {num}: {uid}")
async def faire_toutes_les_requetes_sans_bloquer():
loop = asyncio.get_event_loop()
with aiohttp.ClientSession() as session:
futures = [requete_sans_bloquer(x, session) for x in range(10)]
loop.run_until_complete(asyncio.gather(*futures))
loop.close()
print("Fin de la boucle !")
print("Non bloquant : ")
start = datetime.datetime.now()
faire_toutes_les_requetes_sans_bloquer()
exec_time = (datetime.datetime.now() - start).seconds
print(f"Pour faire 10 requêtes, ça prend {exec_time}s\n")
The first classic part of the code runs correctly, but the second half only produces:
synchronicite.py:43: RuntimeWarning: coroutine 'faire_toutes_les_requetes_sans_bloquer' was never awaited
You made faire_toutes_les_requetes_sans_bloquer an awaitable function, a coroutine, by using async def.
When you call an awaitable function, you create a new coroutine object. The code inside the function won't run until you then await on the function or run it as a task:
>>> async def foo():
... print("Running the foo coroutine")
...
>>> foo()
<coroutine object foo at 0x10b186348>
>>> import asyncio
>>> asyncio.run(foo())
Running the foo coroutine
You want to keep that function synchronous, because you don't start the loop until inside that function:
def faire_toutes_les_requetes_sans_bloquer():
loop = asyncio.get_event_loop()
# ...
loop.close()
print("Fin de la boucle !")
However, you are also trying to use a aiophttp.ClientSession() object, and that's an asynchronous context manager, you are expected to use it with async with, not just with, and so has to be run in aside an awaitable task. If you use with instead of async with a TypeError("Use async with instead") exception will be raised.
That all means you need to move the loop.run_until_complete() call out of your faire_toutes_les_requetes_sans_bloquer() function, so you can keep that as the main task to be run; you can call and await on asycio.gather() directly then:
async def faire_toutes_les_requetes_sans_bloquer():
async with aiohttp.ClientSession() as session:
futures = [requete_sans_bloquer(x, session) for x in range(10)]
await asyncio.gather(*futures)
print("Fin de la boucle !")
print("Non bloquant : ")
start = datetime.datetime.now()
asyncio.run(faire_toutes_les_requetes_sans_bloquer())
exec_time = (datetime.datetime.now() - start).seconds
print(f"Pour faire 10 requêtes, ça prend {exec_time}s\n")
I used the new asyncio.run() function (Python 3.7 and up) to run the single main task. This creates a dedicated loop for that top-level coroutine and runs it until complete.
Next, you need to move the closing ) parenthesis on the await resp.json() expression:
uid = (await response.json())['uuid']
You want to access the 'uuid' key on the result of the await, not the coroutine that response.json() produces.
With those changes your code works, but the asyncio version finishes in sub-second time; you may want to print microseconds:
exec_time = (datetime.datetime.now() - start).total_seconds()
print(f"Pour faire 10 requêtes, ça prend {exec_time:.3f}s\n")
On my machine, the synchronous requests code in about 4-5 seconds, and the asycio code completes in under .5 seconds.
Do not use loop.run_until_complete call inside async function. The purpose for that method is to run an async function inside sync context. Anyway here's how you should change the code:
async def faire_toutes_les_requetes_sans_bloquer():
async with aiohttp.ClientSession() as session:
futures = [requete_sans_bloquer(x, session) for x in range(10)]
await asyncio.gather(*futures)
print("Fin de la boucle !")
loop = asyncio.get_event_loop()
loop.run_until_complete(faire_toutes_les_requetes_sans_bloquer())
Note that alone faire_toutes_les_requetes_sans_bloquer() call creates a future that has to be either awaited via explicit await (for that you have to be inside async context) or passed to some event loop. When left alone Python complains about that. In your original code you do none of that.
Not sure if this was the issue for you, but for me the response from the coroutine was another coroutine, so my code started warning me (note not actually crashing) I had creating coroutines that weren't being called. After I actually called them (although I didn't realy use the response the error went away).
Note main code I added was:
content_from_url_as_str: list[str] = await asyncio.gather(*content_from_url, return_exceptions=True)
inspired after I saw:
response: str = await content_from_url[0]
Full code:
"""
-- Notes from [1]
Threading and asyncio both run on a single processor and therefore only run one at a time [1]. It's cooperative concurrency.
Note: threads.py has a very good block with good defintions for io-bound, cpu-bound if you need to recall it.
Note: coroutine is an important definition to understand before proceeding. Definition provided at the end of this tutorial.
General idea for asyncio is that there is a general event loop that controls how and when each tasks gets run.
The event loop is aware of each task and knows what states they are in.
For simplicitly of exponsition assume there are only two states:
a) Ready state
b) Waiting state
a) indicates that a task has work to do and can be run - while b) indicates that a task is waiting for a response from an
external thing (e.g. io, printer, disk, network, coq, etc). This simplified event loop has two lists of tasks
(ready_to_run_lst, waiting_lst) and runs things from the ready to run list. Once a task runs it is in complete control
until it cooperatively hands back control to the event loop.
The way it works is that the task that was ran does what it needs to do (usually an io operation, or an interleaved op
or something like that) but crucially it gives control back to the event loop when the running task (with control) thinks is best.
(Note that this means the task might not have fully completed getting what is "fully needs".
This is probably useful when the user whats to implement the interleaving himself.)
Once the task cooperatively gives back control to the event loop it is placed by the event loop in either the
ready to run list or waiting list (depending how fast the io ran, etc). Then the event loop goes through the waiting
loop to see if anything waiting has "returned".
Once all the tasks have been sorted into the right list the event loop is able to choose what to run next (e.g. by
choosing the one that has been waiting to be ran the longest). This repeats until the event loop code you wrote is done.
The crucial point (and distinction with threads) that we want to emphasizes is that in asyncio, an operation is never
interrupted in the middle and every switching/interleaving is done deliberately by the programmer.
In a way you don't have to worry about making your code thread safe.
For more details see [2], [3].
Asyncio syntax:
i) await = this is where the code you wrote calls an expensive function (e.g. an io) and thus hands back control to the
event loop. Then the event loop will likely put it in the waiting loop and runs some other task. Likely eventually
the event loop comes back to this function and runs the remaining code given that we have the value from the io now.
await = the key word that does (mainly) two things 1) gives control back to the event loop to see if there is something
else to run if we called it on a real expensive io operation (e.g. calling network, printer, etc) 2) gives control to
the new coroutine (code that might give up control copperatively) that it is awaiting. If this is your own code with async
then it means it will go into this new async function (coroutine) you defined.
No real async benefits are being experienced until you call (await) a real io e.g. asyncio.sleep is the typical debug example.
todo: clarify, I think await doesn't actually give control back to the event loop but instead runs the "coroutine" this
await is pointing too. This means that if it's a real IO then it will actually give it back to the event loop
to do something else. In this case it is actually doing something "in parallel" in the async way.
Otherwise, it is your own python coroutine and thus gives it the control but "no true async parallelism" happens.
iii) async = approximately a flag that tells python the defined function might use await. This is not strictly true but
it gives you a simple model while your getting started. todo - clarify async.
async = defines a coroutine. This doesn't define a real io, it only defines a function that can give up and give the
execution power to other coroutines or the (asyncio) event loop.
todo - context manager with async
ii) awaiting = when you call something (e.g. a function) that usually requires waiting for the io response/return/value.
todo: though it seems it's also the python keyword to give control to a coroutine you wrote in python or give
control to the event loop assuming your awaiting an actual io call.
iv) async with = this creates a context manager from an object you would normally await - i.e. an object you would
wait to get the return value from an io. So usually we swap out (switch) from this object.
todo - e.g.
Note: - any function that calls await needs to be marked with async or you’ll get a syntax error otherwise.
- a task never gives up control without intentionally doing so e.g. never in the middle of an op.
Cons: - note how this also requires more thinking carefully (but feels less dangerous than threading due to no pre-emptive
switching) due to the concurrency. Another disadvantage is again the idisocyncracies of using this in python + learning
new syntax and details for it to actually work.
- understanding the semanics of new syntax + learning where to really put the syntax to avoid semantic errors.
- we needed a special asycio compatible lib for requests, since the normal requests is not designed to inform
the event loop that it's block (or done blocking)
- if one of the tasks doesn't cooperate properly then the whole code can be a mess and slow it down.
- not all libraries support the async IO paradigm in python (e.g. asyncio, trio, etc).
Pro: + despite learning where to put await and async might be annoying it forces your to think carefully about your code
which on itself can be an advantage (e.g. better, faster, less bugs due to thinking carefully)
+ often faster...? (skeptical)
1. https://realpython.com/python-concurrency/
2. https://realpython.com/async-io-python/
3. https://stackoverflow.com/a/51116910/6843734
todo - read [2] later (or [3] but thats not a tutorial and its more details so perhaps not a priority).
asynchronous = 1) dictionary def: not happening at the same time
e.g. happening indepedently 2) computing def: happening independently of the main program flow
couroutine = are computer program components that generalize subroutines for non-preemptive multitasking, by allowing execution to be suspended and resumed.
So basically it's a routine/"function" that can give up control in "a controlled way" (i.e. not randomly like with threads).
Usually they are associated with a single process -- so it's concurrent but not parallel.
Interesting note: Coroutines are well-suited for implementing familiar program components such as cooperative tasks, exceptions, event loops, iterators, infinite lists and pipes.
Likely we have an event loop in this document as an example. I guess yield and operators too are good examples!
Interesting contrast with subroutines: Subroutines are special cases of coroutines.[3] When subroutines are invoked, execution begins at the start,
and once a subroutine exits, it is finished; an instance of a subroutine only returns once, and does not hold state between invocations.
By contrast, coroutines can exit by calling other coroutines, which may later return to the point where they were invoked in the original coroutine;
from the coroutine's point of view, it is not exiting but calling another coroutine.
Coroutines are very similar to threads. However, coroutines are cooperatively multitasked, whereas threads are typically preemptively multitasked.
event loop = event loop is a programming construct or design pattern that waits for and dispatches events or messages in a program.
Appendix:
For I/O-bound problems, there’s a general rule of thumb in the Python community:
“Use asyncio when you can, threading when you must.”
asyncio can provide the best speed up for this type of program, but sometimes you will require critical libraries that
have not been ported to take advantage of asyncio.
Remember that any task that doesn’t give up control to the event loop will block all of the other tasks
-- Notes from [2]
see asyncio_example2.py file.
The sync fil should have taken longer e.g. in one run the async file took:
Downloaded 160 sites in 0.4063692092895508 seconds
While the sync option took:
Downloaded 160 in 3.351937770843506 seconds
"""
import asyncio
from asyncio import Task
from asyncio.events import AbstractEventLoop
import aiohttp
from aiohttp import ClientResponse
from aiohttp.client import ClientSession
from typing import Coroutine
import time
async def download_site(session: ClientSession, url: str) -> str:
async with session.get(url) as response:
print(f"Read {response.content_length} from {url}")
return response.text()
async def download_all_sites(sites: list[str]) -> list[str]:
# async with = this creates a context manager from an object you would normally await - i.e. an object you would wait to get the return value from an io. So usually we swap out (switch) from this object.
async with aiohttp.ClientSession() as session: # we will usually away session.FUNCS
# create all the download code a coroutines/task to be later managed/run by the event loop
tasks: list[Task] = []
for url in sites:
# creates a task from a coroutine todo: basically it seems it creates a callable coroutine? (i.e. function that is able to give up control cooperatively or runs an external io and also thus gives back control cooperatively to the event loop). read more? https://stackoverflow.com/questions/36342899/asyncio-ensure-future-vs-baseeventloop-create-task-vs-simple-coroutine
task: Task = asyncio.ensure_future(download_site(session, url))
tasks.append(task)
# runs tasks/coroutines in the event loop and aggrates the results. todo: does this halt until all coroutines have returned? I think so due to the paridgm of how async code works.
content_from_url: list[ClientResponse.text] = await asyncio.gather(*tasks, return_exceptions=True)
assert isinstance(content_from_url[0], Coroutine) # note allresponses are coroutines
print(f'result after aggregating/doing all coroutine tasks/jobs = {content_from_url=}')
# this is needed since the response is in a coroutine object for some reason
content_from_url_as_str: list[str] = await asyncio.gather(*content_from_url, return_exceptions=True)
print(f'result after getting response from coroutines that hold the text = {content_from_url_as_str=}')
return content_from_url_as_str
if __name__ == "__main__":
# - args
num_sites: int = 80
sites: list[str] = ["https://www.jython.org", "http://olympus.realpython.org/dice"] * num_sites
start_time: float = time.time()
# - run the same 160 tasks but without async paradigm, should be slower!
# note: you can't actually do this here because you have the async definitions to your functions.
# to test the synchronous version see the synchronous.py file. Then compare the two run times.
# await download_all_sites(sites)
# download_all_sites(sites)
# - Execute the coroutine coro and return the result.
asyncio.run(download_all_sites(sites))
# - run event loop manager and run all tasks with cooperative concurrency
# asyncio.get_event_loop().run_until_complete(download_all_sites(sites))
# makes explicit the creation of the event loop that manages the coroutines & external ios
# event_loop: AbstractEventLoop = asyncio.get_event_loop()
# asyncio.run(download_all_sites(sites))
# making creating the coroutine that hasn't been ran yet with it's args explicit
# event_loop: AbstractEventLoop = asyncio.get_event_loop()
# download_all_sites_coroutine: Coroutine = download_all_sites(sites)
# asyncio.run(download_all_sites_coroutine)
# - print stats about the content download and duration
duration = time.time() - start_time
print(f"Downloaded {len(sites)} sites in {duration} seconds")
print('Success.\a')
I am creating a cryptocurrency exchange API client using Python3.5 and Tkinter. I have several displays that I want to update asynchronously every 10 seconds. I am able to update the displays every 10 seconds using Tk.after() like in this example
def updateLoans():
offers = dd.loanOffers()
demands = dd.loanDemands()
w.LoanOfferView.delete(1.0, END)
w.LoanDemandView.delete(1.0, END)
w.LoanOfferView.insert(END, offers)
w.LoanDemandView.insert(END, demands)
print('loans refreshed')
root.after(10000, updateLoans)
In order for the after method to continue to update continuously every 10 seconds the function updateLoans() needs to be passed as a callable into after() inside of the function.
Now the part that is stumping me, when I make this function asynchronous with python's new async and await keywords
async def updateLoans():
offers = await dd.loanOffers()
demands = await dd.loanDemands()
w.LoanOfferView.delete(1.0, END)
w.LoanDemandView.delete(1.0, END)
w.LoanOfferView.insert(END, offers)
w.LoanDemandView.insert(END, demands)
print('loans refreshed')
root.after(10000, updateLoans)
The problem here is that I can not await a callable inside of the parameters for the after method. So I get a runtime warning. RuntimeWarning: coroutine 'updateLoans' was never awaited.
My initial function call IS placed inside of an event loop.
loop = asyncio.get_event_loop()
loop.run_until_complete(updateLoans())
loop.close()
The display populates just fine initially but never updates.
How can I use Tk.after to continuously update a tkinter display asynchronously?
tk.after accepts a normal function, not a coroutine. To run the coroutine to completion, you can use run_until_complete, just as you did the first time:
loop = asyncio.get_event_loop()
root.after(10000, lambda: loop.run_until_complete(updateLoans()))
Also, don't call loop.close(), since you'll need the loop again.
The above quick fix will work fine for many use cases. The fact is, however, that it will render the GUI completely unresponsive if updateLoans() takes a long time due to slow network or a problem with the remote service. A good GUI app will want to avoid this.
While Tkinter and asyncio cannot share an event loop yet, it is perfectly possible to run the asyncio event loop in a separate thread. The main thread then runs the GUI, while a dedicated asyncio thread runs all asyncio coroutines. When the event loop needs to notify the GUI to refresh something, it can use a queue as shown here. On the other hand, if the GUI needs to tell the event loop to do something, it can call call_soon_threadsafe or run_coroutine_threadsafe.
Example code (untested):
gui_queue = queue.Queue()
async def updateLoans():
while True:
offers = await dd.loanOffers()
demands = await dd.loanDemands()
print('loans obtained')
gui_queue.put(lambda: updateLoansGui(offers, demands))
await asyncio.sleep(10)
def updateLoansGui(offers, demands):
w.LoanOfferView.delete(1.0, END)
w.LoanDemandView.delete(1.0, END)
w.LoanOfferView.insert(END, offers)
w.LoanDemandView.insert(END, demands)
print('loans GUI refreshed')
# http://effbot.org/zone/tkinter-threads.htm
def periodicGuiUpdate():
while True:
try:
fn = gui_queue.get_nowait()
except queue.Empty:
break
fn()
root.after(100, periodicGuiUpdate)
# Run the asyncio event loop in a worker thread.
def start_loop():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.create_task(updateLoans())
loop.run_forever()
threading.Thread(target=start_loop).start()
# Run the GUI main loop in the main thread.
periodicGuiUpdate()
root.mainloop()
# To stop the event loop, call loop.call_soon_threadsafe(loop.stop).
# To start a coroutine from the GUI, call asyncio.run_coroutine_threadsafe.
I'm trying to create a function performing some asynchronous operations using asyncio, users of this function should not need to know that asyncio is involved under the hood.
I'm having a very hard time understanding how this shall be done with the asyncio API as most functions seem to operate under some global loop-variable accessed with get_event_loop and calls to this are affected by the global state inside this loop.
I have four examples here where two (foo1 and foo3) seem to be reasonable use cases but they all show very strange behaviors:
async def bar(loop):
# Disregard how simple this is, it's just for example
s = await asyncio.create_subprocess_exec("ls", loop=loop)
def foo1():
# Example1: Just use get_event_loop
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000))
# On exit this is written to stderr:
# Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
# Traceback (most recent call last):
# File "/usr/lib/python3.5/asyncio/base_events.py", line 510, in __del__
# File "/usr/lib/python3.5/asyncio/unix_events.py", line 65, in close
# File "/usr/lib/python3.5/asyncio/unix_events.py", line 146, in remove_signal_handler
# File "/usr/lib/python3.5/signal.py", line 47, in signal
# TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object
def foo2():
# Example2: Use get_event_loop and close it when done
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000)) # RuntimeError: Event loop is closed --- if foo2() is called twice
loop.close()
def foo3():
# Example3: Always use new_event_loop
loop = asyncio.new_event_loop()
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000)) #RuntimeError: Cannot add child handler, the child watcher does not have a loop attached
loop.close()
def foo4():
# Example4: Same as foo3 but also set_event_loop to the newly created one
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop) # Polutes global event loop, callers of foo4 does not expect this.
loop.run_until_complete(asyncio.wait_for(bar(loop), 1000)) # OK
loop.close()
None of these functions work and i don't see any other obvious way to do it, how is asyncio supposed to be used? It's seems like it's only designed to be used under the assumption that the entry point of the application is the only place where you can create and close the global loop. Do i have to fiddle around with event loop policies?
foo3 seems like the correct solution but i get an error even though i explicitly pass along loop, because deep down inside create_subprocess_exec it is using the current policy to get a new loop which is None, is this a bug in asyncio subprocess?
I'm using Python 3.5.3 on Ubuntu.
foo1 error happens because you didn't close event loop, see this issue.
foo2 because you can't reuse closed event loop.
foo3 because you didn't set new event loop as global.
foo4 is almost what you want, all you left to do is store old event loop and set it back as global after bar executed:
import asyncio
async def bar():
# After you set new event loop global,
# there's no need to pass loop as param to bar or anywhere else.
process = await asyncio.create_subprocess_exec("ls")
await process.communicate()
def sync_exec(coro): # foo5
old_loop = asyncio.get_event_loop()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(coro)
finally:
loop.close()
asyncio.set_event_loop(old_loop)
sync_exec(asyncio.wait_for(bar(), 1000))
One more important thing: it's not clear why you want to hide using of asyncio behind some sync functions, but usually it's bad idea. Whole thing about one global event loop is to allow user to run different concurrent jobs in this single event loop. You're trying to take away this possibility. I think you should reconsider this decision.
Upgrade to Python 3.6, then foo1() will work, without the need to explicitly close the default event loop.
Not the answer i was hoping for as we only use 3.5 :(