I'm writing a library which is using Tornado Web's tornado.httpclient.AsyncHTTPClient to make requests which gives my code a async interface of:
async def my_library_function():
return await ...
I want to make this interface optionally serial if the user provides a kwarg - something like: serial=True. Though you can't obviously call a function defined with the async keyword from a normal function without await. This would be ideal - though almost certain imposible in the language at the moment:
async def here_we_go():
result = await my_library_function()
result = my_library_function(serial=True)
I'm not been able to find anything online where someones come up with a nice solution to this. I don't want to have to reimplement basically the same code without the awaits splattered throughout.
Is this something that can be solved or would it need support from the language?
Solution (though use Jesse's instead - explained below)
Jesse's solution below is pretty much what I'm going to go with. I did end up getting the interface I originally wanted by using a decorator. Something like this:
import asyncio
from functools import wraps
def serializable(f):
#wraps(f)
def wrapper(*args, asynchronous=False, **kwargs):
if asynchronous:
return f(*args, **kwargs)
else:
# Get pythons current execution thread and use that
loop = asyncio.get_event_loop()
return loop.run_until_complete(f(*args, **kwargs))
return wrapper
This gives you this interface:
result = await my_library_function(asynchronous=True)
result = my_library_function(asynchronous=False)
I sanity checked this on python's async mailing list and I was lucky enough to have Guido respond and he politely shot it down for this reason:
Code smell -- being able to call the same function both asynchronously
and synchronously is highly surprising. Also it violates the rule of
thumb that the value of an argument shouldn't affect the return type.
Nice to know it's possible though if not considered a great interface. Guido essentially suggested Jesse's answer and introducing the wrapping function as a helper util in the library instead of hiding it in a decorator.
When you want to call such a function synchronously, use run_until_complete:
asyncio.get_event_loop().run_until_complete(here_we_go())
Of course, if you do this often in your code, you should come up with an abbreviation for this statement, perhaps just:
def sync(fn, *args, **kwargs):
return asyncio.get_event_loop().run_until_complete(fn(*args, **kwargs))
Then you could do:
result = sync(here_we_go)
Related
Suppose I have code like this
async def fetch_text() -> str:
return "text "
async def show_something():
something = await fetch_text()
print(something)
Which is fine. But then I want to clean the data, so I do
async def fetch_text() -> str:
return "text "
def fetch_clean_text(text: str) -> str:
text = await fetch_text()
return text.strip(text)
async def show_something():
something = fetch_clean_text()
print(something)
(I could clean text inside show_something(), but let's assume that show_something() can print many things and doesn't or shouldn't know the proper way of cleaning them.)
This is of course a SyntaxError: 'await' outside async function. But—if this code could run—while the await expression is not placed inside a coroutine function, it is executed in the context of one. Why this behavior is not allowed?
I see one pro in this design; in my latter example, you can't see that show_something()'s body is doing something that can result in its suspension. But if I were to make fetch_clean_text() a coroutine, not only would it complicate things but would probably also reduce performance. It just makes little sense to have another coroutine that doesn't perform any I/O by itself. Is there a better way?
You can only use await in an async enviroment. Try changing the function to asnyc:
import asyncio
whatever = . . .
async def function(param) -> asyncio.coroutine:
await param
asyncio.run(function(whatever))
Simple and easy.
I see one pro in this design; in my latter example, you can't see that
show_something()'s body is doing something that can result in its
suspension.
That's exactly why it designed this way. Writing concurrent code can be very tricky and asyncio authors decided that it's critically important to always explicitly mark places of suspend in code.
This article explains it in details (you can start from "Get To The Point Already" paragraph).
But if I were to make fetch_clean_text() a coroutine, not only would
it complicate things but would probably also reduce performance.
You need coroutines almost exclusively when you deal with I/O. I/O always takes much-much more time than overhead for using coroutines. So I guess it can be said - no, comparing to I/O you already deal with, you won't lose any significant amount of execution time for using coroutines.
Is there a better way?
Only way I can suggest: is to maximally split logic that deals with I/O (async part) from rest of the code (sync part).
from typing import Awaitable
def clean_text(text: str) -> str:
return text.strip(text)
async def fetch_text() -> Awaitable[str]:
return "text "
async def fetch_clean_text(text: str) -> Awaitable[str]:
text = await fetch_text()
return clean_text(text)
async def show_something():
something = await fetch_clean_text()
print(something)
So I create a class variable called executor in my class
executor = ThreadPoolExecutor(100)
and instead of having functions and methods and using decorators, I simply use following line to handle my blocking tasks(like io and hash creation and....) in my async methods
result = await to_tornado_future(self.executor.submit(blocking_method, param1, param2)
I decided to use this style cause
1- decorators are slower by nature
2- there is no need for extra methods and functions
3- it workes as expected and creates no threads before it needed
Am I right ? Please use reasons(I want to know if the way I use, is slower or uses more resources or....)
Update
Based on Ben answer, my above approach was not correct
so I ended up using following function as needed, I think it's the best way to go
def pool(pool_executor, fn, *args, **kwargs):
new_future = Future()
result_future = pool_executor.submit(fn, *args, **kwargs)
result_future.add_done_callback(lambda f: new_future.set_result(f.result()))
return new_future
usage:
result = await pool(self.executor, time.sleep, 3)
This is safe as long as all your blocking methods are thread-safe. Since you mentioned doing IO in these threads, I'll point out that doing file IO here is fine but all network IO in Tornado must occur on the IOLoop's thread.
Why do you say "decorators are slower by nature"? Which decorators are slower than what? Some decorators have no performance overhead at all (although most do have some runtime cost). to_tornado_future(executor.submit()) isn't free either. (BTW, I think you want tornado.gen.convert_yielded instead of tornado.platform.asyncio.to_tornado_future. executor.submit doesn't return an asyncio.Future).
As a general rule, running blocking_method on a thread pool is going to be slower than just calling it directly. You should do this only when blocking_method is likely to block for long enough that you want the main thread free to do other things in the meantime.
I'm currently trying to use decorators to implement the functionality to wait for some assert calls in my unittests before i finally let the AssertionError bubble up.
Currently my decorator looks like this:
def waitUpTo(seconds):
def decorator(assert_func_to_decorate):
def wrapper(*args, **kwargs):
for _ in range(seconds):
print(datetime.now())
try:
return(assert_func_to_decorate(*args,**kwargs))
except AssertionError:
time.sleep(1)
return(assert_func_to_decorate(*args,**kwargs))
return wrapper
return decorator
And it works quite well when i call and use it like this:
waitUpTo(3)(self.assertRegexpMatches)('notHello', r'Hello')
But somehow this calls tend to get a little bit long.
Is there a way to use the # syntax for this before the actual call to the function and not before the definition of it?
Ideally such a thing would be nice:
#waitUpTo(3)
self.assertRegexpMatches('notHello', r'Hello')
I am coding a framework, where the framework will call a user-supplied function.
I want to allow the used supplied function to be any of the following:
a plain function
a function returning asyncio.Future
a asyncio.coroutine
That is, the user function can be either synchronous or asynchronous, and the framework does not know in advance, but needs to cope with all variants.
Twisted has defer.maybeDeferred for this. What would be the asyncio way?
I have something like the following (full code here):
import types
types.GeneratorType
def maybe_async(value):
if isinstance(value, types.GeneratorType) or \
isinstance(value, asyncio.futures.Future):
return value
else:
future = asyncio.Future()
future.set_result(value)
return future
and then call the user supplied function f like this in the framework:
res = yield from maybe_async(f(x))
This wraps any plain function return value into a Future - always. And I am wary of the performance or other impacts of this.
Is above the "recommended" way?
Also, the "inline" version of above code does not have this overhead. How could I achieve the best of both: no overhead for the "plain" case, but no code duplication for checking for async returns all over the framework?
To sum up, there seem to be two (main) options:
Idiom 1:
res = f(x)
if yields(res):
res = yield from res
where
def yields(value):
return isinstance(value, asyncio.futures.Future) or inspect.isgenerator(value)
or
Idiom 2:
res = yield from asyncio.coroutine(f)(x)
Instead of isinstance(value, types.GeneratorType), you can write asyncio.iscoroutine(value):
https://docs.python.org/dev/library/asyncio-task.html#asyncio.iscoroutine
Tornado/Twisted newb here. First I just want to confirm what I know (please correct and elaborate if I am wrong):
In order to use #gen.engine and gen.Task in Tornado, I need to feed gen.Task() functions that are:
asynchronous to begin with
has the keyword argument "callback"
calls the callback function at the very end
In other words the function should look something like this:
def function(arg1, arg2, ... , callback=None):
# asynchronous stuff here ...
callback()
And I would call it like this (trivial example):
#gen.engine
def coroutine_call():
yield gen.Task(function, arg1, arg2)
Now I am in a weird situation where I have to use Twisted in a Tornado system for asynchronous client calls to a server (since Tornado apparently does not support it).
So I wrote a function in Twisted (e.g. connects to the server):
import tornado.platform.twisted
tornado.platform.twisted.install()
from twisted.web.xmlrpc import Proxy
class AsyncConnection():
def __init__(self, hostname):
self.proxy = Proxy(hostname)
self.token = False
#defer.inlineCallbacks
def login(self, user, passwd, callback=None):
"""Login to server using given username and password"""
self.token = yield self.proxy.callRemote('login', user, passwd) # twisted function
callback()
And if I run it like so:
#gen.engine
def test():
conn = AsyncConnection("192.168.11.11")
yield gen.Task(conn.login, "user","pwd")
print conn.token
if __name__ == '__main__':
test()
tornado.ioloop.IOLoop.instance().start()
And I DO get the token as I want. But my question is:
I know that Twisted and Tornado can share the same IOLoop. But am I allowed to do this (i.e. use #defer.inlineCallbacks function in gen.Task simply by giving it the callback keyword argument)? I seem to get the right result but is my way really running things asynchronously? Any complications/problems with the IOLoop this way?
I actually posted somewhat related questions on other threads
Is it possible to use tornado's gen.engine and gen.Task with twisted?
Using Tornado and Twisted at the same time
and the answers told me that I should "wrap" the inlineCallback function. I was wondering if adding the callback keyword is enough to "wrap" the twisted function to be suitable for Tornado.
Thanks in advance
What you're doing is mostly fine: adding a callback argument is enough to make a function usable with gen.Task. The only tricky part is exception handling: you'll need to run the callback from an except or finally block to ensure it always happens, and should probably return some sort of value to indicate whether the operation succeeded or not (exceptions do not reliably pass through a gen.Task when you're working with non-Tornado code)
The wrapper approach (which I posted in Is it possible to use tornado's gen.engine and gen.Task with twisted?) has two advantages: it can be used with most Twisted code directly (since Twisted functions usually don't have a callback argument), and exceptions work more like you'd expect (an exception raised in the inner function will be propagated to the outer function).