Develop Reference

How to call asynchronous function in Django?

The following doesn't execute foo and gives
RuntimeWarning: coroutine 'foo' was never awaited
# urls.py
async def foo(data):
# process data ...
#api_view(['POST'])
def endpoint(request):
data = request.data.get('data')
# How to call foo here?
foo(data)
return Response({})

Django is an synchronous language but it supports Async behavior.
Sharing the code snippet which may help.
import asyncio
from channels.db import database_sync_to_async
def get_details(tag):
response = another_sync_function()
# Creating another thread to execute function
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
async_result = loop.run_until_complete(remove_tags(response, tag))
loop.close()
# Async function
async def remove_tags(response, tag_id):
// do something here
# calling another function only for executing database queries
await tag_query(response, tag_id)
#database_sync_to_async
def tag_query(response, tag_id):
Mymodel.objects.get(all_tag_id=tag_id).delete()
This way i called async function in synchronous function.
Reference for database sync to async decorator

Found a way to do it.
Create another file bar.py in the same directory as urls.py.
# bar.py
def foo(data):
// process data
# urls.py
from multiprocessing import Process
from .bar import foo
#api_view(['POST'])
def endpoint(request):
data = request.data.get('data')
p = Process(target=foo, args=(data,))
p.start()
return Response({})

You can't await foo in this context. Seeing that Django is mainly a synchronous library, it doesn't interact well with asynchronous code. The best advice I can give it to try avoid using an asynchronous function here, or perhaps use another method of concurrency (ie threading or multiprocessing).
Note: there is a great answer given about Django's synchronous nature that can be found here: Django is synchronous or asynchronous?.

Python coroutines: Release context manager when pausing

Background: I'm a very experienced Python programmer who is completely clueless about the new coroutines/async/await features. I can't write an async "hello world" to save my life.
My question is: I am given an arbitrary coroutine function f. I want to write a coroutine function g that will wrap f, i.e. I will give g to the user as if it was f, and the user will call it and be none the wiser, since g will be using f under the hood. Like when you decorate a normal Python function to add functionality.
The functionality that I want to add: Whenever the program flow goes into my coroutine, it acquires a context manager that I provide, and as soon as program flow goes out of the coroutine, it releases that context manager. Flow comes back in? Re-acquire the context manager. It goes back out? Re-release it. Until the coroutine is completely finished.
To demonstrate, here is the described functionality with plain generators:
def generator_wrapper(_, *args, **kwargs):
gen = function(*args, **kwargs)
method, incoming = gen.send, None
while True:
with self:
outgoing = method(incoming)
try:
method, incoming = gen.send, (yield outgoing)
except Exception as e:
method, incoming = gen.throw, e
Is it possible to do it with coroutines?

Coroutines are built on iterators - the __await__ special method is a regular iterator. This allows you to wrap the underlying iterator in yet another iterator. The trick is that you must unwrap the iterator of your target using its __await__, then re-wrap your own iterator using your own __await__.
The core functionality that works on instantiated coroutines looks like this:
class CoroWrapper:
"""Wrap ``target`` to have every send issued in a ``context``"""
def __init__(self, target: 'Coroutine', context: 'ContextManager'):
self.target = target
self.context = context
# wrap an iterator for use with 'await'
def __await__(self):
# unwrap the underlying iterator
target_iter = self.target.__await__()
# emulate 'yield from'
iter_send, iter_throw = target_iter.send, target_iter.throw
send, message = iter_send, None
while True:
# communicate with the target coroutine
try:
with self.context:
signal = send(message)
except StopIteration as err:
return err.value
else:
send = iter_send
# communicate with the ambient event loop
try:
message = yield signal
except BaseException as err:
send, message = iter_throw, err
Note that this explicitly works on a Coroutine, not an Awaitable - Coroutine.__await__ implements the generator interface. In theory, an Awaitable does not necessarily provide __await__().send or __await__().throw.
This is enough to pass messages in and out:
import asyncio
class PrintContext:
def __enter__(self):
print('enter')
def __exit__(self, exc_type, exc_val, exc_tb):
print('exit via', exc_type)
return False
async def main_coro():
print(
'wrapper returned',
await CoroWrapper(test_coro(), PrintContext())
)
async def test_coro(delay=0.5):
await asyncio.sleep(delay)
return 2
asyncio.run(main_coro())
# enter
# exit via None
# enter
# exit <class 'StopIteration'>
# wrapper returned 2
You can delegate the wrapping part to a separate decorator. This also ensures that you have an actual coroutine, not a custom class - some async libraries require this.
from functools import wraps
def send_context(context: 'ContextManager'):
"""Wrap a coroutine to issue every send in a context"""
def coro_wrapper(target: 'Callable[..., Coroutine]') -> 'Callable[..., Coroutine]':
#wraps(target)
async def context_coroutine(*args, **kwargs):
return await CoroWrapper(target(*args, **kwargs), context)
return context_coroutine
return coro_wrapper
This allows you to directly decorate a coroutine function:
#send_context(PrintContext())
async def test_coro(delay=0.5):
await asyncio.sleep(delay)
return 2
print('async run returned:', asyncio.run(test_coro()))
# enter
# exit via None
# enter
# exit via <class 'StopIteration'>
# async run returned: 2

Tornado web asynchronous

I am trying to play with this piece of code to understand #tornado.web.asynchronous. The code as intended should handle asynchronous web requests but it doesnt seem to work as intended. There are two end points:
1) http://localhost:5000/A (This is the time consuming request and
takes a few seconds)
2) http://localhost:5000/B (This is the fast request and takes no time to return.
However when I hit the browser to go to http://localhost:5000/A and then while that is running go to http://localhost:5000/B the second request is queued and runs only after A has finished.
In other words one task is time consuming but it blocks the other faster task. What am I doing wrong?
import tornado.web
from tornado.ioloop import IOLoop
import sys, random, signal
class TestHandler(tornado.web.RequestHandler):
"""
In below function goes your time consuming task
"""
def background_task(self):
sm = 0
for i in range(10 ** 8):
sm = sm + 1
return str(sm + random.randint(0, sm)) + "\n"
#tornado.web.asynchronous
def get(self):
""" Request that asynchronously calls background task. """
res = self.background_task()
self.write(str(res))
self.finish()
class TestHandler2(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self):
self.write('Response from server: ' + str(random.randint(0, 100000)) + "\n")
self.finish()
def sigterm_handler(signal, frame):
# save the state here or do whatever you want
print('SIGTERM: got kill, exiting')
sys.exit(0)
def main(argv):
signal.signal(signal.SIGTERM, sigterm_handler)
try:
if argv:
print ":argv:", argv
application = tornado.web.Application([
(r"/A", TestHandler),
(r"/B", TestHandler2),
])
application.listen(5000)
IOLoop.instance().start()
except KeyboardInterrupt:
print "Caught interrupt"
except Exception as e:
print e.message
finally:
print "App: exited"
if __name__ == '__main__':
sys.exit(main(sys.argv))

According to the documentation:
To minimize the cost of concurrent connections, Tornado uses a
single-threaded event loop. This means that all application code
should aim to be asynchronous and non-blocking because only one
operation can be active at a time.
To achieve this goal you need to prepare the RequestHandler properly. Simply adding #tornado.web.asynchronous decorator to any of the functions (get, post, etc.) is not enough if the function performs only synchronous actions.
What does the #tornado.web.asynchronous decorator do?
Let's look at the get function. The statements are executed one after another in a synchronous manner. Once the work is done and the function returns the request is being closed. A call to self.finish() is being made under the hood. However, when we use the #tornado.web.asynchronous decorator the request is not being closed after the function returned. So the self.finish() must be called by the user to finish the HTTP request. Without this decorator the request is automatically finished when the get() method returns.
Look at the "Example 21" from this page - tornado.web.asynchronous:
#web.asynchronous
def get(self):
http = httpclient.AsyncHTTPClient()
http.fetch("http://example.com/", self._on_download)
def _on_download(self, response):
self.finish()
The get() function performs an asynchronous call to the http://example.com/ page. Let's assume this call is a long action. So the http.fetch() function is being called and a moment later the get() function returns (http.fetch() is still running the background). The Tornado's IOLoop can move forward to serve the next request while the data from the http://example.com/ is being fetched. Once the the http.fetch() function call is finished the callback function - self._on_download - is called. Then self.finish() is called and the request is finally closed. This is the moment when the user can see the result in the browser.
It's possible due to the httpclient.AsyncHTTPClient(). If you use a synchronous version of the httpclient.HTTPClient() you will need to wait for the call to http://example.com/ to finish. Then the get() function will return and the next request will be processed.
To sum up, you use #tornado.web.asynchronous decorator if you use asynchronous code inside the RequestHandler which is advised. Otherwise it doesn't make much difference to the performance.
EDIT: To solve your problem you can run your time-consuming function in a separate thread. Here's a simple example of your TestHandler class:
class TestHandler(tornado.web.RequestHandler):
def on_finish(self, response):
self.write(response)
self.finish()
def async_function(base_function):
#functools.wraps(base_function)
def run_in_a_thread(*args, **kwargs):
func_t = threading.Thread(target=base_function, args=args, kwargs=kwargs)
func_t.start()
return run_in_a_thread
#async_function
def background_task(self, callback):
sm = 0
for i in range(10 ** 8):
sm = sm + 1
callback(str(sm + random.randint(0, sm)))
#tornado.web.asynchronous
def get(self):
res = self.background_task(self.on_finish)
You also need to add those imports to your code:
import threading
import functools
import threading
async_function is a decorator function. If you're not familiar with the topic I suggest to read (e.g.: decorators) and try it on your own. In general, our decorator allows the function to return immediately (so the main program execution can go forward) and the processing to take place at the same time in a separate thread. Once the function in a thread is finished we call a callback function which writes out the results to the end user and closes the connection.

Python with Tornado: How to get fetch response in async functions?

I'm trying to create a Python script which running Tornado Async http client with fetch and trying to get the response and print the response.body to the screen.
my class code is:
class MyAsyncHTTPClient(httpclient.AsyncHTTPClient):
#gen.coroutine
def _fetch(self, url):
print('send Asynchronous request ...['+url+"] ")
http_response = yield gen.Task(self.fetch, url)
print(http_response.body)
print('got Asynchronous response !!'+str(http_response))
raise gen.Return(http_response.body)
and I'm calling it this way:
async_http_client = MyAsyncHTTPClient()
res_body = async_http_client._fetch(url)
The issue is that I'm not sure how to deal with this code to get the returned value once it's ready.
can you please help?
Thanks!
Editing
I have also tried implementing this function like:
class MyAsyncHTTPClient(httpclient.AsyncHTTPClient):
#gen.coroutine
def _fetch(self, url):
print('send Asynchronous request ...['+url+"] "+str(self))
http_response = yield self.fetch(url)
print('got Asynchronous response !!')
return http_response.body
But I'm having the same results :(
Editing again
I have succeeded running the async class...but without the inherited object self.
I did it like that:
#gen.coroutine
def _fetch_async(self, url):
print('send Asynchronous request ...[' + url + "] ")
http_response = yield httpclient.AsyncHTTPClient().fetch(url)
#http_response = yield self.fetch(url)
print('got Asynchronous response !!')
return http_response.body
and it worked fine.
The issue is that I need to use the inherited object self, and I'm not sure what am I missing here when defining the class.
When debugging I can see that self is very "thin" with its content..
Please let me know what am I doing wrong here.
Thanks!

Asynchronous functions can only be called from other asynchronous functions. You must call _fetch like this:
#gen.coroutine
def f():
async_http_client = MyAsyncHTTPClient()
res_body = yield async_http_client._fetch(url)
If you're not doing this in the context of a tornado server, the best way to call a coroutine from your __main__ block is like this:
tornado.ioloop.IOLoop.current().run_sync(f)

Python Tornado: Delete request never ends

I am having problems with a delete request in Tornado. The request arrives to the server and everything in the handler work great, but it never returns the response to the client.
I have tried returning something, with only the "return" and even without the "return" and the result is always the same.
I am using Python 3.4, Tornado 4.1 and the RestClient of Firefox.
#web.asynchronous
#gen.coroutine
def delete(self, _id):
try:
model = Model()
model.delete(_id)
self.set_status(204)
except Exception as e:
logging.error(e)
self.set_status(500)
return

Tornado documentation (tornado.web.asynchronous):
If this decorator is given, the response is not finished when the method > returns. It is up to the request handler to call self.finish() to finish > the HTTP request.
You need to call tornado.web.RequestHandler.finish method. This will work:
#web.asynchronous
#gen.coroutine
def delete(self, _id):
try:
model = Model()
model.delete(_id)
self.set_status(204)
except Exception as e:
logging.error(e)
self.set_status(500)
self.finish()
return
However, you don't need asynchronous approach in this example. This will work also in the same way:
def delete(self, _id):
try:
model = Model()
model.delete(_id)
self.set_status(204)
except Exception as e:
logging.error(e)
self.set_status(500)
return
Also, if you are using #gen.coroutine decorator, you don't need to use #web.asynchronous decorator. Simply use only #gen.coroutine, it is the correct way and much more elegant.
Lastly, I think you should read this article for understanding asynchronous programming in Tornado.