Develop Reference

Twisted making a function non-blocking

I am using twisted for making an async webserver (to learn the idea behind even based asynchronous programming). consider this scenario , server when gets a GET request on some endpoint, ex http:localhost:8000/api/v1/calc_fact?num=1000 calculates the factorial of 1000 and return back the result. this part is easy to get. I am also aware of deferred API. how can i define my function calc_factorial() so that it return a deferred and the overall result is non-blocking.
How do I solve the problem?

I have done something similar.
In your resource you need to return a server.NOT_DONE_YET and add the calc_factorial deferred callback like this
def render_GET(self, request):
d = Deferred()
reactor.callLater(1, d.callback, None)
d.addCallback(self.calc_factorial, request)
d.addErrback(rror_handler, request)
return server.NOT_DONE_YET
Then inside the calc_factorial you write into the request:
def calc_factorial(self, request):
# something something
request.write("factorial calc done")
request.finish()
Once you write request finish it will trigger the NOT_DONE_YET

Can't get raise Return to work when using coroutines

I've been experimenting with Python 2.7 and Tornado 3.2. I've been trying to get a simple coroutine example to work, but without much luck:
import tornado.web
from tornado.gen import coroutine
from tornado.httpclient import AsyncHTTPClient
from tornado.gen import Return
class MainHandler(tornado.web.RequestHandler):
# Tried with and without #asynchronous
#tornado.web.asynchronous
def get(self):
data = MainService().get_google_data()
self.write(data)
class MainService:
#coroutine
def get_google_data(self):
response = yield AsyncHTTPClient().fetch("http://www.google.com")
raise Return(value = 'hello')
I expected this would write out 'hello' when cURLing the URL. Instead, I get:
...
File "/vagrant/venv/lib/python2.7/site-packages/tornado/web.py", line 656, in write
raise TypeError("write() only accepts bytes, unicode, and dict objects")
TypeError: write() only accepts bytes, unicode, and dict objects
Apparently, a Future is being returned, but calling result() on the future throws another exception: DummyFuture does not support blocking for results
The Tornado docs say that in order to return a value from a coroutine, you raise a Return exception. Looking at the source code, that indeed seems to be what's expected. Yet, when I run it, it doesn't seem to work.
Appreciate any insights on this!

You need to yield the call to get_google_data():
class MainHandler(tornado.web.RequestHandler):
#coroutine
def get(self):
data = yield MainService().get_google_data()
self.write(data)
Tornado coroutines always return a Future. You wait for the result of that Future by calling yield on it. Without the yield, you just end up getting the Future back immediately, without waiting for the coroutine to complete. You should also use the #coroutine decorator on the get method, in addition to get_google_data. The #asynchronous decorator is generally used if you want to use callbacks, rather than coroutines.

Why asynchronous function in tornado is blocking?

Why incoming request are not being processed while another request is in the "waiting" state?
If you look at the code below, function "get" has a tornado task which is executed with "yield" keyword, which would mean "wait for a callback to be executed". In my code, the callback is never executed. If you run request second time, while first is on hold, the second request is not processed. If you run any other requests, they are being processed just fine.
So, my actions:
1. Start application
2. GET localhost:8080/
- Application is printing output "incoming call"
3. GET localhost:8080/anotherrequest
- Application is printing output "another request"
4. GET localhost:8080/
- Application is not printing any output while I'm expecting it to print "incoming call". Why?
So, why this piece of code gets blocking? Code sample is attached.
I was using tornado 2.1 and python 2.7 to run this sample.
Thank you
import tornado
import tornado.web
from tornado import gen
class AnotherHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self):
print 'another request'
self.finish()
class MainHandler(tornado.web.RequestHandler):
def printStuff(*args, **kwargs):
print 'incoming call'
#tornado.web.asynchronous
#tornado.gen.engine
def get(self):
result = yield tornado.gen.Task(self.printStuff);
application = tornado.web.Application([
(r"/", MainHandler),
(r"/anotherrequest", AnotherHandler)
])
if __name__ == "__main__":
application.listen(8080)
tornado.ioloop.IOLoop.instance().start()

Each new request to "localhost:8080/" will, in fact, cause your application to print "incoming call." However, requests to "localhost:8080/" will never finish. In order for the yield statement to be used, printStuff has to accept a callback and execute it. Also, an asynchronous get function must call self.finish:
class MainHandler(tornado.web.RequestHandler):
def printStuff(self, callback):
print 'incoming call'
callback()
#tornado.web.asynchronous
#tornado.gen.engine
def get(self):
result = yield tornado.gen.Task(self.printStuff)
self.finish()
It's easier to use Tornado's modern "coroutine" interface instead of gen.Task and gen.engine:
class MainHandler(tornado.web.RequestHandler):
#gen.coroutine
def printStuff(self):
print 'incoming call'
#gen.coroutine
def get(self):
result = yield self.printStuff()
self.finish()

Found the problem, it's actually happening when requests are made from the browser. With "curl" everything works as expected. Sorry for inconvenience.

How to keep tornado call async?

While doing an async mongodb query like the one in the below class how is this call really non-blocking if I still have access to an argument like self.get_argument("ip_address") inside the callback function? Or should I not access to the argument like this to keep the call async?
class MainHandler(tornado.web.RequestHandler):
def get(self):
app_key = self.get_argument("app_key")
#async call to mongodb. call _valid_app afterwards
db.apps.find_one({'app_key': app_key}, callback=self._valid_app);
def _valid_app(self, response, error):
if error:
raise tornado.web.HTTPError(500)
if response:
ip_address = self.get_argument("ip_address")
#rest of the code
else:
print("invalid app_key")

The instance self referenced in the callback function will be hanging around until the end of the callback function, therefore self.arguments will always be available inside _valid_app.
Maybe you can be confused by what would happen if another request to the same handler were made during the async call to Mongo. This would not be a problem because, for any new request a new instance of MainHandler is created, not interfering with the previous one.

Tornado celery integration hacks

Since nobody provided a solution to this post plus the fact that I desperately need a workaround, here is my situation and some abstract solutions/ideas for debate.
My stack:
Tornado
Celery
MongoDB
Redis
RabbitMQ
My problem: Find a way for Tornado to dispatch a celery task ( solved ) and then asynchronously gather the result ( any ideas? ).
Scenario 1: (request/response hack plus webhook)
Tornado receives a (user)request, then saves in local memory (or in Redis) a { jobID : (user)request} to remember where to propagate the response, and fires a celery task with jobID
When celery completes the task, it performs a webhook at some url and tells tornado that this jobID has finished ( plus the results )
Tornado retrieves the (user)request and forwards a response to the (user)
Can this happen? Does it have any logic?
Scenario 2: (tornado plus long-polling)
Tornado dispatches the celery task and returns some primary json data to the client (jQuery)
jQuery does some long-polling upon receipt of the primary json, say, every x microseconds, and tornado replies according to some database flag. When the celery task completes, this database flag is set to True, then jQuery "loop" is finished.
Is this efficient?
Any other ideas/schemas?

My solution involves polling from tornado to celery:
class CeleryHandler(tornado.web.RequestHandlerr):
#tornado.web.asynchronous
def get(self):
task = yourCeleryTask.delay(**kwargs)
def check_celery_task():
if task.ready():
self.write({'success':True} )
self.set_header("Content-Type", "application/json")
self.finish()
else:
tornado.ioloop.IOLoop.instance().add_timeout(datetime.timedelta(0.00001), check_celery_task)
tornado.ioloop.IOLoop.instance().add_timeout(datetime.timedelta(0.00001), check_celery_task)
Here is post about it.

Here is our solution to the problem. Since we look for result in several handlers in our application we made the celery lookup a mixin class.
This also makes code more readable with the tornado.gen pattern.
from functools import partial
class CeleryResultMixin(object):
"""
Adds a callback function which could wait for the result asynchronously
"""
def wait_for_result(self, task, callback):
if task.ready():
callback(task.result)
else:
# TODO: Is this going to be too demanding on the result backend ?
# Probably there should be a timeout before each add_callback
tornado.ioloop.IOLoop.instance().add_callback(
partial(self.wait_for_result, task, callback)
)
class ARemoteTaskHandler(CeleryResultMixin, tornado.web.RequestHandler):
"""Execute a task asynchronously over a celery worker.
Wait for the result without blocking
When the result is available send it back
"""
#tornado.web.asynchronous
#tornado.web.authenticated
#tornado.gen.engine
def post(self):
"""Test the provided Magento connection
"""
task = expensive_task.delay(
self.get_argument('somearg'),
)
result = yield tornado.gen.Task(self.wait_for_result, task)
self.write({
'success': True,
'result': result.some_value
})
self.finish()

I stumbled upon this question and hitting the results backend repeatedly did not look optimal to me. So I implemented a Mixin similar to your Scenario 1 using Unix Sockets.
It notifies Tornado as soon as the task finishes (to be accurate, as soon as next task in chain runs) and only hits results backend once. Here is the link.

Now, https://github.com/mher/tornado-celery comes to rescue...
class GenAsyncHandler(web.RequestHandler):
#asynchronous
#gen.coroutine
def get(self):
response = yield gen.Task(tasks.sleep.apply_async, args=[3])
self.write(str(response.result))
self.finish()