Develop Reference

python tornado - how to return real-time data

I am using the tornado library in python. I have a queue where data gets added in. I have to keep connection open so that when client requests I send out items from queue. Here is a simple implementation of mine. The problem I face is when I add new elements to queue, I don't see it being it returned. Infact, I don't see any code executed below IOLoop.current().start() line.
from tornado.ioloop import IOLoop
from tornado.web import RequestHandler, Application, url,asynchronous
from Queue import Queue
import json
q=Queue()
q.put("one")
q.put("two")
class HelloHandler(RequestHandler):
def get(self):
data=q.get()
self.write(data)
def make_app():
return Application([
url(r"/", HelloHandler),
])
def main():
app = make_app()
app.listen(8888)
IOLoop.current().start() # stops here
time.sleep(2)
q.put("three")
print q
if __name__=='__main__':
main()
first time on this :
http://localhost:8888/
returns "one"
second time on this:
http://localhost:8888/
return "two"
Third time on this"
http://localhost:8888/
blocking

The problem you have is that calling IOLoop.current().start() transfers control to Tornado. It loops until IOLoop.stop() is called.
If you need to do something after the IOLoop has started, then you can use one of the callbacks. For example, here is your code modified to use IOLoop.call_later(). You could also use IOLoop.add_timeout() if you are using an earlier (<4.0) version of Tornado.
from tornado.ioloop import IOLoop
from tornado.web import RequestHandler, Application, url,asynchronous
from Queue import Queue
import json
q=Queue()
q.put("one")
q.put("two")
class HelloHandler(RequestHandler):
def get(self):
data=q.get()
self.write(data)
def make_app():
return Application([
url(r"/", HelloHandler),
])
def main():
app = make_app()
app.listen(8888)
IOLoop.current().call_later(2, q.put, "three")
IOLoop.current().start()
if __name__=='__main__':
main()

Creating a processing queue in Tornado

I'm using a Tornado web server to queue up items that need to be processed outside of the request/response cycle.
In my simplified example below, every time a request comes in, I add a new string to a list called queued_items. I want to create something that will watch that list and process the items as they show up in it.
(In my real code, the items are processed and sent over a TCP socket which may or may not be connected when the web request arrives. I want the web server to keep queuing up items regardless of the socket connection)
I'm trying to keep this code simple and not use external queues/programs like Redis or Beanstalk. It's not going to have very high volume.
What's a good way using Tornado idioms to watch the client.queued_items list for new items and process them as they arrive?
import time
import tornado.ioloop
import tornado.gen
import tornado.web
class Client():
def __init__(self):
self.queued_items = []
#tornado.gen.coroutine
def watch_queue(self):
# I have no idea what I'm doing
items = yield client.queued_items
# go_do_some_thing_with_items(items)
class IndexHandler(tornado.web.RequestHandler):
def get(self):
client.queued_items.append("%f" % time.time())
self.write("Queued a new item")
if __name__ == "__main__":
client = Client()
# Watch the queue for when new items show up
client.watch_queue()
# Create the web server
application = tornado.web.Application([
(r'/', IndexHandler),
], debug=True)
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()

There is a library called toro, which provides synchronization primitives for tornado. [Update: As of tornado 4.2, toro has been merged into tornado.]
Sounds like you could just use a toro.Queue (or tornado.queues.Queue in tornado 4.2+) to handle this:
import time
import toro
import tornado.ioloop
import tornado.gen
import tornado.web
class Client():
def __init__(self):
self.queued_items = toro.Queue()
#tornado.gen.coroutine
def watch_queue(self):
while True:
items = yield self.queued_items.get()
# go_do_something_with_items(items)
class IndexHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
yield client.queued_items.put("%f" % time.time())
self.write("Queued a new item")
if __name__ == "__main__":
client = Client()
# Watch the queue for when new items show up
tornado.ioloop.IOLoop.current().add_callback(client.watch_queue)
# Create the web server
application = tornado.web.Application([
(r'/', IndexHandler),
], debug=True)
application.listen(8888)
tornado.ioloop.IOLoop.current().start()
There are a few tweaks required, aside from switching the data structure from a list to a toro.Queue:
We need to schedule watch_queue to run inside the IOLoop using add_callback, rather than trying to call it directly outside of an IOLoop context.
IndexHandler.get needs to be converted to a coroutine, because toro.Queue.put is a coroutine.
I also added a while True loop to watch_queue, so that it will run forever, rather than just processing one item and then exiting.

Why tornado doesn't go concurrently?

The following is my test code. I am using Python2.7, with futures installed using:
pip install futures
The following is my demo code:
import tornado.ioloop
import tornado.web
from tornado.gen import coroutine, Task
from tornado.concurrent import Future
import time
class MainHandler(tornado.web.RequestHandler):
#coroutine
def get(self):
print "in"
res = yield Task(self._work)
self.write(res)
def _work(self, callback):
time.sleep(10)
callback("hello world!")
if __name__ == "__main__":
application = tornado.web.Application([
(r"/", MainHandler),
])
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
This code should go concurrently, shouldn't it? However, it just doesn't.
I tested with Firefox and IE. I think I made some mistakes. It would be nice for you to point out my error.
only one request at a time(http://localhost:8888/:
import tornado.ioloop
import tornado.web
from tornado.gen import coroutine, Return, Task
from tornado.process import Subprocess
from tornado.concurrent import Future
from threading import Thread
import time
#coroutine
def async_sleep(timeout):
""" Sleep without blocking the IOLoop. """
yield Task(tornado.ioloop.IOLoop.instance().add_timeout, time.time() + timeout)
class MainHandler(tornado.web.RequestHandler):
#coroutine
def get(self):
print "in"
res = yield self._work()
self.write(res)
#coroutine
def _work(self):
yield async_sleep(5)
raise Return("hello world!")
if __name__ == "__main__":
application = tornado.web.Application([
(r"/", MainHandler),
])
application.listen(8888)
ioloop=tornado.ioloop.IOLoop.instance()
Thread(target=ioloop.start).start()

Since you indicated in the comments you want to run a subprocess via tornado, here's an example illustrating how to do that. I also fixed a logic error where you were creating a Task when calling _work, which wasn't going to work the way you intended:
import tornado.ioloop
import tornado.web
from tornado.gen import coroutine, Return
from tornado.process import Subprocess
from tornado.concurrent import Future
class MainHandler(tornado.web.RequestHandler):
#coroutine
def get(self):
print "in"
res = yield self._work()
self.write(res)
#coroutine
def _work(self):
p = Subprocess(['sleep', '10'])
f = Future()
p.set_exit_callback(f.set_result)
yield f
raise Return("hello world!")
if __name__ == "__main__":
application = tornado.web.Application([
(r"/", MainHandler),
])
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
As you can see, I made _work a coroutine, and then used tornado's built-in Subprocess class to execute a command. I created a Future object, and instructed the Subprocess to call Future.set_result(return_code_of_subprocess) when it completed. Then I called yield on the Future instance. That makes the code wait until the subprocess completes, without blocking the I/O loop.

The time.sleep in your code is a blocking method.
You need tornado.gen.sleep(a non-blocking method, new in tornado 4.1) here to instead.

Tornado and Autobahn-python listening on the same port

Recently I started a small personal project. It's a realtime web system based on asyncio and autobahn-python. However I also would like to serve some static files via HTTP and do it from the same process. My HTTP server is Tornado sitting on top of asyncio event loop and everything works perfectly fine except that I have to start tornado and autobahn handlers on different ports. Here is a stripped down version of what I currently have:
import six
import datetime
import asyncio
import tornado.web
import tornado.httpserver
import tornado.netutil
from tornado.platform.asyncio import AsyncIOMainLoop
from autobahn.wamp import router
from autobahn.asyncio import wamp, websocket
# WAMP server
class MyBackendComponent(wamp.ApplicationSession):
def onConnect(self):
self.join(u"realm1")
#asyncio.coroutine
def onJoin(self, details):
def utcnow():
now = datetime.datetime.utcnow()
return six.u(now.strftime("%Y-%m-%dT%H:%M:%SZ"))
reg = yield from self.register(utcnow, 'com.timeservice.now')
# HTTP server
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world!")
tornado_app = tornado.web.Application(
[
(r"/", MainHandler),
],
)
if __name__ == '__main__':
router_factory = router.RouterFactory()
session_factory = wamp.RouterSessionFactory(router_factory)
session_factory.add(MyBackendComponent())
transport_factory = websocket.WampWebSocketServerFactory(session_factory,
debug=True,
debug_wamp=True)
AsyncIOMainLoop().install()
tornado_app.listen(80, "127.0.0.1")
loop = asyncio.get_event_loop()
coro = loop.create_server(transport_factory, "127.0.0.1", 8080)
server = loop.run_until_complete(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
server.close()
loop.close()
Question: Is there the Right Way to make autobahn-wamp and tornado handlers listen on the same port?
My initial idea was to implement some kind of socket.socket wrapper and dispatch incoming messages there but it turned out to be awfully messy. I don't want to use any external proxies because the backend should be portable as much as possible.
Also I'm not asking anybody to implement it for me(but of course you can if you want to!) - only to know if somebody have already done something similar before diving into autobahn/tornado code.
Thanks in advance!
PS: Sorry for my poor English - it's not my mother tongue.

How to make tornado call itself asynchronously

Below is the simple hello world app. How can I have tornado call '/' in two threads with a 1 sec sleep. Thus the page will be called 2 times per second. I will need later to extend to a redis call but want to start simple now since I am new to this logic. I need to build a web page that is called repeatably asynchronously.
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8880)
tornado.ioloop.IOLoop.instance().start()

If you are new to tornado, check out tornadogist. They have a lot of useful code snippets. Here is one adapted to your need:
from time import sleep
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.web import Application, asynchronous, RequestHandler
from multiprocessing.pool import ThreadPool
# 2 threads
_workers = ThreadPool(2)
def run_background(func, callback, args=(), kwds={}):
def _callback(result):
IOLoop.instance().add_callback(lambda: callback(result))
_workers.apply_async(func, args, kwds, _callback)
# blocking task like querying to MySQL
def blocking_task(n):
sleep(n)
return n
class MainHandler(RequestHandler):
#asynchronous
def get(self):
run_background(blocking_task, self.on_complete, (1,))
def on_complete(self, res):
self.write("Test {0}<br/>".format(res))
self.finish()
HTTPServer(Application([("/", MainHandler)],debug=True)).listen(8888)
IOLoop.instance().start()