I'm creating a web application in python using tornado web server. I'm using Motor for mongo db connection with tornado web server.
i'm referring to tornado motor docs.
in my application i'm uploading data from csv or other data sources and doing some machine learning operations. all the meta data and other required details have to be stored in mongo DB. Web page is a little heavy(as it has a lot of visualizations other operations).
Some examples i tried for learning motor
Example of call back:
import tornado.web, tornado.ioloop
import motor
class NewMessageHandler(tornado.web.RequestHandler):
def get(self):
"""Show a 'compose message' form."""
self.write('''
<form method="post">
<input type="text" name="msg">
<input type="submit">
</form>''')
# Method exits before the HTTP request completes, thus "asynchronous"
#tornado.web.asynchronous
def post(self):
"""Insert a message."""
msg = self.get_argument('msg')
# Async insert; callback is executed when insert completes
self.settings['db'].messages.insert(
{'msg': msg},
callback=self._on_response)
def _on_response(self, result, error):
if error:
raise tornado.web.HTTPError(500, error)
else:
self.redirect('/')
class MessagesHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self):
"""Display all messages."""
self.write('Compose a message<br>')
self.write('<ul>')
db = self.settings['db']
db.messages.find().sort([('_id', -1)]).each(self._got_message)
def _got_message(self, message, error):
if error:
raise tornado.web.HTTPError(500, error)
elif message:
self.write('<li>%s</li>' % message['msg'])
else:
# Iteration complete
self.write('</ul>')
self.finish()
db = motor.MotorClient().test
application = tornado.web.Application(
[
(r'/compose', NewMessageHandler),
(r'/', MessagesHandler)
],
db=db
)
#print 'Listening on http://localhost:8888'
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
Example of coroutines:
from tornado import gen
import tornado.web, tornado.ioloop
import motor
class NewMessageHandler(tornado.web.RequestHandler):
def get(self):
"""Show a 'compose message' form."""
self.write('''
<form method="post">
<input type="text" name="msg">
<input type="submit">
</form>''')
#tornado.web.asynchronous
#gen.coroutine
def post(self):
"""Insert a message."""
msg = self.get_argument('msg')
db = self.settings['db']
# insert() returns a Future. Yield the Future to get the result.
result = yield db.messages.insert({'msg': msg})
# Success
self.redirect('/')
class MessagesHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
#gen.coroutine
def get(self):
"""Display all messages."""
self.write('Compose a message<br>')
self.write('<ul>')
db = self.settings['db']
cursor = db.messages.find().sort([('_id', -1)])
while (yield cursor.fetch_next):
message = cursor.next_object()
self.write('<li>%s</li>' % message['msg'])
# Iteration complete
self.write('</ul>')
self.finish()
db = motor.MotorClient().test
application = tornado.web.Application(
[
(r'/compose', NewMessageHandler),
(r'/', MessagesHandler)
],
db=db
)
print 'Listening on http://localhost:8881'
application.listen(8881)
tornado.ioloop.IOLoop.instance().start()
How do i compare the benefits or performance of each example ? Both example do the same functionality.
What should i use ? callbacks, co-routines, generators ? I need good performance and flexibility in my application. there are some examples in this link also https://motor.readthedocs.org/en/stable/tutorial.html
Related
When running my tornado script which should send a request and get the body of the response and append this to some html code on a localhost port. However, I get this error:
TypeError: init() got an unexpected keyword argument 'callback'
It turns out from the documentation Asynchronous HTTP client that the callback has been removed for Future
Changed in version 6.0: The callback argument was removed. Use the returned Future instead.
There are a lack of effective examples and so what is the most appropriate method to use Future in replacement for callback in my script?
import asyncio
import os
import json
import tornado.httpclient
import tornado.web
import tornado.httpserver
import tornado.gen
from tornado.options import define, options
define('port', default = 9060, help="run port 9060", type=int)
class requestFour(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
#arg = self.get_argument('stock')
client = tornado.httpclient.AsyncHTTPClient()
yield client.fetch('https://books.toscrape.com', callback=self.on_response)
def on_response(self, response):
body = json.load(response.body)
self.write("""
<div style="text-align: center">
<div style="font-size: 72px">%s</div>
<div style="font-size: 144px">%.02f</div>
<div style="font-size: 24px">tweets per second</div>
</div>
""" % (body))
self.finish()
def my_app():
app = tornado.web.Application(handlers = [(r'/', requestFour)])
http_server = tornado.httpserver.HTTPServer(app)
return http_server
async def main():
app = my_app()
app.listen(options.port)
shutdown_event = asyncio.Event()
await shutdown_event.wait()
if __name__ == '__main__':
asyncio.run(main())
You can just call self.on_response directly:
response = yield client.fetch('https://books.toscrape.com')
self.on_response(response)
Additional notes:
Make your life easier by ditching #gen.coroutine and yield and switching to async/await:
async def get(self):
#arg = self.get_argument('stock')
client = tornado.httpclient.AsyncHTTPClient()
response = await client.fetch('https://books.toscrape.com')
self.on_response(response)
I have requirement to stream huge oracle record set in Python rest API. I am running flask on tornado server. when I use tornado streaming dosent work, whereas on flask native server(werkzeung) it works perfectly. can anyone help me tornado support streaming or not?
Here is a small sample of code just trying to stream by using yield.
import tornado.web
from tornado import gen, httpclient
import asyncio, json, time
class basicReuqestHandler(tornado.web.RequestHandler):
def get(self):
self.write("Helow World!")
class staticReuqestHandler(tornado.web.RequestHandler):
def get(self):
self.render("index.html")
class StreamingHandler(tornado.web.RequestHandler):
#gen.coroutine
def get(self):
self.write("starting ....")
def stream():
a = 1
for i in range(100):
a = a+i
print(i)
print(json.dumps(i))
yield json.dumps(i)
self.write(stream())
self.write("closing...")
self.finish()
if __name__=='__main__':
app = tornado.web.Application([
(r"/", basicReuqestHandler),
(r"/myPage",staticReuqestHandler ),
(r"/StreamTest", StreamingHandler),
])
app.listen(7000)
tornado.ioloop.IOLoop.current().start()
I got my mistake, so answering to my question here to help anyone having similar issue.
here is the code:
import tornado.web
from tornado import gen, httpclient
import asyncio, json, time
class basicReuqestHandler(tornado.web.RequestHandler):
def get(self):
self.write("Helow World!")
class staticReuqestHandler(tornado.web.RequestHandler):
def get(self):
self.render("index.html")
class StreamingHandler(tornado.web.RequestHandler):
#gen.coroutine
def get(self):
self.write("starting ....")
def stream():
for i in range(100):
print(i)
print(json.dumps(i))
self.write(json.dumps(i))
yield self.flush()
self.write("closing...")
self.finish()
if __name__=='__main__':
app = tornado.web.Application([
(r"/", basicReuqestHandler),
(r"/myPage",staticReuqestHandler ),
(r"/StreamTest", StreamingHandler),
])
app.listen(7000)
tornado.ioloop.IOLoop.current().start()
To give you an idea of what I am trying to accomplish with Twisted Web and Autobahn websockets: my UI currently sends an initial HTTP GET request with an upgrade to a websocket in the header. Upon reading that in Twisted Web, the connection needs to switch from HTTP to a websocket protocol to pass data back and forth. Note that this websocket upgrade happens on the same port, port 8000.
Does anyone know how I can implement what I am trying to do? Thank you so much.
EDIT: updated code for working example. You can find it here: Payload from POST Request is Cutoff (Twisted Web)
Here is my code using Twisted Web:
class HttpResource(resource.Resource):
isLeaf = 1
def __init__(self):
self.children = {}
self.ws_port = None
print 'resource invoked'
def render_GET(self, request):
print 'render invoked'
if request.getHeader('Sec-WebSocket-Key'):
# Processing the Key as per RFC 6455
key = request.getHeader('Sec-WebSocket-Key')
h = hashlib.sha1(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11")
request.setHeader('Sec-WebSocket-Accept', base64.b64encode(h.digest()))
# setting response headers
request.setHeader('Upgrade', 'websocket')
request.setHeader('Connection', 'Upgrade')
request.setResponseCode(101)
return ''
else:
log("Regular HTTP GET request.")
return "<html><body style='margin: 0; overflow: hidden;'><iframe style='width: 100%; height: 100%; border: none;' src='http://tsa-graphiql.herokuapp.com/'></iframe></body></html>"
def render_POST(self,request):
log("POST request")
request.setResponseCode(200)
def handle_single_query(self, queryData):
log("Handle single query data.")
return
class HttpWsChannel(http.HTTPChannel):
def dataReceived(self, data):
log('Data received:\n{}'.format(data))
if data.startswith('GET'):
# This will invoke the render method of resource provided
http.HTTPChannel.dataReceived(self, data)
if data.startswith('POST'):
http.HTTPChannel.dataReceived(self, data)
else:
"""
Pass binary data to websocket class.
"""
ws_protocol = self.site.ws_factory.protocol(self.site.ws_factory.connection_subscriptions)
log(ws_protocol)
#just echo for now
# self.transport.write(data)
class HttpFactory(Site):
"""
Factory which takes care of tracking which protocol
instances or request instances are responsible for which
named response channels, so incoming messages can be
routed appropriately.
"""
def __init__(self, resource):
http.HTTPFactory.__init__(self)
self.resource = resource
self.ws_factory = WsProtocolFactory("ws://127.0.0.1:8000")
self.ws_factory.protocol = WsProtocol
def buildProtocol(self, addr):
try:
channel = HttpWsChannel()
channel.requestFactory = self.requestFactory
channel.site = self
return channel
except Exception as e:
log("Could not build protocol: {}".format(e))
site = HttpFactory(HttpResource())
if __name__ == '__main__':
reactor.listenTCP(8000, site)
reactor.run()
EDIT 7/8/2017: Here is the new code I am trying below. The websocket messages are received successfully via the onMessage method. However the HTTP requests are not working. The current error I am getting on a GET request is:
<html>
<head><title>404 - No Such Resource</title></head>
<body>
<h1>No Such Resource</h1>
<p>No such child resource.</p>
</body>
</html>
Python code
from twisted.web.server import (
Site,
)
from twisted.internet import reactor
from twisted.web.resource import (
Resource,
)
from autobahn.twisted.websocket import (
WebSocketServerProtocol,
WebSocketServerFactory,
)
from autobahn.twisted.resource import (
WebSocketResource,
)
class WebSocketProtocol(WebSocketServerProtocol):
def onConnect(self, request):
print("WebSocket connection request: {}".format(request))
def onMessage(self, payload, isBinary):
print("onMessage: {}".format(payload))
if __name__ == '__main__':
factory = WebSocketServerFactory()
factory.protocol = WebSocketProtocol
resource = WebSocketResource(factory)
root = Resource()
root.putChild(b"ws", resource)
site = Site(root)
reactor.listenTCP(8000, site)
reactor.run()
Use WebSocketResource to expose a WebSocketServerFactory as part of a Site.
from twisted.web.server import (
Site,
)
from twisted.web.resource import (
Resource,
)
from autobahn.twisted.websocket import (
WebSocketServerProtocol,
WebSocketServerFactory,
)
from autobahn.twisted.resource import (
WebSocketResource,
)
class YourAppProtocol(WebSocketServerProtocol):
def onConnect(self, request):
...
...
def main():
factory = WebSocketRendezvousFactory()
factory.protocol = YourAppProtocol
resource = WebSocketResource(factory)
root = Resource()
root.putChild(b"some-path-segment", resource)
root.putChild(...)
site = Site(root)
reactor.listenTCP(8080, site)
reactor.run()
The problems with truncated request bodies is probably because your upgrade protocol implementation is buggy. There is no framing applied at the level of dataReceived so you can't expect checks like startswith("GET") to be reliable.
Using WebSocketResource and Site gives you the correct HTTP parsing code from Twisted Web and Autobahn while also allowing you to speak WebSocket to a particular URL and regular HTTP to others.
So after reading a little bit on Google, I found this website that explains how to upgrade the HTTP connection to a websocket connection via Autobahn Twisted: Read and Set request headers via Autobahn Twisted.
The code that I was able to get to work is shown below!
from twisted.web.server import (
Site,
)
from twisted.internet import reactor
from twisted.web.resource import (
Resource,
)
from autobahn.twisted.websocket import (
WebSocketServerProtocol,
WebSocketServerFactory,
)
from autobahn.twisted.resource import (
WebSocketResource,
)
class HttpResource(Resource):
isLeaf = True
def render_GET(self, request):
return "<html><body style='margin: 0; overflow: hidden;'><iframe style='width: 100%; height: 100%; border: none;' src='http://tsa-graphiql.herokuapp.com/'></iframe></body></html>"
class WebSocketProtocol(WebSocketServerProtocol):
def onConnect(self, request):
custom_header = {}
if request.headers['sec-websocket-key']:
custom_header['sec-websocket-protocol'] = 'graphql-ws'
return (None, custom_header)
def onMessage(self, payload, isBinary):
print("onMessage: {}".format(payload))
if __name__ == '__main__':
factory = WebSocketServerFactory()
factory.protocol = WebSocketProtocol
resource = WebSocketResource(factory)
root = Resource()
root.putChild("", HttpResource())
root.putChild(b"ws", ws_resource)
site = Site(root)
reactor.listenTCP(8000, site)
I run a twisted server and It works fine with chrome but in firefox I can't push anything to client. I can post data but when I call request.write nothing happen in firefox!
the code is:
from twisted.web import resource, server, http
from twisted.internet import reactor
import random
# iframe closing tags are apparently necessary.
tophtml = """<html><head><title>Chat channel</title>
<style>
iframe { border: 0 }
</style>
</head><body>
<h1>HTML chat demo</h1>
<p>Inspired by Ka-Ping Yee's awesome <a href="http://zesty.ca/chat/";
>libdynim GIF chat app</a>.</p>
<!-- Like it, this uses no Java, DHTML, or reloading; unlike it,
this uses frames and no JavaScript (Ping's demo works with or without
JavaScript, but without JavaScript, it reloads the page when you speak.) -->
<iframe width="100%%" height="50" src="?frame=form;sess_id=%(sess_id)s">
</iframe>
<iframe width="100%%" height="300" src="?frame=output;sess_id=%(sess_id)s">
</iframe>
</body></html>
"""
formhtml = """<html><head><title>Chat post form</title></head><body>
<form method="POST">
<input name="sess_id" type="hidden" value="%(sess_id)s" />
to: <input name="destination" />
say: <input name="chatline" size="80" />
<input type="submit" value="send" />
</form>
</html>
"""
class ChatSession:
"A persistent connection to a user's browser."
def __init__(self, channel, request, sess_id):
(self.channel, self.request, self.sess_id) = (channel, request, sess_id)
self.deferred = request.notifyFinish()
self.deferred.addCallback(self.stop)
self.deferred.addErrback(self.stop)
def stop(self, reason):
"Called when the request finishes to close the channel."
print "%s stopping: %s" % (self.sess_id, reason)
self.channel.delete_session(self.sess_id)
def sendmsg(self, origin, msg):
"Display a chat message to the user."
self.request.write("""<div>
<%(origin)s> %(msg)s </div>
""" % {'origin': origin, 'msg': msg})
class ChatChannel(resource.Resource):
"A resource representing a chat room, plus private messages."
isLeaf = True
def __init__(self):
resource.Resource.__init__(self) # ??? necessary??
self.sessions = {}
def render_GET(self, request):
"Handle HTTP GET requests by dispatching on 'frame' arg."
if request.args.has_key('frame'):
frame = request.args['frame'][0]
if frame == 'form': return self.render_form(request)
elif frame == 'output': return self.do_chat_output(request)
sess_id = random.randrange(1000) # not secure, like everything here
return tophtml % {'sess_id': sess_id}
def render_form(self, request):
"The form used for posting."
return formhtml % {'sess_id': request.args['sess_id'][0]}
def do_chat_output(self, request):
"Open a persistent ChatSession."
sess_id = request.args['sess_id'][0]
# Note that this may remove a previous ChatSession from
# self.sessions:
self.sessions[sess_id] = ChatSession(self, request, sess_id)
# The next line is a hack due to Donovan Preston: increases
# browsers' per-server connection limit, which is normally 2
# if the server seems to support HTTP 1.1 connection
# keepalive, to 8.
request.setHeader('connection', 'close')
request.write("<html><head><title>session %s</title><body>\n" % sess_id)
return server.NOT_DONE_YET
def render_POST(self, request):
"Send back 204 No Content to an utterance of a chat line."
def arg(name):
return request.args[name][0]
self.handle_chatline(arg('destination'), arg('sess_id'),
arg('chatline'))
request.setResponseCode(http.NO_CONTENT)
return ""
def handle_chatline(self, dest, sess_id, chatline):
"Send a chat line from a source to a destination, '' meaning 'all'."
try:
if dest:
self.sessions[dest].sendmsg(sess_id, '(private) ' + chatline)
self.sessions[sess_id].sendmsg('server', 'private message sent')
else:
for session in self.sessions.values():
session.sendmsg(sess_id, chatline)
except Exception, e:
self.sessions[sess_id].sendmsg('error', str(e))
def delete_session(self, sess_id):
"Delete a session by ID --- if it's there."
try: del self.sessions[sess_id]
except KeyError: pass
if __name__ == '__main__':
port = 8086
reactor.listenTCP(port, server.Site(ChatChannel()))
print "ok, running on port", port
reactor.run()
what is the problem?
I tried to make proxy server by tornado in Python. The simple http proxy server has worked well, but the https proxy has some problem.
Part of my programs which might have problem are below.
import tornado.ioloop
from tornado.web import RequestHandler, Application
from tornado.httpclient import AsyncHTTPClient, HTTPRequest
from tornado.httpserver import HTTPServer
class HTTPSHandler(RequestHandler):
#tornado.web.asynchronous
def get(self):
print self.request.host, self.request.method
def handle_request(response):
if response.error and not isinstance(response.error, tornado.httpclient.HTTPError):
print "Error:", response.error
else:
self.write(response.body)
self.finish(" ")#in case of response.body == None
request = self.request
req = HTTPRequest(url=request.uri, method=request.method,
headers=request.headers, body=request.body,
allow_nonstandard_methods = True, follow_redirects = False,
validate_cert=True)
http_client = AsyncHTTPClient()
try:
http_client.fetch(req, handle_request)
except Exception as e:
print e
#tornado.web.asynchronous
def post(self):
return self.get()
#tornado.web.asynchronous
def head(self):
return self.get()
#tornado.web.asynchronous
def delete(self):
return self.get()
#tornado.web.asynchronous
def patch(self):
return self.get()
#tornado.web.asynchronous
def put(self):
return self.get()
#tornado.web.asynchronous
def options(self):
return self.get()
if __name__ == "__main__":
app2 = Application([(r"https:.*", HTTPSHandler),])
httpsServer = HTTPServer(app2, ssl_options = {
"certfile": "./server.crt",
"keyfile": "./server.key",
})
app2.listen(444)
tornado.ioloop.IOLoop.instance().start()
It outputs a WARNING like below (when I access to https://www.google.com and https://github.com)
WARNING:tornado.access:405 CONNECT www.google.co.jp:443 (127.0.0.1) 0.69ms
WARNING:tornado.access:405 CONNECT github.com:443 (127.0.0.1) 0.58ms
Finally, web pages which use https protocol could not be displayed with browser error.
ERR_TUNNEL_CONNECTION_FAILED
I guess, this is caused by the tornado’s requestHandler because it does not support CONNECT method.
My question is how can I use the CONNECT method?
I have noticed the way to fix this problem.
At first, SUPPORTED_METHOD should be written in HTTPSHandler class.
This can solve the 405 WARNING and browser error.
class HTTPSHandler(RequestHandler):
SUPPORTED_METHODS = ("CONNECT", "GET", "HEAD", "POST", "DELETE", "PATCH", "PUT", "OPTIONS")
#tornado.web.asynchronous
def get(self):
This is written in official document as follows.
If you want to support more methods than the standard GET/HEAD/POST, you
should override the class variable ``SUPPORTED_METHODS`` in your
`RequestHandler` subclass.
Moreover, to handle and process the CONNECT method request, additional method is needed in HTTPSHandler class.
#tornado.web.asynchronous
def connect(self):
print "some specific processings here"
Finally, I took stupid mistakes in regular expressions and ssl_option.
app2 = Application([(r"https:.*", HTTPSHandler),]) # not correct
app2 = Application([(r".*", HTTPSHandler),]) # correct
httpServer = HTTPServer(app2) # ssl_options is not needed
In theory you should be able to implement the CONNECT method in the same way that WebSocketHandler works, by hijacking the underlying connection's IOStream. But be warned that this is uncharted territory; the HTTP proxy protocol has some differences from plain HTTP and I don't know how well it will work to implement a proxy on top of a normal application-level HTTP service.