How can I do async request processing in Twisted like in Node.js?
I wrote sample with Twisted, but my app still waited an answer from long operation(I emulate this with time.sleep).
Also I don't understand how can I use reactor.callLater correct.
This is my sample of twisted app.
from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.defer import Deferred
import time
class Hello(Resource):
def getChild(self, name, request):
if name == '':
return self
print name
return Resource.getChild(self, name, request)
def render_GET(self, req):
d = Deferred()
reactor.callLater(2, d.callback, None)
d.addCallback(lambda _: self.say_hi(req))
return server.NOT_DONE_YET
def say_hi(self, req):
req.setHeader("content-type", "text/html")
time.sleep(5)
req.write("hello!")
req.finish()
class Hello2(Resource):
isLeaf = True
def render_GET(self, req):
req.setHeader("content-type", "text/html")
return "hello2!"
root = Hello()
root.putChild("2", Hello2())
reactor.listenTCP(8080, server.Site(root))
reactor.run()
Edit: Now question is how to write a sync code? Please example.
You're already doing it... sort of.
Your problem here is that time.sleep() is a blocking call, and will therefore make your whole server stop.
If you're using that as a stand-in for something that does network I/O (like urllib), the best option is usually to do the I/O with Twisted (like twisted.web.client.getPage) rather than to try to do something to the blocking code. Twisted has lots of client libraries. These libraries will generally give you a Deferred, which you're already handling.
If you're using it as a stand-in for actually waiting, then you can create a Deferred which waits with deferLater.
If you're using it as a stand-in for something CPU intensive that doesn't grab the GIL (like PIL image encoding, or a native XML parser), or an existing native / proprietary I/O layer (like Oracle or MSSQL database bindings) that you'd prefer not to rewrite to use Twisted properly, you can invoke it with deferToThread.
However you get your Deferred, you're almost set up to handle it. You just need to adjust say_hi to return one:
def say_hi(self, req):
req.setHeader("content-type", "text/html")
d = getADeferredSomehow()
def done(result):
req.write("hello!")
req.finish()
return d.addBoth(done)
Related
I have a python based page which recieves data by POST, which is then forwarded to the Crossbar server using Autobahn (Wamp). It works well the first 1-2 times but when it's called again after that it throws ReactorNotRestartable.
Now, I need this to work whichever way possible, either by reusing this "Reactor" based on a conditional check or by stopping it properly after every run. (The first one would be preferable because it might reduce the execution time)
Thanks for your help!
Edit:
This is in a webpage (Django View) so it needs to run as many times as the page is loaded/data is sent to it via POST.
from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.application.internet import ClientService
from autobahn.wamp.types import ComponentConfig
from autobahn.twisted.wamp import ApplicationSession, WampWebSocketClientFactory
class MyAppSession(ApplicationSession):
def __init__(self, config):
ApplicationSession.__init__(self, config)
def onConnect(self):
self.join(self.config.realm)
def onChallenge(self, challenge):
pass
#inlineCallbacks
def onJoin(self, details):
yield self.call('receive_data', data=message)
yield self.leave()
def onLeave(self, details):
self.disconnect()
def onDisconnect(self):
reactor.stop()
message = "data from POST[]"
session = MyAppSession(ComponentConfig('realm_1', {}))
transport = WampWebSocketClientFactory(session, url='ws://127.0.0.1:8080')
endpoint = TCP4ClientEndpoint(reactor, '127.0.0.1', 8080)
service = ClientService(endpoint, transport)
service.startService()
reactor.run()
I figured out a probably hacky-and-not-so-good way by using multiprocessing and putting reactor.stop() inside onJoin() right after the function call. This way I don't have to bother with the "twisted running in the main thread" thing because its process gets killed as soon as my work is done.
Is there a better way?
I have no idea how to use something like twisted.internet.loopingCall() in twisted.internet.ClientFactory
I need to write python script that scans directory for incoming files with phone numbers, reads them, and makes call using YATE yaypm python module that uses twisted library.
client_factory = yaypm.TCPDispatcherFactory(start_client)
reactor.connectTCP(host, port, client_factory)
reactor.run()
Where yaypm.TCPDispatcherFactory derived from twisted.internet.ClientFactory and start_client is the function that will be executed after successfull connection.
If start_client only makes demonstration call:
def start_client(client_yate):
d = dialer(client_yate)
d.call(caller, target)
Everything is OK.
(dialer is the object that implements yaypm.flow logic, full description placed in http://docs.yate.ro/wiki/YAYPM:Bridge_and_then_unbridge)
I need to write something like this in start_client
d = dialer(client_yate)
files = os.listdir(input_directory)
for filename in files:
<read caller and target numbers from file>
d.call(caller, target)
time.sleep(interval)
I know that using sleep function in the main thread leads to deadlock.
How should I implement the algorithm above?
twisted.internet.task.deferLater behaves like a sleep() call if you use it with the inlineCallbacks decorator. Here is a simplified example that uses ClientFactory:
from twisted.internet import reactor, task
from twisted.internet.defer import inlineCallbacks
from twisted.internet.protocol import Protocol, ClientFactory
class DoNothing(Protocol):
def __init__(self, connection_callback):
self.connection_callback = connection_callback
def connectionMade(self):
self.connection_callback()
return
class ConnectionClientFactory(ClientFactory):
def __init__(self, connection_callback):
self.connection_callback = connection_callback
def buildProtocol(self, addr):
return DoNothing(self.connection_callback)
def sleep(delay):
# Returns a deferred that calls do-nothing function
# after `delay` seconds
return task.deferLater(reactor, delay, lambda: None)
#inlineCallbacks
def repeat_forever(message):
while True:
print(message)
yield sleep(1)
if __name__ == '__main__':
repeat_forever('running')
factory = ConnectionClientFactory(lambda: repeat_forever('connected'))
reactor.connectTCP('example.com', 80, factory)
reactor.run()
The above code is essentially what your library does with the callback you pass in. As you can see, the call to repeat_forever('running') runs concurrently to the one called after the client connects.
Now I wrote ferver by this tutorial:
https://twistedmatrix.com/documents/14.0.0/web/howto/web-in-60/asynchronous-deferred.html
But it seems to be good only for delayng process, not actually concurently process 2 or more requests. My full code is:
from twisted.internet.task import deferLater
from twisted.web.resource import Resource
from twisted.web.server import Site, NOT_DONE_YET
from twisted.internet import reactor, threads
from time import sleep
class DelayedResource(Resource):
def _delayedRender(self, request):
print 'Sorry to keep you waiting.'
request.write("<html><body>Sorry to keep you waiting.</body></html>")
request.finish()
def make_delay(self, request):
print 'Sleeping'
sleep(5)
return request
def render_GET(self, request):
d = threads.deferToThread(self.make_delay, request)
d.addCallback(self._delayedRender)
return NOT_DONE_YET
def main():
root = Resource()
root.putChild("social", DelayedResource())
factory = Site(root)
reactor.listenTCP(8880, factory)
print 'started httpserver...'
reactor.run()
if __name__ == '__main__':
main()
But when I passing 2 requests console output is like:
Sleeping
Sorry to keep you waiting.
Sleeping
Sorry to keep you waiting.
But if it was concurrent it should be like:
Sleeping
Sleeping
Sorry to keep you waiting.
Sorry to keep you waiting.
So the question is how to make twisted not to wait until response is finished before processing next?
Also make_delayIRL is a large function with heavi logic. Basically I spawn lot of threads and make requests to other urls and collecting results intro response, so it can take some time and not easly to be ported
Twisted processes everything in one event loop. If somethings blocks the execution, it also blocks Twisted. So you have to prevent blocking calls.
In your case you have time.sleep(5). It is blocking. You found the better way to do it in Twisted already: deferLater(). It returns a Deferred that will continue execution after the given time and release the events loop so other things can be done meanwhile. In general all things that return a deferred are good.
If you have to do heavy work that for some reason can not be deferred, you should use deferToThread() to execute this work in a thread. See https://twistedmatrix.com/documents/15.5.0/core/howto/threading.html for details.
You can use greenlents in your code (like threads).
You need to install the geventreactor - https://gist.github.com/yann2192/3394661
And use reactor.deferToGreenlet()
Also
In your long-calculation code need to call gevent.sleep() for change context to another greenlet.
msecs = 5 * 1000
timeout = 100
for xrange(0, msecs, timeout):
sleep(timeout)
gevent.sleep()
I'm afraid I'm finding it difficult to work with the adbapi interface for sqlite3 ConnectionPools in twisted.
I've initialized my pool like this in a file I've named db.py:
from twisted.enterprise import adbapi
pool = adbapi.ConnectionPool("sqlite3", db=config.db_file)
pool.start()
def last(datatype, n):
cmd = "SELECT * FROM %s ORDER BY Timestamp DESC LIMIT %i" % (datatype, n)
return pool.runQuery(cmd)
Then, I'm importing db.py and using it inside a particular route handler. Unfortunately, it appears the callback is never triggered. datatype is printed, but response is never printed.
class DataHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self, datatype):
print datatype
data = db.last(datatype, 500)
data.addCallback(self.on_response)
def on_response(self, response):
print response
self.write(json.dumps(response))
self.finish()
Any ideas?
Mixing Tornado and Twisted requires special attention. Try this, as the first lines executed in your entire program:
import tornado.platform.twisted
tornado.platform.twisted.install()
Then, to start your server:
tornado.ioloop.IOLoop.current().start()
What's happening now is, you start the Tornado IOLoop but you never start the Twisted Reactor.
Your Twisted SQLite connection begins an IO operation when you run your query, but since the Reactor isn't running, the operation never completes. In order for the IOLoop and the Reactor to share your process you must run one of them on top of the other. Tornado provides a compatibility layer that allows you to do that.
I'm writing a small web server in Python, using BaseHTTPServer and a custom subclass of BaseHTTPServer.BaseHTTPRequestHandler. Is it possible to make this listen on more than one port?
What I'm doing now:
class MyRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def doGET
[...]
class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
pass
server = ThreadingHTTPServer(('localhost', 80), MyRequestHandler)
server.serve_forever()
Sure; just start two different servers on two different ports in two different threads that each use the same handler. Here's a complete, working example that I just wrote and tested. If you run this code then you'll be able to get a Hello World webpage at both http://localhost:1111/ and http://localhost:2222/
from threading import Thread
from SocketServer import ThreadingMixIn
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-type", "text/plain")
self.end_headers()
self.wfile.write("Hello World!")
class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
daemon_threads = True
def serve_on_port(port):
server = ThreadingHTTPServer(("localhost",port), Handler)
server.serve_forever()
Thread(target=serve_on_port, args=[1111]).start()
serve_on_port(2222)
update:
This also works with Python 3 but three lines need to be slightly changed:
from socketserver import ThreadingMixIn
from http.server import HTTPServer, BaseHTTPRequestHandler
and
self.wfile.write(bytes("Hello World!", "utf-8"))
Not easily. You could have two ThreadingHTTPServer instances, write your own serve_forever() function (don't worry it's not a complicated function).
The existing function:
def serve_forever(self, poll_interval=0.5):
"""Handle one request at a time until shutdown.
Polls for shutdown every poll_interval seconds. Ignores
self.timeout. If you need to do periodic tasks, do them in
another thread.
"""
self.__serving = True
self.__is_shut_down.clear()
while self.__serving:
# XXX: Consider using another file descriptor or
# connecting to the socket to wake this up instead of
# polling. Polling reduces our responsiveness to a
# shutdown request and wastes cpu at all other times.
r, w, e = select.select([self], [], [], poll_interval)
if r:
self._handle_request_noblock()
self.__is_shut_down.set()
So our replacement would be something like:
def serve_forever(server1,server2):
while True:
r,w,e = select.select([server1,server2],[],[],0)
if server1 in r:
server1.handle_request()
if server2 in r:
server2.handle_request()
I would say that threading for something this simple is overkill. You're better off using some form of asynchronous programming.
Here is an example using Twisted:
from twisted.internet import reactor
from twisted.web import resource, server
class MyResource(resource.Resource):
isLeaf = True
def render_GET(self, request):
return 'gotten'
site = server.Site(MyResource())
reactor.listenTCP(8000, site)
reactor.listenTCP(8001, site)
reactor.run()
I also thinks it looks a lot cleaner to have each port be handled in the same way, instead of having the main thread handle one port and an additional thread handle the other. Arguably that can be fixed in the thread example, but then you're using three threads.