I'm afraid I'm finding it difficult to work with the adbapi interface for sqlite3 ConnectionPools in twisted.
I've initialized my pool like this in a file I've named db.py:
from twisted.enterprise import adbapi
pool = adbapi.ConnectionPool("sqlite3", db=config.db_file)
pool.start()
def last(datatype, n):
cmd = "SELECT * FROM %s ORDER BY Timestamp DESC LIMIT %i" % (datatype, n)
return pool.runQuery(cmd)
Then, I'm importing db.py and using it inside a particular route handler. Unfortunately, it appears the callback is never triggered. datatype is printed, but response is never printed.
class DataHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self, datatype):
print datatype
data = db.last(datatype, 500)
data.addCallback(self.on_response)
def on_response(self, response):
print response
self.write(json.dumps(response))
self.finish()
Any ideas?
Mixing Tornado and Twisted requires special attention. Try this, as the first lines executed in your entire program:
import tornado.platform.twisted
tornado.platform.twisted.install()
Then, to start your server:
tornado.ioloop.IOLoop.current().start()
What's happening now is, you start the Tornado IOLoop but you never start the Twisted Reactor.
Your Twisted SQLite connection begins an IO operation when you run your query, but since the Reactor isn't running, the operation never completes. In order for the IOLoop and the Reactor to share your process you must run one of them on top of the other. Tornado provides a compatibility layer that allows you to do that.
Related
I have a python based page which recieves data by POST, which is then forwarded to the Crossbar server using Autobahn (Wamp). It works well the first 1-2 times but when it's called again after that it throws ReactorNotRestartable.
Now, I need this to work whichever way possible, either by reusing this "Reactor" based on a conditional check or by stopping it properly after every run. (The first one would be preferable because it might reduce the execution time)
Thanks for your help!
Edit:
This is in a webpage (Django View) so it needs to run as many times as the page is loaded/data is sent to it via POST.
from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.application.internet import ClientService
from autobahn.wamp.types import ComponentConfig
from autobahn.twisted.wamp import ApplicationSession, WampWebSocketClientFactory
class MyAppSession(ApplicationSession):
def __init__(self, config):
ApplicationSession.__init__(self, config)
def onConnect(self):
self.join(self.config.realm)
def onChallenge(self, challenge):
pass
#inlineCallbacks
def onJoin(self, details):
yield self.call('receive_data', data=message)
yield self.leave()
def onLeave(self, details):
self.disconnect()
def onDisconnect(self):
reactor.stop()
message = "data from POST[]"
session = MyAppSession(ComponentConfig('realm_1', {}))
transport = WampWebSocketClientFactory(session, url='ws://127.0.0.1:8080')
endpoint = TCP4ClientEndpoint(reactor, '127.0.0.1', 8080)
service = ClientService(endpoint, transport)
service.startService()
reactor.run()
I figured out a probably hacky-and-not-so-good way by using multiprocessing and putting reactor.stop() inside onJoin() right after the function call. This way I don't have to bother with the "twisted running in the main thread" thing because its process gets killed as soon as my work is done.
Is there a better way?
Now I wrote ferver by this tutorial:
https://twistedmatrix.com/documents/14.0.0/web/howto/web-in-60/asynchronous-deferred.html
But it seems to be good only for delayng process, not actually concurently process 2 or more requests. My full code is:
from twisted.internet.task import deferLater
from twisted.web.resource import Resource
from twisted.web.server import Site, NOT_DONE_YET
from twisted.internet import reactor, threads
from time import sleep
class DelayedResource(Resource):
def _delayedRender(self, request):
print 'Sorry to keep you waiting.'
request.write("<html><body>Sorry to keep you waiting.</body></html>")
request.finish()
def make_delay(self, request):
print 'Sleeping'
sleep(5)
return request
def render_GET(self, request):
d = threads.deferToThread(self.make_delay, request)
d.addCallback(self._delayedRender)
return NOT_DONE_YET
def main():
root = Resource()
root.putChild("social", DelayedResource())
factory = Site(root)
reactor.listenTCP(8880, factory)
print 'started httpserver...'
reactor.run()
if __name__ == '__main__':
main()
But when I passing 2 requests console output is like:
Sleeping
Sorry to keep you waiting.
Sleeping
Sorry to keep you waiting.
But if it was concurrent it should be like:
Sleeping
Sleeping
Sorry to keep you waiting.
Sorry to keep you waiting.
So the question is how to make twisted not to wait until response is finished before processing next?
Also make_delayIRL is a large function with heavi logic. Basically I spawn lot of threads and make requests to other urls and collecting results intro response, so it can take some time and not easly to be ported
Twisted processes everything in one event loop. If somethings blocks the execution, it also blocks Twisted. So you have to prevent blocking calls.
In your case you have time.sleep(5). It is blocking. You found the better way to do it in Twisted already: deferLater(). It returns a Deferred that will continue execution after the given time and release the events loop so other things can be done meanwhile. In general all things that return a deferred are good.
If you have to do heavy work that for some reason can not be deferred, you should use deferToThread() to execute this work in a thread. See https://twistedmatrix.com/documents/15.5.0/core/howto/threading.html for details.
You can use greenlents in your code (like threads).
You need to install the geventreactor - https://gist.github.com/yann2192/3394661
And use reactor.deferToGreenlet()
Also
In your long-calculation code need to call gevent.sleep() for change context to another greenlet.
msecs = 5 * 1000
timeout = 100
for xrange(0, msecs, timeout):
sleep(timeout)
gevent.sleep()
The code below is taken from Twisted's documentation on AMP (link). When a callback is added to d, there's automatically a "protocol" argument added to it and the deferred is automatically run when reactor.run() is called.
def connect():
endpoint = TCP4ClientEndpoint(reactor, "127.0.0.1", 8750)
factory = Factory()
factory.protocol = AMP
return endpoint.connect(factory)
d = connect()
def connected(protocol):
return protocol.callRemote(
RegisterUser,
username=u'alice'
d.addCallback(connected)
reactor.run()
In my code, everything is exactly the same, except I've been using pyglet-twisted (link) with cocos2d so I can't call reactor.run() because the reactor starts at the same time as the application.
If I call reactor.run(), I get an error saying the reactor is already running.
If I don't, the deferred doesn't seem to get called.
I've been trying to call it with reactor.callLater, reactor.callWhenRunning, but both need an argument. Passing in None doesn't work.
So my question is, how should I make this this deferred run without calling reactor.run().
Thanks!
Few of Twisted's APIs will succeed without a running reactor. The reactor is responsible for doing the I/O. You must have a running reactor in order to set up a connection (regardless of whether you are using an endpoint object or some other API to do so).
As far as I can tell, the pyglet integration reactor does not automatically start itself. Something must call its run method. Your question suggests that you are not calling it, so I'm quite curious what is calling it.
When I modify your example to make it complete and runnable and add error reporting, like this:
from pygletreactor import install
install()
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.internet.protocol import Factory
from twisted.protocols.amp import AMP
from twisted.python.log import err
def connect():
endpoint = TCP4ClientEndpoint(reactor, "127.0.0.1", 8750)
factory = Factory()
factory.protocol = AMP
return endpoint.connect(factory)
d = connect()
def connected(protocol):
return protocol.callRemote(
RegisterUser,
username=u'alice')
d.addCallback(connected)
d.addErrback(err)
reactor.run()
Then I get the behavior I expect, which is for the connection to be attempted and then fail (because I am not running an AMP server anywhere):
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
Perhaps you can compare this to your complete program and find an important difference.
After more research, I found the reason the deferred returned from endpoint.connect() wasn't being called was a bug with cocos2d.
At the bottom of cocos.director, where pyglet.app.event_loop is assigned to handle the director's events in line director.event = event_loop.event.
This needs to be changed to use pygletreactor's eventloop instead. So the above code needs to be changed to the following:
import pygletreactor
event_loop = pygletreactor.EventLoop()
director = Director()
director.event = event_loop.event
How can I do async request processing in Twisted like in Node.js?
I wrote sample with Twisted, but my app still waited an answer from long operation(I emulate this with time.sleep).
Also I don't understand how can I use reactor.callLater correct.
This is my sample of twisted app.
from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.defer import Deferred
import time
class Hello(Resource):
def getChild(self, name, request):
if name == '':
return self
print name
return Resource.getChild(self, name, request)
def render_GET(self, req):
d = Deferred()
reactor.callLater(2, d.callback, None)
d.addCallback(lambda _: self.say_hi(req))
return server.NOT_DONE_YET
def say_hi(self, req):
req.setHeader("content-type", "text/html")
time.sleep(5)
req.write("hello!")
req.finish()
class Hello2(Resource):
isLeaf = True
def render_GET(self, req):
req.setHeader("content-type", "text/html")
return "hello2!"
root = Hello()
root.putChild("2", Hello2())
reactor.listenTCP(8080, server.Site(root))
reactor.run()
Edit: Now question is how to write a sync code? Please example.
You're already doing it... sort of.
Your problem here is that time.sleep() is a blocking call, and will therefore make your whole server stop.
If you're using that as a stand-in for something that does network I/O (like urllib), the best option is usually to do the I/O with Twisted (like twisted.web.client.getPage) rather than to try to do something to the blocking code. Twisted has lots of client libraries. These libraries will generally give you a Deferred, which you're already handling.
If you're using it as a stand-in for actually waiting, then you can create a Deferred which waits with deferLater.
If you're using it as a stand-in for something CPU intensive that doesn't grab the GIL (like PIL image encoding, or a native XML parser), or an existing native / proprietary I/O layer (like Oracle or MSSQL database bindings) that you'd prefer not to rewrite to use Twisted properly, you can invoke it with deferToThread.
However you get your Deferred, you're almost set up to handle it. You just need to adjust say_hi to return one:
def say_hi(self, req):
req.setHeader("content-type", "text/html")
d = getADeferredSomehow()
def done(result):
req.write("hello!")
req.finish()
return d.addBoth(done)
Goal: Show data from server in wxPython GUI on client
Newcomer to Twisted. I have a wxPython GUI running on a Windows 7 client, and I have a program running on an Ubuntu server that produces a log. My current attempt is to tail -f the log, pipe the output to a twisted server, then serve any data that meets my regex conditions to the client. I already have a tunnel open, so I don't need to complicate things with SSH. I've gotten the following block of code running, but it only serves the first line in the input. I know I need to keep checking the input for a newline and then writing it to the transport, but I'm not sure how to do that without breaking the connection.
I haven't been able to find enough information to patch a full solution together. I have also tried various other methods using sockets and file IO, but I think Twisted seems to be a good tool for this issue. Am I on the right track? Any recommendations appreciated. Thanks
#! /usr/bin/python
import optparse, os, sys
from twisted.internet.protocol import ServerFactory, Protocol
def parse_args():
usage = """usage: %prog [options]
"""
parser = optparse.OptionParser(usage)
help = "The port to listen on. Default to a random available port."
parser.add_option('--port', type='int', help=help)
help = "The interface to listen on. Default is localhost."
parser.add_option('--iface', help=help, default='localhost')
options =parser.parse_args()
return options#, log_file
class LogProtocol(Protocol):
def connectionMade(self):
for line in self.factory.log:
self.transport.write(line)
class LogFactory(ServerFactory):
protocol = LogProtocol
def __init__(self,log):
self.log = log
def main():
log = sys.stdin.readline()
options, log_file = parse_args()
factory = LogFactory(log)
from twisted.internet import reactor
port = reactor.listenTCP(options.port or 0, factory,
interface=options.iface)
print 'Serving %s on %s.' % (log_file, port.getHost())
reactor.run()
if __name__ == '__main__':
main()
To answer the first comment, I have also tried to just read the log from within Python, program hangs. Code follows:
#! /usr/bin/python
import optparse, os, sys, time
from twisted.internet.protocol import ServerFactory, Protocol
def parse_args():
usage = """ usage: %prog [options]"""
parser = optparse.OptionParser(usage)
help = "The port to listen on. Default to a random available port"
parser.add_option('--port', type='int', help=help, dest="port")
help = "The logfile to tail and write"
parser.add_option('--file', help=help, default='log/testgen01.log',dest="logfile")
options = parser.parse_args()
return options
class LogProtocol(Protocol):
def connectionMade(self):
for line in self.follow():
self.transport.write(line)
self.transport.loseConnection()
def follow(self):
while True:
line = self.factory.log.readline()
if not line:
time.sleep(0.1)
continue
yield line
class LogFactory(ServerFactory):
protocol = LogProtocol
def __init__(self,log):
self.log = log
def main():
options, log_file = parse_args()
log = open(options.logfile)
factory = LogFactory(log)
from twisted.internet import reactor
port = reactor.listenTCP(options.port or 0, factory) #,interface=options.iface)
print 'Serving %s on %s.' % (options.logfile, port.getHost())
reactor.run()
if __name__ == '__main__':
main()
You've got a few different easily separated goals you're trying to achieve here. First, I'll talk about watching the log file.
Your generator has a couple problems. One of them is big - it calls time.sleep(0.1). The sleep function blocks for the amount of time passed to it. While it is blocking, the thread which called it can't do anything else (that's roughly what "blocking" means, after all). You're iterating over the generator in the same thread as LogProtocol.connectionMade is called in (since connectionMade calls follow). LogProtocol.connectionMade is called in the same thread as the Twisted reactor is running, because Twisted is roughly single threaded.
So, you're blocking the reactor with the sleep calls. As long as sleep is blocking the reactor, the reactor can't do anything - like send bytes over sockets. Blocking is transitive, by the way. So LogProtocol.connectionMade is an even bigger problem: it iterates indefinitely, sleeping and reading. So it blocks the reactor indefinitely.
You need to read lines from the file without blocking. You can do this by polling - which is effectively the approach you're taking now - but avoiding the sleep call. Use reactor.callLater to schedule future reads from the file:
def follow(fObj):
line = fObj.readline()
reactor.callLater(0.1, follow, fObj)
follow(open(filename))
You can also let LoopingCall deal with the part that makes this a loop that runs forever:
def follow(fObj):
line = fObj.readline()
from twisted.internet.task import LoopingCall
loop = LoopingCall(follow, open(filename))
loop.start(0.1)
Either of these will let you read new lines from the file over time without blocking the reactor. Of course, they both just drop the line on the floor after they read it. This leads me to the second problem...
You need to react to the appearance of a new line in the file. Presumably you want to write it out to your connection. This isn't too hard: "reacting" is pretty easy, it usually just means calling a function or a method. In this case, it's easiest to have the LogProtocol set up the log following and supply a callback object to handle lines when they appear. Consider this slight adjustment to the follow function from above:
def follow(fObj, gotLine):
line = fObj.readline()
if line:
gotLine(line)
def printLine(line):
print line
loop = LoopingCall(follow, open(filename), printLine)
loop.start(0.1)
Now you can non-blockingly poll a log file for new lines and learn when one has actually shown up. This is simple to integrate with LogProtocol...
class LogProtocol(Protocol):
def connectionMade(self):
self.loop = LoopingCall(follow, open(filename), self._sendLogLine)
self.loop.start()
def _sendLogLine(self, line):
self.transport.write(line)
One last detail is that you probably want to stop watching the file when the connection is lost:
def connectionLost(self, reason):
self.loop.stop()
So, this solution avoids blocking by using LoopingCall instead of time.sleep and pushes lines to the protocol when they're found using simple method calls.