Develop Reference

Python CGI simultaneous AJAX requests

So here's the deal : I'm writing a simple lightweight IRC app, hosted locally, that basically does the same job as Xchat and works in your browser, just as Sabnzbd. I display search results in the browser as an html table, and using an AJAX GET request with an on_click event, the download is launched. I use another AJAX GET request in a 1 second loop to request the download information (status, progress, speed, ETA, etc.). I hit a bump with the simultaneous AJAX requests, since my CGI handler seems to only be able to handle one thread at a time : indeed, the main thread processes the download, while requests for download status are sent too.
Since I had a Django app somewhere, I tried implementing this IRC app and everything works fine. Simultaneous requests are handled properly.
So is there something I have to know with the HTTP handler ? Is it not possible with the basic CGI handle to deal with simultaneous requests ?
I use the following for my CGI IRC app :
from http.server import BaseHTTPRequestHandler, HTTPServer, CGIHTTPRequestHandler
If it's not about theory but about my code, I can gladly post various python scripts if it helps.

A little bit deeper into the documentation:
These four classes process requests synchronously; each request must be completed before the next request can be started.
TL;DR: Use a real web server.

So, after further research, here's my code, whick works :
from http.server import BaseHTTPRequestHandler, HTTPServer, CGIHTTPRequestHandler
from socketserver import ThreadingMixIn
import threading
import cgitb; cgitb.enable() ## This line enables CGI error reporting
import webbrowser
class HTTPRequestHandler(CGIHTTPRequestHandler):
"""Handle requests in a separate thread."""
def do_GET(self):
if "shutdown" in self.path:
self.send_head()
print ("shutdown")
server.stop()
else:
self.send_head()
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
allow_reuse_address = True
daemon_threads = True
def shutdown(self):
self.socket.close()
HTTPServer.shutdown(self)
class SimpleHttpServer():
def __init__(self, ip, port):
self.server = ThreadedHTTPServer((ip,port), HTTPRequestHandler)
self.status = 1
def start(self):
self.server_thread = threading.Thread(target=self.server.serve_forever)
self.server_thread.daemon = True
self.server_thread.start()
def waitForThread(self):
self.server_thread.join()
def stop(self):
self.server.shutdown()
self.waitForThread()
if __name__=='__main__':
HTTPRequestHandler.cgi_directories = ["/", "/ircapp"]
server = SimpleHttpServer('localhost', 8020)
print ('HTTP Server Running...........')
webbrowser.open_new_tab('http://localhost:8020/ircapp/search.py')
server.start()
server.waitForThread()

Tornado adbapi callback not running

I'm afraid I'm finding it difficult to work with the adbapi interface for sqlite3 ConnectionPools in twisted.
I've initialized my pool like this in a file I've named db.py:
from twisted.enterprise import adbapi
pool = adbapi.ConnectionPool("sqlite3", db=config.db_file)
pool.start()
def last(datatype, n):
cmd = "SELECT * FROM %s ORDER BY Timestamp DESC LIMIT %i" % (datatype, n)
return pool.runQuery(cmd)
Then, I'm importing db.py and using it inside a particular route handler. Unfortunately, it appears the callback is never triggered. datatype is printed, but response is never printed.
class DataHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
def get(self, datatype):
print datatype
data = db.last(datatype, 500)
data.addCallback(self.on_response)
def on_response(self, response):
print response
self.write(json.dumps(response))
self.finish()
Any ideas?

Mixing Tornado and Twisted requires special attention. Try this, as the first lines executed in your entire program:
import tornado.platform.twisted
tornado.platform.twisted.install()
Then, to start your server:
tornado.ioloop.IOLoop.current().start()
What's happening now is, you start the Tornado IOLoop but you never start the Twisted Reactor.
Your Twisted SQLite connection begins an IO operation when you run your query, but since the Reactor isn't running, the operation never completes. In order for the IOLoop and the Reactor to share your process you must run one of them on top of the other. Tornado provides a compatibility layer that allows you to do that.

txredisapi subscribe and listen async

I'm working in a project with Python, Twisted and Redis. So the team decided to use txredisapi for the communication between the Python modules and Redis. This project does a lot of different things and we need to subscribe to several channels for listen the messages sent by Redis without the other functionalities stops (asynchronously).
Can one execution handle all the work and listen the messages sent by Redis at the same time or must we separate and execute the code in differents flows?
We use the following code for listen the messages:
import txredisapi as redis
class RedisListenerProtocol(redis.SubscriberProtocol):
def connectionMade(self):
self.subscribe("channelName")
def messageReceived(self, pattern, channel, message):
print "pattern=%s, channel=%s message=%s" %(pattern, channel, message)
def connectionLost(self, reason):
print "lost connection:", reason
class RedisListenerFactory(redis.SubscriberFactory):
maxDelay = 120
continueTrying = True
protocol = RedisListenerProtocol
We try to listen the messages with:
self.connRedisChannels = yield redis.ConnectionPool()
I'm interested to know how can I specify that the Connection must use the "RedisListenerFactory", then I guess that the function "messageReceived" will be fired when a message arrives.
Any suggestions, example or correction will be apreciated.
Thanks!
The following code solves the problem:
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(HOST, PORT)
Thanks to Philippe T. for the help.

If you want to use directly the redis.Connection() may be you can do this before:
redis.SubscriberFactory.protocol = RedisListenerProtocol
the package make internal call to is factory for connection.
other way is to rewrite *Connection class and make*Connection factory to use your factory.
to make the connection on other part of your code you can do something like this :
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
# some where :
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(__HOST__, __PORT__)
# the defer will have your client when the connection is done

Twisted async request processing

How can I do async request processing in Twisted like in Node.js?
I wrote sample with Twisted, but my app still waited an answer from long operation(I emulate this with time.sleep).
Also I don't understand how can I use reactor.callLater correct.
This is my sample of twisted app.
from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.defer import Deferred
import time
class Hello(Resource):
def getChild(self, name, request):
if name == '':
return self
print name
return Resource.getChild(self, name, request)
def render_GET(self, req):
d = Deferred()
reactor.callLater(2, d.callback, None)
d.addCallback(lambda _: self.say_hi(req))
return server.NOT_DONE_YET
def say_hi(self, req):
req.setHeader("content-type", "text/html")
time.sleep(5)
req.write("hello!")
req.finish()
class Hello2(Resource):
isLeaf = True
def render_GET(self, req):
req.setHeader("content-type", "text/html")
return "hello2!"
root = Hello()
root.putChild("2", Hello2())
reactor.listenTCP(8080, server.Site(root))
reactor.run()
Edit: Now question is how to write a sync code? Please example.

You're already doing it... sort of.
Your problem here is that time.sleep() is a blocking call, and will therefore make your whole server stop.
If you're using that as a stand-in for something that does network I/O (like urllib), the best option is usually to do the I/O with Twisted (like twisted.web.client.getPage) rather than to try to do something to the blocking code. Twisted has lots of client libraries. These libraries will generally give you a Deferred, which you're already handling.
If you're using it as a stand-in for actually waiting, then you can create a Deferred which waits with deferLater.
If you're using it as a stand-in for something CPU intensive that doesn't grab the GIL (like PIL image encoding, or a native XML parser), or an existing native / proprietary I/O layer (like Oracle or MSSQL database bindings) that you'd prefer not to rewrite to use Twisted properly, you can invoke it with deferToThread.
However you get your Deferred, you're almost set up to handle it. You just need to adjust say_hi to return one:
def say_hi(self, req):
req.setHeader("content-type", "text/html")
d = getADeferredSomehow()
def done(result):
req.write("hello!")
req.finish()
return d.addBoth(done)

How do I write a python HTTP server to listen on multiple ports?

I'm writing a small web server in Python, using BaseHTTPServer and a custom subclass of BaseHTTPServer.BaseHTTPRequestHandler. Is it possible to make this listen on more than one port?
What I'm doing now:
class MyRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def doGET
[...]
class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
pass
server = ThreadingHTTPServer(('localhost', 80), MyRequestHandler)
server.serve_forever()

Sure; just start two different servers on two different ports in two different threads that each use the same handler. Here's a complete, working example that I just wrote and tested. If you run this code then you'll be able to get a Hello World webpage at both http://localhost:1111/ and http://localhost:2222/
from threading import Thread
from SocketServer import ThreadingMixIn
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-type", "text/plain")
self.end_headers()
self.wfile.write("Hello World!")
class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
daemon_threads = True
def serve_on_port(port):
server = ThreadingHTTPServer(("localhost",port), Handler)
server.serve_forever()
Thread(target=serve_on_port, args=[1111]).start()
serve_on_port(2222)
update:
This also works with Python 3 but three lines need to be slightly changed:
from socketserver import ThreadingMixIn
from http.server import HTTPServer, BaseHTTPRequestHandler
and
self.wfile.write(bytes("Hello World!", "utf-8"))

Not easily. You could have two ThreadingHTTPServer instances, write your own serve_forever() function (don't worry it's not a complicated function).
The existing function:
def serve_forever(self, poll_interval=0.5):
"""Handle one request at a time until shutdown.
Polls for shutdown every poll_interval seconds. Ignores
self.timeout. If you need to do periodic tasks, do them in
another thread.
"""
self.__serving = True
self.__is_shut_down.clear()
while self.__serving:
# XXX: Consider using another file descriptor or
# connecting to the socket to wake this up instead of
# polling. Polling reduces our responsiveness to a
# shutdown request and wastes cpu at all other times.
r, w, e = select.select([self], [], [], poll_interval)
if r:
self._handle_request_noblock()
self.__is_shut_down.set()
So our replacement would be something like:
def serve_forever(server1,server2):
while True:
r,w,e = select.select([server1,server2],[],[],0)
if server1 in r:
server1.handle_request()
if server2 in r:
server2.handle_request()

I would say that threading for something this simple is overkill. You're better off using some form of asynchronous programming.
Here is an example using Twisted:
from twisted.internet import reactor
from twisted.web import resource, server
class MyResource(resource.Resource):
isLeaf = True
def render_GET(self, request):
return 'gotten'
site = server.Site(MyResource())
reactor.listenTCP(8000, site)
reactor.listenTCP(8001, site)
reactor.run()
I also thinks it looks a lot cleaner to have each port be handled in the same way, instead of having the main thread handle one port and an additional thread handle the other. Arguably that can be fixed in the thread example, but then you're using three threads.