Develop Reference

Using Twisted and Python to create a Router

When a client (ie. web browser) points to localhost:8080, its request should be sent to an address defined by SERVER_ADDR such as http://www.yahoo.com. It's more like a router or load balancer, than a proxy.
Problem: When my web browser points to localhost:8080, nothing is returned to it. On the Python console, the HTTP request header can be seen to have reached the Twisted app. Maybe this cannot be used for redirecting the browser's request?
from twisted.internet import protocol, reactor
LISTEN_PORT = 8080
SERVER_PORT = 80
SERVER_ADDR = 'http://www.yahoo.com'
class ServerProtocol(protocol.Protocol):
def __init__(self):
self.buffer = None
self.client = None
def connectionMade(self):
factory = protocol.ClientFactory()
factory.protocol = ClientProtocol
factory.server = self
reactor.connectTCP(SERVER_ADDR, SERVER_PORT, factory)
# Client => Proxy
def dataReceived(self, data):
print 'Data received from Client:'
if self.client:
self.client.write(data)
else:
data = data.replace('localhost:8080', SERVER_ADDR)
print data
self.buffer = data
# Proxy => Client
def write(self, data):
self.transport.write(data)
class ClientProtocol(protocol.Protocol):
def connectionMade(self):
self.factory.server.client = self
self.write(self.factory.server.buffer)
self.factory.server.buffer = ''
# Server => Proxy
def dataReceived(self, data):
print 'Data received from Server: '
print data
self.factory.server.write(data)
# Proxy => Server
def write(self, data):
if data:
self.transport.write(data)
def main():
factory = protocol.ServerFactory()
factory.protocol = ServerProtocol
reactor.listenTCP(LISTEN_PORT, factory)
reactor.run()
if __name__ == '__main__':
main()

It partially works for me: I get an HTTP error 400 bad request from the Yahoo server in my browser when I visit http://localhost:8080. This is because you're replacing the Host: localhost:8080 section of the GET request with Host: http://www.yahoo.com and the protocol here is invalid. It should just be Host: www.yahoo.com.
That said, it then responds with a HTTP/1.1 301 Moved Permanently redirect and it you run into other problems.
Edit: you should take a look at the proxy example on the twisted website (https://twistedmatrix.com/documents/12.3.0/web/examples/#auto2) and dive into the source to see how it's implemented.
from twisted.web import proxy, http
from twisted.internet import reactor
class ProxyFactory(http.HTTPFactory):
def buildProtocol(self, addr):
return proxy.Proxy()
reactor.listenTCP(8080, ProxyFactory())
reactor.run()

Understanding Autobahn and Twisted integration

I am trying to understand the examples given here: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/wamp/basic/pubsub/basic
I built this script which is supposed to handle multiple pub/sub websocket connections and also open a tcp port ( 8123 ) for incoming control messages. When a message comes on the 8123 port, the application should broadcast to all the connected subscribers the message received on port 8123. How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message.
Another thing that i do not understand is the url. The client javascript connects to the url http://:8080/ws . Where does the "ws" come from ?
Also can someone explain the purpose of RouterFactory, RouterSessionFactory and this bit:
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
my code is below:
import sys, time
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.defer import inlineCallbacks
from autobahn.twisted.wamp import ApplicationSession
from autobahn.twisted.util import sleep
class NotificationProtocol(Protocol):
def __init__(self, factory):
self.factory = factory
def dataReceived(self, data):
print "received new data"
class NotificationFactory(Factory):
protocol = NotificationProtocol
class WsNotificationComponent(ApplicationSession):
#inlineCallbacks
def onJoin(self, details):
counter = 0
while True:
self.publish("com.myapp.topic1", "test %d" % counter )
counter += 1
yield sleep(1)
## we use an Autobahn utility to install the "best" available Twisted reactor
##
from autobahn.twisted.choosereactor import install_reactor
reactor = install_reactor()
## create a WAMP router factory
##
from autobahn.wamp.router import RouterFactory
router_factory = RouterFactory()
## create a WAMP router session factory
##
from autobahn.twisted.wamp import RouterSessionFactory
session_factory = RouterSessionFactory(router_factory)
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
from autobahn.twisted.websocket import WampWebSocketServerFactory
transport_factory = WampWebSocketServerFactory(session_factory)
transport_factory.setProtocolOptions(failByDrop = False)
from twisted.internet.endpoints import serverFromString
## start the server from an endpoint
##
server = serverFromString(reactor, "tcp:8080")
server.listen(transport_factory)
notificationFactory = NotificationFactory()
reactor.listenTCP(8123, notificationFactory)
reactor.run()

"How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message":
Check out one of my other answers on SO: Persistent connection in twisted. Jump down to the example code and model your websocket logic like the "IO" logic and you'll have a good fit (You might also want to see the follow-on answer about the newer endpoint calls from one of the twisted core-team too)
"Where does the "ws" come from ?"
Websockets are implemented by retasking http connections, which by their nature have to have a specific path on the request. That "ws" path typically would map to a special http handler that autobahn is building for you to process websockets (or at least that's what your javascript is expecting...). Assuming thing are setup right you can actually point your web-browswer at that url and it should print back an error about the websocket handshake (Expected WebSocket Headers in my case, but I'm using cyclones websockets not autobahn).
P.S. one of the cool side-effects from "websockets must have a specific path" is that you can actually mix websockets and normal http content on the same handler/listen/port, this gets really handy when your trying to run them all on the same SSL port because your trying to avoid the requirement of a proxy front-ending your code.

python twisted simple server client

I am trying to build a simple "quote of the day" server and client modified from the twisted tutorial documentation. I want the "quote of the day" to be printed from the client to prove the communication. However, from what I can tell the client is not connecting. Here is my code.
Server
from twisted.python import log
from twisted.internet.protocol import Protocol
from twisted.internet.protocol import Factory
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor
class QOTD(Protocol):
def connectionMade(self):
self.transport.write("An apple a day keeps the doctor away\r\n")
self.transport.loseConnection()
class QOTDFactory(Factory):
def buildProtocol(self, addr):
return QOTD()
# 8007 is the port you want to run under. Choose something >1024
endpoint = TCP4ServerEndpoint(reactor, 8007)
endpoint.listen(QOTDFactory())
reactor.run()
Client
import sys
from twisted.python import log
from twisted.internet import reactor
from twisted.internet.protocol import Factory, Protocol
from twisted.internet.endpoints import TCP4ClientEndpoint
class SimpleClient(Protocol):
def connectionMade(self):
log.msg("connection made")
#self.transport.loseConnection()
def lineReceived(self, line):
print "receive:", line
class SimpleClientFactory(Factory):
def buildProtocol(self, addr):
return SimpleClient()
def startlog():
log.startLogging(sys.stdout)
point = TCP4ClientEndpoint(reactor, "localhost", 8007)
d = point.connect(SimpleClientFactory)
reactor.callLater(0.1, startlog)
reactor.run()

pass instance of SimpleClientFactory, not the class itself to point.connect()
subclass SimpleClient from twisted.protocol.basic.LineReceiver instead of Protocol if you use lineReceived
call addErrback on the results of endpoint.listen and point.connect to handle errors

Python Proxy with Twisted

Hello! I have this code:
from twisted.web import proxy, http
from twisted.internet import reactor
class akaProxy(proxy.Proxy):
"""
Local proxy = bridge between browser and web application
"""
def dataReceived(self, data):
print "Received data..."
headers = data.split("\n")
request = headers[0].split(" ")
method = request[0].lower()
action = request[1]
print action
print "ended content manipulation"
return proxy.Proxy.dataReceived(self, data)
class ProxyFactory(http.HTTPFactory):
protocol = akaProxy
def intercept(port):
print "Intercept"
try:
factory = ProxyFactory()
reactor.listenTCP(port, factory)
reactor.run()
except Exception as excp:
print str(excp)
intercept(1337)
I use above code to intercept everything between browser and web site. When using above, I configure my browser settings: to IP: 127.0.0.1 and Port: 1337. I put this script in remote server to act my remote server as proxy server. But when I change browser proxy IP settings to my server's it does not work. What I do wrong? What else I need to configure?

Presumably your dataReceived is raising an exception during its attempts to parse the data passed to it. Try enabling logging so you can see more of what's going on:
from twisted.python.log import startLogging
from sys import stdout
startLogging(stdout)
The reason your parser is likely to raise exceptions is that dataReceived is not called only with a complete request. It is called with whatever bytes are read from the TCP connection. This may be a complete request, a partial request, or even two requests (if pipelining is in use).

dataReceived in the Proxy context is handling "translation of rawData into lines", so it may be too early for trying your manipulation code. You can try overriding allContentReceived instead and you will have access to the complete headers and content. Here is an example that I believe does what you are after:
#!/usr/bin/env python
from twisted.web import proxy, http
class SnifferProxy(proxy.Proxy):
"""
Local proxy = bridge between browser and web application
"""
def allContentReceived(self):
print "Received data..."
print "method = %s" % self._command
print "action = %s" % self._path
print "ended content manipulation\n\n"
return proxy.Proxy.allContentReceived(self)
class ProxyFactory(http.HTTPFactory):
protocol = SnifferProxy
if __name__ == "__main__":
from twisted.internet import reactor
reactor.listenTCP(8080, ProxyFactory())
reactor.run()

Manually giving the twisted (web) network stack a packet to process?

I am running an HTTP server using the twisted framework. Is there any way I can "manually" ask it to process some payload? For example, if I've constructed some Ethernet frame can I ask twisted's reactor to handle it just as if it had just arrived on my network card?

You can do something like this:
from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, ClientFactory
class SomeWebThing(Resource):
def render_GET(self, request):
return "hello\n"
class SomeClient(Protocol):
def dataReceived(self, data):
p = self.factory.site.buildProtocol(self.transport.addr)
p.transport = self.transport
p.dataReceived(data)
class SomeClientFactory(ClientFactory):
protocol = SomeClient
def __init__(self, site):
self.site = site
if __name__ == '__main__':
root = Resource()
root.putChild('thing', SomeWebThing())
site = server.Site(root)
reactor.listenTCP(8000, site)
factory = SomeClientFactory(site)
reactor.connectTCP('localhost', 9000, factory)
reactor.run()
and save it as simpleinjecter.py, if you then do (from the commandline):
echo -e "GET /thing HTTP/1.1\r\n\r\n" | nc -l 9000 # runs a server, ready to send req to first client connection
python simpleinjecter.py
it should work as expected, with the request from the nc server on port 9000 getting funneled as the payload into the twisted web server, and the response coming back as expected.
The key lines are in SomeClient.dataRecieved(). You'll need a transport object with the right methods -- in the example above, I just steal the object from the client connection. If you aren't going to do that, I imagine you'll have to make one up, as the stack will want to do things like call getPeer() on it.

What is the use-case?
Perhaps you want to create your own Datagram Protocol
At the base, the place where you
actually implement the protocol
parsing and handling, is the
DatagramProtocol class. This class
will usually be decended from twisted.internet.protocol.DatagramProtocol.
Most protocol handlers inherit either
from this class or from one of its
convenience children. The
DatagramProtocol class receives
datagrams, and can send them out over
the network. Received datagrams
include the address they were sent
from, and when sending datagrams the
address to send to must be specified.
If you want to see wire-level transmissions rather than inject them, install and run WireShark, the fantastic, free packet sniffer.