I am running an HTTP server using the twisted framework. Is there any way I can "manually" ask it to process some payload? For example, if I've constructed some Ethernet frame can I ask twisted's reactor to handle it just as if it had just arrived on my network card?
You can do something like this:
from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, ClientFactory
class SomeWebThing(Resource):
def render_GET(self, request):
return "hello\n"
class SomeClient(Protocol):
def dataReceived(self, data):
p = self.factory.site.buildProtocol(self.transport.addr)
p.transport = self.transport
p.dataReceived(data)
class SomeClientFactory(ClientFactory):
protocol = SomeClient
def __init__(self, site):
self.site = site
if __name__ == '__main__':
root = Resource()
root.putChild('thing', SomeWebThing())
site = server.Site(root)
reactor.listenTCP(8000, site)
factory = SomeClientFactory(site)
reactor.connectTCP('localhost', 9000, factory)
reactor.run()
and save it as simpleinjecter.py, if you then do (from the commandline):
echo -e "GET /thing HTTP/1.1\r\n\r\n" | nc -l 9000 # runs a server, ready to send req to first client connection
python simpleinjecter.py
it should work as expected, with the request from the nc server on port 9000 getting funneled as the payload into the twisted web server, and the response coming back as expected.
The key lines are in SomeClient.dataRecieved(). You'll need a transport object with the right methods -- in the example above, I just steal the object from the client connection. If you aren't going to do that, I imagine you'll have to make one up, as the stack will want to do things like call getPeer() on it.
What is the use-case?
Perhaps you want to create your own Datagram Protocol
At the base, the place where you
actually implement the protocol
parsing and handling, is the
DatagramProtocol class. This class
will usually be decended from twisted.internet.protocol.DatagramProtocol.
Most protocol handlers inherit either
from this class or from one of its
convenience children. The
DatagramProtocol class receives
datagrams, and can send them out over
the network. Received datagrams
include the address they were sent
from, and when sending datagrams the
address to send to must be specified.
If you want to see wire-level transmissions rather than inject them, install and run WireShark, the fantastic, free packet sniffer.
Related
I am trying to understand the examples given here: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/wamp/basic/pubsub/basic
I built this script which is supposed to handle multiple pub/sub websocket connections and also open a tcp port ( 8123 ) for incoming control messages. When a message comes on the 8123 port, the application should broadcast to all the connected subscribers the message received on port 8123. How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message.
Another thing that i do not understand is the url. The client javascript connects to the url http://:8080/ws . Where does the "ws" come from ?
Also can someone explain the purpose of RouterFactory, RouterSessionFactory and this bit:
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
my code is below:
import sys, time
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.defer import inlineCallbacks
from autobahn.twisted.wamp import ApplicationSession
from autobahn.twisted.util import sleep
class NotificationProtocol(Protocol):
def __init__(self, factory):
self.factory = factory
def dataReceived(self, data):
print "received new data"
class NotificationFactory(Factory):
protocol = NotificationProtocol
class WsNotificationComponent(ApplicationSession):
#inlineCallbacks
def onJoin(self, details):
counter = 0
while True:
self.publish("com.myapp.topic1", "test %d" % counter )
counter += 1
yield sleep(1)
## we use an Autobahn utility to install the "best" available Twisted reactor
##
from autobahn.twisted.choosereactor import install_reactor
reactor = install_reactor()
## create a WAMP router factory
##
from autobahn.wamp.router import RouterFactory
router_factory = RouterFactory()
## create a WAMP router session factory
##
from autobahn.twisted.wamp import RouterSessionFactory
session_factory = RouterSessionFactory(router_factory)
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
from autobahn.twisted.websocket import WampWebSocketServerFactory
transport_factory = WampWebSocketServerFactory(session_factory)
transport_factory.setProtocolOptions(failByDrop = False)
from twisted.internet.endpoints import serverFromString
## start the server from an endpoint
##
server = serverFromString(reactor, "tcp:8080")
server.listen(transport_factory)
notificationFactory = NotificationFactory()
reactor.listenTCP(8123, notificationFactory)
reactor.run()
"How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message":
Check out one of my other answers on SO: Persistent connection in twisted. Jump down to the example code and model your websocket logic like the "IO" logic and you'll have a good fit (You might also want to see the follow-on answer about the newer endpoint calls from one of the twisted core-team too)
"Where does the "ws" come from ?"
Websockets are implemented by retasking http connections, which by their nature have to have a specific path on the request. That "ws" path typically would map to a special http handler that autobahn is building for you to process websockets (or at least that's what your javascript is expecting...). Assuming thing are setup right you can actually point your web-browswer at that url and it should print back an error about the websocket handshake (Expected WebSocket Headers in my case, but I'm using cyclones websockets not autobahn).
P.S. one of the cool side-effects from "websockets must have a specific path" is that you can actually mix websockets and normal http content on the same handler/listen/port, this gets really handy when your trying to run them all on the same SSL port because your trying to avoid the requirement of a proxy front-ending your code.
I'm working in a project with Python, Twisted and Redis. So the team decided to use txredisapi for the communication between the Python modules and Redis. This project does a lot of different things and we need to subscribe to several channels for listen the messages sent by Redis without the other functionalities stops (asynchronously).
Can one execution handle all the work and listen the messages sent by Redis at the same time or must we separate and execute the code in differents flows?
We use the following code for listen the messages:
import txredisapi as redis
class RedisListenerProtocol(redis.SubscriberProtocol):
def connectionMade(self):
self.subscribe("channelName")
def messageReceived(self, pattern, channel, message):
print "pattern=%s, channel=%s message=%s" %(pattern, channel, message)
def connectionLost(self, reason):
print "lost connection:", reason
class RedisListenerFactory(redis.SubscriberFactory):
maxDelay = 120
continueTrying = True
protocol = RedisListenerProtocol
We try to listen the messages with:
self.connRedisChannels = yield redis.ConnectionPool()
I'm interested to know how can I specify that the Connection must use the "RedisListenerFactory", then I guess that the function "messageReceived" will be fired when a message arrives.
Any suggestions, example or correction will be apreciated.
Thanks!
The following code solves the problem:
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(HOST, PORT)
Thanks to Philippe T. for the help.
If you want to use directly the redis.Connection() may be you can do this before:
redis.SubscriberFactory.protocol = RedisListenerProtocol
the package make internal call to is factory for connection.
other way is to rewrite *Connection class and make*Connection factory to use your factory.
to make the connection on other part of your code you can do something like this :
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
# some where :
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(__HOST__, __PORT__)
# the defer will have your client when the connection is done
Im trying to write a program that would be listening for data (simple text messages) on some port (say tcp 6666) and then pass them to one or more different protocols - irc, xmpp and so on. I've tried many approaches and digged the Internet, but I cant find easy and working solution for such task.
The code I am currently fighting with is here: http://pastebin.com/ri7caXih
I would like to know how to from object like:
ircf = ircFactory('asdfasdf', '#asdf666')
get access to self protocol methods, because this:
self.protocol.dupa1(msg)
returns error about self not being passed to active protocol object. Or maybe there is other, better, easier and more kosher way to create single reactor with multiple protocols and have actions triggeres when a message arrives on any of them, and then pass that message to other protocols for handling/processing/sending?
Any help will be highly appreciated!
Here is sample code to read from multiple connections to port 9001 and write out to a connection on port 9000. You would need multiple "PutLine" implementations, one for XMPP, IRC, MSN, etc.
I used a global to store the output connection PutLine but you would want to create a more complex Factory object that would handle this instead.
#!/usr/bin/env python
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.endpoints import clientFromString, serverFromString
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor
queue = []
putter = None
class GetLine(LineReceiver):
delimiter = '\n'
def lineReceived(self, line):
queue.append(line)
putter.have_data()
self.sendLine(line)
class PutLine(LineReceiver):
def __init__(self):
global putter
putter = self
print 'putline init called %s' % str(self)
def have_data(self):
line = queue.pop()
self.sendLine(line)
def main():
f = Factory()
f.protocol = PutLine
endpoint = clientFromString(reactor, "tcp:host=localhost:port=9000")
endpoint.connect(f)
f = Factory()
f.protocol = GetLine
endpoint2 = serverFromString(reactor, "tcp:port=9001")
endpoint2.listen(f)
reactor.run()
if __name__ == '__main__':
main()
Testing:
nc -l 9000
python test.py
nc 9001
Data entered form any number of nc 9001 (or netcat 9001) will appear on nc -l 9000.
This is answered in the FAQ.
http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputononeconnectionresultinoutputonanother
See doc/core/examples/chatserver.py. There they've added hooks to the Protocol's connectionMade and connectionLost methods to maintain a list of connected clients, and then it iterates through all of them when a message arrives to pass on.
This is problem.
My primary work is : deliver "s" object to "handle" method in TestRequestHandler class.
My first step was : deliver "s" object through "point" method to TestServer class, but here im stuck. How to deliver "s" object to TestRequestHandler? Some suggestions?
import threading
import SocketServer
from socket import *
class TestRequestHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server):
SocketServer.BaseRequestHandler.__init__(self, request, client_address, server)
return
def setup(self):
return SocketServer.BaseRequestHandler.setup(self)
def handle(self):
data = self.request.recv(1024)
if (data):
self.request.send(data)
print data
def finish(self):
return SocketServer.BaseRequestHandler.finish(self)
class TestServer(SocketServer.TCPServer):
def __init__(self, server_address, handler_class=TestRequestHandler):
print "__init__"
SocketServer.TCPServer.__init__(self, server_address, handler_class)
return
def point(self,obj):
self.obj = obj
print "point"
def server_activate(self):
SocketServer.TCPServer.server_activate(self)
return
def serve_forever(self):
print "serve_forever"
while True:
self.handle_request()
return
def handle_request(self):
return SocketServer.TCPServer.handle_request(self)
if __name__ == '__main__':
s = socket(AF_INET, SOCK_STREAM)
address = ('localhost', 6666)
server = TestServer(address, TestRequestHandler)
server.point(s)
t = threading.Thread(target=server.serve_forever())
t.setDaemon(True)
t.start()
If I understand correctly, I think you perhaps are misunderstanding how the module works. You are already specifying an address of 'localhost:6666' for the server to bind on.
When you start the server via your call to serve_forever(), this is going to cause the server to start listening to a socket on localhost:6666.
According to the documentation, that socket is passed to your RequestHandler as the 'request' object. When data is received on the socket, your 'handle' method should be able to recv/send from/to that object using the documented socket API.
If you want a further abstraction, it looks like your RequestHandler can extend from StreamRequestHandler and read/write to the socket using file-like objects instead.
The point is, there is no need for you to create an additional socket and then try to force your server to use the new one instead. Part of the value of the SocketServer module is that it manages the lifecycle of the socket for you.
On the flip side, if you want to test your server from a client's perspective, then you would want to create a socket that you can read/write your client requests on. But you would never pass this socket to your server, per se. You would probably do this in a completely separate process and test your server via IPC over the socket.
Edit based on new information
To get server A to open a socket to server B when server A receives data one solution is to simply open a socket from inside your RequestHandler. That said, there are likely some other design concerns that you will need to address based on the requirements of your service.
For example, you may want to use a simple connection pool that say opens a few sockets to server B that server A can use like a resource. There may already be some libraries in Python that help with this.
Given your current design, your RequestHandler has access to the server as a member variable so you could do something like this:
class TestServer(SocketServer.TCPServer):
def point (self, socketB):
self.socketB = socketB # hold serverB socket
class TestRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
data = self.request.recv(1024)
if (data):
self.request.send(data)
print data
self.server.socketB ... # Do whatever with the socketB
But like I said, it may be better for you to have some sort of connection pool or other object that manages your server B socket such that your server A handler can just acquire/release the socket as incoming requests are handled.
This way you can better deal with conditions where server B breaks the socket. Your current design wouldn't be able to handle broken sockets very easily. Just some thoughts...
If the value of s is set once, and not reinitialized - you could make it a class variable as opposed to an instance variable of TestServer, and then have the handler retrieve it via a class method of TestServer in the handler's constructor.
eg: TestServer._mySocket = s
Ok, my main task is this. Construction of the listening server (A-server - localhost, 6666) which during start will open "hard" connection to the different server (B-server - localhost, 7777).
When the customer send data to the A-server this (A-server) sends data (having that hard connection to the B-server) to B-server, the answer receives from the B-server to A-server and answer sends to the customer.
Then again : the customer sends data, A-server receives them, then sends to the B-server, the answer receives data from the B-server and A-server send data to the customer.
And so round and round. The connection to the B-server is closes just when the server A will stop.
All above is the test of making this.
I am looking for a way to periodically send some data over all clients connected to a TCP port. I am looking at twisted python and I am aware of reactor.callLater. But how do I use it to send some data to all connected clients periodically ? The data sending logic is in Protocol class and it is instantiated by the reactor as needed. I don't know how to tie it from reactor to all protocol instances...
You would probably want to do this in the Factory for the connections. The Factory is not automatically notified of every time a connection is made and lost, so you can notify it from the Protocol.
Here is a complete example of how to use twisted.internet.task.LoopingCall in conjunction with a customised basic Factory and Protocol to announce that '10 seconds has passed' to every connection every 10 seconds.
from twisted.internet import reactor, protocol, task
class MyProtocol(protocol.Protocol):
def connectionMade(self):
self.factory.clientConnectionMade(self)
def connectionLost(self, reason):
self.factory.clientConnectionLost(self)
class MyFactory(protocol.Factory):
protocol = MyProtocol
def __init__(self):
self.clients = []
self.lc = task.LoopingCall(self.announce)
self.lc.start(10)
def announce(self):
for client in self.clients:
client.transport.write("10 seconds has passed\n")
def clientConnectionMade(self, client):
self.clients.append(client)
def clientConnectionLost(self, client):
self.clients.remove(client)
myfactory = MyFactory()
reactor.listenTCP(9000, myfactory)
reactor.run()
I'd imagine the easiest way to do that is to manage a list of clients in the protocol with connectionMade and connectionLost in the client and then use a LoopingCall to ask each client to send data.
That feels a little invasive, but I don't think you'd want to do it without the protocol having some control over the transmission/reception. Of course, I'd have to see your code to really know how it'd fit in well. Got a github link? :)