How can i access underlying socket from twisted.web.client.Agent? I need to enable TCP_NODELAY on this socket.
Unfortunately Agent doesn't make it as easy as it would be if you were working directly with a Protocol instance, but it's not impossible either.
The key lies here, in the class definition of Agent:
_protocol = HTTP11ClientProtocol
In order to get access to the transport you could override connectionMade on HTTP11ClientProtocol, as well as the Agent.
So you'd end up with something like:
from twisted.web import client
class MyHTTPClient(client.HTTP11ClientProtocol):
def connectionMade(self):
self.transport.setTcpNoDelay(True)
client.HTTP11ClientProtocol.connectionMade(self) # call the super-class's connectionMade
class MyAgent(client.Agent):
_protocol = MyHTTPClient
Now use MyAgent in lieu of Agent and you'll get TCP nodelay on the client.
** Note **, this isn't the only way to do this, but one way you can do so and continue to use Agent.request. Alternately, write your own agent which crafts the request and connects it to a Client and wires up your request, along with TCP nodelay, in a deferred chain.
** Note 2 ** In this case, it's fine to assume 'transport' has the setTcpNoDelay() method because it's a pretty reasonable assumption you'll be using TCP as the transport for an HTTP request. This may not be a smart idea all over twisted, though.
Related
As title, here is 2 ways to binding a zeromq socket.
socket.bind("tcp://*:port")
socket.bind("tcp://127.0.0.1:port")
Both these two way work for me, but I am still curious about it.
In general, the server binds to an endpoint and the client connects to an endpoint as follows:
# Server
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
connect the socket:
# Client
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
By binding to 127.0.0.1 you restrict requests to the server to 127.0.0.1 only. Running locally will work just fine. But when you use different machines with different IPs you will notice the effect. Hence the use of "*".
To add to sitWolf's answer, note that you can bind a socket multiple times, to multiple protocols. For instance:
socket = context.socket(zmq.REQ);
socket.bind("tcp://127.0.0.1:5555"); // Bind to localhost
socket.bind("ipc:///tmp/mypipe"); // Also bind to a local pipe
socket.bind("tcp://192.168.0.2:4444"); // Also bind to a specific NIC
Also, the socket type is independent of whether you bind it or connect it, though for some socket types it's natural to bind, and others it's natural to connect. So, a PUB socket makes most sense if it's bound, and the corresponding SUB socket connects. Other sockets, one can choose as suits circumstances (e.g. if there is a machine that is more obviously in a server role, that'd be the one to bind).
Not sure if this is the right title for my problem, but here it goes:
I am currently implementing a Distributed Hash Table (DHT) with an API which can be contacted through TCP. It can serve multiple API calls like PUT, GET, Trace, while listening on multiple IP/Port combinations like this:
factory = protocol.ServerFactory()
factory.protocol = DHTServer
for ip in interfaces:
for port in ports:
reactor.listenTCP(int(port), factory, interface=ip)
print ("Listening to: "+ ip +" on Port: "+port)
reactor.run()
Now those "external" API calls are going to be executed by the underlying DHT implementation (Kademlia, Chord or Pastry). Those underlying DHT implementations are using different protocols to communicate with one another. Kademlia for example uses RPC through UDP.
The protocol for the TCP API (DHTServer in the Code above) has an internal DHT protocol like this:
self.protocol = Kademlia(8088, [("192.168.2.1", 8088)])
Now if a client makes two seperate API requests after one another i get this error message on the second request:
line 197, in _bindSocket
raise error.CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:8088: [Errno 10
048] Normalerweise darf jede Socketadresse (Protokoll, Netzwerkadresse oder Ansc
hluss) nur jeweils einmal verwendet werden.
Which basically says that each socket address is only to be used once. I am not quite sure, but i guess it is because for each API request a new DHTServer protocol instance is created, which in turn also creates a new Kademlia instance and both are trying to listen on the same address. But why is this the case? Shouldn't the first DHTServer protocol instance be destroyed after the first request is served? What am i doing wrong? Is there a better way of doing this? I only recently started working with twisted, so please be patient.
Thanks a lot!
I don't know anything about twisted, but kademlia is a stateful network service, having to maintain its routing table and all that.
Consider sharing a single kademlia instance (and thus underlying UDP socket) across your requests.
My solution to this was to write my own Factory with the inner protocol already pre-defined. Thus i can access it from every instance and it stays the same.
I am writing a client that needs to establish several independent communication channels, each with its own unique port on the server, with a server through a series of sending and receiving messages. I know how to do this using socket send and recv, by giving each communication channel a socket, and do send and recv on that socket. I need to make this work in Twisted, and found potentially useful interfaces including Factory and ProcessProtocol. However, the Protocol interfaces do not provide a method to send messages. Is ProcessProtocol a good choice for my task, and how I make ProcessProtocol send messages?
In case you don't know about it, I'd like to give a shout out to the excellent Twisted finger tutorial that goes through the library at a good pace but with enough detail that you know what's going on.
To directly answer your question, though, I'd say you're on the right track with Protocol and (Client)Factory. I think the cleanest way to do what you're looking for (assuming you need to connect to different ports because they're outputs for different data) would be to make a factory/protocol pair for each port you want to connect to/handle, and then use an external class to handle the application logic aggregating all of them. Generally you wouldn't want your application logic mixed deeply with your networking logic.
A simple example: (note the use of self.transport.write to send data)
from twisted.internet.protocol import Protocol, ClientFactory
from sys import stdout
from foobar_application import CustomAppObject
class FooProtocol(Protocol):
def connectionMade(self):
# Use self.transport.write to send data to the server
self.transport.write('Hello server this is the Foo protocol.')
self.factory.do_app_logic()
class FooFactory(ClientFactory):
protocol = FooProtocol
def __init__(self, app_object=None):
self.app = app_object
def do_app_logic(self):
self.app.do_something()
class BarProtocol(Protocol):
def dataReceived(self, data):
stdout.write('Received data from server using the Bar protocol.')
self.factory.do_fancy_logic(data)
class BarFactory(ClientFactory):
protocol = BarProtocol
def __init__(self, app_object=None):
self.app = app_object
def do_fancy_logic(self, data):
self.app.do_something_else(data)
logic_obj = CustomAppObject()
reactor.listenTCP(8888, FooFactory(app_object=logic_obj)
reactor.listenTCP(9999, BarFactory(app_object=logic_obj)
reactor.run()
You might also want to look at the 'Writing Clients' docs on the Twisted site.
I want to add a timeout to individual connections within my request handler for a server using the SocketServer module.
Let me start by saying this is the first time I'm attempting to do network programming using Python. I've sub-classed SocketServer.BaseRequestHandler and SocketServer.ThreadingTCPServer & SocketServer.TCPServer and managed to create two classes with some basic threaded TCP functionality.
However I would like my incoming connections to time-out. Trying to override any of the built in SocketServer time-out values and methods does not work, as the documentation says this works only with forking server. I have managed to create a timer thread that fires after X seconds, but due to the nature of the blocking recv call within the Handler thread, this is of no use, as I would be forced to kill it, and this is something I really want to avoid.
So it is my understanding that I need an asyncore implementation, where I get notified and read certain amount of data. In the event that no data is sent over a period of 5 seconds lets say, I want to close that connection (I know how to cleanly do that).
I have found a few examples of using asyncore with sockets, but none using SocketServer. So, how can I implement asyncore & threadingTCPserver ?
Is it possible?
Has anyone done it?
You can also set a timeout on the recv call, like this:
sock.settimeout(1.0)
Since you use SocketServer, you will have to find the underlying socket somewhere in the SocketServer. Please note that SocketServer will create the socket for you, so there is no need to do that yourself.
You will probably have defined a RequestHandler to go with your SocketServer. It should look something like this:
class RequestHandler(SocketServer.BaseRequestHandler):
def setup(self):
# the socket is called request in the request handler
self.request.settimeout(1.0)
def handle(self):
while True:
try:
data = self.request.recv(1024)
if not data:
break # connection is closed
else:
pass # do your thing
except socket.timeout:
pass # handle timeout
What would be the best method to restrict access to my XMLRPC server by IP address? I see the class CGIScript in web/twcgi.py has a render method that is accessing the request... but I am not sure how to gain access to this request in my server. I saw an example where someone patched twcgi.py to set environment variables and then in the server access the environment variables... but I figure there has to be a better solution.
Thanks.
When a connection is established, a factory's buildProtocol is called to create a new protocol instance to handle that connection. buildProtocol is passed the address of the peer which established the connection and buildProtocol may return None to have the connection closed immediately.
So, for example, you can write a factory like this:
from twisted.internet.protocol import ServerFactory
class LocalOnlyFactory(ServerFactory):
def buildProtocol(self, addr):
if addr.host == "127.0.0.1":
return ServerFactory.buildProtocol(self, addr)
return None
And only local connections will be handled (but all connections will still be accepted initially since you must accept them to learn what the peer address is).
You can apply this to the factory you're using to serve XML-RPC resources. Just subclass that factory and add logic like this (or you can do a wrapper instead of a subclass).
iptables or some other platform firewall is also a good idea for some cases, though. With that approach, your process never even has to see the connection attempt.
Okay, another answer is to get the ip address from the transport, inside any protocol:
d = self.transport.getHost () ; print d.type, d.host, d.port
Then use the value to filter it in any way you want.
I'd use a firewall on windows, or iptables on linux.