Not sure if this is the right title for my problem, but here it goes:
I am currently implementing a Distributed Hash Table (DHT) with an API which can be contacted through TCP. It can serve multiple API calls like PUT, GET, Trace, while listening on multiple IP/Port combinations like this:
factory = protocol.ServerFactory()
factory.protocol = DHTServer
for ip in interfaces:
for port in ports:
reactor.listenTCP(int(port), factory, interface=ip)
print ("Listening to: "+ ip +" on Port: "+port)
reactor.run()
Now those "external" API calls are going to be executed by the underlying DHT implementation (Kademlia, Chord or Pastry). Those underlying DHT implementations are using different protocols to communicate with one another. Kademlia for example uses RPC through UDP.
The protocol for the TCP API (DHTServer in the Code above) has an internal DHT protocol like this:
self.protocol = Kademlia(8088, [("192.168.2.1", 8088)])
Now if a client makes two seperate API requests after one another i get this error message on the second request:
line 197, in _bindSocket
raise error.CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:8088: [Errno 10
048] Normalerweise darf jede Socketadresse (Protokoll, Netzwerkadresse oder Ansc
hluss) nur jeweils einmal verwendet werden.
Which basically says that each socket address is only to be used once. I am not quite sure, but i guess it is because for each API request a new DHTServer protocol instance is created, which in turn also creates a new Kademlia instance and both are trying to listen on the same address. But why is this the case? Shouldn't the first DHTServer protocol instance be destroyed after the first request is served? What am i doing wrong? Is there a better way of doing this? I only recently started working with twisted, so please be patient.
Thanks a lot!
I don't know anything about twisted, but kademlia is a stateful network service, having to maintain its routing table and all that.
Consider sharing a single kademlia instance (and thus underlying UDP socket) across your requests.
My solution to this was to write my own Factory with the inner protocol already pre-defined. Thus i can access it from every instance and it stays the same.
Related
Is it possible to have one protocol connect to another protocol on the same server? My goal is to accept a request for one protocol and then pass that request to a different protocol and have that second protocol return values to whatever client is connected to it.
I'm thinking you would transfer the onMessage request to the other protocol some how.
I don't have any code to show as I don't know where to start, but any code examples would be appreciated.
What you're asking for sounds like a proxy server. A proxy can simply be a middleman that speaks the same protocol out both ends (as in a typical http proxy) or it can be some sort of translator that has one protocol coming in and another protocol going out.
So, supposed you want a browser to be able to use a webSocket connection to speak to some other server that doesn't speak the webSocket protocol. You could implement a proxy server yourself that allows the browser to connect to it and then, via your proxy, it could send/receive messages with the other server that speaks a different protocol.
To implement a proxy server like this, you would do the following:
Create a server process that listens for incoming webSocket connections. This would allow the browser to connect to your proxy.
Once connected, the browser would send a message (of your own design) over the webSocket.
Your proxy would receive that message and translate it to the protocol of the other server,
Your proxy would then connect to that other server and send the message to the other server.
Your proxy could then receive a response from that other server and then, if needed, send a translated response message back to the browser over the webSocket.
It would be the proxy's responsibility to translate each message data from what it received over the webSocket to whatever format/protocol the other server speaks.
It would be an implementation choice whether you maintained a dedicated connection between the proxy and the other server for each webSocket connection or whether you directed all requests over one dedicated connection or whether you created a new connection upon demand only for the duration of a given request. Which makes the most sense depends entirely upon the characteristics of the other server, number of requests and the work that is being done.
This is a method of DatagramProtocol class in Twisted. As I understand UDP protocol doesn't guarantee that someone is listening on the given port even using ConnectedDatagramProtocol.
Can someone explain to me, when this method is called and how I suppose to check if there is someone listening to my transmission using UDP?
If the datagram socket is connected, it can receive ICMP Port Unreachable messages via the Sockets API, which presumably maps into calling this method. Note that I am not speaking of the TCP connect operation here, but the Sockets connect() method, which can be called on a UDP socket, and which presumably maps into some method in the API you are using.
I want to create a simple video streaming (actually, image streaming) server that can manage different protocols (TCP Push/Pull, UDP Push/Pull/Multicast).
I managed to get TCP Push/Pull working with the SocketServer.TCPServer class and ThreadinMixIn for processing each connected client in a different thread.
But now that I'm working on the UDP protocol, I just realized that ThreadinMixIn creates a thread per call of handle() per client query (as there's nothing such as a "connection" in UDP).
The problem is I need to process a sequence of queries by the same client, for all the clients. How could I manage that ?
The only way I see I could handle that is to have a list of (client adresses, processing thread) and send each query to the matching thread (or create a new one if the client haven't sent any thread yet). Is there an easier way to do that ?
Thanks !
P.S : I can't use any external or too "high-level" library for this as it's a school subject meant to understand how sockets work.
Take a look at Twisted. This will remove the need to do any thread dispatch from your application. You still have to match up packets to a particular session in order to handle them, but this isn't difficult (use a port per client and dispatch based on the port, or require packets in a session to always come from the same address and use the peer address, or use one of the existing protocols that solves this problem such as SIP).
How can i access underlying socket from twisted.web.client.Agent? I need to enable TCP_NODELAY on this socket.
Unfortunately Agent doesn't make it as easy as it would be if you were working directly with a Protocol instance, but it's not impossible either.
The key lies here, in the class definition of Agent:
_protocol = HTTP11ClientProtocol
In order to get access to the transport you could override connectionMade on HTTP11ClientProtocol, as well as the Agent.
So you'd end up with something like:
from twisted.web import client
class MyHTTPClient(client.HTTP11ClientProtocol):
def connectionMade(self):
self.transport.setTcpNoDelay(True)
client.HTTP11ClientProtocol.connectionMade(self) # call the super-class's connectionMade
class MyAgent(client.Agent):
_protocol = MyHTTPClient
Now use MyAgent in lieu of Agent and you'll get TCP nodelay on the client.
** Note **, this isn't the only way to do this, but one way you can do so and continue to use Agent.request. Alternately, write your own agent which crafts the request and connects it to a Client and wires up your request, along with TCP nodelay, in a deferred chain.
** Note 2 ** In this case, it's fine to assume 'transport' has the setTcpNoDelay() method because it's a pretty reasonable assumption you'll be using TCP as the transport for an HTTP request. This may not be a smart idea all over twisted, though.
What would be the best method to restrict access to my XMLRPC server by IP address? I see the class CGIScript in web/twcgi.py has a render method that is accessing the request... but I am not sure how to gain access to this request in my server. I saw an example where someone patched twcgi.py to set environment variables and then in the server access the environment variables... but I figure there has to be a better solution.
Thanks.
When a connection is established, a factory's buildProtocol is called to create a new protocol instance to handle that connection. buildProtocol is passed the address of the peer which established the connection and buildProtocol may return None to have the connection closed immediately.
So, for example, you can write a factory like this:
from twisted.internet.protocol import ServerFactory
class LocalOnlyFactory(ServerFactory):
def buildProtocol(self, addr):
if addr.host == "127.0.0.1":
return ServerFactory.buildProtocol(self, addr)
return None
And only local connections will be handled (but all connections will still be accepted initially since you must accept them to learn what the peer address is).
You can apply this to the factory you're using to serve XML-RPC resources. Just subclass that factory and add logic like this (or you can do a wrapper instead of a subclass).
iptables or some other platform firewall is also a good idea for some cases, though. With that approach, your process never even has to see the connection attempt.
Okay, another answer is to get the ip address from the transport, inside any protocol:
d = self.transport.getHost () ; print d.type, d.host, d.port
Then use the value to filter it in any way you want.
I'd use a firewall on windows, or iptables on linux.