I implemented a basic SOCKS4 client with socket, but my Twisted translation isn't coming along too well. Here's my current code:
import struct
import socket
from twisted.python.failure import Failure
from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol, ClientFactory
class Socks4Client(Protocol):
VERSION = 4
HOST = "0.0.0.0"
PORT = 80
REQUESTS = {
"CONNECT": 1,
"BIND": 2
}
RESPONSES = {
90: "request granted",
91: "request rejected or failed",
92: "request rejected because SOCKS server cannot connect to identd on the client",
93: "request rejected because the client program and identd report different user-ids"
}
def __init__(self):
self.buffer = ""
def connectionMade(self):
self.connect(self.HOST, self.PORT)
def dataReceived(self, data):
self.buffer += data
if len(self.buffer) == 8:
self.validateResponse(self.buffer)
def connect(self, host, port):
data = struct.pack("!BBH", self.VERSION, self.REQUESTS["CONNECT"], port)
data += socket.inet_aton(host)
data += "\x00"
self.transport.write(data)
def validateResponse(self, data):
version, result_code = struct.unpack("!BB", data[1:3])
if version != 4:
self.factory.protocolError(Exception("invalid version"))
elif result_code == 90:
self.factory.deferred.callback(self.responses[result_code])
elif result_code in self.RESPONSES:
self.factory.protocolError(Exception(self.responses[result_code]))
else:
self.factory.protocolError(Exception())
self.transport.abortConnection()
class Socks4Factory(ClientFactory):
protocol = Socks4Client
def __init__(self, deferred):
self.deferred = deferred
def clientConnectionFailed(self, connector, reason):
self.deferred.errback(reason)
def clientConnectionLost(self, connector, reason):
print "Connection lost:", reason
def protocolError(self, reason):
self.deferred.errback(reason)
def result(result):
print "Success:", result
def error(reason):
print "Error:", reason
if __name__ == "__main__":
d = Deferred()
d.addCallbacks(result, error)
factory = Socks4Factory(d)
reactor.connectTCP('127.0.0.1', 1080, factory)
reactor.run()
I have a feeling that I'm abusing Deferred. Is this the right way to send results from my client?
I've read a few tutorials, looked at the documentation, and read through most of the protocols bundled with Twisted, but I still can't figure it out: what exactly is a ClientFactory for? Am I using it the right way?
clientConnectionLosts gets triggered a lot. Sometimes I lose the connection and get a successful response. How is that so? What does this mean, and should I treat it as an error?
How do I make sure that my deferred calls only one callback/errback?
Any tips are appreciated.
I have a feeling that I'm abusing Deferred. Is this the right way to send results from my client?
It's not ideal, but it's not exactly wrong either. Generally, you should try to keep the code that instantiates a Deferred as close as possible to the code that calls Deferred.callback or Deferred.errback on that Deferred. In this case, those pieces of code are quite far apart - the former is in __main__ while the latter is in a class created by a class created by code in __main__. This is sort of like the law of Demeter - the more steps between these two things, the more tightly coupled, inflexible, and fragile the software.
Consider giving Socks4Client a method that creates and returns this Deferred instance. Then, try using an endpoint to setup the connection so you can more easily call this method:
endpoint = TCP4StreamClientEndpoint(reactor, "127.0.0.1", 1080)
d = endpoint.connect(factory)
def connected(protocol):
return protocol.waitForWhatever()
d.addCallback(connected)
d.addCallbacks(result, error)
One thing to note here is that using an endpoint, the clientConnectionFailed and clientConnectionLost methods of your factory won't be called. The endpoint takes over the former responsibility (not the latter though).
I've read a few tutorials, looked at the documentation, and read through most of the protocols bundled with Twisted, but I still can't figure it out: what exactly is a ClientFactory for? Am I using it the right way?
It's for just what you're doing. :) It creates protocol instances to use with connections. A factory is required because you might create connections to many servers (or many connections to one server). However, a lot of people have trouble with ClientFactory so more recently introduced Twisted APIs don't rely on it. For example, you could also do your connection setup as:
endpoint = TCP4StreamClientEndpoint(reactor, "127.0.0.1", 1080)
d = connectProtocol(endpoint, Socks4Client())
...
ClientFactory is now out of the picture.
clientConnectionLosts gets triggered a lot. Sometimes I lose the connection and get a successful response. How is that so? What does this mean, and should I treat it as an error?
Every connection must eventually be lost. You have to decide on your own whether this is an error or not. If you have finished everything you wanted to do and you called loseConnection, it is probably not an error. Consider a connection to an HTTP server. If you have sent your request and received your response, then losing the connection is probably not a big deal. But if you have only received half the response, that's a problem.
How do I make sure that my deferred calls only one callback/errback?
If you structure your code as I described in response to your first question above, it becomes easier to do this. When the code that uses callback/errback on a Deferred is spread across large parts of your program, then it becomes harder to do this correctly.
It is just a matter of proper state tracking, though. Once you give a Deferred a result, you have to arrange to know that you shouldn't give it another one. A common idiom for this is to drop the reference to the Deferred. For example, if you are saving it as the value of an attribute on a protocol instance, then set that attribute to None when you have given the Deferred its result.
Related
I have a simple client / server setup. Here's the client code:
from twisted.internet import reactor
from twisted.internet import protocol
from twisted.internet.endpoints import TCP4ClientEndpoint
class MyProtocol(protocol.Protocol):
def connectionMade(self):
print "Hello!"
def dataReceived(self, data):
print data
class MyProtocolFactory(protocol.ClientFactory):
def startedConnecting(self, connector):
print "Starting to connect!"
def buildProtocol(self, addr):
return MyProtocol()
def clientConnectionLost(self, connector, reason):
print "Lost connection, reason = %s" % reason
def clientConnectionFailed(self, connector, reason):
print "Connection failed, reason = %s" % reason
reactor.stop()
endpoint = TCP4ClientEndpoint(reactor, "127.0.0.1", 54321, timeout=5)
endpoint.connect(MyProtocolFactory())
reactor.run()
For some reason, this client will connect to the server and the protocol works correctly (I can see "Hello!" printed, along with data sent by the server upon a successful connection), but it won't call any of the protocol factory methods. startedConnecting doesn't get called, and if I stop the server, I don't see clientConnectionLost get called. If I try to run the client before the server has started, I would also expect to see clientConnectionFailed get called.
Here's the strange part...If I change the last 3 lines in the code above to the following:
reactor.connectTCP("127.0.0.1", 54321, MyProtocolFactory())
reactor.run()
Then everything works as expected, and all methods get called in all of the cases outlined above.
My understanding of endpoints is that they wrap "connectTCP" (among others) with additional behaviour, but I can't figure out why it works in the second code snippet, but not the first.
Any ideas?
The client endpoint interface does not call the extra connection-state notification methods of ClientFactory.
So, while endpoints do in some sense "wrap" connectTCP et al, it's not true that they have the exact same behavior as using those lower-level methods.
With endpoints, the factory's job is to provide protocol instances. The factory is no longer responsible for other aspects of connection management.
Additional note to supplement my discussion above:
If you’ve used ClientFactory before, keep in mind that the connect method takes a Factory, not a ClientFactory. Even if you pass a ClientFactory to endpoint.connect, its clientConnectionFailed and clientConnectionLost methods will not be called. In particular, clients that extend ReconnectingClientFactory won’t reconnect. The next section describes how to set up reconnecting clients on endpoints.
From the endpoint docs found here: http://twistedmatrix.com/documents/current/core/howto/endpoints.html
I am attempting to write a simple TCP server in twisted which has to perform the following operations in sequence:
A client connects to the server and the KEEPALIVE flag for this connection is set to 1.
The server receives data from the client.
It then computes the response which is a list.
The server then sends each item of the list one by one while waiting for explicit ACKs from the client in between, i.e., after sending a single item from the list, the server waits for an ACK packet from the client and only after receiving the ACK does it proceed to send the rest of the items in the same manner.
The following is the code:
class MyFactory(ServerFactory):
protocol = MyProtocol
def __init__(self, service):
self.service = service
class MyProtocol(Protocol):
def connectionMade(self):
try:
self.transport.setTcpKeepAlive(1)
except AttributeError:
pass
self.deferred = Deferred()
self.deferred.addCallback(self.factory.service.compute_response)
self.deferred.addCallback(self.send_response)
def dataReceived(self, data):
self.fire(data)
def fire(self, data):
if self.deferred is not None:
d, self.deferred = self.deferred, None
d.callback(data)
def send_response(self, data):
for item in data:
d = Deferred()
d.addCallback(self.transport.write)
d.addCallback(self.wait_for_ack)
d.callback(item)
return
def wait_for_ack(self, dummy):
try:
self.transport.socket.recv(1024)
except socket.error as e:
print e
return
Upon running the server and the client I get the following exception:
Resource temporarily unavailable
I understand the reason for this exception - I'm trying to call a blocking method on non blocking socket.
Please help me in finding a solution to this problem.
There are some problems with your example:
You don't define compute_response anywhere (among other things) so I can't run your example. Consider making it an http://sscce.org
You should never call either send or recv on a socket underlying a Twisted transport; let Twisted call those methods for you. In the case of recv it will deliver the results of recv to dataReceived.
You can't rely upon dataReceived to receive whole messages; packets may always be arbitrarily fragmented in transit so you need to have a framing protocol for encapsulating your messages.
However, since my other answer was so badly botched, I owe you a more thorough explanation of how to set up what you want to do.
As stipulated in your question, your protocol is not completely defined enough to give an answer; you cannot do requests and responses with raw TCP fragments, because your application can't know where they start and end (see point 3 above). So, I've invented a little protocol to serve for this example: it's a line-delimited protocol where the client sends "request foo\n" and the server immediately sends "thinking...\n", computes a response, then sends "response foo\n" and waits for the client to send "ok"; in response, the server will either send the next "response ..." line, or a "done\n" line indicating that it's finished sending responses.
With that as our protocol, I believe the key element of your question is that you cannot "wait for acknowledgement", or for that matter, anything else, in Twisted. What you need to do is implement something along the lines of "when an acknowledgement is received...".
Therefore, when a message is received, we need to identify the type of the message: acknowledgement or request?
if it's a request, we need to compute a response; when the response is finished being computed, we need to enqueue all the elements of the response and send the first one.
if it's an acknowledgement, we need to examine the outgoing queue of responses, and if it has any contents, send the first element of it; otherwise, send "done".
Here's a full, runnable example that implements the protocol I described in that way:
from twisted.internet.protocol import ServerFactory
from twisted.internet.task import deferLater
from twisted.internet import reactor
from twisted.internet.interfaces import ITCPTransport
from twisted.protocols.basic import LineReceiver
class MyProtocol(LineReceiver):
delimiter = "\n"
def connectionMade(self):
if ITCPTransport.providedBy(self.transport):
self.transport.setTcpKeepAlive(1)
self.pendingResponses = []
def lineReceived(self, line):
split = line.rstrip("\r").split(None, 1)
command = split[0]
if command == b"request":
# requesting a computed response
payload = split[1]
self.sendLine("thinking...")
(self.factory.service.computeResponse(payload)
.addCallback(self.sendResponses))
elif command == b"ok":
# acknowledging a response; send the next response
if self.pendingResponses:
self.sendOneResponse()
else:
self.sendLine(b"done")
def sendOneResponse(self):
self.sendLine(b"response " + self.pendingResponses.pop(0))
def sendResponses(self, listOfResponses):
self.pendingResponses.extend(listOfResponses)
self.sendOneResponse()
class MyFactory(ServerFactory):
protocol = MyProtocol
def __init__(self, service):
self.service = service
class MyService(object):
def computeResponse(self, request):
return deferLater(
reactor, 1.0,
lambda: [request + b" 1", request + b" 2", request + b" 3"]
)
from twisted.internet.endpoints import StandardIOEndpoint
endpoint = StandardIOEndpoint(reactor)
endpoint.listen(MyFactory(MyService()))
reactor.run()
I've made this runnable on standard I/O so that you can just run it and type into it to get a feel how it works; if you want to run it on an actual network port, just substitute that with a different type of endpoint. Hopefully this answers your question.
I'm trying to write unittests for my application that uses Autobahn.
I want to test my controllers which gets received data from protocol, parses it and reacts to it.
But when my test comes to a point when protocol should be disconnected (self.sendClose) then it raises error
exceptions.AttributeError: 'MyProtocol' object has no attribute 'state'.
I was trying to makeConnection using proto_helpers.StringTransport but then I have errors too
exceptions.AttributeError: StringTransport instance has no attribute 'setTcpNoDelay'`
I'm using trial and I don't want to run dummy server/client for testing purposes only, because it's not recommended.
How should I write my tests so I can test functions that sends data, read data, disconnects etc. using fake connection and trial ?
It is difficult to say exactly what is going on without having a peek at MyProtocol class. The problem sounds a lot like it is caused by the fact that you are directly messing round with low level functions and therefore also the state attribute of WebSocket class, which is, well, a representation of the internal state of the WebSocket connection.
According to the autobahn reference doc, the APIs from the WebSicketProtocol that you could directly use and override are:
onOpen
onMessage
onClose
sendMessage
sendClose
Your approach of using the StringTransport to test your protocol is not ideal. The problem lays in the fact that MyProtocol is a tiny layer on top of the WebSocketProtocol framework provided by autobahn which, for better or worse, hides the details about managing the connection, the transport and the internal protocol state.
If you think about it, you want to test your stuff, not WebSocketProtocol and therefore if you do not want to embed a dummy server or client, your best bet is to test directly the methods that MyProtocol overrides.
An example of what I am saying is the following
class MyPublisher(object):
cbk=None
def publish(self, msg):
if self.cbk:
self.cbk(msg)
class MyProtocol(WebSocketServerProtocol):
def __init__(self, publisher):
WebSocketServerProtocol.__init__(self)
#Defining callback for publisher
publisher.cbk = self.sendMessage
def onMessage(self, msg, binary)
#Stupid echo
self.sendMessage(msg)
class NotificationTest(unittest.TestCase):
class MyProtocolFactory(WebSocketServerFactory):
def __init__(self, publisher):
WebSocketServerFactory.__init__(self, "ws://127.0.0.1:8081")
self.publisher = publisher
self.openHandshakeTimeout = None
def buildProtocol(self, addr):
protocol = MyProtocol(self.listener)
protocol.factory = self
protocol.websocket_version = 13 #Hybi version 13 is supported by pretty much everyone (apart from IE <8 and android browsers)
return protocol
def setUp(self):
publisher = task.LoopingCall(self.send_stuff, "Hi there")
factory = NotificationTest.MyProtocolFactory(listener)
protocol = factory.buildProtocol(None)
transport = proto_helpers.StringTransport()
def play_dumb(*args): pass
setattr(transport, "setTcpNoDelay", play_dumb)
protocol.makeConnection(transport)
self.protocol, self.transport, self.publisher, self.fingerprint_handler = protocol, transport, publisher, fingerprint_handler
def test_onMessage(self):
#Following 2 lines are the problematic part. Here you are manipulating explicitly a hidden state which your implementation should not be concerned with!
self.protocol.state = WebSocketProtocol.STATE_OPEN
self.protocol.websocket_version = 13
self.protocol.onMessage("Whatever")
self.assertEqual(self.transport.value()[2:], 'Whatever')
def test_push(self):
#Following 2 lines are the problematic part. Here you are manipulating explicitly a hidden state which your implementation should not be concerned with!
self.protocol.state = WebSocketProtocol.STATE_OPEN
self.protocol.websocket_version = 13
self.publisher.publish("Hi there")
self.assertEqual(self.transport.value()[2:], 'Hi There')
As you might have noticed, using the StringTransport here is very cumbersome. You must have knowledge of the underline framework and bypass its state management, something you don't really want to do. Unfortunately autobahn does not provide a ready-to-use test object that would permit easy state manipulation and therefore my suggestion of using dummy servers and clients is still valid
Testing your server WITH network
The test provided shows how you can test server push, asserting that what your are getting is what you expect, and using also a hook on how to determine when to finish.
The server protocol
from twisted.trial.unittest import TestCase as TrialTest
from autobahn.websocket import WebSocketServerProtocol, WebSocketServerFactory, WebSocketClientProtocol, WebSocketClientFactory, connectWS, listenWS
from twisted.internet.defer import Deferred
from twisted.internet import task
START="START"
class TestServerProtocol(WebSocketServerProtocol):
def __init__(self):
#The publisher task simulates an event that triggers a message push
self.publisher = task.LoopingCall(self.send_stuff, "Hi there")
def send_stuff(self, msg):
#this method sends a message to the client
self.sendMessage(msg)
def _on_start(self):
#here we trigger the task to execute every second
self.publisher.start(1.0)
def onMessage(self, message, binary):
#According to this stupid protocol, the server starts sending stuff when the client sends a "START" message
#You can plug other commands in here
{
START : self._on_start
#Put other keys here
}[message]()
def onClose(self, wasClean, code, reason):
#After closing the connection, we tell the task to stop sending messages
self.publisher.stop()
The client protocol and factory
Next class is the client protocol. It basically tells the server to start pushing messages. It calls the close_condition on them to see if it is time to close the connection and as a last thing, it calls the assertion function on the messages it received to see if the test was successful or not
class TestClientProtocol(WebSocketClientProtocol):
def __init__(self, assertion, close_condition, timeout, *args, **kwargs):
self.assertion = assertion
self.close_condition = close_condition
self._received_msgs = []
from twisted.internet import reactor
#This is a way to set a timeout for your test
#in case you never meet the conditions dictated by close_condition
self.damocle_sword = reactor.callLater(timeout, self.sendClose)
def onOpen(self):
#After the connection has been established,
#you can tell the server to send its stuff
self.sendMessage(START)
def onMessage(self, msg, binary):
#Here you get the messages pushed from the server
self._received_msgs.append(msg)
#If it is time to close the connection
if self.close_condition(msg):
self.damocle_sword.cancel()
self.sendClose()
def onClose(self, wasClean, code, reason):
#Now it is the right time to check our test assertions
self.assertion.callback(self._received_msgs)
class TestClientProtocolFactory(WebSocketClientFactory):
def __init__(self, assertion, close_condition, timeout, **kwargs):
WebSocketClientFactory.__init__(self, **kwargs)
self.assertion = assertion
self.close_condition = close_condition
self.timeout = timeout
#This parameter needs to be forced to None to not leave the reactor dirty
self.openHandshakeTimeout = None
def buildProtocol(self, addr):
protocol = TestClientProtocol(self.assertion, self.close_condition, self.timeout)
protocol.factory = self
return protocol
The trial based test
class WebSocketTest(TrialTest):
def setUp(self):
port = 8088
factory = WebSocketServerFactory("ws://localhost:{}".format(port))
factory.protocol = TestServerProtocol
self.listening_port = listenWS(factory)
self.factory, self.port = factory, port
def tearDown(self):
#cleaning up stuff otherwise the reactor complains
self.listening_port.stopListening()
def test_message_reception(self):
#This is the test assertion, we are testing that the messages received were 3
def assertion(msgs):
self.assertEquals(len(msgs), 3)
#This class says when the connection with the server should be finalized.
#In this case the condition to close the connectionis for the client to get 3 messages
class CommunicationHandler(object):
msg_count = 0
def close_condition(self, msg):
self.msg_count += 1
return self.msg_count == 3
d = Deferred()
d.addCallback(assertion)
#Here we create the client...
client_factory = TestClientProtocolFactory(d, CommunicationHandler().close_condition, 5, url="ws://localhost:{}".format(self.port))
#...and we connect it to the server
connectWS(client_factory)
#returning the assertion as a deferred purely for demonstration
return d
This is obviously just an example, but as you can see I did not have to mess around with makeConnection or any transport explicitly
I have been asked to write a class that connects to a server, asynchronously sends the server various commands, and then provides the returned data to the client. I've been asked to do this in Python, which is a new language to me. I started digging around and found the Twisted framework which offers some very nice abstractions (Protocol, ProtocolFactory, Reactor) that do a lot of the things that I would have to do if I would roll my own socket-based app. It seems like the right choice given the problem that I have to solve.
I've looked through numerous examples on the web (mostly Krondo), but I still haven't seen a good example of creating a client that will send multiple commands across the wire and I maintain the connection I create. The server (of which I have no control over), in this case, doesn't disconnect after it sends the response. So, what's the proper way to design the client so that I can tickle the server in various ways?
Right now I do this:
class TestProtocol(Protocol)
def connectionMade(self):
self.transport.write(self.factory.message)
class TestProtocolFactory(Factory):
message = ''
def setMessage(self, msg):
self.message = msg
def main():
f = TestProtocolFactory()
f.setMessage("my message")
reactor.connectTCP(...)
reactor.run()
What I really want to do is call self.transport.write(...) via the reactor (really, call TestProtocolFactory::setMessage() on-demand from another thread of execution), not just when the connection is made.
Depends. Here are some possibilities:
I'm assuming
Approach 1. You have a list of commands to send the server, and for some reason can't do them all at once. In that case send a new one as the previous answer returns:
class proto(parentProtocol):
def stringReceived(self, data):
self.handle_server_response(data)
next_command = self.command_queue.pop()
# do stuff
Approach 2. What you send to the server is based on what the server sends you:
class proto(parentProtocol):
def stringReceived(self, data):
if data == "this":
self.sendString("that")
elif data == "foo":
self.sendString("bar")
# and so on
Approach 3. You don't care what the server sends to, you just want to periodically send some commands:
class proto(parentProtocol):
def callback(self):
next_command = self.command_queue.pop()
# do stuff
def connectionMade(self):
from twisted.internet import task
self.task_id = task.LoopingCall(self.callback)
self.task_id.start(1.0)
Approach 4: Your edit now mentions triggering from another thread. Feel free to check the twisted documentation to find out if proto.sendString is threadsafe. You may be able to call it directly, but I don't know. Approach 3 is threadsafe though. Just fill the queue (which is threadsafe) from another thread.
Basically you can store any amount of state in your protocol; it will stay around until you are done. The you either send commands to the server as a response to it's messages to you, or you set up some scheduling to do your stuff. Or both.
You may want to use a Service.
Services are pieces of functionality within a Twisted app which are started and stopped, and are nice abstractions for other parts of your code to interact with. For example, in this case you might have a SayStuffToServerService (I know, terrible name, but without knowing more about its job it was the best I could do here :) ) that exposed something like this:
class SayStuffToServerService:
def __init__(self, host, port):
# this is the host and port to connect to
def sendToServer(self, whatToSend):
# send some line to the remote server
def startService(self):
# call me before using the service. starts outgoing connection efforts.
def stopService(self):
# clean reactor shutdowns should call this method. stops outgoing
# connection efforts.
(That might be all the interface you need, but it should be fairly clear where you can add things to this.)
The startService() and stopService() methods here are just what Twisted's Services expose. And helpfully, there is a premade Twisted Service which acts like a TCP client and takes care of all the reactor stuff for you. It's twisted.application.internet.TCPClient, which takes arguments for a remote host and port, along with a ProtocolFactory to take care of handling the actual connection attempt.
Here is the SayStuffToServerService, implemented as a subclass of TCPClient:
from twisted.application import internet
class SayStuffToServerService(internet.TCPClient):
factoryclass = SayStuffToServerProtocolFactory
def __init__(self, host, port):
self.factory = self.factoryclass()
internet.TCPClient.__init__(self, host, port, self.factory)
def sendToServer(self, whatToSend):
# we'll do stuff here
(See below for the SayStuffToServerProtocolFactory.)
Using this Service architecture is convenient in a lot of ways; you can group Services together in one container, so that they all get stopped and started as one when you have different parts of your app that you want active. It may make good sense to implement other parts of your app as separate Services. You can set Services as child services to application- the magic name that twistd looks for in order to know how to initialize, daemonize, and shut down your app. Actually yes, let's add some code to do that now.
from twisted.application import service
...
application = service.Application('say-stuff')
sttss = SayStuffToServerService('localhost', 65432)
sttss.setServiceParent(service.IServiceCollection(application))
That's all. Now when you run this module under twistd (i.e., for debugging, twistd -noy saystuff.py), that application will be started under the right reactor, and it will in turn start the SayStuffToServerService, which will start a connection effort to localhost:65432, which will use the service's factory attribute to set up the connection and the Protocol. You don't need to call reactor.run() or attach things to the reactor yourself anymore.
So we haven't implemented SayStuffToServerProtocolFactory yet. Since it sounds like you would prefer that your client reconnect if it has lost the connection (so that callers of sendToServer can usually just assume that there's a working connection), I'm going to put this protocol factory on top of ReconnectingClientFactory.
from twisted.internet import protocol
class SayStuffToServerProtocolFactory(protocol.ReconnectingClientFactory):
_my_live_proto = None
protocol = SayStuffToServerProtocol
This is a pretty nice minimal definition, which will keep trying to make outgoing TCP connections to the host and port we specified, and instantiate a SayStuffToServerProtocol each time. When we fail to connect, this class will do nice, well-behaved exponential backoff so that your network doesn't get hammered (you can set a maximum wait time). It will be the responsibility of the Protocol to assign to _my_live_proto and call this factory's resetDelay() method, so that exponential backoff will continue to work as expected. And here is that Protocol now:
class SayStuffToServerProtocol(basic.LineReceiver):
def connectionMade(self):
# if there are things you need to do on connecting to ensure the
# connection is "all right" (maybe authenticate?) then do that
# before calling:
self.factory.resetDelay()
self.factory._my_live_proto = self
def connectionLost(self, reason):
self.factory._my_live_proto = None
del self.factory
def sayStuff(self, stuff):
self.sendLine(stuff)
def lineReceived(self, line):
# do whatever you want to do with incoming lines. often it makes sense
# to have a queue of Deferreds on a protocol instance like this, and
# each incoming response gets sent to the next queued Deferred (which
# may have been pushed on the queue after sending some outgoing
# message in sayStuff(), or whatever).
pass
This is implemented on top of twisted.protocols.basic.LineReceiver, but would work as well with any other sort of Protocol, in case your protocol isn't line-oriented.
The only thing left is hooking up the Service to the right Protocol instance. This is why the Factory keeps a _my_live_proto attribute, which should be set when a connection is successfully made, and cleared (set to None) when that connection is lost. Here's the new implementation of SayStuffToServerService.sendToServer:
class NotConnectedError(Exception):
pass
class SayStuffToServerService(internet.TCPClient):
...
def sendToServer(self, whatToSend):
if self.factory._my_live_proto is None:
# define here whatever behavior is appropriate when there is no
# current connection (in case the client can't connect or
# reconnect)
raise NotConnectedError
self.factory._my_live_proto.sayStuff(whatToSend)
And now to tie it all together in one place:
from twisted.application import internet, service
from twisted.internet import protocol
from twisted.protocols import basic
class SayStuffToServerProtocol(basic.LineReceiver):
def connectionMade(self):
# if there are things you need to do on connecting to ensure the
# connection is "all right" (maybe authenticate?) then do that
# before calling:
self.factory.resetDelay()
self.factory._my_live_proto = self
def connectionLost(self, reason):
self.factory._my_live_proto = None
del self.factory
def sayStuff(self, stuff):
self.sendLine(stuff)
def lineReceived(self, line):
# do whatever you want to do with incoming lines. often it makes sense
# to have a queue of Deferreds on a protocol instance like this, and
# each incoming response gets sent to the next queued Deferred (which
# may have been pushed on the queue after sending some outgoing
# message in sayStuff(), or whatever).
pass
class SayStuffToServerProtocolFactory(protocol.ReconnectingClientFactory):
_my_live_proto = None
protocol = SayStuffToServerProtocol
class NotConnectedError(Exception):
pass
class SayStuffToServerService(internet.TCPClient):
factoryclass = SayStuffToServerProtocolFactory
def __init__(self, host, port):
self.factory = self.factoryclass()
internet.TCPClient.__init__(self, host, port, self.factory)
def sendToServer(self, whatToSend):
if self.factory._my_live_proto is None:
# define here whatever behavior is appropriate when there is no
# current connection (in case the client can't connect or
# reconnect)
raise NotConnectedError
self.factory._my_live_proto.sayStuff(whatToSend)
application = service.Application('say-stuff')
sttss = SayStuffToServerService('localhost', 65432)
sttss.setServiceParent(service.IServiceCollection(application))
Hopefully that gives enough of a framework with which to start. There is sometimes a lot of plumbing to do to handle client disconnections just the way you want, or to handle out-of-order responses from the server, or handle various sorts of timeout, canceling pending requests, allowing multiple pooled connections, etc, etc, but this should help.
The twisted framework is event-based programming; and by nature, its method is all called in async, and result is get by defer object.
The framework's nature is approprivate for protocol developing, just you have to change your minding from traditional sequential programming. The Protocol class is like a finite state machine with events like: connection make, connection lost, receive data.
You can convert your client code into FSM and then will be easily to fit into the Protocol class.
Below is an rough example of what I want to express. A bit of rouge, but this is i can provide now:
class SyncTransport(Protocol):
# protocol
def dataReceived(self, data):
print 'receive data', data
def connectionMade(self):
print 'i made a sync connection, wow'
self.transport.write('x')
self.state = I_AM_LIVING
def connectionLost(self):
print 'i lost my sync connection, sight'
def send(self, data):
if self.state == I_AM_LIVING:
if data == 'x':
self.transport.write('y')
if data == 'Y':
self.transport.write('z')
self.state = WAITING_DEAD
if self.state == WAITING_DEAD:
self.transport.close()
The scenario I want two different protocols interact with each other is as below:
A and B is two different protocols.
First A will interact with the server and retrieve some values.
Only after A finishes retrieving the values , B will start to interact with the server.
Now my problem is that is there an elegant way to initial B when A retrieves the values.
Currently I just initial B in A's data process function. But i don't think that this is an elegant way.
What I mean an elegant way is that the initialization of B is done by a flow controller or something like that, but not another protocol.
Is there an elegant way? such using defered or any other things.
I'm just new to twisted, not knowing very much about defered....
Thank you very much!
It sounds like you've gotten past the first hurdle - figuring out how to have A and B interact at all. That's good, since for most people that's the biggest conceptual challenge. As for making it elegant, if you're after an approach that keeps your protocol code isolated from the application code driving it (ie, the "business logic"), there are several options. I'll give an example of one based on Deferreds.
Let's consider two POP3 clients. You want the first to retrieve a message list, then the second to retrieve the first message from the resulting list. This example
from twisted.internet import defer, protocol, reactor
from twisted.mail.pop3 import AdvancedPOP3Client
class MessageDownloader(object):
def __init__(self, host, port, user, password):
self.host = host
self.port = port
self.user = user
self.password = password
self.cc = ClientCreator(reactor, AdvancedPOP3Client)
def connect(self):
"""
Connect to the POP3 server and authenticate. Return a Deferred
which fires with the connected protocol instance.
"""
connDeferred = self.cc.connect(self.host, self.port)
def cbAuthenticate(proto):
loginDeferred = proto.login(user, password)
loginDeferred.addCallback(lambda ignored: proto)
return loginDeferred
connDeferred.addCallback(cbAuthenticate)
return connDeferred
def run(self):
connDeferred = self.connect()
connDeferred.addCallback(self.cbFirstConnection)
return connDeferred
def cbFirstConnection(self, firstProto):
listDeferred = firstProto.listUID()
def cbListed(uidList):
connDeferred = self.connect()
def cbConnected(secondProto):
return secondProto.retrieve(uidList[0])
connDeferred.addCallback(cbConnected)
listDeferred.addCallback(cbListed)
return listDeferred
if __name__ == '__main__':
import sys
MessageDownloader(*sys.argv[1:]).run()
reactor.run()
Here, all of the logic about retrieving a list of UIDs and setting up a new connection to retrieve a message is separate from the actual protocol implementation (which is entirely in Twisted). The Deferreds returned from almost all of the APIs used here allow events to be connected up however your application wants.