Develop Reference

Twisted Python factory methods aren't getting called when using reactor wrappers

I have a simple client / server setup. Here's the client code:
from twisted.internet import reactor
from twisted.internet import protocol
from twisted.internet.endpoints import TCP4ClientEndpoint
class MyProtocol(protocol.Protocol):
def connectionMade(self):
print "Hello!"
def dataReceived(self, data):
print data
class MyProtocolFactory(protocol.ClientFactory):
def startedConnecting(self, connector):
print "Starting to connect!"
def buildProtocol(self, addr):
return MyProtocol()
def clientConnectionLost(self, connector, reason):
print "Lost connection, reason = %s" % reason
def clientConnectionFailed(self, connector, reason):
print "Connection failed, reason = %s" % reason
reactor.stop()
endpoint = TCP4ClientEndpoint(reactor, "127.0.0.1", 54321, timeout=5)
endpoint.connect(MyProtocolFactory())
reactor.run()
For some reason, this client will connect to the server and the protocol works correctly (I can see "Hello!" printed, along with data sent by the server upon a successful connection), but it won't call any of the protocol factory methods. startedConnecting doesn't get called, and if I stop the server, I don't see clientConnectionLost get called. If I try to run the client before the server has started, I would also expect to see clientConnectionFailed get called.
Here's the strange part...If I change the last 3 lines in the code above to the following:
reactor.connectTCP("127.0.0.1", 54321, MyProtocolFactory())
reactor.run()
Then everything works as expected, and all methods get called in all of the cases outlined above.
My understanding of endpoints is that they wrap "connectTCP" (among others) with additional behaviour, but I can't figure out why it works in the second code snippet, but not the first.
Any ideas?

The client endpoint interface does not call the extra connection-state notification methods of ClientFactory.
So, while endpoints do in some sense "wrap" connectTCP et al, it's not true that they have the exact same behavior as using those lower-level methods.
With endpoints, the factory's job is to provide protocol instances. The factory is no longer responsible for other aspects of connection management.

Additional note to supplement my discussion above:
If you’ve used ClientFactory before, keep in mind that the connect method takes a Factory, not a ClientFactory. Even if you pass a ClientFactory to endpoint.connect, its clientConnectionFailed and clientConnectionLost methods will not be called. In particular, clients that extend ReconnectingClientFactory won’t reconnect. The next section describes how to set up reconnecting clients on endpoints.
From the endpoint docs found here: http://twistedmatrix.com/documents/current/core/howto/endpoints.html

Proper use of a client and Deferred with Twisted

I implemented a basic SOCKS4 client with socket, but my Twisted translation isn't coming along too well. Here's my current code:
import struct
import socket
from twisted.python.failure import Failure
from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol, ClientFactory
class Socks4Client(Protocol):
VERSION = 4
HOST = "0.0.0.0"
PORT = 80
REQUESTS = {
"CONNECT": 1,
"BIND": 2
}
RESPONSES = {
90: "request granted",
91: "request rejected or failed",
92: "request rejected because SOCKS server cannot connect to identd on the client",
93: "request rejected because the client program and identd report different user-ids"
}
def __init__(self):
self.buffer = ""
def connectionMade(self):
self.connect(self.HOST, self.PORT)
def dataReceived(self, data):
self.buffer += data
if len(self.buffer) == 8:
self.validateResponse(self.buffer)
def connect(self, host, port):
data = struct.pack("!BBH", self.VERSION, self.REQUESTS["CONNECT"], port)
data += socket.inet_aton(host)
data += "\x00"
self.transport.write(data)
def validateResponse(self, data):
version, result_code = struct.unpack("!BB", data[1:3])
if version != 4:
self.factory.protocolError(Exception("invalid version"))
elif result_code == 90:
self.factory.deferred.callback(self.responses[result_code])
elif result_code in self.RESPONSES:
self.factory.protocolError(Exception(self.responses[result_code]))
else:
self.factory.protocolError(Exception())
self.transport.abortConnection()
class Socks4Factory(ClientFactory):
protocol = Socks4Client
def __init__(self, deferred):
self.deferred = deferred
def clientConnectionFailed(self, connector, reason):
self.deferred.errback(reason)
def clientConnectionLost(self, connector, reason):
print "Connection lost:", reason
def protocolError(self, reason):
self.deferred.errback(reason)
def result(result):
print "Success:", result
def error(reason):
print "Error:", reason
if __name__ == "__main__":
d = Deferred()
d.addCallbacks(result, error)
factory = Socks4Factory(d)
reactor.connectTCP('127.0.0.1', 1080, factory)
reactor.run()
I have a feeling that I'm abusing Deferred. Is this the right way to send results from my client?
I've read a few tutorials, looked at the documentation, and read through most of the protocols bundled with Twisted, but I still can't figure it out: what exactly is a ClientFactory for? Am I using it the right way?
clientConnectionLosts gets triggered a lot. Sometimes I lose the connection and get a successful response. How is that so? What does this mean, and should I treat it as an error?
How do I make sure that my deferred calls only one callback/errback?
Any tips are appreciated.

I have a feeling that I'm abusing Deferred. Is this the right way to send results from my client?
It's not ideal, but it's not exactly wrong either. Generally, you should try to keep the code that instantiates a Deferred as close as possible to the code that calls Deferred.callback or Deferred.errback on that Deferred. In this case, those pieces of code are quite far apart - the former is in __main__ while the latter is in a class created by a class created by code in __main__. This is sort of like the law of Demeter - the more steps between these two things, the more tightly coupled, inflexible, and fragile the software.
Consider giving Socks4Client a method that creates and returns this Deferred instance. Then, try using an endpoint to setup the connection so you can more easily call this method:
endpoint = TCP4StreamClientEndpoint(reactor, "127.0.0.1", 1080)
d = endpoint.connect(factory)
def connected(protocol):
return protocol.waitForWhatever()
d.addCallback(connected)
d.addCallbacks(result, error)
One thing to note here is that using an endpoint, the clientConnectionFailed and clientConnectionLost methods of your factory won't be called. The endpoint takes over the former responsibility (not the latter though).
I've read a few tutorials, looked at the documentation, and read through most of the protocols bundled with Twisted, but I still can't figure it out: what exactly is a ClientFactory for? Am I using it the right way?
It's for just what you're doing. :) It creates protocol instances to use with connections. A factory is required because you might create connections to many servers (or many connections to one server). However, a lot of people have trouble with ClientFactory so more recently introduced Twisted APIs don't rely on it. For example, you could also do your connection setup as:
endpoint = TCP4StreamClientEndpoint(reactor, "127.0.0.1", 1080)
d = connectProtocol(endpoint, Socks4Client())
...
ClientFactory is now out of the picture.
clientConnectionLosts gets triggered a lot. Sometimes I lose the connection and get a successful response. How is that so? What does this mean, and should I treat it as an error?
Every connection must eventually be lost. You have to decide on your own whether this is an error or not. If you have finished everything you wanted to do and you called loseConnection, it is probably not an error. Consider a connection to an HTTP server. If you have sent your request and received your response, then losing the connection is probably not a big deal. But if you have only received half the response, that's a problem.
How do I make sure that my deferred calls only one callback/errback?
If you structure your code as I described in response to your first question above, it becomes easier to do this. When the code that uses callback/errback on a Deferred is spread across large parts of your program, then it becomes harder to do this correctly.
It is just a matter of proper state tracking, though. Once you give a Deferred a result, you have to arrange to know that you shouldn't give it another one. A common idiom for this is to drop the reference to the Deferred. For example, if you are saving it as the value of an attribute on a protocol instance, then set that attribute to None when you have given the Deferred its result.

How to properly trigger a python twisted transport?

I have been asked to write a class that connects to a server, asynchronously sends the server various commands, and then provides the returned data to the client. I've been asked to do this in Python, which is a new language to me. I started digging around and found the Twisted framework which offers some very nice abstractions (Protocol, ProtocolFactory, Reactor) that do a lot of the things that I would have to do if I would roll my own socket-based app. It seems like the right choice given the problem that I have to solve.
I've looked through numerous examples on the web (mostly Krondo), but I still haven't seen a good example of creating a client that will send multiple commands across the wire and I maintain the connection I create. The server (of which I have no control over), in this case, doesn't disconnect after it sends the response. So, what's the proper way to design the client so that I can tickle the server in various ways?
Right now I do this:
class TestProtocol(Protocol)
def connectionMade(self):
self.transport.write(self.factory.message)
class TestProtocolFactory(Factory):
message = ''
def setMessage(self, msg):
self.message = msg
def main():
f = TestProtocolFactory()
f.setMessage("my message")
reactor.connectTCP(...)
reactor.run()
What I really want to do is call self.transport.write(...) via the reactor (really, call TestProtocolFactory::setMessage() on-demand from another thread of execution), not just when the connection is made.

Depends. Here are some possibilities:
I'm assuming
Approach 1. You have a list of commands to send the server, and for some reason can't do them all at once. In that case send a new one as the previous answer returns:
class proto(parentProtocol):
def stringReceived(self, data):
self.handle_server_response(data)
next_command = self.command_queue.pop()
# do stuff
Approach 2. What you send to the server is based on what the server sends you:
class proto(parentProtocol):
def stringReceived(self, data):
if data == "this":
self.sendString("that")
elif data == "foo":
self.sendString("bar")
# and so on
Approach 3. You don't care what the server sends to, you just want to periodically send some commands:
class proto(parentProtocol):
def callback(self):
next_command = self.command_queue.pop()
# do stuff
def connectionMade(self):
from twisted.internet import task
self.task_id = task.LoopingCall(self.callback)
self.task_id.start(1.0)
Approach 4: Your edit now mentions triggering from another thread. Feel free to check the twisted documentation to find out if proto.sendString is threadsafe. You may be able to call it directly, but I don't know. Approach 3 is threadsafe though. Just fill the queue (which is threadsafe) from another thread.
Basically you can store any amount of state in your protocol; it will stay around until you are done. The you either send commands to the server as a response to it's messages to you, or you set up some scheduling to do your stuff. Or both.

You may want to use a Service.
Services are pieces of functionality within a Twisted app which are started and stopped, and are nice abstractions for other parts of your code to interact with. For example, in this case you might have a SayStuffToServerService (I know, terrible name, but without knowing more about its job it was the best I could do here :) ) that exposed something like this:
class SayStuffToServerService:
def __init__(self, host, port):
# this is the host and port to connect to
def sendToServer(self, whatToSend):
# send some line to the remote server
def startService(self):
# call me before using the service. starts outgoing connection efforts.
def stopService(self):
# clean reactor shutdowns should call this method. stops outgoing
# connection efforts.
(That might be all the interface you need, but it should be fairly clear where you can add things to this.)
The startService() and stopService() methods here are just what Twisted's Services expose. And helpfully, there is a premade Twisted Service which acts like a TCP client and takes care of all the reactor stuff for you. It's twisted.application.internet.TCPClient, which takes arguments for a remote host and port, along with a ProtocolFactory to take care of handling the actual connection attempt.
Here is the SayStuffToServerService, implemented as a subclass of TCPClient:
from twisted.application import internet
class SayStuffToServerService(internet.TCPClient):
factoryclass = SayStuffToServerProtocolFactory
def __init__(self, host, port):
self.factory = self.factoryclass()
internet.TCPClient.__init__(self, host, port, self.factory)
def sendToServer(self, whatToSend):
# we'll do stuff here
(See below for the SayStuffToServerProtocolFactory.)
Using this Service architecture is convenient in a lot of ways; you can group Services together in one container, so that they all get stopped and started as one when you have different parts of your app that you want active. It may make good sense to implement other parts of your app as separate Services. You can set Services as child services to application- the magic name that twistd looks for in order to know how to initialize, daemonize, and shut down your app. Actually yes, let's add some code to do that now.
from twisted.application import service
...
application = service.Application('say-stuff')
sttss = SayStuffToServerService('localhost', 65432)
sttss.setServiceParent(service.IServiceCollection(application))
That's all. Now when you run this module under twistd (i.e., for debugging, twistd -noy saystuff.py), that application will be started under the right reactor, and it will in turn start the SayStuffToServerService, which will start a connection effort to localhost:65432, which will use the service's factory attribute to set up the connection and the Protocol. You don't need to call reactor.run() or attach things to the reactor yourself anymore.
So we haven't implemented SayStuffToServerProtocolFactory yet. Since it sounds like you would prefer that your client reconnect if it has lost the connection (so that callers of sendToServer can usually just assume that there's a working connection), I'm going to put this protocol factory on top of ReconnectingClientFactory.
from twisted.internet import protocol
class SayStuffToServerProtocolFactory(protocol.ReconnectingClientFactory):
_my_live_proto = None
protocol = SayStuffToServerProtocol
This is a pretty nice minimal definition, which will keep trying to make outgoing TCP connections to the host and port we specified, and instantiate a SayStuffToServerProtocol each time. When we fail to connect, this class will do nice, well-behaved exponential backoff so that your network doesn't get hammered (you can set a maximum wait time). It will be the responsibility of the Protocol to assign to _my_live_proto and call this factory's resetDelay() method, so that exponential backoff will continue to work as expected. And here is that Protocol now:
class SayStuffToServerProtocol(basic.LineReceiver):
def connectionMade(self):
# if there are things you need to do on connecting to ensure the
# connection is "all right" (maybe authenticate?) then do that
# before calling:
self.factory.resetDelay()
self.factory._my_live_proto = self
def connectionLost(self, reason):
self.factory._my_live_proto = None
del self.factory
def sayStuff(self, stuff):
self.sendLine(stuff)
def lineReceived(self, line):
# do whatever you want to do with incoming lines. often it makes sense
# to have a queue of Deferreds on a protocol instance like this, and
# each incoming response gets sent to the next queued Deferred (which
# may have been pushed on the queue after sending some outgoing
# message in sayStuff(), or whatever).
pass
This is implemented on top of twisted.protocols.basic.LineReceiver, but would work as well with any other sort of Protocol, in case your protocol isn't line-oriented.
The only thing left is hooking up the Service to the right Protocol instance. This is why the Factory keeps a _my_live_proto attribute, which should be set when a connection is successfully made, and cleared (set to None) when that connection is lost. Here's the new implementation of SayStuffToServerService.sendToServer:
class NotConnectedError(Exception):
pass
class SayStuffToServerService(internet.TCPClient):
...
def sendToServer(self, whatToSend):
if self.factory._my_live_proto is None:
# define here whatever behavior is appropriate when there is no
# current connection (in case the client can't connect or
# reconnect)
raise NotConnectedError
self.factory._my_live_proto.sayStuff(whatToSend)
And now to tie it all together in one place:
from twisted.application import internet, service
from twisted.internet import protocol
from twisted.protocols import basic
class SayStuffToServerProtocol(basic.LineReceiver):
def connectionMade(self):
# if there are things you need to do on connecting to ensure the
# connection is "all right" (maybe authenticate?) then do that
# before calling:
self.factory.resetDelay()
self.factory._my_live_proto = self
def connectionLost(self, reason):
self.factory._my_live_proto = None
del self.factory
def sayStuff(self, stuff):
self.sendLine(stuff)
def lineReceived(self, line):
# do whatever you want to do with incoming lines. often it makes sense
# to have a queue of Deferreds on a protocol instance like this, and
# each incoming response gets sent to the next queued Deferred (which
# may have been pushed on the queue after sending some outgoing
# message in sayStuff(), or whatever).
pass
class SayStuffToServerProtocolFactory(protocol.ReconnectingClientFactory):
_my_live_proto = None
protocol = SayStuffToServerProtocol
class NotConnectedError(Exception):
pass
class SayStuffToServerService(internet.TCPClient):
factoryclass = SayStuffToServerProtocolFactory
def __init__(self, host, port):
self.factory = self.factoryclass()
internet.TCPClient.__init__(self, host, port, self.factory)
def sendToServer(self, whatToSend):
if self.factory._my_live_proto is None:
# define here whatever behavior is appropriate when there is no
# current connection (in case the client can't connect or
# reconnect)
raise NotConnectedError
self.factory._my_live_proto.sayStuff(whatToSend)
application = service.Application('say-stuff')
sttss = SayStuffToServerService('localhost', 65432)
sttss.setServiceParent(service.IServiceCollection(application))
Hopefully that gives enough of a framework with which to start. There is sometimes a lot of plumbing to do to handle client disconnections just the way you want, or to handle out-of-order responses from the server, or handle various sorts of timeout, canceling pending requests, allowing multiple pooled connections, etc, etc, but this should help.

The twisted framework is event-based programming; and by nature, its method is all called in async, and result is get by defer object.
The framework's nature is approprivate for protocol developing, just you have to change your minding from traditional sequential programming. The Protocol class is like a finite state machine with events like: connection make, connection lost, receive data.
You can convert your client code into FSM and then will be easily to fit into the Protocol class.
Below is an rough example of what I want to express. A bit of rouge, but this is i can provide now:
class SyncTransport(Protocol):
# protocol
def dataReceived(self, data):
print 'receive data', data
def connectionMade(self):
print 'i made a sync connection, wow'
self.transport.write('x')
self.state = I_AM_LIVING
def connectionLost(self):
print 'i lost my sync connection, sight'
def send(self, data):
if self.state == I_AM_LIVING:
if data == 'x':
self.transport.write('y')
if data == 'Y':
self.transport.write('z')
self.state = WAITING_DEAD
if self.state == WAITING_DEAD:
self.transport.close()

how to send data with twisted protocol via factory

I'm writing a client implementing a custom protocol, and have a factory for it. My problem is the following: my client has bi-dir communication, and sometimes I want to tell it "send this data". But all I have is the factory object:
class MyFactory(ClientFactory):
protocol = MyProtocol
def __init__(self, recv_callback):
self.recv_callback = recv_callback
def send_message(self, msg):
self.protocol.send_message(msg)
So I create a factory and have a factory object, I don't the protocol object. When send_message above is called I get an error because self.protocol is just a class, not an object.
How can I do this? Should I also expose the protocol for connection in addition to the factory?
Thanks

You have access to all of the objects you want. The factory is responsible for creating protocol instances, so if you want to keep the protocol instance around where the factory can use it, override buildProtocol and save the instance:
class MyFactory(ClientFactory):
protocol = MyProtocol
...
def buildProtocol(self, address):
proto = ClientFactory.buildProtocol(self, address)
self.connectedProtocol = proto
return proto
However, this approach is lacking in one important feature. It does not make it easy to tell when buildProtocol has been called and connectedProtocol has been set. If you try to use this attribute naively:
factory = MyFactory()
reactor.connectTCP(host, port, factory)
factory.connectedProtocol.send_message(...)
The code will fail with an AttributeError because the connection has not yet actually been set up. Since Twisted is event driven, you need to make sure to use this code by responding to an event that says the connection has been set up.
You might do this by firing a callback when the protocol is constructed instead of just setting an attribute. Twisted actually has a helper factory which does something like this already:
from twisted.internet.protocol import ClientCreator
cc = ClientCreator(reactor, MyProtocol)
whenConnected = cc.connectTCP(host, port)
# Or the equivalent with endpoints
# from twisted.internet.endpoints import TCP4ClientEndpoint
# from twisted.internet.protocol import ClientFactory
# endpoint = TCP4ClientEndpoint(reactor, host, port)
# factory = ClientFactory()
# factory.protocol = MyProtocol
# whenConnected = endpoint.connect(factory)
def cbConnected(connectedProtocol):
connectedProtocol.send_message(...)
def ebConnectError(reason):
# Connection attempt failed, perhaps retry
...
whenConnected.addCallbacks(cbConnected, ebConnectError)
You could also save the reference to connectedProtocol in the cbConnected callback so that you can continue to use it later on. You might also start whatever other operations want to use the connected protocol in cbConnected, so that they don't try to use the connection before it is actually available.

Running a function periodically in twisted protocol

I am looking for a way to periodically send some data over all clients connected to a TCP port. I am looking at twisted python and I am aware of reactor.callLater. But how do I use it to send some data to all connected clients periodically ? The data sending logic is in Protocol class and it is instantiated by the reactor as needed. I don't know how to tie it from reactor to all protocol instances...

You would probably want to do this in the Factory for the connections. The Factory is not automatically notified of every time a connection is made and lost, so you can notify it from the Protocol.
Here is a complete example of how to use twisted.internet.task.LoopingCall in conjunction with a customised basic Factory and Protocol to announce that '10 seconds has passed' to every connection every 10 seconds.
from twisted.internet import reactor, protocol, task
class MyProtocol(protocol.Protocol):
def connectionMade(self):
self.factory.clientConnectionMade(self)
def connectionLost(self, reason):
self.factory.clientConnectionLost(self)
class MyFactory(protocol.Factory):
protocol = MyProtocol
def __init__(self):
self.clients = []
self.lc = task.LoopingCall(self.announce)
self.lc.start(10)
def announce(self):
for client in self.clients:
client.transport.write("10 seconds has passed\n")
def clientConnectionMade(self, client):
self.clients.append(client)
def clientConnectionLost(self, client):
self.clients.remove(client)
myfactory = MyFactory()
reactor.listenTCP(9000, myfactory)
reactor.run()

I'd imagine the easiest way to do that is to manage a list of clients in the protocol with connectionMade and connectionLost in the client and then use a LoopingCall to ask each client to send data.
That feels a little invasive, but I don't think you'd want to do it without the protocol having some control over the transmission/reception. Of course, I'd have to see your code to really know how it'd fit in well. Got a github link? :)