As I am going through the example in this blog, I found that when ClientFactory instance, which is an argument to PoetryClientFactory, or Protocol instance, which is an argument to PoetryProtocol, is not implementing all the functions defined for ClientFactory or Protocol interface. ClientFactory interface implements startedConnecting, clientConnectionFailed, and clientConnectionLost, but PoetryClientFactory does not implement startedConnecting and clientConnectionLost. What is happening?
# This is the Twisted Get Poetry Now! client, version 2.0.
# NOTE: This should not be used as the basis for production code.
import datetime, optparse
from twisted.internet.protocol import Protocol, ClientFactory
def parse_args():
usage = """usage: %prog [options] [hostname]:port ...
This is the Get Poetry Now! client, Twisted version 2.0.
Run it like this:
python get-poetry.py port1 port2 port3 ...
If you are in the base directory of the twisted-intro package,
you could run it like this:
python twisted-client-2/get-poetry.py 10001 10002 10003
to grab poetry from servers on ports 10001, 10002, and 10003.
Of course, there need to be servers listening on those ports
for that to work.
"""
parser = optparse.OptionParser(usage)
_, addresses = parser.parse_args()
if not addresses:
print parser.format_help()
parser.exit()
def parse_address(addr):
if ':' not in addr:
host = '127.0.0.1'
port = addr
else:
host, port = addr.split(':', 1)
if not port.isdigit():
parser.error('Ports must be integers.')
return host, int(port)
return map(parse_address, addresses)
class PoetryProtocol(Protocol):
poem = ''
task_num = 0
def dataReceived(self, data):
self.poem += data
msg = 'Task %d: got %d bytes of poetry from %s'
print msg % (self.task_num, len(data), self.transport.getPeer())
def connectionLost(self, reason):
self.poemReceived(self.poem)
def poemReceived(self, poem):
self.factory.poem_finished(self.task_num, poem)
class PoetryClientFactory(ClientFactory):
task_num = 1
protocol = PoetryProtocol # tell base class what proto to build
def __init__(self, poetry_count):
self.poetry_count = poetry_count
self.poems = {} # task num -> poem
def buildProtocol(self, address):
proto = ClientFactory.buildProtocol(self, address)
proto.task_num = self.task_num
self.task_num += 1
return proto
def poem_finished(self, task_num=None, poem=None):
if task_num is not None:
self.poems[task_num] = poem
self.poetry_count -= 1
if self.poetry_count == 0:
self.report()
from twisted.internet import reactor
reactor.stop()
def report(self):
for i in self.poems:
print 'Task %d: %d bytes of poetry' % (i, len(self.poems[i]))
def clientConnectionFailed(self, connector, reason):
print 'Failed to connect to:', connector.getDestination()
self.poem_finished()
def poetry_main():
addresses = parse_args()
start = datetime.datetime.now()
factory = PoetryClientFactory(len(addresses))
from twisted.internet import reactor
for address in addresses:
host, port = address
reactor.connectTCP(host, port, factory)
reactor.run()
elapsed = datetime.datetime.now() - start
print 'Got %d poems in %s' % (len(addresses), elapsed)
if __name__ == '__main__':
poetry_main()
I don't think I'm quite sure what you're asking, but essentially if you don't need to act on a certain event (such as when a client starts connecting or a connection is lost), you don't need to implement that function. It's just an interface mostly. If you don't implement those functions, an empty function that does nothing is called, from ClientFactory or Protocol, or whichever class you inherit from..
If you are implementing an interface yourself, it looks like this:
from zope.interface import implementer
from twisted.internet.interfaces import IProtocol
#implementer(IProtocol)
class MyProtocol(object):
" ... "
In this case, you do need to implement all methods, because this #implementer declaration is just saying that you intend to provide all the relevant methods.
However, the more common thing to do in Twisted is to subclass, like this:
from twisted.internet.protocol import Protocol
class MyProtocol(Protocol, object):
" ... "
In this case, you do not need to implement all methods, because the Protocol super-class already provides implementations of all the methods on IProtocol. In general, Twisted provides a superclass that has default or empty versions of all the methods for many of the more frequently-used interfaces that Twisted application developers must implement.
I'm not quite sure what you're asking, too.
IMO, reaching github for their source code is the fastest way to learn. As you can see, there are default implementation for startedConnecting and clientConnectionLost (empty code, though).
Therefore, you just need to implement callbacks you need but not all methods defined in that interface.
Related
Is it possible to share socket objects between 2 running processes?
To simplify the issue, assume that we have two files s1.py and s2.py. Both are tcp servers listening on different ports.
s1.py body
from twisted.internet import reactor, protocol
CLIENTS = []
class Echo(protocol.Protocol):
def connectionMade(self):
CLIENTS.append(self)
def dataReceived(self, data):
self.transport.write(data)
def connectionLost(self):
CLIENTS.remove(self)
def main():
factory = protocol.ServerFactory()
factory.protocol = Echo
reactor.listenTCP(8000, factory)
reactor.run()
main()
and s2.py
from twisted.internet import reactor, protocol
class Echo(protocol.Protocol):
def dataReceived(self, data):
for socket in list_of_serialized_redis_CLIENTS_socket_object:
socket.transport.write(data)
def main():
factory = protocol.ServerFactory()
factory.protocol = Echo
reactor.listenTCP(8001, factory)
reactor.run()
main()
Is it possible to share CLIENTS from s1.py with s2.py?
Maybe there is some way to serialize CLIENTS and store it in redis or so.
Using socket.share() might not be as straight forward as you might think.
First off, it's a Windows exclusive feature and Windows is the one platform that gets the least attention by Twisted.
Getting access to the low-level socket object in Twisted can be tricky because there are a lot of things happening behind the scenes and might cause strange outcomes if changed.
So in other words, it's certainly possible to go this route, but it's not recommended.
If you're open to new solutions, I think an RPC-style server will solve your issue.
My thinking is, since the connection objects live in s1, simply create a remote process that will "do stuff" to those connections.
Other processes can connect to that server and execute functions on objects local to s1. Twisted's Perspective Broker is a ready-made solution you could leverage.
s1.py
from twisted.internet import reactor, protocol, endpoints
from twisted.spread import pb
class Echo(protocol.Protocol):
def connectionMade(self):
self.factory.client_set.add(self)
def connectionLost(self, reason):
self.factory.client_set.remove(self)
class Remote(pb.Root):
def __init__(self, client_set):
self.client_set = client_set
def remote_s1_write(self, data):
msg_tmpl = 'From s1...{0}'
for client in self.client_set:
response = msg_tmpl.format(data).encode('utf-8')
client.transport.write(response)
def main():
client_set = set() # container will be shared between the 2 servers
# setup Echo server
factory = protocol.ServerFactory()
factory.protocol = Echo
factory.client_set = client_set
echo_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=8000)
echo_server.listen(factory)
# setup PB server
pb_root = Remote(client_set)
pb_factory = pb.PBServerFactory(pb_root)
pb_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=7999)
pb_server.listen(pb_factory)
reactor.run()
main()
s2.py
from twisted.internet import reactor, protocol, endpoints
from twisted.spread import pb
class Echo(protocol.Protocol):
def dataReceived(self, data):
remote = self.factory.pb_factory.getRootObject()
remote.addCallback(lambda obj: obj.callRemote('s1_write', data))
remote.addCallback(lambda echo: 's1 said: {0}'.format(data))
remote.addErrback(lambda reason: 'error: {0}'.format(str(reason.value)))
remote.addCallback(lambda response: print(response))
self.transport.write(data)
def main():
# connect to s1's PB server
pb_factory = pb.PBClientFactory()
pb_connection = endpoints.TCP4ClientEndpoint(reactor, host='localhost', port=7999)
pb_connection.connect(pb_factory)
# setup Echo server
factory = protocol.ServerFactory()
factory.protocol = Echo
factory.pb_factory = pb_factory
echo_server = endpoints.TCP4ServerEndpoint(reactor, interface='localhost', port=8001)
echo_server.listen(factory)
reactor.run()
main()
I've taken the liberty and updated the syntax. Try this code and see if it works and ask if you have any issues.
I've inherited a Twisted MultiService that I'm trying to add tests to, but whatever I do, I end up with a DirtyReactorAggregateError. The service connects to a server with a twisted.application.internet.TCPClient. I think the error is because the TCPClient is not disconnecting, but I'm not sure how I'm supposed to disconnect it. What is the proper way to test a Twisted Service with a client in it?
Here's the test case:
from labrad.node import *
from twisted.trial import unittest
import os
from socket import gethostname
class NodeTestCase(unittest.TestCase):
def setUp(self):
import labrad
name = os.environ.get('LABRADNODE', gethostname()) + '_test'
self.node = Node(name,
labrad.constants.MANAGER_HOST,
labrad.constants.MANAGER_PORT)
self.node.startService()
#self.addCleanup(self.node.stopService)
def test_nothing(self):
self.assertEqual(3, 3)
def tearDown(self):
return self.node.stopService()
Here's the Node service itself:
class Node(MultiService):
"""Parent Service that keeps the node running.
If the manager is stopped or we lose the network connection,
this service attempts to restart it so that we will come
back online when the manager is back up.
"""
reconnectDelay = 10
def __init__(self, name, host, port):
MultiService.__init__(self)
self.name = name
self.host = host
self.port = port
def startService(self):
MultiService.startService(self)
self.startConnection()
def startConnection(self):
"""Attempt to start the node and connect to LabRAD."""
print 'Connecting to %s:%d...' % (self.host, self.port)
self.node = NodeServer(self.name, self.host, self.port)
self.node.onStartup().addErrback(self._error)
self.node.onShutdown().addCallbacks(self._disconnected, self._error)
self.cxn = TCPClient(self.host, self.port, self.node)
self.addService(self.cxn)
def stopService(self):
if hasattr(self, 'cxn'):
d = defer.maybeDeferred(self.cxn.stopService)
self.removeService(self.cxn)
del self.cxn
return defer.gatherResults([MultiService.stopService(self), d])
else:
return MultiService.stopService(self)
def _disconnected(self, data):
print 'Node disconnected from manager.'
return self._reconnect()
def _error(self, failure):
r = failure.trap(UserError)
if r == UserError:
print "UserError found!"
return None
print failure.getErrorMessage()
return self._reconnect()
def _reconnect(self):
"""Clean up from the last run and reconnect."""
## hack: manually clearing the dispatcher...
dispatcher.connections.clear()
dispatcher.senders.clear()
dispatcher._boundMethods.clear()
## end hack
if hasattr(self, 'cxn'):
self.removeService(self.cxn)
del self.cxn
reactor.callLater(self.reconnectDelay, self.startConnection)
print 'Will try to reconnect in %d seconds...' % self.reconnectDelay
You should refactor your service so that it can use something like MemoryReactor from twisted.test.proto_helpers (the one public module in the twisted.test package, although hopefully it will move out of twisted.test eventually).
The way that you use MemoryReactor is to pass it into your code as the reactor to use. If you then want to see what happens when the connection succeeds, look at some of its public attributes - tcpClients for connectTCP, tcpServers for listenTCP, etc. Your tests can then pull out the Factory instances which were passed to connectTCP/listenTCP, etc, and call buildProtocol on them and makeConnection on the result. To get an ITransport implementation you can use twisted.test.proto_helpers.StringTransportWithDisconnection. You might even look at the (private API! be careful! it'll break without warning! although it really should be public) twisted.test.iosim.IOPump to relay traffic between clients and servers.
If you really actually need to do whole-system non-deterministic real-world testing, with all the complexity and random unrelated failures that implies, here's an article on actually shutting down a client and server all the way.
I had a similar issue with trying to test a instance of an application. I ended up creating a Python base class that used setUp and tearDown methods to start/stop the application.
from twisted.application.app import startApplication
from twisted.application.service import IServiceCollection
from twisted.internet.defer import inlineCallbacks
from twisted.trial import unittest
class MyTest(unittest.TestCase):
def setUp(self):
startApplication(self.app, save=False)
#inlineCallbacks
def tearDown(self):
sc = self.app.getComponent(IServiceCollection)
yield sc.stopService()
I am creating a robot which is going to be driven by the commands received over TCP connection. Therefore, I will have a robot class with methods (e.g. sense(), drive()...) and the class for TCP connection.
To establish TCP connection, I looked at examples from twisted. On the client side, I have written a client.py script for connection handling:
from twisted.internet import reactor, protocol
import random
from eventhook import EventHook
import common
#from Common.socketdataobjects import response
# a client protocol
class EchoClient(protocol.Protocol):
"""Once connected, send a message, then print the result."""
def connectionMade(self):
self.transport.write("hello, world!")
#the server should be notified that the connection to the robot has been established
#along with robot state (position)
#eventConnectionEstablishedHook.fire()
def dataReceived(self, data):
print "Server said:", data
self.transport.write("Hello %s" % str(random.randint(1,10)))
'''
serverMessage = common.deserializeJson(data)
command = serverMessage.command
arguments = serverMessage.arguments
#here we get for example command = "DRIVE"
#arguments = {motor1Speed: 50, motor2Speed: 40}
instead of above response, used for testing purposes,
the commands should be extracted from the data and according to the command,
the method in Robot instance should be called.
When the command execution finishes, the self.transport.write() method should be called
to notify the server that the command execution finished
'''
def connectionLost(self, reason):
print "connection lost"
class EchoFactory(protocol.ClientFactory):
protocol = EchoClient
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
reactor.stop()
# this connects the protocol to a server runing on port 8000
def initializeEventHandlers(connectionEstablishedHook):
global connection
connection.established = 0
global eventConnectionEstablishedHook
eventConnectionEstablishedHook = connectionEstablishedHook
def main():
f = EchoFactory()
reactor.connectTCP("localhost", 8000, f)
reactor.run()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
Beside this script, I have a robot class:
Class Robot(object():
def __init(self)__:
self.position = (0,0)
def drive(self, speedMotor1, speedMotor2, driveTime)
updateMotor1State(speedMotor1)
updateMotor2State(speedMotor2)
time.sleep(driveTime)
#when the execution finished, the finish status should be sent to client in order to inform the server
return "Finished"
def sense(self)
#logic to get the data from the environment
What I would like to do, is to receive the data(commands) from TCP connection and then call the according method in Robot instance. Some procedures might take longer (e.g. driving), so I tried to use events, but haven't figured out the appropriate way to communicate between TCP client and robot using events:
if __name__ == '__main__':
robotController = Robot()
eventController = Controller()
connectionEstablishedHook = EventHook()
client.initializeEventHandlers(connectionEstablishedHook)
eventController.connection = connectionEstablishedHook
client.main()
I tried to create ClientMainProgram script, where I wanted to create an instance of a robot, an instance of TCP client and implement the communication between them using events.
Previously I have managed to implement event handling using Michael Foord's events pattern on a simpler example. I would be very thankful if anyone could provide the solution to this question or any similar example which might be helpful to solve this problem.
Events are easily represented using regular Python function calls.
For example, if your protocol looks like this:
from twisted.internet.protocol import Protocol
class RobotController(Protocol):
def __init__(self, robot):
self.robot = robot
def dataReceived(self, data):
for byte in data:
self.commandReceived(byte)
def commandReceived(self, command):
if command == "\x00":
# drive:
self.robot.drive()
elif command == "\x01":
# sense:
self.robot.sense()
...
(The specifics of the protocol used in this example are somewhat incidental. I picked this protocol because it's very simple and has almost no parsing logic. For your real application I suggest you use twisted.protocols.amp.)
Then all you need to do is make sure the robot attribute is properly initialized. You can do this easily using the somewhat newer endpoint APIs that can often replace use of factories:
from sys import argv
from twisted.internet.endpoints import clientFromString, connectProtocol
from twisted.internet.task import react
def main(reactor, description):
robot = ...
endpoint = clientFromString(reactor, description)
connecting = connectProtocol(endpoint, RobotController(robot))
def connected(controller):
...
connecting.addCallback(connected)
return connecting
react(main, argv[1:])
I am using Twisted to create a simple TCP server. With the Twisted Protocol, Is it ipossible to specific the exact number of bytes returned by dataReceived?
from twisted.internet.protocol import Factory, Protocol
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor
class TestProtocol(Protocol):
def connectionMade(self):
print(id(self))
self.transport.write('hello')
def connectionLost(self, reason):
print('connection lost called')
def dataReceived(self, data):
# is it possible to specify size of "data"?
print('data received called')
class TestFactory(Factory):
protocol = TestProtocol
endpoint = TCP4ServerEndpoint(reactor, 8007)
endpoint.listen(TestFactory())
reactor.run()
I ask this question because if I can control the number of bytes coming in, I can avoid dealing with partial protocol messages or multiple protocol messages coming in a single dataReceived callback.
I was able to achieve this with asynccore by specifying the exact number of bytes in the recv() method. It would be great to be able to do this in Twisted as well.
Thanks
... Alan
Why not define a buffer and use that to grab messages of the size you want. LineReciever does something similar.
Something along the lines of:
from twisted.internet.protocol import Factory, Protocol
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor
class TestProtocol(Protocol):
def __init__(self):
self.__buffer = ""
self.frame_size = 3 #FRAME SIZE HERE
def connectionMade(self):
print(id(self))
self.transport.write('hello')
def connectionLost(self, reason):
print('connection lost called')
def dataReceived(self, data):
# is it possible to specify size of "data"?
print('data received called')
self.__buffer = self.__buffer+data
frame_size = self.frame_size
while len(self.__buffer) >= frame_size:
self.frame_received(self.__buffer[0:frame_size])
self.__buffer=self.__buffer[frame_size:]
def frame_received(self,data):
print data
class TestFactory(Factory):
protocol = TestProtocol
endpoint = TCP4ServerEndpoint(reactor, 8007)
endpoint.listen(TestFactory())
reactor.run()
Just deal with the partial protocol messages.
Twisted provides numerous classes to help you do this in the twisted.protocols.basic package. Normally one of those should just work for you; it's pretty unusual to have to implement your own custom framing protocol.
If you're designing your own protocol, you should probably just use twisted's built-in protocol construction kit, the Asynchronous Messaging Protocol. (Support for other langauges and frameworks available at amp-protocol.net.
Im trying to write a program that would be listening for data (simple text messages) on some port (say tcp 6666) and then pass them to one or more different protocols - irc, xmpp and so on. I've tried many approaches and digged the Internet, but I cant find easy and working solution for such task.
The code I am currently fighting with is here: http://pastebin.com/ri7caXih
I would like to know how to from object like:
ircf = ircFactory('asdfasdf', '#asdf666')
get access to self protocol methods, because this:
self.protocol.dupa1(msg)
returns error about self not being passed to active protocol object. Or maybe there is other, better, easier and more kosher way to create single reactor with multiple protocols and have actions triggeres when a message arrives on any of them, and then pass that message to other protocols for handling/processing/sending?
Any help will be highly appreciated!
Here is sample code to read from multiple connections to port 9001 and write out to a connection on port 9000. You would need multiple "PutLine" implementations, one for XMPP, IRC, MSN, etc.
I used a global to store the output connection PutLine but you would want to create a more complex Factory object that would handle this instead.
#!/usr/bin/env python
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.endpoints import clientFromString, serverFromString
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor
queue = []
putter = None
class GetLine(LineReceiver):
delimiter = '\n'
def lineReceived(self, line):
queue.append(line)
putter.have_data()
self.sendLine(line)
class PutLine(LineReceiver):
def __init__(self):
global putter
putter = self
print 'putline init called %s' % str(self)
def have_data(self):
line = queue.pop()
self.sendLine(line)
def main():
f = Factory()
f.protocol = PutLine
endpoint = clientFromString(reactor, "tcp:host=localhost:port=9000")
endpoint.connect(f)
f = Factory()
f.protocol = GetLine
endpoint2 = serverFromString(reactor, "tcp:port=9001")
endpoint2.listen(f)
reactor.run()
if __name__ == '__main__':
main()
Testing:
nc -l 9000
python test.py
nc 9001
Data entered form any number of nc 9001 (or netcat 9001) will appear on nc -l 9000.
This is answered in the FAQ.
http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputononeconnectionresultinoutputonanother
See doc/core/examples/chatserver.py. There they've added hooks to the Protocol's connectionMade and connectionLost methods to maintain a list of connected clients, and then it iterates through all of them when a message arrives to pass on.