Develop Reference

Python TCP Sockets - Single Client to Multiple Servers

I have about 200 ip controlled power-bars that I need to connect to and control over TCP sockets. So far I can connect to a single power-bar and control it without issue. Where my issue lies is in how to connect to, play ping pong, as well as send and receive commands for all 200 in one client.
I have researched, hopefully exhaustively, and at most all I can find is a pointer towards select, or twisted - but only for multiple clients connecting to a single server (whereas I need the reverse). All I really need is a prod in the right direction. I can create the sockets for all 200, but I cannot for the life of me figure out how to connect to each device using the IP and Port (60000) and send and receive the proper messages in a non-blocking manner.
Any pointers in the general direction would be greatly appreciated. Hopefully this answer will help someone else with a similar issue to solve. Thanks.

You're learning about non-blocking patterns in Python at the right time ;) There's a plethora of ways to do it so I'm not really surprised you're confused. You've named twisted, which is the most mature framework, and there's also asyncio, which is built into Python 3+. Pick which ever one is easiest for you to learn. As you can see, they're very similar in style.
asyncio_client.py
import asyncio
from uuid import uuid4
class Echo(asyncio.Protocol):
def __init__(self):
self.identity = uuid4().hex
def connection_made(self, transport):
message = '{}: hello world'.format(self.identity)
transport.write(message.encode())
def data_received(self, data):
print(data.decode())
def echo_factory():
return Echo()
async def connect_to_server(loop):
await loop.create_connection(echo_factory, host='127.0.0.1', port=6000)
def main():
loop = asyncio.get_event_loop()
loop.create_task(connect_to_server(loop))
loop.create_task(connect_to_server(loop))
loop.create_task(connect_to_server(loop))
loop.run_forever()
main()
twisted_client.py
from uuid import uuid4
from twisted.internet import endpoints, protocol, reactor
class Echo(protocol.Protocol):
def __init__(self):
self.identity = uuid4().hex
def connectionMade(self):
message = '{}: hello world'.format(self.identity)
self.transport.write(message.encode())
def dataReceived(self, data):
print(data.decode())
def connect_to_server(factory):
return endpoints.clientFromString(reactor, 'tcp:6000:host=127.0.0.1').connect(factory)
def main():
factory = protocol.ClientFactory.forProtocol(Echo)
connect_to_server(factory)
connect_to_server(factory)
connect_to_server(factory)
reactor.run()
main()

Twisted doesn't exit when reactor.stop is called

I am trying to reimplement netcat in Python:
#!/usr/bin/env python2
from sys import stdin, stdout
from twisted.internet import reactor
from twisted.internet.protocol import Protocol
from twisted.internet.endpoints import TCP4ClientEndpoint, connectProtocol
class NcClient(Protocol):
def dataReceived(self, data):
stdout.write(data)
def sendData(self, data):
self.transport.write(data)
self.transport.write("\n")
client = NcClient()
def cmdloop():
while True:
line = stdin.readline()
if line == "":
break
else:
client.sendData(line)
if reactor.running:
reactor.stop()
point = TCP4ClientEndpoint(reactor, "localhost", 6004)
connectProtocol(point, client)
reactor.callInThread(cmdloop)
reactor.run()
When cmdloop detects end of input, it calls reactor.stop.
As I understand, reactor.stop sends shutdown events to all things managed by Twisted, such as threads, connections etc. In response to these events connections are closed, threads wait for the completion of their procedures etc. So when reactor.stop() is called, the connection to localhost:6004 should close, and the program should exit.
However, it doesn't happen immediately, but only when NcClient receives a message from server. As if it is reading those messages blockingly, in a loop, and only when it receives one does it proceed to handle shutdown requests.
How to make it shut down before receiving a message? I know of reactor.crash(), but is there a more polite option?

Your main problem is that cmdloop is running in a non-reactor thread, and yet it is calling reactor methods other than callFromThread (specifically: transport.write via client.sendData). The documentation is quite clear on this:
Methods within Twisted may only be invoked from the reactor thread unless otherwise noted. Very few things within Twisted are thread-safe.
Luckily, implementing the equivalent of netcat doesn't require threading at all. You can simply use Twisted's built-in support for standard I/O as a source of data. Here's an example version:
import sys
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.endpoints import clientFromString
from twisted.internet.stdio import StandardIO
class NcClient(Protocol):
def __init__(self, forwardTo):
self.forwardTo = forwardTo
def connectionMade(self):
self.transport.registerProducer(self.forwardTo.transport, True)
self.forwardTo.transport.resumeProducing()
self.forwardTo.transport.registerProducer(self.transport, True)
def dataReceived(self, data):
self.forwardTo.transport.write(data)
def connectionLost(self, reason):
reactor.stop()
class StdIo(Protocol):
def connectionMade(self):
self.transport.pauseProducing()
f4p = Factory.forProtocol(lambda: NcClient(self))
d = endpoint.connect(f4p)
#d.addCallback
def connected(proto):
self.client = proto
def dataReceived(self, data):
self.client.transport.write(data)
def connectionLost(self, reason):
reactor.stop()
endpoint = clientFromString(reactor, sys.argv[1])
output = StandardIO(proto=StdIo(), reactor=reactor)
reactor.run()
If you still want to use threads for some reason, you can modify NcClient to use callFromThread to invoke sendData instead of calling it directly.

Distinguishing outgoing and incoming connections in twisted

I'm working on a P2P application based on the Twisted framework. As such I can have both incoming as well as outgoing connections. Is there a simple way to distinguish them? Currently I just create another Factory that marks the connection as outgoing and delegates all factory calls to the original factory, but there must be a simpler way.
class OutgoingProtocolFactory(MyProtocolFactory):
"""
A rather simple factory that is used to earmark connections as outgoing.
"""
def __init__(self, parentFactory):
self.factory = parentFactory
def buildProtocol(self, addr):
connection = MyProtocolFactory.buildProtocol(self.factory, addr)
connection.factory = self.factory
connection.incoming = False
return connection
def clientConnectionFailed(self, connector, reason):
self.factory.clientConnectionFailed(connector, reason)
def clientConnectionLost(self, connector, reason):
self.factory.clientConnectionLost(connector, reason)
Any thoughts?

Each protocol instance constructed by any of the factories included in Twisted get a free factory attribute that refers back to the factory that created them.
By convention, accepted (server) connections have a ServerFactory (or subclass) instance as their factory and connected (client) connections have a ClientFactory (or subclass) instance.
However, if you really want your code to be completely general, you can't rely on this, and it's better to keep track of it yourself, as it sounds like you're currently doing.
In case the code you currently have for forwarding calls from one factory to another is more cumbersome than necessary, here's an example of how simple it could be:
from twisted.internet.protocol import ClientFactory
from twisted.protocols.policies import WrappingFactory
IN, OUT = 1, 2
class YourFactory(ClientFactory):
# Your application logic
...
class Incoming(WrappingFactory):
direction = IN
def buildProtocol(self, addr):
protocol = self.wrappedFactory.buildProtocol(addr)
protocol.direction = self.direction
return protocol
class Outgoing(Incoming):
direction = OUT
yourFactory = YourFactory(...)
reactor.listenTCP(0, Incoming(yourFactory))
reactor.connectTCP('example.com', 1234, Outgoing(yourFactory))
If that's still too complicated for you, then I don't know what to say. I'm not clever enough to think of a simpler solution.

Irregular Transmission Problem with Python Twisted Push Producer

I want to transmit data from a Queue using Twisted. I currently use a push producer to poll the queue for items and write to the transport.
class Producer:
implements(interfaces.IPushProducer)
def __init__(self, protocol, queue):
self.queue = queue
self.protocol = protocol
def resumeProducing(self):
self.paused = False
while not self.paused:
try:
data = self.queue.get_nowait()
logger.debug("Transmitting: '%s'", repr(data))
data = cPickle.dumps(data)
self.protocol.transport.write(data + "\r\n")
except Empty:
pass
def pauseProducing(self):
logger.debug("Transmitter paused.")
self.paused = True
def stopProducing(self):
pass
The problem is, that the data are sent very irregularly and if only one item was in the queue, the data is never going to be sent. It seems that Twisted waits until the data to be transmitted has grown to a specific value until it transmits it. Is the way I implemented my producer the right way? Can I force Twisted to transmit data now?
I've also tried using a pull producer, but Twisted does not call the resumeProducing() method of it at all. Do I have to call the resumeProducer() method from outside, when using a pull producer?

It's hard to say why your producer doesn't work well without seeing a complete example (that is, without also seeing the code that registers it with a consumer and the code which is putting items into that queue).
However, one problem you'll likely have is that if your queue is empty when resumeProducing is called, then you will write no bytes at all to the consumer. And when items are put into the queue, they'll sit there forever, because the consumer isn't going to call your resumeProducing method again.
And this generalizes to any other case where the queue does not have enough data in it to cause the consumer to call pauseProducing on your producer. As a push producer, it is your job to continue to produce data on your own until the consumer calls pauseProducing (or stopProducing).
For this particular case, that probably means that whenever you're going to put something in that queue - stop: check to see if the producer is not paused, and if it is not, write it to the consumer instead. Only put items in the queue when the producer is paused.

Here are two possible solutions:
1) Periodically poll your local application to see if you have additional data to send.
NB. This relies on a periodic async callback from the deferLater method in twisted. If you need a responsive application that sends data on demand, or a long running blocking operation (eg. ui that uses its own event loop) it may not be appropriate.
Code:
from twisted.internet.protocol import Factory
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet.interfaces import IPushProducer
from twisted.internet.task import deferLater, cooperate
from twisted.internet.protocol import Protocol
from twisted.internet import reactor
from zope.interface import implementer
import time
# Deferred action
def periodically_poll_for_push_actions_async(reactor, protocol):
while True:
protocol.send(b"Hello World\n")
yield deferLater(reactor, 2, lambda: None)
# Push protocol
#implementer(IPushProducer)
class PushProtocol(Protocol):
def connectionMade(self):
self.transport.registerProducer(self, True)
gen = periodically_poll_for_push_actions_async(self.transport.reactor, self)
self.task = cooperate(gen)
def dataReceived(self, data):
self.transport.write(data)
def send(self, data):
self.transport.write(data)
def pauseProducing(self):
print 'Workload paused'
self.task.pause()
def resumeProducing(self):
print 'Workload resumed'
self.task.resume()
def stopProducing(self):
print 'Workload stopped'
self.task.stop()
def connectionLost(self, reason):
print 'Connection lost'
try:
self.task.stop()
except:
pass
# Push factory
class PushFactory(Factory):
def buildProtocol(self, addr):
return PushProtocol()
# Run the reactor that serves everything
endpoint = TCP4ServerEndpoint(reactor, 8089)
endpoint.listen(PushFactory())
reactor.run()
2) Manually keep track of Protocol instances and use reactor.callFromThread() from a different thread. Lets you get away with a long blocking operation in the other thread (eg. ui event loop).
Code:
from twisted.internet.protocol import Factory
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet.interfaces import IPushProducer
from twisted.internet.task import deferLater, cooperate
from twisted.internet.protocol import Protocol
from twisted.internet import reactor, threads
import time
import random
import threading
# Connection
protocol = None
# Some other thread that does whatever it likes.
class SomeThread(threading.Thread):
def run(self):
while True:
print("Thread loop")
time.sleep(random.randint(0, 4))
if protocol is not None:
reactor.callFromThread(self.dispatch)
def dispatch(self):
global protocol
protocol.send("Hello World\n")
# Push protocol
class PushProtocol(Protocol):
def connectionMade(self):
global protocol
protocol = self
def dataReceived(self, data):
self.transport.write(data)
def send(self, data):
self.transport.write(data)
def connectionLost(self, reason):
print 'Connection lost'
# Push factory
class PushFactory(Factory):
def buildProtocol(self, addr):
return PushProtocol()
# Start thread
other = SomeThread()
other.start()
# Run the reactor that serves everything
endpoint = TCP4ServerEndpoint(reactor, 8089)
endpoint.listen(PushFactory())
reactor.run()
Personally, I find the fact that IPushProducer and IPullProducer require a periodic callback, makes them less useful. Others disagree... shrug. Take your pick.

Running a function periodically in twisted protocol

I am looking for a way to periodically send some data over all clients connected to a TCP port. I am looking at twisted python and I am aware of reactor.callLater. But how do I use it to send some data to all connected clients periodically ? The data sending logic is in Protocol class and it is instantiated by the reactor as needed. I don't know how to tie it from reactor to all protocol instances...

You would probably want to do this in the Factory for the connections. The Factory is not automatically notified of every time a connection is made and lost, so you can notify it from the Protocol.
Here is a complete example of how to use twisted.internet.task.LoopingCall in conjunction with a customised basic Factory and Protocol to announce that '10 seconds has passed' to every connection every 10 seconds.
from twisted.internet import reactor, protocol, task
class MyProtocol(protocol.Protocol):
def connectionMade(self):
self.factory.clientConnectionMade(self)
def connectionLost(self, reason):
self.factory.clientConnectionLost(self)
class MyFactory(protocol.Factory):
protocol = MyProtocol
def __init__(self):
self.clients = []
self.lc = task.LoopingCall(self.announce)
self.lc.start(10)
def announce(self):
for client in self.clients:
client.transport.write("10 seconds has passed\n")
def clientConnectionMade(self, client):
self.clients.append(client)
def clientConnectionLost(self, client):
self.clients.remove(client)
myfactory = MyFactory()
reactor.listenTCP(9000, myfactory)
reactor.run()

I'd imagine the easiest way to do that is to manage a list of clients in the protocol with connectionMade and connectionLost in the client and then use a LoopingCall to ask each client to send data.
That feels a little invasive, but I don't think you'd want to do it without the protocol having some control over the transmission/reception. Of course, I'd have to see your code to really know how it'd fit in well. Got a github link? :)