I've been trying to wrap my mind around how to get Twisted to perform, for lack of a better word, "interactive" client/server behavior.
I managed to put together a pair of Protocol and ClientFactory classes that do connect to a service, and perform an immediate query/response (see: connectionMade -> self.queryStatus). This succeeds as expected and prints the server's response from the Factory class.
My problem now is that I'll have outside events that must cause data to be sent, while always listening for potential incoming data. But once the reactor.run() loop is going, I'm not sure how the rest of my application is meant to trigger a data send.
I've tried a few different approaches since, but this is the simplest approach that did handle the recv part as described:
class myListenerProtocol(LineReceiver):
delimiter = '\n'
def connectionMade(self):
print("Connected to: %s" % self.transport.getPeer())
self.queryStatus(1)
def dataReceived(self, data):
print("Receiving Data from %s" % self.transport.getPeer())
...
self.commandReceived(self.myData)
def commandReceived(self, myData):
self.factory.commandReceived(myData)
def connectionLost(self, reason):
print("Disconnected.")
def queryStatus(self, CommandValue):
...
strSend = CommandValue # or some such
self.transport.write(strSend)
class mySocketFactory(ClientFactory):
protocol = myListenerProtocol
def __init__(self):
pass
def buildProtocol(self, address):
proto = ClientFactory.buildProtocol(self, address)
return proto
def commandReceived(self, myData):
print myData
reactor.stop() # It won't normally stop after recv
def clientConnectionFailed(self, connector, reason):
print("Connection failed.")
reactor.stop()
def main():
f = mySocketFactory()
reactor.connectTCP("10.10.10.1", 1234, f)
reactor.run()
I imagine this is pretty straight-forward, but countless hours into numerous examples and documentation have left me without a good understanding of how I'm meant to deal with this scenario.
My problem now is that I'll have outside events that must cause data to be sent, while always listening for potential incoming data. But once the reactor.run() loop is going, I'm not sure how the rest of my application is meant to trigger a data send.
"Outside events"? Like what? Data arriving on a connection? Great, having the reactor running means you'll actually be able to handle that data.
Or maybe someone is clicking a button in a GUI? Try one of the GUI integration reactors - again, you can't handle those events until you have a reactor running.
You're probably getting stuck because you think your main function should do reactor.run() and then go on to do other things. This isn't how it works. When you write an event-driven program, you define all of your event sources and then let the event loop call your handlers when events arrive on those sources.
Well, there are many approaches to that, and the best one really depends on the context of your application, so I won't detail you one way of doing this here, but rather link you to a reading I had recently on hacker's news:
http://www.devmusings.com/blog/2013/05/23/python-concurrency/
and good use-case example, though it may not apply to what you're working on (or you may have read it):
http://eflorenzano.com/blog/2008/11/17/writing-markov-chain-irc-bot-twisted-and-python/
BTW, you may also have a look at gevent or tornado that are good at handling that kind of things.
If your other "events" are from a GUI toolkit (like GTK or QT) be really careful of the GIL, and even if you just want command line events you'll need threads and still be careful of that.
Finally, if you want to make more interaction, you may as well write different kind of "peers" for your server, that interacts with the different use cases you're working on (one client that connects to a GUI, another with a CLI, another with a database, another with a SAAS' API etc..).
In other words, if your design is not working, try changing your perspective!
Related
I'm using a SocketServer.ThreadingTCPServer to serve socket connections to clients. This provides an interface where users can connect, type commands and get responses. That part I have working well.
However, in some cases I need a separate thread to broadcast a message to all connected clients. I can't figure out how to do this because there is no way to pass arguments to the class instantiated by ThreadingTCPServer. I don't know how to gather a list of socket connections that have been created.
Consider the example here. How could I access the socket created in the MyTCPHandler class from the __main__ thread?
You should not write to the same TCP socket from multiple threads. The writes may be interleaved if you do ("Hello" and "World" may become "HelWloorld").
That being said, you can create a global list to contain references to all the server objects (who would register themselves in __init__()). The question is, what to do with this list? One idea would be to use a queue or pipe to send the broadcast data to each server object, and have the server objects look in that queue for the "extra" broadcast data to send each time their handle() method is invoked.
Alternatively, you could use the Twisted networking library, which is more flexible and will let you avoid threading altogether - usually a superior alternative.
Here is what I've come up with. It isn't thread safe yet, but that shouldn't be a hard fix:
When the socket is accepted:
if not hasattr(self.server, 'socketlist'):
self.server.socketlist = dict()
thread_id = threading.current_thread().ident
self.server.socketlist[thread_id] = self.request
When the socket closes:
del self.server.socketlist[thread_id]
When I want to write to all sockets:
def broadcast(self, message):
if hasattr(self._server, 'socketlist'):
for socket in self._server.socketlist.values():
socket.sendall(message + "\r\n")
It seems to be working well and isn't as messy as I thought it might end up being.
I am writing a client that needs to establish several independent communication channels, each with its own unique port on the server, with a server through a series of sending and receiving messages. I know how to do this using socket send and recv, by giving each communication channel a socket, and do send and recv on that socket. I need to make this work in Twisted, and found potentially useful interfaces including Factory and ProcessProtocol. However, the Protocol interfaces do not provide a method to send messages. Is ProcessProtocol a good choice for my task, and how I make ProcessProtocol send messages?
In case you don't know about it, I'd like to give a shout out to the excellent Twisted finger tutorial that goes through the library at a good pace but with enough detail that you know what's going on.
To directly answer your question, though, I'd say you're on the right track with Protocol and (Client)Factory. I think the cleanest way to do what you're looking for (assuming you need to connect to different ports because they're outputs for different data) would be to make a factory/protocol pair for each port you want to connect to/handle, and then use an external class to handle the application logic aggregating all of them. Generally you wouldn't want your application logic mixed deeply with your networking logic.
A simple example: (note the use of self.transport.write to send data)
from twisted.internet.protocol import Protocol, ClientFactory
from sys import stdout
from foobar_application import CustomAppObject
class FooProtocol(Protocol):
def connectionMade(self):
# Use self.transport.write to send data to the server
self.transport.write('Hello server this is the Foo protocol.')
self.factory.do_app_logic()
class FooFactory(ClientFactory):
protocol = FooProtocol
def __init__(self, app_object=None):
self.app = app_object
def do_app_logic(self):
self.app.do_something()
class BarProtocol(Protocol):
def dataReceived(self, data):
stdout.write('Received data from server using the Bar protocol.')
self.factory.do_fancy_logic(data)
class BarFactory(ClientFactory):
protocol = BarProtocol
def __init__(self, app_object=None):
self.app = app_object
def do_fancy_logic(self, data):
self.app.do_something_else(data)
logic_obj = CustomAppObject()
reactor.listenTCP(8888, FooFactory(app_object=logic_obj)
reactor.listenTCP(9999, BarFactory(app_object=logic_obj)
reactor.run()
You might also want to look at the 'Writing Clients' docs on the Twisted site.
I need to create twisted SSH server that accepts several commands. But the main feature is that server should manage connection. To be more specific, it needs to close open connection if it lasts for more than 10 minutes (for example). Or it should not accept new connection if there are already 10 open connections.
In fact I still cannot fully understand how all these realms, avatars, protocol and portals, etc. interact with each other. And I feel strong lack of documentation. There are several examples, but without any comments about what exactly is happening on each step.
Anyway, trying and failing I was able to add execution of needed commands to twisted simple ssh server example. But I'm totally loss on how I can reject new connection or close existing or add some time flag for new connection that can be used to close connection when it reaches time limit.
Any help would be appreciated. Please be kind to me, I never worked with Twisted and actually I'm newby in python :)
Thank you.
p.s. I'm sorry for possible mistakes, English is not my native.
So, the main problem is to limit the number of connections. This is really depends on the protocol you would like to use. Let's assume, you use LineOnlyReceiver as a base protocol (other inheritors of Prototol will behave the same way, but, for example, AMP will be a bit different case):
from twisted.internet.protocol import ServerFactory
from twisted.protocols.basic import LineOnlyReceiver
class NoConnectionSlots(Exception):
message = "Sorry, bro. There are no free slots for you. Try again later."
class ExampleProtocol(LineOnlyReceiver):
def connectionMade(self):
try:
self.factory.client_connected(self)
except NoConnectionSlots as e:
self.sendLine("{:}\\n".format(unicode(e)))
self.transport.loseConnection()
def connectionLost(self, reason):
self.factory.client_left(self)
class ExampleServerFactory(ServerFactory):
protocol = ExampleProtocol
max_connections = 10
def __init__(self):
self._connections = []
def client_connected(self, connection):
if len(self._connections) < self.max_connections:
raise NoConnectionSlots()
self._connections.append(connection)
def client_left(self, connection):
try:
self._connections.remove(connection)
except ValueError as e:
pass # place for logging
I want to add a timeout to individual connections within my request handler for a server using the SocketServer module.
Let me start by saying this is the first time I'm attempting to do network programming using Python. I've sub-classed SocketServer.BaseRequestHandler and SocketServer.ThreadingTCPServer & SocketServer.TCPServer and managed to create two classes with some basic threaded TCP functionality.
However I would like my incoming connections to time-out. Trying to override any of the built in SocketServer time-out values and methods does not work, as the documentation says this works only with forking server. I have managed to create a timer thread that fires after X seconds, but due to the nature of the blocking recv call within the Handler thread, this is of no use, as I would be forced to kill it, and this is something I really want to avoid.
So it is my understanding that I need an asyncore implementation, where I get notified and read certain amount of data. In the event that no data is sent over a period of 5 seconds lets say, I want to close that connection (I know how to cleanly do that).
I have found a few examples of using asyncore with sockets, but none using SocketServer. So, how can I implement asyncore & threadingTCPserver ?
Is it possible?
Has anyone done it?
You can also set a timeout on the recv call, like this:
sock.settimeout(1.0)
Since you use SocketServer, you will have to find the underlying socket somewhere in the SocketServer. Please note that SocketServer will create the socket for you, so there is no need to do that yourself.
You will probably have defined a RequestHandler to go with your SocketServer. It should look something like this:
class RequestHandler(SocketServer.BaseRequestHandler):
def setup(self):
# the socket is called request in the request handler
self.request.settimeout(1.0)
def handle(self):
while True:
try:
data = self.request.recv(1024)
if not data:
break # connection is closed
else:
pass # do your thing
except socket.timeout:
pass # handle timeout
I'm new to Python (I have been programming in Java for multiple years now though), and I am working on a simple socket-based networking application (just for fun). The idea is that my code connects to a remote TCP end-point and then listens for any data being pushed from the server to the client, and perform some parsing on this.
The data being pushed from server -> client is UTF-8 encoded text, and each line is delimited by CRLF (\x0D\x0A). You probably guessed: the idea is that the client connects to the server (until cancelled by the user), and then reads and parses the lines as they come in.
I've managed to get this to work, however, I'm not sure that I'm doing this quite the right way. So hence my actual questions (code to follow):
Is this the right way to do it in Python (ie. is it really this simple)?
Any tips/tricks/useful resources (apart from the reference documentation) regarding buffers/asyncore?
Currently, the data is being read and buffered as follows:
def handle_read(self):
self.ibuffer = b""
while True:
self.ibuffer += self.recv(self.buffer_size)
if ByteUtils.ends_with_crlf(self.ibuffer):
self.logger.debug("Got full line including CRLF")
break
else:
self.logger.debug("Buffer not full yet (%s)", self.ibuffer)
self.logger.debug("Filled up the buffer with line")
print(str(self.ibuffer, encoding="UTF-8"))
The ByteUtils.ends_with_crlf function simply checks the last two bytes of the buffer for \x0D\x0A. The first question is the main one (answer is based on this), but any other ideas/tips are appreciated. Thanks.
TCP is a stream, and you are not guaranteed that your buffer will not contain the end of one message and the beginning of the next.
So, checking for \n\r at the end of the buffer will not work as expected in all situations. You have to check each byte in the stream.
And, I would strongly recommend that you use Twisted instead of asyncore.
Something like this (from memory, might not work out of the box):
from twisted.internet import reactor, protocol
from twisted.protocols.basic import LineReceiver
class MyHandler(LineReceiver):
def lineReceived(self, line):
print "Got line:", line
f = protocol.ClientFactory()
f.protocol = MyHandler
reactor.connectTCP("127.0.0.1", 4711, f)
reactor.run()
It's even simpler -- look at asynchat and its set_terminator method (and other helpful tidbits in that module). Twisted is orders of magnitude richer and more powerful, but, for sufficiently simple tasks, asyncore and asynchat (which are designed to interoperate smoothly) are indeed very simple to use, as you've started observing.