I want to add a timeout to individual connections within my request handler for a server using the SocketServer module.
Let me start by saying this is the first time I'm attempting to do network programming using Python. I've sub-classed SocketServer.BaseRequestHandler and SocketServer.ThreadingTCPServer & SocketServer.TCPServer and managed to create two classes with some basic threaded TCP functionality.
However I would like my incoming connections to time-out. Trying to override any of the built in SocketServer time-out values and methods does not work, as the documentation says this works only with forking server. I have managed to create a timer thread that fires after X seconds, but due to the nature of the blocking recv call within the Handler thread, this is of no use, as I would be forced to kill it, and this is something I really want to avoid.
So it is my understanding that I need an asyncore implementation, where I get notified and read certain amount of data. In the event that no data is sent over a period of 5 seconds lets say, I want to close that connection (I know how to cleanly do that).
I have found a few examples of using asyncore with sockets, but none using SocketServer. So, how can I implement asyncore & threadingTCPserver ?
Is it possible?
Has anyone done it?
You can also set a timeout on the recv call, like this:
sock.settimeout(1.0)
Since you use SocketServer, you will have to find the underlying socket somewhere in the SocketServer. Please note that SocketServer will create the socket for you, so there is no need to do that yourself.
You will probably have defined a RequestHandler to go with your SocketServer. It should look something like this:
class RequestHandler(SocketServer.BaseRequestHandler):
def setup(self):
# the socket is called request in the request handler
self.request.settimeout(1.0)
def handle(self):
while True:
try:
data = self.request.recv(1024)
if not data:
break # connection is closed
else:
pass # do your thing
except socket.timeout:
pass # handle timeout
Related
I'm using a SocketServer.ThreadingTCPServer to serve socket connections to clients. This provides an interface where users can connect, type commands and get responses. That part I have working well.
However, in some cases I need a separate thread to broadcast a message to all connected clients. I can't figure out how to do this because there is no way to pass arguments to the class instantiated by ThreadingTCPServer. I don't know how to gather a list of socket connections that have been created.
Consider the example here. How could I access the socket created in the MyTCPHandler class from the __main__ thread?
You should not write to the same TCP socket from multiple threads. The writes may be interleaved if you do ("Hello" and "World" may become "HelWloorld").
That being said, you can create a global list to contain references to all the server objects (who would register themselves in __init__()). The question is, what to do with this list? One idea would be to use a queue or pipe to send the broadcast data to each server object, and have the server objects look in that queue for the "extra" broadcast data to send each time their handle() method is invoked.
Alternatively, you could use the Twisted networking library, which is more flexible and will let you avoid threading altogether - usually a superior alternative.
Here is what I've come up with. It isn't thread safe yet, but that shouldn't be a hard fix:
When the socket is accepted:
if not hasattr(self.server, 'socketlist'):
self.server.socketlist = dict()
thread_id = threading.current_thread().ident
self.server.socketlist[thread_id] = self.request
When the socket closes:
del self.server.socketlist[thread_id]
When I want to write to all sockets:
def broadcast(self, message):
if hasattr(self._server, 'socketlist'):
for socket in self._server.socketlist.values():
socket.sendall(message + "\r\n")
It seems to be working well and isn't as messy as I thought it might end up being.
Let's say I want to implement an echo server and client using ZeroMQ (pyzmq) and asyncio (for its event loop, coroutines, etc.).
Now I want to add more reliability by adding a heartbeat. As I don't want to interact too much with my wonderful echo protocol, this heartbeat is done by both client and server on a dedicated pair of sockets.
From what I understand, the way to go™ is to create a new zmq socket in the server class, register it to the existing Poller and let the server class handle everything, from timeout calculation to sending beats. That works, of course.
But this is more complicated than it should be (that's a personal view). From the server point of view, 'heartbeats' are implementation details. What heartbeats are there for is to answer a simple question: "is the client still there?". More technically, I would like to setup and Heartbeat object that takes a timeout and an address. That Heartbeat object would do all the socket setup, beat-related socket polling, send actual beats and receive them.
From the server point of view, I would just use client.is_alive() when required. But that would require two socket pollers to work in parallel. I can achieve that with an executor, but that does not seem right. How would you do that?
I've been trying to wrap my mind around how to get Twisted to perform, for lack of a better word, "interactive" client/server behavior.
I managed to put together a pair of Protocol and ClientFactory classes that do connect to a service, and perform an immediate query/response (see: connectionMade -> self.queryStatus). This succeeds as expected and prints the server's response from the Factory class.
My problem now is that I'll have outside events that must cause data to be sent, while always listening for potential incoming data. But once the reactor.run() loop is going, I'm not sure how the rest of my application is meant to trigger a data send.
I've tried a few different approaches since, but this is the simplest approach that did handle the recv part as described:
class myListenerProtocol(LineReceiver):
delimiter = '\n'
def connectionMade(self):
print("Connected to: %s" % self.transport.getPeer())
self.queryStatus(1)
def dataReceived(self, data):
print("Receiving Data from %s" % self.transport.getPeer())
...
self.commandReceived(self.myData)
def commandReceived(self, myData):
self.factory.commandReceived(myData)
def connectionLost(self, reason):
print("Disconnected.")
def queryStatus(self, CommandValue):
...
strSend = CommandValue # or some such
self.transport.write(strSend)
class mySocketFactory(ClientFactory):
protocol = myListenerProtocol
def __init__(self):
pass
def buildProtocol(self, address):
proto = ClientFactory.buildProtocol(self, address)
return proto
def commandReceived(self, myData):
print myData
reactor.stop() # It won't normally stop after recv
def clientConnectionFailed(self, connector, reason):
print("Connection failed.")
reactor.stop()
def main():
f = mySocketFactory()
reactor.connectTCP("10.10.10.1", 1234, f)
reactor.run()
I imagine this is pretty straight-forward, but countless hours into numerous examples and documentation have left me without a good understanding of how I'm meant to deal with this scenario.
My problem now is that I'll have outside events that must cause data to be sent, while always listening for potential incoming data. But once the reactor.run() loop is going, I'm not sure how the rest of my application is meant to trigger a data send.
"Outside events"? Like what? Data arriving on a connection? Great, having the reactor running means you'll actually be able to handle that data.
Or maybe someone is clicking a button in a GUI? Try one of the GUI integration reactors - again, you can't handle those events until you have a reactor running.
You're probably getting stuck because you think your main function should do reactor.run() and then go on to do other things. This isn't how it works. When you write an event-driven program, you define all of your event sources and then let the event loop call your handlers when events arrive on those sources.
Well, there are many approaches to that, and the best one really depends on the context of your application, so I won't detail you one way of doing this here, but rather link you to a reading I had recently on hacker's news:
http://www.devmusings.com/blog/2013/05/23/python-concurrency/
and good use-case example, though it may not apply to what you're working on (or you may have read it):
http://eflorenzano.com/blog/2008/11/17/writing-markov-chain-irc-bot-twisted-and-python/
BTW, you may also have a look at gevent or tornado that are good at handling that kind of things.
If your other "events" are from a GUI toolkit (like GTK or QT) be really careful of the GIL, and even if you just want command line events you'll need threads and still be careful of that.
Finally, if you want to make more interaction, you may as well write different kind of "peers" for your server, that interacts with the different use cases you're working on (one client that connects to a GUI, another with a CLI, another with a database, another with a SAAS' API etc..).
In other words, if your design is not working, try changing your perspective!
I am developing python service for xbmc and I am hopelessly stuck. XBMC has TCP API that communicates by JSON-RPC. XBMC has server TCP socket that is mainly design to recieve commands and respond, but if something happens in system it sends "Notification" to TCP. The problem is that I need to create TCP client that behaves like server therefore it is able to recieve this "Notification". Wherever I run socket.recv(4096) it waits for data and stucks my code, because I need to loop my code. Structure of code is basically like this:
import xbmc, xbmcgui, xbmcaddon
class XPlayer(xbmc.Player) :
def __init__ (self):
xbmc.Player.__init__(self)
def onPlayBackStarted(self):
if xbmc.Player().isPlayingVideo():
doPlayBackStartedStuff()
player=XPlayer()
doStartupStuff()
while (not xbmc.abortRequested):
if xbmc.Player().isPlayingVideo():
doPlayingVideoStuff()
else:
doPlayingEverythingElseStuff()
xbmc.sleep(500)
# This loop is the most essential part of code
if (xbmc.abortRequested):
closeEverything()
xbmc.log('Aborting...')
I tried everything threading, multiprocessing, blocking, non-blocking and nothing helped.
Thanks,
You likely want select(), poll() or epoll():
http://docs.python.org/library/select.html
This Python pipe-progress-meter application uses select, as an example:
http://stromberg.dnsalias.org/~strombrg/reblock.html
If you know what sort of delimiters are separating the various portions of the protocol, you may also find this useful, without a need for select or similar:
http://stromberg.dnsalias.org/~strombrg/bufsock.html
It deals pretty gracefully with "read to the next null byte", "read a maximum of 100 bytes", etc.
I want a two way communication in Python :
I want to bind to a socket where one client can connect to, and then server and client can "chat" with eachother.
I already have the basic listener :
import socket
HOST='' #localhost
PORT=50008
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM ) #create an INET, STREAMing socket
s.bind((HOST,PORT)) #bind to that port
s.listen(1) #listen for user input and accept 1 connection at a time.
conn, addr = s.accept()
print "The connection has been set up"
bool=1
while bool==1:
data=conn.recv(1024)
print data
if "#!END!#" in data:
print "closing the connection"
s.close()
bool=0
What I want to do now is implement something so this script also accepts user input and after the enter key is hit, send it back to the client.
But I can't figure out how I can do this ? Because if I would do it like this :
while bool==1:
data=conn.recv(1024)
print data
u_input = raw_input("input now")
if u_input != "":
conn.send(u_input)
u_input= ""
Problem is that it probably hangs at the user input prompt, so it does not allow my client to send data.
How do I solve this ?
I want to keep it in one window, can this be solved with threads ?
(I've never used threads in python)
Python's sockets have a makefile tool to make this sort of interaction much easier. After creating a socket s, then run f = s.makefile(). That will return an object with a file-like interface (so you can use readline, write, writelines and other convenient method calls). The Python standard library itself makes use of this approach (see the source for ftplib and poplib for example).
To get text from the client and display it on the server console, write a loop with print f.readline().
To get text from the server console and send it to the client, write a loop with f.write(raw_input('+ ') + '\n').
To be send and receive at the same time, do those two steps separate threads:
Thread(target=read_client_and_print_to_console).start()
Thread(target=read_server_console_and_send).start()
If you prefer async over threads, here are two examples to get you started:
Basic Async HTTP Client
Basic Async Echo Server
The basic problem is that you have two sources of input you're waiting for: the socket and the user. The three main approaches I can think of are to use asynchronous I/O, to use synchronous (blocking) I/O with multiple threads, or to use synchronous I/O with timeouts. The last approach is conceptually the simplest: wait for data on the socket for up to some timeout period, then switch to waiting for the user to enter data to send, then back to the socket, etc.
I know at a lower level, you could implement this relatively easily by treating both the socket and stdin as I/O handles and use select to wait on both of them simultaneously, but I can't recall if that functionality is mapped into Python, or if so, how. That's potentially a very good way of handling this if you can make it work. EDIT: I looked it up, and Python does have a select module, but it sounds like it only functions like this under Unix operating systems--in Windows, it can only accept sockets, not stdin or files.
have you checked twisted? twisted python event driven networking engine and library or
oidranot a python library especially for that based on torando web server