Develop Reference

How does the select() function in the select module of Python exactly work?

I am working on writing a network-oriented application in Python. I had earlier worked on using blocking sockets, but after a better understanding of the requirement and concepts, I am wanting to write the application using non-blocking sockets and thus an event-driven server.
I understand that the functions in the select module in Python are to be used to conveniently see which socket interests us and so forth. Towards that I was basically trying to flip through a couple of examples of an event-driven server and I had come across this one:
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
The parts that I didn't seem to understand are the following:
In the code snippet inputready,outputready,exceptready = select.select(input,[],[]), I believe the select() function returns three possibly empty lists of waitable objects for input, output and exceptional conditions. So it makes sense that the first argument to the select() function is the list containing the server socket and the stdin. However, where I face confusion is in the else block of the code.
Since we are for-looping over the list of inputready sockets, it is clear that the select() function will choose a client socket that is ready to be read. However, after we read data using recv() and find that the socket has actually sent data, we would want to echo it back to the client. My question is how can we write to this socket without adding it to the list passed as second argument to the select() function call? Meaning, how can we call send() on the new socket directly without 'registering' it with select() as a writable socket?
Also, why do we loop only over the sockets ready to be read (inputready in this case)? Isn't it necessary to loop over even the outputready list to see which sockets are ready to be written?
Obviously, I am missing something here.
It would also be really helpful if somebody could explain in a little more detailed fashion the working of select() function or point to good documentation.
Thank you.

Probably that snippet of code is just a simple example and so it is not exhaustive. You are free to write and read in every socket, also if select does not tell you that they are ready. But, of course, if you do this you cannot be sure that your send() won't block.
So, yes, it would be best practice to rely on select also for writing operations.
There are also many other function which have a similar purpose and in many cases they are better then select (e.g. epoll), but they are not available on all platforms.
Information about the select, epoll & other functions may be found in Linux man pages.
However in python there are many nice libraries used to handle many connections, some of these are: Twisted and gevent

Python Socket Library, Errno 10054 When Client Quits

So I'm messing around with a very simple echo server/client kind of deal. I'm eventually going to be making a game server, but for now I'm just trying to setup the basic parts.
Here's the server: http://pastebin.com/qtfrMGur
Here's the client: http://pastebin.com/3VK3VxPX
So, my problem is that when the client disconnects, I get socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host. I was under the impression that select.select() returned false(or an empty list, whatever) when the readable socket list had no pending information. This seems to be the cause of the error, but then why is select.select() returning true(on line 26) if the client is no longer there? The actual error occurs when I try to retrieve some info in line 27. I'm really confused by this, assuming the client-side of the socket is closed, shouldn't it just return [], and then go to the else on line 31 and destroy that client?
I don't really know what else to post, this seems like a very elementary problem, usually I use a more abstract library like twisted that makes it easy, but for this project I must use traditional BSD sockets. If you guys have any other info you need just ask.
Thanks.
EDIT: So I put a print statement in there to print the value of select.select(...)[0], like so:
for cl in clients:
ready = select.select([cl], [], [], .1)[0]
print ready
if ready:
data = cl.recv(size)
if data:
print data
#cl.send(data)
else:
print "Client Quit\n"
clients.remove(cl)
cl.shutdown(socket.SHUT_RDWR)
cl.close()
print len(clients)
As you can see I do a proper shutdown now as well. The problem is, that right as the client quits, select.select(...) returns the client socket as a readable source, so cl.recv gets called on it, which then throws an error because cl is closed on the other end. At least this is my suspicion.
So, could somebody please explain what I may be doing wrong here? Why does the client send a last dying message, is it some kind of functionality that I should be taking advantage of?
Thanks
EDIT2: So guys I fired up my debian machine and the code works flawlessly, it's only in windows that it's doing this. Is this some sort of bug? I was under the impression that the socket library abstracted all the stuff beneath the hood to make the interface identical between OS.

You should call the shutdown method of the socket, before closing it on the client side, like this:
s.shutdown(socket.SHUT_RDWR)
s.close()
That way it won't allow further sending / receiving, check out the Python docs about it, it's nicely explained there.
UPDATE: fixed client code
import socket
host = 'localhost'
port = 50000
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,port))
msg = ''
while msg !='quit':
msg = raw_input()
s.send(msg)
#data = s.recv(size)
s.shutdown(socket.SHUT_RDWR)
s.close()

Trouble with a Python server

I have a few test clients that are encountering the same issue each time. The clients can connect, and they can send their first message, but after that the server stops responding to that client. I suspect that the problem is related to s.accept(), but I'm not sure exactly what is wrong or how to work around it.
def startServer():
host = ''
port = 13572
backlog = 5
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(backlog)
print "Close the command prompt to stop Gamelink"
while 1:
try:
client, address = s.accept()
data = client.recv(size)
if data:
processData(data)
client.send("OK")
else:
print "Disconnecting from client at client's request"
client.close()
except socket.error, (value, message):
if s:
print "Disconnecting from client, socket issue"
s.close()
print "Error opening socket: " + message
break
except:
print "Gamelink encountered a problem"
break
print "End of loop"
client.close()
s.close()
The server is intended to be accessed across a local network, and it needs to be light weight and very quick to respond, so if another implementation (such as thread based) would be better for meeting those requirements please let me know. The intended application is to be used as a remote gaming keyboard, thus the need for low resource use and high speed.

Writing a server using socket directly will be hard. As Keith says, you need to multiplex the connections somehow, like with select or poll or threads or fork. You might think you need only one connection, but what will you do when something hiccups and the connection is lost? Will your server be able to respond to reconnection attempts from the client if it hasn't yet realized the connection is lost?
If your networking needs are basic, you might be able to let something else handle all the listening and accepting and forking stuff for you. You don't specify a platform, but examples of such programs are launchd on Mac OS and xinetd on Linux. The details differ between these tools, but basically you configure them, in some configuration file, to listen for a connection on some port. When they get it, they take care of setting up the connection, then they exec() your program with stdin and stdout aimed at the socket, so you can simply use all the basic IO you probably already know like print and sys.stdin.read().
The trouble with solutions like xinitd and launchd is that for each new connection, they must fork() and exec() a new instance of your program. These are relatively heavy operations so a large number of connections or a high rate of new connections might hit the limits of your server. But worse, since each connection is in a separate process, sharing data between them is hard. Also, most solutions you might find to communicate between processes involve a blocking API, and now you are back to the problem of multiplexing with select or threads or similar.
If that doesn't meet your needs, I think you are better off learning to use a higher-level networking framework which will handle all the problems you will inevitably encounter if you go down the path of socket. One such framework I'd suggest is Twisted. Beyond handling the mundane details of handling connections, and the more complex task of multiplexing IO between them, you will also have a huge library of tools that will make implementing your protocol much easier.

Python "push server" tcp client

I am developing python service for xbmc and I am hopelessly stuck. XBMC has TCP API that communicates by JSON-RPC. XBMC has server TCP socket that is mainly design to recieve commands and respond, but if something happens in system it sends "Notification" to TCP. The problem is that I need to create TCP client that behaves like server therefore it is able to recieve this "Notification". Wherever I run socket.recv(4096) it waits for data and stucks my code, because I need to loop my code. Structure of code is basically like this:
import xbmc, xbmcgui, xbmcaddon
class XPlayer(xbmc.Player) :
def __init__ (self):
xbmc.Player.__init__(self)
def onPlayBackStarted(self):
if xbmc.Player().isPlayingVideo():
doPlayBackStartedStuff()
player=XPlayer()
doStartupStuff()
while (not xbmc.abortRequested):
if xbmc.Player().isPlayingVideo():
doPlayingVideoStuff()
else:
doPlayingEverythingElseStuff()
xbmc.sleep(500)
# This loop is the most essential part of code
if (xbmc.abortRequested):
closeEverything()
xbmc.log('Aborting...')
I tried everything threading, multiprocessing, blocking, non-blocking and nothing helped.
Thanks,

You likely want select(), poll() or epoll():
http://docs.python.org/library/select.html
This Python pipe-progress-meter application uses select, as an example:
http://stromberg.dnsalias.org/~strombrg/reblock.html
If you know what sort of delimiters are separating the various portions of the protocol, you may also find this useful, without a need for select or similar:
http://stromberg.dnsalias.org/~strombrg/bufsock.html
It deals pretty gracefully with "read to the next null byte", "read a maximum of 100 bytes", etc.

Socket : 2 way communication in python

I want a two way communication in Python :
I want to bind to a socket where one client can connect to, and then server and client can "chat" with eachother.
I already have the basic listener :
import socket
HOST='' #localhost
PORT=50008
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM ) #create an INET, STREAMing socket
s.bind((HOST,PORT)) #bind to that port
s.listen(1) #listen for user input and accept 1 connection at a time.
conn, addr = s.accept()
print "The connection has been set up"
bool=1
while bool==1:
data=conn.recv(1024)
print data
if "#!END!#" in data:
print "closing the connection"
s.close()
bool=0
What I want to do now is implement something so this script also accepts user input and after the enter key is hit, send it back to the client.
But I can't figure out how I can do this ? Because if I would do it like this :
while bool==1:
data=conn.recv(1024)
print data
u_input = raw_input("input now")
if u_input != "":
conn.send(u_input)
u_input= ""
Problem is that it probably hangs at the user input prompt, so it does not allow my client to send data.
How do I solve this ?
I want to keep it in one window, can this be solved with threads ?
(I've never used threads in python)

Python's sockets have a makefile tool to make this sort of interaction much easier. After creating a socket s, then run f = s.makefile(). That will return an object with a file-like interface (so you can use readline, write, writelines and other convenient method calls). The Python standard library itself makes use of this approach (see the source for ftplib and poplib for example).
To get text from the client and display it on the server console, write a loop with print f.readline().
To get text from the server console and send it to the client, write a loop with f.write(raw_input('+ ') + '\n').
To be send and receive at the same time, do those two steps separate threads:
Thread(target=read_client_and_print_to_console).start()
Thread(target=read_server_console_and_send).start()
If you prefer async over threads, here are two examples to get you started:
Basic Async HTTP Client
Basic Async Echo Server

The basic problem is that you have two sources of input you're waiting for: the socket and the user. The three main approaches I can think of are to use asynchronous I/O, to use synchronous (blocking) I/O with multiple threads, or to use synchronous I/O with timeouts. The last approach is conceptually the simplest: wait for data on the socket for up to some timeout period, then switch to waiting for the user to enter data to send, then back to the socket, etc.
I know at a lower level, you could implement this relatively easily by treating both the socket and stdin as I/O handles and use select to wait on both of them simultaneously, but I can't recall if that functionality is mapped into Python, or if so, how. That's potentially a very good way of handling this if you can make it work. EDIT: I looked it up, and Python does have a select module, but it sounds like it only functions like this under Unix operating systems--in Windows, it can only accept sockets, not stdin or files.

have you checked twisted? twisted python event driven networking engine and library or
oidranot a python library especially for that based on torando web server