I am trying to find the easiest way to read from multiple (around 100) udp datagram sockets in python. I have looked at tornado, but tornado touts http/tcp rather than udp support.
Right now I have threads dedicated to each udp socket; however, this doesn't seem very efficient.
The SocketServer module has a built-in UDP server with options for threading and forking.
Another option is the use the select module which will allow you to focus only on the sockets where data is already available for reading.
I must confess I never used it, but maybe Twisted will suit your needs.
It supports lots of protocols, even serial connections.
I'd like to add some comments on the initial question even though it already has an accepted answer.
If you have multiple connections which need independent processing (or at least processing with not much synchronization), its okay to use one thread per connection and do blocking reads. Modern schedulers won't kill you for that one. It is a fairly efficient way of handling the connections. If you are concerned about memory footprint, you could reduce the stack size of the threads accordingly (does not apply to python).
The threads/processes will stay in a non-busy waiting state for most of the time (while waiting for new data) and not consume any CPU time.
If you do not want or can not use threads, the select call is definetly the way to go. This is also low-level and efficient waiting and as a bonus, gives you the list of sockets which triggered.
asyncoro supports asynchronous TCP and UDP sockets (among many other features). Unlike with other frameworks, programming with asyncoro is very similar to that of threads. A simple UDP client/server program to illustrate:
import socket, asyncoro
def server_proc(n, sock, coro=None):
for i in xrange(n):
msg, addr = yield sock.recvfrom(1024)
print('Received "%s" from %s:%s' % (msg, addr[0], addr[1]))
sock.close()
def client_proc(host, port, coro=None):
sock = asyncoro.AsynCoroSocket(socket.socket(socket.AF_INET, socket.SOCK_DGRAM))
msg = 'client socket: %s' % (sock.fileno())
yield sock.sendto(msg, (host, port))
sock.close()
if __name__ == '__main__':
sock = asyncoro.AsynCoroSocket(socket.socket(socket.AF_INET, socket.SOCK_DGRAM))
sock.bind(('127.0.0.1', 0))
host, port = sock.getsockname()
n = 100
server_coro = asyncoro.Coro(server_proc, n, sock)
for i in range(n):
asyncoro.Coro(client_proc, host, port)
asyncoro uses efficient polling mechanisms where possible. Only with Windows and UDP sockets it uses inefficient 'select' (but uses efficient Windows I/O Completion Ports for TCP if pywin32 is installed).
I think if you do insist on using tornado's ioloop and want to do UDP socket processing, you should use a UDP version of the tornado IOStream. I have done this with success in my own projects. It is a little bit of a misnomer to call it UDPStream (since it is not quite a stream), but the basic use should be very easy for you to integrate into your application.
See the code at: http://kyle.graehl.org/coding/2012/12/07/tornado-udpstream.html
Related
I am implementing a socket in Python to pass data back and forth between two scripts running on the same machine as part of a single Tkinter application.
This data, in many cases, will be highly sensitive (i.e. personal credit card numbers).
Does passing the data between scripts in this way open me up to any security concerns?
Server side:
import socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 8089))
serversocket.listen(5) # become a server socket, maximum 5 connections
while True:
connection, address = serversocket.accept()
buf = connection.recv(64)
if len(buf) > 0:
print buf
break
Client side:
import socket
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect(('localhost', 8089))
clientsocket.send('hello')
Code source.
Additional considerations:
This will only ever function as part of a single Tkinter application, on a single machine. Localhost will always be specified.
I am unable to use multiprocessing or threading; please no suggestions for using one of those or an alternative, other than varieties of socket. For more info as to why, see this SO question, answers, and comments. It has to do with this needing to function on Windows 7 and *nix, as well as my desired set-up.
Yes, passing the data between scripts in this way may raise a security concerns. If the attacker has an access to the same machine - he can easily sniff the traffic using the tool like tcpdump for example.
To avoid this you should encrypted your traffic - I have posted a comment below your question with an example solution.
I am new to the module socket and I am learning how to implement it. I currently have this basic server written.
import socket
s = socket.socket()
host = "Some IP"
port = 12345
s.bind((host, port))
print "Setting up server on", host + ":" + str(port)
s.listen(5)
while True:
c, addr = s.accept()
print 'Got connection from', addr
c.send('Thank you for connecting.')
print c.recv(1024)
c.close()
However, I realize this server can only deal with one client at a time. To further my training, I want to handle multiple clients at one time. So I made a research and came across the library SocketServer. I also heard the method of threading the regular socket module, so every new connection creates a new thread. The information I found was not enough. Please help me to understand the difference between these two methods I found and which one to use where.
Thanks in advance.
The socket library is very low-level, you really have to implement most things yourself.
SocketServer is more high-level. It uses the socket library internally, and provides an easy interface, that allows you deal with multiple clients at a time, and you don't have to worry about the whole low-level stuff involved in using sockets.
Look at the examples here and compare them to your own code. You'll probably see the differences.
https://docs.python.org/2/library/socketserver.html#examples
so, imagine I have only one socket that I need to manage for IO completion while its alive. With select.select() I would do
import socket
import select
a = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
a.setblocking(True)
connected = True
while connected:
read, write, error = select.select([a], [a], [a])
if read:
#do stuff with data
if write:
#write to socket
is there not a better solution to checking if only one socket is readable/writable? I will only be dealing with a single socket object, and no more. I feel like this method was built in mind with managing multiple sockets as there could be a more effecient way to handle only one.
select() is still the way to go, even for a single socket. The only alternative (ignoring for the sake of discussion alternatives like poll(), which are similar to select()) would be blocking I/O, but if you're blocked in recv() you have no way to be woken up when the socket becomes ready-for-write, and vice versa if you're blocked in send().
So you might as well use select(). There's nothing inefficient about using it for a single socket.
I have a few test clients that are encountering the same issue each time. The clients can connect, and they can send their first message, but after that the server stops responding to that client. I suspect that the problem is related to s.accept(), but I'm not sure exactly what is wrong or how to work around it.
def startServer():
host = ''
port = 13572
backlog = 5
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(backlog)
print "Close the command prompt to stop Gamelink"
while 1:
try:
client, address = s.accept()
data = client.recv(size)
if data:
processData(data)
client.send("OK")
else:
print "Disconnecting from client at client's request"
client.close()
except socket.error, (value, message):
if s:
print "Disconnecting from client, socket issue"
s.close()
print "Error opening socket: " + message
break
except:
print "Gamelink encountered a problem"
break
print "End of loop"
client.close()
s.close()
The server is intended to be accessed across a local network, and it needs to be light weight and very quick to respond, so if another implementation (such as thread based) would be better for meeting those requirements please let me know. The intended application is to be used as a remote gaming keyboard, thus the need for low resource use and high speed.
Writing a server using socket directly will be hard. As Keith says, you need to multiplex the connections somehow, like with select or poll or threads or fork. You might think you need only one connection, but what will you do when something hiccups and the connection is lost? Will your server be able to respond to reconnection attempts from the client if it hasn't yet realized the connection is lost?
If your networking needs are basic, you might be able to let something else handle all the listening and accepting and forking stuff for you. You don't specify a platform, but examples of such programs are launchd on Mac OS and xinetd on Linux. The details differ between these tools, but basically you configure them, in some configuration file, to listen for a connection on some port. When they get it, they take care of setting up the connection, then they exec() your program with stdin and stdout aimed at the socket, so you can simply use all the basic IO you probably already know like print and sys.stdin.read().
The trouble with solutions like xinitd and launchd is that for each new connection, they must fork() and exec() a new instance of your program. These are relatively heavy operations so a large number of connections or a high rate of new connections might hit the limits of your server. But worse, since each connection is in a separate process, sharing data between them is hard. Also, most solutions you might find to communicate between processes involve a blocking API, and now you are back to the problem of multiplexing with select or threads or similar.
If that doesn't meet your needs, I think you are better off learning to use a higher-level networking framework which will handle all the problems you will inevitably encounter if you go down the path of socket. One such framework I'd suggest is Twisted. Beyond handling the mundane details of handling connections, and the more complex task of multiplexing IO between them, you will also have a huge library of tools that will make implementing your protocol much easier.
I am trying to use multiprocessing and sockets to allow multiple connections to the same socket. However, I am having a real hard time because I don't have much experience in this field.
The code I have isn't working
def server(port, listen=10):
connected = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', port))
s.listen(listen)
while True:
conn, address = s.accept()
p = multiprocessing.Process(target=server, args=(port, listen))
p.start()
p.join()
command = raw_input("Command: ")
conn.send(command)
Thanks for the help
This is because you are trying to create multiple servers in loop. Single server is suffucient for your task, no need to open many listening sockets. Every local port may be bound by at most one listening socket -- that's why you see "address in use" error.
Try out the Python standard TCPServer class, this could be much more convenient than to bother with low-level sockets.
For threading server see this example.
At the OS socket level, this scheme needs only one listening socket, which will spawn new socket each time when accepting a new connection (that's the standard way of socketry). Then you'll work with new socket at the separate thread (keep in mind access to common data shared among threads).