I am trying to use multiprocessing and sockets to allow multiple connections to the same socket. However, I am having a real hard time because I don't have much experience in this field.
The code I have isn't working
def server(port, listen=10):
connected = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', port))
s.listen(listen)
while True:
conn, address = s.accept()
p = multiprocessing.Process(target=server, args=(port, listen))
p.start()
p.join()
command = raw_input("Command: ")
conn.send(command)
Thanks for the help
This is because you are trying to create multiple servers in loop. Single server is suffucient for your task, no need to open many listening sockets. Every local port may be bound by at most one listening socket -- that's why you see "address in use" error.
Try out the Python standard TCPServer class, this could be much more convenient than to bother with low-level sockets.
For threading server see this example.
At the OS socket level, this scheme needs only one listening socket, which will spawn new socket each time when accepting a new connection (that's the standard way of socketry). Then you'll work with new socket at the separate thread (keep in mind access to common data shared among threads).
Related
I am implementing a socket in Python to pass data back and forth between two scripts running on the same machine as part of a single Tkinter application.
This data, in many cases, will be highly sensitive (i.e. personal credit card numbers).
Does passing the data between scripts in this way open me up to any security concerns?
Server side:
import socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 8089))
serversocket.listen(5) # become a server socket, maximum 5 connections
while True:
connection, address = serversocket.accept()
buf = connection.recv(64)
if len(buf) > 0:
print buf
break
Client side:
import socket
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect(('localhost', 8089))
clientsocket.send('hello')
Code source.
Additional considerations:
This will only ever function as part of a single Tkinter application, on a single machine. Localhost will always be specified.
I am unable to use multiprocessing or threading; please no suggestions for using one of those or an alternative, other than varieties of socket. For more info as to why, see this SO question, answers, and comments. It has to do with this needing to function on Windows 7 and *nix, as well as my desired set-up.
Yes, passing the data between scripts in this way may raise a security concerns. If the attacker has an access to the same machine - he can easily sniff the traffic using the tool like tcpdump for example.
To avoid this you should encrypted your traffic - I have posted a comment below your question with an example solution.
I am new to the module socket and I am learning how to implement it. I currently have this basic server written.
import socket
s = socket.socket()
host = "Some IP"
port = 12345
s.bind((host, port))
print "Setting up server on", host + ":" + str(port)
s.listen(5)
while True:
c, addr = s.accept()
print 'Got connection from', addr
c.send('Thank you for connecting.')
print c.recv(1024)
c.close()
However, I realize this server can only deal with one client at a time. To further my training, I want to handle multiple clients at one time. So I made a research and came across the library SocketServer. I also heard the method of threading the regular socket module, so every new connection creates a new thread. The information I found was not enough. Please help me to understand the difference between these two methods I found and which one to use where.
Thanks in advance.
The socket library is very low-level, you really have to implement most things yourself.
SocketServer is more high-level. It uses the socket library internally, and provides an easy interface, that allows you deal with multiple clients at a time, and you don't have to worry about the whole low-level stuff involved in using sockets.
Look at the examples here and compare them to your own code. You'll probably see the differences.
https://docs.python.org/2/library/socketserver.html#examples
I've been working on this server in python for a bit, and it seemed to work, but I don't think that the threading works properly. Things seem to happen in a sequential order (the first client to connect gets ALL of the information before the next client begins), is this a problem with this threading interface for servers? Should I change it and how so? Here is some example code:
port = 8123
print port
backlog = 5
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(backlog)
while 1:
client, address = s.accept()
print "client recieved at " + address[0]
thread.start_new_thread(serveClient, (client, address, board))
If you want to re-use the standard Python module, check out SocketServer. It includes a Threading mixin, to make your server multi threaded.
An example test (from the above documentation) is this code:
if __name__ == "__main__":
# Port 0 means to select an arbitrary unused port
HOST, PORT = "localhost", 0
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
ip, port = server.server_address
# Start a thread with the server -- that thread will then start one
# more thread for each request
server_thread = threading.Thread(target=server.serve_forever)
# Exit the server thread when the main thread terminates
server_thread.daemon = True
server_thread.start()
print "Server loop running in thread:", server_thread.name
client(ip, port, "Hello World 1")
client(ip, port, "Hello World 2")
client(ip, port, "Hello World 3")
server.shutdown()
Yes. CPython does not support parallel threads, due to the global interpreter lock.
CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.
From https://docs.python.org/2/library/threading.html
If you want truly parallel processing, you need to either use the Process class or use a different Python implementation (Jython supports parallel threads, I believe IronPython does as well).
How to create multi-server sockets on one client in Python ?
I am thinking about create a List of server socket and make the connection with
non-blocking socket, but i don't find a good tutorial for that, thats why i came here,
to ask for better solution or a guide for non-blocking socket.
Thank you !
Thank for the help, but i mean to something different, i have list of Servers Ip like that:
SERVER_IP = ['127.0.0.1', '127.0.0.2', '127.0.0.3', '127.0.0.4', '127.0.0.5', '127.0.0.6, '127.0.0.7']
I have one option to create a list of sockets by ip, and try to connect to every Ip Server, but i ask
here if i have a different way to connect to all this Servers Ip without a list of sockets, something more convenient.
Thank you.
If you want to have multiple sockets connected to multiple servers, you should check out the select module (http://docs.python.org/2/library/select.html).
Basically, it works like this:
import socket, select
socks = {}
# Connect to different servers here #
sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socks[sock1.fileno()] = sock1
poll = select.poll()
for sock in socks:
poll.register(sock)
while 1:
fd, event = poll.poll() # Optional timeout parameter in seconds
sock = socks[fd]
sock.recv(1024) # Do stuff
A note, the poll.poll() method returns the underlying file number (what your operating system uses to represent files) which is useless to you. I just stores the sockets in a dictionary by that number so you could get the actual socket object from the filenumber that is given by poll. I recommend reading the documentation for select.poll, the link above.
You can use select.
http://pymotw.com/2/select/ plus the select documentation.
Or some third party module such as twisted.
I am trying to find the easiest way to read from multiple (around 100) udp datagram sockets in python. I have looked at tornado, but tornado touts http/tcp rather than udp support.
Right now I have threads dedicated to each udp socket; however, this doesn't seem very efficient.
The SocketServer module has a built-in UDP server with options for threading and forking.
Another option is the use the select module which will allow you to focus only on the sockets where data is already available for reading.
I must confess I never used it, but maybe Twisted will suit your needs.
It supports lots of protocols, even serial connections.
I'd like to add some comments on the initial question even though it already has an accepted answer.
If you have multiple connections which need independent processing (or at least processing with not much synchronization), its okay to use one thread per connection and do blocking reads. Modern schedulers won't kill you for that one. It is a fairly efficient way of handling the connections. If you are concerned about memory footprint, you could reduce the stack size of the threads accordingly (does not apply to python).
The threads/processes will stay in a non-busy waiting state for most of the time (while waiting for new data) and not consume any CPU time.
If you do not want or can not use threads, the select call is definetly the way to go. This is also low-level and efficient waiting and as a bonus, gives you the list of sockets which triggered.
asyncoro supports asynchronous TCP and UDP sockets (among many other features). Unlike with other frameworks, programming with asyncoro is very similar to that of threads. A simple UDP client/server program to illustrate:
import socket, asyncoro
def server_proc(n, sock, coro=None):
for i in xrange(n):
msg, addr = yield sock.recvfrom(1024)
print('Received "%s" from %s:%s' % (msg, addr[0], addr[1]))
sock.close()
def client_proc(host, port, coro=None):
sock = asyncoro.AsynCoroSocket(socket.socket(socket.AF_INET, socket.SOCK_DGRAM))
msg = 'client socket: %s' % (sock.fileno())
yield sock.sendto(msg, (host, port))
sock.close()
if __name__ == '__main__':
sock = asyncoro.AsynCoroSocket(socket.socket(socket.AF_INET, socket.SOCK_DGRAM))
sock.bind(('127.0.0.1', 0))
host, port = sock.getsockname()
n = 100
server_coro = asyncoro.Coro(server_proc, n, sock)
for i in range(n):
asyncoro.Coro(client_proc, host, port)
asyncoro uses efficient polling mechanisms where possible. Only with Windows and UDP sockets it uses inefficient 'select' (but uses efficient Windows I/O Completion Ports for TCP if pywin32 is installed).
I think if you do insist on using tornado's ioloop and want to do UDP socket processing, you should use a UDP version of the tornado IOStream. I have done this with success in my own projects. It is a little bit of a misnomer to call it UDPStream (since it is not quite a stream), but the basic use should be very easy for you to integrate into your application.
See the code at: http://kyle.graehl.org/coding/2012/12/07/tornado-udpstream.html