Develop Reference

Handling multiple connections in python with sockets

I have a code which works perfectly for one connection. I have seen two options for multi-client handling but I don't really understand it.
Here is the server socket code:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listening_sock:
listening_sock.bind(('', port))
listening_sock.listen()
client_soc, client_address = listening_sock.accept()
client_soc.sendall('200#Welcome to my server!'.encode())
print(f'Address {client_soc.getsockname()[0]} connected with port {client_soc.getsockname()[1]}')
while True:
# get message
msg = client_soc.recv(1024).decode()
# receive log print:
print(f'"{msg}" sent from {client_soc.getsockname()[0]}')
if 'Quit' in msg:
client_soc.sendall('200#Thanks for using my server!'.encode())
client_soc.close()
elif '0' < msg.split('#')[0] <= '9': # one of the valid actions
answer = call_action(msg.split('#')[0], db, msg.split('#')[1]) # the answer for given parameter
client_soc.sendall("200#".encode() + answer.encode())
If I have only one connection it works good and last thing I need to add is option for multiple-client handling. What is the shortest and easiest way to do it?

The code only calls accept once. Instead, call accept in a while loop and create a thread for each client connection so they are handled in parallel. Use the following pattern as an example:
import socket
import threading
# Thread to handle each "client_soc" connection
def handler(client_soc):
...
client_soc.close()
with socket.socket() as listening_sock:
listening_sock.bind(('', 8000))
listening_sock.listen()
while True:
client_soc, client_address = listening_sock.accept()
# Send each "client_soc" connection as a parameter to a thread.
threading.Thread(target=handler,args=(client_soc,), daemon=True).start()
There is also a built-in socket server that simplifies this process. Here's a tested example echo server that echoes back newline-terminated data:
from socketserver import ThreadingTCPServer,StreamRequestHandler
class echohandler(StreamRequestHandler):
def handle(self):
print(f'Connected: {self.client_address[0]}:{self.client_address[1]}')
while True:
# get message
msg = self.rfile.readline()
if not msg:
print(f'Disconnected: {self.client_address[0]}:{self.client_address[1]}')
break # exits handler, framework closes socket
print(f'Received: {msg}')
self.wfile.write(msg)
self.wfile.flush()
server = ThreadingTCPServer(('',8000),echohandler)
server.serve_forever()

Your code blocks itself.
For instance: client_soc, client_address = listening_sock.accept()
Accepts client, then while True: runs forever, so you can work with 1 connection only, because socket.accept() is called once. You should learn some of these to solve your problem: asyncio, threading, multiprocessing. These libraries will help your code to accept and work with clients concurrently. Sockets can use every, but often they are paired with asyncio: https://asyncio.readthedocs.io/

Socket Server with multiply Clients

I just started programming Python.
My goal is to built a digital Picture Frame with three Screens. Therefore I use 3 Raspis, one for each Monitor.
For the communication of these Raspis I need to program a server and a Client.
For a first test I want to built a server which is able to send and receive messages to/from multiple clients.
So I started with a few socket tutorials an created the following program.
Server Class (TcpServer.py)
class TcpServer:
clients = []
serverIsRunning = 0
port = 0
def __init__(self, port):
self.port = port
self.serverIsRunning = 0
self.serverRunning = 0
def startServer (self):
print("start Server...")
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind(("", self.port))
self.server.listen(1)
self.serverRunning = 1
while self.serverRunning:
read, write, oob = select.select([self.server] + self.clients, [], [])
for sock in read:
if sock is self.server:
client, addr = self.server.accept()
self.clients.append(client)
print ("+++ Client ", addr[0], " verbunden")
else:
nachricht = sock.recv(1024)
ip = sock.getpeername()[0]
if nachricht:
print (ip, nachricht)
else:
print ("+++ Verbindung zu ", ip , " beendet")
sock.close()
self.clients.remove(sock)
for c in self.clients:
c.close()
self.clients.remove(c)
self.server.close()
def send(self, message):
message = message.encode()
self.server.send(message)
Client class (TcpClient.py)
import socket
class TcpClient:
def __init__(self, ip, port):
self.serverAdress = (ip, port)
self.connected = 0
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(self.serverAdress)
print ("connectet to ", self.serverAdress)
def send(self, message):
message = message.encode()
self.connection.send(message)
Server:
import threading
import TcpServer
tcpServer = TcpServer.TcpServer(50000)
threadTcpServer = threading.Thread(target = tcpServer.startServer)
threadTcpServer.start()
while True:
tcpServer.send(input("Nachricht eingeben: "))
Client:
import threading
import TcpClient
tcpClient = TcpClient.TcpClient("192.168.178.49", 50000)
while True:
tcpClient.send(input("Nachricht eingeben: "))
I can send messages from the Client to the server, but when I want to send a Message from the server to the client it generates the following error:
BrokenPipeError: [Errno 32] Broken pipe
I assume it is because the server thread blocks the socket while waiting of a incoming message. But I have no idea how to handle this.
How can I program a server who can send and receive messages? Can you recommend a tutorial? I didn't found a tutorial who describes a solution for my problem.
Edit:
Now I tried to solve the problem with the socketserver library, but I still can't solve may problem.
here is my new code for the server:
import socketserver
import threading
import time
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
threadTcpServer = threading.Thread(target = server.serve_forever)
threadTcpServer.start()
print("server started")
time.sleep(10)
print("sending Data")
server.request.sendall("Server is sending...")
it generates the error:
AttributeError: 'TCPServer' object has no attribute 'request'
My goal is to write a server with a thread who receives Data and still be able to send data from a other thread.
Is this even possible with only one socket?

You should use the provided socketserver rather than writing all the handling of sockets and select etc.
There are multiple problems with your code -
1 - The server is trying to write to the listening socket!! The client communication socket is the one that you get from the accept() call and that is the one you have to use for reading and writing.
2 - The client is sending the data and completing immediately, but it should really wait for getting a response. Otherwise, the python / OS will close the client socket as soon as the program completes and it will mostly be before the server gets a chance to respond.

I believe with the Handler code you are able to receive the data sent by the client on the server and are also able to send some data back from the Handler to the client? You must have understood that the server cannot send any data back unless there is a client connected to it?
Now, to send data to the client (or clients) from "another" thread, you will need a way to make the handler objects or the client sockets (available inside the Handler object as self.request) available to the "another" thread.
One way is to override the def __init__(self, request, client_address, server): method and save this object's reference in a global list. Remember to do the below as the last line of the overridden init -
# BaseRequestHandler __init__ must be the last statement as all request processing happens in this method
socketserver.BaseRequestHandler.__init__(self, request, client_address, server)
Once you have all the client handlers in the global list, you can easily write to all the clients from any thread as per your needs. You must read about synchronization (Locks) and understand that using same object / socket from multiple threads can create some logical / data issues with your application.
Another thing that you have to worry about and code for is cleaning up this global list whenever a client closes the connection.

How to TCP server serves forever?

Hello everyone!
I'm new to python networking programming.
My development environments are as below.
Windows 7
Python 3.4
I am studying with "Python Network Programming Cookbook". In this book, there's an example of ThreadingMixIn socket server application.
This book's code is written in Python 2.7. So I've modified for python 3.4.
The code is...
# coding: utf-8
import socket
import threading
import socketserver
SERVER_HOST = 'localhost'
SERVER_PORT = 0 # tells the kernel to pick up a port dynamically
BUF_SIZE = 1024
def client(ip, port, message):
""" A client to test threading mixin server"""
# Connect to the server
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip, port))
try:
message = bytes(message, encoding="utf-8")
sock.sendall(message)
response = sock.recv(BUF_SIZE)
print("Client received: {0}".format(response))
finally:
sock.close()
class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):
""" An example of threaded TCP request handler """
def handle(self):
data = self.request.recv(1024)
current_thread = threading.current_thread()
response = "{0}: {0}".format(current_thread.name, data)
response = bytes(response, encoding="utf-8")
self.request.sendall(response)
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
"""Nothing to add here, inherited everything necessary from parents"""
pass
if __name__ == "__main__":
# Run server
server = ThreadedTCPServer((SERVER_HOST, SERVER_PORT),
ThreadedTCPRequestHandler)
ip, port = server.server_address # retrieve ip address
# Start a thread with the server -- one thread per request
server_thread = threading.Thread(target=server.serve_forever)
# Exit the server thread when the main thread exits
server_thread.daemon = True
server_thread.start()
print("Server loop running on thread: {0}".format(server_thread))
# Run clients
client(ip, port, "Hello from client 1")
client(ip, port, "Hello from client 2")
client(ip, port, "Hello from client 3")
This code works perfect. Every client's request processed by new thread. And when the client's request is over, program ends.
I want to make server serves forever. So when the additional client's request has come, server send its response to that client.
What should I do?
Thank you for reading my question.
P.S: Oh, one more. I always write say hello in top of my post of stack overflow. In preview it shows normally. But when the post has saved, first line always gone. Please anyone help me XD

Your program exits because your server thread is a daemon:
# Exit the server thread when the main thread exits
server_thread.daemon = True
You can either remove that line or add server_thread.join() at the bottom of the code to prevent the main thread from exiting early.

You will have to run on an infinite loop and on each loop wait for some data to come from client. This way the connection will be kept alive.
Same infinite loop for the server to accept more clients.
However, you will have to somehow detect when a client closes the connection with the server because in most times the server won't be notified.

python multithreading server

I am new to networking programming and python.
I am trying to figure out how to run different jobs at the server side.
For example, I want one function to create connections for incoming clients but in the same time I can still do some administration work from the terminal.
My code is as below but it doesn't work:
Edited: it doesn't work means it will get stuck in the init_conn() function
Like:
starting up on localhost port 8887
Thread: 0 Connected with 127.0.0.1:48080
# waiting
I am looking into SocketServer framework but don't know how that works.
from thread import *
import socket
def init_conn():
thread_count =0
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Bind the socket to the port
server_address = ('localhost', 8887)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
# Listen for incoming connections
sock.listen(10)
#now keep talking with the client
while 1:
#wait to accept a connection - blocking call
conn, addr = sock.accept()
print 'Thread: '+ str(thread_count) + ' Connected with ' + addr[0] + ':' + str(addr[1])
#start new thread takes 1st argument as a function name to be run, second is the tuple of arguments to the function.
start_new_thread(clientthread ,(conn,))
thread_count +=1
sock.close()
def clientthread(conn):
# receive data from client and send back
def console():
print 'this is console'
option = raw_input('-v view clients')
if option == 'v':
print 'you press v'
def main():
start_new_thread( init_conn(),() )
start_new_thread( console(),() )
if __name__ == "__main__":
main()

Your problem is probably that you start the program, sometimes it prints "this is console" and then it ends.
The first bug is that you call the methods instead of passing the handle to start_new_thread. It must be:
start_new_thread( init_conn, () )
i.e. no () after the function name.
The program doesn't do much because start_new_thread() apparent starts a thread and then waits for it to stop. The documentation is pretty unclear. It's better to use the new threading module; See http://pymotw.com/2/threading/
def main():
t = threading.Thread( target=init_conn )
t.daemon = True
t.start()
console()
so the code will run until console() ends.
I suggest to split the server and the command line tool. Create a client which accepts commands from the command line and sends them to the server. That way, you can start the console from anywhere and you can keep the code for the two separate.

Seeing that you're new to python, have you tried taking a look at the threading module that comes with the standard library?
import threading
... #rest of your code
while conditions==True:
i = threading.Thread(target=init_conn)
c = threading.Thread(target=console)
i.start()
c.start()
Can't say I've done too much with networking programming with python, so I don't really have much to say in that manner, but at least this should get you started with adding multithreading to your project.

Using SocketServer you may implement a client/server system. The documentation gives small examples which may be useful for you. Here is an extended example from there:
server.py :
import SocketServer
import os
import logging
FORMAT = '[%(asctime)-15s] %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)
class MyServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
# By setting this we allow the server to re-bind to the address by
# setting SO_REUSEADDR, meaning you don't have to wait for
# timeouts when you kill the server and the sockets don't get
# closed down correctly.
allow_reuse_address = True
request_queue_size = 10
def __init__(self, port):
self.host = os.uname()[1]
self.port = port
SocketServer.TCPServer.__init__(self, (self.host,self.port), MyTCPHandler)
logging.info( "Server has been started on {h}:{p}".format(h=self.host,p=self.port) )
class MyTCPHandler(SocketServer.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
# max length is here 1024 chars
self.data = self.request.recv(1024).strip()
logging.info( "received: {d}".format(d=self.data) )
# here you may execute different functions according to the
# request string
# here: just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
PORT = 8887
if __name__ == "__main__":
# Create the server, binding to localhost on port 8887
#server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
server = MyServer( PORT )
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
client.py
import socket
import sys
import logging
FORMAT = '[%(asctime)-15s] %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)
HOST, PORT = "workstation04", 8887
logging.info( "connect to server {h}:{p}".format(h=HOST,p=PORT ) )
# read command line
data = " ".join(sys.argv[1:])
# Create a socket (SOCK_STREAM means a TCP socket)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
# Connect to server and send data
sock.connect((HOST, PORT))
sock.sendall(data + "\n")
# Receive data from the server and shut down
received = sock.recv(1024)
finally:
sock.close()
logging.info( "Sent: {}".format(data) )
logging.info( "Received: {}".format(received) )
The output looks something like:
server side:
> python server.py
[2015-05-28 11:17:49,263] Server has been started on disasterarea:8887
[2015-05-28 11:17:50,972] received: my message
client side:
[2015-05-28 11:17:50,971] connect to server disasterarea:8887
[2015-05-28 11:17:50,972] Sent: my message
[2015-05-28 11:17:50,972] Received: MY MESSAGE
You can run several clients (from different consoles) in parallel. You may implement a request processor on the server side which processes the incoming requests and executes certain functions.
Alternatively, you may use the python module ParallelPython which executes python code locally on a multicore system or on a cluster and clusters. Check the http examples.
I had to force pip to install this module:
pip install --allow-external pp --allow-unverified pp pp

Python Game Server

I'm completely lost trying to create a UDP server/client for my game in python. I'm new to the language and only have limited experience with networking. Right now, the server runs, but doesn't seem to be getting any messages from the client.
Server:
class GameServer:
class GameServerUDPHandler(socketserver.BaseRequestHandler):
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
print("{} wrote:".format(self.client_address[0]))
print(data)
socket.sendto(data.upper(), self.client_address)
def __init__(self, port):
self.server = socketserver.UDPServer(("localhost", port), self.GameServerUDPHandler)
def start_server(self):
self.server.serve_forever(
Client:
import socket
import sys
class GameClient:
def __init__(self, port, host):
self.port = port
self.host = host
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def register(self):
self.socket.sendto(bytes("register\n", "utf-8"), (self.host, self.port))
self.numberID = int(self.socket.recv(1024))
print("Received: {}".format(self.numberID))
-Main/Start of program
import gameserver
import gameclient
if __name__ == "__main__":
server = gameserver.GameServer(1300)
server.start_server()
client = gameclient.GameClient(1300, "localhost")
client.register()
NOTE: I'm most likely to multiple things wrong and may be violating several best practices in the language. I really have no clue.

The problem is that some of these calls are blocking. In particular, the serve_forever() method will run forever, so you need to put that on a separate thread if you want the rest of your program to continue:
import threading
if __name__ == "__main__":
server = GameServer(1300)
server_thread = threading.Thread(target=lambda: server.start_server())
server_thread.start()
time.sleep(1) # Give it time to start up; not production quality code of course
client = GameClient(1300, "localhost")
client.register()
socket.recv() is also a blocking call but that might be okay in this case.

Seems like this library isn't asynchronous so your first call to serve_forever will not return and your client never gets started. You can create a new thread to launch the server on or split your client and server into seperate processes.