Develop Reference

Socket Server with multiply Clients

I just started programming Python.
My goal is to built a digital Picture Frame with three Screens. Therefore I use 3 Raspis, one for each Monitor.
For the communication of these Raspis I need to program a server and a Client.
For a first test I want to built a server which is able to send and receive messages to/from multiple clients.
So I started with a few socket tutorials an created the following program.
Server Class (TcpServer.py)
class TcpServer:
clients = []
serverIsRunning = 0
port = 0
def __init__(self, port):
self.port = port
self.serverIsRunning = 0
self.serverRunning = 0
def startServer (self):
print("start Server...")
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind(("", self.port))
self.server.listen(1)
self.serverRunning = 1
while self.serverRunning:
read, write, oob = select.select([self.server] + self.clients, [], [])
for sock in read:
if sock is self.server:
client, addr = self.server.accept()
self.clients.append(client)
print ("+++ Client ", addr[0], " verbunden")
else:
nachricht = sock.recv(1024)
ip = sock.getpeername()[0]
if nachricht:
print (ip, nachricht)
else:
print ("+++ Verbindung zu ", ip , " beendet")
sock.close()
self.clients.remove(sock)
for c in self.clients:
c.close()
self.clients.remove(c)
self.server.close()
def send(self, message):
message = message.encode()
self.server.send(message)
Client class (TcpClient.py)
import socket
class TcpClient:
def __init__(self, ip, port):
self.serverAdress = (ip, port)
self.connected = 0
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(self.serverAdress)
print ("connectet to ", self.serverAdress)
def send(self, message):
message = message.encode()
self.connection.send(message)
Server:
import threading
import TcpServer
tcpServer = TcpServer.TcpServer(50000)
threadTcpServer = threading.Thread(target = tcpServer.startServer)
threadTcpServer.start()
while True:
tcpServer.send(input("Nachricht eingeben: "))
Client:
import threading
import TcpClient
tcpClient = TcpClient.TcpClient("192.168.178.49", 50000)
while True:
tcpClient.send(input("Nachricht eingeben: "))
I can send messages from the Client to the server, but when I want to send a Message from the server to the client it generates the following error:
BrokenPipeError: [Errno 32] Broken pipe
I assume it is because the server thread blocks the socket while waiting of a incoming message. But I have no idea how to handle this.
How can I program a server who can send and receive messages? Can you recommend a tutorial? I didn't found a tutorial who describes a solution for my problem.
Edit:
Now I tried to solve the problem with the socketserver library, but I still can't solve may problem.
here is my new code for the server:
import socketserver
import threading
import time
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
threadTcpServer = threading.Thread(target = server.serve_forever)
threadTcpServer.start()
print("server started")
time.sleep(10)
print("sending Data")
server.request.sendall("Server is sending...")
it generates the error:
AttributeError: 'TCPServer' object has no attribute 'request'
My goal is to write a server with a thread who receives Data and still be able to send data from a other thread.
Is this even possible with only one socket?

You should use the provided socketserver rather than writing all the handling of sockets and select etc.
There are multiple problems with your code -
1 - The server is trying to write to the listening socket!! The client communication socket is the one that you get from the accept() call and that is the one you have to use for reading and writing.
2 - The client is sending the data and completing immediately, but it should really wait for getting a response. Otherwise, the python / OS will close the client socket as soon as the program completes and it will mostly be before the server gets a chance to respond.

I believe with the Handler code you are able to receive the data sent by the client on the server and are also able to send some data back from the Handler to the client? You must have understood that the server cannot send any data back unless there is a client connected to it?
Now, to send data to the client (or clients) from "another" thread, you will need a way to make the handler objects or the client sockets (available inside the Handler object as self.request) available to the "another" thread.
One way is to override the def __init__(self, request, client_address, server): method and save this object's reference in a global list. Remember to do the below as the last line of the overridden init -
# BaseRequestHandler __init__ must be the last statement as all request processing happens in this method
socketserver.BaseRequestHandler.__init__(self, request, client_address, server)
Once you have all the client handlers in the global list, you can easily write to all the clients from any thread as per your needs. You must read about synchronization (Locks) and understand that using same object / socket from multiple threads can create some logical / data issues with your application.
Another thing that you have to worry about and code for is cleaning up this global list whenever a client closes the connection.

got stuck programming https proxy - what to do next?

I'm currently making a proxy which sits between the browser and the web. Everything works except https. I'm having troubles understanding some passages of it and haven't found many resources on the web. And so I'm stuck.
The code I'm using is:
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
enc_req = conn.recv(9999) #this gets an encrypted request
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
client.send(enc_req)
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
Is the resp1 I'm getting the certificate? And what do I need to do after that? (Or, which is the same, what does usually happens next with https?)
P.S. I know the question is somewhat generic, but please don't judge me too harshly. I've tried researching on the web but all I keep finding is the encryption method used for ssl. I really don't know how to proceed.

I haven't tested this code (and it's mainly pseudo code), but this should give you an idea of what you need to do.
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
# Here, parse the CONNECT string and get the host and port (not sure if you were doing that already.
# Then, try to connect *before* you tell the client the connection was established (in case it fails)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
# Then loop until the connections are closed.
while True:
# Read from the client, send the data to the server.
enc_req = conn.recv(9999) #this gets an encrypted request
client.send(enc_req)
# Read from the server, send the data to the client.
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
# The first time it's certainly the Client Hello message, not encrypted, but in a binary format indeed.
# Just send everything you've just read to the server.
conn.send(resp1)
This is just a quick overview of the idea of the loop you need to write. In reality, you may be able to process both in parallel. You'd also want to be a bit more careful when closing the connection (allowing it to happen in any order while still relaying the last data sent by either party).

As mentioned in the comments, a proxy handling encrypted end-to-end traffic can only pass it on.
Here is a fully working proxy written using circuits that has been fully tested with passing and proxying SSH traffic so it should work equally as well as a pass-through TCP proxy even if SSL is involved:
#!/usr/bin/env python
from uuid import uuid4 as uuid
from circuits import Component
from circuits.net.events import close, connect, write
from circuits.net.sockets import TCPClient, TCPServer
class Client(Component):
channel = "client"
def init(self, sock, host, port, channel=channel):
self.sock = sock
self.host = host
self.port = port
TCPClient(channel=self.channel).register(self)
def ready(self, *args):
self.fire(connect(self.host, self.port))
def disconnect(self, *args):
self.fire(close(self.sock), self.parent.channel)
def read(self, data):
self.fire(write(self.sock, data), self.parent.channel)
class Proxy(Component):
channel = "server"
def init(self, bind, host, port):
self.bind = bind
self.host = host
self.port = port
self.clients = dict()
TCPServer(self.bind).register(self)
def connect(self, sock, host, port):
channel = uuid()
client = Client(
sock, self.host, self.port, channel=channel
).register(self)
self.clients[sock] = client
def disconnect(self, sock):
client = self.clients.get(sock)
if client is not None:
client.unregister()
del self.clients[sock]
def read(self, sock, data):
client = self.clients[sock]
self.fire(write(data), client.channel)
app = Proxy(("0.0.0.0", 3333), "127.0.0.1", 22)
from circuits import Debugger
Debugger().register(app)
app.run()

Twisted Python: UDP Broadcast (simple echo server)

I'm trying to adapt the Python Twisted - UDP examples to use UDP broadcast. I can send a message from the client and receive it on the server, however, it isn't sending a message back.
Client:
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
from socket import SOL_SOCKET, SO_BROADCAST
class EchoClientDatagramProtocol(DatagramProtocol):
strings = [
"Hello, world!",
"What a fine day it is.",
"Bye-bye!"
]
def startProtocol(self):
self.transport.socket.setsockopt(SOL_SOCKET, SO_BROADCAST, True)
self.transport.connect("255.255.255.255", 8000)
self.sendDatagram()
def sendDatagram(self):
if len(self.strings):
datagram = self.strings.pop(0)
self.transport.write(datagram)
else:
reactor.stop()
def datagramReceived(self, datagram, host):
print 'Datagram received: ', repr(datagram)
self.sendDatagram()
def main():
protocol = EchoClientDatagramProtocol()
#0 means any port
t = reactor.listenUDP(0, protocol)
reactor.run()
if __name__ == '__main__':
main()
Server:
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
class EchoUDP(DatagramProtocol):
def datagramReceived(self, datagram, address):
print "Received from address: " + str(address)
print str(datagram)
self.transport.write(datagram, address)
print "Finished sending reply."
print "Starting server."
reactor.listenUDP(8000, EchoUDP())
reactor.run()
Console Output:
Server:
Starting server.
Received from address ('192.168.1.137', 53737)
Hello, world!
Finished sending reply.
Client:
no output.

transport.connect creates a connected UDP socket
A connected UDP socket is slightly different from a standard one - it can only send and receive datagrams to/from a single address, but this does not in any way imply a connection. Datagrams may still arrive in any order, and the port on the other side may have no one listening. The benefit of the connected UDP socket is that it it may provide notification of undelivered packages. This depends on many factors, almost all of which are out of the control of the application, but it still presents certain benefits which occasionally make it useful.
I suspect that the response from the server is not being caught by the client as it is listening for responses from the broadcast address, not the specific address of the server.
Instead, just use the self.transport.write(data, (host, port)) form of write without initiating the connection first - this will allow the client to receive packets from any address.
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
from socket import SOL_SOCKET, SO_BROADCAST
class EchoClientDatagramProtocol(DatagramProtocol):
strings = [
"Hello, world!",
"What a fine day it is.",
"Bye-bye!"
]
def startProtocol(self):
self.transport.socket.setsockopt(SOL_SOCKET, SO_BROADCAST, True)
#self.transport.connect("255.255.255.255", 8000) <- not needed
self.sendDatagram()
def sendDatagram(self):
if len(self.strings):
datagram = self.strings.pop(0)
self.transport.write(datagram, ('255.255.255.255', 8000)) # <- write to broadcast address here
else:
reactor.stop()
def datagramReceived(self, datagram, host):
print 'Datagram received: ', repr(datagram)
self.sendDatagram()
def main():
protocol = EchoClientDatagramProtocol()
#0 means any port
t = reactor.listenUDP(0, protocol)
reactor.run()
if __name__ == '__main__':
main()

Python Game Server

I'm completely lost trying to create a UDP server/client for my game in python. I'm new to the language and only have limited experience with networking. Right now, the server runs, but doesn't seem to be getting any messages from the client.
Server:
class GameServer:
class GameServerUDPHandler(socketserver.BaseRequestHandler):
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
print("{} wrote:".format(self.client_address[0]))
print(data)
socket.sendto(data.upper(), self.client_address)
def __init__(self, port):
self.server = socketserver.UDPServer(("localhost", port), self.GameServerUDPHandler)
def start_server(self):
self.server.serve_forever(
Client:
import socket
import sys
class GameClient:
def __init__(self, port, host):
self.port = port
self.host = host
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def register(self):
self.socket.sendto(bytes("register\n", "utf-8"), (self.host, self.port))
self.numberID = int(self.socket.recv(1024))
print("Received: {}".format(self.numberID))
-Main/Start of program
import gameserver
import gameclient
if __name__ == "__main__":
server = gameserver.GameServer(1300)
server.start_server()
client = gameclient.GameClient(1300, "localhost")
client.register()
NOTE: I'm most likely to multiple things wrong and may be violating several best practices in the language. I really have no clue.

The problem is that some of these calls are blocking. In particular, the serve_forever() method will run forever, so you need to put that on a separate thread if you want the rest of your program to continue:
import threading
if __name__ == "__main__":
server = GameServer(1300)
server_thread = threading.Thread(target=lambda: server.start_server())
server_thread.start()
time.sleep(1) # Give it time to start up; not production quality code of course
client = GameClient(1300, "localhost")
client.register()
socket.recv() is also a blocking call but that might be okay in this case.

Seems like this library isn't asynchronous so your first call to serve_forever will not return and your client never gets started. You can create a new thread to launch the server on or split your client and server into seperate processes.

python asynchat: how to store information on individual connections, and how to know when a client disconnects

For fun, I'm writing a minimal IRC server with asynchat. I'm trying to clear up a few fundamentals (my specific questions follow the code). I've decided not to use anything in Twisted just so I can implement a little more myself. First, the code I have:
import asyncore,asynchat
import socket
class Connection(asynchat.async_chat):
def __init__(self, server, sock, addr):
asynchat.async_chat.__init__(self, sock)
self.set_terminator('\n')
self.data = ""
print "client connecting:",addr
# do some IRC protocol initialization stuff here
def collect_incoming_data(self, data):
self.data = self.data + data
def found_terminator(self):
print self.data
self.data = ''
class Server(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind((host, port))
self.listen(5)
def handle_accept(self):
conn, addr = self.accept()
Connection(self, conn, addr)
def handle_close(self):
self.close()
s = Server('127.0.0.1',5006)
asyncore.loop()
So, in my mind, this code structure is similar to a Twisted client factory: the Server class is initialized once and basically instantiates Connection every time a client connects. First question: is the best way to keep track of all connected clients by storing all of the Connections in a list within Server?
Also, I don't understand how I am to know when a specific client closes their connection to my socket? Connection implements asynchat (and by extension asyncore) but adding the handle_close() callback to the Connection class doesn't fire when a client disconnects. It seems to be only for when the bound socket on the server is destroyed. I don't see any methods for this purpose. This socket always stays open, whether or not clients connect, right?

to handle client side closed connections check the handle_error method, does your client issue a clean close connection?
handle_error() :Called when an exception is raised and not otherwise handled. The default version prints a condensed traceback.
hope it helps.