Develop Reference

Python UDP socketserver returning empty message

I have a UDP socketserver program that I use to demonstrate how UDP works (code for the server and client are below). I run this on a server, then have the client.py program send a message and receive a reply. I am unfortunately running into an issue that seems to only occur on campus Wifi. On campus wifi, the client does not receive a response.
Troubleshooting with Wireshark shows the issue. For some reason the UDP server is responding with two UDP messages - one empty, and one containing the response message. These messages are recorded in Wireshark as coming in approximately 0.000002 seconds apart. On a wired network, the one with the response consistently comes first, and on Wifi, the empty message consistently comes first. Since the client is waiting for a single messages response, when the empty message returns, the client prints and exits, and the actual response is never seen.
I know I could write the client to listen for both messages and print out whichever one has the data, but I would rather try to figure out what's going on. Why is the socketserver responding with two messages in the first place, and how can I get it to only send one? OR at least to send the data first.
server.py:
import socketserver
class MyUDPRequestHandler(socketserver.DatagramRequestHandler):
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
# just send back the same data, but lower-cased
socket.sendto(data.lower(), self.client_address)
if __name__ == "__main__":
with socketserver.UDPServer(("0.0.0.0", 9091), MyUDPRequestHandler) as server:
server.serve_forever()
client.py:
import socket
HOST, PORT = "localhost", 9091
message = "NOW I AM SHOUTING" # The UDP server will lowercase the message
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(bytes(message + "\n", "utf-8"), (HOST, PORT))
received = str(sock.recv(1024), "utf-8")
print("Sent: {}".format(message))
print("Received: {}".format(received))

I've repeated the problem and it's socketserver. Notice the definition of DatagramRequestHandler below:
class DatagramRequestHandler(BaseRequestHandler):
"""Define self.rfile and self.wfile for datagram sockets."""
def setup(self):
from io import BytesIO
self.packet, self.socket = self.request
self.rfile = BytesIO(self.packet)
self.wfile = BytesIO()
def finish(self):
self.socket.sendto(self.wfile.getvalue(), self.client_address)
The packet is put into a buffer as rfile and should be read from there, then written back to the wfile buffer. finish sends the packet. The handler shouldn't call sendto itself:
import socketserver
class MyUDPRequestHandler(socketserver.DatagramRequestHandler):
def handle(self):
data = self.rfile.read()
self.wfile.write(data.strip().lower())
if __name__ == "__main__":
with socketserver.UDPServer(("0.0.0.0", 9091), MyUDPRequestHandler) as server:
server.serve_forever()
But just using a simple socket as the server works fine too:
import socket
s = socket.socket(type=socket.SOCK_DGRAM)
s.bind(('', 9091))
while True:
data, client = s.recvfrom(2048)
s.sendto(data.strip().lower(), client)
Note that UDP packets are not guaranteed to be delivered or delivered in the same order, so the original code's issue with the two packets changing order isn't surprising.

Socket programming + HTTP1.1 in Python

The task is building two files client.py and server.py. I am able to connect the client to the server. The problem I encounter is when I trying to send a get request like client.send("bGET /suc.txt HTTP/1.1\r\nHost:127.0.0.1\r\n\r\n"), I do not how to return the file suc.txt to the client from the server side. The scene is a client request file from a server and what the server returns is the respond header and the requested file.
What I wrote so far :
Client:
import socket
target_host = "127.0.0.1"
target_port = 5050
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((target_host,target_port))
client.send("bGET /suc.txt HTTP/1.1\r\nHost:127.0.0.1\r\n\r\n")
response = client.recv(1024)
print(response.decode())
Server:
import socket
import threading
import urllib.request
HEADER = 64
PORT = 5050
HOST = socket.gethostbyname(socket.gethostname())
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((HOST,PORT))
def handleClient(conn, addr):
print (f"[NEW CONNECTION {addr} connected. ")
connected = True
while connected:
conn.send()
def start():
server.listen()
while True:
conn, addr = server.accept()
thread = threading.Thread(target=handleClient, args=(conn,addr))
thread.start()
print(f"[ACTIVE CONNECTIONS] {threading.activeCount()} ")
print ("server is starting...")
start()

client.send("bGET /suc.txt HTTP/1.1\r\nHost:127.0.0.1\r\n\r\n")
The "b..." should be b"...", i.e. you want to specify a sequence of bytes and not a string with a b as first character.
I do not how to return the file suc.txt to the client from the server side
You basically ask very broadly how to read an HTTP request, extract information from it, create a proper response and send it. All what you code so far does is create a listener socket, so you are far away from your ultimate goal.
There are two major ways to tackle this: the easy one is to use a library like http.server to implement the complexity of HTTP for you. The documentation contains actual examples on how to do this and there are many more examples on the internet for this.
The harder option is to study the actual HTTP standard and implement everything yourself based on this standard. Expecting that somebody explains the complex standard here and describes how to implement it would be a too broad question.

Socket Server with multiply Clients

I just started programming Python.
My goal is to built a digital Picture Frame with three Screens. Therefore I use 3 Raspis, one for each Monitor.
For the communication of these Raspis I need to program a server and a Client.
For a first test I want to built a server which is able to send and receive messages to/from multiple clients.
So I started with a few socket tutorials an created the following program.
Server Class (TcpServer.py)
class TcpServer:
clients = []
serverIsRunning = 0
port = 0
def __init__(self, port):
self.port = port
self.serverIsRunning = 0
self.serverRunning = 0
def startServer (self):
print("start Server...")
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind(("", self.port))
self.server.listen(1)
self.serverRunning = 1
while self.serverRunning:
read, write, oob = select.select([self.server] + self.clients, [], [])
for sock in read:
if sock is self.server:
client, addr = self.server.accept()
self.clients.append(client)
print ("+++ Client ", addr[0], " verbunden")
else:
nachricht = sock.recv(1024)
ip = sock.getpeername()[0]
if nachricht:
print (ip, nachricht)
else:
print ("+++ Verbindung zu ", ip , " beendet")
sock.close()
self.clients.remove(sock)
for c in self.clients:
c.close()
self.clients.remove(c)
self.server.close()
def send(self, message):
message = message.encode()
self.server.send(message)
Client class (TcpClient.py)
import socket
class TcpClient:
def __init__(self, ip, port):
self.serverAdress = (ip, port)
self.connected = 0
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(self.serverAdress)
print ("connectet to ", self.serverAdress)
def send(self, message):
message = message.encode()
self.connection.send(message)
Server:
import threading
import TcpServer
tcpServer = TcpServer.TcpServer(50000)
threadTcpServer = threading.Thread(target = tcpServer.startServer)
threadTcpServer.start()
while True:
tcpServer.send(input("Nachricht eingeben: "))
Client:
import threading
import TcpClient
tcpClient = TcpClient.TcpClient("192.168.178.49", 50000)
while True:
tcpClient.send(input("Nachricht eingeben: "))
I can send messages from the Client to the server, but when I want to send a Message from the server to the client it generates the following error:
BrokenPipeError: [Errno 32] Broken pipe
I assume it is because the server thread blocks the socket while waiting of a incoming message. But I have no idea how to handle this.
How can I program a server who can send and receive messages? Can you recommend a tutorial? I didn't found a tutorial who describes a solution for my problem.
Edit:
Now I tried to solve the problem with the socketserver library, but I still can't solve may problem.
here is my new code for the server:
import socketserver
import threading
import time
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
threadTcpServer = threading.Thread(target = server.serve_forever)
threadTcpServer.start()
print("server started")
time.sleep(10)
print("sending Data")
server.request.sendall("Server is sending...")
it generates the error:
AttributeError: 'TCPServer' object has no attribute 'request'
My goal is to write a server with a thread who receives Data and still be able to send data from a other thread.
Is this even possible with only one socket?

You should use the provided socketserver rather than writing all the handling of sockets and select etc.
There are multiple problems with your code -
1 - The server is trying to write to the listening socket!! The client communication socket is the one that you get from the accept() call and that is the one you have to use for reading and writing.
2 - The client is sending the data and completing immediately, but it should really wait for getting a response. Otherwise, the python / OS will close the client socket as soon as the program completes and it will mostly be before the server gets a chance to respond.

I believe with the Handler code you are able to receive the data sent by the client on the server and are also able to send some data back from the Handler to the client? You must have understood that the server cannot send any data back unless there is a client connected to it?
Now, to send data to the client (or clients) from "another" thread, you will need a way to make the handler objects or the client sockets (available inside the Handler object as self.request) available to the "another" thread.
One way is to override the def __init__(self, request, client_address, server): method and save this object's reference in a global list. Remember to do the below as the last line of the overridden init -
# BaseRequestHandler __init__ must be the last statement as all request processing happens in this method
socketserver.BaseRequestHandler.__init__(self, request, client_address, server)
Once you have all the client handlers in the global list, you can easily write to all the clients from any thread as per your needs. You must read about synchronization (Locks) and understand that using same object / socket from multiple threads can create some logical / data issues with your application.
Another thing that you have to worry about and code for is cleaning up this global list whenever a client closes the connection.

Python TCP sockets headers

I need to create a communication between a client and a server with TCP. But I'd like to send and work with "headers". So from the client I'd like to send a header "COMMAND1" and the server returns me something.
I have the following code:
Server
import socket
import threading
bind_ip = '0.0.0.0'
bind_port = 9998
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(5) # max backlog of connections
print ('Listening on {}:{}'.format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(1024)
print ('Received {}'.format(request))
client_socket.send('Response1!'.encode('utf-8'))
client_socket.close()
while True:
client_sock, address = server.accept()
print ('Accepted connection from {}:{}'.format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
Client
import socket
hostname, sld, tld, port = 'www', 'integralist', 'co.uk', 80
target = '{}.{}.{}'.format(hostname, sld, tld)
# create an ipv4 (AF_INET) socket object using the tcp protocol (SOCK_STREAM)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect the client
# client.connect((target, port))
client.connect(('0.0.0.0', 9998))
# send some data (in this case a HTTP GET request)
client.send('hi'.encode('utf-8'))
# receive the response data (4096 is recommended buffer size)
response = client.recv(4096)
print (response)
Anyone knows the best way to return "Response1!" when the header is "COMMAND1" and "Response2!" when the header is "COMMAND2"?
I can't find examples on how to use headers
EDIT: It doesn't have to be "COMMAND1" or "COMMAND2" it can be a "0" or "1", or anything else.

If you want to add your own header, you just have to:
Make sure your programm finds the start of your message (like, every message beginns "!?&")
Send your own header-data just after the start-symbol of your message.
Maybe mark the end of your message with something or pass a length in your header.
Since TCP will give you a stream of data, it might come to a case, where it just gives you 2 or 3 messages at once. You have to separate these messages by yourself (e.g. by using "?!&" as start of every message).
You can always create your own protocoll as payload of another protocoll. Just as TCP is just payload from the ethernet point of view.

You can do something i have done with my program to accept such headers
Use pickle library to encode a dict headers and send it through socket.
Code will look something like this.
import pickle
def handleSocket(headers:dict):
message = pickle.dumps(headers)
socket.send(message)
For server side, you will be handling it
Gonna initialise the socket recv to 100 kb
def handleReceivedSocket(socket):
message:dict = pickle.loads(socket.recv(102400))
Another way to do this. Is sending a raw json string to the server (just change pickle.dumps,pickle.loads by json.dumps,json.loads
But it will be in raw and less secure.
Last way you can do it is uri encoding. Check w3schools

got stuck programming https proxy - what to do next?

I'm currently making a proxy which sits between the browser and the web. Everything works except https. I'm having troubles understanding some passages of it and haven't found many resources on the web. And so I'm stuck.
The code I'm using is:
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
enc_req = conn.recv(9999) #this gets an encrypted request
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
client.send(enc_req)
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
Is the resp1 I'm getting the certificate? And what do I need to do after that? (Or, which is the same, what does usually happens next with https?)
P.S. I know the question is somewhat generic, but please don't judge me too harshly. I've tried researching on the web but all I keep finding is the encryption method used for ssl. I really don't know how to proceed.

I haven't tested this code (and it's mainly pseudo code), but this should give you an idea of what you need to do.
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
# Here, parse the CONNECT string and get the host and port (not sure if you were doing that already.
# Then, try to connect *before* you tell the client the connection was established (in case it fails)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
# Then loop until the connections are closed.
while True:
# Read from the client, send the data to the server.
enc_req = conn.recv(9999) #this gets an encrypted request
client.send(enc_req)
# Read from the server, send the data to the client.
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
# The first time it's certainly the Client Hello message, not encrypted, but in a binary format indeed.
# Just send everything you've just read to the server.
conn.send(resp1)
This is just a quick overview of the idea of the loop you need to write. In reality, you may be able to process both in parallel. You'd also want to be a bit more careful when closing the connection (allowing it to happen in any order while still relaying the last data sent by either party).

As mentioned in the comments, a proxy handling encrypted end-to-end traffic can only pass it on.
Here is a fully working proxy written using circuits that has been fully tested with passing and proxying SSH traffic so it should work equally as well as a pass-through TCP proxy even if SSL is involved:
#!/usr/bin/env python
from uuid import uuid4 as uuid
from circuits import Component
from circuits.net.events import close, connect, write
from circuits.net.sockets import TCPClient, TCPServer
class Client(Component):
channel = "client"
def init(self, sock, host, port, channel=channel):
self.sock = sock
self.host = host
self.port = port
TCPClient(channel=self.channel).register(self)
def ready(self, *args):
self.fire(connect(self.host, self.port))
def disconnect(self, *args):
self.fire(close(self.sock), self.parent.channel)
def read(self, data):
self.fire(write(self.sock, data), self.parent.channel)
class Proxy(Component):
channel = "server"
def init(self, bind, host, port):
self.bind = bind
self.host = host
self.port = port
self.clients = dict()
TCPServer(self.bind).register(self)
def connect(self, sock, host, port):
channel = uuid()
client = Client(
sock, self.host, self.port, channel=channel
).register(self)
self.clients[sock] = client
def disconnect(self, sock):
client = self.clients.get(sock)
if client is not None:
client.unregister()
del self.clients[sock]
def read(self, sock, data):
client = self.clients[sock]
self.fire(write(data), client.channel)
app = Proxy(("0.0.0.0", 3333), "127.0.0.1", 22)
from circuits import Debugger
Debugger().register(app)
app.run()