Develop Reference

Python UDP socketserver returning empty message

I have a UDP socketserver program that I use to demonstrate how UDP works (code for the server and client are below). I run this on a server, then have the client.py program send a message and receive a reply. I am unfortunately running into an issue that seems to only occur on campus Wifi. On campus wifi, the client does not receive a response.
Troubleshooting with Wireshark shows the issue. For some reason the UDP server is responding with two UDP messages - one empty, and one containing the response message. These messages are recorded in Wireshark as coming in approximately 0.000002 seconds apart. On a wired network, the one with the response consistently comes first, and on Wifi, the empty message consistently comes first. Since the client is waiting for a single messages response, when the empty message returns, the client prints and exits, and the actual response is never seen.
I know I could write the client to listen for both messages and print out whichever one has the data, but I would rather try to figure out what's going on. Why is the socketserver responding with two messages in the first place, and how can I get it to only send one? OR at least to send the data first.
server.py:
import socketserver
class MyUDPRequestHandler(socketserver.DatagramRequestHandler):
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
# just send back the same data, but lower-cased
socket.sendto(data.lower(), self.client_address)
if __name__ == "__main__":
with socketserver.UDPServer(("0.0.0.0", 9091), MyUDPRequestHandler) as server:
server.serve_forever()
client.py:
import socket
HOST, PORT = "localhost", 9091
message = "NOW I AM SHOUTING" # The UDP server will lowercase the message
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(bytes(message + "\n", "utf-8"), (HOST, PORT))
received = str(sock.recv(1024), "utf-8")
print("Sent: {}".format(message))
print("Received: {}".format(received))

I've repeated the problem and it's socketserver. Notice the definition of DatagramRequestHandler below:
class DatagramRequestHandler(BaseRequestHandler):
"""Define self.rfile and self.wfile for datagram sockets."""
def setup(self):
from io import BytesIO
self.packet, self.socket = self.request
self.rfile = BytesIO(self.packet)
self.wfile = BytesIO()
def finish(self):
self.socket.sendto(self.wfile.getvalue(), self.client_address)
The packet is put into a buffer as rfile and should be read from there, then written back to the wfile buffer. finish sends the packet. The handler shouldn't call sendto itself:
import socketserver
class MyUDPRequestHandler(socketserver.DatagramRequestHandler):
def handle(self):
data = self.rfile.read()
self.wfile.write(data.strip().lower())
if __name__ == "__main__":
with socketserver.UDPServer(("0.0.0.0", 9091), MyUDPRequestHandler) as server:
server.serve_forever()
But just using a simple socket as the server works fine too:
import socket
s = socket.socket(type=socket.SOCK_DGRAM)
s.bind(('', 9091))
while True:
data, client = s.recvfrom(2048)
s.sendto(data.strip().lower(), client)
Note that UDP packets are not guaranteed to be delivered or delivered in the same order, so the original code's issue with the two packets changing order isn't surprising.

Troubleshooting hanged UDP socket in a Python server application code

I am writing a UDP server application that serves as a back end to Teltonika FMB630 car mounted devices.
I already took care of the protocol specifics and decoding, the problem I am facing relates to the UDP socket used.
My UDP server has to send an acknowledgement to the client device upon receiving a message (that is the protocol), however, if I send those ACKs, the server socket stops receiving data after a while.
The server's UDP socket object is passed to an concurrent.futures.ThreadPoolExecutor that fires a function (send_ack) that sends the ACK, however this is not the issue because I tried calling send_ack in the main thread, after receiving data and the same issue occurs.
I suspect the problem is the remote device somehow breaks the connection or the ISP or MNO doesn't route the reply packet (this is a GPRS device) and then the socket.send() method that is used to send the acknowledge, somehow freezes other socket operations, specifically recvfrom_into called in the main thread loop.
I wrote two scripts to illustrate the situation:
udp_test_echo.py :
#!/usr/env/bin python
import socket
import concurrent.futures
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.connect(addr)
print("connected to {}".format(addr))
sock.send(ack)
print("ACK sent to {}".format(addr))
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(("127.0.0.1", 1337))
data = bytearray([0] * 10)
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
while True:
print("listening")
nbytes, address = s.recvfrom_into(data)
print("Socket Data received {} bytes Address {}".format(nbytes, address))
print("Data received: ", data, " Echoing back to client")
executor.submit(send_ack, s, address, data[:nbytes])
udp_test_client.py:
#!/usr/env/bin python
import socket
import time
import random
def get_random_bytes():
return bytearray([random.randint(0,255) for b in range(10)])
ip = "127.0.0.1"
port = 1337
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((ip, port))
while True:
stuff_to_send = get_random_bytes()
print("Sending stuff", stuff_to_send)
s.sendall(stuff_to_send)
print("reply: ", s.recvfrom(10))
time.sleep(0.1)
Running udp_test_echo.py in one terminal and udp_test_client.py in another, we see normal operation but if you Ctrl+C the test client and re run it, you will see that the server doesn't respond until it is restarted.
Is there a way to timeout a specific sending operation from a specific call to socket.send() method without affecting other calls ? (I want my socket.recvfrom_into call to block on the main thread)
If I settimeout on the entire socket object, I am going to have to deal with many exceptions while waiting for data in the main thread and I don't like to have to rely on exceptions for proper program operation.

The culprit was the socket.connect() call in send_ack, when being called on the server's socket object it causes the socket to no longer be bound and listen on the port specified in the start of the program.
Instead the send_ack function was changed to be:
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.sendto(ack, addr)
print("ACK sent to {}".format(addr))
socket.sendto(data, address) uses the existing connection instead of starting a new one.

Socket Server with multiply Clients

I just started programming Python.
My goal is to built a digital Picture Frame with three Screens. Therefore I use 3 Raspis, one for each Monitor.
For the communication of these Raspis I need to program a server and a Client.
For a first test I want to built a server which is able to send and receive messages to/from multiple clients.
So I started with a few socket tutorials an created the following program.
Server Class (TcpServer.py)
class TcpServer:
clients = []
serverIsRunning = 0
port = 0
def __init__(self, port):
self.port = port
self.serverIsRunning = 0
self.serverRunning = 0
def startServer (self):
print("start Server...")
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind(("", self.port))
self.server.listen(1)
self.serverRunning = 1
while self.serverRunning:
read, write, oob = select.select([self.server] + self.clients, [], [])
for sock in read:
if sock is self.server:
client, addr = self.server.accept()
self.clients.append(client)
print ("+++ Client ", addr[0], " verbunden")
else:
nachricht = sock.recv(1024)
ip = sock.getpeername()[0]
if nachricht:
print (ip, nachricht)
else:
print ("+++ Verbindung zu ", ip , " beendet")
sock.close()
self.clients.remove(sock)
for c in self.clients:
c.close()
self.clients.remove(c)
self.server.close()
def send(self, message):
message = message.encode()
self.server.send(message)
Client class (TcpClient.py)
import socket
class TcpClient:
def __init__(self, ip, port):
self.serverAdress = (ip, port)
self.connected = 0
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(self.serverAdress)
print ("connectet to ", self.serverAdress)
def send(self, message):
message = message.encode()
self.connection.send(message)
Server:
import threading
import TcpServer
tcpServer = TcpServer.TcpServer(50000)
threadTcpServer = threading.Thread(target = tcpServer.startServer)
threadTcpServer.start()
while True:
tcpServer.send(input("Nachricht eingeben: "))
Client:
import threading
import TcpClient
tcpClient = TcpClient.TcpClient("192.168.178.49", 50000)
while True:
tcpClient.send(input("Nachricht eingeben: "))
I can send messages from the Client to the server, but when I want to send a Message from the server to the client it generates the following error:
BrokenPipeError: [Errno 32] Broken pipe
I assume it is because the server thread blocks the socket while waiting of a incoming message. But I have no idea how to handle this.
How can I program a server who can send and receive messages? Can you recommend a tutorial? I didn't found a tutorial who describes a solution for my problem.
Edit:
Now I tried to solve the problem with the socketserver library, but I still can't solve may problem.
here is my new code for the server:
import socketserver
import threading
import time
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
threadTcpServer = threading.Thread(target = server.serve_forever)
threadTcpServer.start()
print("server started")
time.sleep(10)
print("sending Data")
server.request.sendall("Server is sending...")
it generates the error:
AttributeError: 'TCPServer' object has no attribute 'request'
My goal is to write a server with a thread who receives Data and still be able to send data from a other thread.
Is this even possible with only one socket?

You should use the provided socketserver rather than writing all the handling of sockets and select etc.
There are multiple problems with your code -
1 - The server is trying to write to the listening socket!! The client communication socket is the one that you get from the accept() call and that is the one you have to use for reading and writing.
2 - The client is sending the data and completing immediately, but it should really wait for getting a response. Otherwise, the python / OS will close the client socket as soon as the program completes and it will mostly be before the server gets a chance to respond.

I believe with the Handler code you are able to receive the data sent by the client on the server and are also able to send some data back from the Handler to the client? You must have understood that the server cannot send any data back unless there is a client connected to it?
Now, to send data to the client (or clients) from "another" thread, you will need a way to make the handler objects or the client sockets (available inside the Handler object as self.request) available to the "another" thread.
One way is to override the def __init__(self, request, client_address, server): method and save this object's reference in a global list. Remember to do the below as the last line of the overridden init -
# BaseRequestHandler __init__ must be the last statement as all request processing happens in this method
socketserver.BaseRequestHandler.__init__(self, request, client_address, server)
Once you have all the client handlers in the global list, you can easily write to all the clients from any thread as per your needs. You must read about synchronization (Locks) and understand that using same object / socket from multiple threads can create some logical / data issues with your application.
Another thing that you have to worry about and code for is cleaning up this global list whenever a client closes the connection.

Python broadcasting message to all clients in a socket

I have made a simple chat server using threads like the following:
#-*- coding:utf-8 -*-
import _thread as thread
import time
import socket
def now():
return time.asctime(time.localtime())
def handleclient(connection, ADDR):
sod = str(ADDR)
msg = sod+"joined the chat"
msg2 = msg.encode("utf-8")
connection.sendall(msg2)
while True:
recieved = connection.recv(1024)
adsf = recieved.decode("utf-8")
print(now(),"(%s):%s" % (ADDR, recieved))
output = "%s:%s"%(ADDR, recieved.decode("utf-8"))
message = output.encode("utf-8")
connection.sendall(message)
if __name__ == "__main__":
addr = ("", 8080)
r =socket.socket()
print("socket object created at", now())
r.bind(addr)
r.listen(5)
while True:
print("Waiting for clients...")
connection, ADDR = r.accept()
print("We have connection from ", ADDR)
thread.start_new_thread(handleclient, (connection, ADDR))
However, it looks like the sendall isnt working and sending the message to only the person who sent it. How can I make it send it to all clients?

There is nothing like what you're trying to do, because as pointed out in the commends, sendall() means "definitely send all my bytes and keep trying until you have," not "send these bytes to lots of clients."
You will want to use either UDP multicast (if you're on a relatively reliable network which supports it, such as a LAN or corporate WAN), or you will simply need to send explicitly to every connected client. The other alternative is peer-to-peer: send to several clients and instruct those clients to send to more clients until all clients are taken care of. Obviously this requires more coding. :)

You may have a look at Zero MQ, which provides high-level facilities over sockets by implementing several patterns ( publish/subscribe , push/pull, etc...).

Correct multiprocessing to treat UDP in Python

I am trying to implement a simple UDP client and server. Server should receive a message and return a transformed one.
My main technique for server is to listen UDP messages in a loop, then spawn multiprocessing.Process for each incoming message and send the reply within each Process instance:
class InputProcessor(Process):
...
def run(self):
output = self.process_input()
self.sock.sendto(output, self.addr) # send a reply
if __name__ == "__main__":
print "serving at %s:%s" % (UDP_IP, UDP_PORT)
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP,UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message: %s from %s:%s" % (data, addr[0], addr[1])
p = InputProcessor(sock, data, addr)
p.start()
In test client, I do something like this:
def send_message(ip, port, data):
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
print "sending: %s" % data
sock.sendto(data, (ip, port))
sock.close()
for i in xrange(SECONDS*REQUESTS_PER_SECOND):
data = generate_data()
p = multiprocessing.Process(target=send_message, args=(UDP_IP,
UDP_PORT,
data))
p.start()
time.sleep(1/REQUESTS_PER_SECOND)
The problem I am having with the code above is that when REQUESTS_PER_SECOND becomes higher than certain value (~50), it seems some client processes receive responses destinated to different processes, i.e. process #1 receives response for process #2, and vice versa.
Please criticize my code as much as possible, due to I am new to network programming and may miss something obvious. Maybe it's even worth and better for some reason to use Twisted, hovewer, I am highly interested in understanding the internals. Thanks.

As per previous answer, I think that the main reason is that there is a race condition at the UDP port for the clients. I do not see receiving at the client code, but presumably it is similar to the one in server part. What I think happens in concrete terms is that for values under 50 requests / second, the request - response roundtrip gets completed and the client exits. When more requests arrive, there may be multiple processes blocking to read the UDP socket, and then it is probably nondeterministic which client process receives the incoming message. If the network latency is going to be larger in the real setting, this limit will be hit sooner.

Thanks guys a lot! It seems I've found why my code failed before. I was using multiprocessing.Manager().dict() within client to check if the results from server are correct. However, I didn't use any locks to wrap a set of write operations to that dict(), thus got a lot of errors though the output from server was correct.
Shortly, in client, I was doing incorrect checks for correct server responses.