Develop Reference

Python UDP socketserver returning empty message

I have a UDP socketserver program that I use to demonstrate how UDP works (code for the server and client are below). I run this on a server, then have the client.py program send a message and receive a reply. I am unfortunately running into an issue that seems to only occur on campus Wifi. On campus wifi, the client does not receive a response.
Troubleshooting with Wireshark shows the issue. For some reason the UDP server is responding with two UDP messages - one empty, and one containing the response message. These messages are recorded in Wireshark as coming in approximately 0.000002 seconds apart. On a wired network, the one with the response consistently comes first, and on Wifi, the empty message consistently comes first. Since the client is waiting for a single messages response, when the empty message returns, the client prints and exits, and the actual response is never seen.
I know I could write the client to listen for both messages and print out whichever one has the data, but I would rather try to figure out what's going on. Why is the socketserver responding with two messages in the first place, and how can I get it to only send one? OR at least to send the data first.
server.py:
import socketserver
class MyUDPRequestHandler(socketserver.DatagramRequestHandler):
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
# just send back the same data, but lower-cased
socket.sendto(data.lower(), self.client_address)
if __name__ == "__main__":
with socketserver.UDPServer(("0.0.0.0", 9091), MyUDPRequestHandler) as server:
server.serve_forever()
client.py:
import socket
HOST, PORT = "localhost", 9091
message = "NOW I AM SHOUTING" # The UDP server will lowercase the message
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(bytes(message + "\n", "utf-8"), (HOST, PORT))
received = str(sock.recv(1024), "utf-8")
print("Sent: {}".format(message))
print("Received: {}".format(received))

I've repeated the problem and it's socketserver. Notice the definition of DatagramRequestHandler below:
class DatagramRequestHandler(BaseRequestHandler):
"""Define self.rfile and self.wfile for datagram sockets."""
def setup(self):
from io import BytesIO
self.packet, self.socket = self.request
self.rfile = BytesIO(self.packet)
self.wfile = BytesIO()
def finish(self):
self.socket.sendto(self.wfile.getvalue(), self.client_address)
The packet is put into a buffer as rfile and should be read from there, then written back to the wfile buffer. finish sends the packet. The handler shouldn't call sendto itself:
import socketserver
class MyUDPRequestHandler(socketserver.DatagramRequestHandler):
def handle(self):
data = self.rfile.read()
self.wfile.write(data.strip().lower())
if __name__ == "__main__":
with socketserver.UDPServer(("0.0.0.0", 9091), MyUDPRequestHandler) as server:
server.serve_forever()
But just using a simple socket as the server works fine too:
import socket
s = socket.socket(type=socket.SOCK_DGRAM)
s.bind(('', 9091))
while True:
data, client = s.recvfrom(2048)
s.sendto(data.strip().lower(), client)
Note that UDP packets are not guaranteed to be delivered or delivered in the same order, so the original code's issue with the two packets changing order isn't surprising.

Python TCP sockets headers

I need to create a communication between a client and a server with TCP. But I'd like to send and work with "headers". So from the client I'd like to send a header "COMMAND1" and the server returns me something.
I have the following code:
Server
import socket
import threading
bind_ip = '0.0.0.0'
bind_port = 9998
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(5) # max backlog of connections
print ('Listening on {}:{}'.format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(1024)
print ('Received {}'.format(request))
client_socket.send('Response1!'.encode('utf-8'))
client_socket.close()
while True:
client_sock, address = server.accept()
print ('Accepted connection from {}:{}'.format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
Client
import socket
hostname, sld, tld, port = 'www', 'integralist', 'co.uk', 80
target = '{}.{}.{}'.format(hostname, sld, tld)
# create an ipv4 (AF_INET) socket object using the tcp protocol (SOCK_STREAM)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect the client
# client.connect((target, port))
client.connect(('0.0.0.0', 9998))
# send some data (in this case a HTTP GET request)
client.send('hi'.encode('utf-8'))
# receive the response data (4096 is recommended buffer size)
response = client.recv(4096)
print (response)
Anyone knows the best way to return "Response1!" when the header is "COMMAND1" and "Response2!" when the header is "COMMAND2"?
I can't find examples on how to use headers
EDIT: It doesn't have to be "COMMAND1" or "COMMAND2" it can be a "0" or "1", or anything else.

If you want to add your own header, you just have to:
Make sure your programm finds the start of your message (like, every message beginns "!?&")
Send your own header-data just after the start-symbol of your message.
Maybe mark the end of your message with something or pass a length in your header.
Since TCP will give you a stream of data, it might come to a case, where it just gives you 2 or 3 messages at once. You have to separate these messages by yourself (e.g. by using "?!&" as start of every message).
You can always create your own protocoll as payload of another protocoll. Just as TCP is just payload from the ethernet point of view.

You can do something i have done with my program to accept such headers
Use pickle library to encode a dict headers and send it through socket.
Code will look something like this.
import pickle
def handleSocket(headers:dict):
message = pickle.dumps(headers)
socket.send(message)
For server side, you will be handling it
Gonna initialise the socket recv to 100 kb
def handleReceivedSocket(socket):
message:dict = pickle.loads(socket.recv(102400))
Another way to do this. Is sending a raw json string to the server (just change pickle.dumps,pickle.loads by json.dumps,json.loads
But it will be in raw and less secure.
Last way you can do it is uri encoding. Check w3schools

How can I continually send data without shutdown socket connection in python

I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'

You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.

Python Socket errno 10054 only when client.py runs

There are some other posts about this issue but none did help me with mine.
I'm trying to build a total simple server - client relationship in python
server.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1",8889))
s.listen(1)
try:
while True:
client, add = s.accept()
data = client.recv(1024)
if not data:
print 'No data'
print data
finally:
s.close()
client.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1",8889))
try:
while True:
message = 'Foo'
s.send(message)
ans = s.recv(1024)
print ans
finally:
s.close()
I start by running the server first, but when I try to run the client I'm getting this Errno 10054 --> An existing connection was forcibly closed by the remote host
While request the browser with the ip and the related port, I receive some data.
I'm quiet new to networking, so please explain what might be obvious wrong in my code.
EDIT* Main issue is, that the client is somehow wrong, because it returns also an empty string on recv
Thank you in advance

Main issue is, that the client is somehow wrong, because it returns also an empty string on recv
The client isn't receiving anything from the server because the server is not sending anything.
On the server side, after print data, adding client.send(data) will send the string back to the client.

I am guessing:
The server accepts one socket and then does
client, add = s.accept()
data = client.recv(1024)
...
client, add = s.accept()
The client does this in the mean time:
s.send(message)
ans = s.recv(1024) # blocks until timeout
If now an other client connects to the server then client is replaced, the socket garbage collected and closed. s.recv(1024) will then tell that the connection is reset.
Have a look at import select or twisted (google around) to handle multiple connections at once.

Streaming continuous data over a network with python

I have a device that continually outputs data and I would like to send that data to a client on the same network as it is produced and I'm not finding a good solution. Here is what I'm trying.
Server:
import SocketServer
from subprocess import Popen,PIPE
class Handler(SocketServer.BaseRequestHandler):
def handle(self):
if not hasattr(self, 'Proc'):
self.Proc = Popen('r.sh', stdout=PIPE)
socket = self.request[1]
socket.sendto(self.Proc.stdout.readline(),self.client_address)
if __name__ == "__main__":
HOST, PORT = "192.168.1.1", 6001
server = SocketServer.UDPServer((HOST, PORT), Handler)
server.serve_forever()
Client:
import socket
data = " ".join(sys.argv[1:])
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(data + "\n", ("192.168.1.1", 6001))
try:
received = sock.recv(1024)
while True:
print "Sent: {}".format(data)
print "Received: {}".format(received)
sock.sendto('more' + "\n", ("192.168.1.1", 6001))
received = sock.recv(1024)
except:
print "No more messages"
arg[1] for the client is a program that outputs lines of data for several minutes that I need to process as it is created. The problem seems to be that every time the client sends another request, a new Handler object is created, so I loose Proc. How can I stream Proc.stdout?
Edit: The device is a Korebot2, so I have limited access to other python libraries due to space.

Using UDP you get a new "connection" each time you send a datagram, which is the reason you notice that a new object instance is created each time you send something. You're probably using the wrong kind of protocol here... UDP is used mostly for sending distinct "datagrams", or when a longer connection is not needed. TCP is also called a "streaming" protocol, and is often used for data that has no fixed end.
Also remember that UDP is not a reliable protocol, if used over a network it is almost guaranteed that you will loose packets.