Develop Reference

Recvall with while loop doesn't work between two devices in python

I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)

recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...

Reliable UDP implementation using sequence numbers, deadlocking

I am trying to implement a reliable UDP messaging scheme between a single client and a server.
In my current code I can send incrementing numbers by 1 to the server if he only uses the receive command. if the server tries replying with the received data using the send command, it seems to work for 1-3 messages back and forth, and then I enter a deadlock. I do not understand where the source of the deadlock comes from. Below is my implementation of send and receive.
Both the client and server start with their self.seqnumber set to 0 and both sockets are set to timeout after 1 second. Both the client and server share these methods as they belong to a class both the client and server import.
def sendCommand(self, command):
self.s.sendto((self.seqnumber).to_bytes(8, "little") + command, self.address)
try:
data, self.address = self.s.recvfrom(1200)
if int.from_bytes(data[:8], 'little') == self.seqnumber:
self.seqnumber += 1
return 0
else:
return self.sendCommand(command)
except:
return self.sendCommand(command)
def getCommand(self):
while(1):
try:
data, self.address = self.s.recvfrom(1200)
if int.from_bytes(data[:8], 'little') == self.seqnumber:
self.s.sendto(data, self.address)
self.seqnumber += 1
break
elif int.from_bytes(data[:8], 'little') < self.seqnumber:
self.s.sendto(data, self.address)
else:
continue
except:
continue
return data[8:]
The code running on the server (commInf is the class in which the get and send command are defined):
while (1):
command = self.commInf.getCommand()
print(command.decode())
self.commInf.sendCommand(command)
and the code running on the client:
for i in range(100):
self.commInf.sendCommand(f"{i}".encode())
command = self.commInf.getCommand()
print(command.decode())
I expect the output to allow me to reliably send messages and return them using the sendCommand(with the received data) since the returned data from getCommand does not include its sequence number and is just the raw data.

Why does server and client gets out of sync? (python sockets)

I'm currently writing a small client-server application for transferring an arbitrary file from a server to a client, via sockets.
The server will only handle one client at a time, but when a client is served it shall be ready to handle a new client connection.
The client will request a file, if the file exist, the client will receive the file, write it to disk and close the connection.
Server code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
print('The server is ready to receive')
server_socket = socket(AF_INET, SOCK_STREAM)
server_socket.bind(('', PORT))
server_socket.listen(1)
while True:
connection_socket, addr = server_socket.accept()
try:
requested_filepath = connection_socket.recv(BUFSIZE).decode()
print("Client requested the file: " + requested_filepath)
capital_sentence = requested_filepath.upper()
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(filesize.encode())
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode())
finally:
connection_socket.close()
Client code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
servername = argv[0]
filepath = argv[1]
client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect((servername, PORT))
try:
client_socket.send(filepath.encode())
response = client_socket.recv(BUFSIZE).decode()
if(response != "error"):
filesize = int(response)
print("Requested filesize: " + str(filesize))
filename = filepath.split('/')[-1]
with open(filename, 'wb') as f:
while(True):
content = client_socket.recv(BUFSIZE)
if not content:
break
f.write(content)
print('File recived')
else:
print("The requested file did not exist")
finally:
client_socket.close()
I can run the server and get the client to request and get a file, but when i run the client for a second or third time, the server and client seems to get out of sync. Both programs breaks and returns the following error message:
Client error:
Traceback (most recent call last):
File "client.py", line 37, in <module>
main(sys.argv[1:])
File "client.py", line 16, in main
response = client_socket.recv(BUFSIZE).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 6: invalid start byte
Server error:
The server is ready to receive
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
Traceback (most recent call last):
File "server.py", line 44, in <module>
main(sys.argv[1:])
File "server.py", line 30, in main
connection_socket.send(content)
ConnectionResetError: [Errno 104] Connection reset by peer
Am I not closing the socket connection in a proper way?

You have fallen into one of the most common TCP socket programming traps. You assumed your socket would send messages, while it sends and receives only data and is completely agnostic to your messaging structure. Even if you send data using several send calls, your recv calls do not receive this exact structure but whatever happens to be in the buffer. If you sent one byte a thousand times, your recv(1000) would receive a thousand bytes and this is what is going on here.
Your issue is caused by your server being a bit faster than your client. I had to tweak your code to be able to reproduce the code reliably but this does it:
client_socket.send(filepath.encode())
sleep(1)
response = client_socket.recv(BUFSIZE).decode()
This emulates your server being faster than the client, which eventually will happen anyway. By adding sleep we can make it happen every time.
When you call recv on a TCP socket, one of the following five things can happen:
There is no data and the call blocks
You received data and the data you received is exactly one "message", whatever that is in your context
Your server had sent more than one message before you read from the socket and you received them all on one go
Your client was too eager to read and it decided to read when only a part of your first message was available
Combination of 3 and 4: You receive several full messages plus one partial
What happens with your code is that your server has managed to send the encoded file size and some of your data as well. On your client you now assume your first recv receives only the file size, but this is no way guaranteed. There can be already some file data (as you will read BUFSIZE - there can be almost a full buffer of data there) and when you try to decode that as an integer, weird things happen as the data is not what you expected it to be.
The only reliable way to handle TCP sockets is to read from the socket, append to a temporary processing buffer, then parse that buffer and see what is in there. If there is a "message", process it and delete it from the buffer. Whatever remains in the buffer must stay there and your next recv result gets appended to this.
The simplest way to quickfix this is if your server makes the initial message of a fixed length. Then you can safely read exactly this amount of characters from the socket and process this as the size/error message, and the rest will be data. This is a horrible fix in many, many ways and you should aim for something better. The "proper" way is to devise a protocol, where the server puts delimiters in place so that your client can detect which message means what. Your protocol could be for example
SIZE: <decimal>\n
DATA: <data>
or even as simple as assuming everything before a newline is filesize and everything that follows is data.
But this works better even with sleep(1) added as it will now pad the initial message to exactly 100 bytes. This could still go wrong because of (4), so actually you will need to check that you received 100 characters initially and keep reading until you do, but I will leave this for you to implement.
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(("%s" % filesize).encode().ljust(100))
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode().ljust(100))
Client:
try:
client_socket.send(filepath.encode())
sleep(1)
response_raw = client_socket.recv(100)
response = response_raw.strip().decode()
PS your server should catch the "connection reset by peer" error. It is something that can happen if there is a network problem or the client application crashes. The server can safely ignore this error and just stop sending to that particular client socket.

Will select.poll detect read event (select.POLLIN) if the socket is closed

I am not able to detect socket client closing in a particular network. I am running a socket server and once a client connects I am saving the client socket and periodically sending a request to the client . I am using select.poll then to check if there is any data to be read from the socket, and if there is , will read from the socket. All this is fine as of now.
Question is , if the remote socket client is terminated, will select.poll signal a read event in the client socket. If this happens then I can check the data length returned in socket.recv to detect the client has disconnected - as is described here
Adding a code snippet for select
def _wait_for_socket_poller(self, read, write, message=None):
"""
Instead of blockign wait, this polls and check if the read or write socket is ready. If so it proceeds with
reading or writing to the socket. The advantage is that while the poll blocks, it yeilds back to the other
waiting greenlets; poll blocks because we have not given a timeout
:param read: The read function
:param write: The write function
:param message: The CrowdBox API call
:return: The result : In case of read - In JSON format; But catch is that the caller cannot wait on the
result being available,as else the thread will block
"""
if not self.client_socket:
logging.error("CB ID =%d - Connection closed", self.id)
return
poller = select.poll()
# Commonly used flag setes
READ_ONLY = select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR
WRITE_ONLY = select.POLLOUT
READ_WRITE = READ_ONLY | select.POLLOUT
if read and write:
poller.register(self.client_socket, READ_WRITE)
elif write:
poller.register(self.client_socket, WRITE_ONLY)
elif read:
poller.register(self.client_socket, READ_ONLY)
# Map file descriptors to socket objects
fd_to_socket = {self.client_socket.fileno(): self.client_socket, }
result = ''
retry = True
while retry:
# Poll will Block!!
events = poller.poll(
1) # using poll instead of select as the latter runs out of file descriptors on load
# Note here, Poll needs to timeout or will block ,as there is no gevent patched poll, the moment it blocks
# neither greenlets or Twisted Deffered can help -Everything freezes,as all of this is in main thread
if not events:
retry = True
gevent.sleep(0) # This is needed to yeild in case no input comes from CB
else:
retry = False
clientsock = None
fd = None
flag = None
for fd, flag in events:
# Retrieve the actual socket from its file descriptor to map return of poll to socket
clientsock = fd_to_socket[fd]
if clientsock is None:
logging.error("Problem Houston")
raise ValueError("Client Sokcet has Become Invalid")
if flag & select.POLLHUP:
logging.error("Client Socket Closed")
self.client_socket.close()
self.client_socket = None
return None
if flag & (select.POLLIN | select.POLLPRI):
if read:
result = read()
if flag & select.POLLOUT:
if write:
result = write(message)
# poller.uregister(self.client_socket)
return result

In general, yes, a socket will be marked as "readable" when a TCP connection is closed. But this assumes that there was a normal closing, meaning a TCP FIN or RST packet.
Sometimes TCP connections don't end that way. In particular, if TCP Keep-Alive is not enabled (and by default it is not), a network outage between server and client could effectively terminate the connection without either side knowing until they try to send data.
So if you want to make sure you are promptly notified when a TCP connection is broken, you need to send keep-alive messages at either the TCP layer or the application layer.
Keep-alive messages have the additional benefit that they can prevent unused connections from being automatically dropped by various network appliances due to long periods of inactivity.
For more on keep-alive, see here: http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html

Thought of adding an anwer here so that I can post some tcp dump trace. We tested this in a live network. The Socket client process in the remote machine terminated and python socket.send ( on a non blocking socket) client_socket.setblocking(0), did not return any error, for subsequent request send to the client from the server There was no event generated to indicate (EPOLLIN) something to read either.
So to detect the client connection loss, we ping the client periodically and if there is no expected response after three retrials , disconnect the client. Basically handled this in the application layer. Clients also changed to reply with some data for our 'are you alive' requests instead of just ignoring it.
sent = 0
try:
sent = self.client_socket.send(out)
except socket.error as e:
if e.args[0] == errno.EPIPE:
logging.error("Socket connection is closed or broken")
if sent == 0 and self.client_socket is not None:
logging.error("socket connection is already closed by client, cannot write request")
self.close_socket_connection()
else
# send succcessfully
Below is the tcpdump wireshark trace where you can see the re-transmit happening. IP details masked for security

Timers and retransmissions in a UDP file transfer program (using low-level sockets)

I'm trying to implement a UDP client program that receives a file from the server by sending multiple read requests and receiving small pieces of the file in return (1400 bytes, to prevent fragmentation). Each time I send a request, I set up a select() call with a timeout of 1 second. If I receive a response within that time, the client goes on to send the request for the next piece of file. Otherwise, the client resends the request for the same piece of file.
The problem with this sequential design is that the client waits for EACH request to be responded to before it sends the next one. If there is even a small ratio of packet losses, this would increase the time required for the sending of even a moderately large file to pretty unacceptable levels.
I would like to design the client to be able to send all of the read requests without waiting for responses, while simultaneously running a timer and retransmission loop that resends the individual requests that do not receive responses. However, I'm pretty new to coding in general, and can't seem to figure out how to do that. Do I need to open up several sockets at a time and run different loops on each? Or is there a more straightforward way of doing it?
My current code is (forgive the messiness):
def read_service_loop(self):
"""Loop governing the timing, checking, and retransmission or processing of read service. """
#Increment start_position each time packet sent, send a read request packet for each new position.
#Expect to receive a read_response packet for each time read request sent.
recv_data = None
print("Sending request to server to read and receive file...")
start_position = 0
while(self.eof == False):
print("Reading from byte " + str(start_position))
num_retransmits = 0
#Loop for retransmissions of the same start position
while(num_retransmits < 60):
num_retransmits = num_retransmits + 1
self.send_read_request(start_position)
input_socket = [self.client_socket]
inputready,outputready,exceptready = select.select(input_socket,[],[], 1)
if (inputready == []):
continue
else:
recv_data = self.client_socket.recv(self.buffer_)
bit_signature = recv_data[0:4]
response_type = recv_data[4:8]
recv_payload = recv_data[8:]
if bit_signature != "\x00\x00\x00\r":
self.recv_invalid_response(recv_data, "bit_signature")
continue
else:
if response_type == "\x00\x00\x00\x02":
#Packet is valid, proceed to recv_read_response to append this bit of file received into local_filename
self.file_append = open(self.local_filename, 'r+b')
self.recv_read_response(recv_payload)
break
else:
self.recv_invalid_response(recv_data, "response_type")
continue
start_position = start_position + self.NUM_BYTES_TO_READ
if (num_retransmits >= 60):
print ("Exceeded number of retransmissions allowed. Exiting program.")
sys.exit()
return

what you want to implement is called "sliding window", like TCP does. It will be complicated, because you need to take round-trip time into account. It is still evolving, see how TCP implementations of different operating systems have different performances. Perhaps you can find some library that already implements it.
Any reason not to use TCP?