Develop Reference

Why does server and client gets out of sync? (python sockets)

I'm currently writing a small client-server application for transferring an arbitrary file from a server to a client, via sockets.
The server will only handle one client at a time, but when a client is served it shall be ready to handle a new client connection.
The client will request a file, if the file exist, the client will receive the file, write it to disk and close the connection.
Server code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
print('The server is ready to receive')
server_socket = socket(AF_INET, SOCK_STREAM)
server_socket.bind(('', PORT))
server_socket.listen(1)
while True:
connection_socket, addr = server_socket.accept()
try:
requested_filepath = connection_socket.recv(BUFSIZE).decode()
print("Client requested the file: " + requested_filepath)
capital_sentence = requested_filepath.upper()
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(filesize.encode())
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode())
finally:
connection_socket.close()
Client code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
servername = argv[0]
filepath = argv[1]
client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect((servername, PORT))
try:
client_socket.send(filepath.encode())
response = client_socket.recv(BUFSIZE).decode()
if(response != "error"):
filesize = int(response)
print("Requested filesize: " + str(filesize))
filename = filepath.split('/')[-1]
with open(filename, 'wb') as f:
while(True):
content = client_socket.recv(BUFSIZE)
if not content:
break
f.write(content)
print('File recived')
else:
print("The requested file did not exist")
finally:
client_socket.close()
I can run the server and get the client to request and get a file, but when i run the client for a second or third time, the server and client seems to get out of sync. Both programs breaks and returns the following error message:
Client error:
Traceback (most recent call last):
File "client.py", line 37, in <module>
main(sys.argv[1:])
File "client.py", line 16, in main
response = client_socket.recv(BUFSIZE).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 6: invalid start byte
Server error:
The server is ready to receive
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
Traceback (most recent call last):
File "server.py", line 44, in <module>
main(sys.argv[1:])
File "server.py", line 30, in main
connection_socket.send(content)
ConnectionResetError: [Errno 104] Connection reset by peer
Am I not closing the socket connection in a proper way?

You have fallen into one of the most common TCP socket programming traps. You assumed your socket would send messages, while it sends and receives only data and is completely agnostic to your messaging structure. Even if you send data using several send calls, your recv calls do not receive this exact structure but whatever happens to be in the buffer. If you sent one byte a thousand times, your recv(1000) would receive a thousand bytes and this is what is going on here.
Your issue is caused by your server being a bit faster than your client. I had to tweak your code to be able to reproduce the code reliably but this does it:
client_socket.send(filepath.encode())
sleep(1)
response = client_socket.recv(BUFSIZE).decode()
This emulates your server being faster than the client, which eventually will happen anyway. By adding sleep we can make it happen every time.
When you call recv on a TCP socket, one of the following five things can happen:
There is no data and the call blocks
You received data and the data you received is exactly one "message", whatever that is in your context
Your server had sent more than one message before you read from the socket and you received them all on one go
Your client was too eager to read and it decided to read when only a part of your first message was available
Combination of 3 and 4: You receive several full messages plus one partial
What happens with your code is that your server has managed to send the encoded file size and some of your data as well. On your client you now assume your first recv receives only the file size, but this is no way guaranteed. There can be already some file data (as you will read BUFSIZE - there can be almost a full buffer of data there) and when you try to decode that as an integer, weird things happen as the data is not what you expected it to be.
The only reliable way to handle TCP sockets is to read from the socket, append to a temporary processing buffer, then parse that buffer and see what is in there. If there is a "message", process it and delete it from the buffer. Whatever remains in the buffer must stay there and your next recv result gets appended to this.
The simplest way to quickfix this is if your server makes the initial message of a fixed length. Then you can safely read exactly this amount of characters from the socket and process this as the size/error message, and the rest will be data. This is a horrible fix in many, many ways and you should aim for something better. The "proper" way is to devise a protocol, where the server puts delimiters in place so that your client can detect which message means what. Your protocol could be for example
SIZE: <decimal>\n
DATA: <data>
or even as simple as assuming everything before a newline is filesize and everything that follows is data.
But this works better even with sleep(1) added as it will now pad the initial message to exactly 100 bytes. This could still go wrong because of (4), so actually you will need to check that you received 100 characters initially and keep reading until you do, but I will leave this for you to implement.
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(("%s" % filesize).encode().ljust(100))
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode().ljust(100))
Client:
try:
client_socket.send(filepath.encode())
sleep(1)
response_raw = client_socket.recv(100)
response = response_raw.strip().decode()
PS your server should catch the "connection reset by peer" error. It is something that can happen if there is a network problem or the client application crashes. The server can safely ignore this error and just stop sending to that particular client socket.

python recv losing first bytes of data

I have a problem when recieving messages sent over an ssl socket. On rare occasions I lose the first few bytes of data in the message. I am pretty certain this somehow is a speed problem since it only seems to happen when 2 messages are sent in rapid succession (1-2 milliseconds apart). I am running the recieving code in a separate thread with minimal code dumping the messages in a queue as they arrive.
queue = Queue()
...
def read_feed(session_key, hostname, port, ssl_socket):
''' READ whatever is coming on the stream '''
while (1):
try:
output = ssl_socket.recv(2048) # Message size always < 2048
except (ConnectionResetError, OSError):
logger.info("Connecting feed")
try:
ssl_socket.connect((hostname, port))
except ValueError: # Something's wrong, disconnect and do a new round
ssl_socket.close()
else:
cmd = {"cmd":"login", "args":{"session_key":session_key}}
data = str.encode(json.dumps(cmd) + "\n")
num_bytes = ssl_socket.send(data)
else:
queue.put(output)
...
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_socket = ssl.wrap_socket(s)
...
t3 = threading.Thread(target=read_feed, name = 'Read Feed', args=(session_key, hostname, port, ssl_socket))
t3.start()
I was first suspecting that somehow the other threads running was stealing too much CPU time so that the network buffer was filled before this thread got a chance to run, but I have tried to use a multi core machine and the problem persists.
In essense this should be the only code running when I am connected?
while (1):
output = ssl_socket.recv(2048) # Message size always < 2048
queue.put(output)
Or am I making the wrong assumptions here? Maybe the try:/except: construct is costly, or is the queue.put method slow and I should use something else? Or maybe Python is not the right tool for the job?
Any suggestions on how to improve the code so that I don't lose those few precious first bytes?

Sockets Python 3.5: Socket server hangs forever on file receive

I'm trying to write a Python program that can browse directories and grab files w/ sockets if the client connects to the server. The browsing part works fine, it prints out all directories of the client.
Here's a part of the code:
with clientsocket:
print('Connected to: ', addr)
while True:
m = input("Command > ")
clientsocket.send(m.encode('utf-8'))
data = clientsocket.recv(10000)
if m == "exit":
clientsocket.close()
if m.split()[0] == 'get':
inp = input("Filename > ")
while True:
rbuf = clientsocket.recv(8192)
if not rbuf:
break
d = open(inp, "ab")
d.write(rbuf)
d.close()
elif data.decode('utf-8').split()[0] == "LIST":
print(data.decode('utf-8'))
if not data:
break
However, the problem lies in here:
if m.split()[0] == 'get':
inp = input("Filename > ")
while True:
rbuf = clientsocket.recv(8192)
if not rbuf:
break
It seems to be stuck in an infinite loop. What's more interesting is that the file I'm trying to receive is 88.3kb, but what the file returns is 87kb while it's in the loop, which is very close...
I tried receiving a python script at one time as well (without the loop) and it works fine.
Here's some of the client code:
while True:
msg = s.recv(1024).decode('utf-8')
if msg.split()[0] == "list":
dirs = os.listdir(msg.split()[1])
string = ''
for dira in dirs:
string += "LIST " + dira + "\n"
s.send(string.encode('utf-8'))
elif msg == "exit":
break
else:
#bit that sends the file
with open(msg.split()[1], 'rb') as r:
s.sendall(r.read())
So my question is, why is it getting stuck in an infinite loop if I have it set up to close when there is no data, and how can I fix this?
I'm sort of new to network programming in general, so forgive me if I miss something obvious.
Thanks!

I think I know what's the problem, but I may be wrong. It happened to me several times, that the entire message is not received in one recv call, even if I specify the correct length. However, you don't reach the end of stream, so your program keeps waiting for remaining of 8192 bytes which never arrives.
Try this:
Sending file:
#bit that sends the file
with open(msg.split()[1], 'rb') as r:
data = r.read()
# check data length in bytes and send it to client
data_length = len(data)
s.send(data_length.to_bytes(4, 'big'))
s.send(data)
s.shutdown(socket.SHUT_RDWR)
s.close()
Receiving the file:
# check expected message length
remaining = int.from_bytes(clientsocket.recv(4), 'big')
d = open(inp, "wb")
while remaining:
# until there are bytes left...
# fetch remaining bytes or 4094 (whatever smaller)
rbuf = clientsocket.recv(min(remaining, 4096))
remaining -= len(rbuf)
# write to file
d.write(rbuf)
d.close()

There are several issues with your code.
First:
clientsocket.send(m.encode('utf-8'))
data = clientsocket.recv(10000)
This causes the file to be partially loaded to data variable when you issue get statement. That's why you don't get full file.
Now this:
while True:
rbuf = clientsocket.recv(8192)
if not rbuf:
break
...
You indeed load full file but the client never closes the connection (it goes to s.recv() after sending the file) so if statement is never satisfied. Thus this loop gets blocked on the clientsocket.recv(8192) part after downloading the file.
So the problem is that you have to somehow notify the downloader that you've sent all the data even though the connection is still open. There are several ways to do that:
You calculate the size of the file and send it as a first few bytes. For example, say the content of the file is ala ma kota. These are 11 bytes and thus you send \x11ala ma kota. Now receiver knows that first byte is size and it will interpret it as such. Of course one byte header isn't much (you would only be able to send max 256 byte files) so normally you go for for example 4 bytes. So now your protocol between client and server is: first 4 bytes is the size of the file. Thus our initial file would be sent as \x0\x0\x0\x11ala ma kota. The drawback of this solution is that you have to know the size of the content before sending it.
You mark the end of the stream. So you pick a particular character, say X and you read the straem until you find X. If you do then you know that the other side sent everything it has. The drawback is that if X is in the content then you have to escape it, i.e. additional content processing (and interpretation on the other side) is needed.

Python Socket Receive Large Amount of Data

When I try to receive larger amounts of data it gets cut off and I have to press enter to get the rest of the data. At first I was able to increase it a little bit but it still won't receive all of it. As you can see I have increased the buffer on the conn.recv() but it still doesn't get all of the data. It cuts it off at a certain point. I have to press enter on my raw_input in order to receive the rest of the data. Is there anyway I can get all of the data at once? Here's the code.
port = 7777
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('0.0.0.0', port))
sock.listen(1)
print ("Listening on port: "+str(port))
while 1:
conn, sock_addr = sock.accept()
print "accepted connection from", sock_addr
while 1:
command = raw_input('shell> ')
conn.send(command)
data = conn.recv(8000)
if not data: break
print data,
conn.close()

TCP/IP is a stream-based protocol, not a message-based protocol. There's no guarantee that every send() call by one peer results in a single recv() call by the other peer receiving the exact data sent—it might receive the data piece-meal, split across multiple recv() calls, due to packet fragmentation.
You need to define your own message-based protocol on top of TCP in order to differentiate message boundaries. Then, to read a message, you continue to call recv() until you've read an entire message or an error occurs.
One simple way of sending a message is to prefix each message with its length. Then to read a message, you first read the length, then you read that many bytes. Here's how you might do that:
def send_msg(sock, msg):
# Prefix each message with a 4-byte length (network byte order)
msg = struct.pack('>I', len(msg)) + msg
sock.sendall(msg)
def recv_msg(sock):
# Read message length and unpack it into an integer
raw_msglen = recvall(sock, 4)
if not raw_msglen:
return None
msglen = struct.unpack('>I', raw_msglen)[0]
# Read the message data
return recvall(sock, msglen)
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = bytearray()
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data.extend(packet)
return data
Then you can use the send_msg and recv_msg functions to send and receive whole messages, and they won't have any problems with packets being split or coalesced on the network level.

You can use it as: data = recvall(sock)
def recvall(sock):
BUFF_SIZE = 4096 # 4 KiB
data = b''
while True:
part = sock.recv(BUFF_SIZE)
data += part
if len(part) < BUFF_SIZE:
# either 0 or end of data
break
return data

The accepted answer is fine but it will be really slow with big files -string is an immutable class this means more objects are created every time you use the + sign, using list as a stack structure will be more efficient.
This should work better
while True:
chunk = s.recv(10000)
if not chunk:
break
fragments.append(chunk)
print "".join(fragments)

Most of the answers describe some sort of recvall() method. If your bottleneck when receiving data is creating the byte array in a for loop, I benchmarked three approaches of allocating the received data in the recvall() method:
Byte string method:
arr = b''
while len(arr) < msg_len:
arr += sock.recv(max_msg_size)
List method:
fragments = []
while True:
chunk = sock.recv(max_msg_size)
if not chunk:
break
fragments.append(chunk)
arr = b''.join(fragments)
Pre-allocated bytearray method:
arr = bytearray(msg_len)
pos = 0
while pos < msg_len:
arr[pos:pos+max_msg_size] = sock.recv(max_msg_size)
pos += max_msg_size
Results:

You may need to call conn.recv() multiple times to receive all the data. Calling it a single time is not guaranteed to bring in all the data that was sent, due to the fact that TCP streams don't maintain frame boundaries (i.e. they only work as a stream of raw bytes, not a structured stream of messages).
See this answer for another description of the issue.
Note that this means you need some way of knowing when you have received all of the data. If the sender will always send exactly 8000 bytes, you could count the number of bytes you have received so far and subtract that from 8000 to know how many are left to receive; if the data is variable-sized, there are various other methods that can be used, such as having the sender send a number-of-bytes header before sending the message, or if it's ASCII text that is being sent you could look for a newline or NUL character.

Disclaimer: There are very rare cases in which you really need to do this. If possible use an existing application layer protocol or define your own eg. precede each message with a fixed length integer indicating the length of data that follows or terminate each message with a '\n' character. (Adam Rosenfield's answer does a really good job at explaining that)
With that said, there is a way to read all of the data available on a socket. However, it is a bad idea to rely on this kind of communication as it introduces the risk of loosing data. Use this solution with extreme caution and only after reading the explanation below.
def recvall(sock):
BUFF_SIZE = 4096
data = bytearray()
while True:
packet = sock.recv(BUFF_SIZE)
if not packet: # Important!!
break
data.extend(packet)
return data
Now the if not packet: line is absolutely critical!
Many answers here suggested using a condition like if len(packet) < BUFF_SIZE: which is broken and will most likely cause you to close your connection prematurely and loose data. It wrongly assumes that one send on one end of a TCP socket corresponds to one receive of sent number of bytes on the other end. It does not. There is a very good chance that sock.recv(BUFF_SIZE) will return a chunk smaller than BUFF_SIZE even if there's still data waiting to be received. There is a good explanation of the issue here and here.
By using the above solution you are still risking data loss if the other end of the connection is writing data slower than you are reading. You may just simply consume all data on your end and exit when more is on the way. There are ways around it that require the use of concurrent programming, but that's another topic of its own.

A variation using a generator function (which I consider more pythonic):
def recvall(sock, buffer_size=4096):
buf = sock.recv(buffer_size)
while buf:
yield buf
if len(buf) < buffer_size: break
buf = sock.recv(buffer_size)
# ...
with socket.create_connection((host, port)) as sock:
sock.sendall(command)
response = b''.join(recvall(sock))

You can do it using Serialization
from socket import *
from json import dumps, loads
def recvall(conn):
data = ""
while True:
try:
data = conn.recv(1024)
return json.loads(data)
except ValueError:
continue
def sendall(conn):
conn.sendall(json.dumps(data))
NOTE: If you want to shara a file using code above you need to encode / decode it into base64

I think this question has been pretty well answered, but I just wanted to add a method using Python 3.8 and the new assignment expression (walrus operator) since it is stylistically simple.
import socket
host = "127.0.0.1"
port = 31337
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen()
con, addr = s.accept()
msg_list = []
while (walrus_msg := con.recv(3)) != b'\r\n':
msg_list.append(walrus_msg)
print(msg_list)
In this case, 3 bytes are received from the socket and immediately assigned to walrus_msg. Once the socket receives a b'\r\n' it breaks the loop. walrus_msg are added to a msg_list and printed after the loop breaks. This script is basic but was tested and works with a telnet session.
NOTE: The parenthesis around the (walrus_msg := con.recv(3)) are needed. Without this, while walrus_msg := con.recv(3) != b'\r\n': evaluates walrus_msg to True instead of the actual data on the socket.

Modifying Adam Rosenfield's code:
import sys
def send_msg(sock, msg):
size_of_package = sys.getsizeof(msg)
package = str(size_of_package)+":"+ msg #Create our package size,":",message
sock.sendall(package)
def recv_msg(sock):
try:
header = sock.recv(2)#Magic, small number to begin with.
while ":" not in header:
header += sock.recv(2) #Keep looping, picking up two bytes each time
size_of_package, separator, message_fragment = header.partition(":")
message = sock.recv(int(size_of_package))
full_message = message_fragment + message
return full_message
except OverflowError:
return "OverflowError."
except:
print "Unexpected error:", sys.exc_info()[0]
raise
I would, however, heavily encourage using the original approach.

For anyone else who's looking for an answer in cases where you don't know the length of the packet prior.
Here's a simple solution that reads 4096 bytes at a time and stops when less than 4096 bytes were received. However, it will not work in cases where the total length of the packet received is exactly 4096 bytes - then it will call recv() again and hang.
def recvall(sock):
data = b''
bufsize = 4096
while True:
packet = sock.recv(bufsize)
data += packet
if len(packet) < bufsize:
break
return data

This code reads 1024*32(=32768) bytes in 32 iterations from the buffer which is received from Server in socket programming-python:
jsonString = bytearray()
for _ in range(32):
packet = clisocket.recv(1024)
if not packet:
break
jsonString.extend(packet)
Data resides in jsonString variable

Plain and simple:
data = b''
while True:
data_chunk = client_socket.recv(1024)
if data_chunk:
data+=data_chunk
else:
break

Python socket Recv not working properly could someone explain

i am trying to create a gui client for my command line server. However, i am running into some annoying problems i cant seem to fix.
I'm not 100 % sure of what the actual problem is as sometimes the code will work, other times it wont. I think the main problem is that originally i tried the
while 1:
self.data = s.recv(1024)
if not self.data():
break
else:
print self.data()
Then i was sending to it with this
for f in files:
s.send(f)
Each f was a string of a filename. I expected it to come out on the recv side as one file name recieved for each recv call but instead on one recv call i got a big chunk of filenames i assume 1024 chars worth
Which made it impossible to check for the end of the data and thus the loop never exited.
This is the code i have now
def get_data(self,size = 1024):
self.alldata = ""
while 1:
while gtk.events_pending():
gtk.main_iteration()
self.recvdata = self.s.recv(size)
self.alldata += self.recvdata
if self.alldata.find("\r\n\r\nEOF"):
print "recieved end message"
self.rdata = self.alldata[:self.alldata.find("\r\n\r\nEOF")]
break
print "All data Recieved: " + str(len(self.rdata)) + "Bytes"
print "All data :\n" + self.rdata + "\n-------------------------------------------------"
self.infiles = self.rdata.split("-EOS-")
for nf in self.infiles:
if len(nf) > 2:
self.add_message(self.incomingIcon,nf)
At the minute im trying to get the client to read correctly from the server. What i want to happen is when the command list is typed in and sent to the client the server sends back the data and each file is appended to the list store
some times this works ok, other times only one of 1200 files gets returned, if it executes ok, if i try to type another command and send it , the whole gtk window geys out and the program becomes unresponsive.
Sorry i cant explain this question better, ive tried alot of different solutions all of which give different errors.
if someone could explain the recv command and why it may be giving the errors this is how im sending data to the client
if(commands[0] == 'list'):
whatpacketshouldlooklike=""
print "[Request] List files ", address
fil = list_files(path)
for f in fil:
sdata = f
whatpacketshouldlooklike += sdata + "-EOS-"
newSock.send(sdata +"-EOS-")
#print "sent: " + sdata
newSock.send("\r\n\r\nEOF")
whatpacketshouldlooklike += "\r\n\r\nEOF"
print "---------------------------------"
print whatpacketshouldlooklike
print "---------------------------------"

The problem you had in the first part is that sockets are stream based not message based. You need to come up with a message abstraction to layer on top of the stream. This way the other end of the pipe knows what is going on(how much data to expect as a part of one command) and isn't guessing at what is supposed to happen.

Use an abstraction layer (Pyro, XML-RPC, zeromq) or define your own protocol to distinguish messages.
For example as own protocol you can send the length of a message as a "header" before each string. In this case you should use the struct module to parse the length into a binary format. Ask again, if you want to go this way, but I strongly recommend choosing one of the mentioned abstraction layers.

There are different problems with your code.
Let's start with the fundamental that some people already commented, there is no relation between sends() and recv(), you do not control which part of the data is returned on a recv(call), you need some kind of protocol, on your case it could be just as simple as terminating command strings with "\n", and checking for "\n" on the server to consume the data.
Now other problems:
You are using send without checking it's return size, a send() does not guarantee that the data is completely written, if you need that please use sendall().
By using recv(1024) in a blocking socket (default), your server code may wait for 1024 bytes to be received, this will not allow you to process messages until you get the full chunk, you need to use a non blocking socket, and the select module.

My source code:
def readReliably(s,n):
buf = bytearray(n)
view = memoryview(buf)
sz = 0
while sz < n:
k = s.recv_into(view[sz:],n-sz)
sz += k
# print 'readReliably()',sz
return sz,buf
def writeReliably(s,buf,n):
sz = 0
while sz < n:
k = s.send(buf[sz:],n-sz)
sz += k
# obj = s.makefile(mode='w')
# obj.flush()
# print 'writeReliably()',sz
return sz
Usage of these functions:
# Server
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
s.bind((host,port))
s.listen(10) # unaccepted connections
while True:
sk,skfrom = s.accept()
sz,buf = io.readReliably(sk,4)
a = struct.unpack("4B",buf)
print repr(a)
# ...
io.writeReliably(sk,struct.pack("4B",*[0x01,0x02,0x03,0x04]))
See also official docs about recv_into(...), https://docs.python.org/2/library/socket.html#socket.socket.recv_into