Develop Reference

Why is looping Python TCP receiver receives message partially?

I have a server that sends some messages to a client. The print(trades) statement shows that file reader reads the entire csv correctly:
def send_past_trades(self):
with open('OTC_trade_records.csv',newline='') as f:
connectionSocket, addr = self.client
trades = f.read()
#print(trades)
connectionSocket.send(trades.encode())
My client receiver is like this:
msg = b""
while(True):
print("Batch receiving")
tmp = client_socket.recv(4096)
msg += tmp
if len(tmp) < 4096:
print(len(tmp))
break
msg = msg.decode()
print(msg)
The message is always partial. I can see that the statement "Batch receiving" is printed once and when the break statement is initiated, the length of the last message is 1228.
Another point is, this code works fine in my local system. The problem occurs when I put the server program to a remote server machine. Is there a possibility that server intervenes with the message?
Note: I tried different ways to solve the problem such as sending only package size of 1024b messages in a loop. Still partial messages received.

The problem is here:
if len(tmp) < 4096:
print(len(tmp))
break
The point is that bufsize in recv(bufsize) is a maximum size to receive. The recv will return fewer bytes if there are fewer available.
I suggest to define a simple communication protocol that describes the structure of a message with a header and payload. The header must contain the payload size. This allows you to parse data from the incoming TCP stream and get the exact size of the received data. Then you can receive requested amount of data.
A client will look like this:
import struct
# Receive a header
header = connection.recv(8)
(length,) = struct.unpack('>Q', header) # Parse payload length
# Receive the payload
payload = b''
while len(payload) < length:
to_read = length - len(payload)
payload += connection.recv(4096 if to_read > 4096 else to_read)
Server:
import struct
with open('OTC_trade_records.csv',newline='') as f:
connectionSocket, addr = self.client
trades = f.read()
length = struct.pack('>Q', len(trades))
connectionSocket.sendall(length)
connectionSocket.sendall(trades)

Python3 socket, random partial result on socket receive

I've written a basic client/server interface using Python socket (quoted only relevant part of code, for full script: (SERVER: https://github.com/mydomo/ble-presence/blob/master/server.py)
(CLIENT: https://github.com/mydomo/ble-presence/blob/master/clients/DOMOTICZ/ble-presence/plugin.py)
The issue is when the script run from some hours and the result list is getting bigger sometimes the reply is exactly as it should be, other times it's cutted, not complete... it's random, like if the socket closed for no reason earlier or the reply is not fully read.
Can you please help me?
SERVER:
def client_thread(conn, ip, port, MAX_BUFFER_SIZE = 32768):
# the input is in bytes, so decode it
input_from_client_bytes = conn.recv(MAX_BUFFER_SIZE)
# MAX_BUFFER_SIZE is how big the message can be
# this is test if it's too big
siz = sys.getsizeof(input_from_client_bytes)
if siz >= MAX_BUFFER_SIZE:
print("The length of input is probably too long: {}".format(siz))
# decode input and strip the end of line
input_from_client = input_from_client_bytes.decode("utf8").rstrip()
res = socket_input_process(input_from_client)
#print("Result of processing {} is: {}".format(input_from_client, res))
vysl = res.encode("utf8") # encode the result string
conn.sendall(vysl) # send it to client
conn.close() # close connection
##########- END FUNCTION THAT HANDLE SOCKET'S TRANSMISSION -##########
def start_server():
global soc
soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# this is for easy starting/killing the app
soc.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
#print('Socket created')
try:
soc.bind((socket_ip, socket_port))
# print('Socket bind complete')
except socket.error as msg:
# print('Bind failed. Error : ' + str(sys.exc_info()))
sys.exit()
#Start listening on socket
soc.listen(10)
#print('Socket now listening')
# for handling task in separate jobs we need threading
#from threading import Thread
# this will make an infinite loop needed for
# not reseting server for every client
while (not killer.kill_now):
conn, addr = soc.accept()
ip, port = str(addr[0]), str(addr[1])
#print('Accepting connection from ' + ip + ':' + port)
try:
Thread(target=client_thread, args=(conn, ip, port)).start()
except:
print("Terible error!")
import traceback
traceback.print_exc()
soc.close()
CLIENT:
soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SERV_ADDR = str(Parameters["Address"])
SERV_PORT = int(Parameters["Port"])
soc.connect((SERV_ADDR, SERV_PORT))
if BATTERY_REQUEST == True:
clients_input = str(BATTERY_DEVICE_REQUEST)
else:
clients_input = "beacon_data"
soc.send(clients_input.encode()) # we must encode the string to bytes
result_bytes = soc.recv(32768) # the number means how the response can be in bytes
result_string = result_bytes.decode("utf8") # the return will be in bytes, so decode

Method recv() does not guarantee receiving the full message in the first call so you have to try getting the full message by calling recv() multiple times.
If recv() does return an empty string, connection is closed in the client side.
Using this while loop you can get full stream from client into data:
data = b'' # recv() does return bytes
while True:
try:
chunk = conn.recv(4096) # some 2^n number
if not chunk: # chunk == ''
break
data += chunk
except socket.error:
conn.close()
break

TCP is a streaming protocol, meaning it has no concept of what constitutes a complete message. You have to implement your own message protocol layer on top of TCP to make sure you send and receive complete messages. You are responsible for buffering data received until you have a complete message, and you have to define what a complete message is. Some options:
Send fixed length messages.
Send a fixed number of bytes representing the length of the message, then the message.
Separate messages with a sentinel byte.
Then, call recv and accumulate the results until you have a complete message in the buffer.

Transferring file between client and server (Socket error and Attribute error)

I’m trying to send and receive file through TCP socket
There are a lot of problems
1. When the client connects to the server. The server does not “print Client connected ..” but it prints after using the command.
2. When I use the ‘put’ command at the server occur an error socket.error: [Errno 107] Transport endpoint is not connected but the file image is already uploaded to the server.
3. When I use the ‘get’ command at the client. I can’t continue to use another command.
4. The last problem is the client can’t quit and list file from the server. It shows AttributeError: 'module' object has no attribute 'send'
Server
import socket
import sys
import os
HOST = 'localhost'
PORT = 3820
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.bind((HOST, PORT))
socket.listen(1)
while (1):
conn, addr = socket.accept()
print 'Client connected ..'
reqCommand = conn.recv(2048)
print 'Client> %s' %(reqCommand)
if (reqCommand == 'quit'):
break
#list file on server
elif (reqCommand == 'lls'):
start_path = os.listdir('.') # server directory
for path,dirs,files in os.walk(start_path):
for filename in files:
print os.path.join(filename)
else:
string = reqCommand.split(' ', 1) #in case of 'put' and 'get' method
reqFile = string[1]
if (string[0] == 'put'):
with open(reqFile, 'wb') as file_to_write:
while True:
data = socket.recv(1024)
# print data
if not data:
break
# print data
file_to_write.write(data)
file_to_write.close()
break
print 'Receive Successful'
elif (string[0] == 'get'):
with open(reqFile, 'rb') as file_to_send:
for data in file_to_send:
conn.sendall(data)
print 'Send Successful'
conn.close()
socket.close()
Client
import socket
import sys
import os
HOST = 'localhost'
PORT = 3820
def put(commandName):
socket1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket1.connect((HOST, PORT))
socket1.send(commandName)
string = commandName.split(' ', 1)
inputFile = string[1]
with open(inputFile, 'rb') as file_to_send:
for data in file_to_send:
socket1.sendall(data)
print 'PUT Successful'
socket1.close()
return
def get(commandName):
socket1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket1.connect((HOST, PORT))
socket1.send(commandName)
string = commandName.split(' ', 1)
inputFile = string[1]
with open(inputFile, 'wb') as file_to_write:
while True:
data = socket1.recv(2048)
# print data
if not data:
break
# print data
file_to_write.write(data)
file_to_write.close()
print 'GET Successful'
socket1.close()
return
msg = raw_input('Enter your name: ')
while(1):
print 'Instructions'
print '"put [filename]" to upload the file to the server '
print '"get [filename]" to download the file from the server '
print '"ls" to list all files in the client'
print '"lls" to list all files in the server'
print '"quit" to exit'
sys.stdout.write('%s> ' % msg)
inputCommand = sys.stdin.readline().strip()
if (inputCommand == 'quit'):
socket.send('quit')
break
#list file on client
elif (inputCommand == 'ls'):
start_path = os.listdir('.') # client directory
print start_path
#list file on server
elif (inputCommand == 'lls'):
socket.send('lls')
else:
string = inputCommand.split(' ', 1)
if (string[0] == 'put'):
put(inputCommand)
elif (string[0] == 'get'):
get(inputCommand)
socket.close()

Among other things, you need to add "framing" to your transfer protocol. When you do a send on a stream socket, the data gets added to a buffer that will eventually be delivered to the other side. However, the size of that buffer is not transmitted to the other side. In other words, say you send an initial segment with the command "put myfile.txt". Then you send the data from myfile.txt. Now because you are using the file object iterator (for data in file_to_send), you are actually sending it a line at a time (arguably, for a file transfer protocol, it would make more sense to read and send fixed chunks but this would work too). Let's assume the first line of myfile.txt is "The quick brown fox\n"
When the server does its first receive, it could receive "put " or "put myfile.txt" or "put myfile.txtThe quick brown fox\n" or the put command plus the entire file contents. That's because the stream protocol (TCP) does not maintain message boundaries for you.
Now in practice, you may be receiving only the "put myfile.txt" in the first receive, but it's very unwise to count on that because it's dependent on the timing of all sorts of factors on both sending and receiving systems that are outside your control.
So, there are two common ways of handling this:
Add a length at the beginning that delineates the size of the command and any command argument (so that you know where in the stream the actual file data to be transferred begins). (Most binary file transfer protocols work this way.)
Add some known delimiter at the end of your command -- for example, '\n'. HTTP, for example, works this way.
And likewise your receiving side needs to ensure that it is reading exactly the amount needed at each point in order to preserve the full content of the file being transferred.
That means you either (a) are careful to recv exactly the number of bytes you need for the command data, then separately process the file content, or (b) recv an initial chunk of data into a buffer, then carve off exactly what you need for the "command", and ensure the rest will be processed later as file data. Option (b) can often be accomplished by building a file object on top of the socket (see socket.makefile), then using the file object's readline method to obtain only the first line.
Other issues with your code:
The line socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) is a very bad idea. You have just hidden the entire socket module by creating a variable with the same name. For example, if you attempted to refer to socket.AF_INET again on the subsequent line, you'd get the exception AttributeError: '_socketobject' object has no attribute 'AF_INET'. You should name the variable something else, say socket1 as you did on the client side.
On the client side, you have the opposite problem. You're attempting to use a socket object method, but providing the socket module object instead. (That's why you get AttributeError: 'module' object has no attribute 'send'.) You need to reorganize your code so that you call the send method on a connected socket object as you are doing in the put and get functions.
The error socket.error: [Errno 107] Transport endpoint is not connected occurs because you are attempting to recv on the listening socket, not the connected one (conn -- which is returned by socket.accept). The only thing you can do with a listening socket, is accept new connections (or close).
You should be using sendall instead of send to ensure that every byte gets sent. Generally, all the data will get sent with send too, but there are corner cases where that doesn't happen.
Your file-receiving loop in the server begins with while True: but then always breaks. Hence it will only receive the first chunk of data (up to 1024 bytes). If the file is larger than that, you will certainly end up truncating it.
Your server file listing function (lls command) is not sending its output back to the client, it's only printing it to the server side's console. (And don't forget, you will need to provide a delimiter of some kind between the file names you send back or they will end up all concatenated into a single big string.)

TCP sockets unable to send messages in a burst

Hi I have multiple systems communicating via message using TCP connections.
My send function looks like the following
def _send(self, message, dest):
self.sendLock.acquire()
message = pickle.dumps(message)
#sending length
message_length = len(message)
self.outChan[dest].send('<MESSAGELENGTH>%s</MESSAGELENGTH>'
% str(message_length))
for message_i in range(0, message_length, 1024):
self.outChan[dest].send(message[:1024])
message = message[1024:]
self.sendLock.release()
And the receive thread looks like this:
def readlines(self, sock):
while True:
msg = ''
opTag = '<MESSAGELENGTH>'
clTag = '</MESSAGELENGTH>'
while not all(tag in msg for tag in (opTag, clTag)):
msg = sock.recv(1024)
msglen = int(msg.split(clTag)[0].split(opTag)[1])
msg = msg.split(clTag)[1]
while len(msg) < msglen:
msg += sock.recv(msglen-len(msg))
self.rec.put(pickle.loads(msg))
After the message is read from self.rec a confirmation message is sent to the sender.
I have implemented my own buffer to control the traffic in the network. At any moment I would have sent atmost MAX_BUFFER_SIZE messages with no confirmation received.
Here is the problem: When program starts, it sends MAX_BUFFER_SIZE messages without waiting for the confirmation. But only a few of these MAX_BUFFER_SIZE messages are received.
In one of the simulations with MAX_BUFFER_SIZE = 5, total 100 messages were sent and m2,m3 and m4 were not received. All other messages were received (in the order they were sent).
I doubt the error is in the initial sending burst, but I am unable to figure out the exact problem.

There are a few errors in the receive thread:
While inspecting the received message for the opening and closing tags, you are not appending to the already received part, but overwriting it.
After detecting the message length, you are losing the subsequent messages, that have their closing tag already received, but not analyzed yet.
You are possibly putting several messages together into self.rec.
Here is a corrected form, with comments explaining the changes:
def readlines(self, sock):
msg = '' # initialize outside since otherwise remiander of previous message would be lost
opTag = '<MESSAGELENGTH>' # no need to repeat this in each iteration
clTag = '</MESSAGELENGTH>' # no need to repeat this in each iteration
while True:
while not all(tag in msg for tag in (opTag, clTag)):
msg += sock.recv(1024) # += rather than =
msglen = int(msg.split(clTag)[0].split(opTag)[1])
msg = msg.split(clTag, 1)[1] # split just once, starting from the left
while len(msg) < msglen:
msg += sock.recv(msglen-len(msg))
self.rec.put(pickle.loads(msg[:maglen])) # handle just one message
msg = msg[msglen:] # prepare for handling future messages

Python Socket Receive Large Amount of Data

When I try to receive larger amounts of data it gets cut off and I have to press enter to get the rest of the data. At first I was able to increase it a little bit but it still won't receive all of it. As you can see I have increased the buffer on the conn.recv() but it still doesn't get all of the data. It cuts it off at a certain point. I have to press enter on my raw_input in order to receive the rest of the data. Is there anyway I can get all of the data at once? Here's the code.
port = 7777
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('0.0.0.0', port))
sock.listen(1)
print ("Listening on port: "+str(port))
while 1:
conn, sock_addr = sock.accept()
print "accepted connection from", sock_addr
while 1:
command = raw_input('shell> ')
conn.send(command)
data = conn.recv(8000)
if not data: break
print data,
conn.close()

TCP/IP is a stream-based protocol, not a message-based protocol. There's no guarantee that every send() call by one peer results in a single recv() call by the other peer receiving the exact data sent—it might receive the data piece-meal, split across multiple recv() calls, due to packet fragmentation.
You need to define your own message-based protocol on top of TCP in order to differentiate message boundaries. Then, to read a message, you continue to call recv() until you've read an entire message or an error occurs.
One simple way of sending a message is to prefix each message with its length. Then to read a message, you first read the length, then you read that many bytes. Here's how you might do that:
def send_msg(sock, msg):
# Prefix each message with a 4-byte length (network byte order)
msg = struct.pack('>I', len(msg)) + msg
sock.sendall(msg)
def recv_msg(sock):
# Read message length and unpack it into an integer
raw_msglen = recvall(sock, 4)
if not raw_msglen:
return None
msglen = struct.unpack('>I', raw_msglen)[0]
# Read the message data
return recvall(sock, msglen)
def recvall(sock, n):
# Helper function to recv n bytes or return None if EOF is hit
data = bytearray()
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data.extend(packet)
return data
Then you can use the send_msg and recv_msg functions to send and receive whole messages, and they won't have any problems with packets being split or coalesced on the network level.

You can use it as: data = recvall(sock)
def recvall(sock):
BUFF_SIZE = 4096 # 4 KiB
data = b''
while True:
part = sock.recv(BUFF_SIZE)
data += part
if len(part) < BUFF_SIZE:
# either 0 or end of data
break
return data

The accepted answer is fine but it will be really slow with big files -string is an immutable class this means more objects are created every time you use the + sign, using list as a stack structure will be more efficient.
This should work better
while True:
chunk = s.recv(10000)
if not chunk:
break
fragments.append(chunk)
print "".join(fragments)

Most of the answers describe some sort of recvall() method. If your bottleneck when receiving data is creating the byte array in a for loop, I benchmarked three approaches of allocating the received data in the recvall() method:
Byte string method:
arr = b''
while len(arr) < msg_len:
arr += sock.recv(max_msg_size)
List method:
fragments = []
while True:
chunk = sock.recv(max_msg_size)
if not chunk:
break
fragments.append(chunk)
arr = b''.join(fragments)
Pre-allocated bytearray method:
arr = bytearray(msg_len)
pos = 0
while pos < msg_len:
arr[pos:pos+max_msg_size] = sock.recv(max_msg_size)
pos += max_msg_size
Results:

You may need to call conn.recv() multiple times to receive all the data. Calling it a single time is not guaranteed to bring in all the data that was sent, due to the fact that TCP streams don't maintain frame boundaries (i.e. they only work as a stream of raw bytes, not a structured stream of messages).
See this answer for another description of the issue.
Note that this means you need some way of knowing when you have received all of the data. If the sender will always send exactly 8000 bytes, you could count the number of bytes you have received so far and subtract that from 8000 to know how many are left to receive; if the data is variable-sized, there are various other methods that can be used, such as having the sender send a number-of-bytes header before sending the message, or if it's ASCII text that is being sent you could look for a newline or NUL character.

Disclaimer: There are very rare cases in which you really need to do this. If possible use an existing application layer protocol or define your own eg. precede each message with a fixed length integer indicating the length of data that follows or terminate each message with a '\n' character. (Adam Rosenfield's answer does a really good job at explaining that)
With that said, there is a way to read all of the data available on a socket. However, it is a bad idea to rely on this kind of communication as it introduces the risk of loosing data. Use this solution with extreme caution and only after reading the explanation below.
def recvall(sock):
BUFF_SIZE = 4096
data = bytearray()
while True:
packet = sock.recv(BUFF_SIZE)
if not packet: # Important!!
break
data.extend(packet)
return data
Now the if not packet: line is absolutely critical!
Many answers here suggested using a condition like if len(packet) < BUFF_SIZE: which is broken and will most likely cause you to close your connection prematurely and loose data. It wrongly assumes that one send on one end of a TCP socket corresponds to one receive of sent number of bytes on the other end. It does not. There is a very good chance that sock.recv(BUFF_SIZE) will return a chunk smaller than BUFF_SIZE even if there's still data waiting to be received. There is a good explanation of the issue here and here.
By using the above solution you are still risking data loss if the other end of the connection is writing data slower than you are reading. You may just simply consume all data on your end and exit when more is on the way. There are ways around it that require the use of concurrent programming, but that's another topic of its own.

A variation using a generator function (which I consider more pythonic):
def recvall(sock, buffer_size=4096):
buf = sock.recv(buffer_size)
while buf:
yield buf
if len(buf) < buffer_size: break
buf = sock.recv(buffer_size)
# ...
with socket.create_connection((host, port)) as sock:
sock.sendall(command)
response = b''.join(recvall(sock))

You can do it using Serialization
from socket import *
from json import dumps, loads
def recvall(conn):
data = ""
while True:
try:
data = conn.recv(1024)
return json.loads(data)
except ValueError:
continue
def sendall(conn):
conn.sendall(json.dumps(data))
NOTE: If you want to shara a file using code above you need to encode / decode it into base64

I think this question has been pretty well answered, but I just wanted to add a method using Python 3.8 and the new assignment expression (walrus operator) since it is stylistically simple.
import socket
host = "127.0.0.1"
port = 31337
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen()
con, addr = s.accept()
msg_list = []
while (walrus_msg := con.recv(3)) != b'\r\n':
msg_list.append(walrus_msg)
print(msg_list)
In this case, 3 bytes are received from the socket and immediately assigned to walrus_msg. Once the socket receives a b'\r\n' it breaks the loop. walrus_msg are added to a msg_list and printed after the loop breaks. This script is basic but was tested and works with a telnet session.
NOTE: The parenthesis around the (walrus_msg := con.recv(3)) are needed. Without this, while walrus_msg := con.recv(3) != b'\r\n': evaluates walrus_msg to True instead of the actual data on the socket.

Modifying Adam Rosenfield's code:
import sys
def send_msg(sock, msg):
size_of_package = sys.getsizeof(msg)
package = str(size_of_package)+":"+ msg #Create our package size,":",message
sock.sendall(package)
def recv_msg(sock):
try:
header = sock.recv(2)#Magic, small number to begin with.
while ":" not in header:
header += sock.recv(2) #Keep looping, picking up two bytes each time
size_of_package, separator, message_fragment = header.partition(":")
message = sock.recv(int(size_of_package))
full_message = message_fragment + message
return full_message
except OverflowError:
return "OverflowError."
except:
print "Unexpected error:", sys.exc_info()[0]
raise
I would, however, heavily encourage using the original approach.

For anyone else who's looking for an answer in cases where you don't know the length of the packet prior.
Here's a simple solution that reads 4096 bytes at a time and stops when less than 4096 bytes were received. However, it will not work in cases where the total length of the packet received is exactly 4096 bytes - then it will call recv() again and hang.
def recvall(sock):
data = b''
bufsize = 4096
while True:
packet = sock.recv(bufsize)
data += packet
if len(packet) < bufsize:
break
return data

This code reads 1024*32(=32768) bytes in 32 iterations from the buffer which is received from Server in socket programming-python:
jsonString = bytearray()
for _ in range(32):
packet = clisocket.recv(1024)
if not packet:
break
jsonString.extend(packet)
Data resides in jsonString variable

Plain and simple:
data = b''
while True:
data_chunk = client_socket.recv(1024)
if data_chunk:
data+=data_chunk
else:
break