Develop Reference

send a json containing a big buffer (bytearray) through sockets: gets truncated

I'm trying to send a json containing text fields and a buffer in a bytearray, from a micro-controller to a Windows server
msg = {"some_stuff": "some_stuff", "buf": bytearray(b'\xfe\xc2\xf1\xfe\xd5\xc0 ...')}
Note that the buffer is quite long (so that I can't put it here as reference) len(buf) -> 35973
I'm sending the length of the message before to the server so that it knows how long is the message to be received
def send_json(conn, msg):
msg = json.dumps(msg).encode('utf-8')
msg_length = len(msg)
header = str(msg_length).encode('utf-8')
header += b' ' * (64 - len(header))
conn.send(header)
conn.send(msg)
The receiving function is then
def receive_json(conn) -> dict:
msg_length = int(
conn.recv(64).decode('utf-8').replace(' ', '')
)
msg_b = conn.recv(msg_length)
msg_s = msg_b.decode('utf-8')
try:
msg_d = json.loads(msg_s)
except:
msg_d = eval(msg_s)
return msg_d
The problem is that the received message is truncated.
msg_b = b'{"buf": bytearray(b\'\\xfe\\xc2\\xf1 ... \\x06u\\xd0\\xff\\xb'
It's worth mentioning that while in debug, if I stop for a while with a breakpoint on line msg_b = conn.recv(msg_length), before running it, the received message is complete.
So it seems that in the receiving function the conn.recv(msg_length) instruction does not wait to receive a message of the specified length (msg_length)
Why is it the case? What can I do to receive a complete message?
I could introduce time.sleep between receiving the length of the message and the message, but how to know how much to wait depending on the message length?
Thank you

My solution was to check for how much of the message is missing and iterate till the message is complete
def receive_json(conn) -> dict:
msg_length = int(
conn.recv(64).decode('utf-8').replace(' ', '')
)
buf = bytearray(b'')
while len(buf) < msg_length:
missing_length = msg_length - len(buf)
packet = conn.recv(missing_length)
buf.extend(packet)
msg_s = buf.decode('utf-8')
try:
msg_d = json.loads(msg_s)
except:
msg_d = eval(msg_s)
return msg_d

TCP is a streaming protocol that guarantees delivery of bytes in the order sent, but not with the same send breaks. You need to define a protocol (which you have, as a 64-byte header of message size, then the message data), and then buffer reads until you have a complete message.
Python sockets have a .makefile method that handles the buffering for you, where you can .read(n) a specific number of bytes or .readline() to read a newline-terminated line. With this you can implement the following client and server:
server.py
import socket
import json
import time
s = socket.socket()
s.bind(('',5000))
s.listen()
while True:
c,a = s.accept()
print(f'{a} connected')
# wrap socket in a file-like buffer
with c, c.makefile('rb') as r: # read binary so .read(n) gets n bytes
while True:
header = r.readline() # read header up to a newline
if not header: break # if empty string, client closed connection
size = int(header)
data = json.loads(r.read(size)) # read exactly "size" bytes and decode JSON
print(f'{a}: {data}')
print(f'{a} disconnected')
client.py
import socket
import json
def send_json(conn, msg):
# smaller data size if non-ASCII used.
data = json.dumps(msg, ensure_ascii=False).encode()
msg_length = len(data) # length in encoded bytes
# send newline-terminated header, then data
conn.sendall(f'{msg_length}\n'.encode())
conn.sendall(data)
s = socket.socket()
s.connect(('localhost',5000))
with s:
send_json(s, {'name':'马克'}) # check to support non-ASCII properly
send_json(s, [1,2,3])
Start server.py, then run client.py a couple of times:
Output:
('127.0.0.1', 26013) connected
('127.0.0.1', 26013): {'name': '马克'}
('127.0.0.1', 26013): [1, 2, 3]
('127.0.0.1', 26013) disconnected
('127.0.0.1', 26015) connected
('127.0.0.1', 26015): {'name': '马克'}
('127.0.0.1', 26015): [1, 2, 3]
('127.0.0.1', 26015) disconnected

Using a custom socket recvall function works only, if thread is put to sleep

I have the following socket listening on my local network:
def recvall(sock):
BUFF_SIZE = 4096 # 4 KiB
fragments = []
while True:
chunk = sock.recv(BUFF_SIZE)
fragments.append(chunk)
# if the following line is removed, data is omitted
time.sleep(0.005)
if len(chunk) < BUFF_SIZE:
break
data = b''.join(fragments)
return data
def main():
pcd = o3d.geometry.PointCloud()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('192.168.0.22', 2525))
print("starting listening...")
s.listen(1)
counter = 0
while True:
clientsocket, address = s.accept()
print(f"Connection from {address} has been established!")
received_data = recvall(clientsocket)
clientsocket.send(bytes(f"response nr {counter}!", "utf-8"))
counter += 1
print(len(received_data))
if __name__ == "__main__":
main()
To this port, I'm sending byte data with a length of 172800 bytes from an app on my mobile phone.
As one can see, I'm printing the amount of data received. The amount is only correct, if I use time.sleep() as shown in the code above. If I don't use this method, only a part of the data is received.
Obviously this is some timing issue, the question is: How can I be sure to receive all the data all the time without using time.sleep() (since this is also not 100% certain to work, depending on the sleeping time set)

sock.recv() returns the data that is available. The relevant piece from the man page of recv(2) is:
The receive calls normally return any data available, up to the requested amount,
rather than waiting for receipt of the full amount requested.
In your case, time.sleep(0.005) seems to allow for all the remaining data of the message to arrive and be stored in the buffer.
There are some options to eliminate the need for time.sleep(0.005). Which one is the most appropriate depends on your needs.
If the sender sends data, but does not expect a response, you can have the sender close the socket after it sends the data, i.e., sock.close() after sock.sendall(). recv() will return an empty string that can be used to break out of the while loop on the receiver.
def recvall(sock):
BUFF_SIZE = 4096
fragments = []
while True:
chunk = sock.recv(BUFF_SIZE)
if not chunk:
break
fragments.append(chunk)
return b''.join(fragments)
If the sender sends messages of fixed length, e.g., 172800 bytes, you can use recv() in a loop until the receiver receives an entire message.
def recvall(sock, length=172800):
fragments = []
while length:
chunk = sock.recv(length)
if not chunk:
raise EOFError('socket closed')
length -= len(chunk)
fragments.append(chunk)
return b''.join(fragments)
Other options include a. adding a delimiter, e.g., a special character that cannot be part of the data, at the end of the messages that the sender sends; the receiver can then run recv() in a loop until it detects the delimiter and b. prefixing the messages on the sender with their length; the receiver will then know how many bytes to expect for each message.

Python Server-Client Lottery Issue

I am have an issue regarding a python assignment I was doing and would like to ask the community for some guidance. I am to use the socket module and argparse modules only for this assignment. I have created a socketserver.py file and a socketclient.py file. The connection between the server and client is fine. Now the purpose of the assignment is that the client is to send the Lottery game type, # of tickets, and # of lines per tickets using argparse. E.g. syntax for socketclient.py would be python3 -t Lotto_Max -n 2 -l 2. The output for the ticket game, ticket type and number of lines per ticket show up correctly on the server. However, they sometimes don't show correctly on the client, and am really stuck at the moment. Here is my following code....
Server Code
```socketserver.py code```
import socket
from random import sample
def main():
host = input("Please specify an IP address for this server's socket\t")
port = int(input('Please speciy a port number above 1024 for this socket.\t'))
kulmiye = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
kulmiye.bind((host, port))
except socket.error as egeh:
print('Socket bind failed. Error Code : ' + str(egeh[0]) + ' Message ' + egeh[1])
print('Socket bind accomplished...\n')
print("Listening for an incoming connection...")
kulmiye.listen()
conn, addr = kulmiye.accept()
print('Connected with ' + addr[0] + ':' + str(addr[1]))
while True:
server_data = conn.recv(1024).decode("utf-8")
game_data = server_data.split(",")
if not server_data:
break
if game_data[0] == "Lotto_Max":
nval = int(game_data[1])
lval = int(game_data[2])
for nval in range(nval):
for i in range(lval):
numbers = sample(range(1, 50), 7)
numbers.sort()
sortedd = str(numbers)
print(sortedd)
print("--------------------")
conn.sendall(sortedd.encode("utf-8"))
#conn.sendall(bytes(str(numbers),'utf-8'))
liners = "-----------------------"
conn.sendall(liners.encode("utf-8"))
print("From Client: " + str(game_data))
conn.sendall(b'goodbye')
# server_data = input('#\t')
break
else:
conn.close()
if __name__ == '__main__':
main()
Client Code
```socketclient.py code```
import socket
import argparse
def client():
host = input("Please specify the server's IP you want to connect to\t")
port = int(input("Please specify the server's port you want to connect to\t"))
kulmiye = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
kulmiye.connect((host, port))
# client_data = input("#\t")
# while True:
# client_data = input()
# if client_data == 'quit':
# kulmiye.close()
# sys.exit()
# if len(str.encode(client_data)) > 0:
# kulmiye.sendall(str.encode(client_data))
# server_response = str(kulmiye.recv(1024), "utf-8")
# print(server_response, end = " ")
kulmiye.sendall(bytes(tvar.encode("utf-8")))
kulmiye.sendall(bytes(','.encode("utf-8")))
kulmiye.sendall(bytes(str(nval).encode("utf-8")))
kulmiye.sendall(bytes(','.encode("utf-8")))
kulmiye.sendall(bytes(str(lval).encode("utf-8")))
server_data = kulmiye.recv(1024).decode("utf-8")
while server_data != 'goodbye':
server_data = kulmiye.recv(1024).decode("utf-8")
print('Server: \n' + server_data)
# client_data = input("#\t")
if not server_data:
break
kulmiye.close()
# this code block serves to give the user the ability to play lotto max
# with the amount of tickets and lines per ticket they would like
# Using the argparse module to allow the user to input command-line interfaces
parser = argparse.ArgumentParser(description='Welcome to OLG Gaming.')
parser.add_argument(
'-t',
type=str,
help="Pick the lottery you want to play",
required=True)
parser.add_argument(
'-n',
type=int,
help="Pick the amount of lottery tickets you want to play",
required=True)
parser.add_argument(
'-l',
type=int,
help="Pick the amount of lines you would like to play",
required=True)
# parser.add_argument('-o', type = str, help = "This is optional", required = False)
# parse_args will convert the argument strings into objects and will get
# stored in the cmdargs variable
cmdargs = parser.parse_args()
tvar = cmdargs.t # the t string argument that gets parsed into an object will get stored into a variable called tvar
# the n integer argument that gets parsed into an object will get stored
# into a variable called nval
nval = int(cmdargs.n)
# the l integer argument that gets parsed into an object will get stored
# into a variable called lval
lval = int(cmdargs.l)
if __name__ == "__main__":
client()
```code```
Server
python3 socketserver.py
specify localhost as IP
specify a port e.g. 4444
Client
python3 socketclient.py -t Lotto_Max -n 1 -l 1
specify an IP address to connect to the server (e.g. localhost or 127.0.0.1)
specify a port to connect to e.g. 4444
When the connection establishes between client and server, the server receives the client input and prints it on its end the gametype (Lotto_Max), number of tickets and lines per ticket
Server will output the resultse.g.
However, the client won't receive it indefinitely. Usually it'll get it about 25% of the time, and I am not sure why

One problem is here in the server:
server_data = conn.recv(1024).decode("utf-8")
conn.recv(1024) can receive any number of bytes from 0 (closed connection) to 1024. The line above assumes the whole message is received every time, but when it fails it only gets part of the message. Below I've modified your code so the server can process multiple client connections one at a time, and the client will connect/disconnect over and over again to hasten the failure:
Server:
import socket
from random import sample
def main():
host = ''
port = 4444
kulmiye = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
kulmiye.bind((host, port))
print('Socket bind accomplished...\n')
print("Listening for an incoming connection...")
kulmiye.listen()
while True:
conn, addr = kulmiye.accept()
print('Connected with ' + addr[0] + ':' + str(addr[1]))
with conn:
server_data = conn.recv(1024).decode("utf-8")
print(f'server_data={server_data}')
game_data = server_data.split(",")
print("From Client: " + str(game_data))
if server_data:
if game_data[0] == "Lotto_Max":
nval = int(game_data[1])
lval = int(game_data[2])
for nval in range(nval):
for i in range(lval):
numbers = sample(range(1, 50), 7)
numbers.sort()
sortedd = str(numbers)
print(sortedd)
print("--------------------")
conn.sendall(sortedd.encode("utf-8"))
liners = "-----------------------"
conn.sendall(liners.encode("utf-8"))
conn.sendall(b'goodbye')
if __name__ == '__main__':
main()
Client:
import socket
import argparse
def client():
host = 'localhost'
port = 4444
kulmiye = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
kulmiye.connect((host, port))
kulmiye.sendall(bytes(tvar.encode("utf-8")))
kulmiye.sendall(bytes(','.encode("utf-8")))
kulmiye.sendall(bytes(str(nval).encode("utf-8")))
kulmiye.sendall(bytes(','.encode("utf-8")))
kulmiye.sendall(bytes(str(lval).encode("utf-8")))
server_data = kulmiye.recv(1024).decode("utf-8")
while server_data != 'goodbye':
server_data = kulmiye.recv(1024).decode("utf-8")
print('Server: \n' + server_data)
# client_data = input("#\t")
if not server_data:
break
kulmiye.close()
tvar = 'Lotto_Max'
nval = 1
lval = 1
if __name__ == "__main__":
while True:
client()
Here's the result after 100s of successful connections. Note server_data between the successful and unsuccessful connection:
Connected with 127.0.0.1:12175
server_data=Lotto_Max,1,1
From Client: ['Lotto_Max', '1', '1']
[4, 7, 9, 12, 24, 31, 48]
--------------------
Connected with 127.0.0.1:12176
server_data=Lotto_Max,1,
From Client: ['Lotto_Max', '1', '']
Traceback (most recent call last):
File "C:\server.py", line 38, in <module>
main()
File "C:\server.py", line 24, in main
lval = int(game_data[2])
ValueError: invalid literal for int() with base 10: ''
conn.recv(1024) didn't receive the complete message (didn't get the final 1). TCP is a byte streaming protocol with no concept of message boundaries. You're code is responsible to call recv() and buffer the data until you have a complete message. You could send fixed-sized messages and call recv until you have that many bytes, terminate a message with a sentinel byte such as a newline(\n) and read until the sentinel is received, or send the size of the message first followed by the message bytes.
I won't solve it for you since it is an assignment, but as a hint socket.makefile can wrap the socket in a file-like object where you can call .read(n) to read exactly n bytes from a socket, or .readline() to read a \n-terminated line.
Here's a link to an answer of mine that demonstrates this: https://stackoverflow.com/a/55840026/235698
Another example of the streaming nature of TCP is on the client side. Sometimes, it prints:
-----------------------
Server:
goodbye
Server:
-----------------------goodbye
Server:
Server:
-----------------------
Server:
goodbye
The separate lines in the server:
conn.sendall(liners.encode("utf-8"))
conn.sendall(b'goodbye')
Sometimes get received by the single recv in the client:
server_data = kulmiye.recv(1024).decode("utf-8")
Again, this is due to the streaming nature of TCP and no message boundaries. If you sent each line terminated by a newline, you could call recv() a number of times until a newline was detected, then process just that line.
Another answer that illustrates this technique is here: https://stackoverflow.com/a/55186805/235698.

Thank you Mark Tolonen for your guidance. I still need to brush up how to handle TCP data stream. Nevertheless, I was able to achieve more desirable results using numerous
conn.sendall within my nested for loop. Happy camper!

IPID field unexpected increase

Intro:
I have an exercise where I need to send a file from S to D through a third party T only.
T is run on port 10000 or 11000 depending if i use UDP or TCP; i use UDP, and T's ip is given both to S and D.
T is given and all it does is echo messages it got back to the sender.
One of the fields in the header of T's echo message is called ip_id - a counter that goes up by one with each message T receives; in order to view the ip_id value i need to use raw sockets.
S/D sends a message to T and is then supposed to receive back a message. S/D needs to check that the ip and port that it got the message from matches the one it sent to (that is, if S/D sends a message to 1.1.1.1:5 it should receive a message from 1.1.1.1:5).
First I open a socket and a raw socket once
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.raw_socket = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_UDP)
I then get my port by doing:
self.s.sendto("1", (self.remote_server, UDP_PORT))
self.myPort = self.s.getsockname()[1]
UDP_PORT is 10000 and is the port to send messages to.
remote_server is the ip to send messages to.
I also have the following method:
def send_value(self, number_of_packets):
for i in range(0,number_of_packets):
self.s.sendto(BOGUS_DATA, (self.remote_server, UDP_PORT))
self.raw_socket.recvfrom(1024)
#self.raw_socket.recv(1024)
And as well as this one:
def recieve_ip_id(self):
try:
sent = 0
continueFlag = True
while continueFlag:
self.s.sendto(BOGUS_DATA, (self.remote_server, UDP_PORT))
#mypacket = self.raw_socket.recv(1024)
mypacket = self.raw_socket.recvfrom(1024)
mypacket = mypacket[0]
continueFlag = False
if (256*ord(mypacket[20])+ord(mypacket[21]) != UDP_PORT):
continueFlag = True
if (256*ord(mypacket[22])+ord(mypacket[23]) != self.myPort):
continueFlag = True
sent = sent + 1
ip_id = 256 * ord(mypacket[4]) + ord(mypacket[5])
print "packet 20 21 " + str(256*ord(mypacket[20])+ord(mypacket[21])) + " packet 4 5 " + str(256 * ord(mypacket[4]) + ord(mypacket[5])) + " packet 22 23 " + str(256*ord(mypacket[22])+ord(mypacket[23]))
print ip_id
return (ip_id, 0)
except socket.timeout:
# dummy
return (0, 0)
It works good enough in the sense that it checks correctly who it got the message from; The problem is that the ip_id wont advance as expected. after some time the programs are running they each get ip_id as if it was not shared anymore.
am i not checking something correctly, or is it something else?
Thanks.

How to receive data from a socket, process, and send back data in python?

i have the following code:
import socket # Import socket module
import sys
s = socket.socket() # Create a socket object
host = '' # Get local machine name
port = 1234 # Reserve a port for your service.
s.connect((host, port))
while 1:
data = s.recv(1024)
print ' data ' , data
d = data.split('?') # parsing values from server
if len(d) < 2:
# if does not contain ?, do nothing
continue
else:
a = d[0]
b = d[1].replace('\n', '')
# check how a compares to b, and send response accordingly
if (a > b):
s.send('1')
elif (a == b):
s.send('2')
else:
s.send('3')
s.close() # Close the socket when done
Without the processing code I have, it works fine if I just send a random value. But with the code above, I can only parse the first set of line, and then it stops. (I assume it closes the socket or something?)
The data coming from the socket looks like '1 ? 23' or '23 ? 1' , etc. it expects a response that determines how the two numbers relate.
In comparison, if I have this code:
import socket # Import socket module
import sys
s = socket.socket() # Create a socket object
host = '' # Get local machine name
port = 1234 # Reserve a port for your service.
s.connect((host, port))
backlog = ''
while 1:
data = s.recv(1024)
sp = data.split('\n')
if len(sp) < 2:
backlog += data
continue
line = backlog + sp[0]
backlog = sp[1]
data = line
print ' data ' , data
if not data:
break
s.send ('2')
s.close() # Close the socket when done
This code will yield a server response of either 'Correct!' or 'Incorrect...try again!' depending on whether it's right or wrong.

You seem to assume that you always get a full line with each read() call. That is wrong.
You should split your input into lines, and only if you have a full line, you proceed.
backlog = ''
while 1:
data = s.recv(1024)
# do we have a line break?
sp = data.split('\n')
if len(sp) < 2:
# no, we haven't...
backlog += data
continue
# yes, we have.
line = backlog + sp[0] # first part is the now complete line.
backlog = sp[1] # 2nd part is the start of the new line.
print ' line ' , line
d = line.split('?') # parsing values from server
if len(d) < 2:
# if does not contain ?, do nothing
continue
else:
a = int(d[0]) # we want to compare numbers, not strings.
b = int(d[1])
# check how a compares to b, and send response accordingly
if (a > b):
s.send('1')
elif (a == b):
s.send('2')
else:
s.send('3')
Try out what happens now.
Another question which occurs to me is what exactly does the server expect? Really only one byte? Or rather '1\n', '2\n', '3\n'?