Develop Reference

Python UDP socket receive - server.recvfrom package size longer than bufsize

I'm using the socket module for a UDP server。The incoming packets always have a different size(0-65535), so client send package length first, then send the package；server receives data according to the package length and 1024 per server receives。but server didn't work like I thought，it can receives package only once。
# -*- coding: utf-8 -*-
import json
import socket
ip_port = ('127.0.0.1', 8080)
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind(ip_port)
print('server listen......')
def recv_header(s_socket):
try:
msg, addr = s_socket.recvfrom(32)
return json.loads(msg), addr
except Exception as e:
return str(e), None
while True:
header_msg, client_addr = recv_header(s_socket=server)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
receiverBufsize = 1024
if client_addr is None:
print (header_msg)
continue
data_body = ""
package_len = int(header_msg['length'])
print ("package_len: {}".format(package_len))
print ("client_addr: {}".format(str(client_addr)))
while package_len > 0:
if package_len > receiverBufsize:
body_part = server.recvfrom(receiverBufsize)
else:
body_part = server.recvfrom(package_len)
data_body += body_part[0]
package_len -= receiverBufsize
print ("data_body: {}".format(data_body))
# -*- coding: utf-8 -*-
import json
import socket
import random
ip_port = ('127.0.0.1', 8080)
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def get_data(length):
base_str = 'ABCDEFGHIGKLMNOPQRSTUVWXYZabcdefghigklmnopqrstuvwxyz0123456789'
return ''.join([random.choice(base_str) for _ in range(length)])
while 1:
msg = raw_input('>>>：')
if msg == 'q':
break
if not msg:
continue
len = int(msg)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
data = get_data(len)
header = json.dumps({"length": len})
aa = client.sendto(header.encode('utf-8'), ip_port)
print aa
aa = client.sendto(data.encode('utf-8'), ip_port)
print aa
If the packet length is less than 1024(my receiverBufsize), it works normally
enter image description here
enter image description here
If the packet length is more than 1024(my receiverBufsize), socket.recvfrom will block
enter image description here
enter image description here

UDP is a message oriented protocol, not a data stream like TCP. This means in UDP a single send is matched by a single recv.
This means you cannot use multiple recv with a small buffer (size 1024 in your case) to get data which were sent in a single send with a larger buffer - you need to use recv with a large enough buffer instead.
This means also that you don't actually need to send the length upfront, since you don't need to add message semantics to a protocol which already has message semantic. In contrary, sending the length in a separate message as you do could actually be harmful since UDP is an unreliable protocol and you cannot rely on packets getting delivered in order, at all or only once.

How can I receive multiple lists form server with python socket?

I made a very simple server and client program to test. I need to send 3 different lists from server and client must recieve them seperately. Could you please help me ?
Here is server.py
import socket
import pickle
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am01=pickle.dumps(am1)
am02=pickle.dumps(am2)
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
#client.send(am01)
#client.send(am02)
mysocket.close()
here is client.py
import socket
import pickle
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?----------------
data1 = pickle.loads(s.recv(buffer_size))
print(data1)

the pickle protocol includes the length of the pickled buffer and you can use pickle.Unpickler to load one pickled object at a time. The problem is that it wants a file-like object, not a socket. Fortunately, sockets can make themselves look like files using socket.makefile. There are caveats, so reading the referenced doc is worthwhile.
Update your server to write each pickled object and then change the client to
import socket
import pickle
host = '127.0.0.1'
port = 8899
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
unpickler = pickle.Unpickler(s.makefile("rb"))
print(unpickler.load())
print(unpickler.load())
print(unpickler.load())

Here is a more universal method that also works with non pickled data (eg. receiving data from c socket), you first send in the data len and then the actual data in series to the socket, note that the while loop is there because you are not guaranteed to receive the specify number of bytes in a single call, if the data hasn't arrived yet you will receive less. struct.pack packs data you provided to the function into binary data in the order provided by first argument, 'H' represents unsigned short which is 2 bytes long (max integer you can pack into that is 65535)
Server:
import socket
import pickle, struct
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am00 = struct.pack("H", len(am00)) + am00
am01=pickle.dumps(am1)
am01 = struct.pack("H",len(am01)) +am01
am02=pickle.dumps(am2)
am02 = struct.pack("H", len(am02)) + am02
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
client.send(am01)
client.send(am02)
mysocket.close()
Client:
import socket
import pickle, struct
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?---------------
def recv_all(sock:socket.socket):
data_len = 0
buffer = b''
while len(buffer) < 2:
buffer += sock.recv(2-len(buffer))
data_len = struct.unpack("H", buffer)[0]
buffer = b''
while len(buffer) < data_len:
buffer += sock.recv(data_len-len(buffer))
return buffer
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)

How to receive and handle multiple TCP stream sockets?

I would like to send the location of a moving point to a server via TCP with the socket module. That point location is updated at each iteration of a for loop and is sent in the form of a tuple (x, y) that has been serialized with pickle dumps methods.
Problem:
On the server side, it seems that I only get to receive the location from the first iteration of that loop. As if all the following updated positions had been skipped or lost in the process.
I can’t say for sure what is the reason behind this behavior but my bet is that I am not correctly setting things on the server side. I suspect the data to be sent entirely but not processed adequately on reception due to some mistakes that I am probably doing with the socket module (I am completely new to the world of network interfaces).
Code:
--client side--
#Python3.7
import socket
import pickle
import math
HOST = "127.0.0.1"
PORT = 12000
den = 20
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(1000):
i = step%den
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = pickle.dumps((x, y), protocol=0)
sock.sendall(data)
--server side--
#Jython2.7
import pickle
import socket
HOST = "127.0.0.1"
PORT = 12000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
while True:
connection, address = s.accept()
if connection:
data = connection.recv(4096)
print(pickle.loads(data)) # <-- only print once (first location)

You need to put connection, address = s.accept() outside the while loop otherwise your server will wait for a new connection every time.
You also have an issue with the way your are receiving data. connection.recv(4096) will return any amount of bytes between 0 and 4096 not every time a complete "data" message is received. To handle this you could send a header before sending you json indicating how much data should be received
By adding a header, you will make sure the data messages you are sending will be received properly.
The header in this example is a four bytes int indicating the size of data.
Server
import pickle
import socket
import struct
HEADER_SIZE = 4
HOST = "127.0.0.1"
PORT = 12000
def receive(nb_bytes, conn):
# Ensure that exactly the desired amount of bytes is received
received = bytearray()
while len(received) < nb_bytes:
received += conn.recv(nb_bytes - len(received))
return received
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()
while True:
# receive header
header = receive(HEADER_SIZE, connection)
data_size = struct.unpack(">i", header)[0]
# receive data
data = receive(data_size, connection)
print(pickle.loads(data))
Client
import socket
import pickle
import math
HEADER_SIZE = 4
HOST = "127.0.0.1"
PORT = 12000
den = 20
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(1000):
i = step%den
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = pickle.dumps((x, y), protocol=0)
# compute header by taking the byte representation of the int
header = len(data).to_bytes(HEADER_SIZE, byteorder ='big')
sock.sendall(header + data)
Hope it helps

python 3.6 socket pickle data was truncated

I can not send my numpy array in socket. I use pickle but my client pickle crashes with this error: pickle data was truncated
My server :
I create a numpy array and I want to send in my client with pickle (it's work)
import socket, pickle
import numpy as np
from PIL import ImageGrab
import cv2
while(True):
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print ('Connected by', addr)
arr = np.array([[0, 1], [2, 3]])
printscreen_pil=ImageGrab.grab(bbox=(10,10,500,500))
img = np.array(printscreen_pil) ## Transform to Array
data_string = pickle.dumps(img)
conn.send(data_string)
msg_recu = conn.recv(4096)
print(msg_recu.decode())
conn.close()
My client
He has my numpy array, but I can not load with pickle. I have this error.
import socket, pickle
import numpy as np
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
msg_a_envoyer = "hello".encode()
s.send(msg_a_envoyer)
while 1:
data = s.recv(4096)
if not data: break
data_arr = pickle.loads(data)
print (data_arr)
s.close()

the problem is that if the size of the pickled data is > 4096 you only get the first part of the pickled data (hence the pickle data was truncated message you're getting)
You have to append the data and pickle it only when the reception is complete, for example like this:
data = b""
while True:
packet = s.recv(4096)
if not packet: break
data += packet
data_arr = pickle.loads(data)
print (data_arr)
s.close()
increasing a bytes object is not very performant, would be better to store the parts in a list of objects, then join, though. Faster variant:
data = []
while True:
packet = s.recv(4096)
if not packet: break
data.append(packet)
data_arr = pickle.loads(b"".join(data))
print (data_arr)
s.close()

In simple words, the file you are trying to load is not complete. Either you have not downloaded it correctly or it's just that your pickle file is corrupt. You can create a new pickle to solve this issue

how to send an array over a socket in python

I have an array kind of ([1,2,3,4,5,6],[1,2,3,4,5,6]) this. I have to send it over a STREAM/TCP socket in python. Then I have to receive the same array at the receiving end.

Sockets are byte streams, so ideal is to write your protocol (read this)
This is a basic example without protocol and you should care about buffer -> recv(). If it is too small, your data will be chopped off. That's why you should implement a protocol, if you send unknown size of data.
Client:
import socket, pickle
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
arr = ([1,2,3,4,5,6],[1,2,3,4,5,6])
data_string = pickle.dumps(arr)
s.send(data_string)
data = s.recv(4096)
data_arr = pickle.loads(data)
s.close()
print 'Received', repr(data_arr)
Server:
import socket
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(4096)
if not data: break
conn.send(data)
conn.close()

You can Serialize the object before sending to socket and at receiving end Deserialize it. Check this

I solved this problem using json (since I heard pickle is unsafe)
client:
import json
...
arr1 = [1,2,3]
arr2 = [4,5,6]
someVar = 7
data = json.dumps({"a": arr1, "b": arr2, "c": someVar})
socket.send(data.encode())
server:
import json
...
data = socket.recv(1024)
data = json.loads(data.decode())
arr = data.get("a")
var = data.get("c")
Here we deserialize the json string, using data.get("a") which you can interpret as data.a

I solved this issue by going over each item in the array, adding it to a single string, but with a significant character, such as a greek letter or some other uncommon character, then sending that string over the socket, then splitting the recieved string up back into an array on the other side, and removing all of the 'delimiter' items in the new array.
For Example, The Client side
for item in myArray:
print("item: ", item)
myArrayString= myArrayString+ str(item) + "Δ"
print(myArrayString)
myServer.send((myArrayString).encode())
and then on the Server:
files = myconnection.recv(50000)
files = files.decode()
myArray = files.split('Δ')
for myItem in myArray:
print(myItem)
print("End Of Items in Array")
hope this helps! feel free to ask if you need anything clarified! : )