I can not send my numpy array in socket. I use pickle but my client pickle crashes with this error: pickle data was truncated
My server :
I create a numpy array and I want to send in my client with pickle (it's work)
import socket, pickle
import numpy as np
from PIL import ImageGrab
import cv2
while(True):
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print ('Connected by', addr)
arr = np.array([[0, 1], [2, 3]])
printscreen_pil=ImageGrab.grab(bbox=(10,10,500,500))
img = np.array(printscreen_pil) ## Transform to Array
data_string = pickle.dumps(img)
conn.send(data_string)
msg_recu = conn.recv(4096)
print(msg_recu.decode())
conn.close()
My client
He has my numpy array, but I can not load with pickle. I have this error.
import socket, pickle
import numpy as np
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
msg_a_envoyer = "hello".encode()
s.send(msg_a_envoyer)
while 1:
data = s.recv(4096)
if not data: break
data_arr = pickle.loads(data)
print (data_arr)
s.close()
the problem is that if the size of the pickled data is > 4096 you only get the first part of the pickled data (hence the pickle data was truncated message you're getting)
You have to append the data and pickle it only when the reception is complete, for example like this:
data = b""
while True:
packet = s.recv(4096)
if not packet: break
data += packet
data_arr = pickle.loads(data)
print (data_arr)
s.close()
increasing a bytes object is not very performant, would be better to store the parts in a list of objects, then join, though. Faster variant:
data = []
while True:
packet = s.recv(4096)
if not packet: break
data.append(packet)
data_arr = pickle.loads(b"".join(data))
print (data_arr)
s.close()
In simple words, the file you are trying to load is not complete. Either you have not downloaded it correctly or it's just that your pickle file is corrupt. You can create a new pickle to solve this issue
Related
I made a very simple server and client program to test. I need to send 3 different lists from server and client must recieve them seperately. Could you please help me ?
Here is server.py
import socket
import pickle
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am01=pickle.dumps(am1)
am02=pickle.dumps(am2)
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
#client.send(am01)
#client.send(am02)
mysocket.close()
here is client.py
import socket
import pickle
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?----------------
data1 = pickle.loads(s.recv(buffer_size))
print(data1)
the pickle protocol includes the length of the pickled buffer and you can use pickle.Unpickler to load one pickled object at a time. The problem is that it wants a file-like object, not a socket. Fortunately, sockets can make themselves look like files using socket.makefile. There are caveats, so reading the referenced doc is worthwhile.
Update your server to write each pickled object and then change the client to
import socket
import pickle
host = '127.0.0.1'
port = 8899
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
unpickler = pickle.Unpickler(s.makefile("rb"))
print(unpickler.load())
print(unpickler.load())
print(unpickler.load())
Here is a more universal method that also works with non pickled data (eg. receiving data from c socket), you first send in the data len and then the actual data in series to the socket, note that the while loop is there because you are not guaranteed to receive the specify number of bytes in a single call, if the data hasn't arrived yet you will receive less. struct.pack packs data you provided to the function into binary data in the order provided by first argument, 'H' represents unsigned short which is 2 bytes long (max integer you can pack into that is 65535)
Server:
import socket
import pickle, struct
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am00 = struct.pack("H", len(am00)) + am00
am01=pickle.dumps(am1)
am01 = struct.pack("H",len(am01)) +am01
am02=pickle.dumps(am2)
am02 = struct.pack("H", len(am02)) + am02
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
client.send(am01)
client.send(am02)
mysocket.close()
Client:
import socket
import pickle, struct
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?---------------
def recv_all(sock:socket.socket):
data_len = 0
buffer = b''
while len(buffer) < 2:
buffer += sock.recv(2-len(buffer))
data_len = struct.unpack("H", buffer)[0]
buffer = b''
while len(buffer) < data_len:
buffer += sock.recv(data_len-len(buffer))
return buffer
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)
I would like to read one by one the objects coming from a TCP stream, preferably using the MessagePack library.
On one side I have a client that, at each iteration of a loop:
computes a point location (tuple)
packs that location and sends it through a socket
On the other side, a server that:
receive the data when client is detected
unpack that data
For now I am storing the data in a buffer on reception, then proceed to unpacking when the stream is over. My problem is that I need to unpack the tuples one by one as they are sent. In other words I would like to read the data in real time without putting it in a buffer.
Provided it is possible, how could I achieve this using MessagePack ?
-- client side --
#Python3.7
import socket
import msgpack
import math
HOST = "127.0.0.1"
PORT = 9000
den = 40
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(den):
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = msgpack.packb((x, y), use_bin_type = True)
sock.sendall(data)
-- server side --
#Jython2.7 <-- Python 2.7 compatible only
from io import BytesIO
import msgpack
import socket
HOST = "127.0.0.1"
PORT = 9000
buf = BytesIO()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()
while True:
try:
data = connection.recv(1024)
buf.write(data)
except:
buf.seek(0)
unpacker = msgpack.Unpacker(buf, use_list=False, raw=False)
for unpacked in unpacker:
print(unpacked)
buf = BytesIO()
See "Stream Unpacking" section in the README:
https://github.com/msgpack/msgpack-python#streaming-unpacking
You can do like this:
unpacker = msgpack.Unpacker(use_list=False, raw=False)
while True:
data = connection.recv(1024)
if not data:
break
unpacker.feed(data)
for unpacked in unpacker:
print(unpacked)
Python27. I am trying to stream the video from camera between client and server apps.
I am learning python from couple of days... I have already learned that to send the data over UDP I need to serialize the data... I have found the pickle library but the code does not work for me. Is there something more regarding data serialization to be able to send the data over the network? How to divide that into smaller parts of data? When i check the type it says that it is string with the size of about 1MB or even more. I guess that's why I'm getting the error:
socket.error: [Errno 10040] A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram into was smaller than the datagram itself
TCP version of this worked fine, but performance was poor. I wanted to try with UDP
Here's my code:
__AUTHOR__=u"DK"
import cv2
import numpy as np
import socket
import sys
import pickle
import struct
def StartStreamSending():
u"""
Funkcja przesyłająca stream z lokalnej kamery[0] na serwer.
"""
UDP_IP = "127.0.0.1"
UDP_PORT = 8012
cap=cv2.VideoCapture(0)
clientsocket=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
while True:
ret,frame=cap.read()
#result, img_str = cv2.imencode('.jpeg', frame)
#img_str2 = np.array(img_str)
#pik = pickle.Pickler()
#frame = np.reshape(frame, (240,320,3))
data = np.array(frame)
dataToSend = pickle.dumps(data)
size = sys.getsizeof(dataToSend)
print(size)
print(type(dataToSend));
clientsocket.sendto(dataToSend, (UDP_IP, UDP_PORT))
StartStreamSending()
And the server:
__AUTHOR__=u"DK"
import socket
import sys
import cv2
import pickle
import numpy as np
import struct
HOST='localhost'
PORT=8012
def WaitAndReceiveVideoStream(_strHOST, _iPORT):
u"""
Funkcja odbierająca stream z kamery klienta.
"""
s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
print 'Socket created'
s.bind((_strHOST,_iPORT))
print 'Socket bind complete'
data = ""
#payload_size = struct.calcsize("L")
i = 0
while True:
data, addr = s.recvfrom(512)
frame=pickle.loads(data)
i = i + 1
print 'coming frame' + str(i)
#frame = numpy.fromstring(data, dtype=numpy.uint8)
#frame = numpy.reshape(frame, (240,320,3))
#if 3 == i :
# cv2.SaveImage("C:\\image.",frame)
cv2.imshow('frame',frame)
cv2.waitKey(4) #Important delay - for test try 2 or 4 value.
s.close()
WaitAndReceiveVideoStream(HOST, PORT)
Ok so I am trying to send an array over a sockets. The array is the axis of a joy stick so x,y,z,slider.
So then i send it over a socket once and I receive it at the other end and then unpickle it where I get the error
It gives me the error _pickle.UnpicklingError: unpickling stack underflow im not really sure what it means but I have read it is because the array is getting cut off.
Client(Uses Pygame to get joystick axis)
x = j.get_axis(0)
y = j.get_axis(1)
slide = j.get_axis(2)
z = j.get_axis(3)
array = (x, y, slide, z)
send_array = pickle.dumps(array)
and just send the array with socket.send(send_array)
and then I receive and just try to print it for testing purposes.
raw_message = sc.recv(1024)
(data1, data2, data3, data4) = pickle.loads(raw_message)
print (data1)
Thanks
Client:
import socket, pickle
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
arr = (YOUR ARRAY)
data_string = pickle.dumps(arr)
s.send(data_string)
data = s.recv(4096)
data_arr = pickle.loads(data)
s.close()
print 'Received', repr(data_arr)
Server:
import socket
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(4096)
if not data: break
conn.send(data)
conn.close()
I have an array kind of ([1,2,3,4,5,6],[1,2,3,4,5,6]) this. I have to send it over a STREAM/TCP socket in python. Then I have to receive the same array at the receiving end.
Sockets are byte streams, so ideal is to write your protocol (read this)
This is a basic example without protocol and you should care about buffer -> recv(). If it is too small, your data will be chopped off. That's why you should implement a protocol, if you send unknown size of data.
Client:
import socket, pickle
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
arr = ([1,2,3,4,5,6],[1,2,3,4,5,6])
data_string = pickle.dumps(arr)
s.send(data_string)
data = s.recv(4096)
data_arr = pickle.loads(data)
s.close()
print 'Received', repr(data_arr)
Server:
import socket
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(4096)
if not data: break
conn.send(data)
conn.close()
You can Serialize the object before sending to socket and at receiving end Deserialize it. Check this
I solved this problem using json (since I heard pickle is unsafe)
client:
import json
...
arr1 = [1,2,3]
arr2 = [4,5,6]
someVar = 7
data = json.dumps({"a": arr1, "b": arr2, "c": someVar})
socket.send(data.encode())
server:
import json
...
data = socket.recv(1024)
data = json.loads(data.decode())
arr = data.get("a")
var = data.get("c")
Here we deserialize the json string, using data.get("a") which you can interpret as data.a
I solved this issue by going over each item in the array, adding it to a single string, but with a significant character, such as a greek letter or some other uncommon character, then sending that string over the socket, then splitting the recieved string up back into an array on the other side, and removing all of the 'delimiter' items in the new array.
For Example, The Client side
for item in myArray:
print("item: ", item)
myArrayString= myArrayString+ str(item) + "Δ"
print(myArrayString)
myServer.send((myArrayString).encode())
and then on the Server:
files = myconnection.recv(50000)
files = files.decode()
myArray = files.split('Δ')
for myItem in myArray:
print(myItem)
print("End Of Items in Array")
hope this helps! feel free to ask if you need anything clarified! : )