I have an array kind of ([1,2,3,4,5,6],[1,2,3,4,5,6]) this. I have to send it over a STREAM/TCP socket in python. Then I have to receive the same array at the receiving end.
Sockets are byte streams, so ideal is to write your protocol (read this)
This is a basic example without protocol and you should care about buffer -> recv(). If it is too small, your data will be chopped off. That's why you should implement a protocol, if you send unknown size of data.
Client:
import socket, pickle
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
arr = ([1,2,3,4,5,6],[1,2,3,4,5,6])
data_string = pickle.dumps(arr)
s.send(data_string)
data = s.recv(4096)
data_arr = pickle.loads(data)
s.close()
print 'Received', repr(data_arr)
Server:
import socket
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
while 1:
data = conn.recv(4096)
if not data: break
conn.send(data)
conn.close()
You can Serialize the object before sending to socket and at receiving end Deserialize it. Check this
I solved this problem using json (since I heard pickle is unsafe)
client:
import json
...
arr1 = [1,2,3]
arr2 = [4,5,6]
someVar = 7
data = json.dumps({"a": arr1, "b": arr2, "c": someVar})
socket.send(data.encode())
server:
import json
...
data = socket.recv(1024)
data = json.loads(data.decode())
arr = data.get("a")
var = data.get("c")
Here we deserialize the json string, using data.get("a") which you can interpret as data.a
I solved this issue by going over each item in the array, adding it to a single string, but with a significant character, such as a greek letter or some other uncommon character, then sending that string over the socket, then splitting the recieved string up back into an array on the other side, and removing all of the 'delimiter' items in the new array.
For Example, The Client side
for item in myArray:
print("item: ", item)
myArrayString= myArrayString+ str(item) + "Δ"
print(myArrayString)
myServer.send((myArrayString).encode())
and then on the Server:
files = myconnection.recv(50000)
files = files.decode()
myArray = files.split('Δ')
for myItem in myArray:
print(myItem)
print("End Of Items in Array")
hope this helps! feel free to ask if you need anything clarified! : )
Related
I made a very simple server and client program to test. I need to send 3 different lists from server and client must recieve them seperately. Could you please help me ?
Here is server.py
import socket
import pickle
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am01=pickle.dumps(am1)
am02=pickle.dumps(am2)
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
#client.send(am01)
#client.send(am02)
mysocket.close()
here is client.py
import socket
import pickle
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?----------------
data1 = pickle.loads(s.recv(buffer_size))
print(data1)
the pickle protocol includes the length of the pickled buffer and you can use pickle.Unpickler to load one pickled object at a time. The problem is that it wants a file-like object, not a socket. Fortunately, sockets can make themselves look like files using socket.makefile. There are caveats, so reading the referenced doc is worthwhile.
Update your server to write each pickled object and then change the client to
import socket
import pickle
host = '127.0.0.1'
port = 8899
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
unpickler = pickle.Unpickler(s.makefile("rb"))
print(unpickler.load())
print(unpickler.load())
print(unpickler.load())
Here is a more universal method that also works with non pickled data (eg. receiving data from c socket), you first send in the data len and then the actual data in series to the socket, note that the while loop is there because you are not guaranteed to receive the specify number of bytes in a single call, if the data hasn't arrived yet you will receive less. struct.pack packs data you provided to the function into binary data in the order provided by first argument, 'H' represents unsigned short which is 2 bytes long (max integer you can pack into that is 65535)
Server:
import socket
import pickle, struct
am0=['AQ-20', 'A3000', 'AQ-26', 'A5000', 'AQ-33', 'A5000pro', 'AQ-33pro']
am1=['A10000Pro', 'AQ-43', 'AX-48', 'AX-58', 'AX-68']
am2=['Material', 'nan', 'Steel', 'Stainless S.', 'Stainless S. 1.403']
am00=pickle.dumps(am0)
am00 = struct.pack("H", len(am00)) + am00
am01=pickle.dumps(am1)
am01 = struct.pack("H",len(am01)) +am01
am02=pickle.dumps(am2)
am02 = struct.pack("H", len(am02)) + am02
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.bind(('127.0.0.1', 1))
buffer_size = 1024
mysocket.listen()
(client, (ip,port)) = mysocket.accept()
client.send(am00)
#----------------HOW CAN I ADD THEM:----------------
client.send(am01)
client.send(am02)
mysocket.close()
Client:
import socket
import pickle, struct
host = '127.0.0.1'
port = 1
buffer_size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
#----------------HOW CAN I RECEIVE THEM SEPERATELY ?---------------
def recv_all(sock:socket.socket):
data_len = 0
buffer = b''
while len(buffer) < 2:
buffer += sock.recv(2-len(buffer))
data_len = struct.unpack("H", buffer)[0]
buffer = b''
while len(buffer) < data_len:
buffer += sock.recv(data_len-len(buffer))
return buffer
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)
data1 = pickle.loads(recv_all(s))
print(data1)
I would like to read one by one the objects coming from a TCP stream, preferably using the MessagePack library.
On one side I have a client that, at each iteration of a loop:
computes a point location (tuple)
packs that location and sends it through a socket
On the other side, a server that:
receive the data when client is detected
unpack that data
For now I am storing the data in a buffer on reception, then proceed to unpacking when the stream is over. My problem is that I need to unpack the tuples one by one as they are sent. In other words I would like to read the data in real time without putting it in a buffer.
Provided it is possible, how could I achieve this using MessagePack ?
-- client side --
#Python3.7
import socket
import msgpack
import math
HOST = "127.0.0.1"
PORT = 9000
den = 40
rad = 100
theta = math.tau / den
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((HOST, PORT)) #connect to server
for step in range(den):
x = math.cos(i*theta) * rad
y = math.sin(i*theta) * rad
data = msgpack.packb((x, y), use_bin_type = True)
sock.sendall(data)
-- server side --
#Jython2.7 <-- Python 2.7 compatible only
from io import BytesIO
import msgpack
import socket
HOST = "127.0.0.1"
PORT = 9000
buf = BytesIO()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()
while True:
try:
data = connection.recv(1024)
buf.write(data)
except:
buf.seek(0)
unpacker = msgpack.Unpacker(buf, use_list=False, raw=False)
for unpacked in unpacker:
print(unpacked)
buf = BytesIO()
See "Stream Unpacking" section in the README:
https://github.com/msgpack/msgpack-python#streaming-unpacking
You can do like this:
unpacker = msgpack.Unpacker(use_list=False, raw=False)
while True:
data = connection.recv(1024)
if not data:
break
unpacker.feed(data)
for unpacked in unpacker:
print(unpacked)
I can not send my numpy array in socket. I use pickle but my client pickle crashes with this error: pickle data was truncated
My server :
I create a numpy array and I want to send in my client with pickle (it's work)
import socket, pickle
import numpy as np
from PIL import ImageGrab
import cv2
while(True):
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print ('Connected by', addr)
arr = np.array([[0, 1], [2, 3]])
printscreen_pil=ImageGrab.grab(bbox=(10,10,500,500))
img = np.array(printscreen_pil) ## Transform to Array
data_string = pickle.dumps(img)
conn.send(data_string)
msg_recu = conn.recv(4096)
print(msg_recu.decode())
conn.close()
My client
He has my numpy array, but I can not load with pickle. I have this error.
import socket, pickle
import numpy as np
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
msg_a_envoyer = "hello".encode()
s.send(msg_a_envoyer)
while 1:
data = s.recv(4096)
if not data: break
data_arr = pickle.loads(data)
print (data_arr)
s.close()
the problem is that if the size of the pickled data is > 4096 you only get the first part of the pickled data (hence the pickle data was truncated message you're getting)
You have to append the data and pickle it only when the reception is complete, for example like this:
data = b""
while True:
packet = s.recv(4096)
if not packet: break
data += packet
data_arr = pickle.loads(data)
print (data_arr)
s.close()
increasing a bytes object is not very performant, would be better to store the parts in a list of objects, then join, though. Faster variant:
data = []
while True:
packet = s.recv(4096)
if not packet: break
data.append(packet)
data_arr = pickle.loads(b"".join(data))
print (data_arr)
s.close()
In simple words, the file you are trying to load is not complete. Either you have not downloaded it correctly or it's just that your pickle file is corrupt. You can create a new pickle to solve this issue
The problem I'm having is to get a file from the server to client across devices. Everything works fine on localhost.
Lets say I want to "get ./testing.pdf" which sends the pdf from the server to the client. It sends but it is always missing bytes. Is there any problems with how I am sending the data. If so how can I fix it? I left out the code for my other functionalities since they are not used for this function.
sending a txt file with "hello" in it works perfectly
server.py
import socket, os, subprocess # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = ''
port = 5000 # Reserve a port for your service.
bufsize = 4096
s.bind((host, port)) # Bind to the port
s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
while True:
userInput = c.recv(1024)
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
somefile = userInput.split(" ")[1]
size = os.stat(somefile).st_size
print size
c.send(str(size))
bytes = open(somefile).read()
c.send(bytes)
print c.recv(1024)
c.close()
client.py
import socket, os # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = '192.168.0.18'
port = 5000 # Reserve a port for your service.
bufsize = 1
s.connect((host, port))
print s.recv(1024)
print "Welcome to the server :)"
while 1 < 2:
userInput = raw_input()
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
s.send(userInput)
fName = os.path.basename(userInput.split(" ")[1])
myfile = open(fName, 'w')
size = s.recv(1024)
size = int(size)
data = ""
while True:
data += s.recv(bufsize)
size -= bufsize
if size < 0: break
print 'writing file .... %d' % size
myfile = open('Testing.pdf', 'w')
myfile.write(data)
myfile.close()
s.send('success')
s.close
I can see two problems right away. I don't know if these are the problems you are having, but they are problems. Both of them relate to the fact that TCP is a byte stream, not a packet stream. That is, recv calls do not necessarily match one-for-one with the send calls.
size = s.recv(1024) It is possible that this recv could return only some of the size digits. It is also possible that this recv could return all of the size digits plus some of the data. I'll leave it for you to fix this case.
data += s.recv(bufsize) / size -= bufsize There is no guarantee that that the recv call returns bufsize bytes. It may return a buffer much smaller than bufsize. The fix for this case is simple: datum = s.recv(bufsize) / size -= len(datum) / data += datum.
I have been looking at some code for a small chat program that I found online. It was originally written for 2.7, but it seems to work with 3.2. The only problem is that I cannot send strings, only numbers:
The chat.py file source code:
from socket import *
HOST = ''
PORT = 8000
s = socket(AF_INET, SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print ('Connected by ' + str(addr))
i = True
while i is True:
data = conn.recv(1024)
print ("Received " + repr(data))
reply = str(input("Reply: "))
conn.send(reply)
conn.close()
And the client.py source file:
from socket import *
HOST = ''
PORT = 8000
s = socket(AF_INET, SOCK_STREAM)
s.connect((HOST, PORT)) # client-side, connects to a host
while True:
message = str(input("Your Message: "))
s.send(message)
print ("Awaiting reply...")
reply = s.recv(1024) # 1024 is max data that can be received
print ("Received " + repr(reply))
s.close()
When I run these using two separate terminals, they work, but do not send strings.
Thank you
When you work with sockets, the message you're passing around should probably be in bytes, b'bytes'. In Python 2.x, a str is actually what a bytes is in Python 3.x
So your message should be something like:
message = b'Message I want to pass'
Check here http://docs.python.org/3.3/library/stdtypes.html for more information.
According to http://docs.python.org/3/library/functions.html#input input returns a str, which means you'll have to encode message into bytes as such:
message = message.encode()
Do verify that this is the right approach to convert str to bytes by checking the type of message.
Your socket code is correct, it was just failing due to an unrelated error due to raw_input vs input. You probably intended to read a string from the shell instead of reading a string and trying to evaluate it as Python code which is what input does.
Try this instead:
chat.py
from socket import *
HOST = ''
PORT = 8000
s = socket(AF_INET, SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print ('Connected by ' + str(addr))
i = True
while i is True:
data = conn.recv(1024)
print ("Received " + repr(data))
reply = str(raw_input("Reply: "))
conn.send(reply)
conn.close()
client.py
from socket import *
HOST = ''
PORT = 8000
s = socket(AF_INET, SOCK_STREAM)
s.connect((HOST, PORT)) # client-side, connects to a host
while True:
message = str(raw_input("Your Message: "))
s.send(message)
print ("Awaiting reply...")
reply = s.recv(1024) # 1024 is max data that can be received
print ("Received " + repr(reply))
s.close()