Develop Reference

Why it take so much time for the server to receive a message? Python socket

I am trying to code a simple share screen script. For some reason it take a lot of time for the server to receive the message. And I dont know what is the problem.
If you have a solution, or a way to speed the process up it be very helpful. Any way I am appreciate your help.
server:
import socket
import pickle
import select
import numpy
import cv2
def receive_msg(socket, HEADERSIZE):#receive message
try:
readySockets, _, _ = select.select([socket], [], [], 0.02)
if readySockets:
msgLen = socket.recv(HEADERSIZE)#receive the header/the message length
msgLen = int(msgLen)#convert fron bytes to int
msg = socket.recv(msgLen)#resive the size of the message
while len(msg) < msgLen:#if dont receive the full message / if the size of the message smaller than the size that the message sepose to be
msg += socket.recv(msgLen - len(msg))#add to the message the part that missing (the full size of the message - the size of the message that the program received)
msg = pickle.loads(msg)#extract message
else:
msg = False
except:
msg = False
return msg#return the complite massage
HEADERSIZE = 10
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((socket.gethostname(), 1243))
s.listen(5)
clientsocket, address = s.accept()
while True:
# now our endpoint knows about the OTHER endpoint.
msg = receive_msg(clientsocket, HEADERSIZE)
if msg is not False:
image = cv2.cvtColor(msg, cv2.COLOR_RGB2BGR)#numpy array to open cv image
cv2.imshow("image", image)#show image
cv2.waitKey(1)
client:
import socket
import numpy as np
import pyautogui
import pickle
def get_screen():
img = pyautogui.screenshot()# take a screenshot
img = np.array(img)#from image to array
return img
def send_msg(socket, msg, HEADERSIZE):#send a message
msg = pickle.dumps(msg)#comprass the msg
#give the mag a header / signature of the size of the message
msg = bytes(str(len(msg)) + (" " * (HEADERSIZE - len(str(len(msg))))), 'utf-8') + msg
socket.send(msg)#send the msg
HEADERSIZ = 10
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((socket.gethostname(), 1243))
while True:
send_msg(s, get_screen(), HEADERSIZ)#send a image of the screen to the server

Python UDP socket; still receiving data when source is stopped; why?

I am sending 32 bytes packets every 1ms to this socket. I wish to print the data after every 40 ms. And apparently the code does that. But even when I stop sending data, I still continue to see that data is being printed.
Is it holding the data in some cache? or simply the python socket has a huge delay? Why?
The code is as follows:
## Import necessary libraries
import math
import numpy as np
import socket
import struct
import time
from synchrophasor.frame import CommandFrame
from datetime import datetime
## Configure socket for Phasor data ##
UDP_IP = "10.10.114.22"
UDP_PORT = 8208 #UDP phasor values 32 bytes (V,phi,P)
sock_ph = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock_ph.bind((UDP_IP, UDP_PORT))
print("socket bound, waiting for data...")
while True:
raw = sock_ph.recv(32)
#print(raw)
mag = struct.unpack('d', raw[8:16])[0]
# print("mag =",mag,type(mag))
angle = struct.unpack('d', raw[16:24])[0]
# print("angle =",angle,type(angle))
header = struct.unpack('d', raw[0:8])[0]
# print("header =",header,type(header))
phasor = (mag, angle)
Vol_A=raw
VA = float(mag)
phi_A = float(angle)
VB = VA
phi_B = phi_A+(math.pi) * 2 / 3
VC = VA
phi_C = phi_A-(math.pi) * 2 / 3
time.sleep(1/25)
# pmu.send_data(phasors=[(VA,phi_A),(VB,phi_B),(VC,phi_C)],analog=[9.91],digital=[0x0001])
#time.sleep(1/config_rr)
print([(VA,phi_A),(VB,phi_B),(VC,phi_C),datetime.now()])

most programs don't want to discard unread datagrams so most OSs will buffer them for you. your case is somewhat unusual so you'd need to write code to handle this case. I'd change your code to do something like:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('', 8208))
# block until we read an initial packet
raw = s.recv(1024)
s.setblocking(False)
while True:
# unpack
header, mag, angle = struct.unpack('ddd', raw)
# do something with data
print(f'header={header} mag={mag} angle={angle}')
# sleep for some time
time.sleep(1/25)
# discard any packets you've received in the mean time
while True:
try:
raw = s.recv(1024)
except OSError as err:
# OS buffer is empty: we've therefore got the most recent data
if err.errno == socket.EWOULDBLOCK:
break
# something else failing, reraise the error
raise
note that Steffen Ullrich's suggestion of sending the data at the correct rate would be easier, but assumes that you have control over the sending process. the fact that you said "I am sending" suggests you do, and so would likely make a better solution

Python socket Select module showing it received to read requests, when it only received one. Why?

I am creating a program where the client enters in the name of a file containing an image. Then it is converted to a numpy array, pickled and sent to the server. The server uses PIL ImageDraw to draw a red X on top of said image then sends it back to the client. I have noticed that the select function adds the socket twice to the read list for some reason. I know this since the client gets the image back and I added a print function at the end of the sending process and it triggered once, but then an earlier print statement went off again, and returned an error. Here is the server output:
b'56925 '
sent
b''
Traceback (most recent call last):
File "server.py", line 31, in <module>
msglengthi = int(msglength)
ValueError: invalid literal for int() with base 10: ''
Here is the server code:
import socket
import pickle
from PIL import ImageDraw
from PIL import Image
import select
import numpy
IP = socket.gethostbyname(socket.gethostname())
PORT = 4321
HEADERSIZE = 15
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((IP, PORT))
s.listen(5)
socketlist = [s]
readqueue = []
while True:
readqueue, write, exception = select.select(socketlist, [], [])
for socket in readqueue:
if socket == s:
clientsocket, clientaddress = s.accept()
socketlist.append(clientsocket)
print(f"Connection received from {clientaddress}")
clientsocket.send(bytes("Welcome to the server", "utf-8"))
else:
msglen = socket.recv(HEADERSIZE)
msglength = msglen.decode("utf-8")
print(msglen)
msglengthi = int(msglength)
fullmsg = []
x=0
while x<=msglengthi:
msgu = socket.recv(3000)
fullmsg.append(msgu)
x+=3000
fullmsg = b"".join(fullmsg)
msg = pickle.loads(fullmsg)
img = Image.fromarray(msg)
draw = ImageDraw.Draw(img)
width, height = img.size
draw.line((0, 0, width, height), fill="red", width=20)
draw.line((width, 0, 0, height), fill="red", width=20)
payload = pickle.dumps(numpy.array(img))
paylen = len(payload)
socket.send(bytes(f"{paylen:<{HEADERSIZE}}", "utf-8")+payload)
print("sent")
and finally the client code:
import socket
import pickle
from PIL import Image
import numpy
PORT = 4321
IP = socket.gethostbyname(socket.gethostname())
HEADERSIZE = 15
cs = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
cs.connect((IP, PORT))
welcomemsg = cs.recv(21)
welcomemsg = welcomemsg.decode("utf-8")
print(welcomemsg)
img = input("Enter image file: ")
imgo = Image.open(img)
imga = numpy.array(imgo)
imgdump = pickle.dumps(imga)
msglen = len(imgdump)
print(msglen)
cs.send(bytes(f"{msglen:<{HEADERSIZE}}", "utf-8")+imgdump)
msglen = cs.recv(HEADERSIZE)
msglen = msglen.decode("utf-8").strip()
msglen = int(msglen)
x=0
fullmsg=[]
while x<=msglen:
msg = cs.recv(3000)
fullmsg.append(msg)
x+=3000
fullmsg = b"".join(fullmsg)
img = pickle.loads(fullmsg)
img = Image.fromarray(img)
img.show()
Thanks!

I have noticed that the select function adds the socket twice to the read list for some reason.
It doesn't. select returns that the socket is readable whenever there are information on the socket which can be retrieved using recv. While you handle the case that recv returns actual data you don't handle the case when recv returns an empty buffer (i.e. '') which it does if the peer closes the connection. In other words: the problem is not select but your assumption of how select and recv work.

Fast way of transfering large image datasets PYTHON 3

first of all thank you for your time. I have need to transfer images from one PC (Windows) to an other (Linux) and the other way arround. I used sockets and byte streams. How ever it takes 4 seconds to send one image which is insane amount of time. Am I missing some point here? Is there any library that supports fast file transfer on windows as well as on linux?
Server or sender
import numpy
import socket
import threading
import os
from idlelib.IOBinding import encoding
import Communication
import multiprocessing
import time
import cv2
import WrapDatastructures
import sys
import ImageProcessorClient
Queue = multiprocessing.Queue()
def SendFileToClient(name, sock, image, image_number):
#ImageRequest
ImageRequest = sock.recv(1024)
#Decode the Bytestring into ascii characters
ImageRequestAsStr = ImageRequest.decode('ascii')
print("Incoming Request String:",ImageRequestAsStr)
if ImageRequestAsStr == "FILEFLAG":
#Cascade to convert grayscale image to byte array and send it to Image Processor Client
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
#Send size of the image to processor client
memoryConsumption = sys.getsizeof(gray_image)
print("Memory Consumtion",memoryConsumption)
sendStr = "EXISTS" + str(memoryConsumption)
#Convert the string to byte because otherwise it will not be send
sock.send((sendStr.encode(encoding)))
userResponse = sock.recv(1024)
#the Responce will be received in byte and will be converted to a string to make it checkable
userResponceStr = userResponse.decode('ascii')
if userResponceStr[:2] == 'OK':
#Get byte array
print("Convert grayscale_image to bytedata")
data = WrapDatastructures.singleChannelImgToByte(gray_image)
#bytesToSend = data #send 1024 bytes
#sock.send(bytesToSend)
sendData = 0
for i in range(0,memoryConsumption,1024):
sock.send(data[i:i+1024])
else:
print("User response not known")
else:
sendStr = "ERR"
sock.send(sendStr.encode(encoding))
sock.close()
def Main():
host = "127.0.0.1"
port = 5000
s = socket.socket()
s.bind((host,port))
s.listen(5)
#init camera
print("server started.")
imageCounterPerContainer = 0
while(True):
#Take image and put it in Q
#Call the controller to turn motor
#look for client to get request
if Queue.empty() is True:
#create client
print("create Client")
while True:
c, addr = s.accept()
print("client connected ip:< " + str(addr) +">")
imageCounterPerContainer +=1
t = threading.Thread(target = SendFileToClient, args=("rtrThread",c,cv2.imread("image_33.jpg"),imageCounterPerContainer))
t.start()
s.close()
if __name__ == "__main__":
Main()
Processor
import socket
from idlelib.IOBinding import encoding
import WrapDatastructures
import cv2
import time
def Main():
host = "127.0.0.1"
port = 5000
start = time.time()
s = socket.socket()
s.connect((host,port))
getfileRequest = "FILEFLAG"
if getfileRequest != "q":
s.send(getfileRequest.encode())
data = s.recv(1024)
dataStr = data.decode('ascii')
if dataStr[:6] == "EXISTS":
filesize = int(dataStr[6:])
print("Data is available size", filesize)
sendStr = "OK"
s.send(sendStr.encode(encoding))
#create new file new_filename and
img = b""
while True:
data = s.recv(1024)
if len(data) == 0:
break
img = b"".join([img, data])
print("Download Complete")
transferedImg = WrapDatastructures.ByteToCV2(img, (2048,2448))
cv2.imshow("transfered",transferedImg)
end = time.time()
print("Duration",end -start)
cv2.waitKey(0)
s.close()
else:
print("File does not exists")#
s.close()
s.close()
if __name__ == "__main__":
Main()

python 3.6 socket pickle data was truncated

I can not send my numpy array in socket. I use pickle but my client pickle crashes with this error: pickle data was truncated
My server :
I create a numpy array and I want to send in my client with pickle (it's work)
import socket, pickle
import numpy as np
from PIL import ImageGrab
import cv2
while(True):
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print ('Connected by', addr)
arr = np.array([[0, 1], [2, 3]])
printscreen_pil=ImageGrab.grab(bbox=(10,10,500,500))
img = np.array(printscreen_pil) ## Transform to Array
data_string = pickle.dumps(img)
conn.send(data_string)
msg_recu = conn.recv(4096)
print(msg_recu.decode())
conn.close()
My client
He has my numpy array, but I can not load with pickle. I have this error.
import socket, pickle
import numpy as np
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
msg_a_envoyer = "hello".encode()
s.send(msg_a_envoyer)
while 1:
data = s.recv(4096)
if not data: break
data_arr = pickle.loads(data)
print (data_arr)
s.close()

the problem is that if the size of the pickled data is > 4096 you only get the first part of the pickled data (hence the pickle data was truncated message you're getting)
You have to append the data and pickle it only when the reception is complete, for example like this:
data = b""
while True:
packet = s.recv(4096)
if not packet: break
data += packet
data_arr = pickle.loads(data)
print (data_arr)
s.close()
increasing a bytes object is not very performant, would be better to store the parts in a list of objects, then join, though. Faster variant:
data = []
while True:
packet = s.recv(4096)
if not packet: break
data.append(packet)
data_arr = pickle.loads(b"".join(data))
print (data_arr)
s.close()

In simple words, the file you are trying to load is not complete. Either you have not downloaded it correctly or it's just that your pickle file is corrupt. You can create a new pickle to solve this issue