I am capturing screenshots from the server, then sending it to the client, but the images get all sent as one big file to the client that keeps expanding in size. This only happens when i send from one machine to another (I am working on a local netwrok) but when running both client and server from my machine they work fine.
Note: for the client on the other machine, I packaged it into an exe using pyinstaller, since this machine does not have python.
server code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
data = f.read(4096)
while data:
q.send(data)
data = f.read(4096)
q.send(b"full")
i += 1
time.sleep(0.3)
client code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
i = 0
while True:
with open(str(i) + "s.jpg", "wb") as f:
recv_data = s.recv(4096)
while recv_data:
f.write(recv_data)
recv_data = s.recv(4096)
if(recv_data == b"full"):
break
i += 1
There various wrong assumptions here which lead to the problem you see. The wrong assumptions are:
that send(data) will write all data It might send less. You need to check the return value or use sendall to be sure.
that a single send in the sender is matched by exactly a single recv in the recipientTCP is only an unstructured byte stream. send does not add message semantics, so a single send might lead to multiple recv, multiple send might lead to a single recv etc. Specifically send("data") followed by send("full") might be recv(4096) as "datafull", thus missing your code to detect end of image.
As for why does it work on the local machine but not on the remote - the chance in the latter case is higher that send get combined together and recv as one.
As stated by Steffen Ulrich you should use sendall for sending and for receiving we create a specialized function my_recv that will repeatedly call socket.recv until the expected number of bytes have been received. Also, a 4-byte header (you can make the length greater if your file sizes warrant this) that contains a binary representation of the file length precedes the sending of the actual file data. In this way the client knows exactly how much data it should receive for each file.
Server Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
s.listen(1) # This should be called
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
# Get length by positioning to end of file
image_length = f.seek(0, 2)
f.seek(0, 0) # Seek back to beginning of file
# Convert image length to a 4-byte array:
image_length_bytes = image_length.to_bytes(4, 'big')
q.sendall(image_length_bytes)
data = f.read(4096)
while len(data):
q.sendall(data)
data = f.read(4096)
i += 1
Client Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
def my_recv(msg_length):
chunks = []
bytes_to_recv = msg_length
while bytes_to_recv:
chunk = s.recv(bytes_to_recv)
if chunk == b'':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_to_recv -= len(chunk)
return b''.join(chunks)
i = 0
while True:
image_length_bytes = my_recv(4)
image_length = int.from_bytes(image_length_bytes, 'big')
with open(str(i) + "s.jpg", "wb") as f:
bytes_to_recv = image_length
while bytes_to_recv:
recv_data = my_recv(min(4096, bytes_to_recv))
f.write(recv_data)
bytes_to_recv -= len(recv_data)
i += 1
Related
This question already has answers here:
Python TCP Server receives single TCP packet split into multiple packets
(2 answers)
Sending multiple files through a TCP socket
(2 answers)
How does the python socket.recv() method know that the end of the message has been reached?
(1 answer)
How does socket recv function detects end of message
(1 answer)
How to properly use recv function for TCP chat, python3?
(1 answer)
Closed 3 days ago.
So i am trying to build a web chat app that is capable of sending and receiving files. Apparently i am doing all the right steps since i've been reading other people's code but for some reason the server side tends to fail and not read or receive the bytes of data being sent from the client side. i just can't figure out what is causing this issue since it doesn't return any errors whenever I decide to run the code, it just does not write the new file with the data being received. If some cool og programmer can help me it would mean so much.
client.py
import socket
import os
HOST = '127.0.0.1'
PORT = 65432
FORMAT = 'UTF-8'
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((HOST,PORT))
# Declare anything received as the connection status
connection_status = client.recv(1024).decode()
print(connection_status)
# Select Files
files = ["C:/Users/me/mypics/bmw bike.jpg","C:/Users/me/mypics/bmw car.jpg"]
# Files Name
files_name = [os.path.basename(files[0]), os.path.basename(files[1])]
# Files Size
files_size = [os.path.getsize(files[0]), os.path.getsize(files[1])]
# Send files
for file, file_name, file_size in zip(files, files_name, files_size):
# Send file's name and size
client.sendall(f'{file_name}:{file_size}'.encode(FORMAT))
# Open file
with open(file,'rb') as f:
# Declare chunk as 1024 bytes of data to be sent
chunk = f.read(1024)
# Active loop sending chunks of data until the file is complete
while chunk:
# Send chunk
client.sendall(chunk)
# Declare the next 1024 bytes as another chunk until there is none left
chunk = f.read(1024)
# Close file
f.close()
# Close connection
client.close()
server.py
import socket
import os
HOST = ''
PORT = 65432
FORMAT = 'UTF-8'
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((HOST, PORT))
server.listen()
print(f'Listening For Client Connections')
# Accept client connection
client, address = server.accept()
print(f'Connection Established With Client: {address}')
# Inform the client is now connected
client.sendall('You Are Now Connected!'.encode(FORMAT))
# Active loop for incoming files
while True:
# Declare anything received as the file's name and size attributes
file_attributes = client.recv(1024).decode()
# Condition to break loop in case no attributes were received
if not file_attributes:
break
# Split the attributes
file_name, file_size = file_attributes.split(':')
# Convert the file's size into integers
file_size = int(file_size)
# Open file
with open(file_name,'wb') as f:
# Active loop receiving chunks of data until the amount of bytes are completed
while file_size > 0:
# Declare chunk as to the 1024 bytes of data received from the file
chunk = client.recv(1024)
# Write the chunk of data into the new file
f.write(chunk)
# Subtract the number of bytes received from the chunk onto the file's size
file_size -= len(chunk)
# Close file
f.close()
# Close connection
client.close()
File transfer using socket stream is too slow. Almost 100kbps. I used the python socket module to make this code. It sends data to the client when the client sends the file name. How can I increase the speed of this thing?
Below is the server code
import socket
import os
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('localhost', 9999))
server.listen()
client, addr = server.accept()
msg = client.recv(1024).decode()
file = open("Server_files/"+msg, "rb")
file_size = os.path.getsize("Server_files/"+msg)
# client.send("Received_image.png".encode())
client.send(str(file_size).encode())
data = file.read()
client.sendall(data)
client.send(b"<END>")
file.close()
client.close()
Below is the client code
import tqdm
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 9999))
file_name = input("Enter the file name: ")
client.send(file_name.encode())
file_size = client.recv(1024).decode()
print(file_size)
file = open(file_name, "wb")
file_bytes = b""
done = False
progress = tqdm.tqdm(unit="B", unit_scale=True, unit_divisor=1000,
total=int(file_size))
while not done:
data = client.recv(1024)
if file_bytes[-5:] == b"<END>":
done = True
else:
file_bytes += data
progress.update(1024)
file.write(file_bytes)
file.close()
client.close()
Instead of continuously adding incoming data to file_bytes (which requires potentially a lot of RAM, and also requires Python to reallocate the buffer larger multiple times), you should just write the incoming data directly to file as you receive it.
Also it might help to increase your chunk size from 1024 to something larger, maybe 128*1024; that would allow the system calls to do more work in each call.
Here's a complete solution (Python 3.10) that allows multiple files to be downloaded rapidly. It uses socket.makefile to buffer data, with to automatically close sockets, files, and flush tqdm progress, and writes data as received for speed.
Note that TCP is not a messaged-based protocol. It sends bytes and receives bytes in the same order, but not necessarily with the same "breaks". So send('filename') and send('filesize') can be recv(1024) as 'filenamefilesize' or 'filenamefi' and 'lesize' or any other break. So define a protocol:
Server:
read filename followed by '\n'.
If received 0 bytes, connection was closed.
send filesize as a string in base 10 followed by '\n'.
send exactly "filesize" bytes of file data.
repeat.
Client:
send filename followed by '\n'
read filesize followed by '\n', convert to integer of base 10.
read exactly "filesize" bytes of file data
if done, close connection, else repeat.
server.py
import socket
import os
BLOCK = 128 << 10 # 128KB
with socket.socket() as server:
server.bind(('', 9999))
server.listen()
while True:
client, addr = server.accept()
try:
with (client,
client.makefile('rb') as rfile,
client.makefile('wb') as wfile):
while True:
filename = rfile.readline()
if not filename: break
fullname = os.path.join('Server_files', filename.decode().rstrip('\n'))
file_size = os.path.getsize(fullname)
wfile.write(f'{file_size}\n'.encode())
print(f'Sending {fullname}...')
with open(fullname, 'rb') as file:
while data := file.read(BLOCK):
wfile.write(data)
wfile.flush() # make sure anything remaining in makefile buffer is sent.
print(f' Complete ({file_size} bytes).')
except ConnectionResetError:
print('Client aborted.')
client.py
import socket
import tqdm
BLOCK = 1 << 20 # 1MB
with socket.socket() as client:
client.connect(('localhost', 9999))
with (client.makefile('rb') as rfile,
client.makefile('wb') as wfile):
while True:
file_name = input('File name (just ENTER to quit): ')
if not file_name: break
wfile.write(f'{file_name}\n'.encode())
wfile.flush() # make sure makefile buffer is fully sent
file_size = int(rfile.readline().decode())
with (tqdm.tqdm(unit='B',
unit_scale=True,
unit_divisor=1000,
total=file_size) as progress,
open(file_name, 'wb') as file):
remaining = file_size
while remaining:
data = rfile.read(BLOCK if remaining > BLOCK else remaining)
file.write(data)
progress.update(len(data))
remaining -= len(data)
Output example (note download speeds):
File name (just ENTER to quit): bigfile1
100%|██████████████████████████████████████████████████████████| 191M/191M [00:00<00:00, 706MB/s]
File name (just ENTER to quit): bigfile2
100%|██████████████████████████████████████████████████████████| 218M/218M [00:00<00:00, 718MB/s]
File name (just ENTER to quit):
I am trying to send an image via python sockets. I wrote a simple program for doing that.
server.py
import socket
import cv2
import sys
c = 0
q1 = '127.0.0.1'
q2 = 10001
s = socket.socket()
s.bind((q1, q2))
s.listen(5)
image_path = sys.argv[1]
with open(image_path, 'rb') as fh:
print(fh)
print(dir(fh))
l = list(fh)
print(len(l))
c, addr = s.accept()
if c != 0:
for i in fh:
c.send(i)
client.py
import socket
import cv2
import time
s = socket.socket()
q = '127.0.0.1'
p = 10001
condition = True
s.connect((q,p))
counter = 0
with open('recieved_image.jpg', 'wb') as fh:
while condition:
counter = counter+1
img = s.recv(1024)
if img == b'':
condition = False
fh.write(img)
print(counter)
Now this is the problem. In my case, when I run python server.py test.jpg, the length of list(fh) is 374. And those parts from list(fh) are sent one at a time via socket. But when I recieve them from client.py, the counter variable increments only one time. Isn't it supposed to increment until 374? Can someone clarify on the things that I have understood wrong?
It's not a peculiarity of "Python sockets", it's actually how TCP sockets behave. When you call a low level function like send with TCP, it may or may not send every single byte you requested to be sent, and you have to check its return value in order to determine how much was actually sent. In this specific case, you're calling the send function for every line in the file (because you're using the file descriptor as an iterable). On client side, you try to read up to 1024 bytes from the socket, but just like send, it is not guaranteed you will read all of the data within a single recv call. Obviously, since your counter is incremented just one time, that means that recv receives everything in one stand, in that particular occasion. Learn more about sockets, there are many good tutorials and documentations, even on Wikipedia.
Good morning,
I've a simple question.
I need a socket that allow a client to send an XML to a server. I've tried to send a simple string b'simple string just for example' and everything works fine, but when I try to send an XML (as binary) the server isn't able to write that down. I'll paste my code
Client
import socket
socket = socket.socket()
socket.connect(("localhost", 54610))
file = open(r"c:\shared\68118.xml", "rb")
stream = file.read(65536)
while stream:
socket.send(stream)
stream = file.read(65536) # tried with 1024, 2048
socket.close()
Server
import socket
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('', 54610)
socket.bind(server_address)
socket.listen(1)
while True:
inbound_stream, address = socket.accept()
print(address)
counter = 1
file = open(r'c:\shared\68118_' + str(counter) + '.xml', 'wb') # binary
counter += 1
while(True):
stream = inbound_stream.recv(65536)
while stream:
file.write(stream)
stream = inbound_stream.recv(65536) # tried with 1024, 2048
file.close()
inbound_stream.close()
socket.close() # TODO NOT REACHABLE
the output file is 68118_1.xml 0Kb with nothing inside
What I'm doing wrong?
Thank you in advance
I've just remove the stream = file.read(65536) # tried with 1024, 2048 from the loop and everything worked fine
I'm creating a TCP client and server using Python 3.3 and I'm new to using both Python and sockets. My issue is that I need to be able to store anything passed across the connection as a separate variable for writing to various files, etc.
I'm unsure how to do this as it all seems to be one stream of data that I cannot store separately. Below is my latest piece of working code and all it does is send the data I need across the connection. Should I be trying to send all the data across as a list and de-construct it into separate variables? Can I already access them separately and I just haven't figured it out yet? Is my approach all wrong?
Server code:
import os
from socket import *
HOST = '' #We are the host
PORT = 29876
ADDR = (HOST, PORT)
BUFFSIZE = 4096
serv = socket( AF_INET,SOCK_STREAM)
serv.bind(ADDR,) #Create tuple with one element
serv.listen(5)
print ('listening...')
conn,addr = serv.accept()
print (conn,addr)
print ('...connected')
with open(os.path.expanduser("~/.ssh/id_rsa.pub")) as f:
key = f.read()
conn.sendall(key)
print(key)
while True:
data = conn.recv(BUFFSIZE)
print(data)
conn.close()
Client code:
from socket import *
import os
import socket
HOST = 'xxx.xxx.xxx.xxx'
PORT = 29876 #Must match server.py
ADDR = (HOST,PORT)
BUFFSIZE = 4096
cli = socket.socket( AF_INET, SOCK_STREAM)
cli.connect(ADDR,)
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("gmail.com",80))
ip = ((s.getsockname()[0]))
ip = ip.encode('utf-8')
cli.send(ip)
while True:
data = cli.recv(BUFFSIZE)
print (data)
cli.close()
server:
from socket import *
from os.path import isfile
s = socket()
s.bind(('', 1234))
s.listen(4)
ns, na = s.accept()
loopdata = {}
i = 0
while 1:
try:
data = ns.recv(8192)
except:
break
for line data.split('\n'):
if line == 'version':
print na[0] + ' requested a version'
ns.send('1.0\n')
elif line == 'key':
print na[0] + ' is requesting a key'
if isfile(na[0] + '.key'):
with open(na[0] + '.key') as f:
ns.send(f.read())
else:
ns.send('Missing key file!\n')
loopdata[i] = line
#ns.send('OK\n')
i += 1
ns.close()
s.close()
print loopdata # <- Print all the lines
client:
from socket import *
s = socket()
s.connect(('127.0.0.1', 1234))
s.send('version\n')
print 'Client got:', s.recv(8192)
s.close()
I'm not sure as to what you want to save/store/respond to,
You mentioned something about a key in your code and in your client code you created multiple sockets but only really used one.. and that was for printing whatever the server was sednding?
Begin clean, try to figure out things before mixing it all up.
And state a clear goal that you want to achieve with your code? what do you want to send? and when sending X, what do you want the server to respond?
Tactics:
1. You want to define a protocol,
1.1 Command separator
1.2 Command structure (ex: command:parameter:data\n)
1.3 sates (ex login state etc etc..)
After you know what you want to do, ex file share:
c->s: get:file:/root/storage/file.txt\n
c<-s: file:content\n
c<-s: <file data>\n\n
c<-s: file:close\n
c->s: file:recieved
For instance.