This question already has an answer here:
sending multiple files in python
(1 answer)
Closed 3 years ago.
I am using python 3.6 and ubuntu 18.04.
I am able to send single music file using socket-python (in binary mode) and i want to send multiple music files from server to client.
Problem is, at the receiver end (that is client), all the music files (approx 120 files sent from server) gets collected in one single file making it a 9 hour long single music file.
I have tried using time.sleep method (does not work), tried sending bogus element (error was shown) and tried sending some random character to end the file writing at the client side and initiate new file write (but random character requires encoding and decoding, so again error as binary data was unable to decode).
SERVER CODE
import socket
import os
import send_file
import time
s = socket.socket()
host = ""
port = 9997
s.bind((host, port))
s.listen(5)
print("Binding Done\n")
socket_object, address = s.accept()
print("Connection Established\n")
print("Sending file...")
file_class = send_file.send_files() #ignore
file_names = file_class.files #ignore - contains list of path of music file
socket_object.sendall( str(len(file_names)).encode() )
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
print("Files Send")
socket_object.close()
s.close()
CLIENT CODE
import socket
import os
import time
def recv_file(i):
f = open("/home/ravi/PycharmProjects/File_Transfer/B/"+"M"+str(i)+".mp3", 'wb')
buf = s.recv(1024)
while buf:
f.write(buf)
buf = s.recv(1024)
f.close()
s = socket.socket()
host = "127.0.0.1"
port = 9997
s.connect((host, port))
print("Receiving data...")
l = s.recv(1024).decode() #ignore - length of total number of files i.e., 120 approx
for i in range(int(l)):
recv_file(i+1)
print("Files Collected")
s.close()
Any suggestions would be appreciated. Thank You.
TCP is a stream-oriented protocol. It is designed for sending streams of data. At TCP level there are no files. It doesn't split streams in any file-oriented chunks.
Look at your code:
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
You just glue all files in a single stream, and your client has no idea when one file ends and the next file starts.
The task of sending multiple files over TCP could be solved in many ways:
Develop your own protocol. E.g.: first send the number of files, then send an array of 8-byte-encoded file lengths, and then the stream of file contents. The receiving end reads number of files, then parses file lengths. Knowing the lengths the receiver correctly splits the stream into files.
Use existing multi-file packaging formats: tar, cpio, zip, etc. Pack files before sending, then send the resulting package. On the receiving end unpack the package after receiving.
Recommended way Use existing protocols for sending files over TCP: TFTP or FTP.
I'd recommend using TFTP. It is very simple and reasonably efficient. There are several implementations in Python, such as tftpy
On the remote machine where you want to upload your files to, start TFTP server:
import tftpy
server = tftpy.TftpServer('<destination-folder>')
server.listen('0.0.0.0', 54321)
On the machine with files start the client:
import tftpy
client = tftpy.TftpClient('your.server.address', 54321)
for name in ("fileA", "fileB", "fileC"):
# the first argument is the name on the remote machine
# the second argument is the name on the local machine
client.upload(name, name)
Related
This question already has answers here:
Python TCP Server receives single TCP packet split into multiple packets
(2 answers)
Sending multiple files through a TCP socket
(2 answers)
How does the python socket.recv() method know that the end of the message has been reached?
(1 answer)
How does socket recv function detects end of message
(1 answer)
How to properly use recv function for TCP chat, python3?
(1 answer)
Closed 3 days ago.
So i am trying to build a web chat app that is capable of sending and receiving files. Apparently i am doing all the right steps since i've been reading other people's code but for some reason the server side tends to fail and not read or receive the bytes of data being sent from the client side. i just can't figure out what is causing this issue since it doesn't return any errors whenever I decide to run the code, it just does not write the new file with the data being received. If some cool og programmer can help me it would mean so much.
client.py
import socket
import os
HOST = '127.0.0.1'
PORT = 65432
FORMAT = 'UTF-8'
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((HOST,PORT))
# Declare anything received as the connection status
connection_status = client.recv(1024).decode()
print(connection_status)
# Select Files
files = ["C:/Users/me/mypics/bmw bike.jpg","C:/Users/me/mypics/bmw car.jpg"]
# Files Name
files_name = [os.path.basename(files[0]), os.path.basename(files[1])]
# Files Size
files_size = [os.path.getsize(files[0]), os.path.getsize(files[1])]
# Send files
for file, file_name, file_size in zip(files, files_name, files_size):
# Send file's name and size
client.sendall(f'{file_name}:{file_size}'.encode(FORMAT))
# Open file
with open(file,'rb') as f:
# Declare chunk as 1024 bytes of data to be sent
chunk = f.read(1024)
# Active loop sending chunks of data until the file is complete
while chunk:
# Send chunk
client.sendall(chunk)
# Declare the next 1024 bytes as another chunk until there is none left
chunk = f.read(1024)
# Close file
f.close()
# Close connection
client.close()
server.py
import socket
import os
HOST = ''
PORT = 65432
FORMAT = 'UTF-8'
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((HOST, PORT))
server.listen()
print(f'Listening For Client Connections')
# Accept client connection
client, address = server.accept()
print(f'Connection Established With Client: {address}')
# Inform the client is now connected
client.sendall('You Are Now Connected!'.encode(FORMAT))
# Active loop for incoming files
while True:
# Declare anything received as the file's name and size attributes
file_attributes = client.recv(1024).decode()
# Condition to break loop in case no attributes were received
if not file_attributes:
break
# Split the attributes
file_name, file_size = file_attributes.split(':')
# Convert the file's size into integers
file_size = int(file_size)
# Open file
with open(file_name,'wb') as f:
# Active loop receiving chunks of data until the amount of bytes are completed
while file_size > 0:
# Declare chunk as to the 1024 bytes of data received from the file
chunk = client.recv(1024)
# Write the chunk of data into the new file
f.write(chunk)
# Subtract the number of bytes received from the chunk onto the file's size
file_size -= len(chunk)
# Close file
f.close()
# Close connection
client.close()
I am capturing screenshots from the server, then sending it to the client, but the images get all sent as one big file to the client that keeps expanding in size. This only happens when i send from one machine to another (I am working on a local netwrok) but when running both client and server from my machine they work fine.
Note: for the client on the other machine, I packaged it into an exe using pyinstaller, since this machine does not have python.
server code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
data = f.read(4096)
while data:
q.send(data)
data = f.read(4096)
q.send(b"full")
i += 1
time.sleep(0.3)
client code:
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
i = 0
while True:
with open(str(i) + "s.jpg", "wb") as f:
recv_data = s.recv(4096)
while recv_data:
f.write(recv_data)
recv_data = s.recv(4096)
if(recv_data == b"full"):
break
i += 1
There various wrong assumptions here which lead to the problem you see. The wrong assumptions are:
that send(data) will write all data It might send less. You need to check the return value or use sendall to be sure.
that a single send in the sender is matched by exactly a single recv in the recipientTCP is only an unstructured byte stream. send does not add message semantics, so a single send might lead to multiple recv, multiple send might lead to a single recv etc. Specifically send("data") followed by send("full") might be recv(4096) as "datafull", thus missing your code to detect end of image.
As for why does it work on the local machine but not on the remote - the chance in the latter case is higher that send get combined together and recv as one.
As stated by Steffen Ulrich you should use sendall for sending and for receiving we create a specialized function my_recv that will repeatedly call socket.recv until the expected number of bytes have been received. Also, a 4-byte header (you can make the length greater if your file sizes warrant this) that contains a binary representation of the file length precedes the sending of the actual file data. In this way the client knows exactly how much data it should receive for each file.
Server Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
import time
import pyautogui
from socket import *
import os
s=socket(AF_INET, SOCK_STREAM)
s.bind((host,port))
s.listen(1) # This should be called
print("Listening for connections.. ")
q,addr=s.accept()
i = 0
while True:
screenshot = pyautogui.screenshot()
screenshot.save(str(i) + ".jpg")
with open(str(i) + ".jpg", "rb") as f:
# Get length by positioning to end of file
image_length = f.seek(0, 2)
f.seek(0, 0) # Seek back to beginning of file
# Convert image length to a 4-byte array:
image_length_bytes = image_length.to_bytes(4, 'big')
q.sendall(image_length_bytes)
data = f.read(4096)
while len(data):
q.sendall(data)
data = f.read(4096)
i += 1
Client Code
host="192.168.43.79" # Set the server address to variable host
port=4446 # Sets the variable port to 4446
from multiprocessing.reduction import recv_handle
from socket import * # Imports socket module
s=socket(AF_INET, SOCK_STREAM) # Creates a socket
s.connect((host,port))
def my_recv(msg_length):
chunks = []
bytes_to_recv = msg_length
while bytes_to_recv:
chunk = s.recv(bytes_to_recv)
if chunk == b'':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_to_recv -= len(chunk)
return b''.join(chunks)
i = 0
while True:
image_length_bytes = my_recv(4)
image_length = int.from_bytes(image_length_bytes, 'big')
with open(str(i) + "s.jpg", "wb") as f:
bytes_to_recv = image_length
while bytes_to_recv:
recv_data = my_recv(min(4096, bytes_to_recv))
f.write(recv_data)
bytes_to_recv -= len(recv_data)
i += 1
The following codes let me download from server to client three files called tmp.bsp, tmp.seq and tmp.dms. However, just the first file tmp.dms is completely downloaded. The other one tmp.seq is filled up with the informations of tmp.bsp and tmp.bsp stay 0KB.
client:
import socket
import socket
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
sData = "Temp"
sData2 = "Temp"
sData3 = "Temp"
while True:
sData = skClient.recv(1024)
fDownloadFile = open("tmp.dms","wb")
sData2 = skClient.recv(1024)
fDownloadFile2 = open("tmp.seq","wb")
sData3 = skClient.recv(1024)
fDownloadFile3 = open("tmp.bsp","wb")
while sData:
fDownloadFile.write(sData)
sData = skClient.recv(1024)
fDownloadFile.close()
fDownloadFile2.write(sData2)
sData2 = skClient.recv(1024)
fDownloadFile2.close()
fDownloadFile3.write(sData3)
sData3 = skClient.recv(1024)
fDownloadFile3.close()
print "Download over"
break
skClient.close()
n is a counter and the prints are for debugging.
sFileName is to download one file, and used to work but since I want three files I just commented it.
server:
import socket
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
while True:
Content,Address = skServer.accept()
print Address
files = "C:\Users\Name_user\Desktop\Networking\Send_Receive/"
fUploadFile = open(files+str('tmp.dms'),"rb")
sRead = fUploadFile.read(1024)
fUploadFile2 = open(files+str('tmp.seq'),"rb")
sRead2 = fUploadFile2.read(1024)
fUploadFile3 = open(files+str('tmp.bsp'),"rb")
sRead3 = fUploadFile3.read(1024)
while sRead:
Content.send(sRead)
sRead = fUploadFile.read(1024)
Content.send(sRead2)
sRead2 = fUploadFile2.read(1024)
# Content.send(sRead3)
# sRead3 = fUploadFile3.read(1024)
Content.close()
print "Sending is over"
break
skServer.close()
files I'm using:
server2.py is my server
Execution
The main issue with your code is that you're sending / receiving an arbitrary number of data. If your buffer (1024) is smaller than the file size then the client's file will contain less information, and if it's larger the file may contain more information (data from the next file).
You could solve this issue by sending a value that signifies the end of a file. The problem with this method is that this value can't be contained in any file, and the client must be scanning the received data for this value.
Another possible solution is to calculate the file size and send that infomation in front of the file data. This way the cilent will know how many data to expect for each file.
Using struct.pack we can create a minimal four bytes header with the file size.
def send_file(soc, path):
with open(path, 'rb') as f:
data = f.read()
size = struct.pack('!I', len(data))
soc.send(size + data)
Tthen the client can get the file size by reading four bytes and unpacking to int.
def recv_file(soc, path):
size_header = soc.recv(4)
size = struct.unpack('!I', size_header)[0]
data = soc.recv(size)
with open(path, 'wb') as f:
f.write(data)
Note that sending/receiving files with one call may raise a socket error if the file size is larger than the socket buffer. In that case you'll have to read the data in smaller chunks in a loop, or increase the buffer size with socket.setsockopt.
Here is a modified version of the above functions that can handle large files:
import struct
import os.path
def send_file(soc, path):
file_size = os.path.getsize(path)
size_header = struct.pack('!Q', file_size)
soc.send(size_header)
with open(path, 'rb') as f:
while True:
data = f.read(1024)
if not data:
break
soc.send(data)
def recv_file(soc, path):
size_header = soc.recv(8)
file_size = struct.unpack('!Q', size_header)[0]
chunks = [1024 for i in range(file_size / 1024)]
with open(path, 'wb') as f:
for chunk in chunks:
f.write(soc.recv(chunk))
f.write(soc.recv(file_size % 1024))
I haven't tested this code thoroughly, but it should work for files of any size.
An example using the send_file function in your server:
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
Content,Address = skServer.accept()
print Address
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
send_file(Content, file)
Content.close()
print "Sending is over"
skServer.close()
Using recv_file in the client:
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
recv_file(skClient, file)
print "Download over"
skClient.close()
Yes you are right, I did run your program and found exactly same issue. I dont have enough time to work more on this issue but I found few key points which might lead you to the right work around.
https://docs.python.org/2/howto/sockets.html
The above official doc says:
When a recv returns 0 bytes, it means the other side has closed (or is in the process of closing) the connection. You will not receive any more data on this connection. Ever. You may be able to send data successfully
This is what it is happening when the third file returns 0 bytes.
But why 2nd and 3rd file is merged, I guess its because sockets are just buffered files and we might need to try making sure buffer is clear before sending another.
Read this,
Now there are two sets of verbs to use for communication. You can use send and recv, or you can transform your client socket into a file-like beast and use read and write. The latter is the way Java presents its sockets. I’m not going to talk about it here, except to warn you that you need to use flush on sockets. These are buffered “files”, and a common mistake is to write something, and then read for a reply. Without a flush in there, you may wait forever for the reply, because the request may still be in your output buffer.
But if you plan to reuse your socket for further transfers, you need to realize that there is no EOT on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there’s nothing more to read (for now). Now if you think about that a bit, you’ll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).
Hope this helps.
I'm not totally fluent in Python, but I think your while statement should be something like:
while: sData or sData2 or sData3
I may have the syntax wrong, but currently it looks like you will stop when "sData" is done and stop downloading sData2 and aData3 at that time even if they haven't finished.
Hmm--either that or the "While" isn't looping at all and it's just being used as an "if"? hard to tell without knowing the API.
I am working on a client-server file sharing to update a particular folder. Where the client connects to server using tcp-socket and recieve a particular zip-file then unzip it. I have accomplished doing that so far but I want to compare the files in both folders(client and server) to check for difference in files contents(ie: updated files) and only download the files if the contents are different.
My code:
Client:
import socket
import zipfile
import os
def main():
host = '192.168.1.8'
port = 5000
s = socket.socket()
s.connect((host, port))
with open('C:\\file.zip', 'wb') as f:
while True:
data = s.recv(1024)
if not data:
break
f.write(data)
zip_ref = zipfile.ZipFile('C:\\file.zip', 'r')
zip_ref.extractall('C:\\')
zip_ref.close()
os.remove('C:\\file.zip')
s.close()
if __name__ == '__main__':
main()
Server:
import socket
from threading import Thread
def send_file(conn, filename):
with open(filename, 'rb') as f:
print 'Sending file'
data = f.read(1024)
while data:
conn.send(data)
data = f.read(1024)
print 'Finished sending'
conn.close()
def main():
host = '192.168.1.8'
port = 5000
s = socket.socket()
s.bind((host, port))
s.listen(5)
while True:
c, addr = s.accept()
t = Thread(target=send_file, args=(c, 'C:\\file.zip'))
t.start()
if __name__ == '__main__':
main()
What I have tried so far:
I tried filecmp.dircmp but It only checks for different files and not different content of files and also I couldn't compare folder from client with folder from server. I also tried to loop through files and use filecmp on each file but I also couldn't compare it with the same file from server.
Is there an efficient way to do this?
Not sure I understand you are using filecmp to compare contents from the client before downloading from the server. In these cases, usually there are two approaches: Incorporate a protocol to check the modified date of the files in the server (e.g. os.path.getmtime(file_name)) and then be sure to set the modified date when your client downloads the files; or; have the client request hashes for the files and download when the hashes don't match.
I have written two client and server python scripts were server is listening for client to send files following is my server code for listening files. The problems with the code i cannot separate filename from the file data. Following output is received data.txtMyName
def sendFileName(self):
self.clientsocket.send("name:" + self.filename)
print 'filename', self.filename
def sendFile(self):
f=open(self.filename,"rb")
data= f.read(1024)
while (data):
if(self.clientsocket.send(data)):
print "sending data"
data = f.read(1024)
#readByte = open(self.filename, "rb")
#data = readByte.read()
#readByte.close()
#self.gateway.send(data)
self.clientsocket.close()
f.close
def receiveFile(self,sock):
data = sock.recv(1024)
print 'filename', data.strip()
f = open(data.strip(), "wb")
data = sock.recv(1024)
while (data):
f.write(data)
data=sock.recv(1024)
f.close()
self.server_socket.close()
You have to create your own simple protocol on top of TCP/IP to get it work. The simplest I would imagine is to add one special character (for example 0x00) between filename and file content. Receiving site could detect this char and split filename and file content.
More complicated protocol could also send file size so the receiving site could detect when transmission is finally over and if all bytes were send.
It will also be good to send acknowledge message to sender that receiver got whole file.
There is lot of transfer file protocols (TFTP, FTP) and they exist for a reason, because pure TCP/IP without any protocol on top is useless.