Develop Reference

How to send multiple music files using sockets? [duplicate]

This question already has an answer here:
sending multiple files in python
(1 answer)
Closed 3 years ago.
I am using python 3.6 and ubuntu 18.04.
I am able to send single music file using socket-python (in binary mode) and i want to send multiple music files from server to client.
Problem is, at the receiver end (that is client), all the music files (approx 120 files sent from server) gets collected in one single file making it a 9 hour long single music file.
I have tried using time.sleep method (does not work), tried sending bogus element (error was shown) and tried sending some random character to end the file writing at the client side and initiate new file write (but random character requires encoding and decoding, so again error as binary data was unable to decode).
SERVER CODE
import socket
import os
import send_file
import time
s = socket.socket()
host = ""
port = 9997
s.bind((host, port))
s.listen(5)
print("Binding Done\n")
socket_object, address = s.accept()
print("Connection Established\n")
print("Sending file...")
file_class = send_file.send_files() #ignore
file_names = file_class.files #ignore - contains list of path of music file
socket_object.sendall( str(len(file_names)).encode() )
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
print("Files Send")
socket_object.close()
s.close()
CLIENT CODE
import socket
import os
import time
def recv_file(i):
f = open("/home/ravi/PycharmProjects/File_Transfer/B/"+"M"+str(i)+".mp3", 'wb')
buf = s.recv(1024)
while buf:
f.write(buf)
buf = s.recv(1024)
f.close()
s = socket.socket()
host = "127.0.0.1"
port = 9997
s.connect((host, port))
print("Receiving data...")
l = s.recv(1024).decode() #ignore - length of total number of files i.e., 120 approx
for i in range(int(l)):
recv_file(i+1)
print("Files Collected")
s.close()
Any suggestions would be appreciated. Thank You.

TCP is a stream-oriented protocol. It is designed for sending streams of data. At TCP level there are no files. It doesn't split streams in any file-oriented chunks.
Look at your code:
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
You just glue all files in a single stream, and your client has no idea when one file ends and the next file starts.
The task of sending multiple files over TCP could be solved in many ways:
Develop your own protocol. E.g.: first send the number of files, then send an array of 8-byte-encoded file lengths, and then the stream of file contents. The receiving end reads number of files, then parses file lengths. Knowing the lengths the receiver correctly splits the stream into files.
Use existing multi-file packaging formats: tar, cpio, zip, etc. Pack files before sending, then send the resulting package. On the receiving end unpack the package after receiving.
Recommended way Use existing protocols for sending files over TCP: TFTP or FTP.
I'd recommend using TFTP. It is very simple and reasonably efficient. There are several implementations in Python, such as tftpy
On the remote machine where you want to upload your files to, start TFTP server:
import tftpy
server = tftpy.TftpServer('<destination-folder>')
server.listen('0.0.0.0', 54321)
On the machine with files start the client:
import tftpy
client = tftpy.TftpClient('your.server.address', 54321)
for name in ("fileA", "fileB", "fileC"):
# the first argument is the name on the remote machine
# the second argument is the name on the local machine
client.upload(name, name)

Missing one-pixel row while transfering image with TCP socket

I'm facing strange error right now, I have python script, that is sending/receiving data using TCP socket, everything works fine, but when I'm trying to download image with this script, it will download it, but there is a missing one-pixel row. Any ideas on how to fix it?
Server download script:
def download(self, cmd):
try:
self.c.send(str.encode(cmd))
command,filename=cmd.split(' ')
nFile = open(filename, 'wb')
i = self.c.recv(1024)
while not ('complete' in str(i)):
nFile.write(i)
i = self.c.recv(1024)
nFile.close()
self.reset = True
print('\nGot that file')
except Exception as e:
print(e)
Client upload script:
def upload(self, filename):
try:
fd = open(filename, 'rb')
data = fd.read(1024)
while (data):
self.s.sendall(data)
data = fd.read(1024)
self.s.send(str.encode('complete'))
fd.close()
except Exception as e:
print(e)
EXAMPLE - You can see, that last row of pixels is missing:
SOLUTION(1): It's not a solution, just workaround, use the other one!
What happens if you remove the complete part of the payload before
writing the last chunk of data to nFile? – mtrw
The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.
Client upload script:
def upload(self, filename):
try:
fd = open(filename, 'rb')
data = fd.read(1024)
while (data):
self.s.sendall(data)
data = fd.read(1024)
sleep(0.2)
self.s.send(str.encode('complete'))
fd.close()
except Exception as e:
print(e)
SOLUTION(2):
TCP is a stream protocol with no message boundaries. This means that
multiple sends can be received in one recv call, or one send can be
received in multiple recv calls.
The delay work-around may not work reliably. You need to delimit
messages in the stream.
– Maxim Egorushkin
Server download script:
try:
msg_header = self.c.recv(4)
while len(msg_header) != 4:
msg_header += self.c.recv(4 - len(msg_header))
file_len = struct.unpack('<I', msg_header)[0]
nFile = open(filename, 'wb')
data = self.c.recv(file_len)
while len(data) != file_len:
data += self.c.recv(file_len - len(data))
nFile.write(data)
nFile.close()
print('\nGot that file')
except Exception as e:
print(e)
Client upload script:
try:
file_len = os.stat(filename).st_size
msg_header = struct.pack('<I', file_len)
self.s.sendall(msg_header)
fd = open(filename, 'rb')
data = fd.read(file_len)
while (data):
self.s.sendall(data)
data = fd.read(file_len)
fd.close()
except Exception as e:
print(e)

The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.
TCP is a stream protocol with no message boundaries. This means that multiple sends can be received in one recv call, or one send can be received in multiple recv calls.
The delay work-around may not work reliably. You need to delimit messages in the stream.
There are 2 common ways of delimiting messages in a stream:
Prefix messages with a header.
End messages with a suffix.
Since you are sending binary data any suffix can naturally be present in the payload. Unless the suffix is longer than the payload, which isn't practical.
Hence, what you may like to do here is prefix a fixed-size header to your payload. In this particular case, a header with a 4-byte binary file length would suffice. E.g.:
file_len = os.stat(filename).st_size
msg_header = struct.pack('<I', file_len)
self.s.sendall(msg_header)
The receiver needs to read the header first:
msg_header = self.s.recv(4)
while len(msg_header) != 4:
msg_header += self.s.recv(4 - len(msg_header))
file_len = struct.unpack('<I', msg_header)
And then read exactly file_len from the socket.
Knowing the size of the file being received also allows you to preallocate the buffer to avoid memory reallocations and/or preallocate the entire file to minimize disk fragmentation or avoid out of disk space error after the file transfer has started.

Receive files and save them on socket Python

My server is sending serial of files with name like file_1, file_2, and so on.
The sending socket works well and I've checked those files all correct.
Server - sending files
f = open(new_filename, 'rb')
start_ts = ts
seconds += 1
try:
print('Sending %s' % new_filename)
conn.sendall(f.read(99999999))
f.flush()
f.close()
except socket.error:
if errno == errno.ECONNREFUSED:
print(os.strerror(socket.error.errno))
else:
raise
print('Send failed')
sys.exit()
Here on the client side, I want to receive those files and save them as the way it was on the server (save_1, save_2, ...) on my computer.
Client- receive files and try to save them
try:
client.send(b'Receiving Data...\n')
while True:
save_filename = 'savefolder/save_%i.pcap' % file_index
f = open(save_filename, 'wb')
data = client.recv(99999999)
f.write(data)
reply = b'Message Received.\n'
if not data:
break
client.sendall(reply)
f.close()
file_index += 1
except socket.timeout:
print('Done receiving.', end=' ')
client.close()
print('Client socket is closed')
But the saved files look weird. When the server sends one 3,755 KB file, the client writes two files with 192KB and 3,563KB. Worse, the bigger one crashes. Is there a fix for this? I don't know why it happens here with my code.
Is it the only way to input 99999999 value to get the entire file without cutting?
Why is my client not sending message to the server even though I wrote some code in there?

TCP is not a message based protocol but a byte stream protocol. There is no fixed relation between how much was send or sendall and how much gets read with recv.
In order to send multiple messages (files) over the same connection you have to define some application protocol which clearly defines where messages start and end. And alternative would be to use a new TCP connection for each file and read until recv returns '', i.e. indicates that the other site has closed the connection.

Download several files from a local server to a client

The following codes let me download from server to client three files called tmp.bsp, tmp.seq and tmp.dms. However, just the first file tmp.dms is completely downloaded. The other one tmp.seq is filled up with the informations of tmp.bsp and tmp.bsp stay 0KB.
client:
import socket
import socket
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
sData = "Temp"
sData2 = "Temp"
sData3 = "Temp"
while True:
sData = skClient.recv(1024)
fDownloadFile = open("tmp.dms","wb")
sData2 = skClient.recv(1024)
fDownloadFile2 = open("tmp.seq","wb")
sData3 = skClient.recv(1024)
fDownloadFile3 = open("tmp.bsp","wb")
while sData:
fDownloadFile.write(sData)
sData = skClient.recv(1024)
fDownloadFile.close()
fDownloadFile2.write(sData2)
sData2 = skClient.recv(1024)
fDownloadFile2.close()
fDownloadFile3.write(sData3)
sData3 = skClient.recv(1024)
fDownloadFile3.close()
print "Download over"
break
skClient.close()
n is a counter and the prints are for debugging.
sFileName is to download one file, and used to work but since I want three files I just commented it.
server:
import socket
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
while True:
Content,Address = skServer.accept()
print Address
files = "C:\Users\Name_user\Desktop\Networking\Send_Receive/"
fUploadFile = open(files+str('tmp.dms'),"rb")
sRead = fUploadFile.read(1024)
fUploadFile2 = open(files+str('tmp.seq'),"rb")
sRead2 = fUploadFile2.read(1024)
fUploadFile3 = open(files+str('tmp.bsp'),"rb")
sRead3 = fUploadFile3.read(1024)
while sRead:
Content.send(sRead)
sRead = fUploadFile.read(1024)
Content.send(sRead2)
sRead2 = fUploadFile2.read(1024)
# Content.send(sRead3)
# sRead3 = fUploadFile3.read(1024)
Content.close()
print "Sending is over"
break
skServer.close()
files I'm using:
server2.py is my server
Execution

The main issue with your code is that you're sending / receiving an arbitrary number of data. If your buffer (1024) is smaller than the file size then the client's file will contain less information, and if it's larger the file may contain more information (data from the next file).
You could solve this issue by sending a value that signifies the end of a file. The problem with this method is that this value can't be contained in any file, and the client must be scanning the received data for this value.
Another possible solution is to calculate the file size and send that infomation in front of the file data. This way the cilent will know how many data to expect for each file.
Using struct.pack we can create a minimal four bytes header with the file size.
def send_file(soc, path):
with open(path, 'rb') as f:
data = f.read()
size = struct.pack('!I', len(data))
soc.send(size + data)
Tthen the client can get the file size by reading four bytes and unpacking to int.
def recv_file(soc, path):
size_header = soc.recv(4)
size = struct.unpack('!I', size_header)[0]
data = soc.recv(size)
with open(path, 'wb') as f:
f.write(data)
Note that sending/receiving files with one call may raise a socket error if the file size is larger than the socket buffer. In that case you'll have to read the data in smaller chunks in a loop, or increase the buffer size with socket.setsockopt.
Here is a modified version of the above functions that can handle large files:
import struct
import os.path
def send_file(soc, path):
file_size = os.path.getsize(path)
size_header = struct.pack('!Q', file_size)
soc.send(size_header)
with open(path, 'rb') as f:
while True:
data = f.read(1024)
if not data:
break
soc.send(data)
def recv_file(soc, path):
size_header = soc.recv(8)
file_size = struct.unpack('!Q', size_header)[0]
chunks = [1024 for i in range(file_size / 1024)]
with open(path, 'wb') as f:
for chunk in chunks:
f.write(soc.recv(chunk))
f.write(soc.recv(file_size % 1024))
I haven't tested this code thoroughly, but it should work for files of any size.
An example using the send_file function in your server:
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
Content,Address = skServer.accept()
print Address
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
send_file(Content, file)
Content.close()
print "Sending is over"
skServer.close()
Using recv_file in the client:
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
recv_file(skClient, file)
print "Download over"
skClient.close()

Yes you are right, I did run your program and found exactly same issue. I dont have enough time to work more on this issue but I found few key points which might lead you to the right work around.
https://docs.python.org/2/howto/sockets.html
The above official doc says:
When a recv returns 0 bytes, it means the other side has closed (or is in the process of closing) the connection. You will not receive any more data on this connection. Ever. You may be able to send data successfully
This is what it is happening when the third file returns 0 bytes.
But why 2nd and 3rd file is merged, I guess its because sockets are just buffered files and we might need to try making sure buffer is clear before sending another.
Read this,
Now there are two sets of verbs to use for communication. You can use send and recv, or you can transform your client socket into a file-like beast and use read and write. The latter is the way Java presents its sockets. I’m not going to talk about it here, except to warn you that you need to use flush on sockets. These are buffered “files”, and a common mistake is to write something, and then read for a reply. Without a flush in there, you may wait forever for the reply, because the request may still be in your output buffer.
But if you plan to reuse your socket for further transfers, you need to realize that there is no EOT on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there’s nothing more to read (for now). Now if you think about that a bit, you’ll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).
Hope this helps.

I'm not totally fluent in Python, but I think your while statement should be something like:
while: sData or sData2 or sData3
I may have the syntax wrong, but currently it looks like you will stop when "sData" is done and stop downloading sData2 and aData3 at that time even if they haven't finished.
Hmm--either that or the "While" isn't looping at all and it's just being used as an "if"? hard to tell without knowing the API.

Python 3 Socket Programming: saving multiple images

I have been working on an image transfer server and I've run into a problem with saving multiple images. The server can only save multiple images (from more than one client) if the socket on the client end closes which only allows that client to send one image. I've also tried to send the filesize and stop saving when the filesize is met, but the image then becomes impossible to open for whatever reason.
Client Code
f = open(filename, "rb")
while True:
data = f.read(512)
if not data:
break
s.send(data)
f.close()
Server Code
for client in clients:
while True:
data = client[0].recv(512)
if data == "":
break
f.write(data)

Try to send an empty string from the client after the transfer of the image is completed. I believe that the problem lies in the fact that the server does not distinguish how many images are transmitted.
Client:
for file in files:
with open(filename, "rb") as f:
data = f.read()
while data:
s.send(data[:512])
data = data[512:]
s.send('')
Server:
from functools import partial
while True:
with open(filename, "rb") as f:
data = b"".join(iter(partial(client[0].recv, 512), b''))
f.write(data)