I'm facing strange error right now, I have python script, that is sending/receiving data using TCP socket, everything works fine, but when I'm trying to download image with this script, it will download it, but there is a missing one-pixel row. Any ideas on how to fix it?
Server download script:
def download(self, cmd):
try:
self.c.send(str.encode(cmd))
command,filename=cmd.split(' ')
nFile = open(filename, 'wb')
i = self.c.recv(1024)
while not ('complete' in str(i)):
nFile.write(i)
i = self.c.recv(1024)
nFile.close()
self.reset = True
print('\nGot that file')
except Exception as e:
print(e)
Client upload script:
def upload(self, filename):
try:
fd = open(filename, 'rb')
data = fd.read(1024)
while (data):
self.s.sendall(data)
data = fd.read(1024)
self.s.send(str.encode('complete'))
fd.close()
except Exception as e:
print(e)
EXAMPLE - You can see, that last row of pixels is missing:
SOLUTION(1): It's not a solution, just workaround, use the other one!
What happens if you remove the complete part of the payload before
writing the last chunk of data to nFile? – mtrw
The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.
Client upload script:
def upload(self, filename):
try:
fd = open(filename, 'rb')
data = fd.read(1024)
while (data):
self.s.sendall(data)
data = fd.read(1024)
sleep(0.2)
self.s.send(str.encode('complete'))
fd.close()
except Exception as e:
print(e)
SOLUTION(2):
TCP is a stream protocol with no message boundaries. This means that
multiple sends can be received in one recv call, or one send can be
received in multiple recv calls.
The delay work-around may not work reliably. You need to delimit
messages in the stream.
– Maxim Egorushkin
Server download script:
try:
msg_header = self.c.recv(4)
while len(msg_header) != 4:
msg_header += self.c.recv(4 - len(msg_header))
file_len = struct.unpack('<I', msg_header)[0]
nFile = open(filename, 'wb')
data = self.c.recv(file_len)
while len(data) != file_len:
data += self.c.recv(file_len - len(data))
nFile.write(data)
nFile.close()
print('\nGot that file')
except Exception as e:
print(e)
Client upload script:
try:
file_len = os.stat(filename).st_size
msg_header = struct.pack('<I', file_len)
self.s.sendall(msg_header)
fd = open(filename, 'rb')
data = fd.read(file_len)
while (data):
self.s.sendall(data)
data = fd.read(file_len)
fd.close()
except Exception as e:
print(e)
The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.
TCP is a stream protocol with no message boundaries. This means that multiple sends can be received in one recv call, or one send can be received in multiple recv calls.
The delay work-around may not work reliably. You need to delimit messages in the stream.
There are 2 common ways of delimiting messages in a stream:
Prefix messages with a header.
End messages with a suffix.
Since you are sending binary data any suffix can naturally be present in the payload. Unless the suffix is longer than the payload, which isn't practical.
Hence, what you may like to do here is prefix a fixed-size header to your payload. In this particular case, a header with a 4-byte binary file length would suffice. E.g.:
file_len = os.stat(filename).st_size
msg_header = struct.pack('<I', file_len)
self.s.sendall(msg_header)
The receiver needs to read the header first:
msg_header = self.s.recv(4)
while len(msg_header) != 4:
msg_header += self.s.recv(4 - len(msg_header))
file_len = struct.unpack('<I', msg_header)
And then read exactly file_len from the socket.
Knowing the size of the file being received also allows you to preallocate the buffer to avoid memory reallocations and/or preallocate the entire file to minimize disk fragmentation or avoid out of disk space error after the file transfer has started.
Related
I'm writing a file transfer program between two sockets.
The sender has the usual game:
initiate file send
pick file to send, send filesize over socket.
pick recipient
confirm
Then the server reaches out to the recipient:
do you want to accept file of this filesize?
wait for y or n.
Recipient:
yes
receive.
and all the edge cases therein.
I am running a send thread and receive thread on both socket and server.
on the server I'm doing something like this.
import threading
accept_thread = Thread(target=accept)
send_thread = Thread(target=send)
def accept():
while True:
data = client.recv(BUFF)
# break if no data
if not data: break
# if command is /sendfile
if data == b'/sendfile':
# get filesize
filesize = client.recv(BUFF)
# get recipient
recipient = client.recv(BUFF)
# check if recipient is in address list.
if recipient_exists(recipient):
sender_socket.send(b'valid')
ask = recipient_socket.send(b'Do you want to accept?')
if ask == 'n':
sender_socket.send(b'rejected')
else:
data = sender_socket.recv(BUFF)
recipient_socket.send(data)
msg = data
clients.send(msg.encode())
Needless to say, things somehow go out of whack. I keep getting my sendfile tagged onto the end of whatever I send. For instance, if I send a text file, the last bytes will always be '/sendfile'.
And if the sender types anything when they're not supposed to, then that gets added to the file transfer as well.
What do I need to use in order to make this work? I tried putting a thread lock before the receive.
Edit: Here is my send and receive:
def xfer_file(self, path, server_socket):
"""Opens file at path, sends to server_socket."""
with open(path, 'rb') as f:
server_socket.sendfile(f, 0, self.filesize)
def receive_file(self, data, BUFFSIZE, filesize, client):
"""Download file transfer from server."""
try:
data = client.recv(BUFFSIZE)
bytes_recd = len(data)
print("STARTING=========")
# target_path = self._set_target_path()
with open('image(2).jpg', 'wb') as f:
while bytes_recd < int(filesize):
f.write(data)
data = client.recv(BUFFSIZE)
bytes_recd = bytes_recd + len(data)
print(bytes_recd)
print(data)
if data == b'':
raise RuntimeError("socket connection broken")
except ValueError:
pass
print("FINISHED==========")
```
My server is sending serial of files with name like file_1, file_2, and so on.
The sending socket works well and I've checked those files all correct.
Server - sending files
f = open(new_filename, 'rb')
start_ts = ts
seconds += 1
try:
print('Sending %s' % new_filename)
conn.sendall(f.read(99999999))
f.flush()
f.close()
except socket.error:
if errno == errno.ECONNREFUSED:
print(os.strerror(socket.error.errno))
else:
raise
print('Send failed')
sys.exit()
Here on the client side, I want to receive those files and save them as the way it was on the server (save_1, save_2, ...) on my computer.
Client- receive files and try to save them
try:
client.send(b'Receiving Data...\n')
while True:
save_filename = 'savefolder/save_%i.pcap' % file_index
f = open(save_filename, 'wb')
data = client.recv(99999999)
f.write(data)
reply = b'Message Received.\n'
if not data:
break
client.sendall(reply)
f.close()
file_index += 1
except socket.timeout:
print('Done receiving.', end=' ')
client.close()
print('Client socket is closed')
But the saved files look weird. When the server sends one 3,755 KB file, the client writes two files with 192KB and 3,563KB. Worse, the bigger one crashes. Is there a fix for this? I don't know why it happens here with my code.
Is it the only way to input 99999999 value to get the entire file without cutting?
Why is my client not sending message to the server even though I wrote some code in there?
TCP is not a message based protocol but a byte stream protocol. There is no fixed relation between how much was send or sendall and how much gets read with recv.
In order to send multiple messages (files) over the same connection you have to define some application protocol which clearly defines where messages start and end. And alternative would be to use a new TCP connection for each file and read until recv returns '', i.e. indicates that the other site has closed the connection.
The following codes let me download from server to client three files called tmp.bsp, tmp.seq and tmp.dms. However, just the first file tmp.dms is completely downloaded. The other one tmp.seq is filled up with the informations of tmp.bsp and tmp.bsp stay 0KB.
client:
import socket
import socket
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
sData = "Temp"
sData2 = "Temp"
sData3 = "Temp"
while True:
sData = skClient.recv(1024)
fDownloadFile = open("tmp.dms","wb")
sData2 = skClient.recv(1024)
fDownloadFile2 = open("tmp.seq","wb")
sData3 = skClient.recv(1024)
fDownloadFile3 = open("tmp.bsp","wb")
while sData:
fDownloadFile.write(sData)
sData = skClient.recv(1024)
fDownloadFile.close()
fDownloadFile2.write(sData2)
sData2 = skClient.recv(1024)
fDownloadFile2.close()
fDownloadFile3.write(sData3)
sData3 = skClient.recv(1024)
fDownloadFile3.close()
print "Download over"
break
skClient.close()
n is a counter and the prints are for debugging.
sFileName is to download one file, and used to work but since I want three files I just commented it.
server:
import socket
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
while True:
Content,Address = skServer.accept()
print Address
files = "C:\Users\Name_user\Desktop\Networking\Send_Receive/"
fUploadFile = open(files+str('tmp.dms'),"rb")
sRead = fUploadFile.read(1024)
fUploadFile2 = open(files+str('tmp.seq'),"rb")
sRead2 = fUploadFile2.read(1024)
fUploadFile3 = open(files+str('tmp.bsp'),"rb")
sRead3 = fUploadFile3.read(1024)
while sRead:
Content.send(sRead)
sRead = fUploadFile.read(1024)
Content.send(sRead2)
sRead2 = fUploadFile2.read(1024)
# Content.send(sRead3)
# sRead3 = fUploadFile3.read(1024)
Content.close()
print "Sending is over"
break
skServer.close()
files I'm using:
server2.py is my server
Execution
The main issue with your code is that you're sending / receiving an arbitrary number of data. If your buffer (1024) is smaller than the file size then the client's file will contain less information, and if it's larger the file may contain more information (data from the next file).
You could solve this issue by sending a value that signifies the end of a file. The problem with this method is that this value can't be contained in any file, and the client must be scanning the received data for this value.
Another possible solution is to calculate the file size and send that infomation in front of the file data. This way the cilent will know how many data to expect for each file.
Using struct.pack we can create a minimal four bytes header with the file size.
def send_file(soc, path):
with open(path, 'rb') as f:
data = f.read()
size = struct.pack('!I', len(data))
soc.send(size + data)
Tthen the client can get the file size by reading four bytes and unpacking to int.
def recv_file(soc, path):
size_header = soc.recv(4)
size = struct.unpack('!I', size_header)[0]
data = soc.recv(size)
with open(path, 'wb') as f:
f.write(data)
Note that sending/receiving files with one call may raise a socket error if the file size is larger than the socket buffer. In that case you'll have to read the data in smaller chunks in a loop, or increase the buffer size with socket.setsockopt.
Here is a modified version of the above functions that can handle large files:
import struct
import os.path
def send_file(soc, path):
file_size = os.path.getsize(path)
size_header = struct.pack('!Q', file_size)
soc.send(size_header)
with open(path, 'rb') as f:
while True:
data = f.read(1024)
if not data:
break
soc.send(data)
def recv_file(soc, path):
size_header = soc.recv(8)
file_size = struct.unpack('!Q', size_header)[0]
chunks = [1024 for i in range(file_size / 1024)]
with open(path, 'wb') as f:
for chunk in chunks:
f.write(soc.recv(chunk))
f.write(soc.recv(file_size % 1024))
I haven't tested this code thoroughly, but it should work for files of any size.
An example using the send_file function in your server:
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
Content,Address = skServer.accept()
print Address
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
send_file(Content, file)
Content.close()
print "Sending is over"
skServer.close()
Using recv_file in the client:
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
recv_file(skClient, file)
print "Download over"
skClient.close()
Yes you are right, I did run your program and found exactly same issue. I dont have enough time to work more on this issue but I found few key points which might lead you to the right work around.
https://docs.python.org/2/howto/sockets.html
The above official doc says:
When a recv returns 0 bytes, it means the other side has closed (or is in the process of closing) the connection. You will not receive any more data on this connection. Ever. You may be able to send data successfully
This is what it is happening when the third file returns 0 bytes.
But why 2nd and 3rd file is merged, I guess its because sockets are just buffered files and we might need to try making sure buffer is clear before sending another.
Read this,
Now there are two sets of verbs to use for communication. You can use send and recv, or you can transform your client socket into a file-like beast and use read and write. The latter is the way Java presents its sockets. I’m not going to talk about it here, except to warn you that you need to use flush on sockets. These are buffered “files”, and a common mistake is to write something, and then read for a reply. Without a flush in there, you may wait forever for the reply, because the request may still be in your output buffer.
But if you plan to reuse your socket for further transfers, you need to realize that there is no EOT on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there’s nothing more to read (for now). Now if you think about that a bit, you’ll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).
Hope this helps.
I'm not totally fluent in Python, but I think your while statement should be something like:
while: sData or sData2 or sData3
I may have the syntax wrong, but currently it looks like you will stop when "sData" is done and stop downloading sData2 and aData3 at that time even if they haven't finished.
Hmm--either that or the "While" isn't looping at all and it's just being used as an "if"? hard to tell without knowing the API.
I am not sure if this topic have been answered or not, if was I am sorry:
I have a simple python script the sends all files in one folder:
Client:
import os,sys, socket, time
def Send(sok,data,end="292929"):
sok.sendall(data + end);
def SendFolder(sok,folder):
if(os.path.isdir(folder)):
files = os.listdir(folder);
os.chdir(folder);
for file_ in files:
Send(sok,file_);#Send the file name to the server
f = open(file_, "rb");
while(True):
d = f.read();#read file
if(d == ""):
break;
Send(sok, d, "");#Send file data
f.close();
time.sleep(0.8);#Problem here!!!!!!!!!!!
Send(sok,"");#Send termination to the server
time.sleep(1);#Wait the server to write the file
os.chdir("..");
Send(sok,"endfile");#let the server know that we finish sending files
else:
Send("endfile")#If not folder send termination
try:
sok1 = socket.socket();
sok1.connect(("192.168.1.121",4444))#local ip
time.sleep(1);
while(True):
Send(sok1,"Enter folder name to download: ");
r = sok1.recv(1024);
SendFolder(sok1,r);
time.sleep(0.5);
except BaseException, e:
print "Error: " + str(e);
os._exit(1);
Server:
import sys,os,socket,time
# receive data
def Receive(sock, end="292929"):
data = "";
while(True):
if(data.endswith(end)):
break;
else:
data = sock.recv(1024);
return data[:-len(end)];#return data less termination
def FolderReceive(sok):
while(True):
r = Receive(sok);# recv filename or folder termination("endfile")
if(r == "endfolder"):
print "Folder receive complete.";
break;
else:
filename = r;#file name
filedata = Receive(sok);# receive file data
f = open(filename,"wb");
f.write(filedata);
f.close();#finish to write the file
print "Received: " + filename;
try:
sok1 = socket.socket();
sok1.bind(("0.0.0.0",4444));
sok1.listen(5);
cl , addr = sok1.accept();#accepts connection
while(True):
r = Receive(cl);
sys.stdout.write("\n" + r);
next = raw_input();
cl.sendall(next);#send folder name to the client
FolderReceive(cl);
except BaseException, e:
print "Error: " + str(e);
os._exit(1);
I know this not best server ever...but is what I know. This just work for a folder with small files because if I send big files(like 5mb...) it crashes because the time the client waits for the server is not enough.
So my question is how can I send the files to the server without client need to wait??or know exactly how many time the client needs to wait for the server to receive the file?? Some code that does the same but handling any file size, any help?
TCP sockets are byte streams, not message streams. If you want to send a series of separate messages (like your separate files), you need to define a protocol, and then write a protocol handler. There is no way around that; just guessing at the timing or trying to take advantage of packet boundaries cannot possibly work.
The blog post linked above shows one way to do it. But you can do it with string delimiters if you want. But you have to deal with two problems:
The delimiter can appear anywhere in a read packet, not just at the end.
The delimiter can be split on packet boundaries—you may get "2929" at the end of one read, and the other "29" at the start of the next.
The usually way you do that is to accumulate a buffer, and search for the delimiter anywhere in the buffer. Something like this:
def message(sock, delimiter):
buf = ''
while True:
data = sock.read(4096)
if not data:
# If the socket closes with no delimiter, this will
# treat the last "partial file" as a complete file.
# If that's not what you want, just return, or raise.
yield buf
return
buf += data
messages = buf.split(delimiter)
for message in messages[:-1]:
yield message
buf = message[-1]
Meanwhile, you have another problem with your delimiter: There's nothing stopping it from appearing in the files you're trying to transmit. For example, what if you tried to send your script, or this web page?
That's one of the reasons that other protocols are often better than delimiters, but this isn't hard to deal with: just escape any delimiters found in the files. Since you're sending the whole file at once, you can just use replace on right before the sendall, and the reverse replace right before the split.
I have written two client and server python scripts were server is listening for client to send files following is my server code for listening files. The problems with the code i cannot separate filename from the file data. Following output is received data.txtMyName
def sendFileName(self):
self.clientsocket.send("name:" + self.filename)
print 'filename', self.filename
def sendFile(self):
f=open(self.filename,"rb")
data= f.read(1024)
while (data):
if(self.clientsocket.send(data)):
print "sending data"
data = f.read(1024)
#readByte = open(self.filename, "rb")
#data = readByte.read()
#readByte.close()
#self.gateway.send(data)
self.clientsocket.close()
f.close
def receiveFile(self,sock):
data = sock.recv(1024)
print 'filename', data.strip()
f = open(data.strip(), "wb")
data = sock.recv(1024)
while (data):
f.write(data)
data=sock.recv(1024)
f.close()
self.server_socket.close()
You have to create your own simple protocol on top of TCP/IP to get it work. The simplest I would imagine is to add one special character (for example 0x00) between filename and file content. Receiving site could detect this char and split filename and file content.
More complicated protocol could also send file size so the receiving site could detect when transmission is finally over and if all bytes were send.
It will also be good to send acknowledge message to sender that receiver got whole file.
There is lot of transfer file protocols (TFTP, FTP) and they exist for a reason, because pure TCP/IP without any protocol on top is useless.