My server is sending serial of files with name like file_1, file_2, and so on.
The sending socket works well and I've checked those files all correct.
Server - sending files
f = open(new_filename, 'rb')
start_ts = ts
seconds += 1
try:
print('Sending %s' % new_filename)
conn.sendall(f.read(99999999))
f.flush()
f.close()
except socket.error:
if errno == errno.ECONNREFUSED:
print(os.strerror(socket.error.errno))
else:
raise
print('Send failed')
sys.exit()
Here on the client side, I want to receive those files and save them as the way it was on the server (save_1, save_2, ...) on my computer.
Client- receive files and try to save them
try:
client.send(b'Receiving Data...\n')
while True:
save_filename = 'savefolder/save_%i.pcap' % file_index
f = open(save_filename, 'wb')
data = client.recv(99999999)
f.write(data)
reply = b'Message Received.\n'
if not data:
break
client.sendall(reply)
f.close()
file_index += 1
except socket.timeout:
print('Done receiving.', end=' ')
client.close()
print('Client socket is closed')
But the saved files look weird. When the server sends one 3,755 KB file, the client writes two files with 192KB and 3,563KB. Worse, the bigger one crashes. Is there a fix for this? I don't know why it happens here with my code.
Is it the only way to input 99999999 value to get the entire file without cutting?
Why is my client not sending message to the server even though I wrote some code in there?
TCP is not a message based protocol but a byte stream protocol. There is no fixed relation between how much was send or sendall and how much gets read with recv.
In order to send multiple messages (files) over the same connection you have to define some application protocol which clearly defines where messages start and end. And alternative would be to use a new TCP connection for each file and read until recv returns '', i.e. indicates that the other site has closed the connection.
Related
I'm facing strange error right now, I have python script, that is sending/receiving data using TCP socket, everything works fine, but when I'm trying to download image with this script, it will download it, but there is a missing one-pixel row. Any ideas on how to fix it?
Server download script:
def download(self, cmd):
try:
self.c.send(str.encode(cmd))
command,filename=cmd.split(' ')
nFile = open(filename, 'wb')
i = self.c.recv(1024)
while not ('complete' in str(i)):
nFile.write(i)
i = self.c.recv(1024)
nFile.close()
self.reset = True
print('\nGot that file')
except Exception as e:
print(e)
Client upload script:
def upload(self, filename):
try:
fd = open(filename, 'rb')
data = fd.read(1024)
while (data):
self.s.sendall(data)
data = fd.read(1024)
self.s.send(str.encode('complete'))
fd.close()
except Exception as e:
print(e)
EXAMPLE - You can see, that last row of pixels is missing:
SOLUTION(1): It's not a solution, just workaround, use the other one!
What happens if you remove the complete part of the payload before
writing the last chunk of data to nFile? – mtrw
The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.
Client upload script:
def upload(self, filename):
try:
fd = open(filename, 'rb')
data = fd.read(1024)
while (data):
self.s.sendall(data)
data = fd.read(1024)
sleep(0.2)
self.s.send(str.encode('complete'))
fd.close()
except Exception as e:
print(e)
SOLUTION(2):
TCP is a stream protocol with no message boundaries. This means that
multiple sends can be received in one recv call, or one send can be
received in multiple recv calls.
The delay work-around may not work reliably. You need to delimit
messages in the stream.
– Maxim Egorushkin
Server download script:
try:
msg_header = self.c.recv(4)
while len(msg_header) != 4:
msg_header += self.c.recv(4 - len(msg_header))
file_len = struct.unpack('<I', msg_header)[0]
nFile = open(filename, 'wb')
data = self.c.recv(file_len)
while len(data) != file_len:
data += self.c.recv(file_len - len(data))
nFile.write(data)
nFile.close()
print('\nGot that file')
except Exception as e:
print(e)
Client upload script:
try:
file_len = os.stat(filename).st_size
msg_header = struct.pack('<I', file_len)
self.s.sendall(msg_header)
fd = open(filename, 'rb')
data = fd.read(file_len)
while (data):
self.s.sendall(data)
data = fd.read(file_len)
fd.close()
except Exception as e:
print(e)
The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.
TCP is a stream protocol with no message boundaries. This means that multiple sends can be received in one recv call, or one send can be received in multiple recv calls.
The delay work-around may not work reliably. You need to delimit messages in the stream.
There are 2 common ways of delimiting messages in a stream:
Prefix messages with a header.
End messages with a suffix.
Since you are sending binary data any suffix can naturally be present in the payload. Unless the suffix is longer than the payload, which isn't practical.
Hence, what you may like to do here is prefix a fixed-size header to your payload. In this particular case, a header with a 4-byte binary file length would suffice. E.g.:
file_len = os.stat(filename).st_size
msg_header = struct.pack('<I', file_len)
self.s.sendall(msg_header)
The receiver needs to read the header first:
msg_header = self.s.recv(4)
while len(msg_header) != 4:
msg_header += self.s.recv(4 - len(msg_header))
file_len = struct.unpack('<I', msg_header)
And then read exactly file_len from the socket.
Knowing the size of the file being received also allows you to preallocate the buffer to avoid memory reallocations and/or preallocate the entire file to minimize disk fragmentation or avoid out of disk space error after the file transfer has started.
I'm currently writing a small client-server application for transferring an arbitrary file from a server to a client, via sockets.
The server will only handle one client at a time, but when a client is served it shall be ready to handle a new client connection.
The client will request a file, if the file exist, the client will receive the file, write it to disk and close the connection.
Server code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
print('The server is ready to receive')
server_socket = socket(AF_INET, SOCK_STREAM)
server_socket.bind(('', PORT))
server_socket.listen(1)
while True:
connection_socket, addr = server_socket.accept()
try:
requested_filepath = connection_socket.recv(BUFSIZE).decode()
print("Client requested the file: " + requested_filepath)
capital_sentence = requested_filepath.upper()
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(filesize.encode())
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode())
finally:
connection_socket.close()
Client code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
servername = argv[0]
filepath = argv[1]
client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect((servername, PORT))
try:
client_socket.send(filepath.encode())
response = client_socket.recv(BUFSIZE).decode()
if(response != "error"):
filesize = int(response)
print("Requested filesize: " + str(filesize))
filename = filepath.split('/')[-1]
with open(filename, 'wb') as f:
while(True):
content = client_socket.recv(BUFSIZE)
if not content:
break
f.write(content)
print('File recived')
else:
print("The requested file did not exist")
finally:
client_socket.close()
I can run the server and get the client to request and get a file, but when i run the client for a second or third time, the server and client seems to get out of sync. Both programs breaks and returns the following error message:
Client error:
Traceback (most recent call last):
File "client.py", line 37, in <module>
main(sys.argv[1:])
File "client.py", line 16, in main
response = client_socket.recv(BUFSIZE).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 6: invalid start byte
Server error:
The server is ready to receive
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
Traceback (most recent call last):
File "server.py", line 44, in <module>
main(sys.argv[1:])
File "server.py", line 30, in main
connection_socket.send(content)
ConnectionResetError: [Errno 104] Connection reset by peer
Am I not closing the socket connection in a proper way?
You have fallen into one of the most common TCP socket programming traps. You assumed your socket would send messages, while it sends and receives only data and is completely agnostic to your messaging structure. Even if you send data using several send calls, your recv calls do not receive this exact structure but whatever happens to be in the buffer. If you sent one byte a thousand times, your recv(1000) would receive a thousand bytes and this is what is going on here.
Your issue is caused by your server being a bit faster than your client. I had to tweak your code to be able to reproduce the code reliably but this does it:
client_socket.send(filepath.encode())
sleep(1)
response = client_socket.recv(BUFSIZE).decode()
This emulates your server being faster than the client, which eventually will happen anyway. By adding sleep we can make it happen every time.
When you call recv on a TCP socket, one of the following five things can happen:
There is no data and the call blocks
You received data and the data you received is exactly one "message", whatever that is in your context
Your server had sent more than one message before you read from the socket and you received them all on one go
Your client was too eager to read and it decided to read when only a part of your first message was available
Combination of 3 and 4: You receive several full messages plus one partial
What happens with your code is that your server has managed to send the encoded file size and some of your data as well. On your client you now assume your first recv receives only the file size, but this is no way guaranteed. There can be already some file data (as you will read BUFSIZE - there can be almost a full buffer of data there) and when you try to decode that as an integer, weird things happen as the data is not what you expected it to be.
The only reliable way to handle TCP sockets is to read from the socket, append to a temporary processing buffer, then parse that buffer and see what is in there. If there is a "message", process it and delete it from the buffer. Whatever remains in the buffer must stay there and your next recv result gets appended to this.
The simplest way to quickfix this is if your server makes the initial message of a fixed length. Then you can safely read exactly this amount of characters from the socket and process this as the size/error message, and the rest will be data. This is a horrible fix in many, many ways and you should aim for something better. The "proper" way is to devise a protocol, where the server puts delimiters in place so that your client can detect which message means what. Your protocol could be for example
SIZE: <decimal>\n
DATA: <data>
or even as simple as assuming everything before a newline is filesize and everything that follows is data.
But this works better even with sleep(1) added as it will now pad the initial message to exactly 100 bytes. This could still go wrong because of (4), so actually you will need to check that you received 100 characters initially and keep reading until you do, but I will leave this for you to implement.
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(("%s" % filesize).encode().ljust(100))
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode().ljust(100))
Client:
try:
client_socket.send(filepath.encode())
sleep(1)
response_raw = client_socket.recv(100)
response = response_raw.strip().decode()
PS your server should catch the "connection reset by peer" error. It is something that can happen if there is a network problem or the client application crashes. The server can safely ignore this error and just stop sending to that particular client socket.
I’m trying to send and receive file through TCP socket
There are a lot of problems
1. When the client connects to the server. The server does not “print Client connected ..” but it prints after using the command.
2. When I use the ‘put’ command at the server occur an error socket.error: [Errno 107] Transport endpoint is not connected but the file image is already uploaded to the server.
3. When I use the ‘get’ command at the client. I can’t continue to use another command.
4. The last problem is the client can’t quit and list file from the server. It shows AttributeError: 'module' object has no attribute 'send'
Server
import socket
import sys
import os
HOST = 'localhost'
PORT = 3820
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.bind((HOST, PORT))
socket.listen(1)
while (1):
conn, addr = socket.accept()
print 'Client connected ..'
reqCommand = conn.recv(2048)
print 'Client> %s' %(reqCommand)
if (reqCommand == 'quit'):
break
#list file on server
elif (reqCommand == 'lls'):
start_path = os.listdir('.') # server directory
for path,dirs,files in os.walk(start_path):
for filename in files:
print os.path.join(filename)
else:
string = reqCommand.split(' ', 1) #in case of 'put' and 'get' method
reqFile = string[1]
if (string[0] == 'put'):
with open(reqFile, 'wb') as file_to_write:
while True:
data = socket.recv(1024)
# print data
if not data:
break
# print data
file_to_write.write(data)
file_to_write.close()
break
print 'Receive Successful'
elif (string[0] == 'get'):
with open(reqFile, 'rb') as file_to_send:
for data in file_to_send:
conn.sendall(data)
print 'Send Successful'
conn.close()
socket.close()
Client
import socket
import sys
import os
HOST = 'localhost'
PORT = 3820
def put(commandName):
socket1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket1.connect((HOST, PORT))
socket1.send(commandName)
string = commandName.split(' ', 1)
inputFile = string[1]
with open(inputFile, 'rb') as file_to_send:
for data in file_to_send:
socket1.sendall(data)
print 'PUT Successful'
socket1.close()
return
def get(commandName):
socket1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket1.connect((HOST, PORT))
socket1.send(commandName)
string = commandName.split(' ', 1)
inputFile = string[1]
with open(inputFile, 'wb') as file_to_write:
while True:
data = socket1.recv(2048)
# print data
if not data:
break
# print data
file_to_write.write(data)
file_to_write.close()
print 'GET Successful'
socket1.close()
return
msg = raw_input('Enter your name: ')
while(1):
print 'Instructions'
print '"put [filename]" to upload the file to the server '
print '"get [filename]" to download the file from the server '
print '"ls" to list all files in the client'
print '"lls" to list all files in the server'
print '"quit" to exit'
sys.stdout.write('%s> ' % msg)
inputCommand = sys.stdin.readline().strip()
if (inputCommand == 'quit'):
socket.send('quit')
break
#list file on client
elif (inputCommand == 'ls'):
start_path = os.listdir('.') # client directory
print start_path
#list file on server
elif (inputCommand == 'lls'):
socket.send('lls')
else:
string = inputCommand.split(' ', 1)
if (string[0] == 'put'):
put(inputCommand)
elif (string[0] == 'get'):
get(inputCommand)
socket.close()
Among other things, you need to add "framing" to your transfer protocol. When you do a send on a stream socket, the data gets added to a buffer that will eventually be delivered to the other side. However, the size of that buffer is not transmitted to the other side. In other words, say you send an initial segment with the command "put myfile.txt". Then you send the data from myfile.txt. Now because you are using the file object iterator (for data in file_to_send), you are actually sending it a line at a time (arguably, for a file transfer protocol, it would make more sense to read and send fixed chunks but this would work too). Let's assume the first line of myfile.txt is "The quick brown fox\n"
When the server does its first receive, it could receive "put " or "put myfile.txt" or "put myfile.txtThe quick brown fox\n" or the put command plus the entire file contents. That's because the stream protocol (TCP) does not maintain message boundaries for you.
Now in practice, you may be receiving only the "put myfile.txt" in the first receive, but it's very unwise to count on that because it's dependent on the timing of all sorts of factors on both sending and receiving systems that are outside your control.
So, there are two common ways of handling this:
Add a length at the beginning that delineates the size of the command and any command argument (so that you know where in the stream the actual file data to be transferred begins). (Most binary file transfer protocols work this way.)
Add some known delimiter at the end of your command -- for example, '\n'. HTTP, for example, works this way.
And likewise your receiving side needs to ensure that it is reading exactly the amount needed at each point in order to preserve the full content of the file being transferred.
That means you either (a) are careful to recv exactly the number of bytes you need for the command data, then separately process the file content, or (b) recv an initial chunk of data into a buffer, then carve off exactly what you need for the "command", and ensure the rest will be processed later as file data. Option (b) can often be accomplished by building a file object on top of the socket (see socket.makefile), then using the file object's readline method to obtain only the first line.
Other issues with your code:
The line socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) is a very bad idea. You have just hidden the entire socket module by creating a variable with the same name. For example, if you attempted to refer to socket.AF_INET again on the subsequent line, you'd get the exception AttributeError: '_socketobject' object has no attribute 'AF_INET'. You should name the variable something else, say socket1 as you did on the client side.
On the client side, you have the opposite problem. You're attempting to use a socket object method, but providing the socket module object instead. (That's why you get AttributeError: 'module' object has no attribute 'send'.) You need to reorganize your code so that you call the send method on a connected socket object as you are doing in the put and get functions.
The error socket.error: [Errno 107] Transport endpoint is not connected occurs because you are attempting to recv on the listening socket, not the connected one (conn -- which is returned by socket.accept). The only thing you can do with a listening socket, is accept new connections (or close).
You should be using sendall instead of send to ensure that every byte gets sent. Generally, all the data will get sent with send too, but there are corner cases where that doesn't happen.
Your file-receiving loop in the server begins with while True: but then always breaks. Hence it will only receive the first chunk of data (up to 1024 bytes). If the file is larger than that, you will certainly end up truncating it.
Your server file listing function (lls command) is not sending its output back to the client, it's only printing it to the server side's console. (And don't forget, you will need to provide a delimiter of some kind between the file names you send back or they will end up all concatenated into a single big string.)
I am not sure if this topic have been answered or not, if was I am sorry:
I have a simple python script the sends all files in one folder:
Client:
import os,sys, socket, time
def Send(sok,data,end="292929"):
sok.sendall(data + end);
def SendFolder(sok,folder):
if(os.path.isdir(folder)):
files = os.listdir(folder);
os.chdir(folder);
for file_ in files:
Send(sok,file_);#Send the file name to the server
f = open(file_, "rb");
while(True):
d = f.read();#read file
if(d == ""):
break;
Send(sok, d, "");#Send file data
f.close();
time.sleep(0.8);#Problem here!!!!!!!!!!!
Send(sok,"");#Send termination to the server
time.sleep(1);#Wait the server to write the file
os.chdir("..");
Send(sok,"endfile");#let the server know that we finish sending files
else:
Send("endfile")#If not folder send termination
try:
sok1 = socket.socket();
sok1.connect(("192.168.1.121",4444))#local ip
time.sleep(1);
while(True):
Send(sok1,"Enter folder name to download: ");
r = sok1.recv(1024);
SendFolder(sok1,r);
time.sleep(0.5);
except BaseException, e:
print "Error: " + str(e);
os._exit(1);
Server:
import sys,os,socket,time
# receive data
def Receive(sock, end="292929"):
data = "";
while(True):
if(data.endswith(end)):
break;
else:
data = sock.recv(1024);
return data[:-len(end)];#return data less termination
def FolderReceive(sok):
while(True):
r = Receive(sok);# recv filename or folder termination("endfile")
if(r == "endfolder"):
print "Folder receive complete.";
break;
else:
filename = r;#file name
filedata = Receive(sok);# receive file data
f = open(filename,"wb");
f.write(filedata);
f.close();#finish to write the file
print "Received: " + filename;
try:
sok1 = socket.socket();
sok1.bind(("0.0.0.0",4444));
sok1.listen(5);
cl , addr = sok1.accept();#accepts connection
while(True):
r = Receive(cl);
sys.stdout.write("\n" + r);
next = raw_input();
cl.sendall(next);#send folder name to the client
FolderReceive(cl);
except BaseException, e:
print "Error: " + str(e);
os._exit(1);
I know this not best server ever...but is what I know. This just work for a folder with small files because if I send big files(like 5mb...) it crashes because the time the client waits for the server is not enough.
So my question is how can I send the files to the server without client need to wait??or know exactly how many time the client needs to wait for the server to receive the file?? Some code that does the same but handling any file size, any help?
TCP sockets are byte streams, not message streams. If you want to send a series of separate messages (like your separate files), you need to define a protocol, and then write a protocol handler. There is no way around that; just guessing at the timing or trying to take advantage of packet boundaries cannot possibly work.
The blog post linked above shows one way to do it. But you can do it with string delimiters if you want. But you have to deal with two problems:
The delimiter can appear anywhere in a read packet, not just at the end.
The delimiter can be split on packet boundaries—you may get "2929" at the end of one read, and the other "29" at the start of the next.
The usually way you do that is to accumulate a buffer, and search for the delimiter anywhere in the buffer. Something like this:
def message(sock, delimiter):
buf = ''
while True:
data = sock.read(4096)
if not data:
# If the socket closes with no delimiter, this will
# treat the last "partial file" as a complete file.
# If that's not what you want, just return, or raise.
yield buf
return
buf += data
messages = buf.split(delimiter)
for message in messages[:-1]:
yield message
buf = message[-1]
Meanwhile, you have another problem with your delimiter: There's nothing stopping it from appearing in the files you're trying to transmit. For example, what if you tried to send your script, or this web page?
That's one of the reasons that other protocols are often better than delimiters, but this isn't hard to deal with: just escape any delimiters found in the files. Since you're sending the whole file at once, you can just use replace on right before the sendall, and the reverse replace right before the split.
I have written two client and server python scripts were server is listening for client to send files following is my server code for listening files. The problems with the code i cannot separate filename from the file data. Following output is received data.txtMyName
def sendFileName(self):
self.clientsocket.send("name:" + self.filename)
print 'filename', self.filename
def sendFile(self):
f=open(self.filename,"rb")
data= f.read(1024)
while (data):
if(self.clientsocket.send(data)):
print "sending data"
data = f.read(1024)
#readByte = open(self.filename, "rb")
#data = readByte.read()
#readByte.close()
#self.gateway.send(data)
self.clientsocket.close()
f.close
def receiveFile(self,sock):
data = sock.recv(1024)
print 'filename', data.strip()
f = open(data.strip(), "wb")
data = sock.recv(1024)
while (data):
f.write(data)
data=sock.recv(1024)
f.close()
self.server_socket.close()
You have to create your own simple protocol on top of TCP/IP to get it work. The simplest I would imagine is to add one special character (for example 0x00) between filename and file content. Receiving site could detect this char and split filename and file content.
More complicated protocol could also send file size so the receiving site could detect when transmission is finally over and if all bytes were send.
It will also be good to send acknowledge message to sender that receiver got whole file.
There is lot of transfer file protocols (TFTP, FTP) and they exist for a reason, because pure TCP/IP without any protocol on top is useless.