I'm currently writing a small client-server application for transferring an arbitrary file from a server to a client, via sockets.
The server will only handle one client at a time, but when a client is served it shall be ready to handle a new client connection.
The client will request a file, if the file exist, the client will receive the file, write it to disk and close the connection.
Server code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
print('The server is ready to receive')
server_socket = socket(AF_INET, SOCK_STREAM)
server_socket.bind(('', PORT))
server_socket.listen(1)
while True:
connection_socket, addr = server_socket.accept()
try:
requested_filepath = connection_socket.recv(BUFSIZE).decode()
print("Client requested the file: " + requested_filepath)
capital_sentence = requested_filepath.upper()
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(filesize.encode())
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode())
finally:
connection_socket.close()
Client code:
PORT = 9000
BUFSIZE = 1000
def main(argv):
servername = argv[0]
filepath = argv[1]
client_socket = socket(AF_INET, SOCK_STREAM)
client_socket.connect((servername, PORT))
try:
client_socket.send(filepath.encode())
response = client_socket.recv(BUFSIZE).decode()
if(response != "error"):
filesize = int(response)
print("Requested filesize: " + str(filesize))
filename = filepath.split('/')[-1]
with open(filename, 'wb') as f:
while(True):
content = client_socket.recv(BUFSIZE)
if not content:
break
f.write(content)
print('File recived')
else:
print("The requested file did not exist")
finally:
client_socket.close()
I can run the server and get the client to request and get a file, but when i run the client for a second or third time, the server and client seems to get out of sync. Both programs breaks and returns the following error message:
Client error:
Traceback (most recent call last):
File "client.py", line 37, in <module>
main(sys.argv[1:])
File "client.py", line 16, in main
response = client_socket.recv(BUFSIZE).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 6: invalid start byte
Server error:
The server is ready to receive
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
File has been send
Client requested the file: /pepe.jpeg
Traceback (most recent call last):
File "server.py", line 44, in <module>
main(sys.argv[1:])
File "server.py", line 30, in main
connection_socket.send(content)
ConnectionResetError: [Errno 104] Connection reset by peer
Am I not closing the socket connection in a proper way?
You have fallen into one of the most common TCP socket programming traps. You assumed your socket would send messages, while it sends and receives only data and is completely agnostic to your messaging structure. Even if you send data using several send calls, your recv calls do not receive this exact structure but whatever happens to be in the buffer. If you sent one byte a thousand times, your recv(1000) would receive a thousand bytes and this is what is going on here.
Your issue is caused by your server being a bit faster than your client. I had to tweak your code to be able to reproduce the code reliably but this does it:
client_socket.send(filepath.encode())
sleep(1)
response = client_socket.recv(BUFSIZE).decode()
This emulates your server being faster than the client, which eventually will happen anyway. By adding sleep we can make it happen every time.
When you call recv on a TCP socket, one of the following five things can happen:
There is no data and the call blocks
You received data and the data you received is exactly one "message", whatever that is in your context
Your server had sent more than one message before you read from the socket and you received them all on one go
Your client was too eager to read and it decided to read when only a part of your first message was available
Combination of 3 and 4: You receive several full messages plus one partial
What happens with your code is that your server has managed to send the encoded file size and some of your data as well. On your client you now assume your first recv receives only the file size, but this is no way guaranteed. There can be already some file data (as you will read BUFSIZE - there can be almost a full buffer of data there) and when you try to decode that as an integer, weird things happen as the data is not what you expected it to be.
The only reliable way to handle TCP sockets is to read from the socket, append to a temporary processing buffer, then parse that buffer and see what is in there. If there is a "message", process it and delete it from the buffer. Whatever remains in the buffer must stay there and your next recv result gets appended to this.
The simplest way to quickfix this is if your server makes the initial message of a fixed length. Then you can safely read exactly this amount of characters from the socket and process this as the size/error message, and the rest will be data. This is a horrible fix in many, many ways and you should aim for something better. The "proper" way is to devise a protocol, where the server puts delimiters in place so that your client can detect which message means what. Your protocol could be for example
SIZE: <decimal>\n
DATA: <data>
or even as simple as assuming everything before a newline is filesize and everything that follows is data.
But this works better even with sleep(1) added as it will now pad the initial message to exactly 100 bytes. This could still go wrong because of (4), so actually you will need to check that you received 100 characters initially and keep reading until you do, but I will leave this for you to implement.
if(os.path.isfile(requested_filepath)):
filesize = str(os.path.getsize(requested_filepath))
connection_socket.send(("%s" % filesize).encode().ljust(100))
with open(requested_filepath, 'rb') as f:
while(True):
content = f.read(BUFSIZE)
if not content:
break
connection_socket.send(content)
print('File has been send')
else:
error = "error"
connection_socket.send(error.encode().ljust(100))
Client:
try:
client_socket.send(filepath.encode())
sleep(1)
response_raw = client_socket.recv(100)
response = response_raw.strip().decode()
PS your server should catch the "connection reset by peer" error. It is something that can happen if there is a network problem or the client application crashes. The server can safely ignore this error and just stop sending to that particular client socket.
Related
I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)
recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...
My server is sending serial of files with name like file_1, file_2, and so on.
The sending socket works well and I've checked those files all correct.
Server - sending files
f = open(new_filename, 'rb')
start_ts = ts
seconds += 1
try:
print('Sending %s' % new_filename)
conn.sendall(f.read(99999999))
f.flush()
f.close()
except socket.error:
if errno == errno.ECONNREFUSED:
print(os.strerror(socket.error.errno))
else:
raise
print('Send failed')
sys.exit()
Here on the client side, I want to receive those files and save them as the way it was on the server (save_1, save_2, ...) on my computer.
Client- receive files and try to save them
try:
client.send(b'Receiving Data...\n')
while True:
save_filename = 'savefolder/save_%i.pcap' % file_index
f = open(save_filename, 'wb')
data = client.recv(99999999)
f.write(data)
reply = b'Message Received.\n'
if not data:
break
client.sendall(reply)
f.close()
file_index += 1
except socket.timeout:
print('Done receiving.', end=' ')
client.close()
print('Client socket is closed')
But the saved files look weird. When the server sends one 3,755 KB file, the client writes two files with 192KB and 3,563KB. Worse, the bigger one crashes. Is there a fix for this? I don't know why it happens here with my code.
Is it the only way to input 99999999 value to get the entire file without cutting?
Why is my client not sending message to the server even though I wrote some code in there?
TCP is not a message based protocol but a byte stream protocol. There is no fixed relation between how much was send or sendall and how much gets read with recv.
In order to send multiple messages (files) over the same connection you have to define some application protocol which clearly defines where messages start and end. And alternative would be to use a new TCP connection for each file and read until recv returns '', i.e. indicates that the other site has closed the connection.
I’m trying to send and receive file through TCP socket
There are a lot of problems
1. When the client connects to the server. The server does not “print Client connected ..” but it prints after using the command.
2. When I use the ‘put’ command at the server occur an error socket.error: [Errno 107] Transport endpoint is not connected but the file image is already uploaded to the server.
3. When I use the ‘get’ command at the client. I can’t continue to use another command.
4. The last problem is the client can’t quit and list file from the server. It shows AttributeError: 'module' object has no attribute 'send'
Server
import socket
import sys
import os
HOST = 'localhost'
PORT = 3820
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.bind((HOST, PORT))
socket.listen(1)
while (1):
conn, addr = socket.accept()
print 'Client connected ..'
reqCommand = conn.recv(2048)
print 'Client> %s' %(reqCommand)
if (reqCommand == 'quit'):
break
#list file on server
elif (reqCommand == 'lls'):
start_path = os.listdir('.') # server directory
for path,dirs,files in os.walk(start_path):
for filename in files:
print os.path.join(filename)
else:
string = reqCommand.split(' ', 1) #in case of 'put' and 'get' method
reqFile = string[1]
if (string[0] == 'put'):
with open(reqFile, 'wb') as file_to_write:
while True:
data = socket.recv(1024)
# print data
if not data:
break
# print data
file_to_write.write(data)
file_to_write.close()
break
print 'Receive Successful'
elif (string[0] == 'get'):
with open(reqFile, 'rb') as file_to_send:
for data in file_to_send:
conn.sendall(data)
print 'Send Successful'
conn.close()
socket.close()
Client
import socket
import sys
import os
HOST = 'localhost'
PORT = 3820
def put(commandName):
socket1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket1.connect((HOST, PORT))
socket1.send(commandName)
string = commandName.split(' ', 1)
inputFile = string[1]
with open(inputFile, 'rb') as file_to_send:
for data in file_to_send:
socket1.sendall(data)
print 'PUT Successful'
socket1.close()
return
def get(commandName):
socket1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket1.connect((HOST, PORT))
socket1.send(commandName)
string = commandName.split(' ', 1)
inputFile = string[1]
with open(inputFile, 'wb') as file_to_write:
while True:
data = socket1.recv(2048)
# print data
if not data:
break
# print data
file_to_write.write(data)
file_to_write.close()
print 'GET Successful'
socket1.close()
return
msg = raw_input('Enter your name: ')
while(1):
print 'Instructions'
print '"put [filename]" to upload the file to the server '
print '"get [filename]" to download the file from the server '
print '"ls" to list all files in the client'
print '"lls" to list all files in the server'
print '"quit" to exit'
sys.stdout.write('%s> ' % msg)
inputCommand = sys.stdin.readline().strip()
if (inputCommand == 'quit'):
socket.send('quit')
break
#list file on client
elif (inputCommand == 'ls'):
start_path = os.listdir('.') # client directory
print start_path
#list file on server
elif (inputCommand == 'lls'):
socket.send('lls')
else:
string = inputCommand.split(' ', 1)
if (string[0] == 'put'):
put(inputCommand)
elif (string[0] == 'get'):
get(inputCommand)
socket.close()
Among other things, you need to add "framing" to your transfer protocol. When you do a send on a stream socket, the data gets added to a buffer that will eventually be delivered to the other side. However, the size of that buffer is not transmitted to the other side. In other words, say you send an initial segment with the command "put myfile.txt". Then you send the data from myfile.txt. Now because you are using the file object iterator (for data in file_to_send), you are actually sending it a line at a time (arguably, for a file transfer protocol, it would make more sense to read and send fixed chunks but this would work too). Let's assume the first line of myfile.txt is "The quick brown fox\n"
When the server does its first receive, it could receive "put " or "put myfile.txt" or "put myfile.txtThe quick brown fox\n" or the put command plus the entire file contents. That's because the stream protocol (TCP) does not maintain message boundaries for you.
Now in practice, you may be receiving only the "put myfile.txt" in the first receive, but it's very unwise to count on that because it's dependent on the timing of all sorts of factors on both sending and receiving systems that are outside your control.
So, there are two common ways of handling this:
Add a length at the beginning that delineates the size of the command and any command argument (so that you know where in the stream the actual file data to be transferred begins). (Most binary file transfer protocols work this way.)
Add some known delimiter at the end of your command -- for example, '\n'. HTTP, for example, works this way.
And likewise your receiving side needs to ensure that it is reading exactly the amount needed at each point in order to preserve the full content of the file being transferred.
That means you either (a) are careful to recv exactly the number of bytes you need for the command data, then separately process the file content, or (b) recv an initial chunk of data into a buffer, then carve off exactly what you need for the "command", and ensure the rest will be processed later as file data. Option (b) can often be accomplished by building a file object on top of the socket (see socket.makefile), then using the file object's readline method to obtain only the first line.
Other issues with your code:
The line socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) is a very bad idea. You have just hidden the entire socket module by creating a variable with the same name. For example, if you attempted to refer to socket.AF_INET again on the subsequent line, you'd get the exception AttributeError: '_socketobject' object has no attribute 'AF_INET'. You should name the variable something else, say socket1 as you did on the client side.
On the client side, you have the opposite problem. You're attempting to use a socket object method, but providing the socket module object instead. (That's why you get AttributeError: 'module' object has no attribute 'send'.) You need to reorganize your code so that you call the send method on a connected socket object as you are doing in the put and get functions.
The error socket.error: [Errno 107] Transport endpoint is not connected occurs because you are attempting to recv on the listening socket, not the connected one (conn -- which is returned by socket.accept). The only thing you can do with a listening socket, is accept new connections (or close).
You should be using sendall instead of send to ensure that every byte gets sent. Generally, all the data will get sent with send too, but there are corner cases where that doesn't happen.
Your file-receiving loop in the server begins with while True: but then always breaks. Hence it will only receive the first chunk of data (up to 1024 bytes). If the file is larger than that, you will certainly end up truncating it.
Your server file listing function (lls command) is not sending its output back to the client, it's only printing it to the server side's console. (And don't forget, you will need to provide a delimiter of some kind between the file names you send back or they will end up all concatenated into a single big string.)
I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'
You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.
First I am sending the requested file from the server to the client and after that I want to send the computed sha of the file from the server to the client, so that the client can check if both the sha from the sent and the received files are the same.
I manage to send the file but when I try to also send the sha (which is a variable) I receive a error ( i believe that the sha is also added to the file content)
How can i send them separately?
if (reqCommand == 'get'):
with open (reqFile, 'rb') as in_file, open(encFile, "wb") as out_file:
encrypt(in_file, out_file, "abc")
f = open(encFile,'rb')
for data in f:
# print 'here3'
conn.sendall(data)
f.close()
file_sh = hashfile(reqFile)
print 'the sha1 function from the server: ', file_sh
conn.send(file_sh)
and the client:
while True:
data = sock.recv(1024)
if not data:
break
#print data
file_to_write.write(data)
You should redesign a bit how your app works:
First the servers sends to the client the file size
The client reads the file size (converts it to a number) and notifies the server (sends "OK" to the server for example)
The server reads the "OK" from the client and starts to send the file contents (preferably in smaller chunks)
The client keeps reading data until either it reads exactly "file size" bytes or error occurs
If no error occurred the client computes the hash of the file that it just received and sends it to the server
The server reads the hash from client and compares with the one of its local file - if they match it sends "OK" to the client "ERROR" otherwise
The client reads the response from server: if "ERROR" is received the file is deleted
A TCP stream needs to be formatted in both ends, since it is a stream not packets. I suppose you could look for a nullbyte ('\x00') which should signal end of file.