Develop Reference

Multiple clients file transfer in python client-server without threading

I am trying multiple clients to send files to the server simultaneously on one port(i.e. server is running different ports and multiple clients are connected to each port and sending files). I have looked several answers such as this, but they are using different approaches and I just want somebody to pinpoint what I am doing wrong here, so I can use same code which I understand better. Please help me with:
Why my code is not working with multiple file transfer?
I am also calculating the throughput(i.e. actual file transfer), is it the correct method?
Thanks for help.
----- server.py ---
import socket,time
import sys, optparse,datetime
#def client(net,src,dst):
#def server(net,src):
print("we are in server ...")
parser = optparse.OptionParser()
parser.add_option('-i',dest='ip',default='')
parser.add_option('-p',dest='port',type='int',default=5001)
parser.add_option('-t',dest='ftype',type='str',default='.txt')
(options,args) = parser.parse_args()
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
host = socket.gethostname()
server_socket.bind((options.ip, options.port))
server_socket.listen(100)
s = datetime.datetime.now().strftime("%d%m%y_%H%M%S")
f = open('.recfile_%s_%s'%(s,options.port)+options.ftype,'wb')
count = 0
while 1:
client_socket, addr = server_socket.accept()
start_time = datetime.datetime.now()
cl_addr = addr[0]
print 'Got connection from', addr
print("Receiving ...")
data = client_socket.recv(1024)
while(data):
f.write(data)
data = client_socket.recv(1024)
count+=len(data)
continue
f.close()
client_socket.close()
end_time = datetime.datetime.now()
total_time = end_time - start_time
total_time = total_time.total_seconds()
average_speed = round((1024*count*0.001)/(total_time),3)
fd = open('server_data.csv','a+')
fd.write(str(cl_addr)+','+str(start_time)+','+str(end_time)+','+str(total_time)+','+str(average_speed)+','+str(options.port)+'\n\r')
fd.close()
server_socket.close()
client side
----- client.py -----
import socket
import sys, optparse
#def client(net,src,dst):
print("we are in client ..")
parser = optparse.OptionParser()
parser.add_option('-i',dest='ip',default='')
parser.add_option('-p',dest='port',type='int',default=5001)
parser.add_option('-f',dest='fname',type='str',default='hugefile.txt')
(options,args) = parser.parse_args()
client_socket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client_socket.connect((options.ip,options.port))
img_file = options.fname
f = open(img_file,'rb')
data = f.read(1024)
while(data):
client_socket.send(data)
data = f.read(1024)
f.close()
client_socket.shutdown(socket.SHUT_WR)
client_socket.close()
print "Data Sent successfully!"

There is at least one problem: the recfile file is opened before starting the loop, and closed inside the loop. That means that beginning from the second iteration, you will try to write on a closed file and get an exception.
How to avoid: with open(...) as ...: blocks are great because not only they guarantee proper closure in case of errors, but they also ensure a correct bloc structure in your program.
BTW, count should also be reset to 0 inside the loop, and the closer to the loop the better for future readers and maintainers of the code

I found a solution by improvising this .
Multiple connection to same socket is not possible without either multiprocessing or multithreading. Since I am using Python 2.7 multithreading is not an option for me.

How to send multiple music files using sockets? [duplicate]

This question already has an answer here:
sending multiple files in python
(1 answer)
Closed 3 years ago.
I am using python 3.6 and ubuntu 18.04.
I am able to send single music file using socket-python (in binary mode) and i want to send multiple music files from server to client.
Problem is, at the receiver end (that is client), all the music files (approx 120 files sent from server) gets collected in one single file making it a 9 hour long single music file.
I have tried using time.sleep method (does not work), tried sending bogus element (error was shown) and tried sending some random character to end the file writing at the client side and initiate new file write (but random character requires encoding and decoding, so again error as binary data was unable to decode).
SERVER CODE
import socket
import os
import send_file
import time
s = socket.socket()
host = ""
port = 9997
s.bind((host, port))
s.listen(5)
print("Binding Done\n")
socket_object, address = s.accept()
print("Connection Established\n")
print("Sending file...")
file_class = send_file.send_files() #ignore
file_names = file_class.files #ignore - contains list of path of music file
socket_object.sendall( str(len(file_names)).encode() )
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
print("Files Send")
socket_object.close()
s.close()
CLIENT CODE
import socket
import os
import time
def recv_file(i):
f = open("/home/ravi/PycharmProjects/File_Transfer/B/"+"M"+str(i)+".mp3", 'wb')
buf = s.recv(1024)
while buf:
f.write(buf)
buf = s.recv(1024)
f.close()
s = socket.socket()
host = "127.0.0.1"
port = 9997
s.connect((host, port))
print("Receiving data...")
l = s.recv(1024).decode() #ignore - length of total number of files i.e., 120 approx
for i in range(int(l)):
recv_file(i+1)
print("Files Collected")
s.close()
Any suggestions would be appreciated. Thank You.

TCP is a stream-oriented protocol. It is designed for sending streams of data. At TCP level there are no files. It doesn't split streams in any file-oriented chunks.
Look at your code:
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
You just glue all files in a single stream, and your client has no idea when one file ends and the next file starts.
The task of sending multiple files over TCP could be solved in many ways:
Develop your own protocol. E.g.: first send the number of files, then send an array of 8-byte-encoded file lengths, and then the stream of file contents. The receiving end reads number of files, then parses file lengths. Knowing the lengths the receiver correctly splits the stream into files.
Use existing multi-file packaging formats: tar, cpio, zip, etc. Pack files before sending, then send the resulting package. On the receiving end unpack the package after receiving.
Recommended way Use existing protocols for sending files over TCP: TFTP or FTP.
I'd recommend using TFTP. It is very simple and reasonably efficient. There are several implementations in Python, such as tftpy
On the remote machine where you want to upload your files to, start TFTP server:
import tftpy
server = tftpy.TftpServer('<destination-folder>')
server.listen('0.0.0.0', 54321)
On the machine with files start the client:
import tftpy
client = tftpy.TftpClient('your.server.address', 54321)
for name in ("fileA", "fileB", "fileC"):
# the first argument is the name on the remote machine
# the second argument is the name on the local machine
client.upload(name, name)

Python 3: Sending files through socket. (Client-Server Program)

I am having the above issue. The client is suppose to ask for a filename and send the file name to the server after which the server will open the file and display it. Problem is that the server isn't opening the file and displaying it.
Below is the client.
#!/usr/bin/env python3
import socket, os.path, datetime, sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.connect((host, port))
Filename = input("Type in ur file ")
s.send(Filename.encode('utf-8'))
data = s.recv(1024).decode('utf-8')
s.close()
if __name__ == '__main__':
Main()
Below is the server
#!/usr/bin/env python3
import socket
import os
import sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.bind((host,port))
print("server Started")
s.listen(1)
c, addr = s.accept()
print("Connection from: " + str(addr))
while True:
data = c.recv(1024).decode('utf-8')
myfile = open(data, "r")
if not data:
break
print("from connected user: " + myfile)
c.close()
if __name__ == '__main__':
Main()

I've made few minimal adjustments to your code with which it runs as so that server.py continuously listens on a given port and sends back data which each invocation of client.py asks for.
server.py
#!/usr/bin/env python3
import socket
import os
import sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.bind((host,port))
print("server Started")
s.listen(1)
while True:
c, addr = s.accept()
print("Connection from: " + str(addr))
filename = ''
while True:
data = c.recv(1024).decode('utf-8')
if not data:
break
filename += data
print("from connected user: " + filename)
myfile = open(filename, "rb")
c.send(myfile.read())
c.close()
if __name__ == '__main__':
Main()
client.py
#!/usr/bin/env python3
import socket, os.path, datetime, sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.connect((host, port))
Filename = input("Type in ur file ")
s.send(Filename.encode('utf-8'))
s.shutdown(socket.SHUT_WR)
data = s.recv(1024).decode('utf-8')
print(data)
s.close()
if __name__ == '__main__':
Main()
And now a bit of explanation.
On the server side. The outer loop accepts a connection, then reads from the connection until done (more on this later). Prints your debugging info, but note you were trying to print the file object and not the filename (which would fail trying to concatenate). I also open the file in binary mode (that way I can skip the str -> bytes translation.
On the client side. I've added closing the writing end of the socket when the file has been sent. Note you might want to use sendall instead of send for this use case: check those docs links for details. And I've added a print for the incoming data.
Now that bit with shutting down the writing end in the client and the inner loop reading (and also related to the sendall hint. Which BTW also holds true for the server side, otherwise you should loop, as you might see your content truncated; other option is to also have a sending loop.). Stream sockets will guarantee you get your bytes in in order you've send them. On itself it has no idea whether your message is complete and it also does not guarantee in how many and how large chunks will the data be sent and received (resp.).
The inner loop of server keep reading until we see an EOF (we've receive zero length string in python socket). This would happen (be returned by recv when the remote socket (or at least its writing end) has been shut down. Since we still want to reuse the connection, we only do that on the sending end in the client. Hope this helps you to move ahead.

Need to transfer multiple files from client to server

I'm recently working on a project in which I'm basically making a dropbox clone. The server and client are working fine but I'm having a slight issue. I'm able to transfer a single file from the client to the server but when I try to transfer all the files together it gives me an error after the transfer of the first file so basically my code is only working for a single file. I need to make it work for multiple files. Any help will be appreciated. Here's my code
Server Code
import socket
import thread
import hashlib
serversock = socket.socket()
host = socket.gethostname();
port = 9000;
serversock.bind((host,port));
filename = ""
serversock.listen(10);
print "Waiting for a connection....."
clientsocket,addr = serversock.accept()
print("Got a connection from %s" % str(addr))
while True:
size = clientsocket.recv(1)
filesz = clientsocket.recv(1)
if filesz.isdigit():
size += filesz
filesize = int(size)
else:
filesize = int(size)
print filesize
for i in range(0,filesize):
if filesz.isdigit():
filename += clientsocket.recv(1)
else:
filename += filesz
filesz = "0"
print filename
file_to_write = open(filename, 'wb')
while True:
data = clientsocket.recv(1024)
#print data
if not data:
break
file_to_write.write(data)
file_to_write.close()
print 'File received successfully'
serversock.close()
Client Code
import socket
import os
import thread
s = socket.socket()
host = socket.gethostname()
port = 9000
s.connect((host, port))
path = "C:\Users\Fahad\Desktop"
directory = os.listdir(path)
for files in directory:
print files
filename = files
size = bytes(len(filename))
#print size
s.send(size)
s.send(filename)
file_to_send = open(filename, 'rb')
l = file_to_send.read()
s.sendall(l)
file_to_send.close()
print 'File Sent'
s.close()
Here's the error that I get
Waiting for a connection.....
Got a connection from ('192.168.0.100', 58339)
13
Betternet.lnk
File received successfully
Traceback (most recent call last):
File "server.py", line 22, in <module>
filesize = int(size)
ValueError: invalid literal for int() with base 10: ''

There are several minor issues in your snippet. Maybe you could do something like this?
import socket
import thread
import hashlib
serversock = socket.socket()
host = socket.gethostname();
port = 9000;
serversock.bind((host,port));
filename = ""
serversock.listen(10);
print "Waiting for a connection....."
clientsocket,addr = serversock.accept()
print("Got a connection from %s" % str(addr))
while True:
size = clientsocket.recv(16) # Note that you limit your filename length to 255 bytes.
if not size:
break
size = int(size, 2)
filename = clientsocket.recv(size)
filesize = clientsocket.recv(32)
filesize = int(filesize, 2)
file_to_write = open(filename, 'wb')
chunksize = 4096
while filesize > 0:
if filesize < chunksize:
chunksize = filesize
data = clientsocket.recv(chunksize)
file_to_write.write(data)
filesize -= len(data)
file_to_write.close()
print 'File received successfully'
serversock.close()
Client:
import socket
import os
import thread
s = socket.socket()
host = socket.gethostname()
port = 9000
s.connect((host, port))
path = "blah"
directory = os.listdir(path)
for files in directory:
print files
filename = files
size = len(filename)
size = bin(size)[2:].zfill(16) # encode filename size as 16 bit binary
s.send(size)
s.send(filename)
filename = os.path.join(path,filename)
filesize = os.path.getsize(filename)
filesize = bin(filesize)[2:].zfill(32) # encode filesize as 32 bit binary
s.send(filesize)
file_to_send = open(filename, 'rb')
l = file_to_send.read()
s.sendall(l)
file_to_send.close()
print 'File Sent'
s.close()
Here the client also sends the size of the file. Both size and filesize are encoded as binary strings (you could do another method). The filename length (size) can take values up to 2^16 and the send file can have up to 2^32 byte (that is 2^filesize).

I believe you're actually receiving all the files' content but then writing them all to one file.
Your server is only accepting a single connection and it writes whatever data it receives into the file until it receives no more data. That won't happen until the client closes its socket at the end.
There are a couple of ways to fix this.
Move the accept call into the server loop and the connect call into the client loop. Have your client connect, send the filename, transfer the entire content of a single file, then close the connection. On the next iteration, do it all over again.
Alternatively, at the beginning of each file, have the client send to the server both the filename to be transferred and the file size (so the server knows how to find the end of the file content). Then write exactly that many bytes to the server. (But see also below as to transmitting the file size.)
I would recommend (1) as more robust and easier to implement.
Second subject: your mechanism for sending the filename is flawed. If I follow it correctly, your program would not work properly if the filename being transmitted begins with a digit as the server will not be able to determine how many bytes are being used to send the filename length. There are two usual ways of sending numbers:
Use the struct module to format a binary integer in a well-defined way. You actually send the "packed" format, and the server would unpack it. It would then know exactly how many bytes long to receive for the filename.
Just send a header line containing the filename followed by a null byte or newline (or some other well-defined defined terminator byte). Then the server can read a byte at a time of the filename until it sees the terminator.

You could create multiple sockets and write using makefile:
server:
import socket
import threading
import time
serversock = socket.socket()
host = socket.gethostname()
port = 9000
serversock.bind((host, port))
serversock.listen(10)
print("Waiting for a connection.....")
def reader(client):
fle = client.makefile('r')
filename = fle.readline()
client.send("Got file {}\n".format(filename))
file_to_write = open(filename.rstrip(), 'wb')
client.send("Starting writing {}\n".format(filename))
file_to_write.write(fle.read())
file_to_write.close()
client.send("Finished writing {}\n".format(filename))
while True:
client, addr = serversock.accept()
print("Got a connection from %s" % str(addr))
client_serve_thread = threading.Thread(target=reader, args=tuple((client,)))
client_serve_thread.start()
time.sleep(0.001)
serversock.close()
client:
import socket
import os
import thread
import os
host = socket.gethostname()
port = 9000
path = "/home/padraic/t"
directory = os.listdir(path)
for file in directory:
s = socket.socket()
s.connect((host, port))
filename = os.path.join(path, file)
s.send(file+"\n")
print(s.recv(1024))
file_to_send = open(os.path.join(path, file), 'rb')
s.send(file_to_send.read())
print('File Sent\n')
file_to_send.close()
rec = s.recv(1024)
print(rec)
s.close()

Python : How to handle multiple clients and a server

I am implementing a program with a server and multiple clients. All clients send data to the server and a server check out the step of each client. If all client's steps are the same, a server sends new data to all clients to do next step. And it continues this procedure again and again.
However, when I run my program, it cannot communicate each other. Here are my code. Would you give me some hints?
client & server
#client
from socket import *
from sys import *
import time
import stat, os
import glob
# set the socket parameters
host = "localhost"
port = 21567
buf = 1024
data = ''
addr = (host, port)
UDPSock = socket(AF_INET, SOCK_DGRAM)
UDPSock.settimeout(100)
def_msg = "=== TEST ==="
#FILE = open("test.jpg", "w+")
FILE = open("data.txt","w+")
while (1):
#data, addr = UDPSock.recvfrom(buf)
print "Sending"
UDPSock.sendto(def_msg, addr)
#time.sleep(3)
data, addr = UDPSock.recvfrom(buf)
if data == 'done':
FILE.close()
break
FILE.write(data)
print "Receiving"
#time.sleep(3)
UDPSock.close()
# server program for nvt
from socket import *
import os, sys, time, glob
#import pygame
import stat
host = 'localhost'
port = 21567
buf = 1024
addr = (host, port)
print 'test server'
UDPSock = socket(AF_INET, SOCK_DGRAM)
UDPSock.bind(addr)
msg = "send txt file to all clients"
#FILE = open("cam.jpg", "r+")
FILE = open("dna.dat","r+")
sending_data = FILE.read()
FILE.close()
tmp_data = sending_data
while (1):
#UDPSock.listen(1)
#UDPSock.sendto(msg, addr)
#FILE = open("gen1000.dat","r+")
#sending_data = FILE.read()
#FILE.close()
#print 'client is at', addr
data, addr = UDPSock.recvfrom(buf)
#time.sleep(3)
print data
#msg = 'hello'
#
tmp, sending_data = sending_data[:buf-6], sending_data[buf-6:]
if len(tmp) < 1:
msg = 'done'
UDPSock.sendto(msg, addr)
print "finished"
sending_data = tmp_data
UDPSock.sendto(tmp, addr)
print "sending"
#time.sleep(3)
UDPSock.close()

A server must perform the sequence socket(), bind(), listen(), accept() (possibly repeating the accept() to service more than one client), while a client only needs the sequence socket(), connect().
Your missing listen() i saw first. Listen for connections made to the socket.
More on this here: link text

Look at this: http://heather.cs.ucdavis.edu/~matloff/Python/PyNet.pdf
It's a very good Python networking tutorial including working examples of a client and server. Now, I'm not an expert on this, but it looks to me like your code is overcomplicated. And what's the deal with all the commented-out lines?
Quote from question:
#UDPSock.listen(1)
#UDPSock.sendto(msg, addr)
#FILE = open("gen1000.dat","r+")
#sending_data = FILE.read()
#FILE.close()
End quote
Those look like some pretty important lines to me.
Also, make sure the computers are connected. From a prompt run:
ping [IP]
where [IP] is the IP address of the other machine(Note: if you're not connected to the same LAN, this becomes much harder as you might then need port forwarding and possibly static IPs).