I am trying multiple clients to send files to the server simultaneously on one port(i.e. server is running different ports and multiple clients are connected to each port and sending files). I have looked several answers such as this, but they are using different approaches and I just want somebody to pinpoint what I am doing wrong here, so I can use same code which I understand better. Please help me with:
Why my code is not working with multiple file transfer?
I am also calculating the throughput(i.e. actual file transfer), is it the correct method?
Thanks for help.
----- server.py ---
import socket,time
import sys, optparse,datetime
#def client(net,src,dst):
#def server(net,src):
print("we are in server ...")
parser = optparse.OptionParser()
parser.add_option('-i',dest='ip',default='')
parser.add_option('-p',dest='port',type='int',default=5001)
parser.add_option('-t',dest='ftype',type='str',default='.txt')
(options,args) = parser.parse_args()
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
host = socket.gethostname()
server_socket.bind((options.ip, options.port))
server_socket.listen(100)
s = datetime.datetime.now().strftime("%d%m%y_%H%M%S")
f = open('.recfile_%s_%s'%(s,options.port)+options.ftype,'wb')
count = 0
while 1:
client_socket, addr = server_socket.accept()
start_time = datetime.datetime.now()
cl_addr = addr[0]
print 'Got connection from', addr
print("Receiving ...")
data = client_socket.recv(1024)
while(data):
f.write(data)
data = client_socket.recv(1024)
count+=len(data)
continue
f.close()
client_socket.close()
end_time = datetime.datetime.now()
total_time = end_time - start_time
total_time = total_time.total_seconds()
average_speed = round((1024*count*0.001)/(total_time),3)
fd = open('server_data.csv','a+')
fd.write(str(cl_addr)+','+str(start_time)+','+str(end_time)+','+str(total_time)+','+str(average_speed)+','+str(options.port)+'\n\r')
fd.close()
server_socket.close()
client side
----- client.py -----
import socket
import sys, optparse
#def client(net,src,dst):
print("we are in client ..")
parser = optparse.OptionParser()
parser.add_option('-i',dest='ip',default='')
parser.add_option('-p',dest='port',type='int',default=5001)
parser.add_option('-f',dest='fname',type='str',default='hugefile.txt')
(options,args) = parser.parse_args()
client_socket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client_socket.connect((options.ip,options.port))
img_file = options.fname
f = open(img_file,'rb')
data = f.read(1024)
while(data):
client_socket.send(data)
data = f.read(1024)
f.close()
client_socket.shutdown(socket.SHUT_WR)
client_socket.close()
print "Data Sent successfully!"
There is at least one problem: the recfile file is opened before starting the loop, and closed inside the loop. That means that beginning from the second iteration, you will try to write on a closed file and get an exception.
How to avoid: with open(...) as ...: blocks are great because not only they guarantee proper closure in case of errors, but they also ensure a correct bloc structure in your program.
BTW, count should also be reset to 0 inside the loop, and the closer to the loop the better for future readers and maintainers of the code
I found a solution by improvising this .
Multiple connection to same socket is not possible without either multiprocessing or multithreading. Since I am using Python 2.7 multithreading is not an option for me.
Related
I am having the above issue. The client is suppose to ask for a filename and send the file name to the server after which the server will open the file and display it. Problem is that the server isn't opening the file and displaying it.
Below is the client.
#!/usr/bin/env python3
import socket, os.path, datetime, sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.connect((host, port))
Filename = input("Type in ur file ")
s.send(Filename.encode('utf-8'))
data = s.recv(1024).decode('utf-8')
s.close()
if __name__ == '__main__':
Main()
Below is the server
#!/usr/bin/env python3
import socket
import os
import sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.bind((host,port))
print("server Started")
s.listen(1)
c, addr = s.accept()
print("Connection from: " + str(addr))
while True:
data = c.recv(1024).decode('utf-8')
myfile = open(data, "r")
if not data:
break
print("from connected user: " + myfile)
c.close()
if __name__ == '__main__':
Main()
I've made few minimal adjustments to your code with which it runs as so that server.py continuously listens on a given port and sends back data which each invocation of client.py asks for.
server.py
#!/usr/bin/env python3
import socket
import os
import sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.bind((host,port))
print("server Started")
s.listen(1)
while True:
c, addr = s.accept()
print("Connection from: " + str(addr))
filename = ''
while True:
data = c.recv(1024).decode('utf-8')
if not data:
break
filename += data
print("from connected user: " + filename)
myfile = open(filename, "rb")
c.send(myfile.read())
c.close()
if __name__ == '__main__':
Main()
client.py
#!/usr/bin/env python3
import socket, os.path, datetime, sys
def Main():
host = '127.0.0.1'
port = 50001
s = socket.socket()
s.connect((host, port))
Filename = input("Type in ur file ")
s.send(Filename.encode('utf-8'))
s.shutdown(socket.SHUT_WR)
data = s.recv(1024).decode('utf-8')
print(data)
s.close()
if __name__ == '__main__':
Main()
And now a bit of explanation.
On the server side. The outer loop accepts a connection, then reads from the connection until done (more on this later). Prints your debugging info, but note you were trying to print the file object and not the filename (which would fail trying to concatenate). I also open the file in binary mode (that way I can skip the str -> bytes translation.
On the client side. I've added closing the writing end of the socket when the file has been sent. Note you might want to use sendall instead of send for this use case: check those docs links for details. And I've added a print for the incoming data.
Now that bit with shutting down the writing end in the client and the inner loop reading (and also related to the sendall hint. Which BTW also holds true for the server side, otherwise you should loop, as you might see your content truncated; other option is to also have a sending loop.). Stream sockets will guarantee you get your bytes in in order you've send them. On itself it has no idea whether your message is complete and it also does not guarantee in how many and how large chunks will the data be sent and received (resp.).
The inner loop of server keep reading until we see an EOF (we've receive zero length string in python socket). This would happen (be returned by recv when the remote socket (or at least its writing end) has been shut down. Since we still want to reuse the connection, we only do that on the sending end in the client. Hope this helps you to move ahead.
I have a long running python script which updated a list with a few "items"
I need to be able to query this running python script while its running to produce a print out of the content of this list.
Im not sure if this is possible on python2.6?
Added code below which basically generates a list of servers from log file. This is a basic script which ill make a lot more intrusive if i was able to dump the list it generates on a need to basis.
UPDATE
so ive got a script which does what i want in theory. The only thing ive noticed is that during the print from the "buf" received from client comes out on the main tool.
I cant rack my head around how to get it from the main tool to the client tool.
#!/usr/bin/python
import sys
import time
import re
import threading
import socket
import errno
def handle():
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
PORT = 19999
HOST = '127.0.0.1'
MAX_LENGTH = 4096
try:
serversocket.bind((HOST, PORT))
except socket.error, v:
errorcode=v[0]
print "Error found during startup [ %s ] " % (errorcode)
sys.exit(0)
serversocket.listen(10)
(clientsocket, address) = serversocket.accept()
while True:
buf = clientsocket.recv(MAX_LENGTH)
if buf == '': return #client terminated connection
elif buf == "dump":
for item in serverList:
print item
else:
print "Unrecognised command entered"
print buf
def follow(thefile):
thefile.seek(0,2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
handlethread = threading.Thread(target=handle)
handlethread.daemon = True
handlethread.start()
serverList = []
print "hello there"
logfile = open("/var/log/messages","r")
loglines = follow(logfile)
for line in loglines:
servername=line.split(' ')[3]
if servername not in serverList:
serverList.append(servername)
print "added %s to list of servers" % (servername)
CLIENT SCRIPT
#!/usr/bin/python
import socket
import sys
HOST = '127.0.0.1'
PORT = 19999
s = socket.socket()
s.connect((HOST, PORT))
while 1:
msg = raw_input("Command To Send: ")
if msg == "close":
s.close()
sys.exit(0)
s.send(msg)
You could periodically save the contents of the list to your hard drive while the script is running. So in the main loop of the long-running script, simply Pickle the contents of the list. Whenever you need to check the values during execution just un-pickle the saved file. For more information on object serialization in Python see Pickle.
If you're just trying to periodically print out the current list, you can just use the print function. Like so:
vals = []
for elem in range(10):
vals.append(elem)
print vals
this is my UDP-Server very according to the UDP-Server-Example from the python-wiki:
# ----- receiver.py -----
#!/usr/bin/env python
from socket import *
import sys
import select
host="192.168.88.51"
port = 1337
s = socket(AF_INET,SOCK_DGRAM)
s.bind((host,port))
addr = (host,port)
buf=128
data,addr = s.recvfrom(buf)
print "Received File:"
f = open("out.jpg",'wb')
data, addr = s.recvfrom(buf)
try:
while(data):
f.write(data)
s.settimeout(1)
data,addr = s.recvfrom(buf)
except timeout:
f.close()
s.close()
print "File Downloaded"
This code works fine and I'm able to receive ONE file at a time. But I have multiple clients and I'd like to receive every file coming in, so every time a new connection is established (from one certain IP).
Any suggestions?
First of all, if you want an asynchronous server, it's better to not write things from scratch with sockets. Instead, use packages like asyncio or Twisted.
Coming to your problem, it's more convenient to go with a TCP-focused socket, therefore you should use SOCK_STREAM instead of the UDP type SOCK_DGRAM.
First, define a function for downloading:
def get_file(s):
s.settimeout(1)
with open("out.jpg",'wb') as f:
data, addr = s.recv(buf)
try:
while(data):
f.write(data)
data, addr = s.recv(buf)
except timeout:
print "File Downloaded"
After setting up the constants (hostname, port number and so on), do something like the following (and do from threading import Thread first!):
s = socket(AF_INET,SOCK_DGRAM)
s.bind((host,port))
while True:
print "Waiting for connection..."
data, addr = s.recvfrom(buf)
print "... connection from:", addr
Thread(target=get_file, args=(s,)).start() #starts a new thread for file download, addr acts like a filename
I use python program to do traffic generator in Linux Ubuntu, and the code like below:
import socket, sys
host = sys.argv[1] #Server IP Address
textport = sys.argv[2] #Server Binding Port
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) #socket
try:
port = int(textport)
except ValueError:
port = socket.getservbyname(textport, 'udp')
while 1:
try:
data = open('auth3.log')#read file
for each_line in data: #each rows
try:
(role,line_spoken) = each_line.split(': ',1)#split two parts
role = role.strip()
s.sendto(role, (host, port))
print('Send: ' + str(role) + "\n" )
except:
pass
except IOError as err:
print('file isn\'t exist!!~'+str(err))
finally:
if 'data' in locals(): #If data have no object, Don't use data to close!!~
data.close()
print "\n"
The size of auth3.log is about 1.8M.
When I send data to the destination server, I use snmp which OID is ''ifInOctets'' to get the traffic information.
But I the traffic recalculate to unit of ''Kbits'' is about 128.
How can I use this program to fill the bandwidth up to 1Gbits?(In other words, I want to fill out the bandwidth)
Thanks for your helping.
This version of your code implements the first two optimizations suggested by Chris Merck.
import socket, sys, itertools
host = sys.argv[1] #Server IP Address
textport = sys.argv[2] #Server Binding Port
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
port = int(textport)
except ValueError:
port = socket.getservbyname(textport, 'udp')
# preprocess file data
with open('auth3.log') as data:
roles = [role for (role, line_spoken) in line.split(': ', 1) for line in data]
# preprocess everything we can
hp = (host, port)
send = s.sendto
for role in itertools.cycle(roles):
try:
send(role, hp)
except:
pass
For further optimizations, you might want to process it using Cython, which might further speed up the loop. If the code still doesn't generate enough traffic, you'll probably need to launch several processes in parallel.
Your program is not running fast enough to generate 1Gbps on the wire.
To make it run faster, you can:
Remove the call to print after sendto. (Print is slow by nature.)
Preprocess your auth3.log file so that you do not need to process it within your inner loop. (Right now you are looping on .split and .strip, both of which are wasting CPU time.
Rewrite your program to send larger chunks.
But, I fear the result will still not reach 1Gbps. To really max out your line, try using a traffic generation program such as Colasoft Packet Builder (although I'm not sure even that program will do it. 1Gbps is a lot of traffic.)
I am implementing a program with a server and multiple clients. All clients send data to the server and a server check out the step of each client. If all client's steps are the same, a server sends new data to all clients to do next step. And it continues this procedure again and again.
However, when I run my program, it cannot communicate each other. Here are my code. Would you give me some hints?
client & server
#client
from socket import *
from sys import *
import time
import stat, os
import glob
# set the socket parameters
host = "localhost"
port = 21567
buf = 1024
data = ''
addr = (host, port)
UDPSock = socket(AF_INET, SOCK_DGRAM)
UDPSock.settimeout(100)
def_msg = "=== TEST ==="
#FILE = open("test.jpg", "w+")
FILE = open("data.txt","w+")
while (1):
#data, addr = UDPSock.recvfrom(buf)
print "Sending"
UDPSock.sendto(def_msg, addr)
#time.sleep(3)
data, addr = UDPSock.recvfrom(buf)
if data == 'done':
FILE.close()
break
FILE.write(data)
print "Receiving"
#time.sleep(3)
UDPSock.close()
# server program for nvt
from socket import *
import os, sys, time, glob
#import pygame
import stat
host = 'localhost'
port = 21567
buf = 1024
addr = (host, port)
print 'test server'
UDPSock = socket(AF_INET, SOCK_DGRAM)
UDPSock.bind(addr)
msg = "send txt file to all clients"
#FILE = open("cam.jpg", "r+")
FILE = open("dna.dat","r+")
sending_data = FILE.read()
FILE.close()
tmp_data = sending_data
while (1):
#UDPSock.listen(1)
#UDPSock.sendto(msg, addr)
#FILE = open("gen1000.dat","r+")
#sending_data = FILE.read()
#FILE.close()
#print 'client is at', addr
data, addr = UDPSock.recvfrom(buf)
#time.sleep(3)
print data
#msg = 'hello'
#
tmp, sending_data = sending_data[:buf-6], sending_data[buf-6:]
if len(tmp) < 1:
msg = 'done'
UDPSock.sendto(msg, addr)
print "finished"
sending_data = tmp_data
UDPSock.sendto(tmp, addr)
print "sending"
#time.sleep(3)
UDPSock.close()
A server must perform the sequence socket(), bind(), listen(), accept() (possibly repeating the accept() to service more than one client), while a client only needs the sequence socket(), connect().
Your missing listen() i saw first. Listen for connections made to the socket.
More on this here: link text
Look at this: http://heather.cs.ucdavis.edu/~matloff/Python/PyNet.pdf
It's a very good Python networking tutorial including working examples of a client and server. Now, I'm not an expert on this, but it looks to me like your code is overcomplicated. And what's the deal with all the commented-out lines?
Quote from question:
#UDPSock.listen(1)
#UDPSock.sendto(msg, addr)
#FILE = open("gen1000.dat","r+")
#sending_data = FILE.read()
#FILE.close()
End quote
Those look like some pretty important lines to me.
Also, make sure the computers are connected. From a prompt run:
ping [IP]
where [IP] is the IP address of the other machine(Note: if you're not connected to the same LAN, this becomes much harder as you might then need port forwarding and possibly static IPs).