I have just started learning python network programming. I was reading Foundations of Python Network Programming and could not understand the use of s.shutdown(socket.SHUT_WR) where s is a socket object.
Here is the code(where sys.argv[2] is the number of bytes user wants to send, which is rounded off to a multiple of 16) in which it is used:
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while True:
print 'Listening at', s.getsockname()
sc, sockname = s.accept()
print 'Processing up to 1024 bytes at a time from', sockname
n = 0
while True:
message = sc.recv(1024)
if not message:
break
sc.sendall(message.upper()) # send it back uppercase
n += len(message)
print '\r%d bytes processed so far' % (n,),
sys.stdout.flush()
print
sc.close()
print 'Completed processing'
elif len(sys.argv) == 3 and sys.argv[1] == 'client' and sys.argv[2].isdigit():
bytes = (int(sys.argv[2]) + 15) // 16 * 16 # round up to // 16
message = 'capitalize this!' # 16-byte message to repeat over and over
print 'Sending', bytes, 'bytes of data, in chunks of 16 bytes'
s.connect((HOST, PORT))
sent = 0
while sent < bytes:
s.sendall(message)
sent += len(message)
print '\r%d bytes sent' % (sent,),
sys.stdout.flush()
print
s.shutdown(socket.SHUT_WR)
print 'Receiving all the data the server sends back'
received = 0
while True:
data = s.recv(42)
if not received:
print 'The first data received says', repr(data)
received += len(data)
if not data:
break
print '\r%d bytes received' % (received,),
s.close()
else:
print >>sys.stderr, 'usage: tcp_deadlock.py server | client <bytes>'
And this is the explanation that the author provides which I am finding hard to understand:
Second, you will see that the client makes a shutdown() call on the socket after it finishes sending its transmission. This solves an important problem: if the server is going to read forever until it sees end-of-file, then how will the client avoid having to do a full close() on the socket and thus forbid itself from doing the many recv() calls that it still needs to make to receive the server’s response? The solution is to “half-close” the socket—that is, to permanently shut down communication in one direction but without destroying the socket itself—so that the server can no longer read any data, but can still send any remaining reply back in the other direction, which will still be open.
My understanding of what it will do is that it will prevent the client application from further sending the data and thus will also prevent the server side from further attempting to read any data.
What I cant understand is that why is it used in this program and in what situations should I consider using it in my programs?
My understanding of what it will do is that it will prevent the client
application from further sending the data and thus will also prevent
the server side from further attempting to read any data.
Your understanding is correct.
What I cant understand is that why is it used in this program …
As your own statement suggests, without the client's s.shutdown(socket.SHUT_WR) the server would not quit waiting for data, but instead stick in its sc.recv(1024) forever, because there would be no connection termination request sent to the server.
Since the server then would never get to its sc.close(), the client on his part also would not quit waiting for data, but instead stick in its s.recv(42) forever, because there would be no connection termination request sent from the server.
Reading this answer to "close vs shutdown socket?" might also be enlightening.
The explanation is half-baked, it applies only to this specific code and overall I would vote with all-fours that this is bad practice.
Now to understand why is it so, you need to look at a server code. This server works by blocking execution until it receives 1024 bytes. Upon reception it processes the data (makes it upper-case) and sends it back. Now the problem is with hardcoded value of 1024. What if your string is shorter than 1024 bytes?
To resolve this you need to tell the server that - hey there is no more data coming your way, so return from message = sc.recv(1024) and you do this by shutting down the socket in one direction.
You do not want to fully close the socket, because then the server would not be able to send you the reply.
Related
import socket
ip = '127.1.0.0'
port = 2400
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((ip, port))
clientsocket, address = s.accept()
while True:
s.listen(10)
print(f'{address[0]}:{address[1]}')
msg = clientsocket.recv(8196)
print(msg.decode(), "\n")
time.sleep(.05)
This is my server. Whenever I shut down the client server, the while loop continues to print the address and port continuously until I terminate the program. How do I check if the client is still connected, and if not, then close it. I have tried the try and except, and put s.send(bytes()) in the try, and break in the except, but every time I sent a message to the server from the client, it would break the while loop. I am also new to socket programming, so it probably is a little messy.
When the other side disconnected the socket, recv returns 0 bytes (empty string).
Note that this is the only time when recv returns 0 bytes. Otherwise, it keeps waiting until there are some bytes. So you know that if it receives 0 bytes then the other computer disconnected.
Note: in old versions of Python 2, that means recv returns "". In Python 3 it returns b"". I like to write if len(msg) == 0: to cover both cases, but assuming you're using Python 3, if msg == b"": is also fine
I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)
recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...
I wrote a python client to communicate with server side. Each time when I finished sanding out data, I have to call sock.shutdown(socket.SHUT_WR), otherwise the server would not do any response. But after calling sock.shutdown(socket.SHUT_WR), I have to reconnect the connection as sock.connect((HOST, PORT)), other wise I can not send data to server. So how can I keep the connection alive without close it.
My sample code as following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
sock.sendall(data)
sock.shutdown(socket.SHUT_WR)
received = sock.recv(1024)
while len(received)>0:
received = sock.recv(1024)
sock.sendall(newdata) # this would throw exception
The Server Side code as following:
def handle(self):
cur_thread = threading.current_thread()
while True:
self.data = self.rfile.read(bufsiz=100)
if not self.data:
print 'receive none!'
break
try:
data = self.data
print 'Received data, length: %d' % len(data)
self.wfile.write('get received data\n')
except Exception:
print 'exception!!'
You didn't show any server side code but I suspect it simply reads bytes until it gets none anymore.
You can't do this as you found out, because then the only way to tell the server the message is complete is by killing the connection.
Instead you'll have to add some form of framing in your protocol. Possible approaches include a designated stop character that the server recognises (such as a single newline character, or perhaps a 0-byte), sending frames in fixed sizes that your client and server agree upon, or send the frame size first as a network encoded integer followed by exactly the specified number of bytes. The server then first reads the integer and then exactly that same number of bytes from the socket.
That way you can leave the connection open and send multiple messages.
I am working on a proof of concept using python that emulates a server/client communication using sockets to send UDP packets. I can easily do a simple client to server and back to client comms, but I am trying to introduce a "middle-man" into that communication. Conceptually the problem can be descirbed as, if "Joe" is the main client, he will send a message to "Steve" who is the middle man who will do something with that message before sending it to "Carol" who acts as the server that will process the new message and send a response back to the middle-man, "Steve". Eventually the middle-man will then send that message on elsewhere, but at the moment I am not worrying about that.
My current code looks like:
"Joe" (original client) looks like
# create dgram udp socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
except socket.error:
print ('Failed to create socket')
sys.exit()
host = 'localhost'
port = 8888
print("start comms")
while 1:
arr = ['Dog', 'cat', 'treE', 'Paul']
num = random.randrange(0,4)
#Send the string
s.sendto(arr[num].encode(), (host, port))
"Steve" (middle man) looks like
host = ''
hostRT = 'localhost'
portVM = 8888
portRT = 8752
# socket to receive from "Joe"
s1 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s1.bind((host, portVM))
# socket to send to "Carol"
s2 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
print("start comms")
while 1:
# receive from "Joe"
data = s1.recvfrom(1024)
num = data[0].decode()
addrVM = data[1]
# print data from Joe
print(num)
# add some excitement to Joe's message
num += '!!!'
# show received message address + port number
print ("message[" + addrVM[0] + ":" + str(addrVM[1]) + ']')
# Send to "Carol"
s2.sendto(num.encode(), (hostRT, portRT))
# receive from "Carol"
d = s2.recvfrom(1024)
reply = d[0].decode()
addrRT = d[1]
# show received message address + port number
print ("message[" + addrRT[0] + ":" + str(addrRT[1]) + ']')
# show Carol's response
print ('Server reply : ' + reply)
s1.close()
s2.close()
"Carol" (server) looks like
host = ''
port = 8752
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
print ("socket created")
s.bind((host, port))
print ("Socket bind complete")
while 1:
d = s.recvfrom(1024)
data = d[0].decode()
addr = d[1]
print(data)
reply = "Upper case client data = " + data.upper()
print(reply)
s.sendto(reply.encode(), addr)
print ("message[" + addr[0] + ":" + str(addr[1]) + '] - ' + data.strip())
s.close()
Currently I can receive a message from Joe but then it hangs on the sending to the server Carol. I'm completely new to socket programming so any help would be greatly appreciated.
Edit for clarification
Using Python 3.4
Joe is sending packets non stop as to emulate the real life application that this proof of concept is for. Joe will be sending packets at a rate of roughly 1 packet / 4ms, but I am only concerned with the most recent packet. However, since the average turn around time for the round trip from Steve to Carol is around 10ms, I had originally thought to cache Joe's most recent packet in a local memory location and overwrite that location until Steve is ready to send a packet to Carol once she has responded with the last packet. However, for this simple proof of concept I haven't tried to implement that. Any suggestions on that would also be helpful.
There are multiple faults that contribute to the overall failure, some of which are not apparent (i.e. it sort of works until it crashes down somewhere else).
First of all, at the moment sends packets as fast as he cans. That alone can lead to significant packet loss everywhere else (that might be a good thing, since you now have to make sure your code survives packet loss). Unless you truly want to stress the network, something like time.sleep(0.1) would be appropriate in the send loop.
More importantly, steve's socket setup is all messed up. He needs two sockets at the most, not three. The way it is currently set up, carol answers steve to the IP address and port she got the packet from (which is quite sensible), but steve reads on a distinct socket that never gets data sent to.
To make matters worse, the port steve's s3 listens on is actually the same one that carol uses! Since you are not using multicast. You can simply remove every reference to s3 from the code and use s2 exclusively.
Another problem is that you don't deal with packet loss. For example, if a packet gets lost between steve and carol, the code
# Send to "Carol"
s2.sendto(num.encode(), (hostRT, portRT))
# receive from "Carol"
d = s2.recvfrom(1024) # s3 in the original
will hang forever, since Carol does not send any new packets after the one that got lost. As mentioned before, packet loss is way more likely since joe is blasting out packets as fast as he can.
To detect packet loss, there are a few options:
Use multiple threads or processes for sending and receinv. This is going to make your program way more complex.
Switch to asynchronous / non-blocking IO, in Python with the high-level asyncore or the more low-level select.
Set and correctly handle socket timeouts. This is probably the easiest option for now, but is quite limited.
Switch to TCP if you actually need reliable communication.
Apart from the aforementioned network problems, there are also some potential problems or inaccuracies:
If you are using Python 2.x, the behavior of decode and encode on strings depends on the system configuration. Like many other potential problems, this has been fixed in Python 3.x by mandating UTF-8 (in 2.x you have to explicitly request that). In your case, that's fine as long as you only send ASCII characters.
while(1) : looks really strange in Python - why the whitespace after the argument, and why parentheses . Why not while 1: or while True:?
You can use tuple unpacking to great effect. Instead of
data = s1.recvfrom(1024)
num = data[0].decode()
addrVM = data[1]
how about:
data, addrVM = s1.recvfrom(1024)
num = data.decode('utf-8')
I am trying to implement a simple UDP client and server. Server should receive a message and return a transformed one.
My main technique for server is to listen UDP messages in a loop, then spawn multiprocessing.Process for each incoming message and send the reply within each Process instance:
class InputProcessor(Process):
...
def run(self):
output = self.process_input()
self.sock.sendto(output, self.addr) # send a reply
if __name__ == "__main__":
print "serving at %s:%s" % (UDP_IP, UDP_PORT)
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP,UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message: %s from %s:%s" % (data, addr[0], addr[1])
p = InputProcessor(sock, data, addr)
p.start()
In test client, I do something like this:
def send_message(ip, port, data):
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
print "sending: %s" % data
sock.sendto(data, (ip, port))
sock.close()
for i in xrange(SECONDS*REQUESTS_PER_SECOND):
data = generate_data()
p = multiprocessing.Process(target=send_message, args=(UDP_IP,
UDP_PORT,
data))
p.start()
time.sleep(1/REQUESTS_PER_SECOND)
The problem I am having with the code above is that when REQUESTS_PER_SECOND becomes higher than certain value (~50), it seems some client processes receive responses destinated to different processes, i.e. process #1 receives response for process #2, and vice versa.
Please criticize my code as much as possible, due to I am new to network programming and may miss something obvious. Maybe it's even worth and better for some reason to use Twisted, hovewer, I am highly interested in understanding the internals. Thanks.
As per previous answer, I think that the main reason is that there is a race condition at the UDP port for the clients. I do not see receiving at the client code, but presumably it is similar to the one in server part. What I think happens in concrete terms is that for values under 50 requests / second, the request - response roundtrip gets completed and the client exits. When more requests arrive, there may be multiple processes blocking to read the UDP socket, and then it is probably nondeterministic which client process receives the incoming message. If the network latency is going to be larger in the real setting, this limit will be hit sooner.
Thanks guys a lot! It seems I've found why my code failed before. I was using multiprocessing.Manager().dict() within client to check if the results from server are correct. However, I didn't use any locks to wrap a set of write operations to that dict(), thus got a lot of errors though the output from server was correct.
Shortly, in client, I was doing incorrect checks for correct server responses.