Develop Reference

When/why to use s.shutdown(socket.SHUT_WR)?

I have just started learning python network programming. I was reading Foundations of Python Network Programming and could not understand the use of s.shutdown(socket.SHUT_WR) where s is a socket object.
Here is the code(where sys.argv[2] is the number of bytes user wants to send, which is rounded off to a multiple of 16) in which it is used:
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while True:
print 'Listening at', s.getsockname()
sc, sockname = s.accept()
print 'Processing up to 1024 bytes at a time from', sockname
n = 0
while True:
message = sc.recv(1024)
if not message:
break
sc.sendall(message.upper()) # send it back uppercase
n += len(message)
print '\r%d bytes processed so far' % (n,),
sys.stdout.flush()
print
sc.close()
print 'Completed processing'
elif len(sys.argv) == 3 and sys.argv[1] == 'client' and sys.argv[2].isdigit():
bytes = (int(sys.argv[2]) + 15) // 16 * 16 # round up to // 16
message = 'capitalize this!' # 16-byte message to repeat over and over
print 'Sending', bytes, 'bytes of data, in chunks of 16 bytes'
s.connect((HOST, PORT))
sent = 0
while sent < bytes:
s.sendall(message)
sent += len(message)
print '\r%d bytes sent' % (sent,),
sys.stdout.flush()
print
s.shutdown(socket.SHUT_WR)
print 'Receiving all the data the server sends back'
received = 0
while True:
data = s.recv(42)
if not received:
print 'The first data received says', repr(data)
received += len(data)
if not data:
break
print '\r%d bytes received' % (received,),
s.close()
else:
print >>sys.stderr, 'usage: tcp_deadlock.py server | client <bytes>'
And this is the explanation that the author provides which I am finding hard to understand:
Second, you will see that the client makes a shutdown() call on the socket after it finishes sending its transmission. This solves an important problem: if the server is going to read forever until it sees end-of-file, then how will the client avoid having to do a full close() on the socket and thus forbid itself from doing the many recv() calls that it still needs to make to receive the server’s response? The solution is to “half-close” the socket—that is, to permanently shut down communication in one direction but without destroying the socket itself—so that the server can no longer read any data, but can still send any remaining reply back in the other direction, which will still be open.
My understanding of what it will do is that it will prevent the client application from further sending the data and thus will also prevent the server side from further attempting to read any data.
What I cant understand is that why is it used in this program and in what situations should I consider using it in my programs?

My understanding of what it will do is that it will prevent the client
application from further sending the data and thus will also prevent
the server side from further attempting to read any data.
Your understanding is correct.
What I cant understand is that why is it used in this program …
As your own statement suggests, without the client's s.shutdown(socket.SHUT_WR) the server would not quit waiting for data, but instead stick in its sc.recv(1024) forever, because there would be no connection termination request sent to the server.
Since the server then would never get to its sc.close(), the client on his part also would not quit waiting for data, but instead stick in its s.recv(42) forever, because there would be no connection termination request sent from the server.
Reading this answer to "close vs shutdown socket?" might also be enlightening.

The explanation is half-baked, it applies only to this specific code and overall I would vote with all-fours that this is bad practice.
Now to understand why is it so, you need to look at a server code. This server works by blocking execution until it receives 1024 bytes. Upon reception it processes the data (makes it upper-case) and sends it back. Now the problem is with hardcoded value of 1024. What if your string is shorter than 1024 bytes?
To resolve this you need to tell the server that - hey there is no more data coming your way, so return from message = sc.recv(1024) and you do this by shutting down the socket in one direction.
You do not want to fully close the socket, because then the server would not be able to send you the reply.

UDP broadcast and automatic server discovery in python, TCP socket unavailable

I'm developing a reverse shell application in python, and right now I'm trying to implement an autodiscovery feature. It should work as follows:
The server broadcasts the IP/port it listens for connections on, and waits for a client. If no client tries to connect in a few seconds, it broadcasts again (and repeat till a connection).
The client tries to receive the broadcast of the server, and connects to the advertised IP/port.
The broadcast works fine, the client receives the IP/port and successfully connects, however after using the connected pair of ports I get (server side):
socket.error: [Errno 35] Resource temporarily unavailable
Server side test code:
sckt = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sckt.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sckt.settimeout(2)
sckt.bind(('', 9999))
sckt.listen(5)
broadcastSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
broadcastSocket.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
broadcastSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
while True:
broadcastSocket.sendto(socket.gethostbyname(socket.getfqdn()) + ' ' + str(9999), ('<broadcast>', 8888))
try:
sock, address = sckt.accept()
break
except socket.timeout:
pass
broadcastSocket.close()
sckt.settimeout(None)
sock.send('test')
# if I add time.sleep(1) here, it works, but I don't get why
# would sock be unavailable at first, but available a second later
print sock.recv(1) # this is where it fails
# note that it also fails with any recv buffer size, for instance 1024
Why on earth would I want to receive 1 byte of data, you might ask. I have an algorithm which prefixes messages with their lengths, and the receiver reads this prefix byte-by-byte till a delimiter, thats why.
Client side test code:
broadcastSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
broadcastSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
broadcastSocket.settimeout(3)
broadcastSocket.bind(('', 8888))
while True:
try:
data = broadcastSocket.recv(1024)
break
except socket.timeout:
pass
broadcastSocket.close()
sckt = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sckt.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sckt.connect((str(data.split()[0]), int(data.split()[1])))
print sckt.recv(1024)
sckt.send('lel')
If I omit the whole broadcast and autodiscovery part of the code and simply manually enter the IP/port of the server print sock.recv(1) doesn't fail.
Any clues on what the issue might be?

Change sckt.settimeout(None) to sock.settimout(None) in the server code.
You want to have the accepted socket in blocking mode and not the accepting one.
This ensures that the sckt.recv waits for an incoming message from the client.
P.S.
sock.setblocking(1) is exactly the same

Basic python socket server application doesnt result in expected output

Im trying to write a basic server / client application in python, where the clients sends the numbers 1-15 to the server, and the server prints it on the server side console.
Code for client:
import socket
clientsocket.connect(('localhost', 8303))
def updateX():
x = 0
while (x < 15):
x
clientsocket.send(format(x))
print x
x = x+1
updateX()
server:
import socket
HOST = 'localhost'
PORT = 8303
s= socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(5) # become a server socket, maximum 5 connections
connection, address = s.accept()
while True:
connection, address = s.accept()
buf = connection.recv(64)
print buf
The output of running the client while the server is live results in either no output, prints only 1, or prints only 12. Ideas?

Before entering the main loop on the server side, you accept a connection:
connection, address = s.accept()
But then in the loop itself you begin by accepting a connection:
while True:
connection, address = s.accept()
buf = connection.recv(64)
print buf
As a result, you never read from the first connection. That's why you don't see any output.
Note also that it's wrong (for what you're trying to do) to accept a new connection on every iteration. Even if you keep making new client connections, the server will accept a connection on each iteration and read from the socket once, but then continue the next iteration and wait for a new connection, never reading more data sent by a client. You should be making multiple recv calls on the same connection object instead.
You might find this tutorial helpful.

There are multiple errors:
socket.send() might send only partial content, use socket.sendall() instead
format(12) returns '12' therefore even if your code sends all numbers and the server correctly receives them then it sees '01234567891011121314' i.e., individual numbers are not separated
double socket.accept() mentioned by #Alp leads to ignoring the very first connection
socket.recv(64) may return less than 64 bytes, you need a loop until it returns an empty bytestring (meaning EOF) or use socket.makefile()
Client:
#!/usr/bin/env python
"""Send numbers in the range 0..14 inclusive as bytes e.g., 10 -> b'\n'
Usage: python client.py [port]
"""
import sys
import socket
from contextlib import closing
port = 8686 if len(sys.argv) < 2 else int(sys.argv[1])
with closing(socket.create_connection(('localhost', port))) as sock:
sock.sendall(bytearray(range(15))) # send each number as a byte
sock.shutdown(socket.SHUT_RDWR) # no more sends/receives
You need to know how numbers are separated in the data. In this case, a fixed format is used: each number is a separate byte. It is limited to numbers that are less than 256.
And the corresponding server:
#!/usr/bin/env python
"""
Usage: python server.py [port]
"""
from __future__ import print_function
import sys
import socket
from contextlib import closing
host = 'localhost'
port = 8686 if len(sys.argv) < 2 else int(sys.argv[1])
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # ipv4 version
try:
s.bind((host, port))
s.listen(5)
print("listening TCP on {host} port {port}".format(**vars()))
while True:
conn, addr = s.accept()
with closing(conn), closing(conn.makefile('rb')) as file:
for byte in iter(lambda: file.read(1), b''):
# print numerical value of the byte as a decimal number
print(ord(byte), end=' ')
print("") # received all input from the client, print a newline
except KeyboardInterrupt:
print('Keyboard interrupt received, exiting.')
finally:
s.close()

Python TCP socket.recv() returns with nothing as soon as connection is made

I'm trying to implement the most basic python TCP server. Windows 8, Python 2.7, firewall is turned off. Code is from here: https://wiki.python.org/moin/TcpCommunication
If I do the client stuff (socket(...), connect(...), send(...)) via python repl, things work fine, ie the server correctly blocks when calling recv.
However if I run the exact same code via python script (both with and without explicitly calling python.exe at windows command line), the recv returns immediately with no data. I read elsewhere on SO this means it's an invalid socket, but I'm not sure what that means or how to check for it. I'm using the socket returned by accept() not the one used to initiate the connection.
I'm trying to block on recv so I can take advantage of the timeout (I don't want to use select module, which BTW also returns immediately) and process some keyboard stuff between attempts to recv, ie user presses 'q' to quit.
In various experiments I've shown that once this occurs, recv will always return immediately (as will select.select(...)) if I put it in a loop, so it's not like the client is sending a single "bad" packet initially. If the client happens to have sent something, then the recv returns with that data, but it certainly doesn't block waiting for data when put in a tight loop.
Is this behavior expected?
Server code:
import sys
import socket
TCP_IP = '192.168.1.10'
TCP_PORT = 5005
BUFFER_SIZE = 20 # Normally 1024, but we want fast response
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connection address:', addr
while 1:
data = conn.recv(BUFFER_SIZE) # This returns immediately with no data, when client connection is run from script and doesn't send() anything, just connects.
if not data:
print "broken"
break
print "received data:", data
conn.send(data) # echo
conn.close()
sys.exit()
Client code:
import sys
import socket
TCP_IP = '192.168.1.10'
TCP_PORT = 5005
BUFFER_SIZE = 1024
MESSAGE = "Hello, World!"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
# Commenting out the following to prove the recv() call on the other
#end returns with nothing instead of blocking indefinitely. If I
#type the rest of this at the REPL the server behaves correctly,
#ie, the recv call blocks forever until socket.send("bla") from client.
#s.send(MESSAGE) data = s.recv(BUFFER_SIZE)
#s.close()
#print "received data:", data
sys.exit()

Yes, this is expected behavior.
The client does not send anything. And it exit as soon as it connect to the server; cause disconnection.
socket.recv returns an empty string if the peer performed shutdown (disconnect).
While, in the REPL, the socket is not closed until you issue sys.exit() or you quit the interactive shell.

Passing _socketobjects as parameters

I'm writing a multithreaded distributed networking algorithm.
I've one thread that listens to new connections. Every time a new connection is established a separate thread is started for listening to messages from that connection.
My problem is that the socket I open works perfectly in both directions inside the connection listener. After I pass the socket object for that connection to the message listener I can read data from the socket, but sending data through it doesn't reach the remote host.
Here's the essential snip from my code:
def connection_listener(port, start_e, terminate_e):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.settimeout(1)
s.bind(('', port))
s.listen(1)
while (not start_e.isSet()):
try:
conn, addr = s.accept()
msg_in = conn.recv(1024).split(":")
if (msg_in[1]=="hello"):
# If addr sends us a 'id:hello', we reply with a 'my_id:welcome'
conn.send(str(my_id)+":welcome")
t = Thread(target=message_listener, args=(conn, addr[0], terminate_e, ))
t.start()
except:
pass # timeout
def message_listener(conn, address, terminate_e):
while (not terminate_e.isSet()):
try:
msg_in = conn.recv(1024)
# Here I can receive everything that I send from the other end of conn,
# but conn.send("any data") doesn't reach the remote host
What I'd like to do is send acknowledgement-like messages from the message listener thread using the conn. Is this possible somehow or am I thinking and doing it wrong?

I sorted this out myself, so I'll share my answer.
I made the protocol exchange fixed size messages by padding with zeroes up to the desired length. I used a length of 32 bytes, which might be quite tiny from the hardware's point of view. Nevertheless it seems to work as supposed.
Pragmatically my solution looks like:
def send_everyone(message):
for i in range(len(peers)):
chunk = (str(my_id)+":"+message).rjust(32, '0')
peers[i].send(chunk)
And on the receiving side we want only 32 bytes at a time:
def message_listener(conn, address, terminate_e):
while (not terminate_e.isSet()):
try:
msg_in = conn.recv(32)
...