Develop Reference

The server is behaving as if it is blocked but i have set it to non blocking

The server only listens for a message from the first socket to connect, even though it is set to nonblocking, it doesn't skip over it when it doesn't receive data. I'm new to networking and this is my first project, if anyone know of any others good for beginners please let me know. Thanks! Here is the code.
import socket
CONNECTED_SENDERS = []
CONNECTED_LISTENERS = []
def Main():
HOST = socket.gethostname()
PORT = 4444
SERVER_SOCKET = socket.socket()
SERVER_SOCKET.bind((HOST, PORT))
SERVER_SOCKET.listen(1)
for i in range(2):
CONNECTION, ADDRESS = SERVER_SOCKET.accept()
CONNECTED_LISTENERS.append(CONNECTION)
for i in range(2):
CONNECTION, ADDRESS = SERVER_SOCKET.accept()
CONNECTED_SENDERS.append(CONNECTION)
for DEVICE in CONNECTED_LISTENERS:
DEVICE.send(b'SERVER: You have succesfully connected.')
DEVICE.send(b'SERVER: Please wait for permission to talk.')
x = 0
for DEVICE in CONNECTED_LISTENERS:
DEVICE.send(b'SERVER: What is your name?')
Name = CONNECTED_SENDERS[x].recv(1024)
CONNECTED_LISTENERS[x] = (CONNECTED_LISTENERS[x], Name)
x += 1
del x, Name
for DEVICE, _ in CONNECTED_LISTENERS:
DEVICE.send(b'SERVER: You may now talk.')
SERVER_SOCKET.setblocking(0)
LEAVE = False
while LEAVE == False:
try:
MESSAGE = CONNECTED_SENDERS[0].recv(1024)
NAME = CONNECTED_LISTENERS[0][1]
for DEVICE, _ in CONNECTED_LISTENERS:
DEVICE.send(NAME + b': ' + MESSAGE)
if MESSAGE == 'QUIT':
LEAVE = True
except:
try:
MESSAGE = CONNECTED_SENDERS[1].recv(1024)
NAME = CONNECTED_LISTENERS[1][1]
for DEVICE, _ in CONNECTED_LISTENERS:
DEVICE.send(NAME + b': ' + MESSAGE)
if MESSAGE == 'QUIT':
LEAVE = True
except:
pass
for CONNECTION in CONNECTED_LISTENERS:
CONNECTION.close()
for CONNECTION in CONNECTED_SENDERS:
CONNECTION.close()
if __name__ == "__main__":
Main()

There are a number of issues with your code, some small and some big. But the main problem is that you're marking the server socket nonblocking, not any of the sockets on which communication takes place.
In standard TCP socket programming, you set up a server which listens for incoming connections. When that server accepts a new client, this returns a new socket, and it's on this new socket that all communication with the remote client happens. In other words, the server socket is just for accepting new connections, and nothing else. You never write data through the server socket.
So it doesn't matter that SERVER_SOCKET is marked nonblocking, you must do something like this:
conn, addr = server.accept()
conn.setblocking(False)
conn is the new socket through which you talk to the client, and can be used in a nonblocking fashion.
Smaller issues:
I should also point out that you call SERVER_SOCKET.listen(1). That argument of 1 means that the server will only have a backlog of waiting connections from one client. So if a second client connects before the first connection is made, the second client will receive an error, ECONNREFUSED. Given what it looks like you're trying to do, I'd guess SERVER_SOCKET.listen(4) is appropriate.
Next, nonblocking communication is much harder than blocking protocols. I'd suggest you improve your networking skills before tackling them, but when you're ready, look at the select or selectors modules for help. They provide tools to wait for communication from any of a number of clients, rather than looping over them all and checking if data is available, as you've done here. This looping is very inefficient.
Finally, in Python, it's good practice to name variables with lower case, underscore-separated names. UPPER_CASE_NAMES are usually reserved for constants. So change SERVER_SOCKET to server_socket, CONNECTED_LISTENERS to connected_listeners, etc.

When/why to use s.shutdown(socket.SHUT_WR)?

I have just started learning python network programming. I was reading Foundations of Python Network Programming and could not understand the use of s.shutdown(socket.SHUT_WR) where s is a socket object.
Here is the code(where sys.argv[2] is the number of bytes user wants to send, which is rounded off to a multiple of 16) in which it is used:
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
while True:
print 'Listening at', s.getsockname()
sc, sockname = s.accept()
print 'Processing up to 1024 bytes at a time from', sockname
n = 0
while True:
message = sc.recv(1024)
if not message:
break
sc.sendall(message.upper()) # send it back uppercase
n += len(message)
print '\r%d bytes processed so far' % (n,),
sys.stdout.flush()
print
sc.close()
print 'Completed processing'
elif len(sys.argv) == 3 and sys.argv[1] == 'client' and sys.argv[2].isdigit():
bytes = (int(sys.argv[2]) + 15) // 16 * 16 # round up to // 16
message = 'capitalize this!' # 16-byte message to repeat over and over
print 'Sending', bytes, 'bytes of data, in chunks of 16 bytes'
s.connect((HOST, PORT))
sent = 0
while sent < bytes:
s.sendall(message)
sent += len(message)
print '\r%d bytes sent' % (sent,),
sys.stdout.flush()
print
s.shutdown(socket.SHUT_WR)
print 'Receiving all the data the server sends back'
received = 0
while True:
data = s.recv(42)
if not received:
print 'The first data received says', repr(data)
received += len(data)
if not data:
break
print '\r%d bytes received' % (received,),
s.close()
else:
print >>sys.stderr, 'usage: tcp_deadlock.py server | client <bytes>'
And this is the explanation that the author provides which I am finding hard to understand:
Second, you will see that the client makes a shutdown() call on the socket after it finishes sending its transmission. This solves an important problem: if the server is going to read forever until it sees end-of-file, then how will the client avoid having to do a full close() on the socket and thus forbid itself from doing the many recv() calls that it still needs to make to receive the server’s response? The solution is to “half-close” the socket—that is, to permanently shut down communication in one direction but without destroying the socket itself—so that the server can no longer read any data, but can still send any remaining reply back in the other direction, which will still be open.
My understanding of what it will do is that it will prevent the client application from further sending the data and thus will also prevent the server side from further attempting to read any data.
What I cant understand is that why is it used in this program and in what situations should I consider using it in my programs?

My understanding of what it will do is that it will prevent the client
application from further sending the data and thus will also prevent
the server side from further attempting to read any data.
Your understanding is correct.
What I cant understand is that why is it used in this program …
As your own statement suggests, without the client's s.shutdown(socket.SHUT_WR) the server would not quit waiting for data, but instead stick in its sc.recv(1024) forever, because there would be no connection termination request sent to the server.
Since the server then would never get to its sc.close(), the client on his part also would not quit waiting for data, but instead stick in its s.recv(42) forever, because there would be no connection termination request sent from the server.
Reading this answer to "close vs shutdown socket?" might also be enlightening.

The explanation is half-baked, it applies only to this specific code and overall I would vote with all-fours that this is bad practice.
Now to understand why is it so, you need to look at a server code. This server works by blocking execution until it receives 1024 bytes. Upon reception it processes the data (makes it upper-case) and sends it back. Now the problem is with hardcoded value of 1024. What if your string is shorter than 1024 bytes?
To resolve this you need to tell the server that - hey there is no more data coming your way, so return from message = sc.recv(1024) and you do this by shutting down the socket in one direction.
You do not want to fully close the socket, because then the server would not be able to send you the reply.

Python Port Scanner

Am newbie to python and stuck at a point. I want to create port scanner with using only python 3 inbuilt libraries (means avoiding scapy etc) I have following code :
import socket
for i in range(1,26):
s = socket.socket()
s.settimeout(0.5)
ip = "74.207.244.221" #scanme.nmap.org
response = s.connect_ex((ip, i))
if response:
print ("%d\tclose" %i)
else:
print ("%d\topen" %i)
s.close()
Now I want to add 2 functionalities to this : that is
Distinguish between close and filtered ports . In both cases am receiving same errno in return so how can I check if I have received back a rst packet or nothing ? As far as I have tried s.recv() isn't working for this.
I want to control the number of tries (attempts), i.e I want to send only one or two syn packets. I don't want this program to send more than 2 syn packets for probes. How can this thing be achieved ?

Distinguish between close and filtered ports . In both cases am
receiving same errno in return so how can I check if I have received
back a rst packet or nothing
You've probably only checked with servers that send back a RST. Here's what I tried:
First case, normal config:
>>> os.strerror(s.connect_ex((ip, 81)))
'Connection refused'
Second, with manual iptables:
iptables -A OUTPUT -p tcp --dport 81 -j DROP
>>> os.strerror(s.connect_ex((ip, 81)))
'Resource temporarily unavailable'
I want to control the number of tries (attempts), i.e I want to send
only one or two syn packets.
I don't think there's a setsockopt TCP option exposed, but on linux there's:
net.ipv4.tcp_syn_retries
However, since you limited the timeout for the socket, all operations that don't finish within 0.5 seconds will time out. So it's likely only 1 or 2 SYNs will leave the station.

#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET, socekt.SOCK_STREAM)
host = 74.207.244.221
def portscan(port):
try:
s.connect((host,port))
return True
else:
return False
for x in range(1,255):
if portscan(x):
print('Port',x,'Is Open')

Python and UDP listening

I have an app, software defined radio, that broadcasts UDP packets on a port that tell listeners what frequency and demodulation mode have been set (among other things.)
I've written a demo python client (code below) that listens to the port, and dumps out the information in the appropriate packets to the console.
These are both running under OSX 10.6, Snow Leopard. They work there.
The question/issue I have is: the Python app has to be started before the radio app or it claims the port is already in use (ERRNO 47) during bind, and I don't understand why. The radio app is broadcasting UDP; certainly I want to accommodate multiple listeners -- that's the idea of broadcasting, or at least, so I thought.
So here's the Python code (the indent is a little messed up due to stack overflow's really dumb "make-it-code" indent, but I assure you it's ok):
#!/usr/bin/python
import select, socket
# AA7AS - for SdrDx UDP broadcast
# this is a sample python script that captures the UDP messages
# that are coming from SdrDx. SdrDx tells you what frequency and
# mode it has been set to. This, in turn, would be used to tell
# another radio to tune to that frequency and mode.
# UDP packet from SdrDx is zero terminated, but receiving the
# packet makes it seem like the string contains zeros all the
# way out to the 1024th character. This function extracts a
# python string up to the point where it hits the first zero,
# then returns that string.
# -----------------------------------------------------------
def zeroterm(msg):
counter = 0;
for c in msg:
if ord(c) != 0:
counter += 1
strn = msg[:counter]
return strn
port = 58083 # port where we expect to get a msg
bufferSize = 1024 # room for message
# Create port to listen upon
# --------------------------
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
s.bind(('', port))
except:
print 'failure to bind'
s.close()
raise
s.setblocking(0)
# Listen for messages
# -------------------
looping = True
while looping:
try:
result = select.select([s],[],[])
except: # you can kill the client here with control-c in its shell
s.close() # must close socket
print 'Closing, exception encountered during select' # warn
raise SystemExit # and quit
msg = result[0][0].recv(bufferSize) # actually fetch the UDP data
msg = zeroterm(msg) # convert content into python string
# in next line, [] contain optional repeats
# message format is keyword:data[|keyword:data]
# where from 1...n keyword:data pairs may appear, up to 1024 bytes
# ----------------------------------------------------------------
try:
msgs = msg.split('|') # can be more than one message in packet
except: # failed to split
msgs = [] # on the other hand, can just be one. :)
msgs.append(msg) # so build array with that one.
for m in msgs: # now, for every message we have
keyw,data = m.split(':') # break into keyword and data
print keyw + "-->" + data # you'd do something with this
if keyw == "closing": # Our client terminates when SdrDx does
looping = False # loop stops
s.close() # must close socket
print 'Normal termination'
For reference, here's the Qt code that is sending the UDP message:
Setup:
bcast = new QHostAddress("192.168.1.255");
if (bcast)
{
udpSocketSend = new QUdpSocket(0);
if (udpSocketSend)
{
udpSocketSend->bind(*bcast, txudp);
}
}
Broadcast:
if (udpSocketSend)
{
QByteArray *datagram = new QByteArray(1024,0); // datagram is full of zeroes
strcpy(datagram->data(),msg); // except where text message is in it at beginning
udpSocketSend->writeDatagram(*datagram, QHostAddress::Broadcast,txudp); // send
}

You are trying to bind the same port, twice.
You bind it once in the sender:
if (udpSocketSend)
{
udpSocketSend->bind(*bcast, txudp);
}
and again at the receiver
s.bind(('', port))
And since these are running on the same machine, you are getting an error.
Unless you care what the source port is, you don't need to bind() on the sender, just send it and the stack will pick an appropriate outgoing source port number. In the case of a sender, when you transmit a UDP datagram you specify the destination (udpSocketSend->writeDatagram(...)), and the bind actually determines the source of the outgoing datagram. If you don't bind, thats fine, the stack will assign you a port.
If you do care what the source port is, then I suggest you use a different port number for outgoing source port and incoming destination port. Then you would be able to bind both sender and receiver without issue.
Lastly, there is the option to set the SO_REUSEADDR socket option (in whatever language you're using). This would be necessary if you want to run multiple clients on the same machine, as all the clients would have to bind to the same address. But, I'm not sure whether this socket option is cross platform (*nix works fine) and I think the above suggestions are better.

Correct multiprocessing to treat UDP in Python

I am trying to implement a simple UDP client and server. Server should receive a message and return a transformed one.
My main technique for server is to listen UDP messages in a loop, then spawn multiprocessing.Process for each incoming message and send the reply within each Process instance:
class InputProcessor(Process):
...
def run(self):
output = self.process_input()
self.sock.sendto(output, self.addr) # send a reply
if __name__ == "__main__":
print "serving at %s:%s" % (UDP_IP, UDP_PORT)
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP,UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message: %s from %s:%s" % (data, addr[0], addr[1])
p = InputProcessor(sock, data, addr)
p.start()
In test client, I do something like this:
def send_message(ip, port, data):
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
print "sending: %s" % data
sock.sendto(data, (ip, port))
sock.close()
for i in xrange(SECONDS*REQUESTS_PER_SECOND):
data = generate_data()
p = multiprocessing.Process(target=send_message, args=(UDP_IP,
UDP_PORT,
data))
p.start()
time.sleep(1/REQUESTS_PER_SECOND)
The problem I am having with the code above is that when REQUESTS_PER_SECOND becomes higher than certain value (~50), it seems some client processes receive responses destinated to different processes, i.e. process #1 receives response for process #2, and vice versa.
Please criticize my code as much as possible, due to I am new to network programming and may miss something obvious. Maybe it's even worth and better for some reason to use Twisted, hovewer, I am highly interested in understanding the internals. Thanks.

As per previous answer, I think that the main reason is that there is a race condition at the UDP port for the clients. I do not see receiving at the client code, but presumably it is similar to the one in server part. What I think happens in concrete terms is that for values under 50 requests / second, the request - response roundtrip gets completed and the client exits. When more requests arrive, there may be multiple processes blocking to read the UDP socket, and then it is probably nondeterministic which client process receives the incoming message. If the network latency is going to be larger in the real setting, this limit will be hit sooner.

Thanks guys a lot! It seems I've found why my code failed before. I was using multiprocessing.Manager().dict() within client to check if the results from server are correct. However, I didn't use any locks to wrap a set of write operations to that dict(), thus got a lot of errors though the output from server was correct.
Shortly, in client, I was doing incorrect checks for correct server responses.