Develop Reference

Python Socket - How to properly handle timeout properly?

Using
data, addr = sock.recvfrom(1024)
i wonder how someone can handle if a timeout. Let's say i send packets over and recieve them as shown above, everything works, except that when a transmitter stops sending further packets, the socket on the reciever side just waits and continues if a new paket come. However, i would like to have some kind of counter that waits for a specific time and closes the socket after no new paket comes in.
To do so, i have to somehow catch that moment when data is no longer received but i haven't found really a way of doing it properly with socket
I've tried something like
try:
data, addr = sock.recvfrom(1024)
except:
print("connection lost")
but it doesn't work (expected that already). I also tried a while loop like so:
while True:
data, addr = sock.recvfrom(1024)
which works, but again i cannot check if the connection is lost.
Any ideas how to deal with this? Thanks

You can use socket.settimeout() function to set the socket to wait for a particular amount of time for receving data. If you are creating socket as socket.SOCK_DGRAM, it is a UDP socket. UDP is connectionless protocol. With TCP sockets you can simply use recv() and send() method. You can also put your code under "try catch" as shown below to handle socket error...
try:
# Code
except socket.error as exception:
print("socket.error : ", exception)
finally:
pass

Code will not continue after breaking from a while loop receiving client data

I have created a client/server setup for transferring PGP signature information from the client to the server. The code below shows part of the server code, where adds signatures received from the client to an output text file
However, my code isn't able to move on after the second while loop, after breaking.
It receives 2 signatures from the client, and successfully prints the "test" string only twice and adds both received strings to the output file, but the program will not continue after breaking, and doesn't print the other "test2" and "test3" strings.
while True:
# Accepts incoming connection, creating socket used for data transfer to client
conn, addr = server.accept()
print("Connected successfully")
directory = "(Hidden for question)"
outputFile = open((directory + "\\signatures.txt"), "w")
while True:
data = conn.recv(2048)
if not data: break
print("test")
outputFile.write(data.decode()+"\n")
outputFile.flush()
print("test2")
conn.close()
print("test3")
I feel like I am missing something very obvious but cannot figure out what the issue is.

Your loop will never break as the recv function on a socket is a blocking call.
This means the function will not return until it receives some data, there for not data will always be false.
Try sending more information (after the first 2 signatures) into the socket and see that your script will continue to write it into the file.
If you want to receive a specific amount of data/times, track it using a variable and break your loop using that.

Alternatively to #Nadav's answer, remove the inner while loop. Since recv() is synchronous, you don't need to loop.

how do I handle sockets which is readable but recv nothing?

Recently I've been writing a http/https proxy using socket on python, and I use select to decide whether a socket is ready to recv or send, something like this:
inputs = [sock1, sock2...]
outputs = [sock1, sock2...]
while True:
readable, writable, exceptional = select.select(inputs, outputs, inputs)
for s in readable:
do something with s.recv()
for s in writable:
do something with s.send()
and I notice that sometimes a socket is readable but when I do s.recv() it returns empty. And since I have a while True loop outside so it gets nothing again and again. I tried to close the socket at first:
data = s.recv(4096)
if not data:
inputs.remove(s)
s.close()
but after a while some socket comes alive and they can't send msg to me because I've already closed it.
Does that mean I shouldn't close it? So what should I do with that?

When a socket in the receive list returns zero bytes on recv, it means that the socket has been closed from the other side. It will never receive data again and should be shutdown and closed on your side too.
You way "some socket comes alive" - its not the socket on the other side of this connection, that one has already shutdown.

Breaking out of while True loop

I have a python script which will parse xml file for serial numbers and will write them to a text file. The problem with the below code is, It is going on infinite loop. If I am adding a break statement some where after logging to a file, It is writing only one serial number. How do I increase the counter, so that the program will exit after writing all the serial numbers.
try:
while True:
data, addr = s.recvfrom(65507)
mylist=data.split('\r')
url = re.findall('http?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', data)
print url[0]
response = urllib2.urlopen(url[0])
the_page = response.read()
tree = ET.XML(the_page)
with open("temp.xml", "w") as f:
f.write(ET.tostring(tree))
document = parse('temp.xml')
actors = document.getElementsByTagName("ns0:serialNumber")
for act in actors:
for node in act.childNodes:
if node.nodeType == node.TEXT_NODE:
r = "{}".format(node.data)
print r
logToFile(str(r))
time.sleep(10)
s.sendto(msg, ('239.255.255.250', 1900) )
except socket.timeout:
pass

I would normally create a flag so that the while would be
while working == True:
Then reset the flag at the appropriate time.
This allows you to use the else statement to close the text file and output the final results after the while loop is complete. Else clause on Python while statement.
Note that it is always better to explicitly close open files when finished rather than relying on garbage collection. You should also close the file and output a timeout message in the except logic.
For debugging, you can output a statement at each write to the text file.

If your s.recvfrom(65507) is working correctly it should be an easy fix. Write this code just below your data, addr = s.recvfrom(65507)
if not data:
break

You open a UDP socket and you use recvfrom to get data from the socket.
You set a high timeout which makes this function a blocking function. It means when you start listening on the socket, if no data have been sent from the sender your program will be blocked on that line until either the sender sends something or the timeout reaches. In case of timeout and no data the function will raise an Exception.
I see two options:
Send something from the sender that indicates the end of stream (the serial numbers in your case).
Set a small timeout then catch the Exception and use it to break the loop.
Also, take a look at this question: socket python : recvfrom
Hope it helps.

Socket with mysterious buffer

I am building a python based interface for pulling data over TCP from an instrument. The datastream comes as specific events, and the timing is not steady: I get bursts of data and then slow periods. They are small data packets, so for simplicity assume they come across as complete packets.
Here is the behavior I get from the socket:
Send Event #1: socket.recv returns event #1
Send Event #2: socket.recv returns event #2
Quickly Send Event #3-50: socket.recv returns only events #3-30 (returns 27 times)
Slowly send Event #51: socket returns.recv event #31
Slowly send Event #52: socket returns.recv event #32
No data is lost. But there is clearly a buffer somewhere that is filled, and the socket is now returning old data. But shouldn't recv just keep returning till that buffer is empty? Instead, it is only returning when it receives a new packet, despite having a buffer of packets built up. Weird!
Here is the essence of the code (this is for non-blocking, I've also done blocking with just recv - same result). For simplicity I stripped all the packet reassembly stuff. I've carefully traced it back to the socket, so I know that is not to blame.
class mysocket:
def __init__(self,ip,port):
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((ip,port))
self.keepConn = True
self.socket.setblocking(0)
threading.Thread(target = self.rcvThread).start()
threading.Thread(target = self.parseThread).start()
def rcvThread(self):
while self.keepConn:
readable,writable,inError = select([self.socket],[self.socket],[],.1)
if readable:
packet = self.socket.recv(4096)
self.recvqueue.put_nowait(packet)
try:
xmitmsg = self.sendqueue.get_nowait()
except Queue.Empty:
pass
else:
if writable:
self.socket.send(xmitmsg)
def parseThread(self,rest = .1):
while self.keepConn:
try:
output = self.recvqueue.get_nowait()
eventnumber = struct.unpack('<H',output[:2]
print eventnumber
except Queue.Empty:
sleep(rest)
Why can't I get the socket to dump all the data in it's buffer? I can never catch up! This one is too odd. Anybody have pointers?
I'm an amateur but I've really done my homework on this one and am completely baffled.

packet = self.socket.recv(4096)
self.recvqueue.put_nowait(packet)
TCP is a stream-based protocol, not a message-based one. It doesn't preserve message boundaries. Meaning you can't expect to have one recv() call per message. If you send data in a burst, Nagle's algorithm will combine the data into one TCP packet.
Your code assumes that each recv() call returns one "packet", and the parse thread prints the first number from each "packet". But recv() doesn't return packets, it returns chunks of data from the TCP stream. These chunks can contain one message or multiple messages or even partial messages. There's no guarantee that the first two bytes are always event numbers.
Typically, reading data from a TCP connection involves calling recv() multiple times and storing the data you get in a buffer. Once you've received an entire message then you remove the appropriate number of bytes from the buffer and process them.
If you have variable-length messages then you need to keep track of message boundaries yourself. TCP doesn't do it for you like UDP does. That means adding a header containing the message length to the front of each message.
try:
xmitmsg = self.sendqueue.get_nowait()
except Queue.Empty:
pass
else:
if writable:
self.socket.send(xmitmsg)
On another note, it looks like this code has a bug. It removes messages from the sendqueue whether or not the socket is writable. If the socket's not writable it'll silently throw away messages.