Develop Reference

Inconsistent behaviour with TCP socket's recv() watermark on Linux

I was delving into socket tuneables, and I encountered the SO_RCVLOWAT option, so I created a test-case to determine whether the watermark eliminated a specific issue I've had in developing servers before (namely premature data truncation:)
def run_server(host: str, port: int, low_watermark: int):
sk_server = socket.socket()
sk_server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
sk_server.setsockopt(
socket.SOL_SOCKET, socket.SO_RCVLOWAT, low_watermark
)
sk_server.bind((host, port))
sk_server.listen(1)
cl_sockfd, cl_addr = sk_server.accept()
print(f"connected to client, receiving at least {low_watermark}"
" bytes of data")
assert len(cl_sockfd.recv(low_watermark)) == low_watermark
# *** never reached
print("all good!")
if __name__ == "__main__":
host, port = "localhost", 6969
watermark = 100
thd_server = threading.Thread(
target=run_server,
args=(host, port, watermark)
)
thd_server.start()
time.sleep(0.5)
client = socket.socket()
client.connect((host, port))
fragment_1 = b"a" * (n := watermark // 2)
fragment_2 = b"a" * (watermark - n)
print("sending first fragment, and waiting 1 second")
print("sent", client.send(fragment_1), "bytes")
time.sleep(1)
print("sending second fragment")
print("sent", client.send(fragment_2), "bytes")
print("done. waiting for server thread to finish")
thd_server.join()
In short, the client sends exactly watermark bytes to the server which is configured to receive at least watermark bytes, however the principal issue is that the server hangs on the cl_sockfd.recv(low_watermark).
As a sanity-check, I send a single fragment of size watermark, and of course this works as expected, and any single fragments below watermark are rejected. But, the curious observation I make is that with two fragments the only way the server will accept the send() is when the second fragment has size watermark, in which case the two fragments combine as expected, but the remainder length after watermark is discarded, but if I instead send three fragments, the first two of which combine to make length watermark, but the third fragment has size watermark // 2 then it all works, but now the third fragment is discarded.
This does not make sense at all. Is there any explanation for this?
Post-notes:
I experimented by enabling TCP_NODELAY so that send() buffers are immediately flushed, to question whether Linux was post-maturely flushing the buffer (after the third fragment triggered a flush,) but behaviour remained identical
In the scenario of three fragments, totalling 104 bytes of data, no extraneous data is left to recv(), which is expected considering the watermark enforces a minimum, but then after sending the remaining amount to total 2*watermark the recv() works perfectly as expected even though it's spread over two send()s, which again makes no sense considering it failed to receive the initial two fragments, despite them matching the same criteria.

Problems when storing and processing serial data

I am reading serial data and trying to sort and display individual data packets.
The problem is that when I receive a packet, I can print it using print(), and the packet prints perfectly OK. But when I add this packet to the list[] or try to add the packet to Listbox(tkinter), the packets sometimes get combined/scrambled and I get two packets displayed as one longer packet.
The serial data comes in irregular intervals and sometimes in bursts of few packets in a very short time.
Packet length is variable between 9 and 26 ASCII characters, and it is terminated with \r
Serial speed is at the moment limited to 115200, and this is a minimum speed to get bursts of packets transferred without chocking the communication.
Here is what I have tried so far:
tried to run processing without the queue, just directly process packets as they come – result, print() all packets printed ok, but when adding packets to list[] and/or Listbox(tkinter) some packets were combined into single packet.
tried to implement the FIFO queue, and add packets to the queue in one thread, and remove/process them in the other thread, so I thought they will be completely independent - result - same as above, print() all packets printed ok, but when adding packets to list[] and/or Listbox(tkinter) some packets were combined into single packet.
I have tried many variations of the code, and all of it failed. The print function displays packets correctly, but when I try to do anything with them, namely sort them into lists or add them to listbox, some of the packets get combined…. It seems that when packets are coming in bursts with minimal time delay between the packets, python can not process them fast enough? But the print function does keep up, and when I run separate threads for placing packets into queue and separate thread for reading them out of the queue, the timing should not be an issue, right?
Please help, I have exhausted all my ideas…
I am using serial read line function to get packets:
def serial_read_line(port):
ser = serial.Serial(port, 115200, timeout=0.01)
ser.flushInput()
while 1:
x = ser.readline()
if len(x) < 1:
pass
else:
y = str(x.decode('utf-8'))
if y != x:
x = y
return y
The above function is run in a separate thread so it does not block the program.
# *********** Scanning for packets **********
# scan for incoming packets
# set global variable for start/stop packet scan
stop = 1
q1 = queue.Queue()
def ser_packet_scan():
port = serial_port_selection.get()
global q1
while True:
packet_1 = sio.serial_read_line(port)
q1.put(packet_1)
process_packets()
if stop == 1:
break
# starting thread for packet scan
def start_thread():
global stop
stop = 0
th = Thread(target=ser_packet_scan)
th.start()
th2 = Thread(target=process_packets)
th2.start()
# stopping thread for packet scan
def stop_thread():
global stop
stop = 1
The processing is run in a separate thread, I have also tried to run it in the main program thread.
def process_packets():
# sort packets into lists
global q1
pckt_r = q1.get()
print(pckt_r)
pck.sort_into_lists(pckt_r)
# update packet lists
if list_box_selection.get() == 'Running':
list_box_display.insert(0, pckt_r)
# limit listbox display size
if list_box_display.size() > 100:
list_box_display.delete(END)
…
…
# time.sleep(0.5) have tried with the sleep and without it… no diference...
q1.task_done()

OK, I found a solution. I also found a couple of issues.
print() function does automatically new line and it understands \r, so my print output was always correct even though the packet was consisting of two or more packets.
the inbuilt function readline() did not work very well. Some packets were combined even though there was an \r character present.
Solution:
I have replaced inbuilt readline() with custom function:
def read_line_c(ser):
buf = bytearray()
i = buf.find(b"\r")
if i >= 0:
r = buf[:i + 1]
buf[i + 1:]
return r
while True:
i = max(1, min(2048, ser.in_waiting))
data = ser.read(i)
i = data.find(b"\r")
if i >= 0:
r = buf + data[:i + 1]
buf[0:] = data[i + 1:]
return r
else:
buf.extend(data)
The packets appear to be all OK with this custom readline, but I need to test it a bit more to see if there are any issues... Alternatively I could use the inbuilt readline(), and then look for packets which have \r in them and split them...

Serial stops reading after about 6 hours

When reading the serial port in python, all works fine until after hours where receiving data stops. This happens both on a laptop running Linux and on a Raspberry PI 2. Sending data still works at that point.
The source is still sending data (now and again). Restarting the python script helps.
The source sends small packages (4 - 12 bytes) and interval times may vary from minutes to over 12 hours.
My code, just only the receiving part, extracted from a bigger part, looks like this:
#!/usr/bin/python3
import tkinter
import os
import serial
class Transceiver():
def __init__(self, main):
self.port = serial.Serial(
port = "/dev/ttyUSB0",
baudrate = 38400,
bytesize = serial.EIGHTBITS,
parity = serial.PARITY_NONE,
stopbits = serial.STOPBITS_ONE,
timeout = 0
)
self.ReceivedData = bytearray()
self.Time = 0
def Receiver(self):
while (self.port.in_waiting > 0):
# Receiving data
self.Time = 0 # Reset time elapsed to 0
self.ReceivedData += self.port.read(1) # Add data to bytearray
if (self.Time == 20): # Some time elapsed, packet considered complete
print(''.join('{:02x} '.format(x) for x in self.ReceivedData))
self.port.reset_input_buffer()
self.ReceivedData = bytearray()
if (self.Time < 2000): # Just avoid running to 0 again
self.Time += 1 # Increment time elapsed
root.after(5, self.Receiver)
root = tkinter.Tk()
transceiver = Transceiver(root)
transceiver.Receiver()
root.mainloop()
It is just this receiving part that stops working after hours, I am trying to find the root cause.
So the above script is just an extract from a bigger script.
It controls the lights in the house. I want to run it on a raspberry PI, instead of a PC. The R'PI has a little 7" touch screen.
The bigger scripts has some decoding of the received data and updates a very simple GUI, just a bunch of small rectangles with some short name as text in it, indicating which lamps are on and off, and they act as buttons too, to toggle the lamps on and off. All works well, apart from the receiving part, though, in the end it may be possible that some other part disturbs the receiving part.
I have just started running just the above extraction of the script, to see if it keeps working without the GUI updates. I suspect the function Receiver fails to be restarted after some time. I need to have the above script running for a day to see of it keeps receiving though. I will update with the findings.
Update: The above script keeps working. The above is the extraction of a bigger program where only this part stops working after some hours. Receiving stops, but transmitting keeps working, hence I did not include that part.
By now I found the problem. In the receiver function, a decode function is called where above is the print line. Without calling the decode function, there is no problem. The decode function has some conditions, depending on the packet length and some values. Code for one of these conditions contained an error. I always started to run the script again in early evening, and then about 6 hours later a timer (at a fixed time late in the evening) would send the offending packet (a different format) that then caused the problem. The packet was not at fault, but my decoding of that packet.
Strange is that the script kept working, only the reception part stopped. I thought it stopped silently, but I printed quite a lot in this stage, hence I didn't see the error message telling me the problem of the use of an uninitialised variable. It's not in the above code. Hope the above code is useful for other people.

PySerial timeout and callback

I have this short snippet of code here that works fine, but I have problem of getting rid of the hardcoded part.
ser = serial.Serial()
ser.baudrate = 38400
ser.port = '/dev/ttyUSB0'
ser.parity = serial.PARITY_EVEN
ser.timeout = 1
ser.open()
ser.flushInput()
ser.write(command) #command here is a simple request for data to my device
msg = ser.read(200)
ser.close()
While this works fine, the problem I'm having is this. The length of the returned message can vary from 8 byte to almost 200 bytes depending on what was registered. By using a timeout, I prevent my read command from stalling if it doesn't receive 200 bytes. I also don't know ahead the length of the returned message I therefore can't change dynamically the ser.read. Also, there is no constant endline or constant character at the end of the transmission to lock on in a while loop.
Is there a more stable/dynamic way to do this? I could run out of time if the request is too long or I could bust my read buffer without having the complete data transmission. On the other end, increasing the timer mean that my request rate will be slowed down (there is no problem in increasing the read buffer however).

If the reply had a header containing a length field, you could do a fixed-size read() to get the header, then a variable read() to get the rest...
If there is truly no way to tell how big the reply is, then a timeout is the only conceivable solution. However, you have apparently missed the detail that PySerial has two different timeout values: one that applies to the overall operation, and one that applies to gaps between characters. You could set timeout to multiple seconds, so that you never prematurely end a valid reply, and set inter_byte_timeout (was interCharTimeout in older versions) to perhaps 0.1 second, so that your read() will end almost immediately once the device stops sending data. (This assumes that the device never inserts pauses in the middle of sending a reply.)

If the responses (from 8 bytes to 200 bytes) are contiguous, then you could have a loop which concatenates bytes received from calls to ser.read(200), but having set the timeout to something like 1/100th second. Then when you have two timeouts in succession where no bytes were received, then you know you are at the end of the message.
exit = 0
while exit < 2:
more = ser.read(200)
msg += more
if len(more) == 0:
exit += 1
else:
exit = 0

Read from socket: Is it guaranteed to at least get x bytes?

I have a rare bug that seems to occur reading a socket.
It seems, that during reading of data sometimes I get only 1-3 bytes of a data package that is bigger than this.
As I learned from pipe-programming, there I always get at least 512 bytes as long as the sender provides enough data.
Also my sender does at least transmit >= 4 Bytes anytime it does transmit anything -- so I was thinking that at least 4 bytes will be received at once in the beginning (!!) of the transmission.
In 99.9% of all cases, my assumption seems to hold ... but there are really rare cases, when less than 4 bytes are received. It seems to me ridiculous, why the networking system should do this?
Does anybody know more?
Here is the reading-code I use:
mySock, addr = masterSock.accept()
mySock.settimeout(10.0)
result = mySock.recv(BUFSIZE)
# 4 bytes are needed here ...
...
# read remainder of datagram
...
The sender sends the complete datagram with one call of send.
Edit: the whole thing is working on localhost -- so no complicated network applications (routers etc.) are involved. BUFSIZE is at least 512 and the sender sends at least 4 bytes.

I assume you're using TCP. TCP is a stream based protocol with no idea of packets or message boundaries.
This means when you do a read you may get less bytes than you request. If your data is 128k for example you may only get 24k on your first read requiring you to read again to get the rest of the data.
For an example in C:
int read_data(int sock, int size, unsigned char *buf) {
int bytes_read = 0, len = 0;
while (bytes_read < size &&
((len = recv(sock, buf + bytes_read,size-bytes_read, 0)) > 0)) {
bytes_read += len;
}
if (len == 0 || len < 0) doerror();
return bytes_read;
}

As far as I know, this behaviour is perfectly reasonable. Sockets may, and probably will fragment your data as they transmit it. You should be prepared to handle such cases by applying appropriate buffering techniques.
On other hand, if you are transmitting the data on the localhost and you are indeed getting only 4 bytes it probably means you have a bug somewhere else in your code.
EDIT: An idea - try to fire up a packet sniffer and see whenever the packet transmitted will be full or not; this might give you some insight whenever your bug is in your client or in your server.

The simple answer to your question, "Read from socket: Is it guaranteed to at least get x bytes?", is no. Look at the doc strings for these socket methods:
>>> import socket
>>> s = socket.socket()
>>> print s.recv.__doc__
recv(buffersize[, flags]) -> data
Receive up to buffersize bytes from the socket. For the optional flags
argument, see the Unix manual. When no data is available, block until
at least one byte is available or until the remote end is closed. When
the remote end is closed and all data is read, return the empty string.
>>>
>>> print s.settimeout.__doc__
settimeout(timeout)
Set a timeout on socket operations. 'timeout' can be a float,
giving in seconds, or None. Setting a timeout of None disables
the timeout feature and is equivalent to setblocking(1).
Setting a timeout of zero is the same as setblocking(0).
>>>
>>> print s.setblocking.__doc__
setblocking(flag)
Set the socket to blocking (flag is true) or non-blocking (false).
setblocking(True) is equivalent to settimeout(None);
setblocking(False) is equivalent to settimeout(0.0).
From this it is clear that recv() is not required to return as many bytes as you asked for. Also, because you are calling settimeout(10.0), it is possible that some, but not all, data is received near the expiration time for the recv(). In that case recv() will return what it has read - which will be less than you asked for (but consistenty < 4 bytes does seem unlikely).
You mention datagram in your question which implies that you are using (connectionless) UDP sockets (not TCP). The distinction is described here. The posted code does not show socket creation so we can only guess here, however, this detail can be important. It may help if you could post a more complete sample of your code.
If the problem is reproducible you could disable the timeout (which incidentally you do not seem to be handling) and see if that fixes the problem.

This is just the way TCP works. You aren't going to get all of your data at once. There are just too many timing issues between sender and receiver including the senders operating system, NIC, routers, switches, the wires themselves, the receivers NIC, OS, etc. There are buffers in the hardware, and in the OS.
You can't assume that the TCP network is the same as a OS pipe. With the pipe, it's all software so there's no cost in delivering the whole message at once for most messages. With the network, you have to assume there will be timing issues, even in a simple network.
That's why recv() can't give you all the data at once, it may just not be available, even if everything is working right. Normally, you will call recv() and catch the output. That should tell you how many bytes you've received. If it's less than you expect, you need to keep calling recv() (as has been suggested) until you get the correct number of bytes. Be aware that in most cases, recv() returns -1 on error, so check for that and check your documentation for ERRNO values. EAGAIN in particular seems to cause people problems. You can read about it on the internet for details, but if I recall, it means that no data is available at the moment and you should try again.
Also, it sounds like from your post that you're sure the sender is sending the data you need sent, but just to be complete, check this:
http://beej.us/guide/bgnet/output/html/multipage/advanced.html#sendall
You should be doing something similar on the recv() end to handle partial receives. If you have a fixed packet size, you should read until you get the amount of data you expect. If you have a variable packet size, you should read until you have the header that tells you how much data you send(), then read that much more data.

From the Linux man page of recv http://linux.about.com/library/cmd/blcmdl2_recv.htm:
The receive calls normally return any
data available, up to the requested
amount, rather than waiting for
receipt of the full amount requested.
So, if your sender is still transmitting bytes, the call will only give what has been transmitted so far.

If the sender sends 515 bytes, and your BUFSIZE is 512, then the first recv will return 512 bytes, and the next will return 3 bytes... Could this be what's happening?
(This is just one case amongst many which will result in a 3-byte recv from a larger send...)

If you are still interested, patterns like this :
# 4 bytes are needed here ......
# read remainder of datagram...
may create the silly window thing.
Check this out

Use recv_into(...) method from the socket module.
Robert S. Barnes written the example in C.
But you can use Python 2.x with standard python-libraries:
def readReliably(s,n):
buf = bytearray(n)
view = memoryview(buf)
sz = s.recv_into(view,n)
return sz,buf
while True:
sk,skfrom = s.accept()
sz,buf = io.readReliably(sk,4)
a = struct.unpack("4B",buf)
print repr(a)
...
Notice, that sz returned by readReliably() function may be greater than n.