Develop Reference

asyncio pyserial - how to determine if read buffer is empty

I'm using asyncio and trying to asynchronously read from a serial port using pyserial-asyncio which uses asyncio.StreamReader. In synchronous pyserial one can use the client's in_waiting getter to determine if there's anything sitting in the read buffer.
asyncio.StreamReader has the at_eof() member but this relies on the user to assert feed_eof()
Others have suggested simply doing await asyncio.StreamReader.read() and checking for empty bytes but in the case of pyserial-asyncio at least, this never returns empty bytes - it just awaits indefinitely (returns when some bytes are actually received)
I could use asyncio.wait_for() to set a timeout and use this as a clue that the buffer is empty but that's obviously suboptimal.
How can I effectively check for an empty read buffer?

Try to access the buffer directly asyncio.StreamReader._buffer.
if len(my_output._buffer) > 0:
print("not empty")
else:
print("empty")

Strange blocking behaviour when reading sys.stdin in python 2 while having a custom signal handler in place

Consider this small python script odd-read-blocking.py:
#!/usr/bin/python
import signal
import sys
sig = None
def handler(signum, frame):
global sig
sig = signum
signal.signal(signal.SIGINT, handler)
signal.signal(signal.SIGTERM, handler)
x = sys.stdin.read(3)
print 'signal', sig
print 'read bytes', len(x)
exit(0)
I run this and feed it with two bytes of standard input data ('a' + '\n'):
> echo a | ./odd-read-blocking.py
signal None
read bytes 2
>
Fine.
Now I feed it with the same two bytes (by typing 'a' + '\n' into its standard input). Please note that standard input is then not at EOF yet and potentially has more data to come. So the read blocks, as it expects one more byte. I use Ctrl+C on the script.
> ./odd-read-blocking.py
a
^Csignal 2
read bytes 2
>
Fine. We see that two bytes have been read and signal 2 was received.
Now I open a standard input stream, but do not send any byte on it. The read blocks as expected. If I now use Ctrl+C on the script, it will keep sitting there and wait. The read will not be interrupted. The SIGINT will not be processed.
> ./odd-read-blocking.py
^C
Nothing here. Script still running (seemingly blocked at the read).
Now hitting return once, then Ctrl+C again:
^Csignal 2
read bytes 1
>
So, only after receiving at least some data (a single '\n' in this case) on its standard input will the script behave as I expect it and correctly interrupt the blocked read and tell me it has received signal 2 and read 1 byte.
Alternative 1: instead of using Ctrl+C as shown above, I have tried this same thing using kill pid from a separate terminal. The behaviour is the same.
Alternative 2: instead of using the shell standard input as described above, I have done this:
> sleep 2000 | ./odd-read-blocking.py
When using kill pid to send SIGTERM to the odd-read-blocking.py process I get the same behaviour. Here, the script process can only be killed using SIGKILL (9).
Why isn't the read interrupted, when it is blocking on an as yet empty but still active standard input stream?
I find this odd. Who doesn't? Who can explain?

The short version
If a Python signal handler throws an exception to abandon an ongoing file.read, any data already read is lost. (Any asynchronous exception, like the default KeyboardInterrupt, makes it basically impossible to prevent this sort of failure unless you have a way to mask it.)
To minimize the need for this, file.read returns early (i.e., with a shorter string than requested) when it is interrupted by a signal—note that this is in addition to the EOF and non-blocking I/O cases that are documented! However, it can't do this when it has no data yet, since it returns the empty string to indicate EOF.
Details
As always, the way to understand behavior like this is with strace.
read(2)
The actual read system call has a dilemma when a signal arrives while the process is blocked. First, the (C) signal handler gets invoked—but because that could happen between any two instructions, there's very little it can do beyond setting a flag (or writing to a self-pipe). Then what? If SA_RESTART is set, the call is resumed; otherwise…
If no data has been transferred yet, read can fail and the client can check its signal flag. It fails with the special EINTR to clarify that nothing actually went wrong with the I/O.
If some data has already been written into the (userspace) buffer, it can't just return "failure", because data would be lost—the client can't know how much (if any) data is in the buffer. So it just returns success (the number of bytes read so far)! Short reads like this are always a possibility: the client has to call read again to check that it has reached end of file. (Just like file.read, a short read of 0 bytes would be EOF.) The client therefore has to check their signal flag after every read, whether it succeeds or not. (Note that this is still not perfectly reliable, but it's good enough for many interactive use cases.)
file.read()
The system call isn't the whole story: after all, the normal configuration for a terminal has it return immediately after seeing a newline. Python 2's low-level file.read is a wrapper for fread, which will issue another read if one is short. But when a read fails with EINTR, fread returns early and file.read calls your (Python) signal handler. (If you add output to it, you'll see that it's called immediately for each signal you send, even if file.read doesn't return.)
Then it's faced with a dilemma similar to that for the system call: as discussed, a short read can't be empty because it means EOF. Unlike a C signal handler, however, a Python one can do arbitrary work (including raising an exception to abort the I/O immediately, at the cost of risking data loss as mentioned at the beginning), and it's considered a convenient simplification to the interface to hide the possibility EINTR. So the fread call is just silently repeated.
Python 3.5
The rules for retrying changed in 3.5. Now the io.IOBase.read resumes even if it has data in hand; this is more consistent, but it forces the use of exceptions to stop reading, which means that you can't opt to wait on some data in order not to risk losing any you already have. The very heavyweight solution is to switch to multiplexed I/O and use signal.set_wakeup_fd(); this has the added advantage of allowing SIGINT to affect the main thread without having to bother with masking it in all the others.

Breaking out of while True loop

I have a python script which will parse xml file for serial numbers and will write them to a text file. The problem with the below code is, It is going on infinite loop. If I am adding a break statement some where after logging to a file, It is writing only one serial number. How do I increase the counter, so that the program will exit after writing all the serial numbers.
try:
while True:
data, addr = s.recvfrom(65507)
mylist=data.split('\r')
url = re.findall('http?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', data)
print url[0]
response = urllib2.urlopen(url[0])
the_page = response.read()
tree = ET.XML(the_page)
with open("temp.xml", "w") as f:
f.write(ET.tostring(tree))
document = parse('temp.xml')
actors = document.getElementsByTagName("ns0:serialNumber")
for act in actors:
for node in act.childNodes:
if node.nodeType == node.TEXT_NODE:
r = "{}".format(node.data)
print r
logToFile(str(r))
time.sleep(10)
s.sendto(msg, ('239.255.255.250', 1900) )
except socket.timeout:
pass

I would normally create a flag so that the while would be
while working == True:
Then reset the flag at the appropriate time.
This allows you to use the else statement to close the text file and output the final results after the while loop is complete. Else clause on Python while statement.
Note that it is always better to explicitly close open files when finished rather than relying on garbage collection. You should also close the file and output a timeout message in the except logic.
For debugging, you can output a statement at each write to the text file.

If your s.recvfrom(65507) is working correctly it should be an easy fix. Write this code just below your data, addr = s.recvfrom(65507)
if not data:
break

You open a UDP socket and you use recvfrom to get data from the socket.
You set a high timeout which makes this function a blocking function. It means when you start listening on the socket, if no data have been sent from the sender your program will be blocked on that line until either the sender sends something or the timeout reaches. In case of timeout and no data the function will raise an Exception.
I see two options:
Send something from the sender that indicates the end of stream (the serial numbers in your case).
Set a small timeout then catch the Exception and use it to break the loop.
Also, take a look at this question: socket python : recvfrom
Hope it helps.

Python TCP Sockets: How to know if a specific connection has sent information

I have a multi-threaded Python 3 application that on thread #1 accepts TCP socket communications. Thread #2 will check all current connections if they have anything to receive, then act accordingly.
So, currently I have a list called all_connections which is a list of accepted socket connection objects.
Using for connection in all_connections: I can loop through all the connection objects. I know I use conn.recv(256) to check if there is anything ready to recive on this socket. Will this block the loop though untill there is something to receive? I have set conn.setblocking(1) beforehand although Im unsure if this is the best way to get around it:
Here is some example code:
Thread 1
self.all_connections = [] # init a list to hold connection objs
while 1:
try:
conn, address = self.socket.accept()
conn.setblocking(1) # non blocking
except Exception as e:
continue
self.all_connections.append(conn) # Save the connection object
Thread 2
while True:
for connection in self.all_connections:
received = connection.recv(256)
return
So, I'm only interested in connections that have actually sent something, as I will be sending them something back most likely.
I know I can use select.select in order to check if there is anything to receive on the socket, but that wouldn't help me reference the specific connection.

Yes, read() will block; this is the default behaviour. Calling socket.setblocking(1) actually enables blocking, which is opposite of what you wanted. setblocking(False) will set non-blocking mode. I/O on non-blocking sockets requires that you use exception handling.
A better way, and you are already headed in the right direction, is to use select(). You do in fact know which socket sent data because select() returns a list of sockets that are available for reading, writing, or that have an error status. You pass to select() a list of the sockets that you are interested in and it returns those that are available for I/O. Here is the function signature:
select(...)
select(rlist, wlist, xlist[, timeout]) -> (rlist, wlist, xlist)
So the code in thread 2 would look something like this:
from select import select
while True:
rlist, wlist, xlist = select(self.all_connections, [], [])
for connection in rlist:
received = connection.recv(256)
The above code only checks for readable sockets in the list of all connections and reads data from those that are ready. The read will not block.

Non-blocking socket in Python?

Is it me, or can I not find a good tutorial on non-blocking sockets in python?
I'm not sure how to exactly work the .recv and the .send in it. According to the python docs, (my understanding of it, at least) the recv'ed or send'ed data might be only partial data. So does that mean I have to somehow concatenate the data while recv and make sure all data sends through in send. If so, how? An example would be much appreciated.

It doesn't really matter if your socket is in non-blocking mode or not, recv/send work pretty much the same; the only difference is that non-blocking socket throws 'Resource temporarily unavailable' error instead of waiting for data/socket.
recv method returns numbers of bytes received, which is told to be less or equal to the passed bufsize. If you want to receive exactly size bytes, you should do something similar to the following code:
def recvall(sock, size):
data = ''
while len(data) < size:
d = sock.recv(size - len(data))
if not d:
# Connection closed by remote host, do what best for you
return None
data += d
return data
This is important to remember, that in blocking mode you have to do exactly the same. (The number of bytes passed to application layer is for example limited by recv buffer size in the OS.)
send method returns number of bytes sent, which is told to be less or equal to the length of passed string. If you want to ensure the whole message was sent, you should do something similar to the following code:
def sendall(sock, data):
while data:
sent = sock.send(data)
data = data[sent:]
You can use sock.sendall directly, but (according to the documentation) on error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent.
The sockets in Python follow the BSD socket API and behave in the similar way to c-style sockets (the difference is, for example, they throw exception instead of returning error code). You should be happy with any socket tutorial on the web and manpages.

Keep bytes you want to send in a buffer. (A list of byte-strings would be best, since you don't have to concatenate them.) Use the fcntl.fcntl function to set the socket in non-blocking mode:
import fcntl, os
fcntl.fcntl(mysocket, fcntl.F_SETFL, os.O_NONBLOCK)
Then select.select will tell you when it is OK to read and write to the socket. (Writing when it is not OK will give you the EAGAIN error in non-blocking mode.) When you write, check the return value to see how many bytes were actually written. Eliminate that many bytes from your buffer. If you use the list-of-strings approach, you only need to try writing the first string each time.
If you read the empty string, your socket has closed.