I have the following python code:
import pty
import subprocess
os=subprocess.os
from subprocess import PIPE
import time
import resource
pipe=subprocess.Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=PIPE, \
close_fds=True)
skip=[f.fileno() for f in (pipe.stdin, pipe.stdout, pipe.stderr)]
pid, child_fd = pty.fork()
if(pid==0):
max_fd=resource.getrlimit(resource.RLIMIT_NOFILE)[0]
fd=3
while fd<max_fd:
if(fd not in skip):
try:
os.close(fd)
except OSError:
pass
fd+=1
enviroment=os.environ.copy()
enviroment.update({"FD": str(pipe.stdin.fileno())})
os.execvpe("zsh", ["-i", "-s"], enviroment)
else:
os.write(child_fd, "echo a >&$FD\n")
time.sleep(1)
print pipe.stdout.read(2)
How can I rewrite it so that it will not use Popen and cat? I need a way to pass data from a shell function running in the interactive shell that will not mix with data created by other functions (so I cannot use stdout or stderr).
Ok, I think I've got a handle on your question now, and see two different approaches you could take.
If you absolutely want to provide the shell in the child process with an already-open file descriptor, then you can replace the Popen() of cat with a call to os.pipe(). That will give you a connected pair of real file descriptors (not Python file objects). Anything written to the second file descriptor can be read from the first, replacing your jury-rigged cat-pipe. (Although "cat-pipe" is fun to say...). A socket pair (socket.socketpair()) can also be used to achieve the same end if you need a bidirectional pair.
Alternatively, you could simplify your life even further by using a named pipe (aka FIFO). If you aren't familiar with the facility, a named pipe is a uni-directional pipe located in the filesystem namespace. The os.mkfifo() function will create the pipe on the filesystem. You can then open the pipe for reading in your primary process and open it for writing / direct output to it from your shell child process. This should simplify your code and open the option of using an existing library like Pexpect to interact with the shell.
Related
I'm using Popen to run an external process. Here is a deliberately simple example to illustrate; ls writes to standard output, and to standard error for each missing file:
#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE
def test_process(*command):
process = Popen(command)
process.communicate()
test_process('ls', '/etc/hosts', 'bad1', 'bad2', 'bad3')
Output as follows:
/etc/hosts
ls: cannot access 'bad1': No such file or directory
ls: cannot access 'bad2': No such file or directory
ls: cannot access 'bad3': No such file or directory
However, I would like to filter out certain lines from standard error, so I need to pass something into the stderr parameter of the Popen constructor.
According to the Python documentation (and confirmed by stepping through the code in an IDE) the stderr parameter can only be PIPE, DEVNULL, an integer file descriptor or a file-like object wrapping a (valid) file descriptor. I tried creating a class and passing a new instance in stderr, with the idea of implementing the missing methods to do the filtering; but that doesn't work because its fileno method has to return an actual file descriptor.
So I think I need to pass stderr=PIPE instead, and do the filtering on the other end of the pipe that gets created. This was my attempt:
def test_process(*command):
process = Popen(command, stderr=PIPE)
output, error = process.communicate()
if error:
lines = error.decode().splitlines()
for line in lines:
if '2' not in line:
print(line, file=sys.stderr)
However, while I don't expect standard error to contain a lot of lines, it would be more elegant to filter it on the fly. Is there some way I can write some Python code that receives one line at a time, and either discards it or passes it on? I suppose I could create a new subprocess to do the filtering, and chain another Popen onto the first one, but that seems pretty heavyweight for the task.
I certainly don't want to interrupt the standard output stream, which is being written to the actual console, so I can't just read one line at a time from standard error, the way I might for standard output.
Any better ideas?
I've got a main process in which I run a subprocess, which stdin is what I want to pipe. I know I can do it using files:
import subprocess
subprocess.call('shell command', stdin=open('somefile','mode'))
Is there any option to use a custom stdin pipe WITHOUT actual hard drive files? Is there any option, for example, to use string list (each list element would be a newline)?
I know that python subprocess calls .readline() on the pipe object.
First, use subprocess.Popen - .call is just a shortcut for it, and you'll need to access the Popen instance so you can write to the pipe. Then pass subprocess.PIPE flag as the stdin kwarg. Something like:
import subprocess
proc = subprocess.Popen('shell command', stdin=subprocess.PIPE)
proc.stdin.write("my data")
http://docs.python.org/2/library/subprocess.html#subprocess.PIPE
I have an app that reads in stuff from stdin and returns, after a newline, results to stdout
A simple (stupid) example:
$ app
Expand[(x+1)^2]<CR>
x^2 + 2*x + 1
100 - 4<CR>
96
Opening and closing the app requires a lot of initialization and clean-up (its an interface to a Computer Algebra System), so I want to keep this to a minimum.
I want to open a pipe in Python to this process, write strings to its stdin and read out the results from stdout. Popen.communicate() doesn't work for this, as it closes the file handle, requiring to reopen the pipe.
I've tried something along the lines of this related question:
Communicate multiple times with a process without breaking the pipe? but I'm not sure how to wait for the output. It is also difficult to know a priori how long it will take the app to finish to process for the input at hand, so I don't want to make any assumptions. I guess most of my confusion comes from this question: Non-blocking read on a subprocess.PIPE in python where it is stated that mixing high and low level functions is not a good idea.
EDIT:
Sorry that I didn't give any code before, got interrupted. This is what I've tried so far and it seems to work, I'm just worried that something goes wrong unnoticed:
from subprocess import Popen, PIPE
pipe = Popen(["MathPipe"], stdin=PIPE, stdout=PIPE)
expressions = ["Expand[(x+1)^2]", "Integrate[Sin[x], {x,0,2*Pi}]"] # ...
for expr in expressions:
pipe.stdin.write(expr)
while True:
line = pipe.stdout.readline()
if line != '':
print line
# output of MathPipe is always terminated by ';'
if ";" in line:
break
Potential problems with this?
Using subprocess, you can't do this reliably. You might want to look at using the pexpect library. That won't work on Windows - if you're on Windows, try winpexpect.
Also, if you're trying to do mathematical stuff in Python, check out SAGE. They do a lot of work on interfacing with other open-source maths software, so there's a chance they've already done what you're trying to.
Perhaps you could pass stdin=subprocess.PIPE as an argument to subprocess.Popen. This will make the process' stdin available as a general file-like object:
import sys, subprocess
proc = subprocess.Popen(["mathematica <args>"], stdin=subprocess.PIPE,
stdout=sys.stdout, shell=True)
proc.stdin.write("Expand[ (x-1)^2 ]") # Write whatever to the process
proc.stdin.flush() # Ensure nothing is left in the buffer
proc.terminate() # Kill the process
This directs the subprocess' output directly to your python process' stdout. If you need to read the output and do some editing first, that is possible as well. Check out http://docs.python.org/library/subprocess.html#popen-objects.
Again, the same question.
The reason is - I still can't make it work after reading the following:
Real-time intercepting of stdout from another process in Python
Intercepting stdout of a subprocess while it is running
How do I get 'real-time' information back from a subprocess.Popen in python (2.5)
catching stdout in realtime from subprocess
My case is that I have a console app written in C, lets take for example this code in a loop:
tmp = 0.0;
printf("\ninput>>");
scanf_s("%f",&tmp);
printf ("\ninput was: %f",tmp);
It continuously reads some input and writes some output.
My python code to interact with it is the following:
p=subprocess.Popen([path],stdout=subprocess.PIPE,stdin=subprocess.PIPE)
p.stdin.write('12345\n')
for line in p.stdout:
print(">>> " + str(line.rstrip()))
p.stdout.flush()
So far whenever I read form p.stdout it always waits until the process is terminated and then outputs an empty string. I've tried lots of stuff - but still the same result.
I tried Python 2.6 and 3.1, but the version doesn't matter - I just need to make it work somewhere.
Trying to write to and read from pipes to a sub-process is tricky because of the default buffering going on in both directions. It's extremely easy to get a deadlock where one or the other process (parent or child) is reading from an empty buffer, writing into a full buffer or doing a blocking read on a buffer that's awaiting data before the system libraries flush it.
For more modest amounts of data the Popen.communicate() method might be sufficient. However, for data that exceeds its buffering you'd probably get stalled processes (similar to what you're already seeing?)
You might want to look for details on using the fcntl module and making one or the other (or both) of your file descriptors non-blocking. In that case, of course, you'll have to wrap all reads and/or writes to those file descriptors in the appropriate exception handling to handle the "EWOULDBLOCK" events. (I don't remember the exact Python exception that's raised for these).
A completely different approach would be for your parent to use the select module and os.fork() ... and for the child process to execve() the target program after directly handling any file dup()ing. (Basically you'd be re-implement parts of Popen() but with different parent file descriptor (PIPE) handling.
Incidentally, .communicate, at least in Python's 2.5 and 2.6 standard libraries, will only handle about 64K of remote data (on Linux and FreeBSD). This number may vary based on various factors (possibly including the build options used to compile your Python interpreter, or the version of libc being linked to it). It is NOT simply limited by available memory (despite J.F. Sebastian's assertion to the contrary) but is limited to a much smaller value.
Push reading from the pipe into a separate thread that signals when a chunk of output is available:
How can I read all availably data from subprocess.Popen.stdout (non blocking)?
The bufsize=256 argument prevents 12345\n from being sent to the child process in a chunk smaller than 256 bytes, as it will be when omitting bufsize or inserting p.stdin.flush() after p.stdin.write(). Default behaviour is line-buffering.
In either case you should at least see one empty line before blocking as emitted by the first printf(\n...) in your example.
Your particular example doesn't require "real-time" interaction. The following works:
from subprocess import Popen, PIPE
p = Popen(["./a.out"], stdin=PIPE, stdout=PIPE)
output = p.communicate(b"12345")[0] # send input/read all output
print output,
where a.out is your example C program.
In general, for a dialog-based interaction with a subprocess you could use pexpect module (or its analogs on Windows):
import pexpect
child = pexpect.spawn("./a.out")
child.expect("input>>")
child.sendline("12345.67890") # send a number
child.expect(r"\d+\.\d+") # expect the number at the end
print float(child.after) # assert that we can parse it
child.close()
I had the same problem, and "proc.communicate()" does not solve it because it waits for process terminating.
So here is what is working for me, on Windows with Python 3.5.1 :
import subprocess as sp
myProcess = sp.Popen( cmd, creationflags=sp.CREATE_NEW_PROCESS_GROUP,stdout=sp.PIPE,stderr=sp.STDOUT)
while i<40:
i+=1
time.sleep(.5)
out = myProcess.stdout.readline().decode("utf-8").rstrip()
I guess creationflags and other arguments are not mandatory (but I don't have time to test), so this would be the minimal syntax :
myProcess = sp.Popen( cmd, stdout=sp.PIPE)
for i in range(40)
time.sleep(.5)
out = myProcess.stdout.readline()
I have some data that I would like to gzip, uuencode and then print to standard out. What I basically have is:
compressor = Popen("gzip", stdin = subprocess.PIPE, stdout = subprocess.PIPE)
encoder = Popen(["uuencode", "dummy"], stdin = compressor.stdout)
The way I feed data to the compressor is through compressor.stdin.write(stuff).
What I really need to do is to send an EOF to the compressor, and I have no idea how to do it.
At some point, I tried compressor.stdin.close() but that doesn't work -- it works well when the compressor writes to a file directly, but in the case above, the process doesn't terminate and stalls on compressor.wait().
Suggestions? In this case, gzip is an example and I really need to do something with piping the output of one process to another.
Note: The data I need to compress won't fit in memory, so communicate isn't really a good option here. Also, if I just run
compressor.communicate("Testing")
after the 2 lines above, it still hangs with the error
File "/usr/lib/python2.4/subprocess.py", line 1041, in communicate
rlist, wlist, xlist = select.select(read_set, write_set, [])
I suspect the issue is with the order in which you open the pipes. UUEncode is funny is that it will whine when you launch it if there's no incoming pipe in just the right way (try launching the darn thing on it's own in a Popen call to see the explosion with just PIPE as the stdin and stdout)
Try this:
encoder = Popen(["uuencode", "dummy"], stdin=PIPE, stdout=PIPE)
compressor = Popen("gzip", stdin=PIPE, stdout=encoder.stdin)
compressor.communicate("UUencode me please")
encoded_text = encoder.communicate()[0]
print encoded_text
begin 644 dummy
F'XL(`%]^L$D``PL-3<U+SD])5<A-52C(24TL3#4`;2O+"!(`````
`
end
You are right, btw... there is no way to send a generic EOF down a pipe. After all, each program really defines its own EOF. The way to do it is to close the pipe, as you were trying to do.
EDIT: I should be clearer about uuencode. As a shell program, it's default behaviour is to expect console input. If you run it without a "live" incoming pipe, it will block waiting for console input. By opening the encoder second, before you had sent material down the compressor pipe, the encoder was blocking waiting for you to start typing. Jerub was right in that there was something blocking.
This is not the sort of thing you should be doing directly in python, there are eccentricities regarding the how thing work that make it a much better idea to do this with a shell. If you can just use subprocess.Popen("foo | bar", shell=True), then all the better.
What might be happening is that gzip has not been able to output all of its input yet, and the process will no exit until its stdout writes have been finished.
You can look at what system call a process is blocking on if you use strace. Use ps auxwf to discover which process is the gzip process, then use strace -p $pidnum to see what system call it is performing. Note that stdin is FD 0 and stdout is FD 1, you will probably see it reading or writing on those file descriptors.
if you just want to compress and don't need the file wrappers consider using the zlib module
import zlib
compressed = zlib.compress("text")
any reason why the shell=True and unix pipes suggestions won't work?
from subprocess import *
pipes = Popen("gzip | uuencode dummy", stdin=PIPE, stdout=PIPE, shell=True)
for i in range(1, 100):
pipes.stdin.write("some data")
pipes.stdin.close()
print pipes.stdout.read()
seems to work