I would like to get the output (print statements) and return status from a python subprocess along with displaying logging info. I am using Python 2.5 on Windows XP. How do I get the return code as well? Here in this extract, I am reading output so that I can redirect it to display in a wxPython TextCtrl in a separate thread.
self.myproc.poll()
if self.myproc.returncode is None:
# Still running so check stdout
txt = self.myproc.stdout.read(self._readblock)
if txt:
# Add to UI's update queue
self._parent.AppendUpdate(txt)
else:
break
The returncode gets returned by the poll call, and is also (once poll did not return None) accessible via the returncode attribute. You are already using this in your code, so I'm not sure what your problem with that is.
When you want to update your control while your process is running, use readline instead of read: the latter will wait for the entire output to be present, while the former will wait on a newline character. To give a full example using your variable names:
from subprocess import Popen, PIPE
self.myproc = Popen('dir /s', shell=True, stdout=PIPE, stderr=PIPE)
while self.myproc.poll() == None:
self._parent.AppendUpdate(self.myproc.stdout.readline())
self._parent.AppendUpdate('Return code was ' + self.myproc.returncode)
Note that you might want to .strip() the readline result as it will contain the newline character.
Edit: To address your confusion between the script output and its return code, the following. First, a Python script at module level cannot return a value: that would yield a syntax error. So what you should differentiate between is the output of the script, and the return code it gives.
The script's output is in the above example read by using the self.myproc.stdout.readline method. Every time the external process has produced a line of outputted text, calling that function would retrieve it.
The return code however (or exit status) is an integer that gets passed from a child process to the parent (your) process, indicating the state in which the child process exited. In Python, you do this with the sys.exit function. Most commonly, when this value is zero it indicates success; a non-zero value depicts some kind of error.
Say your child process script looks like this:
import sys
# Do some stuff
print 'pass' # Gets to stdout
sys.exit(0) # Return code
Executing this external file (let's call it test.py) with the Popen class, we will get pass when we read out self.myproc.stdout, and 0 when we read out self.myproc.poll (or self.myproc.returncode after the first poll).
The purpose of this return code is that you do not have to parse all of the child process' output to determine if it succeeded in its job: you are free to define your own exit codes. For example, you could consider 0 to be a success, 1 to be a failure, 2 to be some kind of invalid input given, 9 an unknown error, and so on. That way, you can just keep polling the process, and based on the exit code returned by that poll you directly know if it was successful. Note that this is a bit less applicable to your case as you need the output of the child process anyway, but still it is easier to parse a number than a string to determine success.
Related
I am trying to use output of external program run using the run function.
this program regularly throws a row of data which i need to use in mine script
I have found a subprocess library and used its run()/check_output()
Example:
def usual_process():
# some code here
for i in subprocess.check_output(['foo','$$']):
some_function(i)
Now assuming that foo is already in a PATH variable and it outputs a string in semi-random periods.
I want the program to do its own things, and run some_function(i)every time foo sends new row to its output.
which boiles to two problems. piping the output into a for loop and running this as a background subprocess
Thank you
Update: I have managed to get the foo output onto some_function using This
with os.popen('foo') as foos_output:
for line in foos_output:
some_function(line)
According to this os.popen is to be deprecated, but I am yet to figure out how to pipe internal processes in python
Now just need to figure out how to run this function in a background
SO, I have solved it.
First step was to start the external script:
proc=Popen('./cisla.sh', stdout=PIPE, bufsize=1)
Next I have started a function that would read it and passed it a pipe
def foo(proc, **args):
for i in proc.stdout:
'''Do all I want to do with each'''
foo(proc).start()`
Limitations are:
If your wish t catch scripts error you would have to pipe it in.
second is that it leaves a zombie if you kill parrent SO dont forget to kill child in signal-handling
I have a Fortran code that takes a file as input and writes the output into stdout. To avoid read/write cycles, I'd like to run the code inside python and convert the output to a numpy array. I can do this using the following function:
def run_fortran(infile):
import subprocess
import numpy as np
cmd = ['./output.e', infile]
p = subprocess.Popen(cmd, stdout = subprocess.PIPE)
out, err = p.communicate()
if p.returncode == 0:
return np.array(out.split(),dtype=int)
Now I take the array, modify it and write it into a file. The new file is again passed into run_fortran(infile). Can I avoid this step and somehow use the output of run_fortran instead of passing a filename?
I tried two cases with no success:
(1) converting to string:
arr = run_fortran('input.txt')
new_arr = str(arr).replace("[","").replace("]","")
run_fortran(new_arr)
This returns an empty array.
(2) converting to a file type object using StringIO:
from cStringIO import StringIO
run_fortran(StringIO(new_arr))
This returns an error: TypeError: execv() arg 2 must contain only strings which makes sense.
In fortran the read(*,*), read(5,*) or read*, statement will read from from standard input, and if that has the right format, it will work. If you are dealing with formatted data, that is, anything that is human readable, and not a binary file, then you probably need a read loop like:
do line=1,numlines
read(*,*) array(line,:)
enddo
No open or close statements needed. So if what you were writing to a file is passed directly, you should be able to remove the those statements and change the file unit to 5 or *.
Now there are more efficient ways to do this kind of communication, but any solution is a good solution if it suit your purpose.
If your Fortran program (it's that './output.e' AFAICS) can read from stdin, not only from a regular file, you can do without temporary files by passing it stdin=subprocess.PIPE (other valid values are "an existing file descriptor (a positive integer) [or] an existing file object"). In UNIX, there's always /dev/stdin to put on the command line btw and in Windows, there's con.
Still, if the program can only work in "sessions" due to the nature of processing (i.e. cannot run continuously and be fed new data as it's available), you'll have to invoke it repeatedly.
Note that you have to handle different pipes in different threads to avoid deadlocks. So, either use communicate() (but then the program cannot run continuously) or spawn an stdout/stderr thread manually (that's what communicate() does; not stdin because the output reading code has to be running by the time you start writing to stdin or the external program may choke with "no space left on device" while writing).
Here's sample code for running the program continuously:
p=subprocess.Popen(argv,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
while True:
#the only more elegant way to return a value from a thread is with a Thread subclass,
# see http://stackoverflow.com/questions/6893968/how-to-get-the-return-value-from-a-thread-in-python
output_container=[]
ot=threading.Thread(target=_outputreaderthread,args=(p.stdout,output_container,data))
ot.start()
p.stdin.write(data)
ot.join()
output=output_container[0]
data=process_output(output)
if no_more_processing_needed(data): break
p.stdin.close()
if p.wait()!=0:raise subprocess.CalledProcessError(p.returncode,argv)
def _outputreaderthread(stream,container,data):
#since there's no process termination as an end mark here,
# you need to read exactly the right number of bytes to avoid getting blocked.
#use whatever technique is appropriate for your data (e.g. readline()'s)
output=stream.read(calculate_output_size(data))
container.append(output)
I am programming in python which involves me implementing a shell in Python in Linux. I am trying to run standard unix commands by using os.execvp(). I need to keep asking the user for commands so I have used an infinite while loop. However, the infinite while loop doesn't work. I have tried searching online but they're isn't much available for Python. Any help would be appreciated. Thanks
This is the code I have written so far:
import os
import shlex
def word_list(line):
"""Break the line into shell words."""
lexer = shlex.shlex(line, posix=True)
lexer.whitespace_split = False
lexer.wordchars += '#$+-,./?#^='
args = list(lexer)
return args
def main():
while(True):
line = input('psh>')
split_line = word_list(line)
if len(split_line) == 1:
print(os.execvp(split_line[0],[" "]))
else:
print(os.execvp(split_line[0],split_line))
if __name__ == "__main__":
main()
So when I run this and put in the input "ls" I get the output "HelloWorld.py" (which is correct) and "Process finished with exit code 0". However I don't get the output "psh>" which is waiting for the next command. No exceptions are thrown when I run this code.
Your code does not work because it uses os.execvp. os.execvp replaces the current process image completely with the executing program, your running process becomes the ls.
To execute a subprocess use the aptly named subprocess module.
In case of an ill-advised programming exercise then you need to:
# warning, never do this at home!
pid = os.fork()
if not pid:
os.execvp(cmdline) # in child
else:
os.wait(pid) # in parent
os.fork returns twice, giving the pid of child in parent process, zero in child process.
If you want it to run like a shell you are looking for os.fork() . Call this before you call os.execvp() and it will create a child process. os.fork() returns the process id. If it is 0 then you are in the child process and can call os.execvp(), otherwise continue with the code. This will keep the while loop running. You can have the original process either wait for it to complete os.wait(), or continue without waiting to the start of the while loop. The pseudo code on page 2 of this link should help https://www.cs.auckland.ac.nz/courses/compsci340s2c/assignments/A1/A1.pdf
I have a ruby script that gets executed by a python script. From within the python script I want to access to return value of the ruby function.
Imagine, I would have this ruby script test.rb:
class TestClass
def self.test_function(some_var)
if case1
puts "This may take some time"
# something is done here with some_var
puts "Finished"
else
# just do something short with some_var
end
return some_var
end
end
Now, I want to get the return value of that function into my python script, the printed output should go to stdout.
I tried the following (example 1):
from subprocess import call
answer = call(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"])
However, this gives me the whole output on stdout and answer is just the exit code.
Therefore i tried this (example 2):
from subprocess import check_output
answer = check_output(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"])
This gives me the return value of the function in the else case (see test.rb) almost immediately. However, if case1 is true, answer contains the whole output, but while running test.rb nothing gets printed.
Is there any way to get the return value of the ruby function and the statements printed to stdout? Ideally, the solution requires no additional modules to install. Furthermore, I can't change the ruby code.
Edit:
Also tried this, but this also gives no output on stdout while running the ruby script (example 3):
import subprocess
process = subprocess.Popen(["ruby", "-r", "test.rb", "-e", "puts TestClass.test_function('some meaningful text')"], stdout=subprocess.PIPE)
answer = process.communicate()
I also think that this is no matter of flushing the output to stdout in the ruby script. Example 1 gives me the output immediately.
Another way of doing this, without trying to call the ruby script as an external process is to set up a xmlrpc (or jsonrpc) server with the Ruby script, and call the remote functions from Python jsonrpc client (or xmlrpc)- the value would be available inside the Python program, nad even the sntax used would be just like you were dealing with a normal Python function.
Setting up such a server to expose a couple of functions remotely is very easy in Python, and should be the same from Ruby, but I had never tried it.
Check out http://docs.python.org/library/subprocess.html#popen-constructor and look into the ruby means of flushing stdout.
Again, the same question.
The reason is - I still can't make it work after reading the following:
Real-time intercepting of stdout from another process in Python
Intercepting stdout of a subprocess while it is running
How do I get 'real-time' information back from a subprocess.Popen in python (2.5)
catching stdout in realtime from subprocess
My case is that I have a console app written in C, lets take for example this code in a loop:
tmp = 0.0;
printf("\ninput>>");
scanf_s("%f",&tmp);
printf ("\ninput was: %f",tmp);
It continuously reads some input and writes some output.
My python code to interact with it is the following:
p=subprocess.Popen([path],stdout=subprocess.PIPE,stdin=subprocess.PIPE)
p.stdin.write('12345\n')
for line in p.stdout:
print(">>> " + str(line.rstrip()))
p.stdout.flush()
So far whenever I read form p.stdout it always waits until the process is terminated and then outputs an empty string. I've tried lots of stuff - but still the same result.
I tried Python 2.6 and 3.1, but the version doesn't matter - I just need to make it work somewhere.
Trying to write to and read from pipes to a sub-process is tricky because of the default buffering going on in both directions. It's extremely easy to get a deadlock where one or the other process (parent or child) is reading from an empty buffer, writing into a full buffer or doing a blocking read on a buffer that's awaiting data before the system libraries flush it.
For more modest amounts of data the Popen.communicate() method might be sufficient. However, for data that exceeds its buffering you'd probably get stalled processes (similar to what you're already seeing?)
You might want to look for details on using the fcntl module and making one or the other (or both) of your file descriptors non-blocking. In that case, of course, you'll have to wrap all reads and/or writes to those file descriptors in the appropriate exception handling to handle the "EWOULDBLOCK" events. (I don't remember the exact Python exception that's raised for these).
A completely different approach would be for your parent to use the select module and os.fork() ... and for the child process to execve() the target program after directly handling any file dup()ing. (Basically you'd be re-implement parts of Popen() but with different parent file descriptor (PIPE) handling.
Incidentally, .communicate, at least in Python's 2.5 and 2.6 standard libraries, will only handle about 64K of remote data (on Linux and FreeBSD). This number may vary based on various factors (possibly including the build options used to compile your Python interpreter, or the version of libc being linked to it). It is NOT simply limited by available memory (despite J.F. Sebastian's assertion to the contrary) but is limited to a much smaller value.
Push reading from the pipe into a separate thread that signals when a chunk of output is available:
How can I read all availably data from subprocess.Popen.stdout (non blocking)?
The bufsize=256 argument prevents 12345\n from being sent to the child process in a chunk smaller than 256 bytes, as it will be when omitting bufsize or inserting p.stdin.flush() after p.stdin.write(). Default behaviour is line-buffering.
In either case you should at least see one empty line before blocking as emitted by the first printf(\n...) in your example.
Your particular example doesn't require "real-time" interaction. The following works:
from subprocess import Popen, PIPE
p = Popen(["./a.out"], stdin=PIPE, stdout=PIPE)
output = p.communicate(b"12345")[0] # send input/read all output
print output,
where a.out is your example C program.
In general, for a dialog-based interaction with a subprocess you could use pexpect module (or its analogs on Windows):
import pexpect
child = pexpect.spawn("./a.out")
child.expect("input>>")
child.sendline("12345.67890") # send a number
child.expect(r"\d+\.\d+") # expect the number at the end
print float(child.after) # assert that we can parse it
child.close()
I had the same problem, and "proc.communicate()" does not solve it because it waits for process terminating.
So here is what is working for me, on Windows with Python 3.5.1 :
import subprocess as sp
myProcess = sp.Popen( cmd, creationflags=sp.CREATE_NEW_PROCESS_GROUP,stdout=sp.PIPE,stderr=sp.STDOUT)
while i<40:
i+=1
time.sleep(.5)
out = myProcess.stdout.readline().decode("utf-8").rstrip()
I guess creationflags and other arguments are not mandatory (but I don't have time to test), so this would be the minimal syntax :
myProcess = sp.Popen( cmd, stdout=sp.PIPE)
for i in range(40)
time.sleep(.5)
out = myProcess.stdout.readline()