Develop Reference

python output system calls to terminal and logfile

I would like to execute system calls from my (python) script such that the output of the sys-call is generated on the terminal where script is running as well as is captured in a logfile. However I am not able to make this work for interactive system calls.
I started with following code that does not capture the system call in a logfile, but works correctly (as in is able to display output on terminal, and can take inputs I type in terminal) for both basic commands such as system('echo HEYA') as well as interactive commands such as system('python') :
def system(cmd):
log.info(f"Running: {cmd}")
try:
subprocess.run(cmd, check=True, shell=True, executable=SHELL)
except subprocess.CalledProcessError as err:
log.error(err.output)
Notes: log is a logger (created using standard logging module). SHELL variable holds a shell of my choice.
Now, I modify the above to be able to redirect the process output to terminal as well as logfile on disk in realtime with following code:
def system(cmd):
log.info(f"Running: {cmd}")
try:
process = subprocess.Popen(cmd,
shell=True, executable=SHELL, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
o = process.stdout.readline().rstrip('\n')
if o == '' and process.poll() is not None:
break
if o:
syslog.info(o)
ret = process.poll()
if ret:
log.error(f"Failed with exit code: {ret}")
else:
log.info("Done")
except:
err = sys.exc_info()[0]
log.error(err)
raise
Notice that I use a different logger (syslog) to redirect outputs to logfile and terminal. The only reason for this, is I want system command outputs formatted differently from other messages generated in the script.
The second version of system function works for something like system('echo HEYA') but not for interactive calls like system('python'). Any suggestions what I may be doing wrong, and how I can get this to work?

Based on an earlier post of similar nature:
https://stackoverflow.com/a/651718/5150258
I was able to get this work partially by using first form of system definition with a tee:
tee = subprocess.Popen(["tee", "sys_cmds.log"], stdin=subprocess.PIPE)
os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
os.dup2(tee.stdin.fileno(), sys.stderr.fileno()
def system(cmd):
log.info(f"Running: {cmd}")
try:
subprocess.run(cmd, check=True, shell=True, executable=SHELL)
except subprocess.CalledProcessError as err:
log.error(err.output)
However this isn't the perfect solution since I lose formatting control of the output as it goes directly without passing through the logger object.

Python Pandoc Subprocess Not Printing STDOUT When Exception Occurs

I'm trying to run a process with subprocess and print its entire output if and only if an exception occurs.
Where I was before:
try:
proc = subprocess.run(
command,
capture_output=True,
check=True,
text=True,
)
except subprocess.CalledProcessError as error:
print(error.output)
This did not work.
Output when subprocess.CalledProcessError occurs:
b''
Replacing capture_output with stdout=subprocess.PIPE resulted in the output of everything regardless whether an exception occurred or not, error.output was still empty.
So I experimented:
This prints everything I would see if I executed the command in the command-line.
subprocess.run(
command,
stdout=subprocess.PIPE,
)
This prints out nothing.
proc = subprocess.run(
command,
capture_output=True,
)
print(proc.stdout.decode())
I also tried subprocess.check_output() which to my information does the same as subprocess.run() with the flags I set in the first code snippet.
What am I missing here? Thanks.
Addendum
import subprocess
command = ['pandoc', 'file']
try:
proc = subprocess.run(
command,
capture_output=True,
check=True,
)
except subprocess.CalledProcessError as error:
print('Exception:')
print(error.output)
This is an MWE with the specific process I want to run (pandoc)
Output
$ pandoc file
pandoc: file: openBinaryFile: does not exist (No such file or directory)
$ ./samplecode.py
Exception:
b''
So the exception gets triggered, but the output object is empty.

It seems that the error message is present in error.stderr and not in error.output. I tried your example (with a ls of non-existent file) :
import subprocess
command = ['ls', 'file']
try:
proc = subprocess.run(
command,
check=True,
capture_output=True,
text=True
)
except subprocess.CalledProcessError as error:
print('Exception:')
print('output : ' + error.output)
print('stderr : ' + error.stderr)
The output is the following :
Exception:
output :
stderr : ls: file: No such file or directory
Hope it helps.

I believe what you're meaning to run is stderr=subprocess.PIPE. This should print the relevant error code to the standard console error output.
Example:
process = subprocess.Popen(['ls', 'myfile.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output,error) = process.communicate()
if error:
print error

subprocess.CalledProcessError not giving error message

I am using below code for getting output of shell command.
import subprocess
exitcode, err, out = 0, None, None
try:
out = subprocess.check_output(cmd, shell=True, universal_newlines=True)
except subprocess.CalledProcessError as e:
exitcode, err = e.returncode, e.output
print("x{} e{} o{}".format(exitcode, err, out))
When a valid command is being passed for cmd like echo hello, the program is running fine and giving output as (0, None, "hello\n")
But if I give a wrong kind of command I am expecting the error message should come in err, but its getting printed directly. For example if I pass ls -lrt foo in cmd the output is coming as
anirban#desktop> python mytest.py
ls: cannot access foo: No such file or directory
x2 e oNone
So I want ls: cannot access foo: No such file or directory should be coming in err. How to do that?

To capture the error output, you need to pass in another argument to the subprocess.check_output() function. You need to set stderr=subprocess.STDOUT. This will channel the stderr output to e.output.
subprocess.check_output() is a wrapper over subprocess.run(). It makes our lives easier by passing in some sensible defaults. One of those is making stdout=subprocess.PIPE. This directs the standard output of the command you are running back to your program. Similarly, you can direct the standard error output to your program by passing in argument, stderr=subprocess.PIPE, this will populate e.stderr where e is the exception.
Let me know if this is not clear.

subprocess.check_output failing with error 127

I'm attempting to call an outside program from my python application, but it shows no output and fails with error 127. Executing the command from the command line works fine. (and I am in the correct working directory)
def buildContris (self, startUrl, reportArray):
urls = []
for row in reportArray:
try:
url = subprocess.check_output(["casperjs", "casper.js", startUrl, row[0]], shell=True)
print (url)
urls.append(url)
break
except subprocess.CalledProcessError as e:
print ("Error: " + str(e.returncode) + " Output:" + e.output.decode())
return urls
Each loop outputs the following error: (I've also checked e.cmd. It's correct, but long, so I omitted it in this example)
Error: 127 Output:
SOLUTION:
The following code works
app = subprocess.Popen(["./casperjs/bin/casperjs", "casper.js", startUrl, row[0]], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env = {"PATH" : "/usr/local/bin/:/usr/bin"}, universal_newlines=True)
app.wait()
out, errs = app.communicate()

Try adding the full path to casperjs in your subprocess.check_output() call.
Edit: Answeing your 2nd question. My apologies for the formatting as I'm on iPad.
I think you should try Popen instead of check_output so that you can specify environment variables:
p = subprocess.Popen(["/path/to/casperjs", "casper.js", startUrl, row[0]], env={"PATH": "/path/to/phantomjs"})
url, err = p.communicate()

shell=True changes the interpretation of the first argument (args) in check_output() call, from the docs:
On Unix with shell=True, ... If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
Exit status 127 might mean that the shell haven't found casperjs program or casperjs itself exited with that code.
To fix the code: drop shell=True and specify the full path to the casperjs program e.g.:
url = check_output(["./casperjs", "casper.js", startUrl, row[0]])

Try to add explicitly the path in this way.
If the file to call is in the same path (change __file__ if not):
cwd=os.path.dirname(os.path.realpath(__file__))
a = subprocess.check_output(["./casper.js", startUrl, row[0]],cwd=cwd,shell=True)

If you're experiencing this kinda nonsense on macOS: don't use aliases. Lost half a day with that. So, change:
subprocess.check_output(
"scribus-ng -g -ns -py {0} {1}".format(script_path, id),
stderr=subprocess.STDOUT,
shell=True)
to
subprocess.check_output(
"/Applications/Scribus.app/Contents/MacOS/Scribus -g -ns -py {0} {1}".format(script_path, id),
stderr=subprocess.STDOUT,
shell=True)

How to get the return status in python

I am calling the executable from python script using sub process call. these are the following code I have used:
try:
p = subprocess.Popen([abc.exe], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
except Exception as e:
print str(e)
from abc.exe, I have return 1 in failure case and return 0 for success case. But I don't know how to check the return value in python script.
thanks,

Popen.returncode contains the return code when the process has terminated. You can ensure that using Popen.wait.

You've saved as p the output from .communicate(), not Popen object. Perhaps try:
try:
p = subprocess.Popen(['abc.exe'], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except OSError as e:
print str(e)
stdoutdata, stderrdata = p.communicate()
retcode = p.returncode

Another way to do this is to use subprocess.check_output() since you mention Python 2.7. This runs the command with the same arguments as Popen. The output of the command is returned as a string. If the command returns a non-zero value, a subprocess.CalledProcessError exception is raised.
So I think you can rework your code to something like this:
try:
output = subprocess.check_output(['abc.exe'], shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as ex:
# an error occurred
retcode = ex.returncode
output = ex.output
else:
# no error occurred
process(output)
Note that you can't use the stdout argument in check_output since it is used internally. Here are the docs.