The output of subprocess.check_output() looks like this at the moment:
CalledProcessError: Command '['foo', ...]' returned non-zero exit status 1
Is there a way to get a better error message?
I want to see stdout and stderr.
Redirect STDERR to STDOUT.
Example from the interpreter:
>>> try:
... subprocess.check_output(['ls','-j'], stderr=subprocess.STDOUT)
... except subprocess.CalledProcessError as e:
... print('error>', e.output, '<')
...
Will throw:
error> b"ls: invalid option -- 'j'\nTry `ls --help' for more information.\n" <
Explantion
From check_output documentation:
To also capture standard error in the result, use
stderr=subprocess.STDOUT
Don't use check_output(), use Popen and Popen.communicate() instead:
>>> proc = subprocess.Popen(['cmd', '--optional-switch'])
>>> output, errors = proc.communicate()
Here output is data from stdout and errors is data from stderr.
Since I don't want to write more code, just to get a good error message, I wrote subx
From the docs:
subprocess.check_output() vs subx.call()
Look, compare, think and decide what message helps your more.
subprocess.check_output()::
CalledProcessError: Command '['cat', 'some-file']' returned non-zero exit status 1
sub.call()::
SubprocessError: Command '['cat', 'some-file']' returned non-zero exit status 1:
stdout='' stderr='cat: some-file: No such file or directory'
... especially if the code fails in a production environment where
reproducing the error is not easy, subx can call help you to spot the
source of the failure.
In my opinion that a perfect scenario to use sys.excepthook! You just have to filter what you would like to be formatted as you want in the if statement. With this solution, it will cover every exception of your code without having to refract everything!
#!/usr/bin/env python
import sys
import subprocess
# Create the exception handler function
def my_excepthook(type, value, traceback):
# Check if the exception type name is CalledProcessError
if type.__name__ == "CalledProcessError":
# Format the error properly
sys.stderr.write("Error: " + type.__name__ + "\nCommand: " + value.cmd + "\nOutput: " + value.output.strip())
# Else we format the exception normally
else:
sys.stderr.write(str(value))
# We attach every exceptions to the function my_excepthook
sys.excepthook = my_excepthook
# We duplicate the exception
subprocess.check_output("dir /f",shell=True,stderr=subprocess.STDOUT)
You can modify the output as you wish, here is the actual ouput:
Error: CalledProcessError
Command: dir /f
Output: Invalid switch - "f".
Related
Given the following command:
newman run tests.postman_collection.json -e environment.json --reporters testrail,json,html
Raises:
RuntimeError: command 'newman run tests.postman_collection.json -e environment.json --reporters testrail,json,html
' return with error (code 1): b'\nhttps://host.testrail.io/index.php?/runs/view/1234\n'
Py code that executes the command:
try:
newmanCLI_output = subprocess.check_output(npmCLi, shell=True).decode().strip()
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
And yes I do use the check_output return.
The output is a url to test rail reports
That's a misfeature of os.system; it returns the exit code so you can examine it, but doesn't raise an error if something fails.
The check in subprocess.check_output means check that the command succeeded, or raise an exception otherwise. This is generally a good thing, as you don't want processes to die underneath you without a warning.
But you can work around it with subprocess.run if you want to disable it;
import shlex
result = subprocess.run(shlex.split(npmCLi), text=True, capture_output=True)
newmanCLI_output = result.stdout
The switch to avoid shell=True and use shlex.split to parse the string instead is not crucial, but hopefully demonstrates how to do these things properly.
You should still understand why exactly your command fails, and whether it is safe to ignore the failure.
I am making a program that adds additional functionality to the standard command shell in Windows. For instance, typing google followed by keywords will open a new tab with Google search for those keywords, etc. Whenever the input doesn't refer to a custom function I've created, it gets processed as a shell command using subprocess.call(rawCommand, shell=True).
Since I'd like to anticipate when my input isn't a valid command and return something like f"Invalid command: {rawCommand}", how should I go about doing that?
So far I've tried subprocess.call(rawCommand) which also return the standard output as well as the exit code. So that looks like this:
>>> from subprocess import call
>>> a, b = call("echo hello!", shell=1), call("xyz arg1 arg2", shell=1)
hello!
'xyz' is not recognized as an internal or external command,
operable program or batch file.
>>> a
0
>>> b
1
I'd like to simply recieve that exit code. Any ideas on how I can do this?
Should you one day want deal with encoding errors, get back the result of the command you're running, have a timeout or decide which exit codes other than 0 may not trigger errors (i'm looking at you, java runtime !), here's a complete function that does that job:
import os
from logging import getLogger
import subprocess
logger = getLogger()
def command_runner(command, valid_exit_codes=None, timeout=300, shell=False, encoding='utf-8',
windows_no_window=False, **kwargs):
"""
Whenever we can, we need to avoid shell=True in order to preseve better security
Runs system command, returns exit code and stdout/stderr output, and logs output on error
valid_exit_codes is a list of codes that don't trigger an error
windows_no_window will hide the command window (works with Microsoft Windows only)
Accepts subprocess.check_output arguments
"""
# Set default values for kwargs
errors = kwargs.pop('errors', 'backslashreplace') # Don't let encoding issues make you mad
universal_newlines = kwargs.pop('universal_newlines', False)
creationflags = kwargs.pop('creationflags', 0)
if windows_no_window:
creationflags = creationflags | subprocess.CREATE_NO_WINDOW
try:
# universal_newlines=True makes netstat command fail under windows
# timeout does not work under Python 2.7 with subprocess32 < 3.5
# decoder may be unicode_escape for dos commands or utf-8 for powershell
output = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=shell,
timeout=timeout, universal_newlines=universal_newlines, encoding=encoding,
errors=errors, creationflags=creationflags, **kwargs)
except subprocess.CalledProcessError as exc:
exit_code = exc.returncode
try:
output = exc.output
except Exception:
output = "command_runner: Could not obtain output from command."
if exit_code in valid_exit_codes if valid_exit_codes is not None else [0]:
logger.debug('Command [%s] returned with exit code [%s]. Command output was:' % (command, exit_code))
if isinstance(output, str):
logger.debug(output)
return exc.returncode, output
else:
logger.error('Command [%s] failed with exit code [%s]. Command output was:' %
(command, exc.returncode))
logger.error(output)
return exc.returncode, output
# OSError if not a valid executable
except (OSError, IOError) as exc:
logger.error('Command [%s] failed because of OS [%s].' % (command, exc))
return None, exc
except subprocess.TimeoutExpired:
logger.error('Timeout [%s seconds] expired for command [%s] execution.' % (timeout, command))
return None, 'Timeout of %s seconds expired.' % timeout
except Exception as exc:
logger.error('Command [%s] failed for unknown reasons [%s].' % (command, exc))
logger.debug('Error:', exc_info=True)
return None, exc
else:
logger.debug('Command [%s] returned with exit code [0]. Command output was:' % command)
if output:
logger.debug(output)
return 0, output
Usage:
exit_code, output = command_runner('whoami', shell=True)
Some shells have a syntax-checking mode (e.g., bash -n), but that’s the only form of error that’s separable from “try to execute the commands and see what happens”. Defining a larger class of “immediate” errors is a fraught proposition: if echo hello; ./foo is invalid because foo can’t be found as a command, what about false && ./foo, which will never try to run it, or cp /bin/ls foo; ./foo, which may succeed (or might fail to copy)? What about eval $(configure_shell); foo which might or might not manipulate PATH so as to find foo? What about foo || install_foo, where the failure might be anticipated?
As such, anticipating failure is not possible in any meaningful sense: your only real option is to capture the command’s output/error (as mentioned in the comments) and report them in some useful way.
I'm trying to run some commands using the python library subprocess. Some of my commands might get stuck in loops and block the python script. Therefore I'm using check_output() with a timeout argument to limit the commands in time. When the command takes too much time, the function raise a TimeoutExpired error. What I want to do is get what the command has been able to run before being killed by the timeout.
I've except the error and tried "except sp.TimeoutExpired as e:". I read on the doc that if I do e.output it should give me what I want. "Output of the child process if this exception is raised by check_output(). Otherwise, None.". However I don't get anything in the output.
Here is what I did:
import subprocess as sp
try:
out = sp.check_output('ls', stderr=sp.STDOUT, universal_newlines=True, timeout=1)
except sp.TimeoutExpired as e:
print ('output: '+ e.output)
else:
return out
Let say the folder I'm working with is huge and so 1 second isn't enough to ls all its files. Therefore the TimeoutExpired error will be raised. However, I'd like to store what the script was able to get at least. Does someone have an idea?
Found a solution, posting it here in case someone is interested.
In Python 3, the run method allows to get the output.
Using the parameters as shown in the example, TimeoutExpired returns the output before the timeout in stdout:
import subprocess as sp
for cmd in [['ls'], ['ls', '/does/not/exist'], ['sleep', '5']]:
print('Running', cmd)
try:
out = sp.run(cmd, timeout=3, check=True, stdout=sp.PIPE, stderr=sp.STDOUT)
except sp.CalledProcessError as e:
print(e.stdout.decode() + 'Returned error code ' + str(e.returncode))
except sp.TimeoutExpired as e:
print(e.stdout.decode() + 'Timed out')
else:
print(out.stdout.decode())
Possible output:
Running ['ls']
test.py
Running ['ls', '/does/not/exist']
ls: cannot access '/does/not/exist': No such file or directory
Returned error code 2
Running ['sleep', '5']
Timed out
I hope it helps someone.
I am using below code for getting output of shell command.
import subprocess
exitcode, err, out = 0, None, None
try:
out = subprocess.check_output(cmd, shell=True, universal_newlines=True)
except subprocess.CalledProcessError as e:
exitcode, err = e.returncode, e.output
print("x{} e{} o{}".format(exitcode, err, out))
When a valid command is being passed for cmd like echo hello, the program is running fine and giving output as (0, None, "hello\n")
But if I give a wrong kind of command I am expecting the error message should come in err, but its getting printed directly. For example if I pass ls -lrt foo in cmd the output is coming as
anirban#desktop> python mytest.py
ls: cannot access foo: No such file or directory
x2 e oNone
So I want ls: cannot access foo: No such file or directory should be coming in err. How to do that?
To capture the error output, you need to pass in another argument to the subprocess.check_output() function. You need to set stderr=subprocess.STDOUT. This will channel the stderr output to e.output.
subprocess.check_output() is a wrapper over subprocess.run(). It makes our lives easier by passing in some sensible defaults. One of those is making stdout=subprocess.PIPE. This directs the standard output of the command you are running back to your program. Similarly, you can direct the standard error output to your program by passing in argument, stderr=subprocess.PIPE, this will populate e.stderr where e is the exception.
Let me know if this is not clear.
I'm currently converting a shell script to python and I'm having a problem. The current script uses the results of the last ran command like so.
if [ $? -eq 0 ];
then
testPassed=$TRUE
else
testPassed=$FALSE
fi
I have the if statement converted over just not sure about the $? part. As I am new to python I'm wondering if there is a similar way to do this?
You should look into the subprocess module for that. There is a check_call method for looking into exit codes (this is one method, there are others as well). As the manual mentions:
Run command with arguments. Wait for command to complete. If the
return code was zero then return, otherwise raise CalledProcessError.
The CalledProcessError object will have the return code in the
returncode attribute
An example of this is:
import subprocess
command=["ls", "-l"]
try:
exit_code=subprocess.check_call(command)
# Do something for successful execution here
print("Program run")
except subprocess.CalledProcessError as e:
print "Program exited with exit code", e.returncode
# Do something for error here
This will also include output, which you can either redirect to a file or suppress like so:
import subprocess
import os
command=["ls", "-l"]
try:
exit_code=subprocess.check_call(command, stdout=open(os.devnull, "w"))
# Do something for successful execution here
print("Program run")
except subprocess.CalledProcessError as e:
print "Program exited with exit code", e.returncode
# Do something for error here
Here is an example of a call with a non-zero exit code:
import subprocess
import os
command=["grep", "mystring", "/home/cwgem/testdir/test.txt"]
try:
exit_code=subprocess.check_call(command, stdout=open(os.devnull, "w"))
# Do something for successful execution here
print("Program run")
except subprocess.CalledProcessError as e:
print "Program exited with exit code", e.returncode
# Do something for error here
Output:
$ python process_exitcode_test.py
Program exited with exit code 1
Which is captured as an exception that you can handle as above. Note that this will not handle exceptions such as access denied or file not found. You will need to handle them on your own.
You might want use the sh module. It makes shell scripting in Python much more pleasant:
import sh
try:
output = sh.ls('/some/nen-existant/folder')
testPassed = True
except ErrorReturnCode:
testPassed = False