I'm trying to run a process with subprocess and print its entire output if and only if an exception occurs.
Where I was before:
try:
proc = subprocess.run(
command,
capture_output=True,
check=True,
text=True,
)
except subprocess.CalledProcessError as error:
print(error.output)
This did not work.
Output when subprocess.CalledProcessError occurs:
b''
Replacing capture_output with stdout=subprocess.PIPE resulted in the output of everything regardless whether an exception occurred or not, error.output was still empty.
So I experimented:
This prints everything I would see if I executed the command in the command-line.
subprocess.run(
command,
stdout=subprocess.PIPE,
)
This prints out nothing.
proc = subprocess.run(
command,
capture_output=True,
)
print(proc.stdout.decode())
I also tried subprocess.check_output() which to my information does the same as subprocess.run() with the flags I set in the first code snippet.
What am I missing here? Thanks.
Addendum
import subprocess
command = ['pandoc', 'file']
try:
proc = subprocess.run(
command,
capture_output=True,
check=True,
)
except subprocess.CalledProcessError as error:
print('Exception:')
print(error.output)
This is an MWE with the specific process I want to run (pandoc)
Output
$ pandoc file
pandoc: file: openBinaryFile: does not exist (No such file or directory)
$ ./samplecode.py
Exception:
b''
So the exception gets triggered, but the output object is empty.
It seems that the error message is present in error.stderr and not in error.output. I tried your example (with a ls of non-existent file) :
import subprocess
command = ['ls', 'file']
try:
proc = subprocess.run(
command,
check=True,
capture_output=True,
text=True
)
except subprocess.CalledProcessError as error:
print('Exception:')
print('output : ' + error.output)
print('stderr : ' + error.stderr)
The output is the following :
Exception:
output :
stderr : ls: file: No such file or directory
Hope it helps.
I believe what you're meaning to run is stderr=subprocess.PIPE. This should print the relevant error code to the standard console error output.
Example:
process = subprocess.Popen(['ls', 'myfile.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output,error) = process.communicate()
if error:
print error
I'm new to Python and Programming as well. I know from Google's python class how to run external command using:
(status, output) = commands.getstatusoutput(cmd)
if status: ## Error case, print the command's output to stderr and exit
sys.stderr.write(output)
sys.exit(1)
But I perceive that commands module is going obsolete. I want the status and the output, so I can print the output using sys.stderr.write() if there is any error. So, is there any equivalent command in subprocess module? I'm currently using:
subprocess.call(args,shell=False) now.
Thanks!
There is subprocess.getstatusoutput() in Python 3 that could be implemented as:
from subprocess import check_output, CalledProcessError, STDOUT
def getstatusoutput(cmd):
try:
data = check_output(cmd, shell=True, universal_newlines=True, stderr=STDOUT)
status = 0
except CalledProcessError as ex:
data = ex.output
status = ex.returncode
if data[-1:] == '\n':
data = data[:-1]
return status, data
Both return status that is different from the original commands.getstatusoutput(). See Python Issue: Document & unittest the subprocess.getstatusoutput() status value.
It's also possible to do this, with a bit less code
from subprocess import Popen, PIPE
def getstatusoutput(command):
process = Popen(command, stdout=PIPE)
out, _ = process.communicate()
return (process.returncode, out)
code, out = getstatusoutput(["echo", "some text"])
print code
print out
Popen is a nice and easy way to do it:
from subprocess import Popen, PIPE
def local(command):
print 'local', local
process = Popen(command.split(), stdout=PIPE, stderr=PIPE)
out = process.stdout.read()
err = process.stderr.read()
print 'out', out
print 'err', err
returncode = process.wait()
if returncode:
raise Exception(returncode, err)
else:
return out
What getstatusoutput does is gather both stdout and stderr output interleaved in one variable. This will quite closely replicate the actual behaviour of getstatusoutput on those where it does not exist (getstatusoutput and the whole commands module was removed on Python 3 completely), excepting the newline behaviour. The resulting data is in bytes.
def getstatusoutput(cmd):
subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, _ = process.communicate()
if out[-1:] == b'\n':
out = out[:-1]
return (process.returncode, out)
This function returns bytes on Python 3 on purpose, as the Python 2 version returns str
I am calling the executable from python script using sub process call. these are the following code I have used:
try:
p = subprocess.Popen([abc.exe], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
except Exception as e:
print str(e)
from abc.exe, I have return 1 in failure case and return 0 for success case. But I don't know how to check the return value in python script.
thanks,
Popen.returncode contains the return code when the process has terminated. You can ensure that using Popen.wait.
You've saved as p the output from .communicate(), not Popen object. Perhaps try:
try:
p = subprocess.Popen(['abc.exe'], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except OSError as e:
print str(e)
stdoutdata, stderrdata = p.communicate()
retcode = p.returncode
Another way to do this is to use subprocess.check_output() since you mention Python 2.7. This runs the command with the same arguments as Popen. The output of the command is returned as a string. If the command returns a non-zero value, a subprocess.CalledProcessError exception is raised.
So I think you can rework your code to something like this:
try:
output = subprocess.check_output(['abc.exe'], shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as ex:
# an error occurred
retcode = ex.returncode
output = ex.output
else:
# no error occurred
process(output)
Note that you can't use the stdout argument in check_output since it is used internally. Here are the docs.
The problem I'm having is with Eclipse/PyCharm interpreting the results of subprocess's Popen() differently from a standard terminal. All are using python2.6.1 on OSX.
Here's a simple example script:
import subprocess
args = ["/usr/bin/which", "git"]
print "Will execute %s" % " ".join(args)
try:
p = subprocess.Popen(["/usr/bin/which", "git"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# tuple of StdOut, StdErr is the responses, so ..
ret = p.communicate()
if ret[0] == '' and ret[1] <> '':
msg = "cmd %s failed: %s" % (fullcmd, ret[1])
if fail_on_error:
raise NameError(msg)
except OSError, e:
print >>sys.stderr, "Execution failed:", e
With a standard terminal, the line:
ret = p.communicate()
gives me:
(Pdb) print ret
('/usr/local/bin/git\n', '')
Eclipse and PyCharm give me an empty tuple:
ret = {tuple} ('','')
Changing the shell= value does not solve the problem either. On the terminal, setting shell=True, and passing the command in altogether (i.e., args=["/usr/bin/which git"]) gives me the same result: ret = ('/usr/local/bin/git\n', ''). And Eclipse/PyCharm both give me an empty tuple.
Any ideas on what I could be doing wrong?
Ok, found the problem, and it's an important thing to keep in mind when using an IDE in a Unix-type environment. IDE's operate under a different environment context than the terminal user (duh, right?!). I was not considering that the subprocess was using a different environment than the context that I have for my terminal (my terminal has bash_profile set to have more things in PATH).
This is easily verified by changing the script as follows:
import subprocess
args = ["/usr/bin/which", "git"]
print "Current path is %s" % os.path.expandvars("$PATH")
try:
p = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# tuple of StdOut, StdErr is the responses, so ..
out, err = p.communicate()
if err:
msg = "cmd %s failed: %s" % (fullcmd, err)
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Under the terminal, the path includes /usr/local/bin. Under the IDE it does not!
This is an important gotcha for me - always remember about environments!
This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 1 year ago.
I'm trying to make a system call in Python and store the output to a string that I can manipulate in the Python program.
#!/usr/bin/python
import subprocess
p2 = subprocess.Popen("ntpq -p")
I've tried a few things including some of the suggestions here:
Retrieving the output of subprocess.call()
but without any luck.
In Python 2.7 or Python 3
Instead of making a Popen object directly, you can use the subprocess.check_output() function to store output of a command in a string:
from subprocess import check_output
out = check_output(["ntpq", "-p"])
In Python 2.4-2.6
Use the communicate method.
import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()
out is what you want.
Important note about the other answers
Note how I passed in the command. The "ntpq -p" example brings up another matter. Since Popen does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"].
This worked for me for redirecting stdout (stderr can be handled similarly):
from subprocess import Popen, PIPE
pipe = Popen(path, stdout=PIPE)
text = pipe.communicate()[0]
If it doesn't work for you, please specify exactly the problem you're having.
Python 2: http://docs.python.org/2/library/subprocess.html#subprocess.Popen
from subprocess import PIPE, Popen
command = "ntpq -p"
process = Popen(command, stdout=PIPE, stderr=None, shell=True)
output = process.communicate()[0]
print output
In the Popen constructor, if shell is True, you should pass the command as a string rather than as a sequence. Otherwise, just split the command into a list:
command = ["ntpq", "-p"]
process = Popen(command, stdout=PIPE, stderr=None)
If you need to read also the standard error, into the Popen initialization, you should set stderr to PIPE or STDOUT:
command = "ntpq -p"
process = subprocess.Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
output, error = process.communicate()
NOTE: Starting from Python 2.7, you could/should take advantage of subprocess.check_output (https://docs.python.org/2/library/subprocess.html#subprocess.check_output).
Python 3: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
from subprocess import PIPE, Popen
command = "ntpq -p"
with Popen(command, stdout=PIPE, stderr=None, shell=True) as process:
output = process.communicate()[0].decode("utf-8")
print(output)
NOTE: If you're targeting only versions of Python higher or equal than 3.5, then you could/should take advantage of subprocess.run (https://docs.python.org/3/library/subprocess.html#subprocess.run).
In Python 3.7+ you can use the new capture_output= keyword argument for subprocess.run:
import subprocess
p = subprocess.run(["echo", "hello world!"], capture_output=True, text=True)
assert p.stdout == 'hello world!\n'
Assuming that pwd is just an example, this is how you can do it:
import subprocess
p = subprocess.Popen("pwd", stdout=subprocess.PIPE)
result = p.communicate()[0]
print result
See the subprocess documentation for another example and more information.
for Python 2.7+ the idiomatic answer is to use subprocess.check_output()
You should also note the handling of arguments when invoking a subprocess, as it can be a little confusing....
If args is just single command with no args of its own (or you have shell=True set), it can be a string. Otherwise it must be a list.
for example... to invoke the ls command, this is fine:
from subprocess import check_call
check_call('ls')
so is this:
from subprocess import check_call
check_call(['ls',])
however, if you want to pass some args to the shell command, you can't do this:
from subprocess import check_call
check_call('ls -al')
instead, you must pass it as a list:
from subprocess import check_call
check_call(['ls', '-al'])
the shlex.split() function can sometimes be useful to split a string into shell-like syntax before creating a subprocesses...
like this:
from subprocess import check_call
import shlex
check_call(shlex.split('ls -al'))
This works perfectly for me:
import subprocess
try:
#prints results and merges stdout and std
result = subprocess.check_output("echo %USERNAME%", stderr=subprocess.STDOUT, shell=True)
print result
#causes error and merges stdout and stderr
result = subprocess.check_output("copy testfds", stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError, ex: # error code <> 0
print "--------error------"
print ex.cmd
print ex.message
print ex.returncode
print ex.output # contains stdout and stderr together
This was perfect for me.
You will get the return code, stdout and stderr in a tuple.
from subprocess import Popen, PIPE
def console(cmd):
p = Popen(cmd, shell=True, stdout=PIPE)
out, err = p.communicate()
return (p.returncode, out, err)
For Example:
result = console('ls -l')
print 'returncode: %s' % result[0]
print 'output: %s' % result[1]
print 'error: %s' % result[2]
The accepted answer is still good, just a few remarks on newer features. Since python 3.6, you can handle encoding directly in check_output, see documentation. This returns a string object now:
import subprocess
out = subprocess.check_output(["ls", "-l"], encoding="utf-8")
In python 3.7, a parameter capture_output was added to subprocess.run(), which does some of the Popen/PIPE handling for us, see the python docs :
import subprocess
p2 = subprocess.run(["ls", "-l"], capture_output=True, encoding="utf-8")
p2.stdout
I wrote a little function based on the other answers here:
def pexec(*args):
return subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0].rstrip()
Usage:
changeset = pexec('hg','id','--id')
branch = pexec('hg','id','--branch')
revnum = pexec('hg','id','--num')
print('%s : %s (%s)' % (revnum, changeset, branch))
import os
list = os.popen('pwd').read()
In this case you will only have one element in the list.
import subprocess
output = str(subprocess.Popen("ntpq -p",shell = True,stdout = subprocess.PIPE,
stderr = subprocess.STDOUT).communicate()[0])
This is one line solution
The following captures stdout and stderr of the process in a single variable. It is Python 2 and 3 compatible:
from subprocess import check_output, CalledProcessError, STDOUT
command = ["ls", "-l"]
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
If your command is a string rather than an array, prefix this with:
import shlex
command = shlex.split(command)
Use check_output method of subprocess module
import subprocess
address = '192.168.x.x'
res = subprocess.check_output(['ping', address, '-c', '3'])
Finally parse the string
for line in res.splitlines():
Hope it helps, happy coding
For python 3.5 I put up function based on previous answer. Log may be removed, thought it's nice to have
import shlex
from subprocess import check_output, CalledProcessError, STDOUT
def cmdline(command):
log("cmdline:{}".format(command))
cmdArr = shlex.split(command)
try:
output = check_output(cmdArr, stderr=STDOUT).decode()
log("Success:{}".format(output))
except (CalledProcessError) as e:
output = e.output.decode()
log("Fail:{}".format(output))
except (Exception) as e:
output = str(e);
log("Fail:{}".format(e))
return str(output)
def log(msg):
msg = str(msg)
d_date = datetime.datetime.now()
now = str(d_date.strftime("%Y-%m-%d %H:%M:%S"))
print(now + " " + msg)
if ("LOG_FILE" in globals()):
with open(LOG_FILE, "a") as myfile:
myfile.write(now + " " + msg + "\n")