The problem I'm having is with Eclipse/PyCharm interpreting the results of subprocess's Popen() differently from a standard terminal. All are using python2.6.1 on OSX.
Here's a simple example script:
import subprocess
args = ["/usr/bin/which", "git"]
print "Will execute %s" % " ".join(args)
try:
p = subprocess.Popen(["/usr/bin/which", "git"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# tuple of StdOut, StdErr is the responses, so ..
ret = p.communicate()
if ret[0] == '' and ret[1] <> '':
msg = "cmd %s failed: %s" % (fullcmd, ret[1])
if fail_on_error:
raise NameError(msg)
except OSError, e:
print >>sys.stderr, "Execution failed:", e
With a standard terminal, the line:
ret = p.communicate()
gives me:
(Pdb) print ret
('/usr/local/bin/git\n', '')
Eclipse and PyCharm give me an empty tuple:
ret = {tuple} ('','')
Changing the shell= value does not solve the problem either. On the terminal, setting shell=True, and passing the command in altogether (i.e., args=["/usr/bin/which git"]) gives me the same result: ret = ('/usr/local/bin/git\n', ''). And Eclipse/PyCharm both give me an empty tuple.
Any ideas on what I could be doing wrong?
Ok, found the problem, and it's an important thing to keep in mind when using an IDE in a Unix-type environment. IDE's operate under a different environment context than the terminal user (duh, right?!). I was not considering that the subprocess was using a different environment than the context that I have for my terminal (my terminal has bash_profile set to have more things in PATH).
This is easily verified by changing the script as follows:
import subprocess
args = ["/usr/bin/which", "git"]
print "Current path is %s" % os.path.expandvars("$PATH")
try:
p = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# tuple of StdOut, StdErr is the responses, so ..
out, err = p.communicate()
if err:
msg = "cmd %s failed: %s" % (fullcmd, err)
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Under the terminal, the path includes /usr/local/bin. Under the IDE it does not!
This is an important gotcha for me - always remember about environments!
Related
I have a program which can be execute as
./install.sh
This install bunch of stuff and has quite a lot of activity happening on screen..
Now, I am trying to execute it via
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
With the hope that all the activity happening on the screen is captured in out (or err). However, content is printed directly to the terminal while the process is running, and not captured into out or err, which are both empty after the process is run.
What could be happening here? How can this content be captured?
In general, what you're doing is already sufficient to channel all output to your variables.
One exception to that is if the program you're running is using /dev/tty to connect directly to its controlling terminal, and emitting output through that terminal rather than through stdout (FD 1) and stderr (FD 2). This is commonly done for security-sensitive IO such as password prompts, but rarely seen otherwise.
As a demonstration that this works, you can copy-and-paste the following into a Python shell exactly as given:
import subprocess
executable = ['/bin/sh', '-c', 'echo stdout; echo stderr >&2']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out
print "stderr: ", err
...by contrast, for a demonstration of the case that doesn't work:
import subprocess
executable = ['/bin/sh', '-c', 'echo uncapturable >/dev/tty']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out
In this case, content is written to the TTY directly, not to stdout or stderr. This content cannot be captured without using a program (such as script or expect) that provides a fake TTY. So, to use script:
import subprocess
executable = ['script', '-q', '/dev/null',
'/bin/sh', '-c', 'echo uncapturable >/dev/tty']
p = subprocess.Popen(executable, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print "---"
print "output: ", out
I am using:
grepOut = subprocess.check_output("grep " + search + " tmp", shell=True)
To run a terminal command, I know that I can use a try/except to catch the error but how can I get the value of the error code?
I found this on the official documentation:
exception subprocess.CalledProcessError
Exception raised when a process run by check_call() or check_output() returns a non-zero exit status.
returncode
Exit status of the child process.
But there are no examples given and Google was of no help.
You can get the error code and results from the exception that is raised.
This can be done through the fields returncode and output.
For example:
import subprocess
try:
grepOut = subprocess.check_output("grep " + "test" + " tmp", shell=True)
except subprocess.CalledProcessError as grepexc:
print("error code", grepexc.returncode, grepexc.output)
Python 3.5 introduced the subprocess.run() method. The signature looks like:
subprocess.run(
args,
*,
stdin=None,
input=None,
stdout=None,
stderr=None,
shell=False,
timeout=None,
check=False
)
The returned result is a subprocess.CompletedProcess. In 3.5, you can access the args, returncode, stdout, and stderr from the executed process.
Example:
>>> result = subprocess.run(['ls', '/tmp'], stdout=subprocess.DEVNULL)
>>> result.returncode
0
>>> result = subprocess.run(['ls', '/nonexistent'], stderr=subprocess.DEVNULL)
>>> result.returncode
2
is there a way to get a return code without a try/except?
check_output raises an exception if it receives non-zero exit status because it frequently means that a command failed. grep may return non-zero exit status even if there is no error -- you could use .communicate() in this case:
from subprocess import Popen, PIPE
pattern, filename = 'test', 'tmp'
p = Popen(['grep', pattern, filename], stdin=PIPE, stdout=PIPE, stderr=PIPE,
bufsize=-1)
output, error = p.communicate()
if p.returncode == 0:
print('%r is found in %s: %r' % (pattern, filename, output))
elif p.returncode == 1:
print('%r is NOT found in %s: %r' % (pattern, filename, output))
else:
assert p.returncode > 1
print('error occurred: %r' % (error,))
You don't need to call an external command to filter lines, you could do it in pure Python:
with open('tmp') as file:
for line in file:
if 'test' in line:
print line,
If you don't need the output; you could use subprocess.call():
import os
from subprocess import call
try:
from subprocess import DEVNULL # Python 3
except ImportError: # Python 2
DEVNULL = open(os.devnull, 'r+b', 0)
returncode = call(['grep', 'test', 'tmp'],
stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL)
To get both output and return code (without try/except) simply use subprocess.getstatusoutput (Python 3 required)
In Python 2 - use commands module:
import command
rc, out = commands.getstatusoutput("ls missing-file")
if rc != 0: print "Error occurred: %s" % out
In Python 3 - use subprocess module:
import subprocess
rc, out = subprocess.getstatusoutput("ls missing-file")
if rc != 0: print ("Error occurred:", out)
Error occurred: ls: cannot access missing-file: No such file or directory
I am using python 2.7 on windows 7 64 bit machine.
I am calling external application within python code as
os.startfile("D:\\dist\\NewProcess.exe")
This application(used py2exe for converting python script into an exe) uses two strings, which need to pass from parent process.
So, how to pass these two strings, and how to get these strings in NewProcess.py file(may be by sys.argv argument)
You may try this:
import subprocess
import sys
try:
retcode = subprocess.call("D:\\dist\\NewProcess.exe " + sys.argv[1] + " " + sys.argv[2], shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError as e:
print >>sys.stderr, "Execution failed:", e
In sys.argv[0] is the script name, and in sys.argv[1] ... [n] are script arguments. Example above is taken from subprocess module documentation https://docs.python.org/2/library/subprocess.html
I am calling the executable from python script using sub process call. these are the following code I have used:
try:
p = subprocess.Popen([abc.exe], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
except Exception as e:
print str(e)
from abc.exe, I have return 1 in failure case and return 0 for success case. But I don't know how to check the return value in python script.
thanks,
Popen.returncode contains the return code when the process has terminated. You can ensure that using Popen.wait.
You've saved as p the output from .communicate(), not Popen object. Perhaps try:
try:
p = subprocess.Popen(['abc.exe'], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except OSError as e:
print str(e)
stdoutdata, stderrdata = p.communicate()
retcode = p.returncode
Another way to do this is to use subprocess.check_output() since you mention Python 2.7. This runs the command with the same arguments as Popen. The output of the command is returned as a string. If the command returns a non-zero value, a subprocess.CalledProcessError exception is raised.
So I think you can rework your code to something like this:
try:
output = subprocess.check_output(['abc.exe'], shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as ex:
# an error occurred
retcode = ex.returncode
output = ex.output
else:
# no error occurred
process(output)
Note that you can't use the stdout argument in check_output since it is used internally. Here are the docs.
I tried running the myview command and it ran successfully, but I am stuck after this step.
I have to choose from a list of views and have to pass in a number, for example say <1>,<2>..<10>. But when I execute the script it shows me option on the terminal window instead.
Which command should I be using? Because after this I have to run a bunch of other commands as well and basically have to execute them in a particular order. So say cmd should wait for cmd to finish. Thanks in advance for the help.
This is what I have so far.
#! /usr/bin/python
import sys
from subprocess import call
for arg in sys.argv:
print arg
call(["myview"])
Check out the doc for subprocess. I think the API call you need is check_call.
From pydoc subprocess:
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e