Develop Reference

Exit if the called python script encounters an error

I have a central python script that calls various other python scripts and looks like this:
os.system("python " + script1 + args1)
os.system("python " + script2 + args2)
os.system("python " + script3 + args3)
Now, I want to exit from my central script if any of the sub-scripts encounter an error.
What is happening with current code is that let's say script1 encounters an error. The console will display that error and then central script will move onto calling script2 and so on.
I want to display the encountered error and immediately exit my central code.
What is the best way to do this?

Overall this is a terrible way to execute a series of commands from within Python. However here's a minimal way to handle it:
#!python
import os, system
for script, args in some_tuple_of_commands:
exit_code = os.system("python " + script + args)
if exit_code > 0:
print("Error %d running 'python %s %s'" % (
exit_code, script, args), file=sys.stderr)
sys.exit(exit_code)
But, honestly this is all horrible. It's almost always a bad idea to concatenate strings and pass them to your shell for execution from within any programming language.
Look at the subprocess module for much more sane handling of subprocesses in Python.
Also consider trying the sh or the pexpect third party modules depending on what you're trying to do with input or output.

You can try subprocess
import subprocess,sys
try:
output = subprocess.check_output("python test.py", shell=True)
print(output)
except ValueError as e:
print e
sys.exit(0)
print("hello world")

I don't know if it's ideal for you but enclosing these commands in a function seems a good idea to me:
I am using the fact that when a process exits with error os.system(process) returns 256 else it returns 0 as an output respectively.
def runscripts():
if os.system("python " + script1 + args1):return(-1); #Returns -1 if script1 fails and exits.
if os.system("python " + script2 + args2):return(-2); #Returns -2 and exits
if os.system("python " + script3 + args3):return(-3); #Pretty obvious
return(0)
runscripts()
#or if you want to exit the main program
if runscripts():sys.exit(0)

Invoking the operating system like that is a security breach waiting to happen. One should use the subprocess module, because it is more powerful and does not invoke a shell (unless you specifically tell it to). In general, avoid invoking shell whenever possible (see this post).
You can do it like this:
import subprocess
import sys
# create a list of commands
# each command to subprocess.run must be a list of arguments, e.g.
# ["python", "echo.py", "hello"]
cmds = [("python " + script + " " + args).split()
for script, args in [(script1, args1), (script2, args2), (script3,
args3)]]
def captured_run(arglist):
"""Run a subprocess and return the output and returncode."""
proc = subprocess.run( # PIPE captures the output
arglist, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return proc.stdout, proc.stderr, proc.returncode
for cmd in cmds:
stdout, stderr, rc = captured_run(cmd)
# do whatever with stdout, stderr (note that they are bytestrings)
if rc != 0:
sys.exit(rc)
If you don't care about the output, just remove the subprocess.PIPE stuff and return only the returncode from the function. You may also want to add a timeout to the execution, see the subprocess docs linked above for how to do that.

How to run a shell script without having to press enter/confirm s.th. inbetween

I'm currently writing a shell script which is interfacing with numerous python scripts. In one of these Python scripts I'm calling grass without starting it explicitly. When I run my shell script I have to hit enter at the point where I call grass (this is the code I got from the official working with grass page):
startcmd = grass7bin + ' -c ' + file_in2 + ' -e ' + location_path
print startcmd
p = subprocess.Popen(startcmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if p.returncode != 0:
print >>sys.stderr, 'ERROR: %s' % err
print >>sys.stderr, 'ERROR: Cannot generate location (%s)' % startcmd
sys.exit(-1)
else:
print 'Created location %s' % location_path
gsetup.init(gisbase, gisdb, location, mapset)
My problem is that I want this process to run automatically without me having to press enter everytime in between!
I have already tried numerous options such as pexpect, uinput (doesn't work that well because of problems with the module). I know that in windows you have the msvcrt module, but I am working with linux... any ideas how to solve this problem?

Use the pexpect library for expect functionnality.
Here's an example of interaction with a an application requiring user to type in his password:
child = pexpect.spawn('your command')
child.expect('Enter password:')
child.sendline('your password')
child.expect(pexpect.EOF, timeout=None)
cmd_show_data = child.before
cmd_output = cmd_show_data.split('\r\n')
for data in cmd_output:
print data

I finally found an easy and fast way for simulating a key press:
just install xdotool and then use the following code for simulating e.g. the enter key:
import subprocess
subprocess.call(["xdotool","key","Return"])

subprocess.check_output return code

I am using:
grepOut = subprocess.check_output("grep " + search + " tmp", shell=True)
To run a terminal command, I know that I can use a try/except to catch the error but how can I get the value of the error code?
I found this on the official documentation:
exception subprocess.CalledProcessError
Exception raised when a process run by check_call() or check_output() returns a non-zero exit status.
returncode
Exit status of the child process.
But there are no examples given and Google was of no help.

You can get the error code and results from the exception that is raised.
This can be done through the fields returncode and output.
For example:
import subprocess
try:
grepOut = subprocess.check_output("grep " + "test" + " tmp", shell=True)
except subprocess.CalledProcessError as grepexc:
print("error code", grepexc.returncode, grepexc.output)

Python 3.5 introduced the subprocess.run() method. The signature looks like:
subprocess.run(
args,
*,
stdin=None,
input=None,
stdout=None,
stderr=None,
shell=False,
timeout=None,
check=False
)
The returned result is a subprocess.CompletedProcess. In 3.5, you can access the args, returncode, stdout, and stderr from the executed process.
Example:
>>> result = subprocess.run(['ls', '/tmp'], stdout=subprocess.DEVNULL)
>>> result.returncode
0
>>> result = subprocess.run(['ls', '/nonexistent'], stderr=subprocess.DEVNULL)
>>> result.returncode
2

is there a way to get a return code without a try/except?
check_output raises an exception if it receives non-zero exit status because it frequently means that a command failed. grep may return non-zero exit status even if there is no error -- you could use .communicate() in this case:
from subprocess import Popen, PIPE
pattern, filename = 'test', 'tmp'
p = Popen(['grep', pattern, filename], stdin=PIPE, stdout=PIPE, stderr=PIPE,
bufsize=-1)
output, error = p.communicate()
if p.returncode == 0:
print('%r is found in %s: %r' % (pattern, filename, output))
elif p.returncode == 1:
print('%r is NOT found in %s: %r' % (pattern, filename, output))
else:
assert p.returncode > 1
print('error occurred: %r' % (error,))
You don't need to call an external command to filter lines, you could do it in pure Python:
with open('tmp') as file:
for line in file:
if 'test' in line:
print line,
If you don't need the output; you could use subprocess.call():
import os
from subprocess import call
try:
from subprocess import DEVNULL # Python 3
except ImportError: # Python 2
DEVNULL = open(os.devnull, 'r+b', 0)
returncode = call(['grep', 'test', 'tmp'],
stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL)

To get both output and return code (without try/except) simply use subprocess.getstatusoutput (Python 3 required)

In Python 2 - use commands module:
import command
rc, out = commands.getstatusoutput("ls missing-file")
if rc != 0: print "Error occurred: %s" % out
In Python 3 - use subprocess module:
import subprocess
rc, out = subprocess.getstatusoutput("ls missing-file")
if rc != 0: print ("Error occurred:", out)
Error occurred: ls: cannot access missing-file: No such file or directory

Python script to automate in linux based environment

I tried running the myview command and it ran successfully, but I am stuck after this step.
I have to choose from a list of views and have to pass in a number, for example say <1>,<2>..<10>. But when I execute the script it shows me option on the terminal window instead.
Which command should I be using? Because after this I have to run a bunch of other commands as well and basically have to execute them in a particular order. So say cmd should wait for cmd to finish. Thanks in advance for the help.
This is what I have so far.
#! /usr/bin/python
import sys
from subprocess import call
for arg in sys.argv:
print arg
call(["myview"])

Check out the doc for subprocess. I think the API call you need is check_call.

From pydoc subprocess:
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e

subprocess.Popen() has inconsistent behavior between Eclipse/PyCharm and terminal execution

The problem I'm having is with Eclipse/PyCharm interpreting the results of subprocess's Popen() differently from a standard terminal. All are using python2.6.1 on OSX.
Here's a simple example script:
import subprocess
args = ["/usr/bin/which", "git"]
print "Will execute %s" % " ".join(args)
try:
p = subprocess.Popen(["/usr/bin/which", "git"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# tuple of StdOut, StdErr is the responses, so ..
ret = p.communicate()
if ret[0] == '' and ret[1] <> '':
msg = "cmd %s failed: %s" % (fullcmd, ret[1])
if fail_on_error:
raise NameError(msg)
except OSError, e:
print >>sys.stderr, "Execution failed:", e
With a standard terminal, the line:
ret = p.communicate()
gives me:
(Pdb) print ret
('/usr/local/bin/git\n', '')
Eclipse and PyCharm give me an empty tuple:
ret = {tuple} ('','')
Changing the shell= value does not solve the problem either. On the terminal, setting shell=True, and passing the command in altogether (i.e., args=["/usr/bin/which git"]) gives me the same result: ret = ('/usr/local/bin/git\n', ''). And Eclipse/PyCharm both give me an empty tuple.
Any ideas on what I could be doing wrong?

Ok, found the problem, and it's an important thing to keep in mind when using an IDE in a Unix-type environment. IDE's operate under a different environment context than the terminal user (duh, right?!). I was not considering that the subprocess was using a different environment than the context that I have for my terminal (my terminal has bash_profile set to have more things in PATH).
This is easily verified by changing the script as follows:
import subprocess
args = ["/usr/bin/which", "git"]
print "Current path is %s" % os.path.expandvars("$PATH")
try:
p = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# tuple of StdOut, StdErr is the responses, so ..
out, err = p.communicate()
if err:
msg = "cmd %s failed: %s" % (fullcmd, err)
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Under the terminal, the path includes /usr/local/bin. Under the IDE it does not!
This is an important gotcha for me - always remember about environments!