Develop Reference

Assigning shell script execution result to variable

I have shell script. Which has :
./dtapi get_probability decision_tree simulated_diabetes_incidence_data_new.txt AGE 70 weight 34 height 5.5 sex 0 ds1 34
Now I am trying to execute this shell script withing python script and store the result into some variable. test.py contains -
import os, sys
result = os.system("sh cmd_dtapi.sh")
print "Result is : ", result
But it behaves like this:
python test.py
{"risk_of_disease":"2.122e-314"}Result is : 0
Result get printed directly, assignment takes 0 .
How can I store the result into some variable?
Update
After following answer -
import subprocess
import json
result_process_output = subprocess.check_output("sh cmd_dtapi.sh")
result_json = json.loads(result_process_output)
result = result_json["risk_of_disease"]
print "Result is : ", result
Gives
Traceback (most recent call last):
File "test.py", line 3, in <module>
result_process_output = subprocess.check_output("sh cmd_dtapi.sh")
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Here is despcription of os.system():
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command. If command generates any output, it will be sent to the interpreter standard output stream.
Which means, that your shell prints {risk_of_disease":"2.122e-314"} to standard output.
As for os.system() returning 0:
On Windows, the return value is that returned by the system shell after running command. The shell is given by the Windows environment variable COMSPEC: it is usually cmd.exe, which returns the exit status of the command run; on systems using a non-native shell, consult your shell documentation.
So return code of your shell script was 0 and it is assigned to result. So technically you are already storing result in variable.
#edit:
To resolve problem, you'd need to use subprocess module which allows for more detailed manipulation of system calls.
import subprocess
import json
result_process_output = subprocess.check_output("sh cmd_dtapi.sh", shell=True)
result_json = json.loads(result_process_output)
result = result_json["risk_of_disease"]
print "Result is : ", result

subprocess.Popen() error (No such file or directory) when calling command with arguments as a string

I am trying to count the number of lines in a file using Python functions. Within the current directory, while os.system("ls") finds the file, the command subprocess.Popen(["wc -l filename"], stdout=subprocess.PIPE) does not work.
Here is my code:
>>> import os
>>> import subprocess
>>> os.system("ls")
sorted_list.dat
0
>>> p = subprocess.Popen(["wc -l sorted_list.dat"], stdout=subprocess.PIPE)File "<stdin>", line 1, in <module>
File "/Users/a200/anaconda/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/Users/a200/anaconda/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

You should pass the arguments as a list (recommended):
subprocess.Popen(["wc", "-l", "sorted_list.dat"], stdout=subprocess.PIPE)
Otherwise, you need to pass shell=True if you want to use the whole "wc -l sorted_list.dat" string as a command (not recommended, can be a security hazard).
subprocess.Popen("wc -l sorted_list.dat", shell=True, stdout=subprocess.PIPE)
Read more about shell=True security issues here.

The error occurs because you are trying to run a command named wc -l sorted_list.dat, that is, it is trying to find a file named like "/usr/bin/wc -l sorted dat".
Split your arguments:
["wc", "-l", "sorted_list.dat"]

subprocess.check_output(): OSError file not found in Python

Executing following command and its variations always results in an error, which I just cannot figure out:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output([command])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
WHich file it is referring to ? other commands like ls,wc are running correctly though, the command is also running well on terminal but not python script.

Your command is a list with one element. Imagine if you tried to run this at the shell:
/bin/'dd if='/dev/'sda8 count=100 skip=$(expr 19868431049 '/' 512)'
That's effectively what you're doing. There's almost certainly no directory named dd if= in your bin directory, and there's even more almost certainly no dev directory under that with an sd8 count=100 skip=$(expr 19868431049 directory with a program named 512 in it.
What you want is a list where each argument is its own element:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100', 'skip=$(expr 19868431049 / 512)']
print subprocess.check_output(command) # notice no []
But that brings us to your second problem: $(expr 19868431049 / 512) isn't going to be parsed by Python or by dd; that's bash syntax. You can, of course, just do the same thing in Python instead of in bash:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100',
'skip={}'.format(19868431049 // 512)]
print subprocess.check_output(command)
Or, if you really want to use bash for no good reason, pass a string, rather than a list, and use shell=True:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True) # still no []
Although that still isn't going to work portably, because the default shell is /bin/sh, which may not know how to handle bashisms like $(…) (and expr, although I think POSIX requires that expr exist as a separate process…). So:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True, executable='/bin/bash')

This worked for me using subprocess.popen
command = "echo $JAVA_HOME"
proc = subprocess.Popen(command,stdout=subprocess.PIPE,shell=True)

Python subprocess command arguments

Why if I run subprocess.check_output('ls') everything is working but when I add argument to command like: subprocess.check_output('ls -la') I get error:
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
How can I pass command arguments into subprocess.check_output()?

You need to split the arguments into a list:
subprocess.check_output(['ls', '-la'])
The subprocess callables do not parse the command out to individual arguments like the shell does. You either need to do this yourself or you need to tell subprocess to use the shell explicitly:
subprocess.check_output('ls -la', shell=True)
The latter is not recommended as it can expose your application to security vulnerabilities. You can use shlex.split() to parse a shell-like command line if needed:
>>> import shlex
>>> shlex.split('ls -la')
['ls', '-la']

You might find sh.py more friendly:
import sh
print sh.ls("-la")

Popen error: "[Errno 2] No such file or directory" when calling shell function

I have some custom commands.
This works:
subprocess.Popen(['python'], stdout=subprocess.PIPE)
But if I have my own system commands like deactivate, I get that error
Traceback (most recent call last):
File "runner2.py", line 21, in <module>
main()
File "runner2.py", line 18, in main
subprocess.Popen(['deactivate',], stdout=subprocess.PIPE)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Let alone I need to execute this under my sandbox virtualenv.

Try add an extra parameter shell=True to the Popen call.

Just a note. shell=True was likely the correct solution to the o.p., since they did not make the following mistake, but you can also get the "No such file or directory" error if you do not split up your executable from its arguments.
import subprocess as sp, shlex
sp.Popen(['echo 1']) # FAILS with "No such file or directory"
sp.Popen(['echo', '1']) # SUCCEEDS
sp.Popen(['echo 1'], shell=True) # SUCCEEDS, but extra overhead
sp.Popen(shlex.split('echo 1')) # SUCCEEDS, equivalent to #2
Without shell=True, Popen expects the executable to be the first element of args, which is why it fails, there is no "echo 1" executable. Adding shell=True invokes your system shell and passes the first element of args to the shell. i.e. for linux, Popen(['echo 1'], shell=True) is equivalent to Popen('/bin/sh', '-c', 'echo 1') which is more overhead than you may need. See Popen() documentation for cases when shell=True is actually useful.

You have to give the full path to your program deactivate and then it the subprocess module should be able to find it.

I'm spawning subprocesses like that:
SHUTDOWN_CMD = os.path.sep.join(["c:", "windows", "system32", "shutdown.exe"])
def abortShutdown():
os.spawnv(os.P_NOWAIT, SHUTDOWN_CMD,
[SHUTDOWN_CMD, '/A'])
time.sleep(3)
I'm not using subprocess since Python 2.5 does not support it. I had to use use the FULL path to have it working and I guess you also have to use the full path to your custom commands.