I'm trying to perform simple echo operation using subprocess:
import subprocess
import shlex
cmd = 'echo $HOME'
proc = subprocess.Popen(shlex.split(cmd), shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0]
But it prints nothing. Even if I change the command to echo "hello, world" it still prints nothing. Any help is appreciated.
On Unix shell=True implies that 2nd and following arguments are for the shell itself, use a string to pass a command to the shell:
import subprocess
cmd = 'echo $HOME'
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0],
You could also write it as:
import subprocess
cmd = 'echo $HOME'
print subprocess.check_output(cmd, shell=True),
From the subprocess' docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You are confusing the two different invocations of Popen. Either of these will work:
proc=subprocess.Popen(['/bin/echo', 'hello', 'world'], shell=False, stdout=subprocess.PIPE)
or
proc=subprocess.Popen('echo hello world', shell=True, stdout=subprocess.PIPE)
When passing shell=True, the first argument is a string--the shell command line. When not using the shell, the first argument is a list. Both produce this:
print proc.communicate()
('hello world\n', None)
Related
I like shell=False to avoid the various problems with shell=True. Sometimes I see code like this, and it returns True, but it doesn't seem to have printed anything, and I don't know what did happen.
subprocess.run(['echo', 'hi'], shell=True, check=True).returncode == 0
By contrast,
subprocess.run(['echo', 'hi'], shell=False, check=True).returncode == 0
actually prints to the stdout.
What happens when I pass a list as arguments and shell=True?
From the documentation:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.
So your command is equivalent to sh -c "echo" "hi", which simply executes echo without arguments.
A more useful example would be to use this mechanism to pass arbitrary data safely to a shell snippet:
file1='my file.txt'
file2='Author&Title - [FOO] **proper**.mp3'
subprocess.run(
['for f; do printf "Input: %s\n" "$f"; done', '_',
file1, file2],
shell=True, check=True)
This prints out the variables from a shell without having to worry about escaping shell metacharacters. (the extra '_' becomes $0).
I am using subprocess.Popen as opposed to os.fork(), and tryin the "shell=True" construct. However, the arguments to be passed to the child process are getting deleted. Any idea why, and what could be the fix?
The files attached are "/tmp/a.py" and "/tmp/a.pl". If I run a.py without any argument, I get the expected results. With an argument, an error message.
#!/opt/local/bin/python3.6
import sys, subprocess
class child_proc:
def __init__ (self, useShell):
print ("useShell:", useShell)
acmd = ["/tmp/a.pl", "Hello", "World"]
if useShell:
proc = subprocess.Popen(acmd, stdout=subprocess.PIPE, shell=True)
else:
proc = subprocess.Popen(acmd, stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if line:
try:
aline = line.decode().rstrip()
print (aline)
except UnicodeDecodeError as e:
print("Error decoding child output:", line, e)
break
else:
break
child_proc(len(sys.argv) > 1)
The script it is calling -
#!/opt/local/bin/perl -w
die "[a.pl] Missing arguments\n" if $#ARGV < 0;
print "[a.pl] #ARGV\n";
exit 0;
This is on MacOS10.13.1. Thank you for your insight.
From the docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell. ... If
args is a sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell itself.
So if shell=True, you're supposed to pass a string instead of a list of arguments, which will be interpreted literally by the shell. If you pass a sequence as you're doing now, it treats the first element as the entire command and the rest as arguments to the shell. So if you need shell=True, pass a string.
proc = subprocess.Popen("./tmp/a.pl Hello World", stdout=subprocess.PIPE, shell=True)
And be careful not to pass any user input to the shell when you do it this way. As the docs warn, untrusted input to Popen with a shell=True argument is a major security hazard.
When you use a list as the command container, with shell=True, the first element would be treated as the command to run, and then rest would be passed as arguments to the shell itself.
So, if you want to use shell=True to run the command is a shell, make it a string:
acmd = "/tmp/a.pl Hello World"
Note: Be careful about running something directly in the shell in unescaped form.
Or, better, you should drop the shell=True and make subprocess do the fork()-exec() as there is nothing shell specific in your code.
For completeness, if you insist on keeping your current structure, you should do:
if useShell:
proc = subprocess.Popen("/tmp/a.pl Hello World", stdout=subprocess.PIPE, shell=True)
else:
proc = subprocess.Popen(["/tmp/a.pl", "Hello", "World"], stdout=subprocess.PIPE)
Or you can set the acmd at the start based on useShell's value, it gets simpler as you can drop the if...else construct too by setting shell as useShell inside subprocess.Popen:
acmd = "/tmp/a.pl Hello World" if useShell else ["/tmp/a.pl", "Hello", "World"]
proc = subprocess.Popen(acmd, stdout=subprocess.PIPE, shell=useShell)
As a side note, you should use snake_case for variable/function names, and CamelCase for class names.
I want to execute bash command
'/bin/echo </verbosegc> >> /tmp/jruby.log'
in python using Popen. The code does not raise any exception, but none change is made on the jruby.log after execution. The python code is shown below.
>>> command='/bin/echo </verbosegc> >> '+fullpath
>>> command
'/bin/echo </verbosegc> >> /tmp/jruby.log'
>>process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
>>> output= process.communicate()[0]
>>> output
'</verbosegc> >> /tmp/jruby.log\n
I also print out the process.pid and then check the pid using ps -ef | grep pid. The result shows that the process pid has been finished.
Just use pass file object if you want to append the output to a file, you cannot redirect to a file unless you set shell=True:
command = ['/bin/echo', '</verbosegc>']
with open('/tmp/jruby.log',"a") as f:
subprocess.check_call(command, stdout=f,stderr=subprocess.STDOUT)
The first argument to subprocess.Popen is the array ['/bin/echo', '</verbosegc>', '>>', '/tmp/jruby.log']. When the first argument to subprocess.Popen is an array, it does not launch a shell to run the command, and the shell is what's responsible for interpreting >> /tmp/jruby.log to mean "write output to jruby.log".
In order to make the >> redirection work in this command, you'll need to pass command directly to subprocess.Popen() without splitting it into a list. You'll also need to quote the first argument (or else the shell will interpret the "<" and ">" characters in ways you don't want):
command = '/bin/echo "</verbosegc>" >> /tmp/jruby.log'
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
Consider the following:
command = [ 'printf "%s\n" "$1" >>"$2"', # shell script to execute
'', # $0 in shell
'</verbosegc>', # $1
'/tmp/jruby.log' ] # $2
subprocess.Popen(command, shell=True)
The first argument is a shell script referring to $1 and $2, which are in turn passed as separate arguments. Keeping data separate from code, rather than trying to substitute the former into the latter, is a precaution against shell injection (think of this as an analog to SQL injection).
Of course, don't actually do anything like this in Python -- the native primitives for file IO are far more appropriate.
Have you tried without splitting the command and using shell=True? My usual format is:
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
output = process.stdout.read() # or .readlines()
I want to run this git command through a Python script and get the output of it:
git diff --name-only mybranch `git merge-base mybranch develop`
The purpose of the command is to see what changes have been made on mybranch since the last merge with develop.
To achieve this I'm using subprocess.Popen:
output = subprocess.Popen(["git", "diff", "--name-only", "mybranch", "`git merge-base mybranch develop`"], stdout=subprocess.PIPE, shell=True)
However, this does not work. The variable output.communicate()[0] simply gives me a printout of git usage -- essentially telling me the input command is wrong.
I saw that a similar question exists here, but it only told me to use shell=True which didn't solve my problem.
I also attempted to run the two commands in succession, but that gave me the same output as before. It is possible that I am missing something in this step, though.
Any help or tips are appreciated.
Backticks and subprocess
The backtick being a shell feature, you may not have a choice but to use shell=True, however pass in a shell command string, not a list of args
So for your particular command (assuming it works in the first place)
process = subprocess.Popen("git diff --name-only mybranch `git merge-base mybranch develop`", stdout=subprocess.PIPE, shell=True)
Notice when you call Popen() you get a process, shouldn't be called output IMO
Here's a simple example that works with backticks
>>> process = subprocess.Popen('echo `pwd`', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Or you can use the $(cmd) syntax
>>> process = subprocess.Popen('echo $(pwd)', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Here's what did NOT work (for backticks)
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'\n'
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=False)
>>> out, err = process.communicate()
>>> out
'`pwd`\n'
On POSIX, the argument list is passed to /bin/sh -c i.e., only the first argument is recognized as a shell command i.e., the shell runs git without any arguments that is why you see the usage info. You should pass the command as a string if you want to use shell=True. From the subprocess docs:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You don't need shell=True in this case.
#!/usr/bin/env python
from subprocess import check_output
merge_base_output = check_output('git merge-base mybranch develop'.split(),
universal_newlines=True).strip()
diff_output = check_output('git diff --name-only mybranch'.split() +
[merge_base_output])
The subprocess.Popen() lets you pass the shell of your choice via the "executable" parameter.
I have chosen to pass "/bin/tcsh", and I do not want the tcsh to read my ~/.cshrc.
The tcsh manual says that I need to pass -f to /bin/tcsh to do that.
How do I ask Popen to execute /bin/tcsh with a -f option?
import subprocess
cmd = ["echo hi"]
print cmd
proc = subprocess.Popen(cmd, shell=False, executable="/bin/tcsh", stderr=subprocess.PIPE, stdout=subprocess.PIPE)
return_code = proc.wait()
for line in proc.stdout:
print("stdout: " + line.rstrip())
for line in proc.stderr:
print("stderr: " + line.rstrip())
print return_code
Make your life easier:
subprocess.Popen(['/bin/tcsh', '-f', '-c', 'echo hi'],
shell=False, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
I do not understand what the title of your question "Passing arguments to subprocess executable" has to do with the rest of it, especially "I want the tcsh to not to read my ~/.cshrc."
However - I do know that you are not using your Popen correctly.
Your cmd should either be a list or a string, not a list of 1 string.
So cmd = ["echo hi"] should be either cmd = "echo hi" or cmd = ["echo", "hi"]
Then, depending on if it is a string or list you need to set the shell value to True or False. True if it is a string, False if it is a list.
"passing" an argument is a term for functions, using Popen, or subprocess module is not the same as a function, though they are functions, you are actually running a command with them, not passing arguments to them in the traditional sense, so if you want to run a process with '-f' you simply add '-f' to the string or list that you want to run the command with.
To put the whole thing together, you should run something like:
proc = subprocess.Popen('/bin/tcsh -f -c "echo hi"', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)