Develop Reference

What is subprocess doing with a list and shell=True

I like shell=False to avoid the various problems with shell=True. Sometimes I see code like this, and it returns True, but it doesn't seem to have printed anything, and I don't know what did happen.
subprocess.run(['echo', 'hi'], shell=True, check=True).returncode == 0
By contrast,
subprocess.run(['echo', 'hi'], shell=False, check=True).returncode == 0
actually prints to the stdout.
What happens when I pass a list as arguments and shell=True?

From the documentation:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.
So your command is equivalent to sh -c "echo" "hi", which simply executes echo without arguments.
A more useful example would be to use this mechanism to pass arbitrary data safely to a shell snippet:
file1='my file.txt'
file2='Author&Title - [FOO] **proper**.mp3'
subprocess.run(
['for f; do printf "Input: %s\n" "$f"; done', '_',
file1, file2],
shell=True, check=True)
This prints out the variables from a shell without having to worry about escaping shell metacharacters. (the extra '_' becomes $0).

Using back-ticks in Python subprocess

I want to run this git command through a Python script and get the output of it:
git diff --name-only mybranch `git merge-base mybranch develop`
The purpose of the command is to see what changes have been made on mybranch since the last merge with develop.
To achieve this I'm using subprocess.Popen:
output = subprocess.Popen(["git", "diff", "--name-only", "mybranch", "`git merge-base mybranch develop`"], stdout=subprocess.PIPE, shell=True)
However, this does not work. The variable output.communicate()[0] simply gives me a printout of git usage -- essentially telling me the input command is wrong.
I saw that a similar question exists here, but it only told me to use shell=True which didn't solve my problem.
I also attempted to run the two commands in succession, but that gave me the same output as before. It is possible that I am missing something in this step, though.
Any help or tips are appreciated.

Backticks and subprocess
The backtick being a shell feature, you may not have a choice but to use shell=True, however pass in a shell command string, not a list of args
So for your particular command (assuming it works in the first place)
process = subprocess.Popen("git diff --name-only mybranch `git merge-base mybranch develop`", stdout=subprocess.PIPE, shell=True)
Notice when you call Popen() you get a process, shouldn't be called output IMO
Here's a simple example that works with backticks
>>> process = subprocess.Popen('echo `pwd`', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Or you can use the $(cmd) syntax
>>> process = subprocess.Popen('echo $(pwd)', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'
Here's what did NOT work (for backticks)
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'\n'
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=False)
>>> out, err = process.communicate()
>>> out
'`pwd`\n'

On POSIX, the argument list is passed to /bin/sh -c i.e., only the first argument is recognized as a shell command i.e., the shell runs git without any arguments that is why you see the usage info. You should pass the command as a string if you want to use shell=True. From the subprocess docs:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
You don't need shell=True in this case.
#!/usr/bin/env python
from subprocess import check_output
merge_base_output = check_output('git merge-base mybranch develop'.split(),
universal_newlines=True).strip()
diff_output = check_output('git diff --name-only mybranch'.split() +
[merge_base_output])

Why can't we combine arguments in subprocess.Popen?

When using subprocess.Popen, we have to write
with subprocess.Popen(['ls', '-l', '-a'], stdout=subprocess.PIPE) as proc:
print(proc.stdout.read())
instead of
with subprocess.Popen(['ls', '-l -a'], stdout=subprocess.PIPE) as proc:
print(proc.stdout.read())
Why? What ls will get in the second case? Thank you.

When your operating system starts an executable, it does this via a call something very much like this:
execv('/usr/bin/ls', 'ls', '-l', '-a', NULL)
Note that the arguments are already split out into individual words before ls is started; if you're running your program with a shell, then the shell is responsible for doing that splitting; if you're running it via a programming language that lets you control the execv call's arguments directly, then you're deciding how to split the array up yourself.
When ls runs, it's passed those arguments in an array, argv. Witness the usual way a main function is declared in C:
int main(int argc, char *argv[]) {
...
}
It's getting an array of arguments, in a variable conventionally named argv, already broken up into individual words.
The parser for ls, then, can expect that when it's run it will be handed an array that looks like this:
argc = 3 # three arguments, including our own name
argv = ['ls', '-l', '-a'] # first argument is our name, others follow
...so the command-line parser built into ls doesn't need to break up spaces inside of its arguments -- spaces have already been removed, and syntactic quotes honored and stripped, before the ls command is ever started.
Now, when you run ['ls', '-l -a'], you're explicitly specifying an argc of 2, not 3, and a single argument that includes a single string -l -a. To get that behavior from a shell, you'd need to use quoting or escaping:
ls "-l -a"
ls '-l -a'
ls -l\ -a
...and you'll find that ls fails the exact same way as what you get here when invoked from a shell with any of those usages.

In the second case -l -a as a single string will be the first argument to ls, which it won't know what to do with, or at least won't do what you want. In the first case -l is the first argument and -a is the second argument.
If you want to build a string that has the complete command you can use the shell=True flag to Popen, but then your command would be "ls -l -a" not ['ls', '-l -a']
With Popen each argument in the list is an argument passed to the command being executed, it's not a string passed to the shell to be interpreted, unless you ask for it to be passed to the shell to be interpreted.

If you want to use string representation of command to execute, shlex module may be useful.
shlex.split(s[, comments[, posix]])
Split the string s using shell-like syntax. If comments is False (the default), the parsing of comments in the given string will be
disabled (setting the commenters attribute of the shlex instance to
the empty string). This function operates in POSIX mode by default,
but uses non-POSIX mode if the posix argument is false.
assert shlex.split("ls -a -l") == ['ls', '-a', '-l']
subprocess.Popen(shlex.split("ls -a -l"))
It also covers more complex cases like escaping chars or quotes usage:
assert shlex.split("cat 'file with space.txt'") == ['cat', 'file with space.txt']
assert shlex.split(r"cat file\ with\ space.txt") == ['cat', 'file with space.txt']

Why does simple echo in subprocess not working

I'm trying to perform simple echo operation using subprocess:
import subprocess
import shlex
cmd = 'echo $HOME'
proc = subprocess.Popen(shlex.split(cmd), shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0]
But it prints nothing. Even if I change the command to echo "hello, world" it still prints nothing. Any help is appreciated.

On Unix shell=True implies that 2nd and following arguments are for the shell itself, use a string to pass a command to the shell:
import subprocess
cmd = 'echo $HOME'
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
print proc.communicate()[0],
You could also write it as:
import subprocess
cmd = 'echo $HOME'
print subprocess.check_output(cmd, shell=True),
From the subprocess' docs:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])

You are confusing the two different invocations of Popen. Either of these will work:
proc=subprocess.Popen(['/bin/echo', 'hello', 'world'], shell=False, stdout=subprocess.PIPE)
or
proc=subprocess.Popen('echo hello world', shell=True, stdout=subprocess.PIPE)
When passing shell=True, the first argument is a string--the shell command line. When not using the shell, the first argument is a list. Both produce this:
print proc.communicate()
('hello world\n', None)

Passing arguments to "executable" parameter of the subprocess.Popen() call

The subprocess.Popen() lets you pass the shell of your choice via the "executable" parameter.
I have chosen to pass "/bin/tcsh", and I do not want the tcsh to read my ~/.cshrc.
The tcsh manual says that I need to pass -f to /bin/tcsh to do that.
How do I ask Popen to execute /bin/tcsh with a -f option?
import subprocess
cmd = ["echo hi"]
print cmd
proc = subprocess.Popen(cmd, shell=False, executable="/bin/tcsh", stderr=subprocess.PIPE, stdout=subprocess.PIPE)
return_code = proc.wait()
for line in proc.stdout:
print("stdout: " + line.rstrip())
for line in proc.stderr:
print("stderr: " + line.rstrip())
print return_code

Make your life easier:
subprocess.Popen(['/bin/tcsh', '-f', '-c', 'echo hi'],
shell=False, stderr=subprocess.PIPE, stdout=subprocess.PIPE)

I do not understand what the title of your question "Passing arguments to subprocess executable" has to do with the rest of it, especially "I want the tcsh to not to read my ~/.cshrc."
However - I do know that you are not using your Popen correctly.
Your cmd should either be a list or a string, not a list of 1 string.
So cmd = ["echo hi"] should be either cmd = "echo hi" or cmd = ["echo", "hi"]
Then, depending on if it is a string or list you need to set the shell value to True or False. True if it is a string, False if it is a list.
"passing" an argument is a term for functions, using Popen, or subprocess module is not the same as a function, though they are functions, you are actually running a command with them, not passing arguments to them in the traditional sense, so if you want to run a process with '-f' you simply add '-f' to the string or list that you want to run the command with.
To put the whole thing together, you should run something like:
proc = subprocess.Popen('/bin/tcsh -f -c "echo hi"', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)