Develop Reference

subprocess.run failing to execute git command with var=value prefix but os.system does it without issue

I'm running into a bit of a weird issue here.
I'm trying to clone some GitHub repositories automatically but I'm seeing subprocess.run error out in cases where os.system, or just running the command directly in my shell, works fine.
This is the command I'm trying to run:
subprocess.run('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git'.split())
Which leads to this error:
>>> subprocess.run('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git'.split())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 546, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 1022, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 1899, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'GIT_TERMINAL_PROMPT=0'
When the result should be what os.system gives:
>>> os.system('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git')
Cloning into 'fake-repo'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
32768
Would anyone happen to know the reason behind this?

You need to use the following instead:
subprocess.run(['git', 'clone', 'https://github.com/fake-user/fake-repo.git'], env={'GIT_TERMINAL_PROMPT': '0'})
When you run a command via subprocess, you are not using bash or sh or any shell, so environment variables are not interpreted, and must be specified via env, not the CLI.
Note that it is the best practice to explicitly craft the list for the command and its arguments yourself, rather than using a string and .split(). If you have any arguments that contain spaces, rather than including literal quotes in the string that you pass as a list element, you would use a string that contains the full argument. I.e., NOT ['echo', '"foo', 'bar"'] (which would be the result of .split()), but ['echo', 'foo bar'] to include the space.
Another note as to why this occurs is that subprocess.run does technically allow you to pass shell=True, but you should avoid doing so at all costs due to security implications if you are working with any content that may be user-generated. The best practice is to use env={...} as suggested.
If you need to include current environment variables, then you can use the following:
from os import environ
env = environ.copy()
env['GIT_TERMINAL_PROMPT'] = '0'
subprocess.run(['git', 'clone', 'https://github.com/fake-user/fake-repo.git'], env=env)

does python utilize PATH environment?

My python script is being called by kea-dhcp, which has capability of executing external scripts (https://kea.readthedocs.io/en/latest/arm/hooks.html#run-script-run-script-support-for-external-hook-scripts for details)
import subprocess
...
subprocess.check_output(('bridge', arg1, arg2 ...), stderr=subprocess.STDOUT)
Where does python take information about location of bridge binary? I'm getting an error:
FileNotFoundError: [Errno 2] No such file or directory: 'bridge': 'bridge'
I don't think it has anything to do with PYTHONPATH since it is not a problem of importing modules. What can be possibly wrong?
UPDATE
Following Charles Duffy's suggestion, I did:
res = subprocess.check_output([shutil.which('bridge'), '-j', 'fdb', 'show'], stderr=subprocess.STDOUT)
but got error (it points to above subprocess line):
File "/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/lib64/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/lib64/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/lib64/python3.6/subprocess.py", line 1278, in _execute_child
executable = os.fsencode(executable)
File "/lib64/python3.6/os.py", line 800, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
I can't see what is wrong with that, looks perfectly fine.

How unqualified names are handled by subprocess depends on which operating system you're on. Sometimes PATH is honored; never is PYTHONPATH used for the purpose.
To explicitly perform a PATH lookup, change your code to use shutil.which():
import shutil, subprocess
subprocess.check_output([shutil.which('bridge'), arg1, arg2], stderr=subprocess.STDOUT)
Quoting from https://docs.python.org/3/library/subprocess.html#popen-constructor (formatting from the original) --
Warning For maximum reliability, use a fully qualified path for the executable. To search for an unqualified name on PATH, use shutil.which(). On all platforms, passing sys.executable is the recommended way to launch the current Python interpreter again, and use the -m command-line format to launch an installed module.
Resolving the path of executable (or the first item of args) is platform dependent. For POSIX, see os.execvpe(), and note that when resolving or searching for the executable path, cwd overrides the current working directory and env can override the PATH environment variable. For Windows, see the documentation of the lpApplicationName and lpCommandLine parameters of WinAPI CreateProcess, and note that when resolving or searching for the executable path with shell=False, cwd does not override the current working directory and env cannot override the PATH environment variable. Using a full path avoids all of these variations.

The problem is that kea-dhcp does not export system environment variables to scripts it executes, it only supplies its specific environment. Discovered this via os.environ. The only solution to this is provide the full pass to bridge command (or any other launched in this context).

How to python subprocess.check_output linux command on mac os

I am trying to run a linux executable on Max OS X 10.11.6 via python2.7
I would like to use subprocess.check_output.
The command, which works via the terminal is:
mosel -c "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"
However, when I try:
subprocess.check_output(['mosel','-c',cmd])
where
cmd="exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"'
I get:
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 629, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'mosel'
I have been able to get it to "echo" the command to an output file, but I cannot run "which mosel" via python, which leads me to believe that it has to do with check_output using "bin/sh"as the executable.
So, do I need to use "Popen" instead and set
executable=path/to/mosel
?
If so, how do use Python to get the user's path to mosel (i.e. get the output of "which mosel")?
Thanks!
UPDATE:
PyCharm was not seeing the system paths, which I fixed using this answer:
Setting environment variables in OS X?
Now, it appears that
subprocess.check_output(['mosel','-c',cmd])
Is sending the square brackets to the command line, because it now returns:
dyld: Library not loaded: libxprm_mc.dylib
Referenced from: /usr/local/opt/xpress/bin/mosel
Reason: image not found
Traceback (most recent call last):
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 70, in <module>
run(1)
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 44, in run
out = check_output(['mosel', '-c', cmd])
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 219, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['mosel', '-c', cmd]' returned non-zero exit status -5
Or is there still a path issue?! (I can run mosel -c cmd via the mac terminal, but not in pycharm via python, nor in the mac terminal via python).

The problem is you're using check_output's arguments incorrectly. Unless you pass it shell=True, check_output expects a list of parameters as its input, in the form:
check_call(['binary_name','arg1','arg2','arg3', ....])
So in this case, you should do:
subprocess.check_call(['mosel', '-c', "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"])

The root of the issue turns out to be the DYLD_LIBRARY_PATH:
The new OS X release 10.11 "El Capitan" has a "security" feature that
prevents passing DYLD_LIBRARY_PATH to child processes. Somehow, that
variable is stripped from the environment. - (quoted from https://www.postgresql.org/message-id/20151103113612.GW11897#awork2.anarazel.de)
The security feature is called SIP or "System Integrity Protection". Unfortunately, it seems that no one has come up with a solution to this issue (other than work-arounds that must be tailored to each situation).
Here is another example of this issue:
https://github.com/soumith/cudnn.torch/issues/111
Google "mac os inherit dyld_library_path" and you will find many other examples.

python subprocess.call() fails in Sublime Text 2 on this program

I've been stuck on a peculiar error for several hours, searching for solutions in Google and failing, probably because the problem is so specific, but it actually has broader implications, which is why I've been trying hard to fix it.
I am using Python 2.6 and can use the subprocess.call() to run a program called STAMP in the vanilla Python terminal, or even the iPython terminal as such:
>>>Import subprocess
>>>subprocess.call('stamp')
That works fine, but when I execute the same thing via sublime text 2 (ST2) using it's sublimeREPL plug-in for python or iPython (Tools>sublimeREPL>Python>...) it fails with the following error:
> Traceback (most recent call last): File "<stdin>", line 1, in
> <module> File "<string>", line 27, in <module> File
> "/usr/lib64/python2.6/subprocess.py", line 478, in call
> p = Popen(*popenargs, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
> errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
> raise child_exception OSError: [Errno 2] No such file or directory
Which is the same error it gives when you call a program that is not installed on the system. It seems paradoxical that this error does not appear for any other installed programs/commands that I've tested other than 'stamp' (so you would think sublimeREPL is working fine), and yet running subprocess.call('stamp') does work in the native python terminal, as well as iPython (so you would think stamp is working/installed fine). The only clue I had in mind was that I had to install stamp using g++
Summary:
subprocess.call('stamp') works in a native python terminal
subprocess.call('stamp') does not work in ST2's sublimeREPL python terminal
subprocess.call() seems to work fine in both sublimeREPL or native python terminal
Extra info:
Python 2.6.3
Installation procedure for stamp:
Step 1: Install the GNU Scientific Library.
Download from http://www.gnu.org/software/gsl/
Add the 'include' and 'lib' directories to your PATH.
Step 2: Compile the STAMP code.
Use a command such as the following:
g++ -O3 -o stamp Motif.cpp Alignment.cpp ColumnComp.cpp \
PlatformSupport.cpp PlatformTesting.cpp Tree.cpp \
NeuralTree.cpp MultipleAlignment.cpp RandPSSMGen.cpp \
ProteinDomains.cpp main.cpp -lm -lgsl -lgslcblas
Note: if the GSL library is not in the PATH, add the appropriate
directories using the -L and -I compiler options.
Step 3: Test it out!

Converting my comment to answer form so that it's clearer that the problem was solved:
PATH is not set inside the sublimeREPL terminal. If you provide the full path to the stamp executable, subprocess.call will work fine.

What does the 'shell' argument in subprocess mean on Windows?

The docs for the subprocess module state that 'If shell is True, the specified command will be executed through the shell'. What does this mean in practice, on a Windows OS?

It means that the command will be executed using the program specified in the COMSPEC environment variable. Usually cmd.exe.
To be exact, subprocess calls the CreateProcess windows api function, passing "cmd.exe /c " + args as the lpCommandLine argument.
If shell==False, the lpCommandLine argument to CreateProcess is simply args.

When you execute an external process, the command you want may look something like "foo arg1 arg2 arg3". If "foo" is an executable, that is what gets executed and given the arguments.
However, often it is the case that "foo" is actually a script of some sort, or maybe a command that is built-in to the shell and not an actual executable file on disk. In this case the system can't execute "foo" directly because, strictly speaking, these sorts of things aren't executable. They need some sort of "shell" to execute them. On *nix systems this shell is typically (but not necessarily) /bin/sh. On windows it will typically be cmd.exe (or whatever is stored in the COMSPEC environment variable).
This parameter lets you define what shell you wish to use to execute your command, for the relatively rare case when you don't want the default.

In addition to what was said in other answers, it is useful in practice if you want to open a file in the default viewer for that file type. For instance, if you want to open an HTML or PDF file, but will not know which browser or viewer is installed on the systems it will be run on, or have no guarantees as to the path to the executable, you can simply pass the file name as the only argument for the args field, then set shell=True. This will have Windows use whatever program is associated with that file type.
One caveat, if the path to your file has spaces, you need to surround it with two ".
eg.
path = "C:\\Documents and Settings\\Bob\\Desktop\\New Folder\\README.txt"
subprocess.call('""' + path + '""', shell = True)

In using-the-subprocess-module, there is an explicit paragraph:
The executable argument specifies the program to execute. It is very seldom needed: Usually, the program to execute is defined by the args argument. If shell=True, the executable argument specifies which shell to use. On Unix, the default shell is /bin/sh. On Windows, the default shell is specified by the COMSPEC environment variable.
Windows example - the shell (cmd.exe) command date -t will not be recognized without the shell:
>>> p=subprocess.Popen(["date", "/t"], stdout=subprocess.PIPE)
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
>>>
Using a shell, all is well:
>>> p=subprocess.Popen(["date", "/t"], shell=True, stdout=subprocess.PIPE)
>>> p.communicate()
('Wed 04/22/2009 \r\n', None)
>>>