Develop Reference

subprocess.run failing to execute git command with var=value prefix but os.system does it without issue

I'm running into a bit of a weird issue here.
I'm trying to clone some GitHub repositories automatically but I'm seeing subprocess.run error out in cases where os.system, or just running the command directly in my shell, works fine.
This is the command I'm trying to run:
subprocess.run('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git'.split())
Which leads to this error:
>>> subprocess.run('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git'.split())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 546, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 1022, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 1899, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'GIT_TERMINAL_PROMPT=0'
When the result should be what os.system gives:
>>> os.system('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git')
Cloning into 'fake-repo'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
32768
Would anyone happen to know the reason behind this?

You need to use the following instead:
subprocess.run(['git', 'clone', 'https://github.com/fake-user/fake-repo.git'], env={'GIT_TERMINAL_PROMPT': '0'})
When you run a command via subprocess, you are not using bash or sh or any shell, so environment variables are not interpreted, and must be specified via env, not the CLI.
Note that it is the best practice to explicitly craft the list for the command and its arguments yourself, rather than using a string and .split(). If you have any arguments that contain spaces, rather than including literal quotes in the string that you pass as a list element, you would use a string that contains the full argument. I.e., NOT ['echo', '"foo', 'bar"'] (which would be the result of .split()), but ['echo', 'foo bar'] to include the space.
Another note as to why this occurs is that subprocess.run does technically allow you to pass shell=True, but you should avoid doing so at all costs due to security implications if you are working with any content that may be user-generated. The best practice is to use env={...} as suggested.
If you need to include current environment variables, then you can use the following:
from os import environ
env = environ.copy()
env['GIT_TERMINAL_PROMPT'] = '0'
subprocess.run(['git', 'clone', 'https://github.com/fake-user/fake-repo.git'], env=env)

does python utilize PATH environment?

My python script is being called by kea-dhcp, which has capability of executing external scripts (https://kea.readthedocs.io/en/latest/arm/hooks.html#run-script-run-script-support-for-external-hook-scripts for details)
import subprocess
...
subprocess.check_output(('bridge', arg1, arg2 ...), stderr=subprocess.STDOUT)
Where does python take information about location of bridge binary? I'm getting an error:
FileNotFoundError: [Errno 2] No such file or directory: 'bridge': 'bridge'
I don't think it has anything to do with PYTHONPATH since it is not a problem of importing modules. What can be possibly wrong?
UPDATE
Following Charles Duffy's suggestion, I did:
res = subprocess.check_output([shutil.which('bridge'), '-j', 'fdb', 'show'], stderr=subprocess.STDOUT)
but got error (it points to above subprocess line):
File "/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/lib64/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/lib64/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/lib64/python3.6/subprocess.py", line 1278, in _execute_child
executable = os.fsencode(executable)
File "/lib64/python3.6/os.py", line 800, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
I can't see what is wrong with that, looks perfectly fine.

How unqualified names are handled by subprocess depends on which operating system you're on. Sometimes PATH is honored; never is PYTHONPATH used for the purpose.
To explicitly perform a PATH lookup, change your code to use shutil.which():
import shutil, subprocess
subprocess.check_output([shutil.which('bridge'), arg1, arg2], stderr=subprocess.STDOUT)
Quoting from https://docs.python.org/3/library/subprocess.html#popen-constructor (formatting from the original) --
Warning For maximum reliability, use a fully qualified path for the executable. To search for an unqualified name on PATH, use shutil.which(). On all platforms, passing sys.executable is the recommended way to launch the current Python interpreter again, and use the -m command-line format to launch an installed module.
Resolving the path of executable (or the first item of args) is platform dependent. For POSIX, see os.execvpe(), and note that when resolving or searching for the executable path, cwd overrides the current working directory and env can override the PATH environment variable. For Windows, see the documentation of the lpApplicationName and lpCommandLine parameters of WinAPI CreateProcess, and note that when resolving or searching for the executable path with shell=False, cwd does not override the current working directory and env cannot override the PATH environment variable. Using a full path avoids all of these variations.

The problem is that kea-dhcp does not export system environment variables to scripts it executes, it only supplies its specific environment. Discovered this via os.environ. The only solution to this is provide the full pass to bridge command (or any other launched in this context).

How to python subprocess.check_output linux command on mac os

I am trying to run a linux executable on Max OS X 10.11.6 via python2.7
I would like to use subprocess.check_output.
The command, which works via the terminal is:
mosel -c "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"
However, when I try:
subprocess.check_output(['mosel','-c',cmd])
where
cmd="exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"'
I get:
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 629, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'mosel'
I have been able to get it to "echo" the command to an output file, but I cannot run "which mosel" via python, which leads me to believe that it has to do with check_output using "bin/sh"as the executable.
So, do I need to use "Popen" instead and set
executable=path/to/mosel
?
If so, how do use Python to get the user's path to mosel (i.e. get the output of "which mosel")?
Thanks!
UPDATE:
PyCharm was not seeing the system paths, which I fixed using this answer:
Setting environment variables in OS X?
Now, it appears that
subprocess.check_output(['mosel','-c',cmd])
Is sending the square brackets to the command line, because it now returns:
dyld: Library not loaded: libxprm_mc.dylib
Referenced from: /usr/local/opt/xpress/bin/mosel
Reason: image not found
Traceback (most recent call last):
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 70, in <module>
run(1)
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 44, in run
out = check_output(['mosel', '-c', cmd])
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 219, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['mosel', '-c', cmd]' returned non-zero exit status -5
Or is there still a path issue?! (I can run mosel -c cmd via the mac terminal, but not in pycharm via python, nor in the mac terminal via python).

The problem is you're using check_output's arguments incorrectly. Unless you pass it shell=True, check_output expects a list of parameters as its input, in the form:
check_call(['binary_name','arg1','arg2','arg3', ....])
So in this case, you should do:
subprocess.check_call(['mosel', '-c', "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"])

The root of the issue turns out to be the DYLD_LIBRARY_PATH:
The new OS X release 10.11 "El Capitan" has a "security" feature that
prevents passing DYLD_LIBRARY_PATH to child processes. Somehow, that
variable is stripped from the environment. - (quoted from https://www.postgresql.org/message-id/20151103113612.GW11897#awork2.anarazel.de)
The security feature is called SIP or "System Integrity Protection". Unfortunately, it seems that no one has come up with a solution to this issue (other than work-arounds that must be tailored to each situation).
Here is another example of this issue:
https://github.com/soumith/cudnn.torch/issues/111
Google "mac os inherit dyld_library_path" and you will find many other examples.

No such file or directory with subprocess.Popen, not caused by string command

I'm running a webservice which operates on wav-files, but I don't want to allow just any upload, so I check the duration of the uploaded file first with the following code:
os.chdir("/home/me/bin")
proc = subprocess.Popen(['duration',wav],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
out, errors = proc.communicate()
time = int(float(out.strip()))
if time > MAX_TIME:
sys.exit(1)
This has been working fine for several months, but recently (after a migration, I should add) I get the following error:
Traceback (most recent call last):
File "/home/me/webservice.py", line 100, in <module>
proc = subprocess.Popen(['duration',wav],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib64/python2.6/subprocess.py", line 639, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
This error seems to be mostly caused by the use of a string for the command instead of a list, but that's not the case here. When I try to reproduce this error in a separate script, I can't.
Anyone have any idea what might be the cause here?

Commands without a path are always only looked up on the PATH. You cannot cd to a directory and have subprocess (or your shell for that matter) find a command located in that directory without specifying at least a relative path. This is standard command lookup behaviour.
If you want to run a command in the current working directory, you must specify a path of ./ to make it explicit the PATH search semantics are not to be used.
Relative paths in the PATH environment variable are also not supported everywhere; adding . to PATH is considered a security risk and some systems explicitly filter out . from the PATH. You must have migrated away from a system that allowed this, to a system that does disallow . in the PATH.

The error might not be pertaining to the executable you're feeding to Popen, but to an another executable that is subsequently called and is not found in the PATH of the new environment.
This exact situation happened to me, and the problem was that I had not installed the interpreter for the script I was calling. Frustrating to debug because the error message does not tell you which file it could not find.

Consider doing something like the following at the top of you file then there should be no need to chdir unless you have to for another reason.
import os
cmd = os.path.expanduser('~/bin/duration')
if not os.path.isfile(cmd):
raise IOError("command '%s' not found" % cmd)
proc = subprocess.Popen([cmd,wav ...

What does the 'shell' argument in subprocess mean on Windows?

The docs for the subprocess module state that 'If shell is True, the specified command will be executed through the shell'. What does this mean in practice, on a Windows OS?

It means that the command will be executed using the program specified in the COMSPEC environment variable. Usually cmd.exe.
To be exact, subprocess calls the CreateProcess windows api function, passing "cmd.exe /c " + args as the lpCommandLine argument.
If shell==False, the lpCommandLine argument to CreateProcess is simply args.

When you execute an external process, the command you want may look something like "foo arg1 arg2 arg3". If "foo" is an executable, that is what gets executed and given the arguments.
However, often it is the case that "foo" is actually a script of some sort, or maybe a command that is built-in to the shell and not an actual executable file on disk. In this case the system can't execute "foo" directly because, strictly speaking, these sorts of things aren't executable. They need some sort of "shell" to execute them. On *nix systems this shell is typically (but not necessarily) /bin/sh. On windows it will typically be cmd.exe (or whatever is stored in the COMSPEC environment variable).
This parameter lets you define what shell you wish to use to execute your command, for the relatively rare case when you don't want the default.

In addition to what was said in other answers, it is useful in practice if you want to open a file in the default viewer for that file type. For instance, if you want to open an HTML or PDF file, but will not know which browser or viewer is installed on the systems it will be run on, or have no guarantees as to the path to the executable, you can simply pass the file name as the only argument for the args field, then set shell=True. This will have Windows use whatever program is associated with that file type.
One caveat, if the path to your file has spaces, you need to surround it with two ".
eg.
path = "C:\\Documents and Settings\\Bob\\Desktop\\New Folder\\README.txt"
subprocess.call('""' + path + '""', shell = True)

In using-the-subprocess-module, there is an explicit paragraph:
The executable argument specifies the program to execute. It is very seldom needed: Usually, the program to execute is defined by the args argument. If shell=True, the executable argument specifies which shell to use. On Unix, the default shell is /bin/sh. On Windows, the default shell is specified by the COMSPEC environment variable.
Windows example - the shell (cmd.exe) command date -t will not be recognized without the shell:
>>> p=subprocess.Popen(["date", "/t"], stdout=subprocess.PIPE)
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
>>>
Using a shell, all is well:
>>> p=subprocess.Popen(["date", "/t"], shell=True, stdout=subprocess.PIPE)
>>> p.communicate()
('Wed 04/22/2009 \r\n', None)
>>>