Develop Reference

subprocess.run failing to execute git command with var=value prefix but os.system does it without issue

I'm running into a bit of a weird issue here.
I'm trying to clone some GitHub repositories automatically but I'm seeing subprocess.run error out in cases where os.system, or just running the command directly in my shell, works fine.
This is the command I'm trying to run:
subprocess.run('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git'.split())
Which leads to this error:
>>> subprocess.run('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git'.split())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 546, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 1022, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Users/omermikhailk/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py", line 1899, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'GIT_TERMINAL_PROMPT=0'
When the result should be what os.system gives:
>>> os.system('GIT_TERMINAL_PROMPT=0 git clone https://github.com/fake-user/fake-repo.git')
Cloning into 'fake-repo'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
32768
Would anyone happen to know the reason behind this?

You need to use the following instead:
subprocess.run(['git', 'clone', 'https://github.com/fake-user/fake-repo.git'], env={'GIT_TERMINAL_PROMPT': '0'})
When you run a command via subprocess, you are not using bash or sh or any shell, so environment variables are not interpreted, and must be specified via env, not the CLI.
Note that it is the best practice to explicitly craft the list for the command and its arguments yourself, rather than using a string and .split(). If you have any arguments that contain spaces, rather than including literal quotes in the string that you pass as a list element, you would use a string that contains the full argument. I.e., NOT ['echo', '"foo', 'bar"'] (which would be the result of .split()), but ['echo', 'foo bar'] to include the space.
Another note as to why this occurs is that subprocess.run does technically allow you to pass shell=True, but you should avoid doing so at all costs due to security implications if you are working with any content that may be user-generated. The best practice is to use env={...} as suggested.
If you need to include current environment variables, then you can use the following:
from os import environ
env = environ.copy()
env['GIT_TERMINAL_PROMPT'] = '0'
subprocess.run(['git', 'clone', 'https://github.com/fake-user/fake-repo.git'], env=env)

Run Powershell script from Python?

At the end of some processing in Python in Windows I am trying to eject an USB SD card.
Researching here it seems there are two ways in which to do it; call a PowerShell program or run PowerShell within Python.
Can anyone offer me any guidance. Please keep it simple; learning Python is my new year project.
So I have written a PowerShell script (ejectusb.ps1) which works perfectly:
$drive = New-Object -ComObject Shell.Application
$drive.Namespace(17).Parsename("J:").InvokeVerb("Eject")
I then call it from Python using subprocess:
subprocess.run(["E:\Data\Computing\Software\MicroSoft\Programming\Powershell\ejectusb.ps1"])
The SD card is not ejected and I get the error messages:
Traceback (most recent call last):
File "E:/Data/Computing/Software/Scripts/SdCardPlayEachVideo06.py", line 91, in <module>
subprocess.run(["E:\Data\Computing\Software\MicroSoft\Programming\Powershell\ejectusb.ps1"])
File "C:\Users\David\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\David\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1024, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\David\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1493, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 193] %1 is not a valid Win32 application
I don't understand this error message.
So I tried running PowerShell inside Python using:
os.system('powershell $driveEject = New-Object -comObject Shell.Application;
$driveEject.Namespace(17).ParseName("J:").InvokeVerb("Eject")')
An empty PowerShell screen and also what looks like a Windows command screen briefly flash up, but the SD card is not ejected. No error messages.
Can anyone offer me any guidance. Please keep it simple; learning Python is my new year project.

Hamed's solution uses the argument 'powershell' to launch PowerShell. i.e
subprocess.run(["powershell", "-File", "E:\Data\Computing\Software\MicroSoft\Programming\Powershell\ejectusb.ps1"])
I have been using the full path to PowerShell as the argument, i.e
subprocess.run(["C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe", etc
This path is correct (if I type it into the address bar of Windows File Explorer it launches PowerShell. But is causes a 'file not found' error.
So I dont know what the problem is with the full path but I am grateful for Hamed's workaround.

does python utilize PATH environment?

My python script is being called by kea-dhcp, which has capability of executing external scripts (https://kea.readthedocs.io/en/latest/arm/hooks.html#run-script-run-script-support-for-external-hook-scripts for details)
import subprocess
...
subprocess.check_output(('bridge', arg1, arg2 ...), stderr=subprocess.STDOUT)
Where does python take information about location of bridge binary? I'm getting an error:
FileNotFoundError: [Errno 2] No such file or directory: 'bridge': 'bridge'
I don't think it has anything to do with PYTHONPATH since it is not a problem of importing modules. What can be possibly wrong?
UPDATE
Following Charles Duffy's suggestion, I did:
res = subprocess.check_output([shutil.which('bridge'), '-j', 'fdb', 'show'], stderr=subprocess.STDOUT)
but got error (it points to above subprocess line):
File "/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/lib64/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/lib64/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/lib64/python3.6/subprocess.py", line 1278, in _execute_child
executable = os.fsencode(executable)
File "/lib64/python3.6/os.py", line 800, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
I can't see what is wrong with that, looks perfectly fine.

How unqualified names are handled by subprocess depends on which operating system you're on. Sometimes PATH is honored; never is PYTHONPATH used for the purpose.
To explicitly perform a PATH lookup, change your code to use shutil.which():
import shutil, subprocess
subprocess.check_output([shutil.which('bridge'), arg1, arg2], stderr=subprocess.STDOUT)
Quoting from https://docs.python.org/3/library/subprocess.html#popen-constructor (formatting from the original) --
Warning For maximum reliability, use a fully qualified path for the executable. To search for an unqualified name on PATH, use shutil.which(). On all platforms, passing sys.executable is the recommended way to launch the current Python interpreter again, and use the -m command-line format to launch an installed module.
Resolving the path of executable (or the first item of args) is platform dependent. For POSIX, see os.execvpe(), and note that when resolving or searching for the executable path, cwd overrides the current working directory and env can override the PATH environment variable. For Windows, see the documentation of the lpApplicationName and lpCommandLine parameters of WinAPI CreateProcess, and note that when resolving or searching for the executable path with shell=False, cwd does not override the current working directory and env cannot override the PATH environment variable. Using a full path avoids all of these variations.

The problem is that kea-dhcp does not export system environment variables to scripts it executes, it only supplies its specific environment. Discovered this via os.environ. The only solution to this is provide the full pass to bridge command (or any other launched in this context).

"Error The command was not found ...." using wexpect python module on Windows [duplicate]

This question already has answers here:
Running wexpect on windows
(2 answers)
Closed 3 years ago.
So I'm trying to interact with a .exe file (tcpclient.exe) and pexpect docs show that it is perfect since I need to send multiple inputs and interact with the program.
But I cant even get the following code to work
import wexpect
child = wexpect.spawn('dir')
print child
I'm assuming any variable can be print()ed provided the function was valid.
The following is the error message
Traceback (most recent call last):
File "test1.py", line 13, in <module>
child = wexpect.spawn('dir')
File "C:\pytest\wexpect.py", line 279, in spawn
return spawn_windows(command, args, timeout, maxread, searchwindowsize, logfile, cwd, env)
File "C:\pytest\wexpect.py", line 1653, in __init__
self._spawn (command, args)
File "C:\pytest\wexpect.py", line 1698, in _spawn
raise ExceptionPexpect ('The command was not found or was not executable: %s.' % self.command)
ExceptionPexpect: The command was not found or was not executable: dir.
I am using windowsXP I have MinGW installed and Python2.7
I am using wexpect through the pywin32 package
pywin32-217.win32-py2.7.exe
that was the installation .exe name.
My Python path is set as C:\Python27\
I tried setting it to C:\Python27\bin as someone mentioned but then I couldnt execute python after that.
I looked into the source code for wexpect and there is a function "which()" which is returning "None" to its input "dir"
I was unable to modify it to do otherwise.
Please tell me what I did wrong.
Thanks in Advance.

'dir' is an internal command of cmd.exe, and so wexpect can't find it.
You need an absolute path to an executable, and a list of arguments.
In this case, use cmd to execute dir.
>>> import wexpect
>>> child = wexpect.spawn('C:\windows\system32\cmd.exe',['/c','dir'])
>>> print child
<wexpect.spawn_windows object at 0x024926B0> version: 2.3 ($Revision: 399 $)
command: C:\windows\system32\cmd.exe
args: ['C:\\windows\\system32\\cmd.exe', '/c', 'dir']
---snip

No such file or directory with subprocess.Popen, not caused by string command

I'm running a webservice which operates on wav-files, but I don't want to allow just any upload, so I check the duration of the uploaded file first with the following code:
os.chdir("/home/me/bin")
proc = subprocess.Popen(['duration',wav],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
out, errors = proc.communicate()
time = int(float(out.strip()))
if time > MAX_TIME:
sys.exit(1)
This has been working fine for several months, but recently (after a migration, I should add) I get the following error:
Traceback (most recent call last):
File "/home/me/webservice.py", line 100, in <module>
proc = subprocess.Popen(['duration',wav],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib64/python2.6/subprocess.py", line 639, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
This error seems to be mostly caused by the use of a string for the command instead of a list, but that's not the case here. When I try to reproduce this error in a separate script, I can't.
Anyone have any idea what might be the cause here?

Commands without a path are always only looked up on the PATH. You cannot cd to a directory and have subprocess (or your shell for that matter) find a command located in that directory without specifying at least a relative path. This is standard command lookup behaviour.
If you want to run a command in the current working directory, you must specify a path of ./ to make it explicit the PATH search semantics are not to be used.
Relative paths in the PATH environment variable are also not supported everywhere; adding . to PATH is considered a security risk and some systems explicitly filter out . from the PATH. You must have migrated away from a system that allowed this, to a system that does disallow . in the PATH.

The error might not be pertaining to the executable you're feeding to Popen, but to an another executable that is subsequently called and is not found in the PATH of the new environment.
This exact situation happened to me, and the problem was that I had not installed the interpreter for the script I was calling. Frustrating to debug because the error message does not tell you which file it could not find.

Consider doing something like the following at the top of you file then there should be no need to chdir unless you have to for another reason.
import os
cmd = os.path.expanduser('~/bin/duration')
if not os.path.isfile(cmd):
raise IOError("command '%s' not found" % cmd)
proc = subprocess.Popen([cmd,wav ...