Develop Reference

Subprocess can't find module when not run with shell=True (Python)

When I use Popen to start a subprocess, if shell=True isn't set I get the following error message:
Traceback (most recent call last):
File "C:\Users\Student01\PycharmProjects\yolov5-master\detect.py", line 37, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
I don't get this error message if shell=True.
I don't want to need to set shell=True because it causes problems when I then try to call communicate or stdout on the subprocess (It just runs the subprocess into the shell without executing anything bellow the communicate call).
Here is the code (In case it can help):
#!/usr/bin/env python
import subprocess
detectPath = "C:\\Users\\Student01\\PycharmProjects\\yolov5-master\\detect.py"
print("First Print Passed")
process = subprocess.Popen("python {} --source 0".format(detectPath), stdout=subprocess.PIPE, shell=False)
# output = str(process.communicate())
while process.poll() is None:
process.stdout.readline()
print("Poll is None")
I am using Pycharm
I tried adding the yolov5 project files into the included files of my pycharm project (since I'm working in a venv) but still get the error.
The only solution to the moduleNotFound error seems to be to set shell=True in the Popen but that creates other problems.

Launching a detached process with subprocess

I'm currently trying to understand the subprocess.run() api and to do this my goal is to create a detached subprocess that will source another python env, log it's python executable path and exit properly.
Right now, I have this main function that throws an NamedError. I'd really like to rely on the run api but I don't know how to pass the creationflags and even if it's the good way to achieve my goal.
Edit : the error is of course because I didn't import the constants. My question was mainly : how to create such a subprocess with subprocess.run() and without being able to use DETACHED_PROCESS nor CREATE_NEW_PROCESS_GROUP.
import subprocess
import logging
import shlex
import sys
# VENV is virtual env on the model /my/venv/path/bin
def main() -> None:
python_path = sys.executable
logger.debug("Base python executable : %(python_path)s")
tokens = shlex.split(
f"source {VENV} && python3 -c 'import sys;print(sys.executable)'"
)
proc = subprocess.run(
tokens,
shell=True,
text=True,
capture_output=True,
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP,
)
logger.debug(
"In the detached subprocess, the python exec is: %(proc.stdout)s"
)
The error :
Traceback (most recent call last):
File "test.py", line 37, in main
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP,
NameError: name 'DETACHED_PROCESS' is not defined

DETACHED_PROCESS is in the subprocess namespace and it is only there on Windows systems: https://github.com/python/cpython/blob/3.10/Lib/subprocess.py#L85
subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP

Using subprocess for compiling a c++ program from python script

I know there are some similar questions, here Invoking C compiler using Python subprocess command and subprocess, invoke C-program from within Python but I believe my question is in some sense different.
I need to compile a c++ program which uses some ROOT libraries so I need to add some flags and link some libraries for the compilation. Therefore my compilation line on the normal shell is:
> $($ROOTSYS/bin/root-config --cxx) $($ROOTSYS/bin/root-config --cflags --glibs) Analysis.cxx -o analysis.exe
which works nicely. I want to do this compilation from my python script. I have read the documentation for the subprocess module but I could not get a solution without using shell=True in the call of subprocess.Popen and I do not really undestand the difference. If I use:
process = Popen(["$($ROOTSYS/bin/root-config --cxx) $($ROOTSYS/bin/root-config --cflags --glibs) Analysis.cxx -o analysis.exe"], shell=True)
does the job. However, this:
process = Popen(["$($ROOTSYS/bin/root-config --cxx)", "$($ROOTSYS/bin/root-config --cflags --glibs)", "Analysis.cxx", "-o", "analysis.exe"])
I got the following:
Traceback (most recent call last):
File "make_posanalysis.py", line 45, in <module>
"Analysis.cxx", "-o", "analysis.exe"])
File "Python/2.7.15/x86_64-slc6-gcc62-opt/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "Python/2.7.15/x86_64-slc6-gcc62-opt/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I would like to understand the difference between using/not using shell=True since it seems to be the reason behind making the script work or not. Or, is there something else I am missing?

From the documentation:
If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
So it'e executing something equivalent to:
/bin/sh -c '$($ROOTSYS/bin/root-config --cxx)' '$($ROOTSYS/bin/root-config --cflags --glibs)' "Analysis.cxx", "-o", "analysis.exe"
This isn't what you want, because it only performs $(...) expansion in the first argument; everything else is taken literally, and become the positional arguments if the command in the first argument refers to $1, $2, etc.
If you want everything parsed by the shell, just give a single string.

How to python subprocess.check_output linux command on mac os

I am trying to run a linux executable on Max OS X 10.11.6 via python2.7
I would like to use subprocess.check_output.
The command, which works via the terminal is:
mosel -c "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"
However, when I try:
subprocess.check_output(['mosel','-c',cmd])
where
cmd="exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"'
I get:
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 629, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'mosel'
I have been able to get it to "echo" the command to an output file, but I cannot run "which mosel" via python, which leads me to believe that it has to do with check_output using "bin/sh"as the executable.
So, do I need to use "Popen" instead and set
executable=path/to/mosel
?
If so, how do use Python to get the user's path to mosel (i.e. get the output of "which mosel")?
Thanks!
UPDATE:
PyCharm was not seeing the system paths, which I fixed using this answer:
Setting environment variables in OS X?
Now, it appears that
subprocess.check_output(['mosel','-c',cmd])
Is sending the square brackets to the command line, because it now returns:
dyld: Library not loaded: libxprm_mc.dylib
Referenced from: /usr/local/opt/xpress/bin/mosel
Reason: image not found
Traceback (most recent call last):
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 70, in <module>
run(1)
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 44, in run
out = check_output(['mosel', '-c', cmd])
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 219, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['mosel', '-c', cmd]' returned non-zero exit status -5
Or is there still a path issue?! (I can run mosel -c cmd via the mac terminal, but not in pycharm via python, nor in the mac terminal via python).

The problem is you're using check_output's arguments incorrectly. Unless you pass it shell=True, check_output expects a list of parameters as its input, in the form:
check_call(['binary_name','arg1','arg2','arg3', ....])
So in this case, you should do:
subprocess.check_call(['mosel', '-c', "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"])

The root of the issue turns out to be the DYLD_LIBRARY_PATH:
The new OS X release 10.11 "El Capitan" has a "security" feature that
prevents passing DYLD_LIBRARY_PATH to child processes. Somehow, that
variable is stripped from the environment. - (quoted from https://www.postgresql.org/message-id/20151103113612.GW11897#awork2.anarazel.de)
The security feature is called SIP or "System Integrity Protection". Unfortunately, it seems that no one has come up with a solution to this issue (other than work-arounds that must be tailored to each situation).
Here is another example of this issue:
https://github.com/soumith/cudnn.torch/issues/111
Google "mac os inherit dyld_library_path" and you will find many other examples.

Why does Python subprocess.check_call fails to launch abc(1).bat? (bracket in filename)

I run into a problem with Python 2.7.6 on Windows 7 x64. Please help check with it.
I have abc.py
import subprocess
subprocess.check_call('abc(1).bat')
print 'done'
abc(1).bat has only one line:
ver
Running abc.py fails to launch abc(1).bat , with error message on CMD console:
'abc' is not recognized as an internal or external command,
operable program or batch file.
Traceback (most recent call last):
File "C:\test\abc.py", line 3, in <module>
subprocess.check_call('abc(1).bat')
File "C:\Python27\lib\subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'abc(1).bat' returned non-zero exit status 1
If I do not use brackets in my .bat file name, this problem does not occur. Can someone help explain this?

This is because even in the command line, running abc(1).bat returns an error. It needs to run as "abc(1).bat". If you fix your Python code to.
subprocess.check_call('"abc(1).bat"')
This works all fine.
The reason for this is stated in a Microsoft FAQ Question.