Develop Reference

Subprocess can't find module when not run with shell=True (Python)

When I use Popen to start a subprocess, if shell=True isn't set I get the following error message:
Traceback (most recent call last):
File "C:\Users\Student01\PycharmProjects\yolov5-master\detect.py", line 37, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
I don't get this error message if shell=True.
I don't want to need to set shell=True because it causes problems when I then try to call communicate or stdout on the subprocess (It just runs the subprocess into the shell without executing anything bellow the communicate call).
Here is the code (In case it can help):
#!/usr/bin/env python
import subprocess
detectPath = "C:\\Users\\Student01\\PycharmProjects\\yolov5-master\\detect.py"
print("First Print Passed")
process = subprocess.Popen("python {} --source 0".format(detectPath), stdout=subprocess.PIPE, shell=False)
# output = str(process.communicate())
while process.poll() is None:
process.stdout.readline()
print("Poll is None")
I am using Pycharm
I tried adding the yolov5 project files into the included files of my pycharm project (since I'm working in a venv) but still get the error.
The only solution to the moduleNotFound error seems to be to set shell=True in the Popen but that creates other problems.

Python subprocess error: Popen, call, run "No such file or directory: error

I'm having issues with the subprocess module. I'm trying to run a terminal command in Python, which works perfectly fine in the terminal. The command is:
hrun SomeAction LogFile
I've tried a variety of options, including call(), run(), check_output(), and Popen(). No matter which method I use, I get the error:
FileNotFoundError: [Errno 2] No such file or directory: 'hrun': 'hrun'
My code is:
output = Popen(["hrun", "SomeAction", log_file_name], stdout=PIPE, stderr=PIPE)
where "hrun" and "SomeAction" are strings and log_file_name is a string variable.
I found other SO issues and, most (if not all) were resolved with either shell=True (which I dont want), or because the issue was due to a string instead of a list argument.
Thanks!

If you are just trying to run a command from a prompt within a script why not use something like
import os
os.system("your command")
You should be able to just run it like
os.system("hrun SomeAction LogFile")

How to python subprocess.check_output linux command on mac os

I am trying to run a linux executable on Max OS X 10.11.6 via python2.7
I would like to use subprocess.check_output.
The command, which works via the terminal is:
mosel -c "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"
However, when I try:
subprocess.check_output(['mosel','-c',cmd])
where
cmd="exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"'
I get:
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 629, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'mosel'
I have been able to get it to "echo" the command to an output file, but I cannot run "which mosel" via python, which leads me to believe that it has to do with check_output using "bin/sh"as the executable.
So, do I need to use "Popen" instead and set
executable=path/to/mosel
?
If so, how do use Python to get the user's path to mosel (i.e. get the output of "which mosel")?
Thanks!
UPDATE:
PyCharm was not seeing the system paths, which I fixed using this answer:
Setting environment variables in OS X?
Now, it appears that
subprocess.check_output(['mosel','-c',cmd])
Is sending the square brackets to the command line, because it now returns:
dyld: Library not loaded: libxprm_mc.dylib
Referenced from: /usr/local/opt/xpress/bin/mosel
Reason: image not found
Traceback (most recent call last):
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 70, in <module>
run(1)
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 44, in run
out = check_output(['mosel', '-c', cmd])
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 219, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['mosel', '-c', cmd]' returned non-zero exit status -5
Or is there still a path issue?! (I can run mosel -c cmd via the mac terminal, but not in pycharm via python, nor in the mac terminal via python).

The problem is you're using check_output's arguments incorrectly. Unless you pass it shell=True, check_output expects a list of parameters as its input, in the form:
check_call(['binary_name','arg1','arg2','arg3', ....])
So in this case, you should do:
subprocess.check_call(['mosel', '-c', "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"])

The root of the issue turns out to be the DYLD_LIBRARY_PATH:
The new OS X release 10.11 "El Capitan" has a "security" feature that
prevents passing DYLD_LIBRARY_PATH to child processes. Somehow, that
variable is stripped from the environment. - (quoted from https://www.postgresql.org/message-id/20151103113612.GW11897#awork2.anarazel.de)
The security feature is called SIP or "System Integrity Protection". Unfortunately, it seems that no one has come up with a solution to this issue (other than work-arounds that must be tailored to each situation).
Here is another example of this issue:
https://github.com/soumith/cudnn.torch/issues/111
Google "mac os inherit dyld_library_path" and you will find many other examples.

os.system(<command>) execution through Python :: Limitations?

I'm writing a python (ver 2.7) script to automate the set of commands in this Getting Started example for INOTOOL.
Problem: When I run this entire script, I repeatedly encounter these errors:
Current Directory is not empty
No project is found in this directory
No project is found in this directory
But, when I run a first script only up till the code line marked, and manually type in the next three lines, or when I run these last three lines (starting from the "ino init -t blink" line) after manually accessing the beep folder, then I am able to successfully execute the same code.
Is there a limitation with os.system() that I'm encountering?
My code:
import os,sys
def upload()
os.system("cd /home/pi/Downloads")
os.system("mkdir beep")
os.system("cd beep") #will refer to this code junction in question description
os.system("ino init -t blink")
os.system("ino build")
os.system("ino upload")
sys.exit(0)

Yes, when os.system() commands are run for cd , it does not actually change the current directory for the python process' context. From documentation -
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.
So even though you are changing directory in os.system() call, the next os.system call still occurs in same directory. Which could be causing your issue.
You shoud try using os.chdir() to change the directory instead of os.system() calls.
The Best would be to use subprocess module as #PadraicCunningham explains in his answer.

You can use the subprocess module and os.mkdir to make the directory, you can pass the current working directory cwd to check_callso you actually execute the command in the directory:
from subprocess import check_call
import os
def upload():
d = "/home/pi/Downloads/beep"
os.mkdir(d)
check_call(["ino", "init", "-t", "blink"],cwd=d)
check_call(["ino", "build"],cwd=d)
check_call(["ino", "upload"],cwd=d)
A non-zero exit status will raise CalledProcessError which you may want to catch but once successful you know the commands all returned a 0 exit status.

starting new python subprocess containig module import with popen and current env giving ImportError

I'm trying to start a python program as a subprocess using the following code but the subprocess outputs ImportErrors to stderr.
The code
import subprocess
import sys
import os
environment = os.environ
command = ["python", "-u", "/test/my_python_program.py"]
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=environment)
for line in iter(p.stdout.readline, ''):
line = line.replace('\r', '').replace('\n', '')
print line
sys.stdout.flush()
The error
Traceback (most recent call last):
File "/test/my_python_program.py", line 31, in <module>
from PySide import QtGui, QtCore, QtNetwork
ImportError: dlopen(/Applications/Autodesk/maya2014/Maya.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python2.7/site-packages/PySide/QtGui.so, 2): Library not loaded: #executable_path/libpyside-python2.7.1.1.dylib
Referenced from: /Applications/Autodesk/maya2014/Maya.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python2.7/site-packages/PySide/QtGui.so
Reason: image not found
I'm running my code from a version of python embedded into another program (Autodesk Maya). The module's that cause the ImportError are importable from the main processes python interpreter. My assumption was that I could supply the main process's python's env to the subprocess to give it access to the modules that the main python instance had access too.
Does the approach make sense? and if so what am I doing wrong?
Edit1
I also tried to use the embedded version of python used in the popen call and the errors remained. If I remove the Import the embedded version of python runs just fine.
Edit 2
These are the environment variables that os.environ returns
MAYA_MODULE_PATH
AUTOLOADER_LAPS
LOGNAME
USER
SUBSTANCES_LOCATION
PATH
HOME
MAYA_SCRIPT_BASE
MENTALRAY_INCLUDE_LOCATION
SHELL
MAYA_LICENSE_METHOD
MAYA_LICENSE
QT_MAC_NO_NATIVE_MENUBAR
MAYA_SCRIPT_PATH
MAYA_REVERSE_FILEFORMAT_EXT
WF_IMF_CIN_WHITE_POINT
MAYA_LOCATION
PYTHONPATH
SSH_AUTH_SOCK
MENTALRAY_SHADERS_LOCATION
Apple_PubSub_Socket_Render
MAYA_PRESET_PATH
XBMLANGPATH
MAYA_RENDER_DESC_PATH
MAYA_SHADER_LIBRARY_PATH
MENTALRAY_LOCATION
TMPDIR
__KMP_REGISTERED_LIB_5123
MAYA_PLUG_IN_PATH
MAYA_APP_DIR
PYTHONHOME
MAYA_PLUG_IN_RESOURCE_PATH
__CF_USER_TEXT_ENCODING
IMF_PLUG_IN_PATH
__CHECKFIX1436934
WF_IMF_CIN_CORRECTION
Edit 3
Here's the contents of PYHOME
/Applications/Autodesk/maya2014/Maya.app/Contents/Frameworks/Python.framework/Versions/Current

The Python running inside Maya is not completely standard...
When run from within Maya, you might need to use the mayapy command to launch Python. This bootstraps Python according to Autodesk's requirements and should ensure the environment required to run Python is correct. (Although I think this may be what you are referring to in Edit1).
Have you also tried not passing through the environment to the popen command? I think the default behaviour of popen is to inherit the current process' environment which is what you want. I would expect the environment to be setting DYLD_LIBRARY_PATH and/or DYLD_FRAMEWORK_PATH at some point to tell it where Qt is.