I'm currently trying to understand the subprocess.run() api and to do this my goal is to create a detached subprocess that will source another python env, log it's python executable path and exit properly.
Right now, I have this main function that throws an NamedError. I'd really like to rely on the run api but I don't know how to pass the creationflags and even if it's the good way to achieve my goal.
Edit : the error is of course because I didn't import the constants. My question was mainly : how to create such a subprocess with subprocess.run() and without being able to use DETACHED_PROCESS nor CREATE_NEW_PROCESS_GROUP.
import subprocess
import logging
import shlex
import sys
# VENV is virtual env on the model /my/venv/path/bin
def main() -> None:
python_path = sys.executable
logger.debug("Base python executable : %(python_path)s")
tokens = shlex.split(
f"source {VENV} && python3 -c 'import sys;print(sys.executable)'"
)
proc = subprocess.run(
tokens,
shell=True,
text=True,
capture_output=True,
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP,
)
logger.debug(
"In the detached subprocess, the python exec is: %(proc.stdout)s"
)
The error :
Traceback (most recent call last):
File "test.py", line 37, in main
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP,
NameError: name 'DETACHED_PROCESS' is not defined
DETACHED_PROCESS is in the subprocess namespace and it is only there on Windows systems: https://github.com/python/cpython/blob/3.10/Lib/subprocess.py#L85
subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP
Related
When I use Popen to start a subprocess, if shell=True isn't set I get the following error message:
Traceback (most recent call last):
File "C:\Users\Student01\PycharmProjects\yolov5-master\detect.py", line 37, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
I don't get this error message if shell=True.
I don't want to need to set shell=True because it causes problems when I then try to call communicate or stdout on the subprocess (It just runs the subprocess into the shell without executing anything bellow the communicate call).
Here is the code (In case it can help):
#!/usr/bin/env python
import subprocess
detectPath = "C:\\Users\\Student01\\PycharmProjects\\yolov5-master\\detect.py"
print("First Print Passed")
process = subprocess.Popen("python {} --source 0".format(detectPath), stdout=subprocess.PIPE, shell=False)
# output = str(process.communicate())
while process.poll() is None:
process.stdout.readline()
print("Poll is None")
I am using Pycharm
I tried adding the yolov5 project files into the included files of my pycharm project (since I'm working in a venv) but still get the error.
The only solution to the moduleNotFound error seems to be to set shell=True in the Popen but that creates other problems.
So here's an example of the terminal line I'm trying to run after importing the OS into a Python Script of mine:
user$ echo variable | thecommand
Even though OS imports have been working for me lately, the fact that the variable is in the MIDDLE of the imported OS command is not allowing my code to run:
#! /bin/python
import os
variable = 'thevariable'
os.system ("echo "+variable +" | thecommand")
the above is what I have tried in a few different syntax's with no success. Is there a way to accomplish what I'm looking to do using the os.system method?
Don't use os.system(). it is deprecated.
Instead try
import subprocess
variable = 'thevariable'
subprocess.call("echo "+variable +" | thecommand", shell=True)
the shell=True means that the command will be run in a bash process so that echo and the pipe would work.
I successfully ran this script in Python, but IPython notebook threw an exception.
What could be the problem?
import os
from subprocess import call, Popen, PIPE, STDOUT
command = "java -cp \"C:\mallet-2.0.7\class;C:\mallet-2.0.7\lib\mallet-deps.jar\" cc.mallet.fst.SimpleTagger --train true --model-file nouncrf train.txt"
p = Popen(command, stdout=PIPE, stderr=STDOUT, shell = True)
for line in p.stdout:
print line
Exception in thread "main" java.io.FileNotFoundException: train.txt (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileReader.<init>(Unknown Source)
at cc.mallet.fst.SimpleTagger.main(SimpleTagger.java:504)
Apparently the question is not related to Python at all, as it is a Java exception, not a Python one. Most probably, the file (train.txt) is looked for in a current directory. When you run your script in Python, the current directory may differ from what it is in IPython. Is the Java program able to accept absolute file path? If yes, you may specify the absolute path in command line for Java program.
I'm trying to start a python program as a subprocess using the following code but the subprocess outputs ImportErrors to stderr.
The code
import subprocess
import sys
import os
environment = os.environ
command = ["python", "-u", "/test/my_python_program.py"]
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=environment)
for line in iter(p.stdout.readline, ''):
line = line.replace('\r', '').replace('\n', '')
print line
sys.stdout.flush()
The error
Traceback (most recent call last):
File "/test/my_python_program.py", line 31, in <module>
from PySide import QtGui, QtCore, QtNetwork
ImportError: dlopen(/Applications/Autodesk/maya2014/Maya.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python2.7/site-packages/PySide/QtGui.so, 2): Library not loaded: #executable_path/libpyside-python2.7.1.1.dylib
Referenced from: /Applications/Autodesk/maya2014/Maya.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python2.7/site-packages/PySide/QtGui.so
Reason: image not found
I'm running my code from a version of python embedded into another program (Autodesk Maya). The module's that cause the ImportError are importable from the main processes python interpreter. My assumption was that I could supply the main process's python's env to the subprocess to give it access to the modules that the main python instance had access too.
Does the approach make sense? and if so what am I doing wrong?
Edit1
I also tried to use the embedded version of python used in the popen call and the errors remained. If I remove the Import the embedded version of python runs just fine.
Edit 2
These are the environment variables that os.environ returns
MAYA_MODULE_PATH
AUTOLOADER_LAPS
LOGNAME
USER
SUBSTANCES_LOCATION
PATH
HOME
MAYA_SCRIPT_BASE
MENTALRAY_INCLUDE_LOCATION
SHELL
MAYA_LICENSE_METHOD
MAYA_LICENSE
QT_MAC_NO_NATIVE_MENUBAR
MAYA_SCRIPT_PATH
MAYA_REVERSE_FILEFORMAT_EXT
WF_IMF_CIN_WHITE_POINT
MAYA_LOCATION
PYTHONPATH
SSH_AUTH_SOCK
MENTALRAY_SHADERS_LOCATION
Apple_PubSub_Socket_Render
MAYA_PRESET_PATH
XBMLANGPATH
MAYA_RENDER_DESC_PATH
MAYA_SHADER_LIBRARY_PATH
MENTALRAY_LOCATION
TMPDIR
__KMP_REGISTERED_LIB_5123
MAYA_PLUG_IN_PATH
MAYA_APP_DIR
PYTHONHOME
MAYA_PLUG_IN_RESOURCE_PATH
__CF_USER_TEXT_ENCODING
IMF_PLUG_IN_PATH
__CHECKFIX1436934
WF_IMF_CIN_CORRECTION
Edit 3
Here's the contents of PYHOME
/Applications/Autodesk/maya2014/Maya.app/Contents/Frameworks/Python.framework/Versions/Current
The Python running inside Maya is not completely standard...
When run from within Maya, you might need to use the mayapy command to launch Python. This bootstraps Python according to Autodesk's requirements and should ensure the environment required to run Python is correct. (Although I think this may be what you are referring to in Edit1).
Have you also tried not passing through the environment to the popen command? I think the default behaviour of popen is to inherit the current process' environment which is what you want. I would expect the environment to be setting DYLD_LIBRARY_PATH and/or DYLD_FRAMEWORK_PATH at some point to tell it where Qt is.
import shlex,subprocess,os
cmd = "/Applications/LibreOffice.app/Contents/MacOS/swriter --headless --invisible --convert-to pdf:writer_pdf_Export --outdir ~/Downloads ~/Downloads/HS303.xlsx"
#This works
os.popen(cmd)
#This doesnot work
subprocess.call(shlex.split(cmd))
Subprocess calls are not working. This was done in Mac OSX.
Any idea as to why this is happening ?
The problem
The problem is the ~/Downloads path. the ~ is expanded by the shell environment which wasn't enabled when you called subprocess.call. Below is a simplified demonstration of the problem:
>>> import shlex, subprocess, os, os.path
>>> cmd = "ls ~/Downloads"
>>> print os.popen(cmd).read()
ubuntu-11.04-desktop-i386.iso
ubuntu-11.04-server-i386.iso
>>> print subprocess.check_output(shlex.split(cmd))
ls: cannot access ~/Downloads: No such file or directory
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['ls', '~/Downloads']' returned non-zero exit status 2
The solutions
There are two solutions you could use, either expand the ~ in python using os.path.expanduser or call subprocess.call/subprocess.check_output with argument shell=True. I prefer to use check_output over call because it returns any output that might have been produced by the command. Either solution below should solve your problem.
import shlex, subprocess, os, os.path
cmd = 'ls ' + os.path.expanduser('~/Downloads')
print subprocess.check_output(shlex.split(cmd))
cmd = 'ls ~/Downloads'
print subprocess.check_output(cmd, shell=True)