Environment:
Windows: 10
Python: 2.7.13 and 3.8.1
default Python launcher: py -3
Default python: 2.7.13
> python -V
> Python 2.7.13
> py -3 -V
> Python 3.8.1
launcher.py:
import subprocess
subprocess.call(['python', '-V'])
Test 1: py -3 launcher.py
Output: python 3.8.1 (HOW!)
Test 2: py -2 launcher.py
Output: Python 2.7.13
The output should be Python 2.7.13 only, even the launcher runs under py 3!
note that, adding shell=True will work, but the idea is not to use it, and if I ran
subprocess.call(['python', 'script_under_py_2.py']) # Will run python 3 with script python 2!
Thanks
Adam
This behavior is due to the inconsistent handling of the PATH environment variable when calling Popen on Windows and Unix.
Windows creates its sub-processes using CreateProcess function. The search path for CreateProcess includes parent process directory, which may be the reason why you are having different binaries executed.
On Windows, PATH is only considered when shell=True is also passed.
You can learn more about the issue here:
subprocess PATH semantics and portability (notice the "needs patch" stage)
If the problem is that subprocess.call is failing to honour os.environ['PATH'] then you could always explicitly search that path for the executable you want to call, and then call it by its absolute path, as follows:
# Manually search the operating system's PATH for python[.exe]
import os, sys
ospath = os.environ['PATH'].split(os.path.pathsep)
_, extension = os.path.splitext(sys.executable)
target = 'python' + extension
matches = [os.path.join(directory, target) for directory in ospath]
matches = [match for match in matches if os.path.isfile(match)]
if not matches: raise OSError('{!r} was not found in any PATH directory'.format(target))
python = matches[0]
# Test:
import subprocess
print('I am Python {} and I am calling "{}" -V'.format(sys.version.split()[0], python))
subprocess.call([python, '-V'])
Of course, that won't do what you want if the py -3 wrapper works by actually changing the path before launching Python... I'm not familiar with that wrapper so I don't know whether that will be a problem.
Related
I have two python environments with different versions running in parallel. When I execute a python script (test2.py) from one python environment in the other python environment, I get the following error:
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
^
SyntaxError: invalid syntax
So my setup is this:
Python 3.7
(test.py)
│
│ Python 3.5.6
├───────────────────────────────┐
┆ │
┆ execute test2.py
┆ │
┆ 🗲 Error
How can I fix this?
For dm-script-people: How can I execute a module with a different python version in Digital Micrograph?
Details
I have two python files.
File 1 (test.py):
# execute in Digital Micrograph
import os
import subprocess
command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe',
os.path.join(os.getcwd(), 'test2.py')]
print(*command)
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("Subprocess result: '{}', '{}'".format(result.stdout.decode("utf-8"), result.stderr.decode("utf-8")))
and File 2 (test2.py)
# only executable in python 3.5.6
print("Hi")
in the same directory. test.py is executing test2.py with a different python version (python 3.5.6, legacy environment).
My python script (test.py) is running in the python interpreter in a third party program (Digital Micrograph). This program installs a miniconda python enviromnent called GMS_VENV_PYTHON (python version 3.7.x) which can be seen in the above traceback. The legacy miniconda environment is used only for running test2.py (from test.py) in python version 3.5.6.
When I run test.py from the command line (also in the conda GMS_VENV_PYTHON environment), I get the expected output from test2.py in test.py. When I run the exact same file in Digital Micrograph, I get the response
Subprocess result: '', 'Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
^
SyntaxError: invalid syntax
'
This tells me the following (I guess):
The test2.py is called since this is the error output of the subprocess call. So the subprocess.run() function seems to work fine
The paths are in the GMS_VENV_PYTHON environment which is wrong in this case. Since this is test2.py, they should be in the legacy paths
There is a SyntaxError because a f-string (Literal String Interpolation) is used which is introduced with python 3.6. So the executing python version is before 3.6. So the legacy python environment is used.
test2.py uses either use io nor abc (I don't know what to conclude here; are those modules loaded by default when executing python?)
So I guess this means, that the standard modules are loaded (I don't know why, probably because they are always loaded) from the wrong destination.
How can I fix this? (See What I've tried > PATH for more details)
What I've tried so far
Encoding
I came across this post "Fatal Python error: Py_Initialize: can't initialize sys standard streams LookupError: unknown encoding: 65001" telling me, that there might be problems with the encoding. I know that Digital Micrograph internally uses ISO 8859-1. I tried to use python -X utf8 and python -X utf8 (test2.py doesn't care about UTF-8, it is ASCII only) as shown below. But neither of them worked
command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe',
"-X", "utf8=0",
os.path.join(os.getcwd(), 'test2.py')]
PATH
As far as I can tell, I think this is the problem. The answer "https://stackoverflow.com/a/31877629/5934316" of the post "PyCharm: Py_Initialize: can't initialize sys standard streams" suggests to change the PYTHONPATH.
So to specify my question:
Is this the way to go?
How can I set the PYTHONPATH for only the subprocess (while executing python with other libraries in the main thread)?
Is there a better way to have two different python versions at the same time?
Thank you for your help.
Background
I am currently writing a program for handling an electron microscope. I need the "environment" (the graphical interface, the help tools but also hardware access) from Digital Micrograph. So there is no way around using it. And DigitalMicrograph does only support python 3.7.
On the other hand I need an external module which is only available for python 3.5.6. Also there is no way around using this module since it controlls other hardware.
Both rely on python C modules. Since they are compiled already, there is no possibility to check if they work with other versions. Also they are controlling highly sensitive aperatures where one does not want to change code. So in short words: I need two python versions parallel.
I was actually quite close. The problem is, that python imports invalid modules from a wrong location. In my case modules were imported from another python installation due to a wrong path. Modifying the PYTHONPATH according to "https://stackoverflow.com/a/4453495/5934316" works for my example.
import os
my_env = os.environ.copy()
my_env["PYTHONHOME"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy"
my_env["PYTHONPATH"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy;"
my_env["PATH"] = my_env["PATH"].replace("C:\\ProgramData\\Miniconda3\\envs\\GMS_VENV_PYTHON",
"C:\\ProgramData\\Miniconda3\\envs\\legacy")
command = ["C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe",
os.path.join(path, "test2.py")]
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)
For Digital Micrograph users: The python environment is saved in the global tags in "Private:Python:Python Path". So replace:
import DigitalMicrograph as DM
# ...
success, gms_venv = DM.GetPersistentTagGroup().GetTagAsString("Private:Python:Python Path")
if not success:
raise KeyError("Python path is not set.")
my_env["PATH"] = my_env["PATH"].replace(gms_venv, "C:\\ProgramData\\Miniconda3\\envs\\legacy")
I had set "PYTHONPATH" as "D:\ProgramData\Anaconda3" for my python (base) python environment before, but i found when I had changed to another env my python still import basic python package from "D:\ProgramData\Anaconda3",which means it get the wrong basic package with the wrong "System environment variables" config.
so I delete "PYTHONPATH" from my windows "System environment variables", and that will be fixed.
When I make a python venv on Windows:
C:\> mkdir C:\tvenv
C:\> cd C:\tvenv
C:\> python -m venv v
And then create these three files:
t.bat
call "C:\tvenv\v\Scripts\activate.bat"
python t1.py
t1.py
import subprocess
import sys
print('T1', sys.executable)
subprocess.run(['python', 't2.py'])
t2.py
import sys
print('T2', sys.executable)
And then I run t.bat:
C:\> t.bat
OBSERVED OUTPUT
T1 C:\tvenv\v\Scripts\python.exe
T2 C:\Program Files\Python38\python.exe
The following happens:
t.bat activates a venv and calls t1.py.
t1.py correctly reports the sys.executable from the venv
t1.py then calls subprocess.run(['python', 't2.py'])
t2.py then reports the system-wide sys.executable, not the one from the venv
ie I would have expected the output to be:
EXPECTED OUTPUT
T1 C:\tvenv\v\Scripts\python.exe
T2 C:\tvenv\v\Scripts\python.exe
as activate.bat sets:
set PATH=%VIRTUAL_ENV%\Scripts;%PATH%
It puts the venv Scripts dir at the front of the PATH.
so why doesn't subprocess.run(['python']) find the venv python instead of the system-wide one?
Update
I am on latest Windows 10 x64. I just completely reinstalled Python 3.9.1 from the standard python.org Windows installer, and didn't even put it in my PATH. Problem is still present.
The issue is indeed linked to Windows behavior and in fact the main solution is to use sys.exectuable instead of python to actually launch the expected python binary instead of letting the OS resolving that by himself.
Much more explanations on the issue and how OS are impacting the behavior of subprocess.run command: https://github.com/python/cpython/issues/86207
Suppose I have a file called a_python2_script.py with content
print "Hello"
and call it in Python 3 with:
from subprocess import Popen
Popen(["python2", "a_python2_script.py"])
This works, but I do not want to hardcode python2 interpreter call, since the program should run on Windows and Linux.
I know about sys.executable but that would give me the path of the current (i.e. Python 3) interpreter. What is the pythonic way to get the Python 2 interpreter path from within the Python 3 script?
Note: I am calling an external Python 2 library and have no chance to convert it to Python 3.
Use os.get_exec_path() (available from 3.2) to get a list of directories that will be searched for an executable.
In [1]: import os
In [2]: paths = os.get_exec_path()
In [5]: for p in paths:
...: for prog in ["python2", "python2.exe"]:
...: attempt = os.path.join(p, prog)
...: if os.path.isfile(attempt):
...: print(attempt)
...:
/usr/local/bin/python2
This might be one possible way to figure it out. I am currently running Windows and have Python installed using Anaconda and Anaconda3. My PATH contains both of these folders.
A simple script could check each PATH location for the presence of the Python executable. You could then execute python --version at each location:
import sys, os
from subprocess import Popen
python_paths = []
for path in os.environ["PATH"].split(os.pathsep):
for executable in ["python", "python.exe", "python2"]:
if os.path.isfile(os.path.join(path, executable)):
python_paths.append(path)
for path in python_paths:
Popen([os.path.join(path, "python"), "--version"])
print(python_paths)
On my Windows machine this gives:
['d:\\Anaconda', 'd:\\Anaconda3']
Python 2.7.9 :: Anaconda 2.2.0 (32-bit)
Python 3.4.3 :: Anaconda 2.3.0 (32-bit)
Tested using Python 3.4.3. I am not currently able to check the outcome for Linux though.
I'm new to Python and programming in general.
I'm trying to force my scripts to use Python3.4 as installed using the python installer from python.org.
My script has this.
#!/usr/local/bin/python3.4
import sys
print(sys.version)
print("Hello, World!")
Terminal returns this:
$ python pyscript.py
2.7.5 (default, Aug 25 2013, 00:04:04)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)]
Hello, World!
The shebang path is correct, according to "which python3.4"
By calling python on your script you are using the python in your path. As the comment suggests run the file directly after setting the execution bit.
Shebang is only used when you run your script directly using command like this
$ ./pyscript.py
When you python interpreter from command line, the shell doesn't consult shebang line. It simply runs the first python executable it finds on the command line.
To change the default python executable, adjust your path. Or better yet use python virtual environment. More information on virtual environment is here
Depending on your installation, python can be installed in a multitude of places on OS X.
Without any other additional changes, you are likely running the python located in /usr/bin/python. This can be verified by typing which python
~ $ which python
/usr/bin/python
Note that while /usr/local/bin/python3.4 may be the correct path to python 3.4, when you type python script.py you are not invoking the command you found when you did which python3.4.
To fix this you can do one of:
Invoke the script directly. Type script.py (with the executable bit) rather than as python script.py. You may need to change the permissions on the script to executable with chmod u+x script.py.
Change the version in /usr/bin/python to the version you want. Note that this may be very dangerous in that other things expecting the base install python of 2.7.5 can become very unhappy
Change your $PATH to put /usr/local/bin/ before /usr/bin and have python in /usr/local/bin/python point to the 3.4 version.
The shebang path is only used by the OS when you make the file executable and run it directly, like this:
chmod 755 pyscript.py
./pyscript.py
For python, the shebang is only a comment. The only way to force it even when calling the interpreter directly on the CLI is to compare the version and if it's less re-launch it using os.execv or similar. Something like this should do (not tested though):
#!/usr/local/bin/python3.4
import sys
import os
if (sys.hexversion < 0x3040000):
sys.argv.insert(0, '/usr/local/bin/python3.4')
os.execv(sys.argv[0], sys.argv)
print(sys.version)
print("Hello, World!")
You might want to use env on the shebang to avoid specifying the path, and avoid hardcoding the path on the python code too...
For example, in a python script file called script.py, in my case I have a 3.7 Python at the following path:
#! /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
import sys
print("--- Python version ---")
print(sys.version)
print("--- Python version info ---")
print(sys.version_info)
print("--- Python path executable ---")
print(sys.executable)
As others say, give permissions and do a proper execution so that shebang can be applied:
In the terminal type the following to make script.py a executable file for your user, that is:
$ chmod u+x script.py
In the terminal script.py is run with ./ at the beginning (not using python):
$ ./script.py
In my case I've got:
--- Python version ---
3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21)
[Clang 6.0 (clang-600.0.57)]
--- Python version info ---
sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)
--- Python path executable ---
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3
Considerations:
shebang path cannot contain spaces.
"Python path executable" not necessarily match with shebang path as long as the python at the shebang path can be an "alias". Aliases in Mac point to other file, you can see it with right click on a file and select "Get info" to check the "Original" value.
New to Python and programming in general. I want to "install" a module from the command line for v 2.6, but it looks like my default Python is 2.5. (python --version returns 2.5.4)
How can I run my python setup.py build/install on 2.6 instead?
Many thanks in advance,
Brock
You can use explicit paths:
c:\python26\python setup.py install
c:\python25\python setup.py install
Recent versions of Python install PyLauncher. It is installed in the path so no need to add an explicit Python to the path, and it allows easy switching between multiple Python versions.
Examples:
py -3 setup.py # run latest Python 3
py -2 setup.py # run latest Python 2
py -3.3
py -2.7-32 # use 32-bit version
py # run default version
The default version can be specified in the environment variable PY_PYTHON, e.g. PY_PYTHON=3 (latest Python 3).
It depends on your operating system. If you have python 2.6 installed, you need to change your environment path to point to the 2.6 executable rather than the 2.5 executable. Do a Google search for changing your PATH variable on your operating system.
If you're on Windows and you just need to run a different version of Python temporarily or, as was the case for me, a third party program needs to run a different version of Python, then modify your path at the command prompt:
> python --version
> set PATH=<path-to-desired-python-version>;%PATH%
> python --version
For me it was:
> python --version
Python 3.4.2
> set PATH=C:\tools\python2\;%PATH%
> python --version
Python 2.7.9
> npm install...
(success)
This allowed the third party program to install successfully. The PATH modification only affects programs running in the same command prompt session and only lasts as long as the command prompt session..
They are a couple of ways you can do this
1) python virtual environment
2) pylauncher
3) Changing your windows path variable, tedious to say the least
All three outlined in this video https://www.youtube.com/watch?v=ynDlb0n27cw
It sounds like you are on windows. If so, run this with the python you want, to set that python as the windows one. (not my code)
import sys
from _winreg import *
# tweak as necessary
version = sys.version[:3]
installpath = sys.prefix
regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version)
installkey = "InstallPath"
pythonkey = "PythonPath"
pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % (
installpath, installpath, installpath
)
def RegisterPy():
try:
reg = OpenKey(HKEY_LOCAL_MACHINE, regpath)
except EnvironmentError:
try:
reg = CreateKey(HKEY_LOCAL_MACHINE, regpath)
except Exception, e:
print "*** Unable to register: %s" % e
return
SetValue(reg, installkey, REG_SZ, installpath)
SetValue(reg, pythonkey, REG_SZ, pythonpath)
CloseKey(reg)
print "--- Python %s at %s is now registered!" % (version, installpath)
if __name__ == "__main__":
RegisterPy()
Download Python v2.6.