Develop Reference

Fix "Fatal Python error: Py_Initialize: can't initialize sys standard streams"

I have two python environments with different versions running in parallel. When I execute a python script (test2.py) from one python environment in the other python environment, I get the following error:
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
^
SyntaxError: invalid syntax
So my setup is this:
Python 3.7
(test.py)
│
│ Python 3.5.6
├───────────────────────────────┐
┆ │
┆ execute test2.py
┆ │
┆ 🗲 Error
How can I fix this?
For dm-script-people: How can I execute a module with a different python version in Digital Micrograph?
Details
I have two python files.
File 1 (test.py):
# execute in Digital Micrograph
import os
import subprocess
command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe',
os.path.join(os.getcwd(), 'test2.py')]
print(*command)
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("Subprocess result: '{}', '{}'".format(result.stdout.decode("utf-8"), result.stderr.decode("utf-8")))
and File 2 (test2.py)
# only executable in python 3.5.6
print("Hi")
in the same directory. test.py is executing test2.py with a different python version (python 3.5.6, legacy environment).
My python script (test.py) is running in the python interpreter in a third party program (Digital Micrograph). This program installs a miniconda python enviromnent called GMS_VENV_PYTHON (python version 3.7.x) which can be seen in the above traceback. The legacy miniconda environment is used only for running test2.py (from test.py) in python version 3.5.6.
When I run test.py from the command line (also in the conda GMS_VENV_PYTHON environment), I get the expected output from test2.py in test.py. When I run the exact same file in Digital Micrograph, I get the response
Subprocess result: '', 'Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
^
SyntaxError: invalid syntax
'
This tells me the following (I guess):
The test2.py is called since this is the error output of the subprocess call. So the subprocess.run() function seems to work fine
The paths are in the GMS_VENV_PYTHON environment which is wrong in this case. Since this is test2.py, they should be in the legacy paths
There is a SyntaxError because a f-string (Literal String Interpolation) is used which is introduced with python 3.6. So the executing python version is before 3.6. So the legacy python environment is used.
test2.py uses either use io nor abc (I don't know what to conclude here; are those modules loaded by default when executing python?)
So I guess this means, that the standard modules are loaded (I don't know why, probably because they are always loaded) from the wrong destination.
How can I fix this? (See What I've tried > PATH for more details)
What I've tried so far
Encoding
I came across this post "Fatal Python error: Py_Initialize: can't initialize sys standard streams LookupError: unknown encoding: 65001" telling me, that there might be problems with the encoding. I know that Digital Micrograph internally uses ISO 8859-1. I tried to use python -X utf8 and python -X utf8 (test2.py doesn't care about UTF-8, it is ASCII only) as shown below. But neither of them worked
command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe',
"-X", "utf8=0",
os.path.join(os.getcwd(), 'test2.py')]
PATH
As far as I can tell, I think this is the problem. The answer "https://stackoverflow.com/a/31877629/5934316" of the post "PyCharm: Py_Initialize: can't initialize sys standard streams" suggests to change the PYTHONPATH.
So to specify my question:
Is this the way to go?
How can I set the PYTHONPATH for only the subprocess (while executing python with other libraries in the main thread)?
Is there a better way to have two different python versions at the same time?
Thank you for your help.
Background
I am currently writing a program for handling an electron microscope. I need the "environment" (the graphical interface, the help tools but also hardware access) from Digital Micrograph. So there is no way around using it. And DigitalMicrograph does only support python 3.7.
On the other hand I need an external module which is only available for python 3.5.6. Also there is no way around using this module since it controlls other hardware.
Both rely on python C modules. Since they are compiled already, there is no possibility to check if they work with other versions. Also they are controlling highly sensitive aperatures where one does not want to change code. So in short words: I need two python versions parallel.

I was actually quite close. The problem is, that python imports invalid modules from a wrong location. In my case modules were imported from another python installation due to a wrong path. Modifying the PYTHONPATH according to "https://stackoverflow.com/a/4453495/5934316" works for my example.
import os
my_env = os.environ.copy()
my_env["PYTHONHOME"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy"
my_env["PYTHONPATH"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy;"
my_env["PATH"] = my_env["PATH"].replace("C:\\ProgramData\\Miniconda3\\envs\\GMS_VENV_PYTHON",
"C:\\ProgramData\\Miniconda3\\envs\\legacy")
command = ["C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe",
os.path.join(path, "test2.py")]
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)
For Digital Micrograph users: The python environment is saved in the global tags in "Private:Python:Python Path". So replace:
import DigitalMicrograph as DM
# ...
success, gms_venv = DM.GetPersistentTagGroup().GetTagAsString("Private:Python:Python Path")
if not success:
raise KeyError("Python path is not set.")
my_env["PATH"] = my_env["PATH"].replace(gms_venv, "C:\\ProgramData\\Miniconda3\\envs\\legacy")

I had set "PYTHONPATH" as "D:\ProgramData\Anaconda3" for my python (base) python environment before, but i found when I had changed to another env my python still import basic python package from "D:\ProgramData\Anaconda3",which means it get the wrong basic package with the wrong "System environment variables" config.
so I delete "PYTHONPATH" from my windows "System environment variables", and that will be fixed.

Installing "scripts" in setup.py as part of a Python package, on user's path and recognized as Python scripts [duplicate]

This question already has answers here:
Getting Python error "from: can't read /var/mail/Bio"
(6 answers)
Closed 8 months ago.
I'm trying to have a Python script available on a user's path when they install my package from PyPI using pip:
pip install MyPackage
MyPackage is on PyPI and installs successfully--apparently--in a conda virtual environment. The setup.py file (excerpted) looks like this:
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
config = {
'name': 'MyPackage',
'version': '0.2.1.dev',
'install_requires': [...],
'packages': [
'MyPackage', 'MyPackage.utils'
],
'py_modules': [
'MyCLI',
],
'scripts': [
'MyPackage/MyCLI.py',
],
...
On GNU/Linux, when I type MyCLI and hit Tab, it successfully auto-completes to MyCLI.py. When I ask which MyCLI.py it shows me the fully qualified path to Python script in the virtual environment folder:
$ which MyCLI.py
/home/arthur/Applications/miniconda3/envs/MyPackage/bin/MyCLI.py
MyCLI.py uses fire to wrap a Python class, expose its methods at the command line, present docstrings as help documentation, and parse arguments. It looks like:
'''
My Command Line Interface
'''
class CLIRuntime(object):
def run(self):
do_something()
if __name__ == '__main__':
import fire
fire.Fire(CLIRuntime)
If I run this script with the Python interpreter, it executes correctly.
python $(which MyCLI.py)
The problem is, when I try to run it without specifying the Python interpreter, it seems to think it is a bash script or binary file and wrecks my terminal session:
$ MyCLI.py
/home/arthur/Applications/miniconda3/envs/MyPackage/bin/MyCLI.py: line 8:
My Command Line Interface
: No such file or directory
from: can't read /var/mail/__future__
How can I change setup.py so that this script is available on a user's path but also known as/ runs as a Python script?
I want to note that if I install this package from source using pip in editable mode (pip install -e .), MyCLI.py is on my path and runs correctly as a Python script. It just doesn't appear to work when installing from PyPI.

To tell your shell what program should be used to execute a script file, you need to add a "hash-bang" declaration as the first line in the script. For Python executing inside of a virtualenv, either
#!python
or
#!/usr/bin/env python
will do the trick. If you're using Python 3, use python3 instead.

How to (hermetically) include Python interpreter in Bazel to build Python library (sdist)

How can I (hermetically) include python as an (executable) input to my genrule?
Conceptually, I'm aware of the following approaches:
include a python interpreter in the source repo at third_party/python
have Bazel fetch python ala rules-go
There also seem to be a couple methods for doing so:
py_runtime
py_toolchain (doesn't seem to be available yet)
py_distribution_package (doesn't seem to be available yet)
genrules.tools
genrules.toolchains
Example:
I have a python library:
myproject
- setup.py
- mylibrary
- __init__.py
- myfunction.py
I'd like to produce a pip install-able distribution with Bazel so that I can do:
pip install <path/to/distribution>
python
>>> from mylibrary.myfunction import helloworld
>>> helloworld()
I added a (empty) WORKSPACE file at myproject/WORKSPACE and followed the example in this medium article:
# myproject/BUILD
genrule(
name = "mylibrary_sdist",
srcs = glob(["**/*.py"]),
outs = ["mylibrary.tar.gz"],
cmd = "python setup.py sdist && mv dist/mylibrary*.tar.gz $(location mylibrary.tar.gz)"
)
This works (ie. produces bazel-genfiles/mylibrary.tar.gz), but I'm shelling out to python.
How can I (hermetically) include python as an (executable) input to my genrule?

Unless I'm misunderstanding your question, this should just be a matter of passing the actual path to your target python executable. I would check the python executable into the third_party dir and instead of invoking your genrule using simply python I'd pass the relative path to that executable relative to WORKSPACE.

Calling python compiled files in Python 3

I have multiple Python versions installed (2.7 and 3.4)
I want to run a .pyc with specified version of Python
#! C:\python34\python
import sys
print("Hello",sys.version.split()[0])
input()
This sheebang works fine on Windows because I use pylauncher
So I can compile like that
c:\python34\python -m compileall print.py -b
But the sheebang is not recognized when I execute the pyc file.
This works, but I wouldn't like to repeat the C:\python34\python
Because the current script will be already running under the Python version I asked in the shebang.
Therefore I would like to make the sub program start with the same version of the Python.
So far, I tried:
#! C:\python34\python
import os
os.system("C:\python34\python print.pyc")
This would be perfect, but doesn't like pyc files. And the following doesn't works either:
exec( open('print.pyc').read() )
Does someone knows how to call the pyc files in the code?

#! C:\python34\python
import print # imports print.pyc
#now you can use the pyc as a module.
DoSomething()

How can I install Python modules programmatically / through a Python script? [duplicate]

This question already has answers here:
How can I Install a Python module within code?
(12 answers)
Closed 2 years ago.
Can I download and install Python modules from PyPi strictly inside a script, without using a shell at all?
I use a non-standard Python environment, Autodesk Maya's Python interpreter. This does not come with "easy_install," and there is no "shell," only a python script interpreter invoked by the main Maya executable. Copying and pasting ez_setup.py's contents into the script editor window and running it correctly installs an easy_install somewhere into Maya's directory, but the script incorrectly records the Python interpreter as "...maya.exe" instead of "...mayapy.exe" Furthermore, using easy_install requires a shell.
The objective is to deliver a Python script that, e.g., installs NumPy into the Maya Python system. This could be accomplished by dropping eggs into the site-packages directory, but that requires manual user intervention. Anything an end user has to do outside the Maya environment is essentially untouchable, especially messing with the file system. But messing with the filesystem through a script? That's fine.
Is there something more elegant than ez_setup.py + editing the resulting easy_install...py's + subprocess calls? I feel like this is a basic feature. I see documentation online for programmatic module installation through pip... but pip needs to be installed first!
What is the most elegant way to install a module strictly within the confines of a script?

Installing easy_install for Maya on windows.
Download ez_setup.py.
open windows cmd elevated (start, type cmd, rmb click on it ->run as administrator)
change the cmd directory to x:\maya install dir\bin
example: cd c:\Program Files\MayaXX\bin
execute following command mayapy x:\WhereYouSaved\ez_setup.py
Now easy install should be set up properly. You may want to still do following steps:
cd x:\maya install dir\python\scripts
rename all files in this folder to start with ma
example: for %i in (*) do ren %i ma%i
add this folder to your path
hit win+e
rmb my computer and choose properties
Advanced system settings -> Environment variables
search variable path edit it and append ;x:\maya install dir\python\scripts
Now you can call maeasy_install pythonModule from cmd for installing stuff. Also you can call following inside Maya to install modules:
from setuptools.command import easy_install
easy_install.main( ["pythonModule"] )
NOTE: If Maya is installed in program files then you can not really install stuff without elevating. Unless you change disk permissions to the Maya python directory.

#!/usr/bin/env python
from __future__ import print_function
REQUIREMENTS = [ 'distribute', 'version', 'Cython', 'sortedcollection' ]
try:
from setuptools import find_packages
from distutils.core import setup
from Cython.Distutils import build_ext as cython_build
import sortedcollection
except:
import os, pip
pip_args = [ '-vvv' ]
proxy = os.environ['http_proxy']
if proxy:
pip_args.append('--proxy')
pip_args.append(proxy)
pip_args.append('install')
for req in REQUIREMENTS:
pip_args.append( req )
print('Installing requirements: ' + str(REQUIREMENTS))
pip.main(initial_args = pip_args)
# do it again
from setuptools import find_packages
from distutils.core import setup
from Cython.Distutils import build_ext as cython_build
import sortedcollection

To make it work, open the ez_setup.py file and simply add an s after http at this line:
DEFAULT_URL = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
so that it becomes
DEFAULT_URL = "https://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]