Develop Reference

Fix "Fatal Python error: Py_Initialize: can't initialize sys standard streams"

I have two python environments with different versions running in parallel. When I execute a python script (test2.py) from one python environment in the other python environment, I get the following error:
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
^
SyntaxError: invalid syntax
So my setup is this:
Python 3.7
(test.py)
│
│ Python 3.5.6
├───────────────────────────────┐
┆ │
┆ execute test2.py
┆ │
┆ 🗲 Error
How can I fix this?
For dm-script-people: How can I execute a module with a different python version in Digital Micrograph?
Details
I have two python files.
File 1 (test.py):
# execute in Digital Micrograph
import os
import subprocess
command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe',
os.path.join(os.getcwd(), 'test2.py')]
print(*command)
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("Subprocess result: '{}', '{}'".format(result.stdout.decode("utf-8"), result.stderr.decode("utf-8")))
and File 2 (test2.py)
# only executable in python 3.5.6
print("Hi")
in the same directory. test.py is executing test2.py with a different python version (python 3.5.6, legacy environment).
My python script (test.py) is running in the python interpreter in a third party program (Digital Micrograph). This program installs a miniconda python enviromnent called GMS_VENV_PYTHON (python version 3.7.x) which can be seen in the above traceback. The legacy miniconda environment is used only for running test2.py (from test.py) in python version 3.5.6.
When I run test.py from the command line (also in the conda GMS_VENV_PYTHON environment), I get the expected output from test2.py in test.py. When I run the exact same file in Digital Micrograph, I get the response
Subprocess result: '', 'Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
^
SyntaxError: invalid syntax
'
This tells me the following (I guess):
The test2.py is called since this is the error output of the subprocess call. So the subprocess.run() function seems to work fine
The paths are in the GMS_VENV_PYTHON environment which is wrong in this case. Since this is test2.py, they should be in the legacy paths
There is a SyntaxError because a f-string (Literal String Interpolation) is used which is introduced with python 3.6. So the executing python version is before 3.6. So the legacy python environment is used.
test2.py uses either use io nor abc (I don't know what to conclude here; are those modules loaded by default when executing python?)
So I guess this means, that the standard modules are loaded (I don't know why, probably because they are always loaded) from the wrong destination.
How can I fix this? (See What I've tried > PATH for more details)
What I've tried so far
Encoding
I came across this post "Fatal Python error: Py_Initialize: can't initialize sys standard streams LookupError: unknown encoding: 65001" telling me, that there might be problems with the encoding. I know that Digital Micrograph internally uses ISO 8859-1. I tried to use python -X utf8 and python -X utf8 (test2.py doesn't care about UTF-8, it is ASCII only) as shown below. But neither of them worked
command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe',
"-X", "utf8=0",
os.path.join(os.getcwd(), 'test2.py')]
PATH
As far as I can tell, I think this is the problem. The answer "https://stackoverflow.com/a/31877629/5934316" of the post "PyCharm: Py_Initialize: can't initialize sys standard streams" suggests to change the PYTHONPATH.
So to specify my question:
Is this the way to go?
How can I set the PYTHONPATH for only the subprocess (while executing python with other libraries in the main thread)?
Is there a better way to have two different python versions at the same time?
Thank you for your help.
Background
I am currently writing a program for handling an electron microscope. I need the "environment" (the graphical interface, the help tools but also hardware access) from Digital Micrograph. So there is no way around using it. And DigitalMicrograph does only support python 3.7.
On the other hand I need an external module which is only available for python 3.5.6. Also there is no way around using this module since it controlls other hardware.
Both rely on python C modules. Since they are compiled already, there is no possibility to check if they work with other versions. Also they are controlling highly sensitive aperatures where one does not want to change code. So in short words: I need two python versions parallel.

I was actually quite close. The problem is, that python imports invalid modules from a wrong location. In my case modules were imported from another python installation due to a wrong path. Modifying the PYTHONPATH according to "https://stackoverflow.com/a/4453495/5934316" works for my example.
import os
my_env = os.environ.copy()
my_env["PYTHONHOME"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy"
my_env["PYTHONPATH"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy;"
my_env["PATH"] = my_env["PATH"].replace("C:\\ProgramData\\Miniconda3\\envs\\GMS_VENV_PYTHON",
"C:\\ProgramData\\Miniconda3\\envs\\legacy")
command = ["C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe",
os.path.join(path, "test2.py")]
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)
For Digital Micrograph users: The python environment is saved in the global tags in "Private:Python:Python Path". So replace:
import DigitalMicrograph as DM
# ...
success, gms_venv = DM.GetPersistentTagGroup().GetTagAsString("Private:Python:Python Path")
if not success:
raise KeyError("Python path is not set.")
my_env["PATH"] = my_env["PATH"].replace(gms_venv, "C:\\ProgramData\\Miniconda3\\envs\\legacy")

I had set "PYTHONPATH" as "D:\ProgramData\Anaconda3" for my python (base) python environment before, but i found when I had changed to another env my python still import basic python package from "D:\ProgramData\Anaconda3",which means it get the wrong basic package with the wrong "System environment variables" config.
so I delete "PYTHONPATH" from my windows "System environment variables", and that will be fixed.

Running SikuliX 1.1.4 Python scripts from Command Line

I have some Python (jython actually) scripts that run with Sikulix.
I was previously using version 1.1.1 and was using the command line (after doinmg the setup):
java -Dsikuli.Debug=-2 -cp sikulix.jar org.python.util.jython main.py
With version 1.1.4, there is no more setup, and jython has been removed from sikulix.jar and sikulixapi.jar. Jython is in another jar file (jython-standalone-2.7.1.jar).
I tried to run with following command line
java -Dsikuli.Debug=-2 -cp "sikulix.jar;jython-standalone-2.7.1.jar" org.python.util.jython main.py
But I get the following error
Traceback (most recent call last):
File "test.py", line 3, in <module>
from sikuli.Sikuli import *
ImportError: No module named sikuli
The documentation is not fully updated on how to do it. They mention installation of jython, jip and other stuff but nothing managed to work.
Any idea on how to do it?
Thanks
==PS==:
After doing the following it almost worked:
Installing jython
Setting CLASSPATH to absolute path of sikulixapi.jar
Running jython main.py
I got the following error:
[error] RunTimeINIT: *** terminating: Java arch not 64 Bit or not detected (java 8-32 version 1.8 vm 25.121-b13 class 52.0 arch null)
I have installed Jython with a 32-bits Java and it seems 64-bits Java is required.
I will probably try again with 64-bits Java JDK.

Add the following line at the beginning of your script
import org.sikuli.script.SikulixForJython
This will help to look for sikuli module in Java classes.
You can then run with the command line mentionned previously:
java -cp "sikulixapi.jar;jython-standalone-2.7.1.jar" org.python.util.jython main.py
Example of Python script (main.py):
import org.sikuli.script.SikulixForJython
from sikuli.Sikuli import *
notepad = App('notepad.exe')
notepad.open()
sleep(1)
type("It is working!")
notepad.close()

Executing a python script from scala giving error while running the same script from the command line does NOT

This is my python script:
!/usr/bin/python
import sys
import magic
m=magic.from_file('<file with absolute path goes here>')
print(m)
On running this from the command line:
$ python script.py
Microsoft Word 2007+
results in the output of the TYPE of the document that file is. Using python-magic.
Now running the same script from scala using below code:
import sys.process._
def compWithLibmagic(){
val result = "python /script.py" !
}
throws the below error:
Traceback (most recent call last):
File "script.py", line 3, in <module>
import magic
ImportError: No module named magic
PS: I have both python 2.7 and python 3.6 installed on my machine and running the script using any of them from the command line runs just fine so I guess both of them are bundled with the MAGIC packages correctly.
Would highly appreciate any kind of help.

I will try to answer my own question in the best possible way.
While trying to execute the script.py from the command line using python compiler python2.7 was being used and when I tried to invoke the same script.py from the scala compiler using
import sys.process._
def compWithLibmagic(){
val result = "python /script.py" !
}
it was using python3 which was getting unnoticed. It took me a couple of days to actually figure this thing out and finally I removed all the python2.7 and python3 libraries and installed python 2.7 again and it worked like a charm.

It works from both python3 command and Scala code for me.
This should work.
import sys.process._
def compWithLibmagic(){
val result = "python3 script.py".!!
}

Unable to run python script during Jenkins build

The problem:
I have a Jenkins build step that executes a shell script. This script in turn calls a python script which is performing some cryptographic functions. However when the build executes, I am getting the following error.
Traceback (most recent call last):
File "./xyz.py", line 4, in <module>
import rsa
ImportError: No module named 'rsa'
The Jenkins' node has two versions of python - 2.7 (default) and 3.4 and rsa is installed for both of them. I even ran the script (using version 3.4) on the slave itself and it worked fine.
What I have done so far:
I am using EnvInject plugin to point PYTHONPATH to correct location. Without it, I found that PYTHONPATH was undefined.
With Python 2.7
Using default version, my script starts with: #!/usr/bin/env python
Jenkins output:
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
PYTHONPATH=/usr/local/lib/python2.7
[EnvInject] - Variables injected successfully.
[demo] $ /bin/sh -xe /tmp/hudson9217742060700174209.sh
+ export PYTHONPATH=/jenkins/workspace/demo:/usr/local/lib/python2.7
+ echo /jenkins/workspace/demo:/usr/local/lib/python2.7
/jenkins/workspace/demo:/usr/local/lib/python2.7
+ ./abc.sh
/usr/bin/env: python: No such file or directory
With Python 3.4
The Shebang in this case is #!/usr/bin/env python3
Jenkins output:
[EnvInject] - Executing scripts and injecting environment variables after the SCM step.
[EnvInject] - Injecting as environment variables the properties content
PYTHONPATH=/usr/local/lib/python3.4/
[EnvInject] - Variables injected successfully.
[demo] $ /bin/sh -xe /tmp/hudson4592372533933414288.sh
+ export PYTHONPATH=/jenkins/workspace/demo:/usr/local/lib/python3.4/
+ echo /jenkins/workspace/demo:/usr/local/lib/python3.4/
/jenkins/workspace/demo:/usr/local/lib/python3.4/
+ ./abc.sh
Traceback (most recent call last):
File "./xyz.py", line 4, in <module>
import rsa
ImportError: No module named 'rsa'
I even tried doing sys.path.append(os.environ['/usr/local/lib/python3.4/dist-packages/rsa']) in the script itself, but the problem persists.
Can anyone help me with this issue? Thanks.
P.S.- My knowledge of Python is very limited.

As far as I can remember it, I had similar problems with my Python-script and I finally solved it by using
python xyz.py
instead of
./xyz.py
Unfortunately I can not explain why it works in this way, but not the other, but in my case it solved the problem.

How to import 'GDB' in Python

I am using Python 2.7 and Python 3.1.3. But in my Python I am unable to "import gdb".
It is giving me an error as:
>>> import gdb
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
ImportError: No module named gdb
>>>
What's a reason for this? How should I solve this problem?

import gdb only works when your Python code is running within the GDB process. It's not supposed to work from the regular system Python interpreter.
Explanation
GDB embeds the Python interpreter so it can use Python as an extension language.
You can't just import gdb from /usr/bin/python like it's an ordinary Python library because GDB isn't structured as a library.
What you can do is source MY-SCRIPT.py from within gdb (equivalent to running gdb -x MY-SCRIPT.py).
Example Program
Here's a self contained example. Save the file below to t.py:
import gdb
gdb.execute('file /bin/cat')
o = gdb.execute('disassemble exit', to_string=True)
print(o)
gdb.execute('quit')
run:
$ gdb -q -x t.py
and you'll see the PLT stub for exit() disassembled. On x86-64 Linux:
Dump of assembler code for function exit#plt:
0x0000000000401ae0 <+0>: jmpq *0x20971a(%rip) # 0x60b200 <exit#got.plt>
0x0000000000401ae6 <+6>: pushq $0x3d
0x0000000000401aeb <+11>: jmpq 0x401700
End of assembler dump.
I've collected some resources on learning the GDB Python API here.

You can follow this tutorial to install PythonGDB. The Python code depends on a C extension.
For Windows, there is a recent enough gdb build in MinGW, but it doesn't seem to include the Python module you can import (still supports Python scripting in gdb). You have to install MinGW and then install the gbd package using mingw-get install gdb.
If you use Cygwin, there's again a recent enough gdb in Cygwin Ports, without a Python module but with Python scripting support.
I suppose it'd be possible to build gdb from source in either platform and get the Python module.

I just ran into the similar situation when trying to debug Webkit:
$ python Tools/gdb/webkit.py
Traceback (most recent call last):
File "Tools/gdb/webkit.py", line 38, in <module>
import gdb
ImportError: No module named gdb
I then realized that this script should be invoked in gdb to make it working:
(gdb) source Tools/gdb/webkit.py
(gdb) p run
$1 = (const WebCore::TextRun &) #0x7fffffffa450: {m_characters = "Plugin Testsa", m_len = 12, m_xpos = 0,
m_padding = 0, m_allowTabs = false, m_rtl = false, m_directionalOverride = false,
m_applyRunRounding = true, m_applyWordRounding = true, m_disableSpacing = false}
Hope this helps.

I can't test now, but I think you need to configure and build a Python-enabled GDB. Take a look at this guide.
I hope that helps.
This is outdated, I think. Anyway, you always need to build and configure a Python enabled GDB.
You can script GDB using the Python programming language. This feature is available only if GDB was configured using --with-python.
You have to configure GDB using that option:
--with-python=location
Where location is the location of python you would like to use GDB with.