I'm compiling a python program using cython like this:
cython --embed myprg.py && gcc -Os -I /usr/include/python3.5m -o myprg myprg.c -lpython3.5m -lpthread -lm -lutil -ldl
If I run myprg I geht the following error message:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 890, in _find_spec
AttributeError: 'ModuleImporterFromVariables' object has no attribute 'find_spec'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "myprg.py", line 9, in init myprg (myprg.c:2219)
import json
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 892, in _find_spec
File "<frozen importlib._bootstrap>", line 873, in _find_spec_legacy
File "/usr/local/lib/python3.5/dist-packages/sh.py", line 3256, in find_module
while in_importlib(parent_frame):
File "/usr/local/lib/python3.5/dist-packages/sh.py", line 3204, in in_importlib
return frame.f_code.co_filename == "<frozen importlib._bootstrap>"
AttributeError: 'NoneType' object has no attribute 'f_code'
My python program is this:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import sh
import os
import traceback
import shutil
import sys
import json
print(sys.argv[1:])
It seems that the compiled program is not capable of loading the json module while the other modules work well. Some tests showed this seems to be the case with the argparse module as well.
My question: Why is this the case and how can I work around/fix that?
--
Additional notes:
As far as I understand this after compiling with cython the binary will make use of some kind of method of "sh.py" - part of the "sh" package. In this package for some reason a method checks for imports using the following code:
""" helper for checking if a filename is in importlib guts """
return frame.f_code.co_filename == "<frozen importlib._bootstrap>"
As there does not seem to be a stack frame available running the program based on the cython output, this method fails with an exception.
As far as I know this error did not occur some time ago. It is new. It seems that something changed in python in the last months that breaks the cython way of building a binary. Has anyone more information about this?
Related
I am trying to compile a python program which imports configargparse, but pyinstaller doesn't find it, even if I specify it with hidden imports. I use following command: pyinstaller.exe --onefile --hidden-import=configargparse some_folder\viewer.py --log-level=DEBUG --debug=imports but when running the exe I get:
# configargparse not found in PYZ
Traceback (most recent call last):
File "some_folder\viewer.py", line 2, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'configargparse'
[5976] Failed to execute script 'viewer' due to unhandled exception!
Especially weird is that I get this debug message when compiling:
2711 DEBUG: Hidden import 'configargparse' already found
which obviously when running it doesn't seem to be the case
I tried with a small sample program:
import configargparse
print(configargparse)
but that doesn't work either, same error as for my more complex program.
I could once compile this program, but I'm not sure anymore how and the obvious way doesn't seem to work. What am I missing?
So I had to add my venv/lib/site-packages as an absolute path to pathex in the spec file.
I have recently performed various software updates as suggested by the "Software Center" on my Ubuntu machine (Ubuntu 18.04.5 LTS).
Now, when I try to import numba (numba==0.51.2) via
python3 -c 'import numba'
I get the following error
double free or corruption (top) Aborted (core dumped)
The same happens when I create a new conda environment with a fresh numba install.
I have looked at the core dump via
gdb -c core
with
thread apply all bt full
but I only get memory address information. I use python 3.6.9 on my machine, but I have also tried 3.8 in a new conda environment, where I get the same error.
I suspect that the software update is the reason for the error described above. But I might be mistaken and something else goes on here.
Is there any other way to get more info on where python crashes? I really don't want to go through the updated libraries one by one and roll them back to find the error.
At least I have now found the library that causes this error.
What I did were the following steps:
put import numba into a file, e.g. importNumba.py
locate python3.X-gdb.py via locate --regex python3.*-gdb.py. In my case it is in /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py
run python in debug mode via gdb python3 - the gdb console opens
execute source /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py in the gdb console - this will load the python extensions into gdb
execute run importNumba.py in the gdb console - this will produce above error
execute py-bt in the gdb console
This gives
Traceback (most recent call first):
File "/usr/local/lib/python3.6/dist-packages/llvmlite/binding/ffi.py", line 113, in __call__
return self._cfn(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/llvmlite/binding/dylib.py", line 29, in load_library_permanently
_encode_string(filename), outerr):
File "/usr/local/lib/python3.6/dist-packages/numba/__init__.py", line 151, in _try_enable_svml
llvmlite.binding.load_library_permanently("libsvml.so")
File "/usr/local/lib/python3.6/dist-packages/numba/__init__.py", line 201, in <module>
config.USING_SVML = _try_enable_svml()
<built-in method exec of module object at remote 0x7ffff7fb7638>
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "importNumba.py", line 1, in <module>
import numba
So it seems something is wrong with libsvml.so.
I found out that numba allows to disable SVML via
setting the environment flag NUMBA_DISABLE_INTEL_SVML to something other than 0, see https://numba.pydata.org/numba-doc/dev/reference/envvars.html
Changing importNumba.py to
import os
# note that this must be executed before 'import numba'
os.environ['NUMBA_DISABLE_INTEL_SVML'] = '1'
import numba
and running it via python3 importNumba.py now works without error.
These were a few useful resources that I used:
https://fedoraproject.org/wiki/Features/EasierPythonDebugging#New_gdb_commands
https://wiki.python.org/moin/DebuggingWithGdb
I need a lldb python library to debug my python script. I made my python environment configuration following the lldb.llvm.org's instructions. But I got some errors as follow:
/Users/heping/Desktop/Scripts/.env/python-3.7.3/bin/python /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 57996 --file /Users/heping/Desktop/Scripts/RevealServerCommands.py
pydev debugger: process 59879 is connecting
Connected to pydev debugger (build 193.5662.61)
Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/__init__.py", line 35, in <module>
import _lldb
ModuleNotFoundError: No module named '_lldb'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/__init__.py", line 38, in <module>
from . import _lldb
ImportError: dynamic module does not define module export function (PyInit__lldb)
And the PyCharm project structure is as picture showing blow:
The lldb python module shipped with Xcode builds against a specific version of Python.
Prior to Xcode 11 lldb was built against the Python2.7.1 in /System/Library/Frameworks. Starting with Xcode 11, lldb is built against the version of Python 3 (currently 3.7.3) that ships with the Xcode from which you got your lldb. You can locate the proper python3 command line tool by running xcrun python3.
We haven't had much success getting the lldb module we build against this 3.7.3 Python to load into other hand-built Pythons. I'm not sure that this is particularly well supported by Python, though I don't know of anybody who has looked into what it would take to support this.
We do use a lot of the Python C API's in the lldb bindings, so we are more bound to the Python version than pure Python modules. Anyway, at present if you need to load the lldb module into a python you have installed from elsewhere, you will most likely need to hand-build lldb against that python library.
On MacOS PyCharm go Preferences\Python Interpreter\ Then click on the Settings buttons and Show All.
Other answers said you need this:
import sys
sys.path.append('/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python3')
import lldb
With the above setting, it worked with just import lldb.
I am trying to set up python in Vim, but I failed to get it to work. It always throws an exception named UnicodeDecodeError.
I have installed gvim on windows 10. And also installed python 3 with the corresponding version.
Vim can find the python37.dll and the command
:echo has('python3')
returns 1 as expected.
My vim with python works only when no modules except for the builtin ones are imported.
For example:
:py3 print('a')
works pretty well.
:py3 import vim
or
:py3 import sys
are also working.
However, if I write a simple python script vim_test.py like this
print('This is for vim test')
then try to import it in vim as
:py3 import vim_test
it will give an exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 963, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 906, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1280, in find_spec
File "<frozen importlib._bootstrap_external>", line 1252, in _get_spec
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb3 in position 9: invalid start byte
It cannot import any python module in Vim.
But it can run this script directly by
:py3file vim_test.py
if the script file vim_test.py is in the current directory.
What could be the reason for this problem?
And how can I solve it?
I expected to be able to use vim plugins written in python.
With this problem, I cannot achieve that.
System: windows 7 64 bit, python 3.5, anaconda 3 (64 bit) , django 1.10.1
I'm trying to compile my django project in 2 ways:
First:
[Anaconda3] c:\compilation\Gui>pyinstaller --name=gui --exclude-module=PyQt4 --exclude-module=matplotlib --clean --win-private-assemblies manage.py
Second according to this soloution:
[Anaconda3] c:\compilation\Gui>pyinstaller --name=gui --exclude-module=PyQt4 --exclude-module=matplotlib --clean --win-private-assemblies --runtime-hook=pyi_rth_django.py manage.py
When I try to run the output:
c:\compilation\Gui\dist\gui>gui.exe runserver
I get (for the 2 versions I get the same output):
c:\compilation\Gui\dist\gui>gui.exe runserver
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x00000000044E9D90>
Traceback (most recent call last):
File "site-packages\django\utils\autoreload.py", line 226, in wrapper
File "site-packages\django\core\management\commands\runserver.py", line 113, in inner_run
File "site-packages\django\utils\autoreload.py", line 249, in raise_last_exception
File "site-packages\django\utils\six.py", line 685, in reraise
File "site-packages\django\utils\autoreload.py", line 226, in wrapper
File "site-packages\django\__init__.py", line 27, in setup
File "site-packages\django\apps\registry.py", line 85, in populate
File "site-packages\django\apps\config.py", line 116, in create
File "importlib\__init__.py", line 126, in import_module
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named 'django.contrib.admin.apps'
Please advice.
What is your files layout? According to these pyinstaller docs https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Executable-From-Django there could be two solutions.
run your command from parent directory, i.e. instead of
c:\compilation\Gui>pyinstaller --name=gui manage.py
do
c:\compilation>pyinstaller --name=gui Gui\manage.py
try to add import django.contrib.admin.apps to your manage.py and make sure it exists
report bug
To fix the "ImportError: No module named django.contrib.admin.apps" problem you have to create a directory, let's call it "your_project/other_hooks".
Inside that directory create a file called hook-django.contrib.py with this content:
#!/usr/bin/python
# -*- coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('django.contrib')
Now call pyinstaller in this way:
pyinstaller --name=yourProject --additional-hooks-dir=your_project\other_hooks your_project\manage.py
While you can use any name for the directory, the file name is mandatory, it has to be "hook-django.contrib.py".
Hope this helps.
References:
https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Executable-From-Django
https://github.com/pyinstaller/pyinstaller/issues/2332
https://pythonhosted.org/PyInstaller/hooks.html#how-a-hook-is-loaded