I'm running into an issue trying to get python-markdown to work in pyinstaller. I have this code to demonstrate the issue in file called test.py:
import markdown
testMarkdown = "blahdy blah blah"
print(markdown.markdown(testMarkdown))
print(markdown.markdown(testMarkdown, extensions=["extra"]))
When I run it using python3, I get as desired:
(venv) C:\Users\madgrizzle>python3 test.py
<p>blahdy blah blah</p>
<p>blahdy blah blah</p>
I run pyinstaller as follows:
(venv) C:\Users\madgrizzle>pyinstaller test.py
and run the resulting code, I get the following:
(venv) C:\Users\madgrizzle\dist\test>test
<p>blahdy blah blah</p>
Traceback (most recent call last):
File "test.py", line 5, in <module>
File "lib\site-packages\markdown\core.py", line 390, in markdown
File "lib\site-packages\markdown\core.py", line 100, in __init__
File "lib\site-packages\markdown\core.py", line 126, in registerExtensions
File "lib\site-packages\markdown\core.py", line 166, in build_extension
File "importlib\__init__.py", line 126, in import_module
File "<frozen importlib._bootstrap>", line 985, in _gcd_import
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
ImportError: No module named 'extra'
[14432] Failed to execute script test
I tried to rebuild using:
(venv) C:\Users\madgrizzle>pyinstaller --hidden-import="markdown.extensions.extra" test.py
but I get the same error message.
Is there something special that's needed for including markdown extensions?
Additional Information:
It appears that the 'extra' extension might be causing the problem. Per https://python-markdown.github.io/extensions/extra/, 'extra' is a compilation of multiple extensions, including fenced_code and tables. If I just try to use the tables extension by itself, pyinstaller works IF I use the full-name as follows:
markdown.markdown(testMarkdown, extensions=["markdown.extensions.tables"])
If instead of using 'markdown.extensions.tables' I use 'markdown.extensions.extra', compile using pyinstaller, and run it, it responds back with a missing "fenced_code" module. Basically, it seems I have to avoid 'extra' with pyinstaller.
Short names for extensions like extra and table are setuptools entrypoints. I expect that pyinstaller does not play nice with entrypoints. Therefore, you need to use the full importable string name for all extensions.
However, as extra uses the short names internally, that means you cannot use extra. You will need to call each of the nested extensions separately by their full importable string name:
markdown.markdown(
testMarkdown,
extensions=[
"markdown.extensions.abbr",
"markdown.extensions.attr_list",
"markdown.extensions.def_list",
"markdown.extensions.fenced_code",
"markdown.extensions.footnotes",
"markdown.extensions.tables"
]
)
The only problem with that is that you won't get any extra specific behavior. However, when version 3.2 is released, the changes here will be available and you can add markdown.extensions.md_in_html to the list of extensions. At that point, you will get all of extra without needing to include extra at all.
Related
I am trying to compile a python program which imports configargparse, but pyinstaller doesn't find it, even if I specify it with hidden imports. I use following command: pyinstaller.exe --onefile --hidden-import=configargparse some_folder\viewer.py --log-level=DEBUG --debug=imports but when running the exe I get:
# configargparse not found in PYZ
Traceback (most recent call last):
File "some_folder\viewer.py", line 2, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'configargparse'
[5976] Failed to execute script 'viewer' due to unhandled exception!
Especially weird is that I get this debug message when compiling:
2711 DEBUG: Hidden import 'configargparse' already found
which obviously when running it doesn't seem to be the case
I tried with a small sample program:
import configargparse
print(configargparse)
but that doesn't work either, same error as for my more complex program.
I could once compile this program, but I'm not sure anymore how and the obvious way doesn't seem to work. What am I missing?
So I had to add my venv/lib/site-packages as an absolute path to pathex in the spec file.
As a user of the iapws package, I got hit by a general issue that I don't manage to solve.
The package is small, and a good candidate look at.
A simples script is the following (I called it main.py):
from iapws import IAPWS97
def main():
h = IAPWS97(P=1, x=1).h
print(f"h = {h:.5g} kJ/kg")
if __name__ == "__main__":
main()
This script works fine. Packaging this via pyinstaller main.py creates the dist\main folder with main.exe inside.
Executing main.exe yields
Traceback (most recent call last):
File "main.py", line 3, in <module>
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
File "iapws\__init__.py", line 15, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\...\\dist\\main\\iapws\\VERSION'
[22492] Failed to execute script main
The issue here is the missing VERSION file, i.e. a data file of the iapws package, which I as the packager of my importing package should not have to bother about. I cloned the iapws package and tried to specify the VERSION data file in the setup.py in several ways, but pyInstaller doesn't pick it up.
What can how be specified inside the iapws package to trigger pyInstaller to pick such datafiles up?
This issue is communicated with the iapws developer(s) here.
For this purpose, pyInstaller allows hooks to be applied. In this case, create a hook called hook-iapws.py with the content
from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files('iapws')
As a user of the package, the hook is found by pointing on the containing directory in the pyinstaller specfile:
a = Analysis(...
hookspath=["pyinstaller-hooks"],
...)
As a developer of the package, the hook can be made found by pyInstaller as described in this link or by providing the hook-file to the pyInstaller developers.
I answered on another thread, but same applies here:
You need to force pyinstaller to include iapws files, as it seems to have issues generating the appropriate hooks. This worked for me:
pyinstaller "your_script.py" --collect-all iapws
I reviewed command options here which helped: https://pyinstaller.org/en/stable/usage.html
I have recently performed various software updates as suggested by the "Software Center" on my Ubuntu machine (Ubuntu 18.04.5 LTS).
Now, when I try to import numba (numba==0.51.2) via
python3 -c 'import numba'
I get the following error
double free or corruption (top) Aborted (core dumped)
The same happens when I create a new conda environment with a fresh numba install.
I have looked at the core dump via
gdb -c core
with
thread apply all bt full
but I only get memory address information. I use python 3.6.9 on my machine, but I have also tried 3.8 in a new conda environment, where I get the same error.
I suspect that the software update is the reason for the error described above. But I might be mistaken and something else goes on here.
Is there any other way to get more info on where python crashes? I really don't want to go through the updated libraries one by one and roll them back to find the error.
At least I have now found the library that causes this error.
What I did were the following steps:
put import numba into a file, e.g. importNumba.py
locate python3.X-gdb.py via locate --regex python3.*-gdb.py. In my case it is in /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py
run python in debug mode via gdb python3 - the gdb console opens
execute source /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py in the gdb console - this will load the python extensions into gdb
execute run importNumba.py in the gdb console - this will produce above error
execute py-bt in the gdb console
This gives
Traceback (most recent call first):
File "/usr/local/lib/python3.6/dist-packages/llvmlite/binding/ffi.py", line 113, in __call__
return self._cfn(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/llvmlite/binding/dylib.py", line 29, in load_library_permanently
_encode_string(filename), outerr):
File "/usr/local/lib/python3.6/dist-packages/numba/__init__.py", line 151, in _try_enable_svml
llvmlite.binding.load_library_permanently("libsvml.so")
File "/usr/local/lib/python3.6/dist-packages/numba/__init__.py", line 201, in <module>
config.USING_SVML = _try_enable_svml()
<built-in method exec of module object at remote 0x7ffff7fb7638>
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "importNumba.py", line 1, in <module>
import numba
So it seems something is wrong with libsvml.so.
I found out that numba allows to disable SVML via
setting the environment flag NUMBA_DISABLE_INTEL_SVML to something other than 0, see https://numba.pydata.org/numba-doc/dev/reference/envvars.html
Changing importNumba.py to
import os
# note that this must be executed before 'import numba'
os.environ['NUMBA_DISABLE_INTEL_SVML'] = '1'
import numba
and running it via python3 importNumba.py now works without error.
These were a few useful resources that I used:
https://fedoraproject.org/wiki/Features/EasierPythonDebugging#New_gdb_commands
https://wiki.python.org/moin/DebuggingWithGdb
Normally PyInstaller works fine for me but i saw a problem using the python-module pycountry.
I tried this very simple code:
import pycountry
land="DE"
country = pycountry.countries.get (alpha_2=land)
print(country.name)
Compiled it with pyinstaller:
pyinstaller --onefile xyz.py
But i want to execute the compiled exe i get this error:
Traceback (most recent call last):
File "temp2.py", line 1, in <module>
import pycountry
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "c:\users\polzi\appdata\local\programs\python\python37\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\pycountry\__init__.py", line 12, in <module>
File "site-packages\pkg_resources\__init__.py", line 481, in get_distribution
File "site-packages\pkg_resources\__init__.py", line 357, in get_provider
File "site-packages\pkg_resources\__init__.py", line 900, in require
File "site-packages\pkg_resources\__init__.py", line 786, in resolve
pkg_resources.DistributionNotFound: The 'pycountry' distribution was not found and is required by the application
[45548] Failed to execute script temp2
Is there any workaround i can get pycountry functionality to running with pyinstaller?
UPDATE:
found a workaround / solution for my problem-
use command <pyi-makespec --onefile temp2.py> to generate a temp2.spec file
change filename.spec => from PyInstaller.utils.hooks import copy_metadata (in the header) => in the a = Analysis(...) section change " datas = []," to <datas = copy_metadata("pycountry"),>
use pyinstaller to compile exe as above
Alternative: compile program before - change spec - an use command <pyInstaller --clean temp2.spec> – Rapid1898 just now Edit
Your update was really helpful to me. But I needed a variation in order for it to work. I would just order the ideas to show clearer the way it must be done for pycountry library to work properly.
Use command pyi-makespec --onefile name-of-your-file.py to generate a .spec file
with name name-of-your-file.spec
Open name-of-your-file.spec with the text editor of your preference.
Add to the top of the .spec file the following line from PyInstaller.utils.hooks import copy_metadata
Replace datas = [] for datas = copy_metadata("pycountry")
Then rebuild using the following command PyInstaller --clean name-of-your-file.spec
I hope it is helpful for anyone facing the same error.
You can read the documentation for using spec files.
And you can find the same process in this link.
I need a lldb python library to debug my python script. I made my python environment configuration following the lldb.llvm.org's instructions. But I got some errors as follow:
/Users/heping/Desktop/Scripts/.env/python-3.7.3/bin/python /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 57996 --file /Users/heping/Desktop/Scripts/RevealServerCommands.py
pydev debugger: process 59879 is connecting
Connected to pydev debugger (build 193.5662.61)
Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/__init__.py", line 35, in <module>
import _lldb
ModuleNotFoundError: No module named '_lldb'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/__init__.py", line 38, in <module>
from . import _lldb
ImportError: dynamic module does not define module export function (PyInit__lldb)
And the PyCharm project structure is as picture showing blow:
The lldb python module shipped with Xcode builds against a specific version of Python.
Prior to Xcode 11 lldb was built against the Python2.7.1 in /System/Library/Frameworks. Starting with Xcode 11, lldb is built against the version of Python 3 (currently 3.7.3) that ships with the Xcode from which you got your lldb. You can locate the proper python3 command line tool by running xcrun python3.
We haven't had much success getting the lldb module we build against this 3.7.3 Python to load into other hand-built Pythons. I'm not sure that this is particularly well supported by Python, though I don't know of anybody who has looked into what it would take to support this.
We do use a lot of the Python C API's in the lldb bindings, so we are more bound to the Python version than pure Python modules. Anyway, at present if you need to load the lldb module into a python you have installed from elsewhere, you will most likely need to hand-build lldb against that python library.
On MacOS PyCharm go Preferences\Python Interpreter\ Then click on the Settings buttons and Show All.
Other answers said you need this:
import sys
sys.path.append('/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python3')
import lldb
With the above setting, it worked with just import lldb.