Develop Reference

Importing numba crashes with core dump

I have recently performed various software updates as suggested by the "Software Center" on my Ubuntu machine (Ubuntu 18.04.5 LTS).
Now, when I try to import numba (numba==0.51.2) via
python3 -c 'import numba'
I get the following error
double free or corruption (top) Aborted (core dumped)
The same happens when I create a new conda environment with a fresh numba install.
I have looked at the core dump via
gdb -c core
with
thread apply all bt full
but I only get memory address information. I use python 3.6.9 on my machine, but I have also tried 3.8 in a new conda environment, where I get the same error.
I suspect that the software update is the reason for the error described above. But I might be mistaken and something else goes on here.
Is there any other way to get more info on where python crashes? I really don't want to go through the updated libraries one by one and roll them back to find the error.

At least I have now found the library that causes this error.
What I did were the following steps:
put import numba into a file, e.g. importNumba.py
locate python3.X-gdb.py via locate --regex python3.*-gdb.py. In my case it is in /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py
run python in debug mode via gdb python3 - the gdb console opens
execute source /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py in the gdb console - this will load the python extensions into gdb
execute run importNumba.py in the gdb console - this will produce above error
execute py-bt in the gdb console
This gives
Traceback (most recent call first):
File "/usr/local/lib/python3.6/dist-packages/llvmlite/binding/ffi.py", line 113, in __call__
return self._cfn(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/llvmlite/binding/dylib.py", line 29, in load_library_permanently
_encode_string(filename), outerr):
File "/usr/local/lib/python3.6/dist-packages/numba/__init__.py", line 151, in _try_enable_svml
llvmlite.binding.load_library_permanently("libsvml.so")
File "/usr/local/lib/python3.6/dist-packages/numba/__init__.py", line 201, in <module>
config.USING_SVML = _try_enable_svml()
<built-in method exec of module object at remote 0x7ffff7fb7638>
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "importNumba.py", line 1, in <module>
import numba
So it seems something is wrong with libsvml.so.
I found out that numba allows to disable SVML via
setting the environment flag NUMBA_DISABLE_INTEL_SVML to something other than 0, see https://numba.pydata.org/numba-doc/dev/reference/envvars.html
Changing importNumba.py to
import os
# note that this must be executed before 'import numba'
os.environ['NUMBA_DISABLE_INTEL_SVML'] = '1'
import numba
and running it via python3 importNumba.py now works without error.
These were a few useful resources that I used:
https://fedoraproject.org/wiki/Features/EasierPythonDebugging#New_gdb_commands
https://wiki.python.org/moin/DebuggingWithGdb

Cannot load German BERT model in spaCy

Here is my problem: I am working on the German text classification project. I use spacy for that and decided to fine-tune its pretrained BERT model to get better results. However, when I try to load it to the code, it shows me errors.
Here is what I've done:
Installed spacy-transformers: pip install spacy-transformers
Downloaded German BERT model: python -m spacy download de_trf_bertbasecased_lg. It was downloaded successfully and showed me: ✔ Download and installation successful
You can now load the model via spacy.load('de_trf_bertbasecased_lg')
Wrote the following code:
import spacy
nlp = spacy.load('de_trf_bertbasecased_lg')
And the output was:
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
nlp = spacy.load('de_trf_bertbasecased_lg')
File "C:\Python\Python37\lib\site-packages\spacy\__init__.py", line 30, in load
return util.load_model(name, **overrides)
File "C:\Python\Python37\lib\site-packages\spacy\util.py", line 164, in load_model
return load_model_from_package(name, **overrides)
File "C:\Python\Python37\lib\site-packages\spacy\util.py", line 185, in load_model_from_package
return cls.load(**overrides)
File "C:\Python\Python37\lib\site-packages\de_trf_bertbasecased_lg\__init__.py", line 12, in load
return load_model_from_init_py(__file__, **overrides)
File "C:\Python\Python37\lib\site-packages\spacy\util.py", line 228, in load_model_from_init_py
return load_model_from_path(data_path, meta, **overrides)
File "C:\Python\Python37\lib\site-packages\spacy\util.py", line 196, in load_model_from_path
cls = get_lang_class(lang)
File "C:\Python\Python37\lib\site-packages\spacy\util.py", line 70, in get_lang_class
if lang in registry.languages:
File "C:\Python\Python37\lib\site-packages\catalogue.py", line 56, in __contains__
has_entry_point = self.entry_points and self.get_entry_point(name)
File "C:\Python\Python37\lib\site-packages\catalogue.py", line 140, in get_entry_point
return entry_point.load()
File "C:\Python\Python37\lib\site-packages\importlib_metadata\__init__.py", line 94, in load
module = import_module(match.group('module'))
File "C:\Python\Python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "C:\Python\Python37\lib\site-packages\spacy_transformers\__init__.py", line 1, in <module>
from .language import TransformersLanguage
File "C:\Python\Python37\lib\site-packages\spacy_transformers\language.py", line 5, in <module>
from .util import is_special_token, pkg_meta, ATTRS, PIPES, LANG_FACTORY
File "C:\Python\Python37\lib\site-packages\spacy_transformers\util.py", line 2, in <module>
import transformers
File "C:\Python\Python37\lib\site-packages\transformers\__init__.py", line 20, in <module>
from .file_utils import (TRANSFORMERS_CACHE, PYTORCH_TRANSFORMERS_CACHE, PYTORCH_PRETRAINED_BERT_CACHE,
File "C:\Python\Python37\lib\site-packages\transformers\file_utils.py", line 37, in <module>
import torch
File "C:\Python\Python37\lib\site-packages\torch\__init__.py", line 81, in <module>
ctypes.CDLL(dll)
File "C:\Python\Python37\lib\ctypes\__init__.py", line 356, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found
If I run the same code in PyCharm, it also shows me these two lines before all of those above:
2020-05-19 18:00:55.132721: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not
load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2020-05-19 18:00:55.132990: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart
dlerror if you do not have a GPU set up on your machine.
If I got it right, these two lines complain that I don't have a GPU. However, according to the docs, I should be able to use BERT even without GPU.
So I am really stuck right now and looking for your help.
I should also mention, that I used de_core_news_sm model before and it worked fine.
I have also already tried several solutions, but none of them worked. I tried:
this and this. I have also tried to uninstall all spacy-related libraries and installed them again. Didn't help either.
I am working with:
Windows 10 Home
Python: 3.7.2
Spacy: 2.2.4
Spacy-transformers: 0.5.1
Would appreciate any help or advice!

It's probably a problem with your installation of torch. Start in a clean virtual environment and install torch using the instructions here with CUDA as None: https://pytorch.org/get-started/locally/. Then install spacy-transformers with pip.

Try:
python -m spacy download de_trf_bertbasecased_lg
python -m spacy download de_trf_bertbasecased_lg-2.2.0
python -m spacy link de_trf_bertbasecased_lg
python -m spacy link de_trf_bertbasecased_lg-2.2.0
Or:
python -m spacy download de
python -m spacy.de.download all
Or download directly from:
https://deepset.ai/german-bert
Then load:
nlp = spacy.load(r'Path_To_File\de_trf_bertbasecased_lg-2.2.0')

How to import lldb module for python on Mac?

I need a lldb python library to debug my python script. I made my python environment configuration following the lldb.llvm.org's instructions. But I got some errors as follow:
/Users/heping/Desktop/Scripts/.env/python-3.7.3/bin/python /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 57996 --file /Users/heping/Desktop/Scripts/RevealServerCommands.py
pydev debugger: process 59879 is connecting
Connected to pydev debugger (build 193.5662.61)
Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/__init__.py", line 35, in <module>
import _lldb
ModuleNotFoundError: No module named '_lldb'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/__init__.py", line 38, in <module>
from . import _lldb
ImportError: dynamic module does not define module export function (PyInit__lldb)
And the PyCharm project structure is as picture showing blow:

The lldb python module shipped with Xcode builds against a specific version of Python.
Prior to Xcode 11 lldb was built against the Python2.7.1 in /System/Library/Frameworks. Starting with Xcode 11, lldb is built against the version of Python 3 (currently 3.7.3) that ships with the Xcode from which you got your lldb. You can locate the proper python3 command line tool by running xcrun python3.
We haven't had much success getting the lldb module we build against this 3.7.3 Python to load into other hand-built Pythons. I'm not sure that this is particularly well supported by Python, though I don't know of anybody who has looked into what it would take to support this.
We do use a lot of the Python C API's in the lldb bindings, so we are more bound to the Python version than pure Python modules. Anyway, at present if you need to load the lldb module into a python you have installed from elsewhere, you will most likely need to hand-build lldb against that python library.

On MacOS PyCharm go Preferences\Python Interpreter\ Then click on the Settings buttons and Show All.
Other answers said you need this:
import sys
sys.path.append('/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python3')
import lldb
With the above setting, it worked with just import lldb.

How to fix 'AttributeError' on using PyInstaller on a project with Tensorflow and Keras?

The --onefile exe PyInstaller outputs crashes upon startup. The error is:
AttributeError: module 'tensorflow' has no attribute 'keras'
[11288] Failed to execute script main
Note that the main.py script works fine though and never had this problem.
I'm doing all of this with PyCharm in Windows 10.
These commands were used to create the .spec file and to build the executable file:
pyi-makespec --onefile --name app main.py
pyinstaller --clean app.spec
Upon checking the generated warn-app.txt, I've seen this line:
missing module named keras - imported by ai (top-level)
I got rid of it by adding the path of tensorflow.keras to the generated .spec file before running the second command.
The path was determined through use of the terminal and the venv of my project:
import tensorflow as tf
print(tf.keras.__file__)
Or:
from tensorflow import keras
print(keras.__file__)
Which both yield <path to venv folder>\lib\site-packages\tensorflow\python\keras\api\_v2\keras\__init__.py
This were then added to the .spec file under pathex such that:
a = Analysis(['main.py'],
pathex=['<path to project folder>', '<path to venv folder>\lib\site-packages\tensorflow\python\keras\api\_v2\keras\__init__.py'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
Upon running pyinstaller --clean app.spec again, the missing module warning on the warn-app.txt was gone, but the generated --onefile still crashes the error.
The code which requires the tensorflow.keras module is as simple as this:
In ai.py:
from tensorflow import keras
model = keras.load_model(`./model.hf5`)
The basic requirements (imported by the project itself) of the project as displayed by pip freeze:
numpy==1.16.4
opencv-python==4.1.0.25
pandas==0.25.0
PyInstaller==3.5
tensorflow==2.0.0a0
The extended requirements (downloaded and installed too when installing the basic requirements):
absl-py==0.7.1
altgraph==0.16.1
astor==0.8.0
future==0.17.1
gast==0.2.2
google-pasta==0.1.7
grpcio==1.22.0
h5py==2.9.0
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
Markdown==3.1.1
numpy==1.17.0
opencv-python==4.1.0.25
pandas==0.25.0
pefile==2019.4.18
protobuf==3.9.0
PyInstaller==3.5
python-dateutil==2.8.0
pytz==2019.1
pywin32-ctypes==0.2.0
PyYAML==5.1.2
scipy==1.3.0
six==1.12.0
tb-nightly==1.14.0a20190301
tensorflow==2.0.0a0
termcolor==1.1.0
tf-estimator-nightly==1.14.0.dev2019030115
Werkzeug==0.15.5
The whole error:
D:\Shared\CMSC\190-2\readr>readr.exe
Limited tf.compat.v2.summary API due to missing TensorBoard installation
Limited tf.summary API due to missing TensorBoard installation
Traceback (most recent call last):
File "main.py", line 4, in <module>
from modules.model import models
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "d:\shared\cmsc\190-2\venv\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "modules\model.py", line 4, in <module>
models = load_models(n=num_models)
File "modules\ai.py", line 28, in load_models
model = tf.keras.models.load_model(os.path.join('.', 'models', ('m' + str(i) + 'eF.h5')))
AttributeError: module 'tensorflow' has no attribute 'keras'
[11288] Failed to execute script main
I honestly thought getting rid of that missing module warning would get it fixed, but it's still here, and I don't know why.

Downgrading to tensorflow==1.14.0 fixed the error for me.
This is now the basic requirements of my program:
numpy==1.16.4
opencv-python==4.1.0.25
pandas==0.25.0
PyInstaller==3.5
tensorflow==1.14.0
In the .spec file of my program, I also added ['tensorflow.keras'] to hiddenimports.

even though "which tensorboard" showed that tensorboard was already installed, "pip install tensorboard" fixed the following error: Limited tf.compat.v2.summary API due to missing TensorBoard installation.

Install tensorflow GPU

i'm trying install tensorflow gpu on win10.
my steps:
install Python 3.5.2
install Nvidia CUDA(v9)+cudnn
copy cudnn files on prog.files directory NVIDIA GPU Computing Toolkit
install tensorflow gpu (pip3 install --upgrade tensorflow-gpu)
C:\Python>python
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import tensorflow as tf
Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Python\lib\importlib__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 666, in _load_unlocked
File "", line 577, in module_from_spec
File "", line 906, in create_module
File "", line 222, in _call_with_frames_removed
ImportError: DLL load failed: Не найден указанный модуль.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Python\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python\lib\site-packages\tensorflow\__init__.py", line 24, in <module>
from tensorflow.python import *
File "C:\Python\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 52, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Python\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
File "<frozen importlib._bootstrap>", line 577, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 906, in create_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: Не найден указанный модуль.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Python\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/install_sources#common_installation_problems
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
If I install tensorflow without GPU- all OK.

It is really important to follow the TensorFlow install instructions, thereby choosing the correct versions:
CUDA® Toolkit 8.0. For details, see NVIDIA's documentation Ensure that you append the relevant Cuda pathnames to the %PATH% environment variable as described in the NVIDIA documentation.
The NVIDIA drivers associated with CUDA Toolkit 8.0.
cuDNN v6 or v6.1. For details, see NVIDIA's documentation. Note that cuDNN is typically installed in a different location from the other CUDA DLLs. Ensure that you add the directory where you installed the cuDNN DLL to your %PATH% environment variable.
GPU card with CUDA Compute Capability 3.0 or higher. See NVIDIA documentation for a list of supported GPU cards.
Currently, the newest version for the CUDA Toolkit is 9.0, but for Tensorflow you need version 8, which is available on the CUDA toolkit archive:
https://developer.nvidia.com/cuda-80-ga2-download-archive
And for the cuDNN (Deep Neural Network libraries), you need to select cuDNN v6, and not the newest version v7: https://developer.nvidia.com/rdp/cudnn-download
Finally, be sure to put the Windows %PATH% variable to the CUDA bin directory (where the dll libraries are).
Then, you can install tensorflow:
pip3 install --upgrade tensorflow-gpu
If it does not work yet, you might want to restart first.

This is how i resolved it,
If you are using pip or pip3 you have less chance succeeding in first time.
Uninstall Python,CUDA, CuNN, Tensorflow-gpu (if you already installed)
Install ANACONDA or MINICONDA
Open ANACONDA prompt and run following command:
conda create --name tf_gpu tensorflow-gpu
This will create an environment tf_gpu whcih will install all compatible versions of Python, CUDA, CuNN and Tensorflow
once all the packages installed open the ANACONDA prompt and type the following command
conda activate tf_gpu
This will change the conda '(base)' to '(tf_gpu)' environment
Then type python as usual to activate the python prompt
If you want to deactivate the 'tf_gpu' environment
conda deactivate

Currently, these versions are supported. Do not try to install more recent versions.
Visual Studio 2015.
CUDA 9.0.
cuDNN 7.0 (make sure to copy the dlls to the right directory).
Python 3.5.
Then you can install tf

I have tried so many blogs myself. I always got one or the other errors.
However, the following worked perfectly for me.
conda create --name tf_gpu tensorflow-gpu
You simply need to have Nvidia drivers installed on your system. You don't need to install CUDA and CudNN, and worry about the version compatibility.
This command will install the CUDA and CudNN alongwith the compatible version of Tensorflow for you.

Finally this documentation worked for me: Anaconda | TensorFlow
The document is self-explanatory. If you already have an environment, just run:
conda install tensorflow-gpu
Important notes:
GPU TensorFlow uses CUDA.
On Windows and Linux only CUDA 10.0 is supported for the TensorFlow 2.0 release. Previous versions of TensorFlow support other version of CUDA.

Ok, so I have referred to many blogs and StackOverflow answers, but none of them worked for me. However, I have at last found a procedure that worked perfectly fine for me. So, I have an NVIDIA GTX 1650 GPU, and below are the steps that I followed to get GPU enabled TensorFlow up and running:
1) Install the NVIDIA Drivers (Studio Driver) and restart your PC.
2) Download the visual studio 2019 Community version (In it, install all the c++, python/AI/DL dependencies).
3) Install Cuda 10.1 (update 2,released in august,2019).
4) Install CuDNN 7.6.x (choose x accordingly, so as per Cuda 10.1 (august,2019), the appropriate value for x is 3). So I basically installed CuDNN 7.6.3 for Cuda 10.1
5) Then, we have to add the bin, include, lib\x64 folders of Cuda to the System Path. (Note that include, lib\x64 will be added by default to the path after installing Cuda. So we just have to add the bin folder of Cuda to the path).
6) Next, we have to add the bin folder of CuDNN to the System Path.
7) Finally, type pip install TensorFlow in CMD, and restart your PC.
Note that if you have a different GPU, you can check the official websites of Cuda and CuDNN to check which versions of Cuda and CuDNN are compatible with respect to your GPU.

If you are using anaconda then run this command in your anaconda prompt
conda install -c anaconda tensorflow-gpu
This will also install the cuda toolkit and cudnn for you and you are good to go
At website it mentions that this command is for linux so I am not sure whether it will work on window, but you can try