Getting the error message when using matplotlib:
Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib
already initialized
OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can
degrade performance or cause incorrect results. The best thing to do
is to ensure that only a single OpenMP runtime is linked into the
process, e.g. by avoiding static linking of the OpenMP runtime in any
library. As an unsafe, unsupported, undocumented workaround you can
set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the
program to continue to execute, but that may cause crashes or silently
produce incorrect results. For more information, please see
http://www.intel.com/software/products/support/.
Do the following to solve the issue:
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
Answer found at: https://github.com/dmlc/xgboost/issues/1715
Be aware of potential side-effects:
but that may cause crashes or silently produce incorrect results.
This is a better solution, if applicable. Else, anyway gcamargo’s solution is likely to work. However, it comes with a warning "that it may cause crashes or silently produce incorrect results"
I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with
conda install nomkl
Answer found at: https://github.com/dmlc/xgboost/issues/1715
I had the same issue on macOS and found the following reasons:
Problem:
I had a conda environment where Numpy, SciPy and TensorFlow were installed.
Conda is using Intel(R) MKL Optimizations, see docs:
Anaconda has packaged MKL-powered binary versions of some of the most popular numerical/scientific Python libraries into MKL Optimizations for improved performance.
The Intel MKL functions (e.g. FFT, LAPACK, BLAS) are threaded with the OpenMP technology.
But on macOS you do not need MKL, because the Accelerate Framework comes with its own optimization algorithms and already uses OpenMP. That is the reason for the error message: OMP Error #15: ...
Workaround:
You should install all packages without MKL support:
conda install nomkl
and then use
conda install numpy scipy pandas tensorflow
followed by
conda remove mkl mkl-service
For more information see conda MKL Optimizations.
I had the same issue in a conda environment where TensorFlow was installed. After doing
pip uninstall tensorflow
pip install tensorflow
the problem was gone.
For me, this problem came up when I imported pytorch after numpy. Importing them in this order fixed my problem:
import torch
import numpy as np
Had same issue in OSX when updating tensoflow to 1.13 using conda.
Solution 1: /gcamargo worked but 3x slower per training epoch.
Solution 2: /sjcoding worked and removed serious warining but also 3x
slower in training.
Solution 3: that restored performance was: Install pip in new conda
env and use pip to install tensorflow. Using conda-forge also worked
but version of tf is old.
Apparently the new Intel-MKL optimizations in Anaconda are broken for OSX tensorflow.
Check if there's an update for the mkl package in your env (anaconda).
I was able to solve my case simply by updating mkl.
conda install -c intel mkl
(macOS Catalina 10.15.5)
I am using Macbook M1 and I faced the same issue. I solved this problem after removing the mkl package from my conda environment using the following command:
conda remove mkl
This issue happens because mkl is developed for Intel users and many of my packages were comming from mkl. After you remove this package, you will be required to reinstall many packages that you use through mkl. Also, in my case, when I tried to install pandas afterwards, there was a dependency issue. I solved this issue as well after updating all conda packages using the following command:
conda update --all
A useful link that helped me figure this out:
So, for those of you getting this same issue with lightgbm, I found in the documentation that you can
pip uninstall lightgbm
pip install lightgbm
Run the following in anaconda environmnet (if you're running Conda)
ln -sf `ls -d "$(brew --cellar libomp)"/*/lib`/* $CONDA_PREFIX/lib
These three things worked for me.
Try to change the backend of matplotlib.
For example, Tkagg backend causes this problem in my case. I changed it to Qt5Agg
matplotlib.use('Qt5Agg')
and it helps.
Confronted with the same error #15, none of the solutions to-date (5 Feb 2021) fully worked despite being helpful. However, I did manage to solve it while avoiding: dithering with dylib libraries, installing from source, or setting the environment variable KMP_DUPLICATE_LIB_OK=TRUE and its downsides of being an “unsafe, unsupported, undocumented workaround” and its potential “crashes or silently produce incorrect results”.
The trouble was that conda wasn’t picking up the non-mkl builds of tensorflow (v2.0.0) despite loading the nomkl package. What finally made this solution work was to:
ensure I was loading packages from the defaults channel (ie. from a channel with a non-mkl version of tensorflow. As of 5 Feb 2021, conda-forge does not have a tensorflow version of 2.0 or
greater).
specify the precise build of the tensorflow version I wanted: tensorflow>=2.*=eigen_py37h153756e_0. Without this, conda kept loading the mkl_... version of the package despite the nomkl package also being loaded.
I created a conda environment using the following environment.yml file (as per the conda documentation for managing environments) :
name: tf_nomkl
channels:
- conda-forge
- defaults
dependencies:
- nomkl
- python>=3.7
- numpy
- scipy
- pandas
- jupyter
- jupyterlab
- nb_conda
- nb_conda_kernels
- ipykernel
- pathlib
- matplotlib
- seaborn
- tensorflow>=2.*=eigen_py37h153756e_0
You could try to do the same without an environment.yml file, but it’s better to load all the packages you want in an environment in one go if you can.
This solution works on MacOS Big Sur v11.1.
conda install --revision 0 doesn't solve UnsatisfiableError: The following specifications... for me. So I manually install nomkl and remove mkl and mil-service in Anaconda-Navigator environment, and it works great for me!
I was getting the same error as mentioned in the original question when I ran a code with Tensorflow on my macOS Monterey. I tried installing nomkl and removing mkl as suggested in many of the previous answers. However this gave me trouble on running readcsv module of pandas and many other modules from different packages. A friend told me that newer versions of macOS have trouble with the usual Tensorflow and therefore pypi has launched a special version of TF called tf-nightly.
https://pypi.org/project/tf-nightly/#description
This installation solved the problem for me.
I had the same problem. Nothing you suggested solved the issue. I found that a possible cause is that you have multiple OpenMP libraries installed on your machine and they conflict with each other. Plus, I found that the problem was numpy and I did the upgrade (conda update numpy) and FINALLY IT WORKED!!!
Related
I am trying to debug an error with tensorflow. When I import tensorflow
I get the following error
import tensorflow as tf
terminate called after throwing an instance of 'Xbyak::Error'
what(): internal error
Aborted (core dumped)
Here are the install details
OS >> Ubuntu 14.04
Install Method Anaconda >> conda 4.4.11 (conda install tensorflow)
python3 --version >> Python 3.6.4 :: Anaconda, Inc.
If anyone has experience or knowledge on how to fix this issue, I would be most grateful. A search here, on google and on github yielded suggested the issue is likely with MKL-DNN, has anyone found a work around?
As others have mentioned, it is probably a better practice to install from source using official documentation:
https://www.tensorflow.org/install/install_linux#installing_with_anaconda
That's the only method which google supports.
The conda forge version (1.5.0) works fine. Using
conda install -c conda-forge tensorflow
p.s.: Make sure you don't edit as that will throw back the original error.
As stated in my comment, I had the same problem as you. Anaconda seems to not fully support tensorflow or have an older version, because when I installed tensorflow using pip install tensorflow I had no problems. I hope this solves it for you as well.
Even I kept getting this error when I created another environment and installed TF in it. Seems like Tensorflow is the main issue here. What did solve my problem was I simply installed Tensorflow using this command pip3 install tensorflow. I hope this solves your problem.
Just got the same error using the conda distribution. Tried the source forge version and it worked great. but the conda is TF 1.7 and the source forge is 1.5.
I am trying to develop some time-series sequence prediction, using the latest resources available. To that end, I did check the example code from TensorFlow time-series, but I'm getting this error:
AttributeError: module 'tensorflow.python.pywrap_tensorflow' has no attribute 'TFE_Py_RegisterExceptionClass'
I'm using Anaconda. The current environment is Python 3.5 and TensorFlow 1.2.1. Also tried TensorFlow 1.3, but nothing changed.
Here is the code I'm trying to run. I did not find anything useful related to the issue on Google. Any ideas on how to solve it?
As Conan.Net wrote:
I tried to remove/clean some environments from anaconda and install
all again and it work this time.
This solution worked for me as well, so though not ideal, it will solve the problem. If you are using anaconda, it might happen when installing some packages and then removing them (e.g. tensorflow vs tensorflow-gpu) leaves some dependencies hanging. In my case, I used:
conda remove --name py2_tf_gpu --all
then
conda create --name py2_tf_gpu python=2 anaconda pandas numpy scipy jupyter
source activate py2_tf_gpu
pip install --ignore-installed --upgrade tensorflow-gpu
pip currently installs a later(1.4) than anaconda(1.3) version and I had need for it.
Maybe the version of tensorflow doesn't match the version of keras.
Using a lower version of keras solve this problem
I use pip to install lightgbm on windows,Does it work?
Since I have an error while importing LGBMRegressor.
The error shows as below
"cannot import name 'LGBMRegressor'"
I tried installing it using an anaconda environment and was able to install it successfully.
Open Anaconda prompt and type:
conda install -c conda-forge lightgbm
It will take a few minutes for installation.
I am writing an answer as I don't have enough reputation to leave a comment yet. Valid solutions have been offered but I think that some subtleties should be addressed for those that come here and are new(ish) to Python.
There are differences between pip and conda and if I understand it well, one ought to keep using conda as much as possible within their conda environments to avoid running into time-wasting package conflicts. I advise you read this article to get a better grasp on which does what. Myth #2 and #10 are particularly relevant here.
For Lightgbm the obvious solution is to use conda-forge as mentioned above. That is if you are running a conda environment, be it anaconda or miniconda. But you should be aware that for the moment anaconda distribution (AD) and conda-forge (CF) are not 100% compatible, as you can read in this thread. From their own words:
Conda-forge and The Anaconda Distribution are incompatible with each other at present.
...Please pick one or the other for now. CF or AD, never both.
So far it seems not to be a problem for lightbgm though.
Happy coding!
I used this and it worked for me even with Python 3.6
import sys
!{sys.executable} -m pip install lightgbm
I am running a python script and I get this error:
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Both files are present in the anaconda2/lib directory. How can I fix this error? Thanks.
If you use conda, try with these two commands:
conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service
It should fix your problem.
I ran into this problem after installing anaconda3 (vesion 4.2.0). The fix for me was simple, and I was able to keep using mkl. Just update to the latest numpy version.
conda update numpy
Just wanted to note that Anaconda 4.0.0, shipped with mkl enabled by default, has this issue.
The problem is indeed with Anaconda, as it can be reproduced with the simple python test below.
The actual issue is that Anaconda linked with mkl, but not with libmkl_core.so, thus it has a missing symbol, and can be seen by running:
$ LD_DEBUG=symbols python -c 'import sklearn.linear_model.tests.test_randomized_l1' 2>&1 | grep -i error
2200: /opt/anaconda/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)
I didn't want to uninstall mkl, as I'd like to have the performance boost, so I found a workaround which worked for me - preload libmkl_core.so before execution.
$ python -c 'import sklearn.linear_model.tests.test_randomized_l1'
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.
$
$ LD_PRELOAD=/opt/anaconda/lib/libmkl_core.so python -c 'import sklearn.linear_model.tests.test_randomized_l1'
$
I had this same issue using scikit-learn 0.19 and numpy 1.13.3 when running MLPRegressor (and also with a package called pyearth running an algorithm called MARS). I believe the root of the problem was that our python is part of an Anaconda install, but scikit-learn and numpy were installed via pip, and their expectations for mkl must not agree.
Unfortunately my framework is managed by some dedicated company admins, not by me, so I haven't gotten my guy to try recompiling numpy yet. But I was able to find a workaround based on this thread: Adding export LD_PRELOAD=/path/to/anaconda/lib/libmkl_def.so:/path/to/anaconda/lib/libmkl_avx.so:/path/to/anaconda/lib/libmkl_core.so:/path/to/anaconda/lib/libmkl_intel_lp64.so:/path/to/anaconda/lib/libmkl_intel_thread.so:/path/to/anaconda/lib/libiomp5.so to my ~/.bashrc causes the problem to disappear. It's super hacky, and I'd be lying if I said I knew exactly what it's doing (but this is helpful), so I'm hoping a recompile of numpy is a cleaner fix. But at least it works.
Note it is better performance-wise to have the versions of these packages that use mkl. Installing the nomkl versions is a workaround but not a true solution.
Got the same problem and solved by using the command:
conda install nomkl
I got the solution from this discussion https://github.com/BVLC/caffe/issues/3884
It is also possible that you are running Python in a folder which does not exist.
Solution is simply go to another folder.
See this question: sh: 0: getcwd() failed: No such file or directory on cited drive
Wanted to add on Valilutzik and Zaikun Xu's thread. I can't add comments since I don't have enough points yet.
conda install nomkl numpy scipy scikit-learn numexpr worked for me without having to remove mkl and mkl-service.
Adding an answer to Lee's question if nomkl will be slower:
mkl is a Intel math kernel library and is hand-optimized for intel cpus. nomkl uses OpenBlas according to this: https://docs.continuum.io/mkl-optimizations/
It seems that mkl is quite faster on many matrix operations for intel cpus (see https://software.intel.com/en-us/articles/performance-comparison-of-openblas-and-intel-math-kernel-library-in-r)
I've seen someone saying nomkl is faster for AMD cpus (maybe because mkl doesn't work correctly in AMD?)
All the solutions provided above did not work for me, but I found a good compromise:
for someone who has the same error and want's to keep Anaconda on his computer, and steel use mkl for (numpy and scipy) good process performance, the solution I propose:
Edit your .bashrc file.
Look for something like export PATH="/home/anaconda2/bin:$PATH" in the file.
Put a # at the beginning to comment it from the script:
#export PATH="/home/anaconda2/bin:$PATH"
Open a new terminal and you should be running the base python installation.
It worked for me, hope it will be helpfull.
In case anyone has a similar issue and gets errors that libmkl_p4m.so or libmkl_p4.so cannot be found (this occurred for me when calling certain numpy functions), I tried reinstalling / updating different python modules and reverting / updating to various versions of Anaconda, and neither worked.
However I found that completely uninstalling anaconda and re-installing it (to version 4.4.10) solved the issue.
try export LD_DEBUG=libs and execute your program again.
It will tell you which .so is causing the problem.
Then you can do something, e.g. upgrading the package.
For me, I find upgrade numexpr useful as it prints the following message just before the error:
calling init: xxxxx/python3.6/site-packages/numexpr/interpreter.cpython-36m-x86_64-linux-gnu.so
The following worked for me
conda install -f numpy
None of the above answers worked for me. In my case, #Nehal J Wani's comment below the OP's question led to the solution: As stated in the linked Intel support document, a single dynamic library can be used to link to the MKL.
After editing my project's Makefile to replace the following libraries
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core
with
-lmkl_rt and re-building, the error disappeared.
For me downgrading mkl solved it:
conda install mkl=2021.2.0
(Ubuntu 21.04, Python 3.8)
solved by reinstallation:
pip install tensorflow
I am trying to use the One Million Song Dataset, for this i had to install python tables, numpy, cython, hdf5, numexpr, and so.
Yesterday i managed to install all i needed, and after having some troubles with hdf5, i downloaded the precompiled binary packages and saved them in my /bin folder, and the respective libraries in /lib , after that i tested this python script : http://labrosa.ee.columbia.edu/millionsong/sites/default/files/tutorial1.py.txt
and it worked fine, to be clear the way i made it work was to first run the script and start installing the needed dependencies, but today i restarted my laptop and it didn't work, now it throws me this error on the console :
python2.7 script.py
returns :
import numpy as np # get it at: http://numpy.scipy.org/
from . import random
from .mtrand import *
ImportError: /home/francisco/.local/lib/python2.7/site-packages/numpy/random/mtrand.so: undefined symbol: PyFPE_jbuf
seems to me that there is a missing variable in such file, my guess is that the script is looking for the numpy library in the wrong place, since i made so many failed installations maybe i broke something and it only worked out because it was loaded in the temporal memory of the computer.
I tried installing Anaconda, and i created a new environment and installed the packaged with the anaconda package manager, and even thought i listed all packaged and it returns :
# packages in environment at /home/francisco/anaconda2/envs/Music:
#
biopython 1.66 np110py27_0
cython 0.23.4 <pip>
hdf5 1.8.15.1 2
mkl 11.3.1 0
numexpr 2.5 np110py27_0
numpy 1.10.4 py27_1
openssl 1.0.2g 0
pillow 3.1.1 <pip>
pip 8.1.1 py27_0
pytables 3.2.2 np110py27_1
python 2.7.11 0
python-ldap 2.4.25 <pip>
readline 6.2 2
reportlab 3.3.0 <pip>
requirements 0.1 <pip>
setuptools 20.3 py27_0
sqlite 3.9.2 0
tables 3.2.2 <pip>
tk 8.5.18 0
wheel 0.29.0 py27_0
zlib 1.2.8 0
i still get the same error. I really need help and don't know what else to try. Thanks.
I had the same problem, probably you have installed numpy without Anaconda, so there is a conflict because of this, which numpy to use: that one installed with pip or with conda. When I removed non-Anaconda numpy, error gone.
pip uninstall numpy
For cython users:
This post was helpful. The post explains, that there is some flag (--with-fpectl), which is either set during the compilation of cpython or not. When a library has been compiled using a cpython without that flag, it is incompatible to a version with that flag being set. This effect does only show up when you use cython, as numpy itself does not use this extension.
As further stated in that post, my Ubuntu 16.04 has been created with this flag and Conda without it. For me, it was specifically the module hmmlearn throwing the undefined symbol error. This must be the case because it was shipped by Ubuntu with the flag being set and not by anaconda. So I uninstalled hmmlearn and manually installed it anew from sourcescode (Anaconda, unfortunately, does not offer hmmlearn). --> Works!
irony at it's best, i restarted my laptop without doing nothing, and it worked. Can't understand why.
I agree with previous posts that this seems to be caused by having multiple versions of numpy installed. For me, it wasn't enough to just use pip, as I also had multiple versions of pip installed.
Specifying the specific pip solved the problem:
/usr/bin/pip3 uninstall numpy
Initially, I installed cython using system /usr/bin/pip and anconda pip. I uninstalled system cython using system pip and reinstalled using
conda install cython. Works now.
First remove numpy from /usr/local/lib/python2.7/dist-packages/numpy-1.11.0-py2.7-linux-x86_64.egg
and then use the following command
sudo pip install numpy scipy
I had solve this error in my case.
I ran into this problem in a particular situation. Using Anaconda (3 I think) I was creating a new environment. Previously I had created a py3 env with numpy, Not sure if related. But when creating my new py2.7 environment I went to install a particular package Ta-lib via pip, but then had this same Question's import error relating to numpy for the particular case of Ta-lib.
From this post Gaurav suggested the use of pip flag --no-cache-dir to ensure a rebuild during install of numpy. I uninstalled my Ta-lib and numpy and then reinstalled them with this flag via pip and everything worked great.