Develop Reference

"Illegal instruction" if "LD_LIBRARY_PATH" is set correctly when importing tensorflow

I have cuda 8.0, 9.0 and 9.1 installed on my system with /usr/local/cuda pointing to /usr/local/cuda-9.0 and if LD_LIBRARY_PATH is empty I get the typical ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory error when importing tensorflow (tensorflow-gpu==1.7.0 installed over pip).
If I set LD_LIBRARY_PATH and PATH according to the Nvidia documentation to LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64 and PATH=/usr/local/cuda-9.0/bin:... I get an Illegal instruction (core dumped) when importing tensorflow. The same thing happens if i set CUDA_HOME to /usr/local/cuda-9.0/.
cuDNN is version 7.0.4.
Can someone help me with this issue?

PyDev tells "error == cudaSuccess (35 vs. 0)CUDA driver version is insufficient for CUDA runtime version",but command line works fine

My configuration like this:
Ubuntu 16.04
Java 1.8
Python 2.7.12
Caffe 1.0
Cuda 8.0
Nvidia driver 375-66
PyDev 5.7.0.201704111357
And I tried to run this in bash:
https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation/tree/master/testing/python
by
python -m Demo
It works fine,but when try to run from eclipse-pydev,got this error:
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0606 09:34:43.905447 15924 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.
0
E0606 09:34:43.905640 15924 common.cpp:121] Cannot create Curand generator. Curand won't be available.
F0606 09:34:43.905845 15924 common.cpp:152] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version
*** Check failure stack trace: ***
I think this problem comes from that I start up OS with nvidia driver then switchs to intel like this:
This is what I want due to hope to use embeded intel VC to handle OS related work,and leave Nvidia VC for caffe(a deep learning framework) jobs.The
question is :
Why,for same python wrapped caffe job,does command line work fine but PyDev give the these error?

Usually this means that you have some environment variable in your command line that's not replicated in PyDev.
The usual fix is to launch Eclipse from the command line, so that it inherits the variables set there.

Thanks the tips from #Fabio Zadrozny
Window->Preferences->PyDev->Interpreters->Python Interpreter->Click Environment->New
create a env variable:
Name:LD_LIBRARY_PATH Value:as in you system env

Optimization failure in theano

I am using Fedora with the Anaconda Python environment. I have a 960m nvidia gpu, for which I have installed the required drivers and the CUDA toolkit. But when I try to run the theano tests, I end up getting the following error (in a huge error output):
EE.EEEERROR (theano.gof.opt): Optimization failure due to: constant_folding
ERROR (theano.gof.opt): node: DimShuffle{x}(TensorConstant{2})
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
I was trying to compile a simple function y, when I first saw the error. Searching for a solution led me to find that a lot of people had the same problem with the test function, but without any definite solutions. I followed the theano documentations and set the $CUDA_ROOT to my cuda root folder, but to no avail.
I'm using theano version 0.8.2 and Numpy 1.11.1, both from the conda repos. Seems like it is a GPU issue. But if it has problems, shouldn't it fallback to the CPU?
Any help would be highly appreciated. Thanks!

Inception v3 guide on tensorflow broken for C++ and python

I'm following the guide here on running the pretrained inception v3 https://www.tensorflow.org/versions/r0.11/tutorials/image_recognition/index.html
However, when I try the python version, I get:
python classify_image.py
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
Traceback (most recent call last):
File "classify_image.py", line 227, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
TypeError: run() got an unexpected keyword argument 'argv'
Ok.. Fine nevermind let me try the C++ Version.
Downloaded the model, run the bazel command:
➜ tensorflow git:(master) ✗ bazel build tensorflow/examples/label_image/...
.......
ERROR: /storage/git/tensorflow/tensorflow/tensorflow.bzl:636:21: syntax error at '=': expected expression.
ERROR: /storage/git/tensorflow/tensorflow/tensorflow.bzl:711:1: nested functions are not allowed. Move the function to top-level.
ERROR: /storage/git/tensorflow/tensorflow/tensorflow.bzl:739:1: nested functions are not allowed. Move the function to top-level.
ERROR: /storage/git/tensorflow/tensorflow/tensorflow.bzl:773:1: nested functions are not allowed. Move the function to top-level.
ERROR: /storage/git/tensorflow/tensorflow/tensorflow.bzl:776:1: nested functions are not allowed. Move the function to top-level.
ERROR: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package '': Extension 'tensorflow/tensorflow.bzl' has errors.
INFO: Elapsed time: 0.600s
...Okay then. Neither seems to work. Or perhaps I'm doing this wrong. Anyone has any guidance? :)
Using tensorflow 0.11 on Ubuntu 16, Anaconda distribution python 3.5
Thanks!

If it helps anyone:
Solving the C++ problem: Update Bazel to the correct version (you likely installed tensorflow ages ago and git pulled the latest which requires a new bazel version)
Solving the python problem: Remove the argv command.

Python pyopencl DLL load failed even with latest drivers

I've installed the latest CUDA and driver for my GPU. I'm using Python 2.7.10 on Win7 64bit.
I tried installing pyopencl from:
a. the unofficial windows binaries at http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyopencl
b. by compiling my own after getting the sources from https://pypi.python.org/pypi/pyopencl
The installation was successful on both cases but I get the same error message once I try to import it:
>>> import pyopencl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\pyopencl-2015.1-py2.7-win-amd64.egg\pyope
cl\__init__.py", line 30, in <module>
import pyopencl._cl as _cl
ImportError: DLL load failed: The specified procedure could not be found.
>>>
I have Visual C++ Redistributable for Visual Studio 2015 installed from https://www.microsoft.com/en-us/download/details.aspx?id=48145 .
I also tried with 2 different versions of the GPU driver (including latest). Same thing.
A lot of people seem to get the same error and on some forums I read that by updating the GPU drivers to latest, it works fine. But not for me.
Anyone knows how to fix this?

I'm affraid there isn't one right answer to this problem. Each case is different. It depends on what is installed in the OS.
To track down such problems I normally use Dependency Walker.
In this specific case I would open _cl.pyd (usually in C:\Python27\Lib\site-packages\pyopencl) in Dependency Walker to check if there aren't any missing dependencies or if for example OpenCL.dll is actually the one which should be used. OpenCL.dll may be installed by other programs and their path added to PATH. Also OpenCL.dll in System32 may be too old. Basically trial and error renaming all but one OpenCL.dll into OpenCL.dll.bak and/or removing paths from PATH may get you there.

I had this same problem and discovered it was caused by AMD OpenCL.dll not having a function introduced in OpenCL 2.1. The Gohlke site only has OpenCL 2.1 and 1.2, while AMD drivers support 2.0.
Because I wanted 2.0, the easy fix was to manually replace the AMD System32/OpenCL.dll with the one from Intel SDK with experimental 2.1 support.

I had the same problem here, the way I resolved it was:
Make sure you have downloaded and installed the right OpenCL SDK. For example
Intel
NVIDIA
Open the Windows Command Prompt cmd and set the LIB and INCLUDE environment variables. For example
Intel:
set INCLUDE=C:\Program Files (x86)\IntelSWTools\system_studio_2020\OpenCL\sdk\include
set LIB=C:\Program Files (x86)\IntelSWTools\system_studio_2020\OpenCL\sdk\lib\x64
NVIDIA:
set LIB=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v3.2\lib\x64
set INCLUDE=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v3.2\include
now run pip install pyopencl --no-cache-dir
open Python and test import pyopencl
there might be a way to install PyOpenCL via pipwin or by using the --global-option to set the include and library folders. But I haven't succeeded so far.
P.S. The above mentioned NVIDIA OpenCL SDK (i.e., CUDA toolkit) turns out to be very outdated. please don't use it. If you have that installed, please uninstall and install the newer versions.

Try both the versions 1.2 and 2.1 I was trying with later and got this issue. Switched the whl and it works but used the Intel GPU. NVidia OpenCL.dll is 2.0 and that is not working still.
Just checked the cl.get_platforms array and found them
0. Intel
1. NVidia
pyopencl.Platform Intel(R) OpenCL & pyopencl.Device Intel(R) Core(TM) ... Intel(R) OpenCL
pyopencl.Platform NVIDIA CUDA & pyopencl.Device Quadro ... NVIDIA CUDA

I had the same problem in my Lenovo yoga 720. It has NVidia Geforce GTX1050 and intel i7 630 CPU/GPU.
I installed a long time ago update drivers and SDK for Nvidia CUDA. But now I what to run python OpenGL and I install intel SDK also. Pip install pyopencl without problems but import pyopengl give me dll load failure.
Solution was to change Windows\system32\opencl.dll to a new one. The old one was NVidia signed (you can see it in properties of file opencl.dll). The new one is Microsoft signed version 2.1.1.0 Khronos OpenCL ICD
I hope this is useful for you. Solution arrived after a long time trying a lot of things... but nothing worked except the new opencl.dll file