I am trying to build pytorch v1.4.0 from source because I need it for another module.
I have succeeded in building pytorch from source but when I try to run the intended python script I get this error:
RuntimeError: PyTorch was compiled without NumPy support
So I looked up what I did wrong and it turns out I needed to install numpy before i build pytorch from source, so thats what I did.
I used the command:
pip3 install numpy==1.19.4
When installing pytorch the console gives me a lot of information, including the build settings. The build settings looked like this:
-- Compile definitions : ONNX_ML=1;ONNX_NAMESPACE=onnx_torch;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
-- CMAKE_PREFIX_PATH : /home/elvygcp/venv/lib/python3.6/site-packages;/usr/local/cuda
-- CMAKE_INSTALL_PREFIX : /home/elvygcp/venv/pytorch-1.4.0/torch
--
-- TORCH_VERSION : 1.4.0
-- CAFFE2_VERSION : 1.4.0
-- BUILD_CAFFE2_MOBILE : ON
-- USE_STATIC_DISPATCH : OFF
-- BUILD_BINARY : OFF
-- BUILD_CUSTOM_PROTOBUF : ON
-- Link local protobuf : ON
-- BUILD_DOCS : OFF
-- BUILD_PYTHON : True
-- Python version : 3.6.9
-- Python executable : /home/elvygcp/venv/bin/python3
-- Pythonlibs version : 3.6.9
-- Python library : /usr/lib/libpython3.6m.so.1.0
-- Python includes : /usr/include/python3.6m
-- Python site-packages: lib/python3.6/site-packages
-- BUILD_CAFFE2_OPS : ON
-- BUILD_SHARED_LIBS : ON
-- BUILD_TEST : True
-- BUILD_JNI : OFF
-- INTERN_BUILD_MOBILE :
-- USE_ASAN : OFF
-- USE_CUDA : ON
-- CUDA static link : OFF
-- USE_CUDNN : OFF
-- CUDA version : 10.2
-- CUDA root directory : /usr/local/cuda
-- CUDA library : /usr/local/cuda/lib64/stubs/libcuda.so
-- cudart library : /usr/local/cuda/lib64/libcudart.so
-- cublas library : /usr/lib/x86_64-linux-gnu/libcublas.so
-- cufft library : /usr/local/cuda/lib64/libcufft.so
-- curand library : /usr/local/cuda/lib64/libcurand.so
-- nvrtc : /usr/local/cuda/lib64/libnvrtc.so
-- CUDA include path : /usr/local/cuda/include
-- NVCC executable : /usr/local/cuda/bin/nvcc
-- CUDA host compiler : /usr/bin/cc
-- USE_TENSORRT : OFF
-- USE_ROCM : OFF
-- USE_EIGEN_FOR_BLAS : ON
-- USE_FBGEMM : ON
-- USE_FFMPEG : OFF
-- USE_GFLAGS : OFF
-- USE_GLOG : OFF
-- USE_LEVELDB : OFF
-- USE_LITE_PROTO : OFF
-- USE_LMDB : OFF
-- USE_METAL : OFF
-- USE_MKL : OFF
-- USE_MKLDNN : ON
-- USE_MKLDNN_CBLAS : OFF
-- USE_NCCL : ON
-- USE_SYSTEM_NCCL : OFF
-- USE_NNPACK : ON
-- USE_NUMPY : OFF
-- USE_OBSERVERS : ON
-- USE_OPENCL : OFF
-- USE_OPENCV : OFF
-- USE_OPENMP : ON
-- USE_TBB : OFF
-- USE_PROF : OFF
-- USE_QNNPACK : ON
-- USE_REDIS : OFF
-- USE_ROCKSDB : OFF
-- USE_ZMQ : OFF
-- USE_DISTRIBUTED : ON
-- USE_MPI : OFF
-- USE_GLOO : ON
-- BUILD_NAMEDTENSOR : OFF
There are 2 things I do not understand:
The line: USE_CUDNN : OFF, which I think is strange since i followed the pytorch build instructions from their github page which mentions I need CuDNN to build pytorch from source, and CuDNN is installed on my system.
The line: USE_NUMPY : OFF, I dont know why but pytorch does not recognize numpy.
My system:
OS: Ubuntu 18.04 LTS
Cuda version: 10.2
CuDNN version: 8
python venv in Google Cloud Compute Engine VM
How I build pytorch 1.4.0 from source:
git clone --branch v1.4.0 https://github.com/pytorch/pytorch.git pytorch-1.4.0
cd pytorch-1.4.0/
git submodule update --init --recursive
sudo apt install cmake -y
sudo apt-get update
cd ../
sudo apt install python3-venv -y
python3 -m venv venv/
cd venv
source bin/activate
cd pytorch-1.4.0/
pip install pyyaml
python3 setup.py install
cd ../
git clone --branch v0.5.0 https://github.com/pytorch/vision.git torchvision-0.5.0
cd torchvision-0.5.0/
python3 setup.py install
cd ../
If anyone could tell how I make pytorch recognize and build with numpy I'd be very grateful.
If I need to supply more information I'd be happy to.
Okay so I don't exactly know what the solution was since I did two things:
I installed a lower version of numpy since my torch version is also a bit older.
I cleared all the pytorch install cache with the command: sudo USE_ROCM=1 USE_LMDB=1 USE_OPENCV=1 MAX_JOBS=15 python3 setup.py clean
Downgrading numpy might have been unnecessary, since I can't remember if I cleared all the pytorch installation cache after installing numpy and trying again.
Now my installation log contains USE_NUMPY : ON.
Dont know if this fixed all my problems, building pytorch takes forever so I'll just have to wait and see, but at least it fixed this one :)
I am trying to install nlopt onto macOS 10.15.5. I downloaded the nlopt-2.6.2.tar.gz file from the NLopt docs and ran the following from the nlopt-2.6.2 directory:
mkdir build
cd build
cmake -DNLOPT_OCTAVE=Off -DNLOPT_MATLAB=Off -DNLOPT_GUILE=Off ..
make
sudo make install
I got the following output: cmake.txt.
The header file (nlopt.h) installs correctly to /usr/local/include and the dynamic library (libnlopt.dylib) installs correctly to /usr/local/lib/, but neither the dist-info file nor the nlopt module itself installs.
I have also tried installing via pip, brew, and conda, none of which have worked. I have also tried cloning from this Github, which didn't work either.
I appreciate any help with this, because I am completely lost. I am relatively new to this kind of stuff, and I couldn't find any good answers online.
The official docs are somewhat laconic about the exact steps required for building nlopt with Python bindings. First of all, you'll need SWIG installed:
$ brew install swig
Then, you'll need numpy to be available for the target Python interpreter. It is already preinstalled for the system Python, otherwise install it via Homebrew or pip, depending on your Python installation.
Now run cmake:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF
This will build bindings against the default Python 2.7 installation preinstalled on MacOS. If you need to build against custom Python installation (e.g. when you've installed Python 3 via Homebrew or PKG installer from https://www.python.org/downloads), pass it via the PYTHON_EXECUTABLE arg:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF -DPYTHON_EXECUTABLE=/usr/local/bin/python3
Inspect the log now - Python, SWIG and numpy headers should be successfully located. Example output snippet (you may have different paths/versions printed):
-- Found PythonInterp: /usr/local/bin/python3.8 (found version "3.8.3")
-- Found PythonLibs: /Library/Frameworks/Python.framework/Versions/3.8/lib/libpython3.8.dylib (found suitable exact version "3.8.3")
-- Found NumPy: /Users/hoefling/Library/Python/3.8/lib/python/site-packages/numpy/core/include (found version "1.19")
-- Found SWIG: /usr/local/bin/swig (found version "4.0.2")
If any of those conditions is not satisfied (e.g. you see Could NOT find NumPy, Could NOT find PythonLibs or Could NOT find SWIG), then stop and make sure the configuration succeeds before proceeding next.
Now compile:
$ make
...
Scanning dependencies of target nlopt_python_swig_compilation
[ 96%] Swig compile nlopt.i for python
[ 96%] Built target nlopt_python_swig_compilation
Scanning dependencies of target nlopt_python
[ 98%] Building CXX object src/swig/CMakeFiles/nlopt_python.dir/CMakeFiles/nlopt_python.dir/nloptPYTHON_wrap.cxx.o
[100%] Linking CXX shared module _nlopt.so
[100%] Built target nlopt_python
Install:
$ make install
...
-- Installing: /usr/local/lib/python3.8/site-packages/nlopt.py
-- Installing: /usr/local/lib/python3.8/site-packages/_nlopt.so
Test the Python bindings are importable:
$ python -c "import nlopt; print(nlopt.__version__)"
2.6.2
I have downloaded the official conda recipe of opencv in AnacondaRecipes.
I have tried to build this package executing:
conda-build recipe -c conda-forge
I am getting the following error when the recipe compiles opencv, when doing [ 72%] Built target opencv_dnn . The error is the following:
[ 67%] Building CXX object modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/src/grfmt_png.cpp.o
/opt/conda/conda-bld/opencv_1521187259162/work/modules/imgcodecs/src/grfmt_png.cpp:62:10: fatal error: libpng/png.h: No such file or directory
#include <libpng/png.h>
^~~~~~~~~~~~~~
compilation terminated.
modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/build.make:326: recipe for target 'modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/src/grfmt_png.cpp.o' failed
make[2]: *** [modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/src/grfmt_png.cpp.o] Error 1
CMakeFiles/Makefile2:4645: recipe for target 'modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/all' failed
make[1]: *** [modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Lookin in the $PREFIX directory, there is not libpng folder, only a libpng16 folder:
/opt/conda/conda-bld/opencv_1521187259162/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include/
I suspect it must be something related to how conda-build manages the path environment, but I do not have any clue of how to solve it.
Environment: conda 4.4.11, OS: Ubuntu 16.04.
UPDATE 23/03/2018
I have also tried:
add libpng/png.h to the location pointed by the $PREFIX directory.
add /usr/include to the $PATH
No success in either case.
UPDATE 04/03/2018
libpng-dev is installed in the environment.
UPDATE 09/04/2018
Docker recipe to reproduce environment
UPDATE 12/04/2018
Docker recipe using miniconda3 and installing conda in /usr/local/conda instead of /opt/conda
So the issue you have here is that you need the libpng-dev library installed on your OS. Since you mentioned you're operating in a Ubuntu env, you can simply type
sudo apt-get install libpng-dev
Hope this resolves the issue!
UPDATE
Why didn't you try to install opencv with the commands provided in the link:
conda config --add channels conda-forge
conda install opencv
When I type gcc -v, I get :
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-apple-darwin14.0.0/4.9.2/lto-wrapper
Target: x86_64-apple-darwin14.0.0
Configured with: ../gcc-4.9-20141029/configure --enable-languages=c++,fortran
Thread model: posix
gcc version 4.9.2 20141029 (prerelease) (GCC)
So my gcc version is 4.9.2
However, when I try to install gcc again by typing : brew tap homebrew/versions; brew install gcc --without-multilib
I get :
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 3 taps (homebrew/core, homebrew/science, homebrew/versions).
==> New Formulae
git-quick-stats homebrew/science/med-file ktmpl
==> Updated Formulae
aha homebrew/science/bowtie mpd
akka homebrew/science/cdo mysql-cluster
arangodb homebrew/science/matplotlib no-more-secrets
buku homebrew/science/packmol nvm
bullet homebrew/science/paraview protobuf-swift
casperjs homebrew/science/pymol qca
certigo imagemagick ✔ ringojs
chicken imagemagick#6 rust
cjdns jenkins skinny
coffeescript jigdo sqldiff
compcert jump sqlite-analyzer
crystal-lang kobalt svtplay-dl
dbhash libhdhomerun swiftgen
digdag libphonenumber vim
direnv libsigsegv wakatime-cli
elixirscript mapserver wolfssl
embulk mecab-jumandic yank
geoserver mkvtoolnix yaz
gitlab-ci-multi-runner mosquitto
==> Deleted Formulae
bip homebrew/versions/openssl101 probatron4j
edelta lcab rtpbreak
esound malaga
Updating Homebrew...
Warning: gcc-6.3.0_1 already installed
This output says at the end that I have gcc 6.3 .
Am I working with gcc 4.9.2 or gcc 6.3 ?
P.S : If you are interested in why I executed the brew tap & brew install commands, it is because I am trying to install xgboost in python. In the github repo of xgboost, they're advising to run the above commands before installation to get the latest g++ compiler.
To address your question "Am I working with gcc 4.9.2 or gcc 6.3"
You are still working with gcc 4.9.2.
How can you make gcc-6 your default compiler? See below:
sudo ln -s <path-to>/gcc-6 <path-to>/gcc
Reload and verify
Now exit the terminal session and open a new terminal (so the links update) and type gcc -v and see if the version is now the latest.
Other related tools
NOTICE: You may also wish to update c++ -> c++-6, g++ -> g++-6, and cpp -> cpp-6 in a similar fashion, just repeat the above step for each in turn, ALWAYS BACKUP in case something goes wrong so you can undo everything if necessary.
NOTE:
If you get a "Operation not permitted" warning and you are using El Capitan or newer version of Mac OS X please see this question/answer on how to disable the security feature that is preventing you from making changes to /usr/bin directory:
https://superuser.com/questions/933019/sudo-cant-create-file-in-usr-bin-in-el-capitan
Getting this error:
sudo: unable to resolve host coderw#ll
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108
(message):
Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE)
Call Stack (most recent call first):
/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-2.8/Modules/FindPythonInterp.cmake:139
(FIND_PACKAGE_HANDLE_STANDARD_ARGS)
Code/cmake/Modules/FindNumPy.cmake:10 (find_package)
CMakeLists.txt:114 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeOutput.log".
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeError.log".
I have already installed:
sudo apt-get install python-dev
Environment variable are already set as follow:
PYTHON_INCLUDE_DIRS=/usr/include/python2.7
PYTHON_LIBRARIES=/usr/lib/python2.7/config/libpython2.7.so
Location of python.h : /usr/lib/include/python2.7/python.h
Location of python libs: /usr/lib/python2.7/
How to solve this?
You can fix the errors by appending to the cmake command the -DPYTHON_LIBRARY and -DPYTHON_INCLUDE_DIR flags filled with the respective folders.
Thus, the trick is to fill those parameters with the returned information from the python interpreter, which is the most reliable. This may work independently of your python location/version (also for Anaconda users):
$ cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "import sysconfig; print(sysconfig.get_path('include'))") \
-DPYTHON_LIBRARY=$(python -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR'))")
If the version of python that you want to link against cmake is Python3.X and the default python symlink points to Python2.X, python3 -c ... can be used instead of python -c ....
In case that the error persists, you may need to update the cmake to a higher version as stated by #pdpcosta and repeat the process again.
For me this is helpful:
# if using python2
apt-get install python-dev
# if using python3
apt-get install python3-dev
I hit the same issue,and discovered the error message gives misleading variable names. Try setting the following (singular instead of plural):
PYTHON_INCLUDE_DIR=/usr/include/python2.7
PYTHON_LIBRARY=/usr/lib/python2.7/config/libpython2.7.so
The (plural) variables you see error messages about are values that the PythonLibs sets up when it is initialised correctly.
Even after adding -DPYTHON_INCLUDE_DIR and -DPYTHON_LIBRARY as suggested above, I was still facing the error Could NOT find PythonInterp. What solved it was adding -DPYTHON_EXECUTABLE:FILEPATH= to cmake as suggested in https://github.com/pybind/pybind11/issues/99#issuecomment-182071479:
cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
-DPYTHON_LIBRARY=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))") \
-DPYTHON_EXECUTABLE:FILEPATH=`which python`
I was facing this problem while trying to compile OpenCV 3 on a Xubuntu 14.04 Thrusty Tahr system.
With all the dev packages of Python installed, the configuration process was always returning the message:
Could NOT found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.6", minimum required is "2.7")
Could NOT find PythonLibs (missing: PYTHON_INCLUDE_DIRS) (found suitable exact version "2.7.6")
Found PythonInterp: /usr/bin/python3.4 (found suitable version "3.4", minimum required is "3.4")
Could NOT find PythonLibs (missing: PYTHON_LIBRARIES) (Required is exact version "3.4.0")
The CMake version available on Thrusty Tahr repositories is 2.8.
Some posts inspired me to upgrade CMake.
I've added a PPA CMake repository which installs CMake version 3.2.
After the upgrade everything ran smoothly and the compilation was successful.
Some last version of Ubuntu installs Python 3.4 by default and the CMake version from Ubuntu (2.8) only searches up to Python 3.3.
Try to add set(Python_ADDITIONAL_VERSIONS 3.4) before the find_package statement.
Remember to clean CMakeCache.txt too.
This problem can also happen in Windows. Cmake looks into the registry and sometimes python values are not set. For those with similar problem:
http://ericsilva.org/2012/10/11/restoring-your-python-registry-in-windows/
Just create a .reg file to set the necessary keys and edit accordingly to match your setup.
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Python]
[HKEY_CURRENT_USER\Software\Python\Pythoncore]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\InstallPath]
#="C:\\python26"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\PythonPath]
#="C:\\python26;C:\\python26\\Lib\\;C:\\python26\\DLLs\\"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\InstallPath]
#="C:\\python27"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\PythonPath]
#="C:\\python27;C:\\python27\\Lib\\;C:\\python27\\DLLs\\"
Note that if you are using cMake version 3.12 or later, variable PythonInterp and PythonLibs has been changed into Python.
So we use:
find_package(Python ${PY_VERSION} REQUIRED)
instead of:
find_package(PythonInterp ${PY_VERSION} REQUIRED)
find_package(PythonLibs ${PY_VERSION} REQUIRED)
see https://cmake.org/cmake/help/v3.12/module/FindPython.html for details.
I had upgraded to python3.8 on my system and had an incomplete install. Managed to fix it by installing the rest of the 3.8 packages:
sudo apt-get install python3.8 python3.8-dev python3.8-distutils python3.8-venv
Paste this into your CMakeLists.txt:
# find python
execute_process(COMMAND python-config --prefix OUTPUT_VARIABLE PYTHON_SEARCH_PATH)
string(REGEX REPLACE "\n$" "" PYTHON_SEARCH_PATH "${PYTHON_SEARCH_PATH}")
file(GLOB_RECURSE PYTHON_DY_LIBS ${PYTHON_SEARCH_PATH}/lib/libpython*.dylib ${PYTHON_SEARCH_PATH}/lib/libpython*.so)
if (PYTHON_DY_LIBS)
list(GET PYTHON_DY_LIBS 0 PYTHON_LIBRARY)
message("-- Find shared libpython: ${PYTHON_LIBRARY}")
else()
message(WARNING "Cannot find shared libpython, try find_package")
endif()
find_package(PythonInterp)
find_package(PythonLibs ${PYTHON_VERSION_STRING} EXACT)
In case that might help, I found a workaround for a similar problem, looking at the cmake doc :
https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html
You must set two env vars for cmake to find coherent versions.
Unfortunately this is not a generic solution...
cmake -DPYTHON_LIBRARY=${HOME}/.pyenv/versions/3.8.0/lib/libpython3.8.a -DPYTHON_INCLUDE_DIR=${HOME}/.pyenv/versions/3.8.0/include/python3.8/ cern_root/
In Python 3.2 and onward distutils.sysconfig is deprecated in favor of sysconfig.
To get all the variable names in data structure and inspect the situation we can use get_paths function
import sysconfig
sysconfig.get_paths()
which will return us a dict with all the relevant variable names as keys, and corresponding paths as values.
When we know the key we can get the value dynamically,
>>> sysconfig.get_path("include")
'C:\\Program Files\\Python310\\Include'
However, the most convenient feature of sysutils for this situation is that it can list all the variables at once from the command-prompt by invoking python -m sysconfig:
$ python -m sysconfig
Platform: "macosx-10.4-i386"
Python version: "3.2"
Current installation scheme: "posix_prefix"
Paths:
data = "/usr/local"
include = "/Users/tarek/Dev/svn.python.org/py3k/Include"
platinclude = "."
platlib = "/usr/local/lib/python3.2/site-packages"
platstdlib = "/usr/local/lib/python3.2"
purelib = "/usr/local/lib/python3.2/site-packages"
scripts = "/usr/local/bin"
stdlib = "/usr/local/lib/python3.2"
Variables:
AC_APPLE_UNIVERSAL_BUILD = "0"
AIX_GENUINE_CPLUSPLUS = "0"
AR = "ar"
ARFLAGS = "rc"
...
Maybe below command line works for you if all of above methods didn't work.
sudo apt-get install cython cython3
On Ubuntu 20 at least, since Python 2.7 is deprecated, you can:
sudo apt install python2.7-dev
And then PYTHON_INCLUDE_DIR can be set to /usr/include/python2.7