Develop Reference

How to use the Python version active in an Anaconda environment in CMake builds?

I'm building OpenVDB on Windows 10 with Python support, but I need to compile it against Python 3.7.
The system-wide version of Python is 3.10 and the build works fine with that.
I use Anaconda to set up a Py3.7 environment with
conda create -n py37 python=3.7
Then I modify the CMakeLists.txt to look for this version of Python:
find_package(Python 3.7...<3.8 REQUIRED COMPONENTS Development.Module Interpreter)
In this environment, without any other changes, CMake still finds the system version of Python:
CMake Error at C:/Program Files/CMake/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python: Found unsuitable version "3.10.2", required range is
"3.7...<3.8" (found
C:/dev/vcpkg/installed/x64-windows/tools/python3/python.exe, found
components: Development.Module Interpreter)
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:592 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/CMake/share/cmake-3.22/Modules/FindPython/Support.cmake:3172 (find_package_handle_standard_args)
C:/Program Files/CMake/share/cmake-3.22/Modules/FindPython.cmake:519 (include)
C:/dev/vcpkg/installed/x64-windows/share/python/vcpkg-cmake-wrapper.cmake:67 (_find_package)
C:/dev/vcpkg/scripts/buildsystems/vcpkg.cmake:761 (include)
openvdb/openvdb/python/CMakeLists.txt:120 (find_package)
Adding -DPython_EXECUTABLE=C:\Users\me\Anaconda3\envs\py37\python.exe at command line helps with finding the right the Python version, but there's still trouble with CMake:
CMake Error at C:/Program Files/CMake/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python (missing: Development.Module) (found suitable version
"3.7.13", required range is "3.7...<3.8")
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files/CMake/share/cmake-3.22/Modules/FindPython/Support.cmake:3172 (find_package_handle_standard_args)
C:/Program Files/CMake/share/cmake-3.22/Modules/FindPython.cmake:519 (include)
C:/dev/vcpkg/installed/x64-windows/share/python/vcpkg-cmake-wrapper.cmake:67 (_find_package)
C:/dev/vcpkg/scripts/buildsystems/vcpkg.cmake:761 (include)
openvdb/openvdb/python/CMakeLists.txt:120 (find_package)
What is exactly Development.Module, and is it really missing from the environment?
If yes, then how do I install it?
If no, then what do I need to add to the CMake scripts or command line to make it work?

Dlib installing error: Could NOT find Boost

(Please, excuse my bad English!) Got into trap while trying to install dlib library on my computer(Windows 10 Pro, 64x), but all the time get the same error: Could NOT find Boost
What I have already done:
Installed Visual Studio Community 2017 version 15.1
Installed Anaconda: Python 3.6.0
|Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Downloaded boost_1_64_0.7z and unpacked it
Ran bootstrap
b2 -a --with-python address-model=64 toolset=msvc runtime-link=static
got the message: The Boost C++ Libraries were successfully built!
6.Next:
set BOOST_LIBRARYDIR=C:\Users\Lelek\boost_1_64_0\stage\lib
set BOOST_INCLUDEDIR=C:\Users\Lelek\boost_1_64_0
set BOOST_ROOT=C:\Users\Lelek\boost_1_64_0
set PATH=%PATH%;%BOOST_ROOT%;%BOOST_LIBRARYDIR%
7. On this stage if I try to run python setup.py install from dlib directory I get the output (the same error appears when try to call cmake -G "Visual Studio 15 2017 Win64" -DPYTHON3=1 .\tools\python):
CMake Warning at C:/Program Files/Cmake/share/cmake-3.8/Modules/FindBoost.cmake:1521 (message):
No header defined for python-py34; skipping header check
Call Stack (most recent call first):
C:/Users/Lelek/dlib-19.4/dlib/cmake_utils/add_python_module:61(FIND_PACKAGE)
CMakeLists.txt:6 (include)
-- Could NOT find Boost
CMake Warning at C:/Program Files/Cmake/share/cmake-3.8/Modules/FindBoost.cmake:1521 (message):
No header defined for python-py35; skipping header check
Call Stack (most recent call first):
C:/Users/Lelek/dlib-19.4/dlib/cmake_utils/add_python_module:63 (FIND_PACKAGE)
CMakeLists.txt:6 (include)
-- Could NOT find Boost
CMake Warning at C:/Program Files/Cmake/share/cmake-3.8/Modules/FindBoost.cmake:1521 (message):
No header defined for python3; skipping header check
Call Stack (most recent call first):
C:/Users/Lelek/dlib-19.4/dlib/cmake_utils/add_python_module:66 (FIND_PACKAGE)
CMakeLists.txt:6 (include)
-- Could NOT find Boost
-- Could NOT find Boost
-- Found PythonLibs: C:/ProgramData/Anaconda3/libs/python36.lib (found suitable version "3.6.0", minimum required is "3.4")
-- We couldn't find the right version of boost python. If you installed boost and you are still getting this error then you might have installed a version of boost that was compiled with a different version of visual studio than the one you are using. So you have to make sure that the version of visual studio is the same version that was used to compile the copy of boost you are using.
-- Set the BOOST_ROOT and BOOST_LIBRARYDIR environment variables before running cmake.
-- E.g. Something like this:
-- set BOOST_ROOT=C:\local\boost_1_57_0
-- set BOOST_LIBRARYDIR=C:\local\boost_1_57_0\stage\lib
--
-- You will also likely need to compile boost yourself rather than using one of the precompiled
-- windows binaries. Do this by going to the folder tools\build\ within boost and running
-- bootstrap.bat. Then run the command:
-- b2 install
-- And then add the output bin folder to your PATH. Usually this is the C:\boost-build-engine\bin
-- folder. Finally, go to the boost root and run a command like this:
-- b2 -a --with-python address-model=64 toolset=msvc runtime-link=static
-- When it completes, set BOOST_LIBRARYDIR equal to wherever b2 put the compiled libraries.
-- Note that you will need to set the address-model based on if you want a 32 or 64bit python library.
--
-- Next, when you invoke cmake to compile dlib you may have to use cmake's -G option to set the
-- 64 vs. 32bit mode of visual studio. Also, if you want a Python3 library you will need to
-- add -DPYTHON3=1. You do this with a statement like:
-- cmake -G "Visual Studio 12 2013 Win64" -DPYTHON3=1 ..\..\tools\python
-- Rather than:
-- cmake ..\..\tools\python
-- Which will build a 32bit Python2 module by default on most systems.
--
--
CMake Error at C:/Users/Lelek/dlib-19.4/dlib/cmake_utils/add_python_module:116 (message):
Boost python library not found.
Call Stack (most recent call first):
CMakeLists.txt:6 (include)
-- Configuring incomplete, errors occurred!
See also "C:/Users/Lelek/dlib-19.4/tools/python/build/CMakeFiles/CMakeOutput.log".
error: cmake configuration failed!

As usual for me, I found the answer, as soon as I asked question. So, if someone will get the same issue:
To solve his problem I deleted Anaconda and installed only Python(3.5.0). Then updated dlib using PyCharm. And again, excuse my bad English, please.

If anyone still has this problem, try changing all calls to FIND_PACKAGE inside dlib\cmake_utils\add_python_module from 'Boost' to 'boost'. Worked for me.
If, after finding boost, the installation tries to open python27.lib and fails:
Remove boost.
Repeat steps 2--6 from the original question AFTER activating your py36 environment in Anaconda.
Re-run the dlib setup.
The latter was a stupid mistake of mine, but just in case.

Installing bsddb3 6.1.1 in Windows: FileNotFoundError: 'db/include\\db.h'

I'm running Windows 7 x64, with Python 3.4. When I run pip install bsddb3 I get:
λ pip install bsddb3
Collecting bsddb3
Using cached bsddb3-6.1.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "C:\Users\User\AppData\Local\Temp\pip-build-soqf0_qb\bsddb3\setup.py", line 42, in <module>
import setup3
File "C:\Users\User\AppData\Local\Temp\pip-build-soqf0_qb\bsddb3\setup3.py", line 375, in <module>
with open(os.path.join(incdir, 'db.h'), 'r') as f :
FileNotFoundError: [Errno 2] No such file or directory: 'db/include\\db.h'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\User\AppData\Local\Temp\pip-build-soqf0_qb\bsddb3
So I tried downloading the sources for bsddb3. I made sure I had the BSD DB Windows binaries from Oracle installed; I downloaded them from Berkeley DB 6.1.26.msi. Moving the include and lib directories from the DB install to a /db directory in the bsddb3 folder fixed the problem of not being able to find the libraries. But then the setup failed on a missing variable that's in the posix section but not properly declared in the Windows section.
Fixing that, and a couple of other adjustments, and python setup.py build actually ran the build, but after a bunch of warnings about unsafe conversions it failed with an error:
warning: I don't know what to do with 'runtime_library_dirs': ['db/lib']
error: don't know how to set runtime library search path for MSVC++
I'm not sure what to do next. Other than rewrite the library I was originally trying to use so that it uses SQLAlchemy instead or something.

One don't really want to spend couple of nights compiling that on Windows, so the best option is to use pre-built binary from Unofficial Windows Binaries for Python Extension Packages.
Download the package for your Python version (cpXX part of the name) and architecture (win_amd64 or win32), like bsddb3-6.1.1-cp34-none-win_amd64.whl, and install it with pip:
pip install bsddb3-6.1.1-cp34-none-win_amd64.whl

Based on cyberj0g's answer.
https://www.lfd.uci.edu/~gohlke/pythonlibs/#bsddb3
pip install bsddb3-6.1.1-cp34-none-win_amd64.whl
Note: the cp part is corresponding to the CPython version.
which means cp36 is for CPython 3.6.And divided into 32-bit and 64-bit versions for windows
bsddb3‑6.2.5‑cp36‑cp36m‑win32.whl
bsddb3‑6.2.5‑cp36‑cp36m‑win_amd64.whl

Cmake is not able to find Python-libraries

Getting this error:
sudo: unable to resolve host coderw#ll
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108
(message):
Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE)
Call Stack (most recent call first):
/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-2.8/Modules/FindPythonInterp.cmake:139
(FIND_PACKAGE_HANDLE_STANDARD_ARGS)
Code/cmake/Modules/FindNumPy.cmake:10 (find_package)
CMakeLists.txt:114 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeOutput.log".
See also "/home/coderwall/Desktop/rdkit/build/CMakeFiles/CMakeError.log".
I have already installed:
sudo apt-get install python-dev
Environment variable are already set as follow:
PYTHON_INCLUDE_DIRS=/usr/include/python2.7
PYTHON_LIBRARIES=/usr/lib/python2.7/config/libpython2.7.so
Location of python.h : /usr/lib/include/python2.7/python.h
Location of python libs: /usr/lib/python2.7/
How to solve this?

You can fix the errors by appending to the cmake command the -DPYTHON_LIBRARY and -DPYTHON_INCLUDE_DIR flags filled with the respective folders.
Thus, the trick is to fill those parameters with the returned information from the python interpreter, which is the most reliable. This may work independently of your python location/version (also for Anaconda users):
$ cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "import sysconfig; print(sysconfig.get_path('include'))") \
-DPYTHON_LIBRARY=$(python -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR'))")
If the version of python that you want to link against cmake is Python3.X and the default python symlink points to Python2.X, python3 -c ... can be used instead of python -c ....
In case that the error persists, you may need to update the cmake to a higher version as stated by #pdpcosta and repeat the process again.

For me this is helpful:
# if using python2
apt-get install python-dev
# if using python3
apt-get install python3-dev

I hit the same issue,and discovered the error message gives misleading variable names. Try setting the following (singular instead of plural):
PYTHON_INCLUDE_DIR=/usr/include/python2.7
PYTHON_LIBRARY=/usr/lib/python2.7/config/libpython2.7.so
The (plural) variables you see error messages about are values that the PythonLibs sets up when it is initialised correctly.

Even after adding -DPYTHON_INCLUDE_DIR and -DPYTHON_LIBRARY as suggested above, I was still facing the error Could NOT find PythonInterp. What solved it was adding -DPYTHON_EXECUTABLE:FILEPATH= to cmake as suggested in https://github.com/pybind/pybind11/issues/99#issuecomment-182071479:
cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
-DPYTHON_LIBRARY=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))") \
-DPYTHON_EXECUTABLE:FILEPATH=`which python`

I was facing this problem while trying to compile OpenCV 3 on a Xubuntu 14.04 Thrusty Tahr system.
With all the dev packages of Python installed, the configuration process was always returning the message:
Could NOT found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.6", minimum required is "2.7")
Could NOT find PythonLibs (missing: PYTHON_INCLUDE_DIRS) (found suitable exact version "2.7.6")
Found PythonInterp: /usr/bin/python3.4 (found suitable version "3.4", minimum required is "3.4")
Could NOT find PythonLibs (missing: PYTHON_LIBRARIES) (Required is exact version "3.4.0")
The CMake version available on Thrusty Tahr repositories is 2.8.
Some posts inspired me to upgrade CMake.
I've added a PPA CMake repository which installs CMake version 3.2.
After the upgrade everything ran smoothly and the compilation was successful.

Some last version of Ubuntu installs Python 3.4 by default and the CMake version from Ubuntu (2.8) only searches up to Python 3.3.
Try to add set(Python_ADDITIONAL_VERSIONS 3.4) before the find_package statement.
Remember to clean CMakeCache.txt too.

This problem can also happen in Windows. Cmake looks into the registry and sometimes python values are not set. For those with similar problem:
http://ericsilva.org/2012/10/11/restoring-your-python-registry-in-windows/
Just create a .reg file to set the necessary keys and edit accordingly to match your setup.
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Python]
[HKEY_CURRENT_USER\Software\Python\Pythoncore]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\InstallPath]
#="C:\\python26"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.6\PythonPath]
#="C:\\python26;C:\\python26\\Lib\\;C:\\python26\\DLLs\\"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7]
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\InstallPath]
#="C:\\python27"
[HKEY_CURRENT_USER\Software\Python\Pythoncore\2.7\PythonPath]
#="C:\\python27;C:\\python27\\Lib\\;C:\\python27\\DLLs\\"

Note that if you are using cMake version 3.12 or later, variable PythonInterp and PythonLibs has been changed into Python.
So we use:
find_package(Python ${PY_VERSION} REQUIRED)
instead of:
find_package(PythonInterp ${PY_VERSION} REQUIRED)
find_package(PythonLibs ${PY_VERSION} REQUIRED)
see https://cmake.org/cmake/help/v3.12/module/FindPython.html for details.

I had upgraded to python3.8 on my system and had an incomplete install. Managed to fix it by installing the rest of the 3.8 packages:
sudo apt-get install python3.8 python3.8-dev python3.8-distutils python3.8-venv

Paste this into your CMakeLists.txt:
# find python
execute_process(COMMAND python-config --prefix OUTPUT_VARIABLE PYTHON_SEARCH_PATH)
string(REGEX REPLACE "\n$" "" PYTHON_SEARCH_PATH "${PYTHON_SEARCH_PATH}")
file(GLOB_RECURSE PYTHON_DY_LIBS ${PYTHON_SEARCH_PATH}/lib/libpython*.dylib ${PYTHON_SEARCH_PATH}/lib/libpython*.so)
if (PYTHON_DY_LIBS)
list(GET PYTHON_DY_LIBS 0 PYTHON_LIBRARY)
message("-- Find shared libpython: ${PYTHON_LIBRARY}")
else()
message(WARNING "Cannot find shared libpython, try find_package")
endif()
find_package(PythonInterp)
find_package(PythonLibs ${PYTHON_VERSION_STRING} EXACT)

In case that might help, I found a workaround for a similar problem, looking at the cmake doc :
https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html
You must set two env vars for cmake to find coherent versions.
Unfortunately this is not a generic solution...
cmake -DPYTHON_LIBRARY=${HOME}/.pyenv/versions/3.8.0/lib/libpython3.8.a -DPYTHON_INCLUDE_DIR=${HOME}/.pyenv/versions/3.8.0/include/python3.8/ cern_root/

In Python 3.2 and onward distutils.sysconfig is deprecated in favor of sysconfig.
To get all the variable names in data structure and inspect the situation we can use get_paths function
import sysconfig
sysconfig.get_paths()
which will return us a dict with all the relevant variable names as keys, and corresponding paths as values.
When we know the key we can get the value dynamically,
>>> sysconfig.get_path("include")
'C:\\Program Files\\Python310\\Include'
However, the most convenient feature of sysutils for this situation is that it can list all the variables at once from the command-prompt by invoking python -m sysconfig:
$ python -m sysconfig
Platform: "macosx-10.4-i386"
Python version: "3.2"
Current installation scheme: "posix_prefix"
Paths:
data = "/usr/local"
include = "/Users/tarek/Dev/svn.python.org/py3k/Include"
platinclude = "."
platlib = "/usr/local/lib/python3.2/site-packages"
platstdlib = "/usr/local/lib/python3.2"
purelib = "/usr/local/lib/python3.2/site-packages"
scripts = "/usr/local/bin"
stdlib = "/usr/local/lib/python3.2"
Variables:
AC_APPLE_UNIVERSAL_BUILD = "0"
AIX_GENUINE_CPLUSPLUS = "0"
AR = "ar"
ARFLAGS = "rc"
...

Maybe below command line works for you if all of above methods didn't work.
sudo apt-get install cython cython3

On Ubuntu 20 at least, since Python 2.7 is deprecated, you can:
sudo apt install python2.7-dev
And then PYTHON_INCLUDE_DIR can be set to /usr/include/python2.7

Torch7 Mac Installation error

I am trying to install in mac machine Torch7. When I am trying to install torch using this tutorial https://github.com/torch/torch7/wiki/Cheatsheet#installing-and-running-torch on the second line(curl -sk https://raw.githubusercontent.com/torch/ezinstall/master/install-luajit+torch | bash) I am facing the following error:
-- Looking for QT_MAC_USE_COCOA - found
CMake Error at /usr/local/Cellar/cmake/2.8.12.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
Could NOT find Qt4 (missing: QT_MOC_EXECUTABLE QT_RCC_EXECUTABLE
QT_UIC_EXECUTABLE) (found suitable version "4.7.4", minimum required is
"4.3.0")
Call Stack (most recent call first):
/usr/local/Cellar/cmake/2.8.12.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)
/usr/local/Cellar/cmake/2.8.12.2/share/cmake/Modules/FindQt4.cmake:1393 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:27 (FIND_PACKAGE)
Have you experienced this problem before?
Thanks

After running curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash as advised at https://github.com/torch/ezinstall I got the same error, i.e.
Could NOT find Qt4 (missing:
QT_MOC_EXECUTABLE QT_RCC_EXECUTABLE
QT_UIC_EXECUTABLE) (found suitable version "4.8.6", minimum required is
"4.3.0")
When I ran which qmake I found that qmake was located in ~/anaconda/bin/qmake. I tried a bunch of different things to set the QT_MOC_EXECUTABLE, QT_RCC_EXECUTABLE, and QT_UIC_EXECUTABLE paths correctly e.g.
qmake -set "QT_MOC_EXECUTABLE" /usr/local/Cellar/qt/4.8.6/bin/moc
qmake -set "QT_RCC_EXECUTABLE" /usr/local/Cellar/qt/4.8.6/bin/rcc
qmake -set "QT_UIC_EXECUTABLE" /usr/local/Cellar/qt/4.8.6/bin/uic
But nothing worked.
So what I did was remove the anaconda installation via rm -rf ~/anaconda
I ran curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash again, and everything install correctly. Then I just reinstalled anaconda.
That was a relief. It's probably not the best solution, but I hope this can also work for you.