Develop Reference

How do I import a module created with pybind11 on Ubuntu

I am trying to setup a CMake project that creates python bindings for its c++ functions using pybind11 on Ubuntu.
The directory structure is:
pybind_test
arithmetic.cpp
arithmetic.h
bindings.h
CMakeLists.txt
main.cpp
pybind11 (github repo clone)
Repo contents (https://github.com/pybind/pybind11)
The CMakeLists.txt file:
cmake_minimum_required(VERSION 3.10)
project(pybind_test)
set(CMAKE_CXX_STANDARD 17)
find_package(PythonLibs REQUIRED)
include_directories(${PYTHON_INCLUDE_DIRS})
include_directories(pybind11/include/pybind11)
add_executable(pybind_test main.cpp arithmetic.cpp)
add_subdirectory(pybind11)
pybind11_add_module(arithmetic arithmetic.cpp)
target_link_libraries(pybind_test ${PYTHON_LIBRARIES})
The repository builds successfully and the file arithmetic.cpython-36m-x86_64-linux-gnu.so is produced. How do I import this shared object file into python?
The documentation in the pybind11 docs has this line
$ c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix`
but I want to build using CMake and I also don't want to have to specify extra include directories every time I run python to use this module.
How would I import this shared object file into python like a normal python module?
I am using Ubuntu 16.04.

If you open a terminal, go to the directory where arithmetic.cpython-36m-x86_64-linux-gnu.so is located and run python followed by import arithmetic the module will get imported just like any other module.
Another options is to use the method of
import sys
sys.path.insert(0, 'path/to/directory/where/so-file/is')
import arithmetic
With this method you can use both relative and absolute path.

Besides the solution of setting the path in the Python script that is presented by #super, you have two more generic solutions.
Setting PYTHONPATH
There is an environment variable in Linux (and macOS) called PYTHONPATH. If you add the path that contains your *.so to the PYTHONPATH before you call Python, Python will be able to find your library.
To do this:
export PYTHONPATH="/path/that/contains/your/so":"${PYTHONPATH}"
To apply this 'automatically' for every session you can add this line to ~/.bash_profile or ~/.bashrc (see the same reference). In that case, Python will always be able to find your library.
Copying your to a path already in Python's path
You can also 'install' the library. The usual way to do this is to create a setup.py file. If set up correctly you can build and install your library using
python setup.py build
python setup.py install
(Python will know where to put your library. You can 'customize' a bit with an option like --user to use your home-folder, but this doesn't seems to be of particular interest to you.)
The question remains: How to write setup.py? For your case you can actually call CMake. In fact there exists an example that does exactly that: pybind/cmake_example. You can basically copy-paste from there.

./python: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

I need to try python 3.7 with openssl-1.1.1 in Ubuntu 16.04. Both python and openssl versions are pre-release. Following instructions on how to statistically link openssl to python in a previous post, I downloaded the source for opnssl-1.1.1.
Then navigate to the source code for openssl and execute:
./config
sudo make
sudo make install
Then, edit Modules/Setup.dist to uncomment the following lines:
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
Then download python 3.7 source code. Then, navigate inside the source code and execute:
./configure
make
make install
After I execute make install I got this error at the end of the terminal output:
./python: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
generate-posix-vars failed
Makefile:596: recipe for target 'pybuilddir.txt' failed
make: *** [pybuilddir.txt] Error 1
I could not figure out what is the problem and what I need to do.

This has (should have) nothing to do with Python or OpenSSL versions.
Python build process, includes some steps when the newly built interpreter is launched, and attempts to load some of the newly built modules - including extension modules (which are written in C and are actually shared objects (.sos)).
When an .so is loaded, the loader must find (recursively) all the .so files that the .so needs (depends on), otherwise it won't be able to load it.
Python has some modules (e.g. _ssl*.so, _hashlib*.so) that depend on OpenSSL libs. Since you built yours against OpenSSL1.1.1 (the lib names differ from what comes by default on the system: typically 1.0.*), the loader won't be able to use the default ones.
What you need to do, is instruct the loader (check [Man7]: LD.SO(8) for more details) where to look for "your" OpenSSL libs (which are located under /usr/local/ssl/lib). One way of doing that is adding their path in ${LD_LIBRARY_PATH} env var (before building Python):
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/ssl/lib
./configure
make
make install
You might also want to take a look at [Python.Docs]: Configure Python - Libraries options (--with-openssl, --with-openssl-rpath).
Check [SO]: How to enable FIPS mode for libcrypto and libssl packaged with Python? (#CristiFati's answer) for details on a wider problem (remotely) related to yours.

What I have done to fix this :
./configure --with-ssl=./libssl --prefix=/subsystem
sed -i 's!^RUNSHARED=!RUNSHARED=LD_LIBRARY_PATH=/path/to/own/libssl/lib!' Makefile
make
make install
Setting LD_LIBRARY_PATH with export was not sufficient

With Python-3.6.5 and openssl-1.1.0h i get stuck in the same problem. I have uncomment _socket socketmodule.c.

Packaging and installing python bindings with CMake

I have a C++ project that I have generated Python bindings for using SWIG. I am now trying to finish the CMake file for the project by adding an install operation. But whenever I finish the install and try to call my functions, I get an error stating foo has no attribute bar().
It has to do with the fact that Python doesn't know where the .so file that the bindings rely on is. If both foo.py and _foo.so are in the same directory I can use the bindings perfectly. I am struggling with figuring out how I am supposed to "install" both the Python bindings and the .so they depend on, all in a portable manner.
Obviously I could just export the install path of the .so to LD_LIBRARY_PATH, but this seems like a hacky work around for what must have a proper solution.
My CMakeLists.txt. I have cut out the bits related to compiling of my C++ lib RTK:
# Project
##
# TODO this actually needs 3.3+
cmake_minimum_required(VERSION 2.6)
project(RTKLIB)
FIND_PACKAGE(SWIG REQUIRED)
INCLUDE(${SWIG_USE_FILE})
FIND_PACKAGE(PythonLibs 3 REQUIRED)
INCLUDE_DIRECTORIES(${PYTHON_INCLUDE_PATH})
find_program(PYTHON "python3" REQUIRED)
include(GNUInstallDirs)
# Variable declarations
##
# Define this directory
set(RTKLIB_ROOT ${CMAKE_CURRENT_SOURCE_DIR})
# Define the build dir
set(RTKLIB_BIN_DIR "${RTKLIB_ROOT}/build")
list(APPEND CMAKE_MODULE_PATH "${RTKLIB_ROOT}/cmake")
# Setup python vars
set(SETUP_PY_IN "${RTKLIB_ROOT}/setup.py.in") # initial version of setup.py
set(SETUP_PY "${RTKLIB_BIN_DIR}/setup.py") # cmake generated setup.py
set(OUTPUT "${RTKLIB_BIN_DIR}/python_timestamp") # Timestamp used as dep
set(RTKLIB_PY "rtk_lib") # name of the python lib
# Set the output dir for SWIG
set(CMAKE_SWIG_OUTDIR ${RTKLIB_BIN_DIR}/${RTKLIB_PY})
# Generate Python bindings
##
# SWIG Config
SET_PROPERTY(SOURCE include/rtk_lib.i PROPERTY CPLUSPLUS ON)
SWIG_ADD_MODULE(${RTKLIB_PY} python include/rtk_lib.i) # Generate C-Python bindings
SWIG_LINK_LIBRARIES(${RTKLIB_PY} RTK ${PYTHON_LIBRARIES}) # Link the bindings with python
# Generate the setup.py file
configure_file(${SETUP_PY_IN} ${SETUP_PY})
# Build command that depends on the SWIG output files and updates the timestamp
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON} ${SETUP_PY} build
COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}
DEPENDS ${RTKLIB_BIN_DIR}\${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME})
# Custom target that depends on the timestamp file generated by the custom command
add_custom_target(ALL DEPENDS ${OUTPUT})
# Install the shared library
install(TARGETS ${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
# Install to user's packages
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install --user)")
And here is my setup.py.in if its any help:
from distutils.core import setup
setup(name='rtk_lib',
version='${PACKAGE_VERSION}',
description="""Python bindings for rtk_lib, allowing for serial and
and file interfaces with RTK messages.""",
packages=['${RTKLIB_PY}'])
Quick Summary of the code: It generates wrapper classes for the C++ that are Python compatible, then it compiles and links the wrapper classes with the Python libs and the original RTK C++ library. After that you have a directory called rtk_lib which has both your wrapper classes and the rtk_lib.py module. Outside of this rtk_lib directory is the outputted _rtk_lib.so shared library that the rtk_lib.py relies on. So in order to get the bindings to work, I copy _rtk_lib.so in to that rtk_lib directory and call python3. Then I can import the lib and everything is great.
I try to install the shared lib, but even then I still get the same rtk_lib has no attribute blablabla().

Looks like an old question, but here goes anyway.
See this example swig_examples_cpp showing simple C++ functions wrapped by SWIG, using CMake and CLion to build it. The C version is here
Here's the full Python Cmake file:
project(python_example)
find_package(SWIG REQUIRED)
include(${SWIG_USE_FILE})
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_PATH})
include_directories(${CMAKE_CURRENT_SOURCE_DIR})
set(CMAKE_SWIG_FLAGS "")
set_source_files_properties(../src/example.i PROPERTIES CPLUSPLUS ON)
swig_add_library(python_example
TYPE MODULE
LANGUAGE python
OUTPUT_DIR ../../py_out # move the .so to py_out
OUTFILE_DIR . # leave the .cpp in cmake-build-debug
SOURCES ../src/example.i
../src/example.cpp ../src/example.h
)
set_target_properties(python_example PROPERTIES
LIBRARY_OUTPUT_DIRECTORY ../../py_out # must match dir in OUTPUT_DIR
)
After building it, run python test.py to see it go. Note it's all in bash/Ubuntu, so MacOs should be ok, but windows may cause you some churn.
See the README for the full details.

Platform independent way to find path to libpython (e.g., for use in cmake)

Is there any platform independent way to get the path to a python installation's libpython, for use as a cmake argument? sysconfig.get_config_var gives some pieces but there's no consistent way I can get this working.
On OSX:
No variable contains the actual basename of the library (libpython2.7.dylib)
sysconfig.get_config_var('LIBDIR') returns the directory libpython2.7.dylib is in (/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib)
edit: There seems to be a bit of a discrepancy in when sysconfig is reporting a symlink and when it's reporting a real path. LIBDIR does in fact contain libpython.2.7.dylib, but what I just noticed is that this is a symlink to /usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/Python.
INSTSONAME points to Python.framework/Versions/2.7/Python, the basename of that path, but no variable I can find points to the parent part of that path.
On Ubuntu:
sysconfig.get_config_var('INSTSONAME') gives me the name of the library (libpython2.7.so.1.0).
No variable contains the directory it's in (/usr/lib/x86_64-linux-gnu/).
LIBDIR returns only /usr/lib

CMake python modules: FindPythonLibs and FindPythonInterp.
If those don't work for you just set the next vars when calling cmake -DPYTHON_EXECUTABLE:FILEPATH, -DPYTHON_LIBRARY:FILEPATH and -DPYTHON_INCLUDE_DIR:FILEPAT, for more info look here.
Example:
cmake .. -G "Sublime Text 2 - Ninja"
-DCMAKE_BUILD_TYPE=Release
-DPYTHON_EXECUTABLE:FILEPATH=d:\virtualenvs\python362_32
-DPYTHON_LIBRARY:FILEPATH=d:\virtualenvs\python362_32\libs\python36.lib
-DPYTHON_INCLUDE_DIR:FILEPAT=d:\virtualenvs\python362_32\include

CMake Error: The following variables are used in this project, but they are set to NOTFOUND. PYTHON_LIBRARIES (ADVANCED)

I'm trying to install this software http://www.openstructure.org/docs/1.4/install/ and I get this error:
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
PYTHON_LIBRARIES (ADVANCED)
linked by target "_ost_conop" in directory /opt/ost/modules/conop/pymod
linked by target "_ost_gfx" in directory /opt/ost/modules/gfx/pymod
linked by target "_ost_img" in directory /opt/ost/modules/img/base/pymod
linked by target "_ost_img_alg" in directory /opt/ost/modules/img/alg/pymod
linked by target "ost_gui" in directory /opt/ost/modules/gui/src
linked by target "_ost_gui" in directory /opt/ost/modules/gui/pymod
linked by target "_ost_io" in directory /opt/ost/modules/io/pymod
linked by target "_ost_info" in directory /opt/ost/modules/info/pymod
linked by target "_ost_seq" in directory /opt/ost/modules/seq/base/pymod
linked by target "_ost_seq_alg" in directory /opt/ost/modules/seq/alg/pymod
linked by target "_ost_mol" in directory /opt/ost/modules/mol/base/pymod
linked by target "_ost_mol_alg" in directory /opt/ost/modules/mol/alg/pymod
linked by target "_ost_geom" in directory /opt/ost/modules/geom/pymod
linked by target "_ost_base" in directory /opt/ost/modules/base/pymod
-- Configuring incomplete, errors occurred!
See also "/opt/ost/CMakeFiles/CMakeOutput.log".
See also "/opt/ost/CMakeFiles/CMakeError.log".
I'm guessing I have to pass the path to PYTHON_LIBRARIES to cmake but I have no idea what. I have the correct python installed and all other dependencies.

First I took this advice https://askubuntu.com/questions/479260/cmake-can-not-find-pythonlibs and installed the latest cmake (3.4.1) from source. This did not solve the problem and I got the same error. However I did solve it using cmake 3.4, not sure if it made a difference though.
I had to run cmake explicitly showing it PYTHON_LIBRARIES suggested from that same thread.
cmake . -DPYTHON_LIBRARIES=/usr/lib/python2.7/config/libpython2.7.so
It was not simply enough to use
export PYTHON_LIBRARIES=/usr/lib/python2.7/config/libpython2.7.so
and then cmake ..