I have a C++ project that I have generated Python bindings for using SWIG. I am now trying to finish the CMake file for the project by adding an install operation. But whenever I finish the install and try to call my functions, I get an error stating foo has no attribute bar().
It has to do with the fact that Python doesn't know where the .so file that the bindings rely on is. If both foo.py and _foo.so are in the same directory I can use the bindings perfectly. I am struggling with figuring out how I am supposed to "install" both the Python bindings and the .so they depend on, all in a portable manner.
Obviously I could just export the install path of the .so to LD_LIBRARY_PATH, but this seems like a hacky work around for what must have a proper solution.
My CMakeLists.txt. I have cut out the bits related to compiling of my C++ lib RTK:
# Project
##
# TODO this actually needs 3.3+
cmake_minimum_required(VERSION 2.6)
project(RTKLIB)
FIND_PACKAGE(SWIG REQUIRED)
INCLUDE(${SWIG_USE_FILE})
FIND_PACKAGE(PythonLibs 3 REQUIRED)
INCLUDE_DIRECTORIES(${PYTHON_INCLUDE_PATH})
find_program(PYTHON "python3" REQUIRED)
include(GNUInstallDirs)
# Variable declarations
##
# Define this directory
set(RTKLIB_ROOT ${CMAKE_CURRENT_SOURCE_DIR})
# Define the build dir
set(RTKLIB_BIN_DIR "${RTKLIB_ROOT}/build")
list(APPEND CMAKE_MODULE_PATH "${RTKLIB_ROOT}/cmake")
# Setup python vars
set(SETUP_PY_IN "${RTKLIB_ROOT}/setup.py.in") # initial version of setup.py
set(SETUP_PY "${RTKLIB_BIN_DIR}/setup.py") # cmake generated setup.py
set(OUTPUT "${RTKLIB_BIN_DIR}/python_timestamp") # Timestamp used as dep
set(RTKLIB_PY "rtk_lib") # name of the python lib
# Set the output dir for SWIG
set(CMAKE_SWIG_OUTDIR ${RTKLIB_BIN_DIR}/${RTKLIB_PY})
# Generate Python bindings
##
# SWIG Config
SET_PROPERTY(SOURCE include/rtk_lib.i PROPERTY CPLUSPLUS ON)
SWIG_ADD_MODULE(${RTKLIB_PY} python include/rtk_lib.i) # Generate C-Python bindings
SWIG_LINK_LIBRARIES(${RTKLIB_PY} RTK ${PYTHON_LIBRARIES}) # Link the bindings with python
# Generate the setup.py file
configure_file(${SETUP_PY_IN} ${SETUP_PY})
# Build command that depends on the SWIG output files and updates the timestamp
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON} ${SETUP_PY} build
COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}
DEPENDS ${RTKLIB_BIN_DIR}\${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME})
# Custom target that depends on the timestamp file generated by the custom command
add_custom_target(ALL DEPENDS ${OUTPUT})
# Install the shared library
install(TARGETS ${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
# Install to user's packages
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install --user)")
And here is my setup.py.in if its any help:
from distutils.core import setup
setup(name='rtk_lib',
version='${PACKAGE_VERSION}',
description="""Python bindings for rtk_lib, allowing for serial and
and file interfaces with RTK messages.""",
packages=['${RTKLIB_PY}'])
Quick Summary of the code: It generates wrapper classes for the C++ that are Python compatible, then it compiles and links the wrapper classes with the Python libs and the original RTK C++ library. After that you have a directory called rtk_lib which has both your wrapper classes and the rtk_lib.py module. Outside of this rtk_lib directory is the outputted _rtk_lib.so shared library that the rtk_lib.py relies on. So in order to get the bindings to work, I copy _rtk_lib.so in to that rtk_lib directory and call python3. Then I can import the lib and everything is great.
I try to install the shared lib, but even then I still get the same rtk_lib has no attribute blablabla().
Looks like an old question, but here goes anyway.
See this example swig_examples_cpp showing simple C++ functions wrapped by SWIG, using CMake and CLion to build it. The C version is here
Here's the full Python Cmake file:
project(python_example)
find_package(SWIG REQUIRED)
include(${SWIG_USE_FILE})
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_PATH})
include_directories(${CMAKE_CURRENT_SOURCE_DIR})
set(CMAKE_SWIG_FLAGS "")
set_source_files_properties(../src/example.i PROPERTIES CPLUSPLUS ON)
swig_add_library(python_example
TYPE MODULE
LANGUAGE python
OUTPUT_DIR ../../py_out # move the .so to py_out
OUTFILE_DIR . # leave the .cpp in cmake-build-debug
SOURCES ../src/example.i
../src/example.cpp ../src/example.h
)
set_target_properties(python_example PROPERTIES
LIBRARY_OUTPUT_DIRECTORY ../../py_out # must match dir in OUTPUT_DIR
)
After building it, run python test.py to see it go. Note it's all in bash/Ubuntu, so MacOs should be ok, but windows may cause you some churn.
See the README for the full details.
Related
How can I (hermetically) include python as an (executable) input to my genrule?
Conceptually, I'm aware of the following approaches:
include a python interpreter in the source repo at third_party/python
have Bazel fetch python ala rules-go
There also seem to be a couple methods for doing so:
py_runtime
py_toolchain (doesn't seem to be available yet)
py_distribution_package (doesn't seem to be available yet)
genrules.tools
genrules.toolchains
Example:
I have a python library:
myproject
- setup.py
- mylibrary
- __init__.py
- myfunction.py
I'd like to produce a pip install-able distribution with Bazel so that I can do:
pip install <path/to/distribution>
python
>>> from mylibrary.myfunction import helloworld
>>> helloworld()
I added a (empty) WORKSPACE file at myproject/WORKSPACE and followed the example in this medium article:
# myproject/BUILD
genrule(
name = "mylibrary_sdist",
srcs = glob(["**/*.py"]),
outs = ["mylibrary.tar.gz"],
cmd = "python setup.py sdist && mv dist/mylibrary*.tar.gz $(location mylibrary.tar.gz)"
)
This works (ie. produces bazel-genfiles/mylibrary.tar.gz), but I'm shelling out to python.
How can I (hermetically) include python as an (executable) input to my genrule?
Unless I'm misunderstanding your question, this should just be a matter of passing the actual path to your target python executable. I would check the python executable into the third_party dir and instead of invoking your genrule using simply python I'd pass the relative path to that executable relative to WORKSPACE.
I am trying to setup a CMake project that creates python bindings for its c++ functions using pybind11 on Ubuntu.
The directory structure is:
pybind_test
arithmetic.cpp
arithmetic.h
bindings.h
CMakeLists.txt
main.cpp
pybind11 (github repo clone)
Repo contents (https://github.com/pybind/pybind11)
The CMakeLists.txt file:
cmake_minimum_required(VERSION 3.10)
project(pybind_test)
set(CMAKE_CXX_STANDARD 17)
find_package(PythonLibs REQUIRED)
include_directories(${PYTHON_INCLUDE_DIRS})
include_directories(pybind11/include/pybind11)
add_executable(pybind_test main.cpp arithmetic.cpp)
add_subdirectory(pybind11)
pybind11_add_module(arithmetic arithmetic.cpp)
target_link_libraries(pybind_test ${PYTHON_LIBRARIES})
The repository builds successfully and the file arithmetic.cpython-36m-x86_64-linux-gnu.so is produced. How do I import this shared object file into python?
The documentation in the pybind11 docs has this line
$ c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix`
but I want to build using CMake and I also don't want to have to specify extra include directories every time I run python to use this module.
How would I import this shared object file into python like a normal python module?
I am using Ubuntu 16.04.
If you open a terminal, go to the directory where arithmetic.cpython-36m-x86_64-linux-gnu.so is located and run python followed by import arithmetic the module will get imported just like any other module.
Another options is to use the method of
import sys
sys.path.insert(0, 'path/to/directory/where/so-file/is')
import arithmetic
With this method you can use both relative and absolute path.
Besides the solution of setting the path in the Python script that is presented by #super, you have two more generic solutions.
Setting PYTHONPATH
There is an environment variable in Linux (and macOS) called PYTHONPATH. If you add the path that contains your *.so to the PYTHONPATH before you call Python, Python will be able to find your library.
To do this:
export PYTHONPATH="/path/that/contains/your/so":"${PYTHONPATH}"
To apply this 'automatically' for every session you can add this line to ~/.bash_profile or ~/.bashrc (see the same reference). In that case, Python will always be able to find your library.
Copying your to a path already in Python's path
You can also 'install' the library. The usual way to do this is to create a setup.py file. If set up correctly you can build and install your library using
python setup.py build
python setup.py install
(Python will know where to put your library. You can 'customize' a bit with an option like --user to use your home-folder, but this doesn't seems to be of particular interest to you.)
The question remains: How to write setup.py? For your case you can actually call CMake. In fact there exists an example that does exactly that: pybind/cmake_example. You can basically copy-paste from there.
I need to try python 3.7 with openssl-1.1.1 in Ubuntu 16.04. Both python and openssl versions are pre-release. Following instructions on how to statistically link openssl to python in a previous post, I downloaded the source for opnssl-1.1.1.
Then navigate to the source code for openssl and execute:
./config
sudo make
sudo make install
Then, edit Modules/Setup.dist to uncomment the following lines:
SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
Then download python 3.7 source code. Then, navigate inside the source code and execute:
./configure
make
make install
After I execute make install I got this error at the end of the terminal output:
./python: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
generate-posix-vars failed
Makefile:596: recipe for target 'pybuilddir.txt' failed
make: *** [pybuilddir.txt] Error 1
I could not figure out what is the problem and what I need to do.
This has (should have) nothing to do with Python or OpenSSL versions.
Python build process, includes some steps when the newly built interpreter is launched, and attempts to load some of the newly built modules - including extension modules (which are written in C and are actually shared objects (.sos)).
When an .so is loaded, the loader must find (recursively) all the .so files that the .so needs (depends on), otherwise it won't be able to load it.
Python has some modules (e.g. _ssl*.so, _hashlib*.so) that depend on OpenSSL libs. Since you built yours against OpenSSL1.1.1 (the lib names differ from what comes by default on the system: typically 1.0.*), the loader won't be able to use the default ones.
What you need to do, is instruct the loader (check [Man7]: LD.SO(8) for more details) where to look for "your" OpenSSL libs (which are located under /usr/local/ssl/lib). One way of doing that is adding their path in ${LD_LIBRARY_PATH} env var (before building Python):
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/ssl/lib
./configure
make
make install
You might also want to take a look at [Python.Docs]: Configure Python - Libraries options (--with-openssl, --with-openssl-rpath).
Check [SO]: How to enable FIPS mode for libcrypto and libssl packaged with Python? (#CristiFati's answer) for details on a wider problem (remotely) related to yours.
What I have done to fix this :
./configure --with-ssl=./libssl --prefix=/subsystem
sed -i 's!^RUNSHARED=!RUNSHARED=LD_LIBRARY_PATH=/path/to/own/libssl/lib!' Makefile
make
make install
Setting LD_LIBRARY_PATH with export was not sufficient
With Python-3.6.5 and openssl-1.1.0h i get stuck in the same problem. I have uncomment _socket socketmodule.c.
I have a small project where I use the CMake system to create a Python module out of C++ files. In the CMakeLists.txt file I have Swig integrated as follows:
# only the Swig part here
find_package(SWIG REQUIRED)
include(${SWIG_USE_FILE})
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_PATH})
set(CMAKE_SWIG_OUTDIR ${PROJECT_BINARY_DIR}/../lib/Foo)
SET_SOURCE_FILES_PROPERTIES(swig/interface.i PROPERTIES CPLUSPLUS ON)
set_source_files_properties(swig/interface.i SWIG_FLAGS "-includeall;-c++;-shadow")
swig_add_module(Foo python swig/interface.i code/foo.cpp)
swig_link_libraries(Foo foolib ${PYTHON_LIBRARIES})
My first question is why not both the Foo.py and the _Foo.so are created in the location specified by CMAKE_SWIG_OUTDIR? Only the .py file is created in that directory. Is this a bug of the CMake UseSWIG.cmake file? The .so file is still in the PROJECT_BINARY_DIR location. As a result, I can't load the module in Python if only the location CMAKE_SWIG_OUTDIR is in the PYTHON_PATH environment variable. So to solve this problem I could either:
Add the PROJECT_BINARY_DIR directory to the PYTHON_PATH.
Copy the .so file to CMAKE_SWIG_OUTDIR or create a symbolic link using the CMake system.
Don't set the CMAKE_SWIG_OUTDIR variable so that everything is created in the PROJECT_BINARY_DIR and add only this location to PYTHON_PATH.
But none of these seem to be the logic thing to do, for the CMAKE_SWIG_OUTDIR should be used to output both the .py and the .so files. Am I missing something here?
I'm not sure why CMAKE_SWIG_OUTDIR does not affect the location of where the .so file gets generated. However, I can tell you of a simpler and cleaner way of indicating where the .so file should get generated.
After your swig_add_module(Foo ...), a CMake target called _Foo gets created. You can then edit the target's properties and change where its library (.so file) gets generated - set_target_properties(.. LIBRARY_OUTPUT_DIRECTORY <so_file_output_dir>).
So in your code, just add the set_target_properties(..) line shown below:
...
swig_add_module(Foo python swig/interface.i code/foo.cpp)
set_target_properties(_Foo PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_SWIG_OUTDIR})
swig_link_libraries(Foo foolib ${PYTHON_LIBRARIES})
...
I'm trying to set up Boost.Python 1.54.0 on Windows 7, python 2.7.3. I installed boost_1_54_0 and built it with Visual Studio 2010. I can build and run the quickstart and tutorial examples in the distribution (I had to add some MSVC lib dirs as library-path requirements in Jamroot, but that's OK). But when I copy the tutorial example to a different dir, not inside the boost dist dir, to make it more like a real project, bjam gives build errors. I changed the use-project boost location in Jamroot to /local/boost_1_54_0/ which is where boost wanted to install itself, and added a simple one-line boost-build.jam (based on something I found online):
boost-build C:/local/boost_1_54_0/tools/build/v2 ;
but ultimately bjam gives me this error:
% bjam
notice: no Python configured in user-config.jam
notice: will use default configuration
C:/local/boost_1_54_0/tools/build/v2/build\project.jam:262: in find-jamfile from module project
error: Unable to load Jamfile.
error: Could not find a Jamfile in directory '/local/boost_1_54_0'.
error: Attempted to find it with pattern '[Bb]uild.jam [Jj]amfile.v2 [Jj]amfile [Jj]amfile.jam'.
error: Please consult the documentation at 'http://www.boost.org'.
C:/local/boost_1_54_0/tools/build/v2/build\project.jam:280: in load-jamfile from module project
C:/local/boost_1_54_0/tools/build/v2/build\project.jam:64: in load from module project
C:/local/boost_1_54_0/tools/build/v2/build\project.jam:89: in load-used-projects from module project
C:/local/boost_1_54_0/tools/build/v2/build\project.jam:75: in load from module project
C:/local/boost_1_54_0/tools/build/v2/build\project.jam:145: in project.find from module project
C:/local/boost_1_54_0/tools/build/v2\build-system.jam:535: in load from module build-system
C:\local\boost_1_54_0\tools\build\v2/kernel\modules.jam:289: in import from module modules
C:\local\boost_1_54_0\tools\build\v2/kernel/bootstrap.jam:139: in boost-build from module
C:\tmp\tutorial\boost-build.jam:1: in module scope from module
What else do I need to add, or what am I doing wrong?
I ended up giving up on bjam for this, and just used SCons. A simple SConstruct was enough:
# SConstruct for building boost python tutorial example
import os
boost_python_lib = 'boost_python-vc100-gd-1_54'
boost_top = 'c:/boost'
python_top = 'c:/python27'
env=Environment(TARGET_ARCH='x86',
CCFLAGS=['/MDd', '/DEBUG', '/EHsc'],
CPPPATH=[os.path.join(boost_top,'include/boost-1_54'),
os.path.join(python_top, 'include')],
LIBPATH=[os.path.join(boost_top, 'lib/i386'),
os.path.join(python_top, 'libs')])
dll=env.SharedLibrary('hello_ext', 'hello.cpp',
LIBS=boost_python_lib)
env.InstallAs('hello_ext.pyd', dll[0])
# Copy the boost python lib into this dir so hello_ext will find it at runtime
env.Install('.', os.path.join(boost_top, 'lib/i386', '%s.dll'%boost_python_lib))
Of course you could make a real SCons Tool out of this, but that was enough to get me going. Hope it's useful to others.