I have a small project where I use the CMake system to create a Python module out of C++ files. In the CMakeLists.txt file I have Swig integrated as follows:
# only the Swig part here
find_package(SWIG REQUIRED)
include(${SWIG_USE_FILE})
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_PATH})
set(CMAKE_SWIG_OUTDIR ${PROJECT_BINARY_DIR}/../lib/Foo)
SET_SOURCE_FILES_PROPERTIES(swig/interface.i PROPERTIES CPLUSPLUS ON)
set_source_files_properties(swig/interface.i SWIG_FLAGS "-includeall;-c++;-shadow")
swig_add_module(Foo python swig/interface.i code/foo.cpp)
swig_link_libraries(Foo foolib ${PYTHON_LIBRARIES})
My first question is why not both the Foo.py and the _Foo.so are created in the location specified by CMAKE_SWIG_OUTDIR? Only the .py file is created in that directory. Is this a bug of the CMake UseSWIG.cmake file? The .so file is still in the PROJECT_BINARY_DIR location. As a result, I can't load the module in Python if only the location CMAKE_SWIG_OUTDIR is in the PYTHON_PATH environment variable. So to solve this problem I could either:
Add the PROJECT_BINARY_DIR directory to the PYTHON_PATH.
Copy the .so file to CMAKE_SWIG_OUTDIR or create a symbolic link using the CMake system.
Don't set the CMAKE_SWIG_OUTDIR variable so that everything is created in the PROJECT_BINARY_DIR and add only this location to PYTHON_PATH.
But none of these seem to be the logic thing to do, for the CMAKE_SWIG_OUTDIR should be used to output both the .py and the .so files. Am I missing something here?
I'm not sure why CMAKE_SWIG_OUTDIR does not affect the location of where the .so file gets generated. However, I can tell you of a simpler and cleaner way of indicating where the .so file should get generated.
After your swig_add_module(Foo ...), a CMake target called _Foo gets created. You can then edit the target's properties and change where its library (.so file) gets generated - set_target_properties(.. LIBRARY_OUTPUT_DIRECTORY <so_file_output_dir>).
So in your code, just add the set_target_properties(..) line shown below:
...
swig_add_module(Foo python swig/interface.i code/foo.cpp)
set_target_properties(_Foo PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_SWIG_OUTDIR})
swig_link_libraries(Foo foolib ${PYTHON_LIBRARIES})
...
Related
I am trying to bind my CPP function and use it in python code using pybind11. I have created a visual studio project that includes the function (does not have the main function). I cloned pybind11 from git and added it and python38 /include folders to Configuration Properties/VC++ Directories/Include Directories and python38 /libs folder to Configuration Properties/VC++ Directories/Library Directories. Also, I added the python38.lib file to Linker/input/Additional Dependencies.
I added
#include <Python.h>
#include <pybind11/pybind11.h>
to the top of my code. And the lines
And I get build errors of the form "unresolved external symbol
"LNK2001 unresolved external symbol __imp_PyThread_tss_create"
PYBIND11_MODULE(my_kmeans, handle)
{
handle.doc() = "Kmeans Clustering";
handle.def("my_kmeans", &kmeans, "kmeans function");
}
Can you please help me fix them?
I tried looking for a .lib file in pybind11 folder but I did not find one.
I am trying to setup a CMake project that creates python bindings for its c++ functions using pybind11 on Ubuntu.
The directory structure is:
pybind_test
arithmetic.cpp
arithmetic.h
bindings.h
CMakeLists.txt
main.cpp
pybind11 (github repo clone)
Repo contents (https://github.com/pybind/pybind11)
The CMakeLists.txt file:
cmake_minimum_required(VERSION 3.10)
project(pybind_test)
set(CMAKE_CXX_STANDARD 17)
find_package(PythonLibs REQUIRED)
include_directories(${PYTHON_INCLUDE_DIRS})
include_directories(pybind11/include/pybind11)
add_executable(pybind_test main.cpp arithmetic.cpp)
add_subdirectory(pybind11)
pybind11_add_module(arithmetic arithmetic.cpp)
target_link_libraries(pybind_test ${PYTHON_LIBRARIES})
The repository builds successfully and the file arithmetic.cpython-36m-x86_64-linux-gnu.so is produced. How do I import this shared object file into python?
The documentation in the pybind11 docs has this line
$ c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix`
but I want to build using CMake and I also don't want to have to specify extra include directories every time I run python to use this module.
How would I import this shared object file into python like a normal python module?
I am using Ubuntu 16.04.
If you open a terminal, go to the directory where arithmetic.cpython-36m-x86_64-linux-gnu.so is located and run python followed by import arithmetic the module will get imported just like any other module.
Another options is to use the method of
import sys
sys.path.insert(0, 'path/to/directory/where/so-file/is')
import arithmetic
With this method you can use both relative and absolute path.
Besides the solution of setting the path in the Python script that is presented by #super, you have two more generic solutions.
Setting PYTHONPATH
There is an environment variable in Linux (and macOS) called PYTHONPATH. If you add the path that contains your *.so to the PYTHONPATH before you call Python, Python will be able to find your library.
To do this:
export PYTHONPATH="/path/that/contains/your/so":"${PYTHONPATH}"
To apply this 'automatically' for every session you can add this line to ~/.bash_profile or ~/.bashrc (see the same reference). In that case, Python will always be able to find your library.
Copying your to a path already in Python's path
You can also 'install' the library. The usual way to do this is to create a setup.py file. If set up correctly you can build and install your library using
python setup.py build
python setup.py install
(Python will know where to put your library. You can 'customize' a bit with an option like --user to use your home-folder, but this doesn't seems to be of particular interest to you.)
The question remains: How to write setup.py? For your case you can actually call CMake. In fact there exists an example that does exactly that: pybind/cmake_example. You can basically copy-paste from there.
I have a C++ project that I have generated Python bindings for using SWIG. I am now trying to finish the CMake file for the project by adding an install operation. But whenever I finish the install and try to call my functions, I get an error stating foo has no attribute bar().
It has to do with the fact that Python doesn't know where the .so file that the bindings rely on is. If both foo.py and _foo.so are in the same directory I can use the bindings perfectly. I am struggling with figuring out how I am supposed to "install" both the Python bindings and the .so they depend on, all in a portable manner.
Obviously I could just export the install path of the .so to LD_LIBRARY_PATH, but this seems like a hacky work around for what must have a proper solution.
My CMakeLists.txt. I have cut out the bits related to compiling of my C++ lib RTK:
# Project
##
# TODO this actually needs 3.3+
cmake_minimum_required(VERSION 2.6)
project(RTKLIB)
FIND_PACKAGE(SWIG REQUIRED)
INCLUDE(${SWIG_USE_FILE})
FIND_PACKAGE(PythonLibs 3 REQUIRED)
INCLUDE_DIRECTORIES(${PYTHON_INCLUDE_PATH})
find_program(PYTHON "python3" REQUIRED)
include(GNUInstallDirs)
# Variable declarations
##
# Define this directory
set(RTKLIB_ROOT ${CMAKE_CURRENT_SOURCE_DIR})
# Define the build dir
set(RTKLIB_BIN_DIR "${RTKLIB_ROOT}/build")
list(APPEND CMAKE_MODULE_PATH "${RTKLIB_ROOT}/cmake")
# Setup python vars
set(SETUP_PY_IN "${RTKLIB_ROOT}/setup.py.in") # initial version of setup.py
set(SETUP_PY "${RTKLIB_BIN_DIR}/setup.py") # cmake generated setup.py
set(OUTPUT "${RTKLIB_BIN_DIR}/python_timestamp") # Timestamp used as dep
set(RTKLIB_PY "rtk_lib") # name of the python lib
# Set the output dir for SWIG
set(CMAKE_SWIG_OUTDIR ${RTKLIB_BIN_DIR}/${RTKLIB_PY})
# Generate Python bindings
##
# SWIG Config
SET_PROPERTY(SOURCE include/rtk_lib.i PROPERTY CPLUSPLUS ON)
SWIG_ADD_MODULE(${RTKLIB_PY} python include/rtk_lib.i) # Generate C-Python bindings
SWIG_LINK_LIBRARIES(${RTKLIB_PY} RTK ${PYTHON_LIBRARIES}) # Link the bindings with python
# Generate the setup.py file
configure_file(${SETUP_PY_IN} ${SETUP_PY})
# Build command that depends on the SWIG output files and updates the timestamp
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON} ${SETUP_PY} build
COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}
DEPENDS ${RTKLIB_BIN_DIR}\${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME})
# Custom target that depends on the timestamp file generated by the custom command
add_custom_target(ALL DEPENDS ${OUTPUT})
# Install the shared library
install(TARGETS ${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
# Install to user's packages
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install --user)")
And here is my setup.py.in if its any help:
from distutils.core import setup
setup(name='rtk_lib',
version='${PACKAGE_VERSION}',
description="""Python bindings for rtk_lib, allowing for serial and
and file interfaces with RTK messages.""",
packages=['${RTKLIB_PY}'])
Quick Summary of the code: It generates wrapper classes for the C++ that are Python compatible, then it compiles and links the wrapper classes with the Python libs and the original RTK C++ library. After that you have a directory called rtk_lib which has both your wrapper classes and the rtk_lib.py module. Outside of this rtk_lib directory is the outputted _rtk_lib.so shared library that the rtk_lib.py relies on. So in order to get the bindings to work, I copy _rtk_lib.so in to that rtk_lib directory and call python3. Then I can import the lib and everything is great.
I try to install the shared lib, but even then I still get the same rtk_lib has no attribute blablabla().
Looks like an old question, but here goes anyway.
See this example swig_examples_cpp showing simple C++ functions wrapped by SWIG, using CMake and CLion to build it. The C version is here
Here's the full Python Cmake file:
project(python_example)
find_package(SWIG REQUIRED)
include(${SWIG_USE_FILE})
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_PATH})
include_directories(${CMAKE_CURRENT_SOURCE_DIR})
set(CMAKE_SWIG_FLAGS "")
set_source_files_properties(../src/example.i PROPERTIES CPLUSPLUS ON)
swig_add_library(python_example
TYPE MODULE
LANGUAGE python
OUTPUT_DIR ../../py_out # move the .so to py_out
OUTFILE_DIR . # leave the .cpp in cmake-build-debug
SOURCES ../src/example.i
../src/example.cpp ../src/example.h
)
set_target_properties(python_example PROPERTIES
LIBRARY_OUTPUT_DIRECTORY ../../py_out # must match dir in OUTPUT_DIR
)
After building it, run python test.py to see it go. Note it's all in bash/Ubuntu, so MacOs should be ok, but windows may cause you some churn.
See the README for the full details.
I connected yesterday using the SSH protocol to another computer and tried to load, through Python, a SO file (which would be compiled C). Here is what I got in the CLI:
The file that is being requested (libLMR_Demodulator.so) next to "OSError:" is in the same dir as the file I want to load (libDemodulatorJNI_lmr.so).
The python code (v3.5.2) is the following one:
import ctypes
sh_obj = ctypes.cdll.LoadLibrary('./libLMR_Demodulator.so')
actual_start_frequency = sh_obj.getActualStartFrequency(ctypes.c_long(0))
print('The Current Actual Frequency Is: ' + str(actual_start_frequency))
#Charles Duffy is right. The issue come from dependencies. You can verify this by command:
ldd libLMR_Demodulator.so
You have several ways to fix this issue:
Put all the lib to /lib, /usr/lib paths, or directly install them to your system.
Put the libs' path to /etc/ld.so.conf file, then run ldconfig to refresh cache.
use LD_LIBRARY_PATH to add the libs' path, then try to run you script
LD_LIBRARY_PATH=[..path] python [script.py]
or
export LD_LIBRARY_PATH=[..path]
python [script.py]
You can check with manual of dlopen to get more details.
I got here looking for how to ensure that a module / package with a .so file was able to load another .so file that it depends upon -- changing the current directory to the location of the first .so file (i.e., in the directory where the module is) seems to work for me:
import os,sys,inspect
cwd = os.getcwd()
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
os.chdir(currentdir)
import _myotherlib
os.chdir(cwd) # go back
might also work for the OP case?
I am trying to integrate a third party library written in C with my python application using Cython. I have all of the python code written for a test. I am having trouble finding an example for setting this up.
I have a pyd/pyx file I created manually. The third party has given me a header file (*.h) and a shared library (*.so). As far as I can tell, there are no other dependencies. Can someone provide an example of how to set this up using Cython and disutils?
Thanks
Sure !
(In the following, I assume that you already know how to deal with cimport and the interactions between .pxd and .pyx. If this is not completely the case, just ask and I will develop that part as well)
The sample (grabbed from a C++ project of mine, but a C project would work pretty much the same) :
1. The Distutils setup file :
Assuming that the extension to be created will be called myext and the 3rd party shared library is libexternlib.so (note the lib* prefix, here)...
# setup.py file
import sys
import os
import shutil
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("myext") and not(name.endswith(".pyx") or name.endswith(".pxd"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
# build "myext.so" python extension to be added to "PYTHONPATH" afterwards...
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("myext",
sources=["myext.pyx",
"SomeAdditionalCppClass1.cpp",
"SomeAdditionalCppClass2.cpp"
],
libraries=["externlib"], # refers to "libexternlib.so"
language="c++", # remove this if C and not C++
extra_compile_args=["-fopenmp", "-O3"],
extra_link_args=["-DSOME_DEFINE_OPT",
"-L./some/extra/dependency/dir/"]
)
]
)
Note : Your external .so file is linked via the libraries option :
libraries=["externlib"] # Without the 'lib' prefix and the '.so' extension...
Note : the sources option can be used to get some additional source files compiled.
Important : myext.pxd (do not confound with .pyd - Windows stuff) and myext.pyx should be in the same directory. At compile time the definition file, if it exists, is processed first (more).
2. Then run it as follows :
After having changed directory to the one containing your myext.pxd, your myext.pyx, as well as the above setup.py script :
# setup.sh
# Make the "myext" Python Module ("myext.so")
CC="gcc" \
CXX="g++" \
CFLAGS="-I./some/path/to/includes/ -I../../../DEPENDENCIES/python2.7/inc -I../../../DEPENDENCIES/gsl-1.15" \
LDFLAGS="-L./some/path/to/externlib/" \
python setup.py build_ext --inplace
Where :
libexternlib.so is assumed to be located at ./some/path/to/externlib/
yourheader.h is assumed to be located at ./some/path/to/includes/
Note : CFLAGS could also have been setup using the extra_compile_args option :
extra_compile_args=["-I./some/path/to/includes/", "-fopenmp", "-O3"]
Note : LDFLAGS could also have been setup using the extra_link_args option :
extra_link_args=["-L./some/path/to/externlib/", "-DSOME_DEFINE_OPT", "-L./some/extra/dependency/dir/"]
Once distutils is done with the build, you get some new files, specially the myext.cpp, myext.h and most importantly, the myext.so.
3. After that, you're good to go :
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./some/path/to/externlib/
export PYTHONPATH=$PYTHONPATH:./some/path/to/myext/
# Run some script requiring "myext.so"
python somescript.py
Where your freshly created Python extension can be imported by its name :
# somescript.py
import myext
from myext import PySomeFeature
...
Note about Optimization : By default -O2 is used for compiling the extension, but this can be overloaded (see above setup where -O3 is specified).
Note about Cython paths : If Cython was installed in a custom directory, you might want to add it to your environment, before all :
PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18 export PYTHONPATH;
PATH=$PATH:../../../DEPENDENCIES/Cython-0.18/bin; export PATH;
Well, hope I covered the main points...