I successfully used Boost Python to write C++ functions with Numpy array arguments.
But I'm struggling with a very basic case: when my function has to return a value. There's no problem until I try to read the int returned by my function.
I strongly suspect my setup has something to do with this, but I'm not sure where to begin. I commented my CMakeLists.txt to explain how I did setup all the libs.
I tried to play with return_value_policy but it seems irrelevant for simple scalar values.
Environment:
MacOS 11.0.1 (20B50)
CLion for C++ edition and build
Pycharm to run Python code, configured using conda environment with Python 3.8
Boost setup described in CMakeLists.txt
C++ code:
int testFunction() {
std::cout << "Test Function 1\n";
return 1;
}
void testFunction2() {
std::cout << "Test Function 2\n";
}
BOOST_PYTHON_MODULE (mylib) {
np::initialize();
py::def("test_function", testFunction);
py::def("test_function_2", testFunction2);
}
My CMakeLists.txt:
cmake_minimum_required(VERSION 3.17)
project(mylib_project)
set(CMAKE_VERBOSE_MAKEFILE ON)
set(CMAKE_CXX_STANDARD 20)
if(NOT CMAKE_BUILD_TYPE)
set(CMAKE_BUILD_TYPE Release)
endif()
set(CMAKE_CXX_FLAGS "-Wall -Wextra")
set(CMAKE_CXX_FLAGS_DEBUG "-g")
set(CMAKE_CXX_FLAGS_RELEASE "-O3")
find_package(PythonInterp 3)
find_package(PythonLibs 3)
# Standard boost installation
# Download boost https://www.boost.org/users/download/ and unzip
# Inside boost dir:
# ./bootstrap.sh
# ./b2
# ./b2 install
# Other (brew / macports) install will have all chances to fail (at the time of writing)
# Components should be found in path with name libboost_NAME.dylib, eg: /usr/local/lib/libboost_numpy38.dylib
# This is additional to standard boost installation
# For both python38 and numpy38 to be produced, you have to:
# - be in the right python environment (conda python 3.8 for example)
# - install numpy: pip install numpy
# - inside boost dir:
# ./bootstrap.sh --with-libraries=python
# ./b2
# ./b2 install
find_package(Boost 1.75.0 COMPONENTS python38 numpy38 REQUIRED)
# Python library, will produce a mylib.so file
python_add_module(mylib python-mylib.cpp)
target_include_directories(mylib PRIVATE ${PYTHON_INCLUDE_DIRS})
target_include_directories(mylib PRIVATE ${Boost_INCLUDE_DIRS})
target_link_libraries(mylib ${PYTHON_LIBRARY})
target_link_libraries(mylib ${Boost_PYTHON_LIBRARY})
target_link_libraries(mylib ${Boost_LIBRARIES})
I'm then producing a mylib.so that I provide to the path of my python code.
Python code:
if __name__ == '__main__':
mylib.test_function_2()
result = mylib.test_function()
print(result)
Output:
Test Function 2
Test Function 1
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
My naive question is: how to avoid this crash ?
But after some further investigations (see comments) I'd rather ask: how can I proceed to find (and then probably fix) the cause of this issue in my setup ?
Related
I'm using PythonKit in my Swift project for MacOS. At the moment I'm using Python 2.7 but from MacOs 12.3 it isn't no more supported so I'm trying to migrate to Python 3 but it doesn't work.
func applicationDidFinishLaunching(_ notification: Notification) {
if #available(OSX 12, *) {
PythonLibrary.useVersion(3)
}
else {
PythonLibrary.useVersion(2)
}
let sys = Python.import("sys")
print("Python \(sys.version_info.major).\(sys.version_info.minor)")
print("Python Version: \(sys.version)")
print("Python Encoding: \(sys.getdefaultencoding().upper())")
sys.path.append(Bundle.main.resourceURL!.absoluteURL.path)
let checker = Python.import("checkLibrary")
_ = Array(checker.check())
}
This is the error message:
PythonKit/PythonLibrary.swift:46: Fatal error: Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.
The code fail on MacOs 12 on line 9th line (let sys = Python.import("sys")), so I can't interact so sys in any way.
I've already tried to disable sandbox and Hardened Runtime but is seems useless.
I was having the same issue.
where python3
which python3
type -a python3
I could clearly see that Python3 was present using any of the above commands from terminal. Python3 wasnt something that I directly installed, (probably something I added during an install of XCode) but I could see it located at "/usr/bin/python3"
I had already removed the sandbox and disabled the hardened runtime.
Adding the environment variable to XCode did not work and Google ultimately told me to do what had already been done.
Finally, I decided to just perform a fresh install, following the blog as guidance.
https://www.dataquest.io/blog/installing-python-on-mac/#installing-python-mac
https://www.python.org/downloads/macos/ (direct URL for Python download)
After installing, everything worked as expected.
PythonLibrary.useVersion(3)
PythonLibrary.useLibrary(at: "/usr/local/bin/python3")
I could use either of the above methods, without adding the environment variable to the XCode scheme.
Hopefully that helps
I have a C++ project that I have generated Python bindings for using SWIG. I am now trying to finish the CMake file for the project by adding an install operation. But whenever I finish the install and try to call my functions, I get an error stating foo has no attribute bar().
It has to do with the fact that Python doesn't know where the .so file that the bindings rely on is. If both foo.py and _foo.so are in the same directory I can use the bindings perfectly. I am struggling with figuring out how I am supposed to "install" both the Python bindings and the .so they depend on, all in a portable manner.
Obviously I could just export the install path of the .so to LD_LIBRARY_PATH, but this seems like a hacky work around for what must have a proper solution.
My CMakeLists.txt. I have cut out the bits related to compiling of my C++ lib RTK:
# Project
##
# TODO this actually needs 3.3+
cmake_minimum_required(VERSION 2.6)
project(RTKLIB)
FIND_PACKAGE(SWIG REQUIRED)
INCLUDE(${SWIG_USE_FILE})
FIND_PACKAGE(PythonLibs 3 REQUIRED)
INCLUDE_DIRECTORIES(${PYTHON_INCLUDE_PATH})
find_program(PYTHON "python3" REQUIRED)
include(GNUInstallDirs)
# Variable declarations
##
# Define this directory
set(RTKLIB_ROOT ${CMAKE_CURRENT_SOURCE_DIR})
# Define the build dir
set(RTKLIB_BIN_DIR "${RTKLIB_ROOT}/build")
list(APPEND CMAKE_MODULE_PATH "${RTKLIB_ROOT}/cmake")
# Setup python vars
set(SETUP_PY_IN "${RTKLIB_ROOT}/setup.py.in") # initial version of setup.py
set(SETUP_PY "${RTKLIB_BIN_DIR}/setup.py") # cmake generated setup.py
set(OUTPUT "${RTKLIB_BIN_DIR}/python_timestamp") # Timestamp used as dep
set(RTKLIB_PY "rtk_lib") # name of the python lib
# Set the output dir for SWIG
set(CMAKE_SWIG_OUTDIR ${RTKLIB_BIN_DIR}/${RTKLIB_PY})
# Generate Python bindings
##
# SWIG Config
SET_PROPERTY(SOURCE include/rtk_lib.i PROPERTY CPLUSPLUS ON)
SWIG_ADD_MODULE(${RTKLIB_PY} python include/rtk_lib.i) # Generate C-Python bindings
SWIG_LINK_LIBRARIES(${RTKLIB_PY} RTK ${PYTHON_LIBRARIES}) # Link the bindings with python
# Generate the setup.py file
configure_file(${SETUP_PY_IN} ${SETUP_PY})
# Build command that depends on the SWIG output files and updates the timestamp
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON} ${SETUP_PY} build
COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}
DEPENDS ${RTKLIB_BIN_DIR}\${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME})
# Custom target that depends on the timestamp file generated by the custom command
add_custom_target(ALL DEPENDS ${OUTPUT})
# Install the shared library
install(TARGETS ${SWIG_MODULE_${RTKLIB_PY}_REAL_NAME}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
# Install to user's packages
install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install --user)")
And here is my setup.py.in if its any help:
from distutils.core import setup
setup(name='rtk_lib',
version='${PACKAGE_VERSION}',
description="""Python bindings for rtk_lib, allowing for serial and
and file interfaces with RTK messages.""",
packages=['${RTKLIB_PY}'])
Quick Summary of the code: It generates wrapper classes for the C++ that are Python compatible, then it compiles and links the wrapper classes with the Python libs and the original RTK C++ library. After that you have a directory called rtk_lib which has both your wrapper classes and the rtk_lib.py module. Outside of this rtk_lib directory is the outputted _rtk_lib.so shared library that the rtk_lib.py relies on. So in order to get the bindings to work, I copy _rtk_lib.so in to that rtk_lib directory and call python3. Then I can import the lib and everything is great.
I try to install the shared lib, but even then I still get the same rtk_lib has no attribute blablabla().
Looks like an old question, but here goes anyway.
See this example swig_examples_cpp showing simple C++ functions wrapped by SWIG, using CMake and CLion to build it. The C version is here
Here's the full Python Cmake file:
project(python_example)
find_package(SWIG REQUIRED)
include(${SWIG_USE_FILE})
find_package(PythonLibs)
include_directories(${PYTHON_INCLUDE_PATH})
include_directories(${CMAKE_CURRENT_SOURCE_DIR})
set(CMAKE_SWIG_FLAGS "")
set_source_files_properties(../src/example.i PROPERTIES CPLUSPLUS ON)
swig_add_library(python_example
TYPE MODULE
LANGUAGE python
OUTPUT_DIR ../../py_out # move the .so to py_out
OUTFILE_DIR . # leave the .cpp in cmake-build-debug
SOURCES ../src/example.i
../src/example.cpp ../src/example.h
)
set_target_properties(python_example PROPERTIES
LIBRARY_OUTPUT_DIRECTORY ../../py_out # must match dir in OUTPUT_DIR
)
After building it, run python test.py to see it go. Note it's all in bash/Ubuntu, so MacOs should be ok, but windows may cause you some churn.
See the README for the full details.
I have protobuf compiler version 3.0 and need to install grpc and grpc python plugin. Following the tutorial, I added deb http://http.debian.net/debian jessie-backports main to my sources.list file and did sudo apt-get update and sudo apt-get install libgrpc-dev which returned
Package libgrpc-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'libgrpc-dev' has no installation candidate
So, I decided to compile it from source as mentioned in INSTALL notes and did:
$ git clone https://github.com/grpc/grpc.git
$ cd grpc
$ git submodule update --init
$ make
$ [sudo] make install
However, on the make step, I get
[MAKE] Generating cache.mk
make: Circular /home/vagrant/grpc2/grpc/libs/opt/libboringssl.a <- /home/vagrant/grpc2/grpc/libs/opt/libboringssl.a dependency dropped.
[C] Compiling third_party/boringssl/crypto/bio/connect.c
third_party/boringssl/crypto/bio/connect.c: In function 'split_host_and_port':
third_party/boringssl/crypto/bio/connect.c:127:17: error: declaration of 'close' shadows a global declaration [-Werror=shadow]
cc1: all warnings being treated as errors
make: *** [/home/vagrant/grpc2/grpc/objs/opt/third_party/boringssl/crypto/bio/connect.o] Error 1
On switching to the release-0_11 branch, running make results in
[HOSTCXX] Compiling src/compiler/csharp_generator.cc
src/compiler/csharp_generator.cc:47:43: error: 'google::protobuf::compiler::csharp::GetUmbrellaClassName' has not been declared
src/compiler/csharp_generator.cc: In function 'void grpc_csharp_generator::{anonymous}::GenerateServiceDescriptorProperty(grpc::protobuf::io::Printer*, const ServiceDescriptor*)':
src/compiler/csharp_generator.cc:237:62: error: 'GetUmbrellaClassName' was not declared in this scope
make: *** [/home/vagrant/grpc2/grpc/objs/opt/src/compiler/csharp_generator.o] Error 1
I can't figure out how to install this. Any help would be appreciated.
For me the issue got fixed after i made change in the file as:
diff --git a/src/compiler/csharp_generator.cc
b/src/compiler/csharp_generator.cc
index 7b497df..5a8746d 100644
--- a/src/compiler/csharp_generator.cc
+++ b/src/compiler/csharp_generator.cc
## -44,7 +44,7 ##
using google::protobuf::compiler::csharp::GetFileNamespace;
using google::protobuf::compiler::csharp::GetClassName;
-using google::protobuf::compiler::csharp::GetUmbrellaClassName;
+using google::protobuf::compiler::csharp::GetReflectionClassName;
using grpc::protobuf::FileDescriptor;
using grpc::protobuf::Descriptor;
using grpc::protobuf::ServiceDescriptor;
## -234,7 +234,7 ## void GenerateServiceDescriptorProperty(Printer* out, const ServiceDescriptor *se
out->Print("public static global::Google.Protobuf.Reflection.ServiceDescriptor Descriptor\n");
out->Print("{\n");
out->Print(" get { return $umbrella$.Descriptor.Services[$index$]; }\n",
- "umbrella", GetUmbrellaClassName(service->file()), "index",
+ "umbrella", GetReflectionClassName(service->file()), "index",
index.str());
out->Print("}\n");
out->Print("\n");
More specifically, please open the file src/compiler/csharp_generator.cc and replace all references of GetUmbrellaClassName by GetReflectionClassName
Working backwards:
for release-0_11: it looks like you're trying to compile against the most recent protobuf. Since we're both in development right now there's occasional breakage - but grpc does track the version of protobuf it's tested against in third_party/protobuf. Try checking out and installing that version. I filed https://github.com/grpc/grpc/issues/4697 to update to the latest protobuf 3.0 version.
for master from github: which compiler and OS are you using? I recently checked in the boringssl integration work, so it's fresh, and not nearly as battle-tested. I'd like to get it battle-tested. That said, if you do a 'make EMBED_OPENSSL=false' then things should work out for you.
for the debian package problem: I'm not sure what's going on. I'm happy to try and spin up a VM with your OS and repro if you can let me know which OS it is.
I am trying to setup and compile the Hello World example for Boost.Python: http://www.boost.org/doc/libs/1_57_0/libs/python/doc/tutorial/doc/html/python/hello.html
I installed bjam, boost, boost-build, and boost-python from Homebrew:
brew install bjam
brew install boost
brew install boost-build
brew install boost-python
My python install is also via Homebrew. I am not sure how to properly modify the example Jamroot file so that it is compatible with my system setup. I changed the boost path to : /usr/local/Cellar/boost; but I'm not sure of the other paths that need to be changed. The current setup gives me the following error:
> bjam
notice: no Python configured in user-config.jam
notice: will use default configuration
Jamroot:26: in modules.load
*** argument error
* rule use-project ( id : where )
* called with: ( boost : /usr/local/Cellar/boost; project : requirements <library>/boost/python//boost_python <implicit-dependency>/boost//headers : usage-requirements <implicit-dependency>/boost//headers )
* extra argument project
/usr/local/share/boost-build/build/project.jam:1138:see definition of rule 'use-project' being called
/usr/local/share/boost-build/build/project.jam:311: in load-jamfile
/usr/local/share/boost-build/build/project.jam:64: in load
/usr/local/share/boost-build/build/project.jam:145: in project.find
/usr/local/share/boost-build/build-system.jam:535: in load
/usr/local/share/boost-build/kernel/modules.jam:289: in import
/usr/local/share/boost-build/kernel/bootstrap.jam:139: in boost-build
/usr/local/share/boost-build/boost-build.jam:8: in module scope
Summary
Don't use BJAM it's a waste of your time - I am assuming your interest in BJAM is a side-product of getting your code to actually work
Here is the quick-link to my github page where I do a hello_world example using namespace boost::python
See my github for linking multiple boost files into one import library
Longer Answer
I have exactly the same setup as you. I spent ages getting this to work as the documentation is really shady (as you know) and before you know it you go down some weird rabbit hole trying to hack make files and BJAM installations.
You can use a setup.py as you would normally with C code as follows...
Installation
You can obtain the correct boost-python through homebrew via the command:
brew install boost --with-python --build-from-source
I think brew install boost should work but it's a big install and life is short to do it twice
Boost code
Assume the following code in hello_ext.cpp
#include <boost/python.hpp>
char const* greet()
{
return "Greetings!";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
Python setup
Then you can write setup.py file as
from distutils.core import setup
from distutils.extension import Extension
hello_ext = Extension(
'hello_ext',
sources=['hello_ext.cpp'],
libraries=['boost_python-mt'],
)
setup(
name='hello-world',
version='0.1',
ext_modules=[hello_ext])
Compiling
The following example can be used by:
python setup.py build_ext --inplace
which will create the following build/ directory and file:
build/
hello_ext.so
Running
This can be directly now called by python with:
In [1]: import hello_ext
In [2]: hello_ext.greet()
Out[2]: 'Greetings!'
What am I missing in my Boost.Python configuration/installation?
I'm trying to compile tutorial example, and I get error with libboost_python not found
cd /usr/share/doc/libboost1.42-doc/examples/libs/python/example/tutorial
bjam
error: Unable to find file or target named
error: 'libboost_python'
error: referred from project at
error: '.'
But the library is there, ldconfig.real has been run:
/usr/lib/libboost_python.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py26.a -> libboost_python-py26.a
/usr/lib/libboost_python-mt-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-mt-py27.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py26.a
/usr/lib/libboost_python-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py27.a
/usr/lib/libboost_python-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py27.so.1.42.0
/usr/lib/libboost_python.so -> libboost_python-py27.so
I'm using default libboost packages from Ubuntu 11.04.
My user-config.jam is
using python : 2.7 ;
I had a similar problem on ubuntu 12.04 where I installed all the boost libraries as a package. I found the solution here:
http://jayrambhia.wordpress.com/2012/06/25/configuring-boostpython-and-hello-boost/
It turns out that you do not need to use bjam at all. A makefile suffices. I will repeat the the solution from the above link here:
1.) Install the libboost-python package
2.) Create a hello world source file called 'hello_ext.c':
char const* greet()
{
return "hello, world";
}
#include<boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet",greet);
}
3.) Create a makefile:
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = /usr/include
BOOST_LIB = /usr/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr /lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).c
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).c
4.) make
make
5.) Ready to use. In python:
import hello_ext
print hello_ext.greet()
Still not sure it that's the proper way, seems little hackish, but following helped:
In Jamroot file replaced
project
: requirements <library>libboost_python ;
with
project
: requirements <library>/usr/lib/libboost_python.so ;
You could have a site-config file with something like the following ;
using boost : 1.48 : <include>/usr/include/boost-1_48 <library>/usr/lib ;
(you need the < library > bit, not sure why)
then you can do stuff like.
project foo
: <library>/boost//python
Makes things easier in the long run, as you inevitably will have to change boost version at some point.