I just recently discovered Boost.Python and I am trying to figure out how it works. I tried to go through the tutorial on the official website. However, I got
link.jam: No such file or directory
when running bjam as in the example (which appears to be just a warning),
and
Traceback (most recent call last):
File "hello.py", line 7, in <module>
import hello_ext
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
when running python hello.py.
I also tried to compile a module as described in another tutorial with similar results. I am running Ubuntu14.04 with boost1.55 compiled myself.
I tried to compile the following:
#include <boost/python.hpp>
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
with the following command from command line:
g++ -o hello_ext.so hello.cpp -I /usr/include/python2.7/ -I /home/berardo/boost_1_55_0/ -L /usr/lib/python2.7/ -L /home/berardo/boost/lib/ -lboost_python -lpython2.7 -Wl, -fPIC -expose-dynamic
which still gives me a:
/usr/bin/ld: impossibile trovare : File o directory non esistente
collect2: error: ld returned 1 exit status.
Finally, I was able to make it work. First, I fixed the linker issues, as suggested by Dan. It finally compiled but I still got:
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
The problem was that the python module was not able to load correctly so I needed to add another linker option. Here, I report the final Makefile:
# location of the Python header file
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = ${HOME}/boost/include
BOOST_LIB = ${HOME}/boost/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,-rpath,$(BOOST_LIB) -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr/lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).C
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).C
Notice the -Wl,-rpath, option, which apparently makes the new created shared library available to the python script.
#Dan: Thanks for the valuable hints.
Related
I currently have a source files written in C++ and wrapped into a Python module (uses Boost and Parser libs).
I ported the C++ folder under go/src and .so files along with main.go
Program Structure
src
/main
main.go
network.so
/network
file1.cpp (this has a function **object DBdata::getTable())
file1.hpp (#define FILE1_H_)
main.go
package main
// #cgo pkg-config: python3
// #cgo CFLAGS : -I./ -I/usr/include/python3.6
// #cgo LDFLAGS: -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6 -lpthread -ldl -lutil -lm
// #include <Python.h>
import "C"
import "fmt"
func main() {
cmd := exec.Command("python", "-c", "import network; network.getTable()")
cmd.Dir = "/home/username/go/src/network" //directory where is my python code
out,err := cmd.CombinedOutput()
}
After building this main.go, I get the error as
/usr/include/boost/config/no_tr1/memory.hpp: fatal error: memory: No
such file or directory
> # include
> ^~~~~~~~
> compilation terminated.
How to import .so as Python modules in Go?
Can Swig be used in this place?
What is the better approach to expose Python module to Go?
Issue fixed. The .so placed in /bin and the Go was build and could access the functions under network
I am working on writing a wrapper around C++ functions to use in Python. Thus, I am trying to use Boost.Python as an experiment first. Below are the function I want to wrap:
hello_exp.cpp:
char const* greet()
{
return "hello, world!";
}
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
my Makefile:
COMPILER = g++
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
# Python and BoostPython links.
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.64.0_1/include/boost/
BOOSTLIBRARIES = -L/usr/local/Cellar/boost-python/1.64.0/lib/
BOOSTLIB = -lboost_python
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/include/python3.6m
PYTHONLIBRARIES = -L/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib
PYTHONLIB = -lpython3.6
# Collect links.
LIBRARIES = $(BOOSTLIBRARIES) $(PYTHONLIBRARIES) $(PYTHONLIB) $(BOOSTLIB)
HEADERS = $(BOOSTHEADERS) $(PYTHONHEADERS)
# Build target.
TARGET = hello_ext
# BEGIN MAKE
all: $(TARGET)
$(TARGET): $(TARGET).cpp
$(COMPILER) -shared $(TARGET).cpp $(LIBRARIES) $(HEADERS) -o $(TARGET).so
clean:
$(RM) $(TARGET)
However, after some experiments, I am perpetually stuck at this error...:
Undefined symbols for architecture x86_64:
"boost::python::detail::init_module(PyModuleDef&, void (*)())", referenced from:
_PyInit_hello_ext in hello_ext-476eb2.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [hello_ext.so] Error 1
So I have a installed python3 through HomeBrew, as well as boost and boost-python. Note that I actually installed the boost-python library without python2 support and only with python3 support.
Thank you in advance!
After some fixing and digging around, it turns out the problem is that the boost-python library I installed is still in python2. so instead, make sure you do
brew rm boost-python
brew install boost-python --with-python3 --without-python
to get the right version. Then the make file is just change
BOOSTLIB = -lboost_python
to
BOOSTLIB = -lboost_python3
And then just hit make :)
I was looking at here to see how to expose c++ to Python. I have built Python deep learning code which uses boost-python to connect c++ and python and it is running ok, so my system has things for boost-python alread setup.
Here is my hello.cpp code (where I used WorldC and WorldP to clearly show the C++ and Python class name usage in the declaration. I don't know why the original web page is using the same class name World to cause confusion to beginners.)
#include <boost/python.hpp>
using namespace boost::python;
struct WorldC
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
BOOST_PYTHON_MODULE(hello)
{
class_<WorldC>("WorldP")
.def("greet", &WorldC::greet)
.def("set", &WorldC::set)
;
}
and this is how I made hello.so
g++ -shared -c -o hello.so -fPIC hello.cpp -lboostpython -lpython2.7 -I/usr/local/include/python2.7
When I run import hello in python, it gives me this error.
>>> import hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./hello.so: only ET_DYN and ET_EXEC can be loaded
Can anybody tell me what's wrong?
(I'm using anaconda2 under my home directory for python environment, but since my deep learning code builds ok with boost-python, there should be no problem including boost/python.hpp in my system directory)
I've long forgotten about this problem and got to revisit this issue today.
I found two problems. The first problem was that I gave -c option which made the compiler only compile the source and not link. The second problem was that the library name was wrong(I search /usr/lib64 and there was libboost_python.so, so it should be -lboost_python instead of -lboostpython). So the correct way to build it is :
g++ -shared -o hello.so -fPIC hello.cpp -lboost_python -lpython2.7 -I/usr/local/include/python2.7
I found it runs ok in python. :)
I am trying to wrap c++ functions into python using swig. I am using following commands
swig -c++ -python helloworld.i
g++ -O2 -fPIC -c hello.cpp
g++ -O2 -fPIC -c helloworld_wrap.cxx -I//anaconda/include/python2.7
g++ -lpython -dynamclib hello.o helloworld_wrap.o -o _helloworld.so
with hello.cpp being initial file with functions and helloworld.i being file with wrapper. These commands creates the library helloworldbut I can only import it through default python in /usr/bin/python
If I try to import it through python installed through anaconda it gives following error:
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6
Can you tell me how can I wrap the codes with python from anaconda?
Found a solution : Python.h not found using swig and Anaconda Python
In above question, the top answer gives an explanation of using disutilsand a set up file in python to build the library. This works wonders :)
The next problem I am having for wrapping simple class:
my class code from [example] (http://web.mit.edu/svn/src/swig-1.3.25/Examples/python/class/index.html)
/* File : example.h */
class Shape {
public:
Shape() {
nshapes++;
}
virtual ~Shape() {
nshapes--;
};
double x, y;
void move(double dx, double dy);
virtual double area() = 0;
virtual double perimeter() = 0;
static int nshapes;
};
class Circle : public Shape {
private:
double radius;
public:
Circle(double r) : radius(r) { };
virtual double area();
virtual double perimeter();
};
class Square : public Shape {
private:
double width;
public:
Square(double w) : width(w) { };
virtual double area();
virtual double perimeter();
};
My setup.py file :
#setup.py file:
from setuptools import setup, Extension
setup(name='example',
version='0.1',
ext_modules=[Extension('_example', ['example.h', 'example.i'],
swig_opts=['-c++'],
)],
)
Code I am using to wrap :
python setup.py build_ext --inplace
Error message:
running build_ext
building '_example' extension
swigging example.i to example_wrap.cpp
swig -python -c++ -o example_wrap.cpp example.i
error: unknown file type '.h' (from 'example.h')
Can you suggest what is wrong here. I suppose it is not recognizing '.h' file but as it is header file, I thought it could be kept as it is. Also if my setup.py file is correct or not? I am just trying to follow example for simple wrapping, there is no simple tutorial online apparently.
I can also ask this question on other different question but thought just continuing here for now.
Answer by Warren Weckesser in similar question
. I used the setup.py file as suggested in the answer and added the path to the library to sys.path and it work wonders :)
Use the option -I/Users/myuser/anaconda/include/python2.7 in the gcc command. (That's assuming you are using python 2.7. Change the name to match the version of python that you are using.) You can use the command python-config --cflags to get the full set of recommended compilation flags:
$ python-config --cflags
-I/Users/myuser/anaconda/include/python2.7 -I/Users/myuser/anaconda/include/python2.7 -fno-strict-aliasing -I/Users/myuser/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
However, to build the extension module, I recommend using a simple setup script, such as the following setup.py, and let distutils figure out all the compiling and linking options for you.
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example_wrap.c', 'example.c'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
Then you can run:
$ swig -python example.i
$ python setup.py build_ext --inplace
(Take a look at the compiler commands that are echoed to the terminal when setup.py is run.)
distutils knows about SWIG, so instead of including example_wrap.c in the list of source files, you can include example.i, and swig will be run automatically by the setup script:
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example.c', 'example.i'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
With the above version of setup.py, you can build the extension module with the single command
$ python setup.py build_ext --inplace
Once you've built the extension module, you should be able to use it in python:
>>> import example
>>> example.fact(5)
120
If you'd rather not use the script setup.py, here's a set of commands that worked for me:
$ swig -python example.i
$ gcc -c -I/Users/myuser/anaconda/include/python2.7 example.c example_wrap.c
$ gcc -bundle -undefined dynamic_lookup -L/Users/myuser/anaconda/lib example.o example_wrap.o -o _example.so
Note: I'm using Mac OS X 10.9.4:
$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.3.0
Thread model: posix
I'm writing some libraries in C which contain functions that I want to call from Python via ctypes.
I've done this successfully another library, but that library had only very vanilla dependencies (namely fstream, math, malloc, stdio, stdlib). The other library I'm working on has more complicated dependencies.
For example, I'll try to use fftw3. As a test, I'll just try to compile a simple .cpp file containing:
int foo()
{
void *p = fftw_malloc( sizeof(fftw_complex)*64 );
fftw_free(p);
printf("foo called.\n");
return 0;
}
I compile it as:
icpc -Wall -fPIC -c waveprop.cpp -o libwaveprop.o $std_link
icpc -shared -Wl,-soname,libwaveprop.so.1 -o libwaveprop.so.1.0 libwaveprop.o
cp waveprop.so.1.0 /usr/local/lib/
rm waveprop.so.1.0
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so.1
This all works. Now I test it with another .cpp file containing:
int main()
{
foo();
}
Result:
icpc test.cpp -lwaveprop
/lib/../lib64/libwaveprop.so: undefined reference to `fftw_free'
/lib/../lib64/libwaveprop.so: undefined reference to `fftw_malloc'
Which is entirely reasonable. Next I try:
icpc test.cpp -lwaveprop -lfftw3
./a.out
foo called.
Great! But now when I try to load the library with ctypes:
>>> from ctypes import *
>>> print cdll.LoadLibrary('/usr/local/lib/libwaveprop.so.1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/ctypes/__init__.py", line 431, in LoadLibrary
return self._dlltype(name)
File "/usr/lib64/python2.6/ctypes/__init__.py", line 353, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/libwaveprop.so.1: undefined symbol: fftw_free
So it's the same problem, but I have no idea how to resolve it for ctypes. I've tried various things without any success, and I'm pretty stuck at this point.
OK, thanks for your help.
to get this to work I had to include the dependencies when linking (duh). I had tried this before but got an error, so solve this I had to recompile fftw with '-fpic' as a CPP flag. all works now.
icpc -Wall -fPIC -c waveprop.cpp -o libwaveprop.o $std_link
icpc -shared -Wl,-soname,libwaveprop.so.1 -o libwaveprop.so.1.0 libwaveprop.o -lfftw3
cp waveprop.so.1.0 /usr/local/lib/
rm waveprop.so.1.0
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so.1
thanks,
-nick
You need to link libwaveprop.so itself against the fftw3 library. Otherwise Python simply won't know where to go to get those missing symbols; mind-reading isn't implemented in any programming language.