Develop Reference

Loading C++ dll in python doesn't work with C++ libs

So I am trying to make a C/C++ dll for a project but any C++ library I include in my header, ends up causing loading problems when I try to load it using ctypes in python. I'm guessing maybe ctypes doesn't have c++ libs paths included? I made a simple demonstration to my problem.
init2.h: Generic Header file for dll and execs using the dll
#ifndef INIT2_H
#define INIT2_H
#ifdef __cplusplus
extern "C" {
#endif
#include <stdio.h>
void func11();
#ifdef __cplusplus
}
#endif
#endif
init2.cc: dll code (loads fine)
// This works fine
#include "init2.h"
void func11() {
printf("Func11\n");
}
init.c: exe code that loads dll
#include "init2.h"
int main() {
func11();
return 0;
}
Makefile
CFLAGS = -Wall -std=c11 -g
CCFLAGS = -Wall -std=c++11 -g
all: libmain.dll main
main.o: init2.cc
g++ $(CCFLAGS) init2.cc -fpic -c -o main.o
libmain.dll: main.o
g++ $(CCFLAGS) -shared main.o -o libmain.dll
main: init.c
gcc $(CFLAGS) -L. -lmain init.c -o main
clean:
del *.o *.exe *.dll *.txt
script.py
from ctypes import *
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
lib.func11()
The above compiles, links and loads fine in the generated main.exe and script.py. The problem occurs when I add a c++ lib in my dll.
init2.cc: With iostream (loads in main.exe only)
#include "init2.h"
#include <iostream>
void func11() {
std::cout << "Func11\n";
}
Compiling that into the dll, while everything stays the same loads fine into the main.exe (init.c code) but gives me the following loading error in script.py.
Traceback (most recent call last):
File "D:\Projects\Learning-C++\script.py", line 3, in <module>
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'D:\Projects\Learning-C++\libmain.dll' (or one of its dependencies). Try using the full path with constructor syntax.
How can I fix this? My bad if these seems obvious, but I'm fairly new to C++ and haven't got the integration down yet. Thanks in advance!

The problem was as I expected due to cdll not able to find the C++ library. The file name for my case was libstdc++-6.dll. Which is located in the bin folder in my compiler's directory. To make cdll search for the dependencies in that folder I did the following:
import os
# add dependency directory
os.add_dll_directory('C:\\Program Files\\CodeBlocks\\MinGW\\bin')
# load the dll
lib = CDLL('D:\\Projects\\Learning-C++\\libmain.dll')
lib.func11()
The dll may have many dependencies located in other directories e.g. You defined libDLL.dll and bin/libDependency.dll. You will need to add the /path/to/dir/bin directory using the above method if you're trying to load libDLL.dll in python. You can find all the dependencies of a dll using dependency walker or
dumpbin /dependents libDLL.dll
in the VS dev powershell.

How to import a binary module generated by pybind11?

I'm trying to use pybind11 for the first time by following the tutorial documentation.
As the documentation says, I created the following example.cpp file with the following content:
#include <pybind11/pybind11.h>
int add(int i, int j) {
return i + j;
}
PYBIND11_MODULE(example, m) {
m.doc() = "pybind11 example plugin"; // optional module docstring
m.def("add", &add, "A function which adds two numbers");
}
And since I'm on macOS, based on this tutorial, I compiled the example file with the following command: (not sure if this is correct but seems to work)
c++ -O3 -Wall -shared -std=c++11 -undefined dynamic_lookup -I /usr/local/Cellar/pybind11/2.4.3/include example.cpp -o example`python3-config --extension-suffix --includes`
This command generates the example.cpython-37m-darwin.so file in the same directory.
And then I created example.py with the following content:
import example
example.add(1, 2)
But when I run the script with python example.py, it shows the following error:
Traceback (most recent call last):
File "example.py", line 1, in <module>
import example
File "/Users/cuinjune/pybind11/build/example.py", line 2, in <module>
example.add(1, 2)
AttributeError: 'module' object has no attribute 'add'
What is wrong and how to fix this?
EDIT: Previously, it somehow generated example.pyc file but I don't know how the file was generated and now it just says: ImportError: No module named example.

This line:
/Users/cuinjune/pybind11/build/example.py
says that there is (or was) another file in your work directory (or perhaps /Users/cuinjune/pybind11/build is in your PYTHONPATH) that is a pure Python file, and unrelated to your built example.cpython-37m-darwin.so. When that example.py was imported, presumably some time in the past, the example.pyc you saw was generated. The .pyc remains fully functional standalone, even when the originating example.py was removed. The .pyc file will have the full directory to the original file hardcoded in it, and thus will continue to refer to that file, even when it is removed.
From your update, now all other example.py, .pyc, .pyo are gone, right?
First make sure that the python interpreter you are running matches the tag that you used to build. For example by running:
python3 -c 'import distutils.sysconfig as ds; print(ds.get_config_var("EXT_SUFFIX"))'
(where I'm assuming you used python3 as the command to start the Python interpreter; if not, then change it to the relevant command).
Second, verify the PYTHONPATH envar to see whether the local directory (either fully spelled out, or .) is in it. If not, add it.
With those two covered, it should work ...
Personally, however, I'd first simplify my life by renaming the module to something more unique (rename both example in the .cpp as well as the target on the CLI).

Cython Compile Error: Python3 Compatibility

I am running Python 3.4.4 :: Anaconda 4.0.0 (x86_64) on OS X Yosemite. My Cython's version is 0.23.4. I'm trying to embed some very trivial Cython code test.pyx into C code testcc.c. The problem is, if I use python2.7-config then everything works well (Python 2.7 is the built-in version on OS X). However if I use python3.4-config the following errors raised:
Undefined symbols for architecture x86_64:
"_inittest", referenced from:
_main in testcc-b22dcf.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I have to use Python3 since all my other codes are written in it. Please help me solve this problem.
The following are my source files:
test.pyx:
cdef public void pythonAdd(int[] a):
a[1] = 5
a[0] = 4
testcc.c:
#include "Python.h"
#include "test.h"
#include <math.h>
#include <stdio.h>
int main(int argc, char **argv) {
Py_Initialize();
inittest();
int a [2] = {0 , 0};
pythonAdd(a);
printf("fist: %d, second: %d", a[0], a[1]);
Py_Finalize();
return 0;
}
And Compiling those two files using following setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("testc", sources=["test.pyx"])
setup(name="testc", ext_modules=cythonize(ext))
The following is the command I compile those c files:
ldflags:=$(shell $(python3.4-config) --ldflags)
cflags:=$(shell $(python3.4-config) --cflags)
python setup.py build_ext --inplace
cython test.pyx
gcc $(cflags) $(ldflags) test.c testcc.c -o cysvm.out
Update:
I changed the inittest() to PyInit_test() as Jim suggested. The code compiles successfully. However when I run ./cysvm.out the following errors occured:
./cysvm.out
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
Current thread 0x00007fff772f5300 (most recent call first):
Update
I solved this by adding the following line before Py_Initialize(); in my c code:
Py_SetPythonHome(L"/PATH/to/python3");

This is probably due to the fact that in Python 3.x initialization of modules is not performed by calling init<module_name> but rather with PyInit_<module_name> (See PEP 3121). So, if you are linking with Python 3.x and executing via 3.x you need to change the initialization call.
In short, changing the call that initializes the module from:
inittest();
To:
PyInit_test();
and recompiling, should do the trick.
As for your second problem, an alternate solution other than using Py_SetPythonHome is setting PYTHONHOME to the output of python3.4-config --exec-prefix (or sys.exec_prefix) prior to compilation.

How to use Boost.Python

I just recently discovered Boost.Python and I am trying to figure out how it works. I tried to go through the tutorial on the official website. However, I got
link.jam: No such file or directory
when running bjam as in the example (which appears to be just a warning),
and
Traceback (most recent call last):
File "hello.py", line 7, in <module>
import hello_ext
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
when running python hello.py.
I also tried to compile a module as described in another tutorial with similar results. I am running Ubuntu14.04 with boost1.55 compiled myself.
I tried to compile the following:
#include <boost/python.hpp>
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
with the following command from command line:
g++ -o hello_ext.so hello.cpp -I /usr/include/python2.7/ -I /home/berardo/boost_1_55_0/ -L /usr/lib/python2.7/ -L /home/berardo/boost/lib/ -lboost_python -lpython2.7 -Wl, -fPIC -expose-dynamic
which still gives me a:
/usr/bin/ld: impossibile trovare : File o directory non esistente
collect2: error: ld returned 1 exit status.

Finally, I was able to make it work. First, I fixed the linker issues, as suggested by Dan. It finally compiled but I still got:
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
The problem was that the python module was not able to load correctly so I needed to add another linker option. Here, I report the final Makefile:
# location of the Python header file
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = ${HOME}/boost/include
BOOST_LIB = ${HOME}/boost/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,-rpath,$(BOOST_LIB) -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr/lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).C
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).C
Notice the -Wl,-rpath, option, which apparently makes the new created shared library available to the python script.
#Dan: Thanks for the valuable hints.

bjam `Unable to find file or target named 'libboost_python'`

What am I missing in my Boost.Python configuration/installation?
I'm trying to compile tutorial example, and I get error with libboost_python not found
cd /usr/share/doc/libboost1.42-doc/examples/libs/python/example/tutorial
bjam
error: Unable to find file or target named
error: 'libboost_python'
error: referred from project at
error: '.'
But the library is there, ldconfig.real has been run:
/usr/lib/libboost_python.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py26.a -> libboost_python-py26.a
/usr/lib/libboost_python-mt-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-mt-py27.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py26.a
/usr/lib/libboost_python-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py27.a
/usr/lib/libboost_python-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py27.so.1.42.0
/usr/lib/libboost_python.so -> libboost_python-py27.so
I'm using default libboost packages from Ubuntu 11.04.
My user-config.jam is
using python : 2.7 ;

I had a similar problem on ubuntu 12.04 where I installed all the boost libraries as a package. I found the solution here:
http://jayrambhia.wordpress.com/2012/06/25/configuring-boostpython-and-hello-boost/
It turns out that you do not need to use bjam at all. A makefile suffices. I will repeat the the solution from the above link here:
1.) Install the libboost-python package
2.) Create a hello world source file called 'hello_ext.c':
char const* greet()
{
return "hello, world";
}
#include<boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet",greet);
}
3.) Create a makefile:
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = /usr/include
BOOST_LIB = /usr/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr /lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).c
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).c
4.) make
make
5.) Ready to use. In python:
import hello_ext
print hello_ext.greet()

Still not sure it that's the proper way, seems little hackish, but following helped:
In Jamroot file replaced
project
: requirements <library>libboost_python ;
with
project
: requirements <library>/usr/lib/libboost_python.so ;

You could have a site-config file with something like the following ;
using boost : 1.48 : <include>/usr/include/boost-1_48 <library>/usr/lib ;
(you need the < library > bit, not sure why)
then you can do stuff like.
project foo
: <library>/boost//python
Makes things easier in the long run, as you inevitably will have to change boost version at some point.