I'm writing some libraries in C which contain functions that I want to call from Python via ctypes.
I've done this successfully another library, but that library had only very vanilla dependencies (namely fstream, math, malloc, stdio, stdlib). The other library I'm working on has more complicated dependencies.
For example, I'll try to use fftw3. As a test, I'll just try to compile a simple .cpp file containing:
int foo()
{
void *p = fftw_malloc( sizeof(fftw_complex)*64 );
fftw_free(p);
printf("foo called.\n");
return 0;
}
I compile it as:
icpc -Wall -fPIC -c waveprop.cpp -o libwaveprop.o $std_link
icpc -shared -Wl,-soname,libwaveprop.so.1 -o libwaveprop.so.1.0 libwaveprop.o
cp waveprop.so.1.0 /usr/local/lib/
rm waveprop.so.1.0
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so.1
This all works. Now I test it with another .cpp file containing:
int main()
{
foo();
}
Result:
icpc test.cpp -lwaveprop
/lib/../lib64/libwaveprop.so: undefined reference to `fftw_free'
/lib/../lib64/libwaveprop.so: undefined reference to `fftw_malloc'
Which is entirely reasonable. Next I try:
icpc test.cpp -lwaveprop -lfftw3
./a.out
foo called.
Great! But now when I try to load the library with ctypes:
>>> from ctypes import *
>>> print cdll.LoadLibrary('/usr/local/lib/libwaveprop.so.1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/ctypes/__init__.py", line 431, in LoadLibrary
return self._dlltype(name)
File "/usr/lib64/python2.6/ctypes/__init__.py", line 353, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/libwaveprop.so.1: undefined symbol: fftw_free
So it's the same problem, but I have no idea how to resolve it for ctypes. I've tried various things without any success, and I'm pretty stuck at this point.
OK, thanks for your help.
to get this to work I had to include the dependencies when linking (duh). I had tried this before but got an error, so solve this I had to recompile fftw with '-fpic' as a CPP flag. all works now.
icpc -Wall -fPIC -c waveprop.cpp -o libwaveprop.o $std_link
icpc -shared -Wl,-soname,libwaveprop.so.1 -o libwaveprop.so.1.0 libwaveprop.o -lfftw3
cp waveprop.so.1.0 /usr/local/lib/
rm waveprop.so.1.0
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so
ln -sf /usr/local/lib/waveprop.so.1.0 /usr/local/lib/waveprop.so.1
thanks,
-nick
You need to link libwaveprop.so itself against the fftw3 library. Otherwise Python simply won't know where to go to get those missing symbols; mind-reading isn't implemented in any programming language.
Related
So I am trying to make a C/C++ dll for a project but any C++ library I include in my header, ends up causing loading problems when I try to load it using ctypes in python. I'm guessing maybe ctypes doesn't have c++ libs paths included? I made a simple demonstration to my problem.
init2.h: Generic Header file for dll and execs using the dll
#ifndef INIT2_H
#define INIT2_H
#ifdef __cplusplus
extern "C" {
#endif
#include <stdio.h>
void func11();
#ifdef __cplusplus
}
#endif
#endif
init2.cc: dll code (loads fine)
// This works fine
#include "init2.h"
void func11() {
printf("Func11\n");
}
init.c: exe code that loads dll
#include "init2.h"
int main() {
func11();
return 0;
}
Makefile
CFLAGS = -Wall -std=c11 -g
CCFLAGS = -Wall -std=c++11 -g
all: libmain.dll main
main.o: init2.cc
g++ $(CCFLAGS) init2.cc -fpic -c -o main.o
libmain.dll: main.o
g++ $(CCFLAGS) -shared main.o -o libmain.dll
main: init.c
gcc $(CFLAGS) -L. -lmain init.c -o main
clean:
del *.o *.exe *.dll *.txt
script.py
from ctypes import *
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
lib.func11()
The above compiles, links and loads fine in the generated main.exe and script.py. The problem occurs when I add a c++ lib in my dll.
init2.cc: With iostream (loads in main.exe only)
#include "init2.h"
#include <iostream>
void func11() {
std::cout << "Func11\n";
}
Compiling that into the dll, while everything stays the same loads fine into the main.exe (init.c code) but gives me the following loading error in script.py.
Traceback (most recent call last):
File "D:\Projects\Learning-C++\script.py", line 3, in <module>
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'D:\Projects\Learning-C++\libmain.dll' (or one of its dependencies). Try using the full path with constructor syntax.
How can I fix this? My bad if these seems obvious, but I'm fairly new to C++ and haven't got the integration down yet. Thanks in advance!
The problem was as I expected due to cdll not able to find the C++ library. The file name for my case was libstdc++-6.dll. Which is located in the bin folder in my compiler's directory. To make cdll search for the dependencies in that folder I did the following:
import os
# add dependency directory
os.add_dll_directory('C:\\Program Files\\CodeBlocks\\MinGW\\bin')
# load the dll
lib = CDLL('D:\\Projects\\Learning-C++\\libmain.dll')
lib.func11()
The dll may have many dependencies located in other directories e.g. You defined libDLL.dll and bin/libDependency.dll. You will need to add the /path/to/dir/bin directory using the above method if you're trying to load libDLL.dll in python. You can find all the dependencies of a dll using dependency walker or
dumpbin /dependents libDLL.dll
in the VS dev powershell.
I am working on writing a wrapper around C++ functions to use in Python. Thus, I am trying to use Boost.Python as an experiment first. Below are the function I want to wrap:
hello_exp.cpp:
char const* greet()
{
return "hello, world!";
}
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
my Makefile:
COMPILER = g++
CPPFLAGS = -g -Wall -std=c++11 -stdlib=libc++
# Python and BoostPython links.
BOOSTHEADERS = -I/usr/local/Cellar/boost/1.64.0_1/include/boost/
BOOSTLIBRARIES = -L/usr/local/Cellar/boost-python/1.64.0/lib/
BOOSTLIB = -lboost_python
PYTHONHEADERS = -I/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/include/python3.6m
PYTHONLIBRARIES = -L/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib
PYTHONLIB = -lpython3.6
# Collect links.
LIBRARIES = $(BOOSTLIBRARIES) $(PYTHONLIBRARIES) $(PYTHONLIB) $(BOOSTLIB)
HEADERS = $(BOOSTHEADERS) $(PYTHONHEADERS)
# Build target.
TARGET = hello_ext
# BEGIN MAKE
all: $(TARGET)
$(TARGET): $(TARGET).cpp
$(COMPILER) -shared $(TARGET).cpp $(LIBRARIES) $(HEADERS) -o $(TARGET).so
clean:
$(RM) $(TARGET)
However, after some experiments, I am perpetually stuck at this error...:
Undefined symbols for architecture x86_64:
"boost::python::detail::init_module(PyModuleDef&, void (*)())", referenced from:
_PyInit_hello_ext in hello_ext-476eb2.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [hello_ext.so] Error 1
So I have a installed python3 through HomeBrew, as well as boost and boost-python. Note that I actually installed the boost-python library without python2 support and only with python3 support.
Thank you in advance!
After some fixing and digging around, it turns out the problem is that the boost-python library I installed is still in python2. so instead, make sure you do
brew rm boost-python
brew install boost-python --with-python3 --without-python
to get the right version. Then the make file is just change
BOOSTLIB = -lboost_python
to
BOOSTLIB = -lboost_python3
And then just hit make :)
I was looking at here to see how to expose c++ to Python. I have built Python deep learning code which uses boost-python to connect c++ and python and it is running ok, so my system has things for boost-python alread setup.
Here is my hello.cpp code (where I used WorldC and WorldP to clearly show the C++ and Python class name usage in the declaration. I don't know why the original web page is using the same class name World to cause confusion to beginners.)
#include <boost/python.hpp>
using namespace boost::python;
struct WorldC
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
BOOST_PYTHON_MODULE(hello)
{
class_<WorldC>("WorldP")
.def("greet", &WorldC::greet)
.def("set", &WorldC::set)
;
}
and this is how I made hello.so
g++ -shared -c -o hello.so -fPIC hello.cpp -lboostpython -lpython2.7 -I/usr/local/include/python2.7
When I run import hello in python, it gives me this error.
>>> import hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./hello.so: only ET_DYN and ET_EXEC can be loaded
Can anybody tell me what's wrong?
(I'm using anaconda2 under my home directory for python environment, but since my deep learning code builds ok with boost-python, there should be no problem including boost/python.hpp in my system directory)
I've long forgotten about this problem and got to revisit this issue today.
I found two problems. The first problem was that I gave -c option which made the compiler only compile the source and not link. The second problem was that the library name was wrong(I search /usr/lib64 and there was libboost_python.so, so it should be -lboost_python instead of -lboostpython). So the correct way to build it is :
g++ -shared -o hello.so -fPIC hello.cpp -lboost_python -lpython2.7 -I/usr/local/include/python2.7
I found it runs ok in python. :)
I just recently discovered Boost.Python and I am trying to figure out how it works. I tried to go through the tutorial on the official website. However, I got
link.jam: No such file or directory
when running bjam as in the example (which appears to be just a warning),
and
Traceback (most recent call last):
File "hello.py", line 7, in <module>
import hello_ext
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
when running python hello.py.
I also tried to compile a module as described in another tutorial with similar results. I am running Ubuntu14.04 with boost1.55 compiled myself.
I tried to compile the following:
#include <boost/python.hpp>
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
with the following command from command line:
g++ -o hello_ext.so hello.cpp -I /usr/include/python2.7/ -I /home/berardo/boost_1_55_0/ -L /usr/lib/python2.7/ -L /home/berardo/boost/lib/ -lboost_python -lpython2.7 -Wl, -fPIC -expose-dynamic
which still gives me a:
/usr/bin/ld: impossibile trovare : File o directory non esistente
collect2: error: ld returned 1 exit status.
Finally, I was able to make it work. First, I fixed the linker issues, as suggested by Dan. It finally compiled but I still got:
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
The problem was that the python module was not able to load correctly so I needed to add another linker option. Here, I report the final Makefile:
# location of the Python header file
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = ${HOME}/boost/include
BOOST_LIB = ${HOME}/boost/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,-rpath,$(BOOST_LIB) -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr/lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).C
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).C
Notice the -Wl,-rpath, option, which apparently makes the new created shared library available to the python script.
#Dan: Thanks for the valuable hints.
I have a firmware for an USB module I can already control by visual C. Now I want to port this to python. for this I need the octopus library which is written in c. I found a file called octopus_wrap which was created by SWIG!
then I found a makefile which says:
python2.5:
swig -python -outdir ./ ../octopus.i
gcc -fPIC -c ../../liboctopus/src/octopus.c
gcc -fPIC -c ../octopus_wrap.c -I /usr/include/python2.5
gcc -fPIC -shared octopus_wrap.o octopus.o /usr/lib/libusb.so -o _octopus.so
python2.4:
swig -python -outdir ./ ../octopus.i
gcc -fPIC -c ../../liboctopus/src/octopus.c
gcc -fPIC -c ../octopus_wrap.c -I /usr/include/python2.4
gcc -fPIC -shared octopus_wrap.o octopus.o /usr/lib/libusb.so -o _octopus.so
win:
gcc -fPIC -c ../../liboctopus/src/octopus.c -I /c/Programme/libusb-win32-device-bin-0.1.10.1/include
gcc -fPIC -c octopus_wrap.c -I /c/Python25/libs -lpython25 -I/c/Python25/include -I /c/Programme/libusb-win32-device-bin-0.1.10.1/include
gcc -fPIC -shared *.o -o _octopus.pyd -L/c/Python25/libs -lpython25 -lusb -L/c/Programme/libusb-win32-device-bin-0.1.10.1/lib/gcc
clean:
rm -f octopus* _octopus*
install_python2.4:
cp _octopus.so /usr/local/lib/python2.4/site-packages/
cp octopus.py /usr/local/lib/python2.4/site-packages/
install_python2.5:
cp _octopus.so /usr/local/lib/python2.5/site-packages/
cp octopus.py /usr/local/lib/python2.5/site-packages/
I dont know how to handle this but as far as I can see octopus.py and _octopus.so are the resulting output files which are relevant to python right?
luckily someone already did that and so I put those 2 files to my "python26/lib" folder (hope it doesnt matter if it´s python 2.5 or 2.6?!)
So when working with the USB device the octopus.py is the library to work with!
Importing this file makes several problems:
>>>
Traceback (most recent call last):
File "C:\Users\ameise\My Dropbox\µC\AVR\OCTOPUS\octopususb-0.5\demos\python \blink_status.py", line 8, in <module>
from octopus import *
File "C:\Python26\lib\octopus.py", line 7, in <module>
import _octopus
ImportError: DLL load failed: module not found.
and here´s the related line 7 :
import _octopus
So there´s a problem considering the .so file!
What could be my next step?
I know that´s a lot of confusing stuff but I hope anyone of you could bring some light in my mind!
thy in advance
You should link and compile for the python2.6 -lpython26.
Also the file extension for windows is .pyd no .so