Develop Reference

Compiling source code that uses Boost, wxWidgets and Python with MingW

I have been trying to compile a source code but every time I use the make command, it gives me the error:
In file included from ../src/MadEditFrame.cpp:60:0:
../src/EmbeddedPython.hpp:11:20: fatal error: Python.h: No such file or directory
#include <Python.h>
I had been able to pass the Boost and wxWidget part. This Python part is what makes no sense. I have Python 3.8 installed, it is recognised by system when in command line i use the Python --version command as 3.8.2 g++ too is recognised as 6.3.0 with Mingw.
Update:
The code has three makefiles with terminations: install, mac and mingw. I'm using the one with mingw by copying and deleting termination. The content of said file is too long, but there is a part that goes this:
...
./Release/nsUTF8Prober.obj \
$(RES)
LINKOBJ = $(OBJ)
#WXWIN = D:\wxWidgets28_SVN
WXWIN = C:\wxWidgets-3.1.5
#XPRESSIVE_DIR = N:/boost
XPRESSIVE_DIR = ../xpressive
#BOOST_DIR = D:/boost
BOOST_DIR = C:\Programs\boost_1_77_0\boost
LIBS = -L$(WXWIN)/lib/gcc_lib -L"../libunicows" -mwindows -lwxmsw28u_core -lwxbase28u -lunicows -lkernel32 -luser32 -lgdi32 -lcomdlg32 -lwinspool -lwinmm -lshell32 -lcomctl32 -lole32 -loleaut32 -luuid -lrpcrt4 -ladvapi32 -lwsock32 -lodbc32 -lopengl32 -lgmon -s
#-g -pg
CXXINCS = -I$(XPRESSIVE_DIR) -I$(BOOST_DIR) -I"../charset-detector/include" -I"../charset-detector/src" -I"../charset-detector/src/tables" -I$(WXWIN)/include -I$(WXWIN)/lib/gcc_lib/mswu
BIN = Release/MadEdit.exe
CXXFLAGS = $(CXXINCS) -D_UNICODE -D__WXMSW__ -D__GNUWIN32__ -D__WIN95__ -DWIN32 -O2
#-g -pg
RM = rm -f
.PHONY: all all-before all-after clean clean-custom
all: all-before Release/MadEdit.exe all-after
...
I added the paths under the commented ones for WXWIN and BOOSTDIR

Tensorflow custom op with OpenCV undefined symbol

I'm writing a custom operation for Tensorflow that is supposed to load a video. For this, I need to include OpenCV.
For now, the operation simply tries to open a VideoCapture and returns an empty tensor.
Here's the C++ code:
#include "opencv2/opencv.hpp"
#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"
#include "tensorflow/core/framework/op.h"
#include "tensorflow/core/framework/shape_inference.h"
#include "tensorflow/core/framework/op_kernel.h"
#include <iostream>
using namespace tensorflow;
using namespace cv;
using namespace std;
using shape_inference::ShapeHandle;
using shape_inference::DimensionHandle;
REGISTER_OP("LoadVideo")
.Input("filename: string")
.Output("frame: float32")
.SetShapeFn([](::tensorflow::shape_inference::InferenceContext* c) {
TensorShape outputTensorShape({224, 224, 3});
ShapeHandle outputShapeHandle;
c->MakeShapeFromTensorShape(outputTensorShape, &outputShapeHandle);
c->set_output(0, outputShapeHandle);
return Status::OK();
});
class LoadVideoOp : public OpKernel {
public:
explicit LoadVideoOp(OpKernelConstruction* context) : OpKernel(context) {}
void Compute(OpKernelContext* context) override {
// Grab the input tensor
const Tensor& input_tensor = context->input(0);
auto input = input_tensor.flat<string>();
string filename = input(0);
VideoCapture cap = VideoCapture("data/0eRkpTGq5pA.mp4");
Tensor* output_tensor = NULL;
OP_REQUIRES_OK(context, context->allocate_output(0, {224, 224, 3}, &output_tensor));
}
};
REGISTER_KERNEL_BUILDER(Name("LoadVideo").Device(DEVICE_CPU), LoadVideoOp);
Then, I use the following command to compile the code:
g++ -std=c++11 -shared -fPIC \
-I /home/master/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/include \
-I ~/anaconda3/envs/tf/include/opencv2/ -I ~/anaconda3/envs/tf/include/opencv/ -O2 \
-L ~/anaconda3/envs/tf/lib \
load_video.cc -o load_video.so \
-lopencv_core -lopencv_videoio -lopencv_highgui \
-lopencv_imgproc -lopencv_video -lopencv_objdetect
When I load the compiled code into a Python script (using tf.load_op_library) and try to run the op I get the following error:
tensorflow.python.framework.errors_impl.NotFoundError: lib/ops/load_video.so: undefined symbol: _ZN2cv12VideoCaptureC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
It looks like the compiled C++ code cannot access the appropriate OpenCV object. I don't know much about C++ compilation and linking, so the problem probably is that I'm compiling the custom op in a wrong way.
Could you please help me compile the op in such way, that it can be successfully loaded and run by tensorflow?
EDIT 1:
This is the Python script I use to load the custom op:
import tensorflow as tf
load_video_module = tf.load_op_library('lib/ops/load_video.so')
with tf.Session():
x = load_video_module.load_video("data/0eRkpTGq5pA.mp4").eval()
print(x)
The error happens on the line 2 (i.e. when trying to load the compiled C++ code).
Solution:
I managed to successfully compile and run the custom tensorflow op after rebuilding OpenCV. The compilation command is:
g++ -std=c++11 -ggdb -shared -I`python -c 'import tensorflow as tf; print(tf.sysconfig.get_include())'` `pkg-config --cflags opencv` -o load_video.so load_video.cc `pkg-config --libs opencv` -fPIC

You can check if your library requires some missing library using ldd.
Just check ldd load_video.so.
However, probably you're not linking some shared library that some OpenCV method you're using requires.
To be sure to link and include every library needed you can use pkg-config.
Remove your manually -I and -l flags that point to OpenCV libs and just add pkg-config --libs --cflags opencv that do the complete work (of including and linking libraries) for you

If you are running on Linux, cd into the directory that contains your video file and then perform, without the parentheses:
sudo chmod 777 (the name of your video file)
This should give your program access to the video file.
I don't know much about C++, but TensorFlow often throws this error when it's denied permission, so give it a shot and good luck!

exposing C++ class in Python ( only ET_DYN and ET_EXEC can be loaded)

I was looking at here to see how to expose c++ to Python. I have built Python deep learning code which uses boost-python to connect c++ and python and it is running ok, so my system has things for boost-python alread setup.
Here is my hello.cpp code (where I used WorldC and WorldP to clearly show the C++ and Python class name usage in the declaration. I don't know why the original web page is using the same class name World to cause confusion to beginners.)
#include <boost/python.hpp>
using namespace boost::python;
struct WorldC
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
BOOST_PYTHON_MODULE(hello)
{
class_<WorldC>("WorldP")
.def("greet", &WorldC::greet)
.def("set", &WorldC::set)
;
}
and this is how I made hello.so
g++ -shared -c -o hello.so -fPIC hello.cpp -lboostpython -lpython2.7 -I/usr/local/include/python2.7
When I run import hello in python, it gives me this error.
>>> import hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./hello.so: only ET_DYN and ET_EXEC can be loaded
Can anybody tell me what's wrong?
(I'm using anaconda2 under my home directory for python environment, but since my deep learning code builds ok with boost-python, there should be no problem including boost/python.hpp in my system directory)

I've long forgotten about this problem and got to revisit this issue today.
I found two problems. The first problem was that I gave -c option which made the compiler only compile the source and not link. The second problem was that the library name was wrong(I search /usr/lib64 and there was libboost_python.so, so it should be -lboost_python instead of -lboostpython). So the correct way to build it is :
g++ -shared -o hello.so -fPIC hello.cpp -lboost_python -lpython2.7 -I/usr/local/include/python2.7
I found it runs ok in python. :)

How to use Boost.Python

I just recently discovered Boost.Python and I am trying to figure out how it works. I tried to go through the tutorial on the official website. However, I got
link.jam: No such file or directory
when running bjam as in the example (which appears to be just a warning),
and
Traceback (most recent call last):
File "hello.py", line 7, in <module>
import hello_ext
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
when running python hello.py.
I also tried to compile a module as described in another tutorial with similar results. I am running Ubuntu14.04 with boost1.55 compiled myself.
I tried to compile the following:
#include <boost/python.hpp>
char const* greet()
{
return "hello, world";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
with the following command from command line:
g++ -o hello_ext.so hello.cpp -I /usr/include/python2.7/ -I /home/berardo/boost_1_55_0/ -L /usr/lib/python2.7/ -L /home/berardo/boost/lib/ -lboost_python -lpython2.7 -Wl, -fPIC -expose-dynamic
which still gives me a:
/usr/bin/ld: impossibile trovare : File o directory non esistente
collect2: error: ld returned 1 exit status.

Finally, I was able to make it work. First, I fixed the linker issues, as suggested by Dan. It finally compiled but I still got:
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
The problem was that the python module was not able to load correctly so I needed to add another linker option. Here, I report the final Makefile:
# location of the Python header file
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = ${HOME}/boost/include
BOOST_LIB = ${HOME}/boost/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,-rpath,$(BOOST_LIB) -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr/lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).C
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).C
Notice the -Wl,-rpath, option, which apparently makes the new created shared library available to the python script.
#Dan: Thanks for the valuable hints.

Swig wrapping C++ to python is using default Python rather than Anaconda python in Mac OS X

I am trying to wrap c++ functions into python using swig. I am using following commands
swig -c++ -python helloworld.i
g++ -O2 -fPIC -c hello.cpp
g++ -O2 -fPIC -c helloworld_wrap.cxx -I//anaconda/include/python2.7
g++ -lpython -dynamclib hello.o helloworld_wrap.o -o _helloworld.so
with hello.cpp being initial file with functions and helloworld.i being file with wrapper. These commands creates the library helloworldbut I can only import it through default python in /usr/bin/python
If I try to import it through python installed through anaconda it gives following error:
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6
Can you tell me how can I wrap the codes with python from anaconda?
Found a solution : Python.h not found using swig and Anaconda Python
In above question, the top answer gives an explanation of using disutilsand a set up file in python to build the library. This works wonders :)
The next problem I am having for wrapping simple class:
my class code from [example] (http://web.mit.edu/svn/src/swig-1.3.25/Examples/python/class/index.html)
/* File : example.h */
class Shape {
public:
Shape() {
nshapes++;
}
virtual ~Shape() {
nshapes--;
};
double x, y;
void move(double dx, double dy);
virtual double area() = 0;
virtual double perimeter() = 0;
static int nshapes;
};
class Circle : public Shape {
private:
double radius;
public:
Circle(double r) : radius(r) { };
virtual double area();
virtual double perimeter();
};
class Square : public Shape {
private:
double width;
public:
Square(double w) : width(w) { };
virtual double area();
virtual double perimeter();
};
My setup.py file :
#setup.py file:
from setuptools import setup, Extension
setup(name='example',
version='0.1',
ext_modules=[Extension('_example', ['example.h', 'example.i'],
swig_opts=['-c++'],
)],
)
Code I am using to wrap :
python setup.py build_ext --inplace
Error message:
running build_ext
building '_example' extension
swigging example.i to example_wrap.cpp
swig -python -c++ -o example_wrap.cpp example.i
error: unknown file type '.h' (from 'example.h')
Can you suggest what is wrong here. I suppose it is not recognizing '.h' file but as it is header file, I thought it could be kept as it is. Also if my setup.py file is correct or not? I am just trying to follow example for simple wrapping, there is no simple tutorial online apparently.
I can also ask this question on other different question but thought just continuing here for now.

Answer by Warren Weckesser in similar question
. I used the setup.py file as suggested in the answer and added the path to the library to sys.path and it work wonders :)
Use the option -I/Users/myuser/anaconda/include/python2.7 in the gcc command. (That's assuming you are using python 2.7. Change the name to match the version of python that you are using.) You can use the command python-config --cflags to get the full set of recommended compilation flags:
$ python-config --cflags
-I/Users/myuser/anaconda/include/python2.7 -I/Users/myuser/anaconda/include/python2.7 -fno-strict-aliasing -I/Users/myuser/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
However, to build the extension module, I recommend using a simple setup script, such as the following setup.py, and let distutils figure out all the compiling and linking options for you.
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example_wrap.c', 'example.c'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
Then you can run:
$ swig -python example.i
$ python setup.py build_ext --inplace
(Take a look at the compiler commands that are echoed to the terminal when setup.py is run.)
distutils knows about SWIG, so instead of including example_wrap.c in the list of source files, you can include example.i, and swig will be run automatically by the setup script:
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example.c', 'example.i'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
With the above version of setup.py, you can build the extension module with the single command
$ python setup.py build_ext --inplace
Once you've built the extension module, you should be able to use it in python:
>>> import example
>>> example.fact(5)
120
If you'd rather not use the script setup.py, here's a set of commands that worked for me:
$ swig -python example.i
$ gcc -c -I/Users/myuser/anaconda/include/python2.7 example.c example_wrap.c
$ gcc -bundle -undefined dynamic_lookup -L/Users/myuser/anaconda/lib example.o example_wrap.o -o _example.so
Note: I'm using Mac OS X 10.9.4:
$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.3.0
Thread model: posix