Develop Reference

Loading C++ dll in python doesn't work with C++ libs

So I am trying to make a C/C++ dll for a project but any C++ library I include in my header, ends up causing loading problems when I try to load it using ctypes in python. I'm guessing maybe ctypes doesn't have c++ libs paths included? I made a simple demonstration to my problem.
init2.h: Generic Header file for dll and execs using the dll
#ifndef INIT2_H
#define INIT2_H
#ifdef __cplusplus
extern "C" {
#endif
#include <stdio.h>
void func11();
#ifdef __cplusplus
}
#endif
#endif
init2.cc: dll code (loads fine)
// This works fine
#include "init2.h"
void func11() {
printf("Func11\n");
}
init.c: exe code that loads dll
#include "init2.h"
int main() {
func11();
return 0;
}
Makefile
CFLAGS = -Wall -std=c11 -g
CCFLAGS = -Wall -std=c++11 -g
all: libmain.dll main
main.o: init2.cc
g++ $(CCFLAGS) init2.cc -fpic -c -o main.o
libmain.dll: main.o
g++ $(CCFLAGS) -shared main.o -o libmain.dll
main: init.c
gcc $(CFLAGS) -L. -lmain init.c -o main
clean:
del *.o *.exe *.dll *.txt
script.py
from ctypes import *
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
lib.func11()
The above compiles, links and loads fine in the generated main.exe and script.py. The problem occurs when I add a c++ lib in my dll.
init2.cc: With iostream (loads in main.exe only)
#include "init2.h"
#include <iostream>
void func11() {
std::cout << "Func11\n";
}
Compiling that into the dll, while everything stays the same loads fine into the main.exe (init.c code) but gives me the following loading error in script.py.
Traceback (most recent call last):
File "D:\Projects\Learning-C++\script.py", line 3, in <module>
lib = CDLL('D:\Projects\Learning-C++\libmain.dll')
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3312.0_x64__qbz5n2kfra8p0\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'D:\Projects\Learning-C++\libmain.dll' (or one of its dependencies). Try using the full path with constructor syntax.
How can I fix this? My bad if these seems obvious, but I'm fairly new to C++ and haven't got the integration down yet. Thanks in advance!

The problem was as I expected due to cdll not able to find the C++ library. The file name for my case was libstdc++-6.dll. Which is located in the bin folder in my compiler's directory. To make cdll search for the dependencies in that folder I did the following:
import os
# add dependency directory
os.add_dll_directory('C:\\Program Files\\CodeBlocks\\MinGW\\bin')
# load the dll
lib = CDLL('D:\\Projects\\Learning-C++\\libmain.dll')
lib.func11()
The dll may have many dependencies located in other directories e.g. You defined libDLL.dll and bin/libDependency.dll. You will need to add the /path/to/dir/bin directory using the above method if you're trying to load libDLL.dll in python. You can find all the dependencies of a dll using dependency walker or
dumpbin /dependents libDLL.dll
in the VS dev powershell.

Tensorflow custom op with OpenCV undefined symbol

I'm writing a custom operation for Tensorflow that is supposed to load a video. For this, I need to include OpenCV.
For now, the operation simply tries to open a VideoCapture and returns an empty tensor.
Here's the C++ code:
#include "opencv2/opencv.hpp"
#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"
#include "tensorflow/core/framework/op.h"
#include "tensorflow/core/framework/shape_inference.h"
#include "tensorflow/core/framework/op_kernel.h"
#include <iostream>
using namespace tensorflow;
using namespace cv;
using namespace std;
using shape_inference::ShapeHandle;
using shape_inference::DimensionHandle;
REGISTER_OP("LoadVideo")
.Input("filename: string")
.Output("frame: float32")
.SetShapeFn([](::tensorflow::shape_inference::InferenceContext* c) {
TensorShape outputTensorShape({224, 224, 3});
ShapeHandle outputShapeHandle;
c->MakeShapeFromTensorShape(outputTensorShape, &outputShapeHandle);
c->set_output(0, outputShapeHandle);
return Status::OK();
});
class LoadVideoOp : public OpKernel {
public:
explicit LoadVideoOp(OpKernelConstruction* context) : OpKernel(context) {}
void Compute(OpKernelContext* context) override {
// Grab the input tensor
const Tensor& input_tensor = context->input(0);
auto input = input_tensor.flat<string>();
string filename = input(0);
VideoCapture cap = VideoCapture("data/0eRkpTGq5pA.mp4");
Tensor* output_tensor = NULL;
OP_REQUIRES_OK(context, context->allocate_output(0, {224, 224, 3}, &output_tensor));
}
};
REGISTER_KERNEL_BUILDER(Name("LoadVideo").Device(DEVICE_CPU), LoadVideoOp);
Then, I use the following command to compile the code:
g++ -std=c++11 -shared -fPIC \
-I /home/master/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/include \
-I ~/anaconda3/envs/tf/include/opencv2/ -I ~/anaconda3/envs/tf/include/opencv/ -O2 \
-L ~/anaconda3/envs/tf/lib \
load_video.cc -o load_video.so \
-lopencv_core -lopencv_videoio -lopencv_highgui \
-lopencv_imgproc -lopencv_video -lopencv_objdetect
When I load the compiled code into a Python script (using tf.load_op_library) and try to run the op I get the following error:
tensorflow.python.framework.errors_impl.NotFoundError: lib/ops/load_video.so: undefined symbol: _ZN2cv12VideoCaptureC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
It looks like the compiled C++ code cannot access the appropriate OpenCV object. I don't know much about C++ compilation and linking, so the problem probably is that I'm compiling the custom op in a wrong way.
Could you please help me compile the op in such way, that it can be successfully loaded and run by tensorflow?
EDIT 1:
This is the Python script I use to load the custom op:
import tensorflow as tf
load_video_module = tf.load_op_library('lib/ops/load_video.so')
with tf.Session():
x = load_video_module.load_video("data/0eRkpTGq5pA.mp4").eval()
print(x)
The error happens on the line 2 (i.e. when trying to load the compiled C++ code).
Solution:
I managed to successfully compile and run the custom tensorflow op after rebuilding OpenCV. The compilation command is:
g++ -std=c++11 -ggdb -shared -I`python -c 'import tensorflow as tf; print(tf.sysconfig.get_include())'` `pkg-config --cflags opencv` -o load_video.so load_video.cc `pkg-config --libs opencv` -fPIC

You can check if your library requires some missing library using ldd.
Just check ldd load_video.so.
However, probably you're not linking some shared library that some OpenCV method you're using requires.
To be sure to link and include every library needed you can use pkg-config.
Remove your manually -I and -l flags that point to OpenCV libs and just add pkg-config --libs --cflags opencv that do the complete work (of including and linking libraries) for you

If you are running on Linux, cd into the directory that contains your video file and then perform, without the parentheses:
sudo chmod 777 (the name of your video file)
This should give your program access to the video file.
I don't know much about C++, but TensorFlow often throws this error when it's denied permission, so give it a shot and good luck!

exposing C++ class in Python ( only ET_DYN and ET_EXEC can be loaded)

I was looking at here to see how to expose c++ to Python. I have built Python deep learning code which uses boost-python to connect c++ and python and it is running ok, so my system has things for boost-python alread setup.
Here is my hello.cpp code (where I used WorldC and WorldP to clearly show the C++ and Python class name usage in the declaration. I don't know why the original web page is using the same class name World to cause confusion to beginners.)
#include <boost/python.hpp>
using namespace boost::python;
struct WorldC
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
BOOST_PYTHON_MODULE(hello)
{
class_<WorldC>("WorldP")
.def("greet", &WorldC::greet)
.def("set", &WorldC::set)
;
}
and this is how I made hello.so
g++ -shared -c -o hello.so -fPIC hello.cpp -lboostpython -lpython2.7 -I/usr/local/include/python2.7
When I run import hello in python, it gives me this error.
>>> import hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./hello.so: only ET_DYN and ET_EXEC can be loaded
Can anybody tell me what's wrong?
(I'm using anaconda2 under my home directory for python environment, but since my deep learning code builds ok with boost-python, there should be no problem including boost/python.hpp in my system directory)

I've long forgotten about this problem and got to revisit this issue today.
I found two problems. The first problem was that I gave -c option which made the compiler only compile the source and not link. The second problem was that the library name was wrong(I search /usr/lib64 and there was libboost_python.so, so it should be -lboost_python instead of -lboostpython). So the correct way to build it is :
g++ -shared -o hello.so -fPIC hello.cpp -lboost_python -lpython2.7 -I/usr/local/include/python2.7
I found it runs ok in python. :)

Py_DECREF segmentation fault

I am having trouble embedding Python in C++. I am using Mingw w64 gcc and 64 bit Python 2.7.11.
#include <Python.h>
int main(int argc, char *argv[]) {
Py_Initialize();
PyObject* pName = PyString_FromString("test");
Py_DECREF(pName);
Py_Finalize();
return 0;
}
Calls to compiler:
g++ "-IC:\\Python27\\include" -O0 -g3 -Wall -c -fmessage-length=0 -std=c++11 -o "src\\main.o" "..\\src\\main.cpp"
g++ "-LC:\\Python27\\libs" -std=c++11 -o pytest.exe "src\\main.o" -lpython27
The problem is that it segfaults in Py_DECREF. I have tried expanding the macros, and have traced segfault to the following statement:
((*(((PyObject*) ((PyObject *) (pName)))->ob_type)->tp_dealloc)((PyObject *) ((PyObject *) (pName))));
Turns out, tp_dealloc points to 0x1.
The same problem happens in the example code provided in Python docs:
https://docs.python.org/2/extending/embedding.html#pure-embedding
If I remove some of the calls to Py_DECREF(pName) and Py_DECREF(pArgs), code from the docs works as intended. Yet every example I've found on the web (including the one from the Python docs) does call Py_DECREF.
What could be the cause of this error? Could there be some inconsistency in my build environment?

So, apparently, something is wrong with my environment. I tried compiling the same code on another PC and there was no segfault anymore.

Can C programs have Python GUI?

My friend has an application written in C that comes with a GUI made using GTK under Linux. Now we want to rewrite the GUI in python (wxpython or PyQT).
I don't have experience with Python and don't know how to make Python communicate with C. I'd like to know if this is possible and if yes, how should I go about implementing it?

Yes its possible to call 'C' functions from Python.
Please look into SWIG(deprecated) also Python provides its own Extensibility API. You might want to look into that.
Also google CTypes.
LINKS:
Python Extension
A simple example:
I used Cygwin on Windows for this. My python version on this machine is 2.6.8 - tested it with test.py loading the module called "myext.dll" - it works fine. You might want to modify the Makefile to make it work on your machine.
original.h
#ifndef _ORIGINAL_H_
#define _ORIGINAL_H_
int _original_print(const char *data);
#endif /*_ORIGINAL_H_*/
original.c
#include <stdio.h>
#include "original.h"
int _original_print(const char *data)
{
return printf("o: %s",data);
}
stub.c
#include <Python.h>
#include "original.h"
static PyObject *myext_print(PyObject *, PyObject *);
static PyMethodDef Methods[] = {
{"printx", myext_print, METH_VARARGS,"Print"},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC initmyext(void)
{
PyObject *m;
m = Py_InitModule("myext",Methods);
}
static PyObject *myext_print(PyObject *self, PyObject *args)
{
const char *data;
int no_chars_printed;
if(!PyArg_ParseTuple(args, "s", &data)){
return NULL;
}
no_chars_printed = _original_print(data);
return Py_BuildValue("i",no_chars_printed);
}
Makefile
PYTHON_INCLUDE = -I/usr/include/python2.6
PYTHON_LIB = -lpython2.6
USER_LIBRARY = -L/usr/lib
GCC = gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -DMAJOR_VERSION=1 -DMINOR_VERSION=0 -I/usr/include -I/usr/include/python2.6
win32 : myext.o
- gcc -shared myext.o $(USER_LIBRARY) $(PYTHON_LIB) -o myext.dll
linux : myext.o
- gcc -shared myext.o $(USER_LIBRARY) $(PYTHON_LIB) -o myext.so
myext.o: stub.o original.o
- ld -r stub.o original.o -o myext.o
stub.o: stub.c
- $(GCC) -c stub.c -o stub.o
original.o: original.c
- $(GCC) -c original.c -o original.o
clean: myext.o
- rm stub.o original.o stub.c~ original.c~ Makefile~
test.py
import myext
myext.printx('hello world')
OUTPUT
o: hello world

Sorry but i don't have python experience so don't know how to make Python communicate with C program.
Yes, that's exactly how you do it. Turn your C code into a Python module, and then you can write the entire GUI in Python. See Extending and Embedding the Python Interpreter.

If you have an option between C or C#(Sharp) then go with C# and use visual studio, you can build the GUI by dragging and dropping components easy. If you want to do something in python look up wxPython. Java has a built in GUI builder known as swing. You'll need some tutorials, but unless this program doesn't need to be portable just go with C# and build it in 10 minutes .
Also, you can write your code in C and export it as a python module which you can load from python. It`s not very complicated to set up some C functions and have a python GUI which calls them. To achieve this you can use SWIG, Pyrex, BOOST.