Develop Reference

How to call function/classes in c++ .so files, generated by Bazel, in Python?

Let's say I have a simple class in hello.h
#ifndef LIB_HELLO_GREET_H_
#define LIB_HELLO_GREET_H_
class A{
public:
int a = 0;
int b = 0;
int add(){
return a+b;
}
};
#endif
with bazel build file:
load("#rules_cc//cc:defs.bzl", "cc_binary", "cc_library")
cc_library(
name = "hello",
hdrs = ["hello.h"],
)
cc_binary(
name = "hello.so",
deps = [
":hello",
],
linkshared=True,
linkstatic=False
)
After I run bazel build hello.so, there is a shared object file generated in bazel-bin/main and bazel-bin/main/hello.so.runfiles/__main__/main/hello.so. Using those files, I want call class A with a python script. Ideally, I'd want to use cppyy or something similar.
I've tried with simple python scripts
import cppyy
cppyy.load_reflection_info('hello')
print(dir(cppyy.gbl))
or
import cppyy
cppyy.load_library('hello')
print(dir(cppyy.gbl))
with both .so files, but cppyy can't seem to detect class A - it is never inside cppyy.gbl
I'm wondering the best way to solve this problem, any help appreciated!

With cppyy, you also need to give it the header file, that is:
cppyy.include("hello.h")
and optionally use cppyy.add_include_path() to add the directory where hello.h resides, so that it can be found.
An alternative is to use so-called "dictionaries." These package (customizable) directory locations, names of necessary headers files and (through linking) the shared library with the implementation into a single, new, shared library (named "dictionary" because of history). There can be a so-called "rootmap" file on the side that allows a class loader to find A automatically on first use. See the example in the documentation.

Loading C library in Python that dlopens another C library - unresolved shared symbol

I have a commercial c library (a.so) that has several functions. When you call the a.open() function, it performs a dlopen() call for another library dynamically. If calling a.open('b'), it will open b.so. If calling a.open('c'), it will open c.so.
The problem is that a.so and b.so share a global variable defined in a.so, but referenced by b.so (and c.so,etc.). I am able to load a.so correctly in python using ctypes and see all the symbols in Python. However, when I call a.open('b'), it attempts to load b.so but returns undefined symbol.
//a.c -source for a.so library
int aglobal = 0;
void open(char* lib)
{ dlopen(lib); }
//b.c - source for b.so library
extern int aglobal;
Here is my python code to load:
from ctypes import cdll
p = ctypes.CDLL('a.so')
p.open('b')
returns the error code: undefined symbol: aglobal
Some other notes:
files are linked with -fPIC -rdynamic -shared
When I write a C program that does the same as the python program,
there is not a problem.
I've tried swig to wrap the library also, and numerous other things, build options, etc. but same results.
Is Python binding the symbols differently or something?

You need to use RTLD_GLOBAL when you load a.so.
The object's symbols shall be made available for the relocation processing of any other object. In addition, symbol lookup using dlopen(0, mode) and an associated dlsym() allows objects loaded with this mode to be searched.
The Linux man page is a bit more straightforward:
The symbols defined by this shared object will be made
available for symbol resolution of subsequently loaded shared
objects.
The python documentation describes the presence of the option but does not describe what it does.
New in version 2.6: The use_last_error and use_errno optional parameters were added.
ctypes.RTLD_GLOBAL
Flag to use as mode parameter. On platforms where this flag is not available, it is defined as the integer zero.
ctypes.RTLD_LOCAL
Flag to use as mode parameter. On platforms where this is not available, it is the same as RTLD_GLOBAL.
ctypes.DEFAULT_MODE
The default mode which is used to load shared libraries. On OSX 10.3, this is RTLD_GLOBAL, otherwise it is the same as RTLD_LOCAL.
From the documentation, it appears you are on a system where DEFAULT_MODE is the same as RTLD_LOCAL, which is the converse of RTLD_GLOBAL.

How to enable the pyd to call functions in external C++ dll?

I'm trying to design an external DLL for my python program. Now I could use the C++ and Visual Studio 2010 to produce a file with a postfix of ".pyd". If the .pyd is not attached with other .dll files produced by C++, this python library could work well.
However, now I need to design a .pyd file with such a struction:
A.pyd -> 1.dll
-> 2.dll
in which the files 1,2 are C libraries. The functions in these libraries are called when producing A.pyd.
Although this .pyd file could be produced without errors by VS 2010, it could not work in python 3.6. The error report is as follow:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed, could not find the target program.
Even when the referred 1.dll and 2.dll are stored in the same folder that contains A.pyd, this error still exists. I wonder how could I enable the python-C++ library to call the functions in these dynamic link libraries based on C.

OK! Now I have found the correct way to perform this operation!
Wrap the DLL
At first, if the dll (I call it dll-A) that you need to call has not exported any function (you could use dumpbin in VS command to check the exported functions), you need to wrap the original dll-A. Use the implicit call to include dll-A. If there is a function declared in the original .h/.lib, like this:
int func(int a, int b);
Then you need to create a new dll project and wrap the above function by this:
.h
extern "C" _declspec(dllexport) int w_func(int a, int b);
.cpp
int w_func(int a, int b){
return func(a, b);
}
Certainly, if the dumpbin shows that dll-A have avaliable exported functions, you could skip this step.
After exporting this dll (I call it dll-B), you will get 'dll-B' and its depending files (including dll-A and dll-A's depending files).
Write the .pyd file
Use the explicit call to refer dll-B. When using this method, you should not include any lib/.h files, because dll-B itself could provide you with enough interfaces. You could load the dll by this method:
.h
typedef int (*func_ptr)(int a, int b);
.cpp (part of a function named Testdll when you write the .pyd project)
func_ptr FUNC_API = NULL;
HINSTANCE h = LoadLibraryA("libdll/dllB.dll");
//Certainly! you could set the folder of stored dlls!
if (h){
FUNC_API = (func_ptr)GetProcAddress(h, "w_func");
//You could load more functions here.
}
else{ // If the dll could not be found
FreeLibrary(h);
return 0x100;
}
int errorflag=0;
if (FUNC_API==NULL){ //Check whether the function is valid.
cout << "Could not find: func" << endl;
errorflag = errorflag | 0x001;
}
//You could check more functions here.
if (errorflag!=0){ // if any function could not be found.
FreeLibrary(h);
return 0x100 | errorflag;
}
//process functions.
int a,b,c;
c = FUNC_API(a,b);
//Free the lib
if (h)
FreeLibrary(h);
After build your own .pyd, you could get your python database (I call it pyd-C).
Link the .pyd with Python
In python project, you could test this file by this method:
.py
import pydC
if __name__ == '__main__':
X = pydC.cclass()
X.Testdll();
Then you can find that the function is performed well.
Noted that your .pyd should be in the same folder with where .py is. Because you have set dll-B in libdll/dllB.dll, dll-B should be put in the folder named libdll. However, because dll-B calls dll-A and other depending dlls implicitly, dll-A and other files should be in your workspace folder, i.e. the same folder with where .py is.
In short, you need to enter the workspace folder, and the folder formation is as follow:
./
Test.py
pydC.pyd
dllA.dll
dllA_depended1.dll
...
libdll/
dllB.dll

Python callback invocation from C++ fails on native site

I have a shared library (DLL) in C++ with some C-style API which I use from Python. There is a function which takes a C callback as an argument. As far as I understood, Python extension module (a separate DLL) is required to do that. This module shall pass a native "proxy" callback to API, then call Python object from that callback. I try do use it as follows (it fails):
from ctypes import CDLL
from test_extension import set_python_callback
def callback():
pass
if __name__ == '__main__':
library = CDLL('test_dll.shared-library')
set_python_callback(library, callback)
library.trigger_callback()
I'm using Windows 8 x64 environment with Python 2.7.6 (32 bit) and MinGW 32-bit compiler (mingw32). Python extension is built by distutils. API library can be changed if necessary, however, Python-specific code has to be kept in a separate library.
Here is my code, reduced. All error-checking was removed, however, when performed it showed no errors either from Python API or from Windows API.
API library:
/* typedef void (*callback_t)(); */
static callback_t s_callback = nullptr;
void DLL_PUBLIC set_callback(callback_t callback) {
s_callback = callback;
}
void DLL_PUBLIC trigger_callback() {
s_callback(); // <--------------------------(1)
}
Python extension module function set_python_callback(library, callback) takes a ctypes.CDLL object and Python callable. It then extracts native DLL handle from the former:
PyObject* handle_object = PyObject_GetAttrString(library, "_handle");
const HMODULE handle = static_cast<const HMODULE>(
PyLong_AsVoidPtr(handle_object));
const native_set_callback_t native_api_routine =
reinterpret_cast<const native_set_callback_t>(
GetProcAddress(handle, "set_callback"));
native_api_routine(common_callback);
C function common_callback() is implemented as follows:
extern "C" void DLL_PUBLIC common_callback() {
// <--------------------------------------- (2)
PyObject *arguments = Py_BuildValue("()");
PyObject *callback_result = PyEval_CallObject(s_callback, arguments);
Py_DECREF(arguments);
Py_XDECREF(callback_result);
}
The error message says:
Invoking callback from DLL... Traceback (most recent call last):
File "script.py", line 14, in <module>
library.trigger_callback()
WindowsError: exception: access violation reading 0x00000028
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute. The strange thing is, changing the way common_callback() call Python code (e. g. passing Py_None instead of empty tuple) changes the address in the error message.
Maybe this is somehow related to the calling convention, but I have no idea where it's wrong exactly or how to fix it.

Solved the issue by registering the thread for GIL as described in documentation.
Please note that original API DLL, Python extension DLL, and the script were executed in single-threaded mode and no additional thread had been created, either Python-managed, or not. However, there had been a synchronization issue for sure, because using Py_AddPendingCall() also worked. (Using is is discouraged; I found it analyzing signals module sources, it lead to the solution above.)
This statement of mine was incorrect:
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute.
I was mislead by the fact that some of Python API functions still worked (e. g. Py_BuildValue or Py_Initialize), but most didn't (e. g. PySys_WriteStdout, PyObject_Call, etc).

list exported functions from dll with ctypes

Is there any way to know which functions are exported from the dll through python foreign function library ctypes?
And if possible to know details about the exported functions through ctypes.
If yes, could someone provide a snippet of code?

If you are on Linux, there is a handy utility nm to list the content of a shared library (there is always a handy utility on Linux, especially for C stuff).
Here is the question about it.
You use it with the -D flag: nm -D ./libMyLib.so

I don't think ctypes offers this functionality. On Windows with visual studio:
DUMPBIN -EXPORTS XXX.DLL
Or for mingw on windows:
objdump -p XXX.dll

In general, this is not possible, because, again in general, dynamically loaded libraries do not carry the meta-information you require. It may be possible to obtain that information in certain special cases through system-specific ways, but ctypes itself does not fetch that information. You can record such info via ctypes (see e.g. the restype and argtypes attributes of function pointers), but only after you have obtained it by different means.

The below approach worked for both Windows and Ubuntu. For Windows, Cygwin is required.
Suppose, there is a c file like below which name is test.c.
int func1(int a, int b){
return a + b;
}
int func2(int a, int b){
return a - b;
}
And the above c codes were compiled to test.dll file with the below commands:
gcc -shared -Wl,-soname,adder -o test.dll -fPIC test.c
And the below Python script finds which functions of the test.dll can be used by Python.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from subprocess import Popen, PIPE
out = Popen(
args="nm ./test.dll",
shell=True,
stdout=PIPE
).communicate()[0].decode("utf-8")
attrs = [
i.split(" ")[-1].replace("\r", "")
for i in out.split("\n") if " T " in i
]
from ctypes import CDLL
functions = [i for i in attrs if hasattr(CDLL("./test.dll"), i)]
print(functions)
The output I got in Windows is as below:
['func1', 'func2']
The output I got in Ubuntu is as below:
['_fini', 'func1', 'func2', '_init']
The items of the above list are objects of _FuncPtr class.

#Mark's answer uses Visual Studio tools.
On windows you can also use Dependency Walker to get the function names of dll exports.
Sometimes names are mangled and can't be used as a valid python function name.
You can use getattr to get a handle to mangled functions, e.g:
mylib = ctypes.cdll('mylib.dll')
my_func = getattr(mylib, '_my_func#0')
my_func()

If you've also got the source for said library, and you're looking for a fully automated all-python way, you could use pycparser
for the file: prog.c
typedef short int ret_t;
typedef short int param_t;
ret_t add(param_t a, param_t b) {
return (ret_t)(a + b);
}
ret_t passthrough(ret_t (* func)(param_t a, param_t b), param_t a, param_t b) {
// parameter intentionally altered.
// if this isn't done, compiler will deem entire function redundant
return func(a, b + 1);
}
compiling with gcc
gcc -I. -E ./prog.c > prog-preproc.c
gives us the pre-processed c file: prog-preproc.c
then in python:
import pycparser
parser = pycparser.c_parser.CParser()
with open('prog-preproc.c', 'r') as fh:
ast = parser.parse(fh.read())
class FunctionVisitor(pycparser.c_ast.NodeVisitor):
def visit_FuncDef(self, node):
print("found function: %s" % node.decl.name)
#node.show()
FunctionVisitor().visit(ast)
yields
found function: add
found function: passthrough
To dig further you can also get parameter and return types.
Uncomment node.show() for more information from within the Abstract Syntax Tree (AST)
edit: I've written a library to do a lot of the heavy lifting on this : https://github.com/fragmuffin/prlsc/tree/main/lib/py (examples can be found in that project)

YES! there is a very clever native method to do it.
let's say you are using Python ctypes. put something like this in your C code:
1) in your C code:
#define PYEXPORT extern "C" __declspec(dllexport)
now put PYEXPORT above the function you want to export:
PYEXPORT
int myfunc(params){
2) After compiling, go back into Python and open your .c file, and parse it similar to this:
source1_ = open(cfile_name + '.c')
source1 = source1_.read()
source1_.close()
fn = source1.split('PYEXPORT')[-1].split('(')[0].split(' ')[1]
shell input: fn
shell output: 'myfunc'
3) Now here's the clever part: define a new function in a string:
a1 = """
global get_c_fn
def get_c_fn(dll):
func = dll."""
a2 = """
return func"""
a3 = a1 + fn + a2
print(a3)
global get_c_fn
def get_c_fn(dll):
func = dll.myfunc
return func
NOW do exec(a3) and it will turn that string into a function that you can use.
4) do the usual:
mydll = ctypes.CDLL(cfile_name + '.dll')
c_fn = get_cuda_fn(mydll)
c_fn.argtypes = func_params (an array of C-converted inputs you need)
c_fn( *[params] )
and there you have a python wrapper for a C script without having to modify ten different things every time something changes.

Internally ctypes uses functions provided by dynamic link library (dlopen/dlsym on unix, LoadLibrary/GetProcAddress on windows) to load library and find address of function specified by function name; and then use cffi library to pass parameter dynamically.
Problem is that the dynamic link library that ctypes depends on doesn't include function to list symbol from the shared library, that's why you can't list symbol by ctypes.
To do that, you have to use specific tools to dump elf file (readelf on unix) and pe file for dll (dumpbin on windows).