Develop Reference

How to call function/classes in c++ .so files, generated by Bazel, in Python?

Let's say I have a simple class in hello.h
#ifndef LIB_HELLO_GREET_H_
#define LIB_HELLO_GREET_H_
class A{
public:
int a = 0;
int b = 0;
int add(){
return a+b;
}
};
#endif
with bazel build file:
load("#rules_cc//cc:defs.bzl", "cc_binary", "cc_library")
cc_library(
name = "hello",
hdrs = ["hello.h"],
)
cc_binary(
name = "hello.so",
deps = [
":hello",
],
linkshared=True,
linkstatic=False
)
After I run bazel build hello.so, there is a shared object file generated in bazel-bin/main and bazel-bin/main/hello.so.runfiles/__main__/main/hello.so. Using those files, I want call class A with a python script. Ideally, I'd want to use cppyy or something similar.
I've tried with simple python scripts
import cppyy
cppyy.load_reflection_info('hello')
print(dir(cppyy.gbl))
or
import cppyy
cppyy.load_library('hello')
print(dir(cppyy.gbl))
with both .so files, but cppyy can't seem to detect class A - it is never inside cppyy.gbl
I'm wondering the best way to solve this problem, any help appreciated!

With cppyy, you also need to give it the header file, that is:
cppyy.include("hello.h")
and optionally use cppyy.add_include_path() to add the directory where hello.h resides, so that it can be found.
An alternative is to use so-called "dictionaries." These package (customizable) directory locations, names of necessary headers files and (through linking) the shared library with the implementation into a single, new, shared library (named "dictionary" because of history). There can be a so-called "rootmap" file on the side that allows a class loader to find A automatically on first use. See the example in the documentation.

Call function from python in C++

I'm currently working on a project with a separate cli and gui. The gui part is written in C++ to keep the exectuable small. I compiled my python program with pyinstaller and need to call some functions from the C++ program.
Python part:
import configparser
def getconfig() -> list:
config = configparser.ConfigParser()
config.read("config.ini")
section1 = config["general"]
section2 = section["section2"]
return [section1, section2] #should be a list of dict?
Compiling it via pyinstaller --onefile --clean myprogram.py
What i would like to do in C++ is :
//please correct me if this is the wrong type,
//i know i probably need to do a bit of parsing between python and C++ types
std::vector<std::unordered_map<std::string, std::string>> > = myprogram.getconfig()
I just don't want to do the config parsing in C++ again, or would you recommend this as it's probably easier to call the compiled python binary?
This program just needs to run on linux, windows is not necessary.

If you are only planning to run on POSIX-compatible systems (ie. GNU/Linux), you could spawn a new process to run the Python script, e.g. using the C function exec. Example:
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
int main()
{
char* args[] = { "pythonscript.py", NULL };
int t = execv("pythonscript.py", args);
if(t < 0) printf("%s\n", strerror(errno));
return 0;
}
And then in pythonscript.py:
#!/usr/bin/python3
print("hello, world!")
This trick works with bash scripts as well.
The script must be executable though, so remember to run chmod +x pythonscript.py.
EDIT:
You will likely need to use pipes or other mechanisms for inter-process communication. (this is not my expertise!)

How can I save a python function to a static c++ container using pybind11?

Essentially, on the C++ side I have a container that holds a certain type of function. Now I would like to expose this container to python with the possibility for the users to provide their own python functions.
The simplest example would look like this:
#include "pybind/common/Common.h"
using CppFunc = std::function< int (int) >;
PYBIND11_MODULE( test, m )
{
m.def("addFunc", [](const pybind11::function& f){
static std::vector<CppFunc> vec{};
vec.push_back(f.cast<CppFunc>());
});
}
Then in python I would like to just do something like this.
import test
def myFunc(number):
return number+1
test.addFunc(myFunc)
Interestingly enough, this works fine. However, if I run the script with "python script.py" it runs through and then never terminates. In an interactive console, the same code works fine until you try to close the console: the process gets stuck.
How can I safely store this python function in the C++ container?

static std::vector<CppFunc> vec{} stores references to python objects (user functions) which never get released due to static storage, therefore interpreter cannot terminate.
To ensure interpreter termination you can call a cleanup function at module termination:
#include "pybind11/pybind11.h"
#include "pybind11/functional.h"
namespace py = pybind11;
using CppFunc = std::function< int (int) >;
PYBIND11_MODULE( test , m )
{
static std::vector<CppFunc> vec{};
m.def("addFunc", [](CppFunc f){
vec.push_back(std::move(f));
});
m.add_object("_cleanup", py::capsule([]{ vec.clear(); }));
}
See doc for more details: https://pybind11.readthedocs.io/en/stable/advanced/misc.html#module-destructors

Cpp compilation from python fail but not in the shell

I have a cpp file that compiles fine with g++ by using the shell:
extern "C"{
#include <quadmath.h>
}
inline char* print( const __float128& val)
{
char* buffer = new char[128];
quadmath_snprintf(buffer,128,"%+.34Qe", val);
return buffer;
}
int main(){
__float128 a = 1.0;
print(a);
return 0;
}
However, when I try to compile it via a python scrit, it fails with the following error:
"undefined reference to quadmath_snprintf"
Here the code of the python script:
import commands
import string
import os
(status, output) = commands.getstatusoutput("(g++ test/*.c -O3 -lquadmath -m64)")
Any idea how to solve this? Thanks.

When you open a shell a whole of stuff is silently initialized for you, and most important for your issue, environment variables are set. What you most likely miss is the definition of LIBRARY_PATH, which is the variable used by the linker to look for libraries matching the ones you instruct it to link using the -lNAME flags.
What the linker needs is a list of directories where it will search for files matching libNAME.{a,so}. You can also pass these directories directly using the -L flag, but in general, you should probably try to use a program like CMake, Make or any other build tool.
This will give you access to commands like find_package and target_link_libraries (CMake), to find, respectively add libraries to your build targets, instead of having to maintain your python to compile your stuff.

Call Python code from LLVM JIT

I write a language lexer/parser/compiler in python, that should run in the LLVM JIT-VM (using llvm-py) later. The first two steps are quite straightforward for now, but (even if I didn't start the compile-task yet) I see a problem, when my code wants to call Python-Code (in general), or interact with the Python lexer/parser/compiler (in special) respectively. My main concern is, that the code should be able to dynamically load additional code into the VM at runtime and thus it must trigger the whole lexer/parser/compiler-chain in Python from within the VM.
First of all: Is this even possible, or is the VM "unmutable" once it is started?
If it is I currently see 3 possible solutions (I am open for other suggestions)
"Break out" of the VM and make it possible to call Python functions of the main process directly (maybe by registering it as a LLVM-function, that redirects to the main process somehow). I didn't found anything about this and anyway I am not sure, if this is a good idea (security and such).
Compile the runtime (statically or dynamically at runtime) into LLVM-Assembly/-IR. This requires, that the IR-code is able to modify the VM it runs in
Compile the runtime (statically) into a library and load it directly into the VM. Again it must be able to add functions (etc) to the VM it runs in.

Like Eli said, there's not stopping you from calling out to the Python C-API. When you call an external function from inside of the LLVM JIT it effectively just uses dlopen() on the process space so if you're running from inside of llvmpy you already have all the Python interpreter symbols accessible, you can even interact with the active interpreter that invoked the ExecutionEngine or you can spin a new Python interpreter if needed.
To get you started, create a new C file with our evaluator.
#include <Python.h>
void python_eval(const char* s)
{
PyCodeObject* code = (PyCodeObject*) Py_CompileString(s, "example", Py_file_input);
PyObject* main_module = PyImport_AddModule("__main__");
PyObject* global_dict = PyModule_GetDict(main_module);
PyObject* local_dict = PyDict_New();
PyObject* obj = PyEval_EvalCode(code, global_dict, local_dict);
PyObject* result = PyObject_Str(obj);
// Print the result if you want.
// PyObject_Print(result, stdout, 0);
}
Here's a little Makefile to compile that:
CC = gcc
LPYTHON = $(shell python-config --includes)
CFLAGS = -shared -fPIC -lpthread $(LPYTHON)
.PHONY: all clean
all:
$(CC) $(CFLAGS) cbits.c -o cbits.so
clean:
-rm cbits.c
Then we start with the usual boilerplate for LLVM but use ctypes to load the shared object of our cbits.so shared library into the global process space so that we have the python_eval symbol. Then just create a simple LLVM module with a function, allocate a string with some Python source with ctypes and pass the pointer to the ExecutionEngine running the JIT'd function from our module, which in turns passes the Python source to the C-function which invokes the Python C-API and then yields back to the LLVM JIT.
import llvm.core as lc
import llvm.ee as le
import ctypes
import inspect
ctypes._dlopen('./cbits.so', ctypes.RTLD_GLOBAL)
pointer = lc.Type.pointer
i32 = lc.Type.int(32)
i64 = lc.Type.int(64)
char_type = lc.Type.int(8)
string_type = pointer(char_type)
zero = lc.Constant.int(i64, 0)
def build():
mod = lc.Module.new('call python')
evalfn = lc.Function.new(mod,
lc.Type.function(lc.Type.void(),
[string_type], False), "python_eval")
funty = lc.Type.function(lc.Type.void(), [string_type])
fn = lc.Function.new(mod, funty, "call")
fn_arg0 = fn.args[0]
fn_arg0.name = "input"
block = fn.append_basic_block("entry")
builder = lc.Builder.new(block)
builder.call(evalfn, [fn_arg0])
builder.ret_void()
return fn, mod
def run(fn, mod, buf):
tm = le.TargetMachine.new(features='', cm=le.CM_JITDEFAULT)
eb = le.EngineBuilder.new(mod)
engine = eb.create(tm)
ptr = ctypes.cast(buf, ctypes.c_voidp)
ax = le.GenericValue.pointer(ptr.value)
print 'IR'.center(80, '=')
print mod
mod.verify()
print 'Assembly'.center(80, '=')
print mod.to_native_assembly()
print 'Result'.center(80, '=')
engine.run_function(fn, [ax])
if __name__ == '__main__':
# If you want to evaluate the source of an existing function
# source_str = inspect.getsource(mypyfn)
# If you want to pass a source string
source_str = "print 'Hello from Python C-API inside of LLVM!'"
buf = ctypes.create_string_buffer(source_str)
fn, mod = build()
run(fn, mod, buf)
You should the following output:
=======================================IR=======================================
; ModuleID = 'call python'
declare void #python_eval(i8*)
define void #call(i8* %input) {
entry:
call void #python_eval(i8* %input)
ret void
}
=====================================Result=====================================
Hello from Python C-API inside of LLVM!

You can call external C functions from LLVM JIT-ed code. What else do you need?
These external functions will be found in the executing process, meaning that if you link Python into your VM you can call Python's C API functions.
The "VM" is probably less magic than you think it is :-) In the end, it's just machine code that gets emitted at runtime into a buffer and executed from there. To the extent that this code has access to other symbols in the process in which it's running, it can do everything any other code in that process can do.