I have a C++ function that accepts a callback, like this:
void func(std::function<void(A, B)> callback) { ... }
I want to call this function from Cython by giving it a closure, i.e. something I would have done with a lambda if I was calling it from C++. If this was a C function, it would have some extra void* arguments:
typedef void(*callback_t)(int, int, void*);
void func(callback_t callback, void *user_data) {
callback(1, 2, user_data);
}
and then I would just pass PyObject* as user_data (there is a more detailed example here).
Is there way to do this more in C++ way, without having to resort to explicit user_data?
What I believe you're aiming to do is pass a callable Python object to something accepting a std::function. You need to do create a bit of C++ code to make it happen, but it's reasonably straightforward.
Starting by defining "accepts_std_function.hpp" as simply as possible to provide an illustrative example:
#include <functional>
#include <string>
inline void call_some_std_func(std::function<void(int,const std::string&)> callback) {
callback(5,std::string("hello"));
}
The trick is then to create a wrapper class that holds a PyObject* and defines operator(). Defining operator() allows it to be converted to a std::function. Most of the class is just refcounting. "py_obj_wrapper.hpp":
#include <Python.h>
#include <string>
#include "call_obj.h" // cython helper file
class PyObjWrapper {
public:
// constructors and destructors mostly do reference counting
PyObjWrapper(PyObject* o): held(o) {
Py_XINCREF(o);
}
PyObjWrapper(const PyObjWrapper& rhs): PyObjWrapper(rhs.held) { // C++11 onwards only
}
PyObjWrapper(PyObjWrapper&& rhs): held(rhs.held) {
rhs.held = 0;
}
// need no-arg constructor to stack allocate in Cython
PyObjWrapper(): PyObjWrapper(nullptr) {
}
~PyObjWrapper() {
Py_XDECREF(held);
}
PyObjWrapper& operator=(const PyObjWrapper& rhs) {
PyObjWrapper tmp = rhs;
return (*this = std::move(tmp));
}
PyObjWrapper& operator=(PyObjWrapper&& rhs) {
held = rhs.held;
rhs.held = 0;
return *this;
}
void operator()(int a, const std::string& b) {
if (held) { // nullptr check
call_obj(held,a,b); // note, no way of checking for errors until you return to Python
}
}
private:
PyObject* held;
};
This file uses a very short Cython file to do the conversions from C++ types to Python types. "call_obj.pyx":
from libcpp.string cimport string
cdef public void call_obj(obj, int a, const string& b):
obj(a,b)
You then just need to create the Cython code wraps these types. Compile this module and call test_func to run this. ("simple_version.pyx":)
cdef extern from "py_obj_wrapper.hpp":
cdef cppclass PyObjWrapper:
PyObjWrapper()
PyObjWrapper(object) # define a constructor that takes a Python object
# note - doesn't match c++ signature - that's fine!
cdef extern from "accepts_std_func.hpp":
void call_some_std_func(PyObjWrapper) except +
# here I lie about the signature
# because C++ does an automatic conversion to function pointer
# for classes that define operator(), but Cython doesn't know that
def example(a,b):
print(a,b)
def test_call():
cdef PyObjWrapper f = PyObjWrapper(example)
call_some_std_func(f)
The above version works but is somewhat limited in that if you want to do this with a different std::function specialization you need to rewrite some of it (and the conversion from C++ to Python types doesn't naturally lend itself to a template implementation). One easy way round this is to use the Boost Python library object class, which has a templated operator(). This comes at the cost of introducing an extra library dependency.
First defining the header "boost_wrapper.hpp" to simplify the conversion from PyObject* to boost::python::object
#include <boost/python/object.hpp>
inline boost::python::object get_as_bpo(PyObject* o) {
return boost::python::object(boost::python::handle<>(boost::python::borrowed(o)));
}
You then just need to Cython code to wrap this class ("boost_version.pyx"). Again, call test_func
cdef extern from "boost_wrapper.hpp":
cdef cppclass bpo "boost::python::object":
# manually set name (it'll conflict with "object" otherwise
bpo()
bpo get_as_bpo(object)
cdef extern from "accepts_std_func.hpp":
void call_some_std_func(bpo) except + # again, lie about signature
def example(a,b):
print(a,b)
def test_call():
cdef bpo f = get_as_bpo(example)
call_some_std_func(f)
A "setup.py"
from distutils.core import setup, Extension
from Cython.Build import cythonize
extensions = [
Extension(
"simple_version", # the extension name
sources=["simple_version.pyx", "call_obj.pyx" ],
language="c++", # generate and compile C++ code
),
Extension(
"boost_version", # the extension name
sources=["boost_version.pyx"],
libraries=['boost_python'],
language="c++", # generate and compile C++ code
)
]
setup(ext_modules = cythonize(extensions))
(A final option is to use ctypes to generate a C function pointer from a Python callable. See Using function pointers to methods of classes without the gil (bottom half of answer) and http://osdir.com/ml/python-cython-devel/2009-10/msg00202.html. I'm not going to go into detail about this here.)
Related
I have include a minimal working example below - it can be compiled using the typical pybind11 instructions (I use cmake).
I have an abstract base class, Abstract, which is pure virtual. I can easily wrap this in pybind11 using a "trampoline class" (this is well documented by pybind11).
Further, I have a concrete implementation of Abstract, ToBeWrapped, that is also wrapped using pybind11.
My issue is that I have some client code which accepts an arbitrary PyObject* (or, in the case of this example, pybind11's wrapper py::object) and expects to cast this to Abstract*.
However, as illustrated in my example, I am unable to cast the py::object to Abstract*.
I have no problem casting to ToBeWrapped* and then storing that as an Abstract*', however this would require my client code to know ahead of time what kind ofAbstract*` the python interpreter is sending, which defeats the purpose of the abstract base class.
TL;DR
Is it possible to modify this code such that the client accessMethod is able to arbitrarily handle an Abstract* passed from the python interpreter?
#include <pybind11/pybind11.h>
#include <iostream>
namespace py = pybind11;
// abstract base class - cannot be instantiated on its own
class Abstract
{
public:
virtual ~Abstract() = 0;
virtual std::string print() const = 0;
};
Abstract::~Abstract(){}
// concrete implementation of Abstract
class ToBeWrapped : public Abstract
{
public:
ToBeWrapped(const std::string& msg = "heh?")
: myMessage(msg){};
std::string print() const override
{
return myMessage;
}
private:
const std::string myMessage;
};
// We need a trampoline class in order to wrap this with pybind11
class AbstractPy : public Abstract
{
public:
using Abstract::Abstract;
std::string print() const override
{
PYBIND11_OVERLOAD_PURE(
std::string, // return type
Abstract, // parent class
print, // name of the function
// arguments (if any)
);
}
};
// I have client code that accepts a raw PyObject* - this client code base implements its
// own python interpreter, and calls this "accessMethod" expecting to convert the python
// object to its c++ type.
//
// Rather than mocking up the raw PyObject* method (which would be trivial) I elected to
// keep this minimal example 100% pybind11
void accessMethod(py::object obj)
{
// runtime error: py::cast_error
//Abstract* casted = obj.cast<Abstract*>();
// this works
Abstract* casted = obj.cast<ToBeWrapped*>();
}
PYBIND11_MODULE(PyMod, m)
{
m.doc() = R"pbdoc(
This is a python module
)pbdoc";
py::class_<Abstract, AbstractPy>(m, "Abstract")
.def("print", &Abstract::print)
;
py::class_<ToBeWrapped>(m, "WrappedClass")
.def(py::init<const std::string&>())
;
m.def("access", &accessMethod, "This method will attempt to access the wrapped type");
}
You need to declare the hierarchy relationship, so this:
py::class_<ToBeWrapped>(m, "WrappedClass")
should be:
py::class_<ToBeWrapped, Abstract>(m, "WrappedClass")
In general what is the recommended way to pass a Python function through Boost-Python for use later in C++ code (i.e. as a callback in a C++ object)?
More specifically, I have a C++ class FooCPP that I've successfully exposed to Python via Boost-Python; the user interacts with the Python class Foo that runs the C++ counterpart under the hood. Contrived example:
# Foo.py
from foo_base import FooBase
class Foo(FooBase):
...
def callback(val=42.):
return val
foo = Foo()
foo.run(callback)
And the Boost Python bindings:
// foo_bindings.cpp
#include "foo.hpp"
#include <boost/python.hpp>
namespace bp = boost::python;
FooPython::Run(const bp::object& py_callback)
// TODO: Do something with the python callback to make it a C++ function!
std::function<double(double)> cpp_callback;
FooCPP::Run(cpp_callback);
)
BOOST_PYTHON_MODULE(foo_base){
bp::class_<FooPython>("FooBase")
.def("run", &FooPython::Run)
;
}
So how can I address the TODO comment in foo_bindings.cpp?
I've gone through a number of related SO questions -- e.g. pass python function to boost c and sending py function as boost function arg -- and I'm familiar with the Boost-Python docs, but have not found a good solution/explanation. Thanks in advance!
Notes: C++11, boost v1.58.0, ubuntu 16.04
Update
I may have just found a solution, where I can implement a functor in foo_bindings.cpp, e.g.,
struct PythonCallback {
public:
PythonCallback(bp::object cb_func) : cb_func_(cb_func) {}
double operator() (const double& val) {
// Call the callback function in python
return cb_func_(val);
}
private:
bp::object cb_func_;
};
But then what should the FooCPP::Run signature be? I.e. what type is defined for the cpp_callback passed in?
And does the BOOST_PYTHON_MODULE code need to change for this callback functor?
Implement a functor in foo_bindings.cpp, where the callback is invoked with call:
#include <boost/python.hpp>
#include <boost/python/call.hpp>
struct PythonCallback : {
public:
PythonCallback(PyObject* func) : cb_(func) {}
double operator() (const double& value) {
// Call the callback function in python
return boost::python::call<double>(cb_, value);
}
private:
PyObject* cb_;
};
I have a C++ function which returns a raw float pointer, and another C++ function which accepts a raw float pointer as an argument. Something like:
float* ptr = something;
float* get_ptr(void) { return ptr; }
void use_ptr(float* ptr) { do_work(ptr); }
I want to be able to pass around pointers using Python. Something like this:
import my_native_functions as native
ptr = native.get_ptr()
native.use_ptr(ptr)
I am using pybind11 to create my native python module but I don't know how to create the bindings for the get_ptr() function. If I just do the following:
PYBIND11_MODULE(my_native_functions, m)
{
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
the get_ptr() function returns a Python Float object. I guess this makes sense because there are no pointer types in python. However, because this is now a simple Float, when I call the use_ptr() function and iterate over the pointer in C/C++, only the first element of the array is correct. The rest are garbage. To fix this, in C++, I have to cast my pointer to/from std::size_t. By doing this, everything works just fine.
However, I would like to ask: Is there a "right way" of achieving the above without the casting to/from std::size_t with pybind11?
In case you are curious why I do that:
I do understand that what I am doing is not type-safe. Also, I never touch the pointer/integer on the Python side. I just retrieve it from one native module and pass it to another. Also, I cannot cast the pointer to some kind of numpy view because the pointer is not always on the CPU. Sometimes I want to pass around CUDA pointers. Creating a py::array_t from a CUDA pointer is not possible unless I copy the data (and I do not want to do that).
Thank you.
Wrap the raw pointer in a custom "smart" pointer class (only pretending to be smart really) as described here. You can add some additional information to this class while you're at it, such as the size of the array element and the number of elements. This would make it a generalised array descriptor on the C++ side (but not on the Python side because you are not exposing the raw pointer to Python).
For a simpler option, just wrap your pointer in any old class in order to hide it from Python. No need to expose it to Python as a custom smart pointer. Here's an example that does just that:
#include <pybind11/pybind11.h>
#include <memory>
#include <iostream>
namespace py = pybind11;
template <class T> class ptr_wrapper
{
public:
ptr_wrapper() : ptr(nullptr) {}
ptr_wrapper(T* ptr) : ptr(ptr) {}
ptr_wrapper(const ptr_wrapper& other) : ptr(other.ptr) {}
T& operator* () const { return *ptr; }
T* operator->() const { return ptr; }
T* get() const { return ptr; }
void destroy() { delete ptr; }
T& operator[](std::size_t idx) const { return ptr[idx]; }
private:
T* ptr;
};
float array[3] = { 3.14, 2.18, -1 };
ptr_wrapper<float> get_ptr(void) { return array; }
void use_ptr(ptr_wrapper<float> ptr) {
for (int i = 0; i < 3; ++i)
std::cout << ptr[i] << " ";
std::cout << "\n";
}
PYBIND11_MODULE(Ptr,m)
{
py::class_<ptr_wrapper<float>>(m,"pfloat");
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
I am trying to compile a C++ module to use in scipy.weave that is composed of several headers and source C++ files. These files contain classes and methods that extensively use the Numpy/C-API interface. But I am failing to figure out how to include import_array() successfully. I have been struggling on this for the past week and I am going nuts. I hope you could help me with it because the weave help is not very explanatory.
In practice I have first a module called pycapi_utils that contains some routines to interface C objects with Python objects. It consists of a header file pycapi_utils.h and a source file pycapi_utils.cpp such as:
//pycapi_utils.h
#if ! defined _PYCAPI_UTILS_H
#define _PYCAPI_UTILS_H 1
#include <stdlib.h>
#include <Python.h>
#include <numpy/arrayobject.h>
#include <tuple>
#include <list>
typedef std::tuple<const char*,PyObject*> pykeyval; //Tuple type (string,Pyobj*) as dictionary entry (key,val)
typedef std::list<pykeyval> kvlist;
//Declaration of methods
PyObject* array_double_to_pyobj(double* v_c, long int NUMEL); //Convert from array to Python list (double)
...
...
#endif
and
//pycapi_utils.cpp
#include "pycapi_utils.h"
PyObject* array_double_to_pyobj(double* v_c, long int NUMEL){
//Convert a double array to a Numpy array
PyObject* out_array = PyArray_SimpleNew(1, &NUMEL, NPY_DOUBLE);
double* v_b = (double*) ((PyArrayObject*) out_array)->data;
for (int i=0;i<NUMEL;i++) v_b[i] = v_c[i];
free(v_c);
return out_array;
}
Then I have a further module model that contains classes and routines dealing with some mathematical model. Again it consists of a header and source file like:
//model.h
#if ! defined _MODEL_H
#define _MODEL_H 1
//model class
class my_model{
int i,j;
public:
my_model();
~my_model();
double* update(double*);
}
//Simulator
PyObject* simulate(double* input);
#endif
and
//model.cpp
#include "pycapi_utils.h"
#include "model.h"
//Define class and methods
model::model{
...
...
}
...
...
double* model::update(double* input){
double* x = (double*)calloc(N,sizeof(double));
...
...
// Do something
...
...
return x;
}
PyObject* simulate(double* input){
//Initialize Python interface
Py_Initialize;
import_array();
model random_network;
double* output;
output = random_network.update(input);
return array_double_to_pyobj(output); // from pycapi_utils.h
}
The above code is included in a scipy.weave module in Python with
def model_py(input):
support_code="""
#include "model.h"
"""
code = """
return_val = simulate(input.data());
"""
libs=['gsl','gslcblas','m']
vars = ['input']
out = weave.inline(code,
vars,
support_code=support_code,
sources = source_files,
libraries=libs
type_converters=converters.blitz,
compiler='gcc',
extra_compile_args=['-std=c++11'],
force=1)
It fails to compile giving:
error: int _import_array() was not declared in this scope
Noteworthy is that if I lump into pycapi_utils.h also the source pycapi_utils.cpp, everything works fine. But I don't want to use this solution, as in practice my modules here need to be included in several other modules that also use PyObjects and need call import_array().
I was looking to this post on stack exchange, but I cannot figure out if and how to properly define the #define directives in my case. Also the example in that post is not exactly my case as there, import_array() is called within the global scope of main() whereas in my case import_array() is called within my simulate routine which is invoked by main() build by scipy.weave.
I had a similar problem, as the link you've posted points out, the root of all evil is that the PyArray_API is defined static, which means that each translation unit has it's own PyArray_API which is initialized with PyArray_API = NULL by default. Thus import_array() must be called once for every *.cpp file. In your case it should be sufficient to call it in pycapi_utils.cpp and also once in model.cpp. You can also test if array_import is necessary before actualy calling it with:
if(PyArray_API == NULL)
{
import_array();
}
So apparently if I include in the pycapi_utils module a simple initialization routine such as:
//pycapi_utils.h
...
...
void init_numpy();
//pycapi_utils.cpp
...
...
void init_numpy(){
Py_Initialize;
import_array();
}
and then I invoke this routine at the beginning of any function / method that uses Numpy objects in my C code, it works. That is, the above code is edited as:
//pycapi_utils.cpp
...
...
PyObject* array_double_to_pyobj(...){
init_numpy();
...
...
}
//model.cpp
...
...
PyObject* simulate(...){
init_numpy();
...
...
}
My only concern at this point is whether there is a way to minimize number of calls to init_numpy(), or regardless I have to call it from any function that I define within my CPP modules that uses Numpy objects...
I'm trying to wrap a C++ library in which the logic is implemented as templatized functors in .hpp files, and I'm struggling to find the right way to expose the C++ functors as Cython/Python functions. How are functors like the one below supposed to be wrapped in Cython?
I believe this should be possible, at least for template classes and functions, according to the Cython 0.20 docs.
Note: I think I've figured out how to wrap normal C++ functions—the problem occurs when I'm trying to wrap a templatized functor, i.e. a template struct that overloads the () operator (making it act like a function when a data type is fixed).
Disclaimer: I'm a total novice in C++ and very new to Cython so apologies if I'm making obvious mistakes here.
The functor I'm trying to wrap:
#include <vector>
#include "EMD_DEFS.hpp"
#include "flow_utils.hpp"
template<typename NUM_T, FLOW_TYPE_T FLOW_TYPE= NO_FLOW>
struct emd_hat_gd_metric {
NUM_T operator()(const std::vector<NUM_T>& P, const std::vector<NUM_T>& Q,
const std::vector< std::vector<NUM_T> >& C,
NUM_T extra_mass_penalty= -1,
std::vector< std::vector<NUM_T> >* F= NULL);
};
My wrapper.pyx file:
# distutils: language = c++
from libcpp.vector cimport vector
cdef extern from "lib/emd_hat.hpp":
# Apparently `cppclass` is necessary here even though
# `emd_hat_gd_metric` is not a class...?
cdef cppclass emd_hat_gd_metric[NUM_T]:
NUM_T operator()(vector[NUM_T]& P,
vector[NUM_T]& Q,
vector[vector[NUM_T]]& C) except +
cdef class EMD:
cdef emd_hat_gd_metric *__thisptr
def __cinit__(self):
self.__thisptr = new emd_hat_gd_metric()
def __dealloc__(self):
del self.__thisptr
def calculate(self, P, Q, C):
# What goes here? How do I call the functor as a function?
return self.__thisptr(P, Q, C)
The above just gives a Calling non-function type 'emd_hat_gd_metric[NUM_T]' error when I try to compile it with cython --cplus wrapper.pyx.
Here's the full library I'm trying to wrap.
End goal: to be able to call emd_hat_gd_metric as a Cython/Python function, with arguments being NumPy arrays.
I couldn't find a real solution, but here's a workaround (that requires modifying the C++ code): just instantiate the template function with the data type you need in the C++ header, then declare that function normally in your .pyx file.
It's a little unwieldy if you need many different data types, but I only needed double. It would also be nicer if it wasn't necessary to modify the external library… but it works.
In the C++ some_library.hpp file:
Instantiate the functor with the data type you need (say, double):
template<typename T>
struct some_template_functor {
T operator()(T x);
};
// Add this:
some_template_functor<double> some_template_functor_double;
In the Cython .pyx file:
Declare the function normally (no need for cppclass):
cdef extern from "path/to/some_library.hpp":
cdef double some_template_functor_double(double x)
Then you can call some_template_functor_double from within Cython.