Develop Reference

pybind11 buffer protocol with class members

I'm trying to use pybind11 to bind a struct that looks like this
struct myStruct {
int na;
int nb;
double* a;
double* b;
}
I'm not sure the right way to go about it. The examples in the pybind11 documentation show how to attach buffer protocol semantics to an object, but not to a class member.
I don't have the luxury of changing the interface of myStruct to contain std::vectors either which would allow me to use the usual .def_readwrite().
I've tried doing something like this
py::class_<myStruct>(m, "myStruct")
.def_property("a",
[](myStruct &s) {return py::array<double>({s.na}, {sizeof(double), s.a};)},
[](myStruct &s, py::array_t<double> val) {std::copy((double*) val.request().ptr, (double*) val.request().ptr + s.na, s.a);)}
)
Which compiles, but in python I don't see changes persist in the underlying data
print(my_struct.a[0]) # prints 0.0
my_struct.a[0] = 123.0
print(my_struct.a[0]) # still prints 0.0

Hey most likely not the most elegant answer, but maybe it gives you a starting point and temporary solution. I think what you need to do is use shared pointers.
Under https://github.com/pybind/pybind11/issues/1150 someone asked something similar but I was not able to adapt it to your example and only got the same result to yours with no changes to the data.
What worked for me in your specific example was using the shared_ptr and defining setter and getter functions for the pointers with a simple def_property for the pybin11 class.
class class_DATA{
public:
int na;
std::shared_ptr<double> a;
void set_a(double a){*class_DATA::a = a; };
double get_a(void){return *class_DATA::a; };
};
PYBIND11_MODULE(TEST,m){
m.doc() = "pybind11 example plugin";
//the costum class
py::class_<class_DATA>(m, "class_DATA", py::dynamic_attr())
.def(py::init<>()) //needed to define constructor
.def_readwrite("na", &class_DATA::na)
.def_property("a", &class_DATA::get_a, &class_DATA::set_a, py::return_value_policy::copy);
}

Can you create a python class via pybind11?

Currently when working with python + pybind11 I find it frustrating to work with the typed c++ classes/structs.
I would like to change my bindings so that they generate a simple python class, with an __init__ and a simple function like shown below. Is something like the feasible?
Reasoning:
I currently have a struct that I generate via c++, but it has a lot of heavy std::vector<float>s that I would like to pass to python, and keep as numpy arrays inside a similar interfacing python class. (bonus points if you can tell me how to move vectors to be numpy arrays quickly!)
I have already completely bound my c++ struct with pybind11, so I feel like I know what I'm doing... however I can't seem to figure out if this is possible!
So, as a learning exercise, can I make the following python class via pybind11?
>>> python
class MyStruct:
def __init__(self, A_in, descriptor_in):
self.A = A_in
self.descriptor = descriptor_in
def add_to_vec(f_in):
self.A.append(f_in)
<<< python
Edit:
I want to say I 'think' that this is doable with the python C api, but I'd like to avoid using that directly if I can. (but if you think that's the only way, please let me know :) )
Edit2: (response to #Erwan)
The only way I'm aware of to get class variables individually is this (shown below). You cannot use the pybind advertised buffer_protocol interface if you have more than one numpy array in the struct you would like to get. However this requires creating a python-interface only function .def (not ideal) that points to (what I think is a copy) of the original data (so it's probably slow, i haven't benchmarked it, but I'm not sure if this was is a hack or the correct way to get vectors into numpy arrays).
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <vector>
#include <string>
struct Pet {
Pet(const std::string &name) : name(name) {
bdata.push_back(22.);
bdata.push_back(23.1);
bdata.push_back(24.);
bdata.push_back(2222.);
}
void setName(const std::string &name_) { name = name_; }
const std::string &getName() const { return name; }
std::string name;
std::vector<float> bdata;
};
namespace py = pybind11;
PYBIND11_MODULE(example, m) {
py::class_<Pet>(m, "Pet")
.def(py::init<const std::string &>())
.def("setName", &Pet::setName)
.def("getName", &Pet::getName)
.def("bdata", [](Pet &m) -> py::array {
py::buffer_info buff_info(py::buffer_info(
m.bdata.data(), /* Pointer to buffer */
sizeof(float), /* Size of one scalar */
py::format_descriptor<float>::format(), /* Python struct-style format descriptor */
m.bdata.size() /* Number of dimensions */
));
return py::array(buff_info);
});
}

I don't understand your question in whole, but I'll take this part:
bonus points if you can tell me how to move vectors to be numpy arrays quickly!
If you use the return result of bdata.data() combined with numpy.frombuffer() and bdata.size() if need be, you can get a view on the vector data, which is guaranteed to be contiguous as of C++11. (The normal numpy.array() call will not honor copy=False in this case, but frombuffer acts like a cast.) Since there is no copy, that's probably as quick as it gets.
Below is an example in cppyy (which allows for easy testing, but the use of which is otherwise immaterial to the answer of how to mix std::vector and numpy.array per se). The gravy is in the last few lines: the update to 'arr' will show up in the original vector (and v.v.) b/c frombuffer is a view, not a copy:
import cppyy
import numpy as np
# load struct definition
cppyy.cppdef("""
struct Pet {
Pet(const std::string &name) : name(name) {
bdata.push_back(22.);
bdata.push_back(23.1);
bdata.push_back(24.);
bdata.push_back(2222.);
}
void setName(const std::string &name_) { name = name_; }
const std::string &getName() const { return name; }
std::string name;
std::vector<float> bdata;
};""")
# create a pet object
p = cppyy.gbl.Pet('fido')
print(p.bdata[0]) # for reference (prints 22, per above)
# create a numpy view on the std::vector's data
# add count=p.bdata.size() if need be
arr = np.frombuffer(p.bdata.data(), dtype=np.float32)
# prove that it worked as intended
arr[0] = 14
print(p.bdata[0]) # shows update to 14
p.bdata[2] = 17.5
print(arr[2]) # shows update to 17.5
which will print:
22.0
14.0
17.5
'arr' may become invalid if the std::vector resizes. If you know the maximum size, however, and it is not too large or will be fully used for sure, you can reserve that, so the vector's internal data will not be reallocated.
Depending on how/where you store the numpy array, I also recommend tying the life time of 'p' (and hence 'p.bdata') to 'arr' eg. by keeping them both as data members in an instance of the wrapper class you're after.
If you want to do the conversion in C++ instead, use PyArray_FromBuffer from NumPy's array API.

Exposing virtual member functions from C++ to Python using boost::python

I try to expose two different classes to python, but I don't get it to compile. I tried to follow the boost::python example, which works quite well. But if I try to write the wrapper classes for my classes it doesn't work. I have provided two minimal examples below:
struct Base
{
virtual ~Base() {}
virtual std::unique_ptr<std::string> f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
std::unique_ptr<std::string> f()
{
return this->get_override("f")();
}
};
and
struct Base
{
virtual ~Base() {}
virtual void f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
void f()
{
return this->get_override("f")();
}
};
The first one does not compile because of the unique pointer(I think boost::python does not use unique pointers?) and the second example complains about the return statement inside the void function. Can someone help me how to solve this problems?

The examples are failing to compile because:
The first example attempts to convert an unspecified type (the return type of override::operator()) to an incompatible type. In particular, Boost.Python does not currently support std::unique_ptr, and hence will not convert to it.
The second example attempts to return the unspecified type mentioned above when the calling function declares that it returns void.
From a Python perspective, strings are immutable, and attempting to transferring ownership of a string from Python to C++ violates semantics. However, one could create a copy of a string within C++, and pass ownership of the copied string to C++. For example:
std::unique_ptr<std::string> BaseWrap::f()
{
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("f")();
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
The object returned from this->get_override("f")() has an unspecified type, but can be used to convert to C++ types. The invocation of the override will throw if Python throws, and the conversion to the C++ type will throw if the object returned from Python is not convertible to the C++ type.
Here is a complete example demonstrating two ways to adapt the returned Python object to a C++ object. As mentioned above, the override conversion can be used. Alternatively, one can use boost::python::extract<>, allowing one to check if the conversion will fail before performing the conversion:
#include <memory> // std::unique_ptr
#include <boost/algorithm/string.hpp> // boost::to_upper_copy
#include <boost/python.hpp>
struct base
{
virtual ~base() {}
virtual std::unique_ptr<std::string> perform() = 0;
};
struct base_wrap : base, boost::python::wrapper<base>
{
std::unique_ptr<std::string> perform()
{
namespace python = boost::python;
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("perform")();
// Alternatively, an extract could be used to defer extracting the
// result.
python::object method(this->get_override("perform"));
python::extract<std::string> extractor(method());
// Check that extractor contains a std::string without throwing.
assert(extractor.check());
// extractor() would throw if it did not contain a std::string.
assert(result == extractor());
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<base_wrap, boost::noncopyable>("Base", python::init<>())
.def("perform", python::pure_virtual(&base::perform))
;
python::def("make_upper", +[](base* object) {
auto result = object->perform(); // Force dispatch through base_wrap.
assert(result);
return boost::to_upper_copy(*result);
});
}
Interactive usage:
>>> import example
>>> class Derived(example.Base):
... def perform(self):
... return "abc"
...
>>> derived = Derived()
>>> assert("ABC" == example.make_upper(derived))

Boost Python - Cast to incomplete class

Is there a way to cast between classes where only the interface is known in boost python?
Consider the following example:
// ---------- Module A ----------
class A_ifc
{
public:
virtual int foo() = 0;
};
class A_impl : public A_ifc
{
public:
int foo() {...}
};
BOOST_PYTHON_MODULE(Module_A)
{
class_<A_impl, boost::noncopyable>("A", no_init)
.def("foo", &A_impl::foo)
;
}
// ---------- Module B ----------
class B_ifc
{
public:
virtual int cross_module_operation(A_ifc* a) = 0;
};
class B_impl : public B_ifc
{
public:
int cross_module_operation(A_ifc* a) {...}
};
BOOST_PYTHON_MODULE(Module_B)
{
class_<B_impl, boost::noncopyable>("B", no_init)
.def("cross_module_operation", &B_impl::cross_module_operation)
;
}
In my C code none of the classes know anything of each other except for their respective interfaces (here 'A_ifc' and 'B_ifc'). All cross-module operations pass only interface pointers.
When I want to achieve the same behavior in Python however this wouldn't work, because for Boost Python the whole class definition must be known for every referenced class.
I have already tried opaque pointers (see http://www.boost.org/doc/libs/1_55_0/libs/python/doc/v2/return_opaque_pointer.html), but whenever I compile any Boost code that contains "boost::python::return_value_policy", the application crashes with an access violation on startup.
I also looked into custom converters (see Boost.Python custom converter), but I have no idea how to apply this concept to my problem.
Any help is greatly appreciated

As it turns out, casting to a an abstract base class is indeed possible as long as as run-time type information is enabled.
In other words this will work: objects::register_conversion<A_impl, A_ifc>();
I feel a bit silly for not having tried such an obvious solution earlier.

Can I export c++ template class to C and therefore to python with ctypes?

For a non template class I would write something like that
But I don't know what should I do if my class is a template class.
I've tried something like that and it's not working.
extern "C" {
Demodulator<double>* Foo_new_double(){ return new Demodulator<double>(); }
Demodulator<float>* Foo_new_float(){ return new Demodulator<float>(); }
void demodulateDoubleMatrix(Demodulator<double>* demodulator, double * input, int rows, int columns){ demodulator->demodulateMatrixPy(input, rows, columns) }
}

Note: Your question contradicts the code partially, so I'm ignoring the code for now.
C++ templates are an elaborated macro mechanism that gets resolved at compile time. In other words, the binary only contains the code from template instantiations (which is what you get when you apply parameters, typically types, to the the template), and those are all that you can export from a binary to other languages. Exporting them is like exporting any regular type, see for example std::string.
Since the templates themselves don't survive compilation, there is no way that you can export them from a binary, not to C, not to Python, not even to C++! For the latter, you can provide the templates themselves though, but that doesn't include them in a binary.
Two assumptions:
Exporting/importing works via binaries. Of course, you could write an import that parses C++.
C++ specifies (or specified?) export templates, but as far as I know, this isn't really implemented in the wild, so I left that option out.

The C++ language started as a superset of C: That is, it contains new keywords, syntax and capabilities that C does not provide. C does not have the concept of a class, has no concept of a member function and does not support the concept of access restrictions. C also does not support inheritance. The really big difference, however, is templates. C has macros, and that's it.
Therefore no, you can't directly expose C++ code to C in any fashion, you will have to use C-style code in your C++ to expose the C++ layer.
template<T> T foo(T i) { /* ... */ }
extern "C" int fooInt(int i) { return foo(i); }
However C++ was originally basically a C code generator, and C++ can still interop (one way) with the C ABI: member functions are actually implemented by turning this->function(int arg); into ThisClass0int1(this, arg); or something like that. In theory, you could write something to do this to your code, perhaps leveraging clang.
But that's a non-trivial task, something that's already well-tackled by SWIG, Boost::Python and Cython.
The problem with templates, however, is that the compiler ignores them until you "instantiate" (use) them. std::vector<> is not a concrete thing until you specify std::vector<int> or something. And now the only concrete implementation of that is std::vector<int>. Until you've specified it somewhere, std::vector<string> doesn't exist in your binary.
You probably want to start by looking at something like this http://kos.gd/2013/01/5-ways-to-use-python-with-native-code/, select a tool, e.g. SWIG, and then start building an interface to expose what you want/need to C. This is a lot less work than building the wrappers yourself. Depending which tool you use, it may be as simple as writing a line saying using std::vector<int> or typedef std::vector<int> IntVector or something.
---- EDIT ----
The problem with a template class is that you are creating an entire type that C can't understand, consider:
template<typename T>
class Foo {
T a;
int b;
T c;
public:
Foo(T a_) : a(a_) {}
void DoThing();
T GetA() { return a; }
int GetB() { return b; }
T GetC() { return c; }
};
The C language doesn't support the class keyword, never mind understand that members a, b and c are private, or what a constructor is, and C doesn't understand member functions.
Again it doesn't understand templates so you'll need to do what C++ does automatically and generate an instantiation, by hand:
struct FooDouble {
double a;
int b;
double c;
};
Except, all of those variables are private. So do you really want to be exposing them? If not, you probably just need to typedef "FooDouble" to something the same size as Foo and make a macro to do that.
Then you need to write replacements for the member functions. C doesn't understand constructors, so you will need to write a
extern "C" FooDouble* FooDouble_construct(double a);
FooDouble* FooDouble_construct(double a) {
Foo* foo = new Foo(a);
return reinterept_cast<FooDouble*>(foo);
}
and a destructor
extern "C" void FooDouble_destruct(FooDouble* foo);
void FooDouble_destruct(FooDouble* foo) {
delete reinterpret_cast<Foo*>(foo);
}
and a similar pass-thru for the accessors.