I have two c++ libraries that expose python API but using two different frameworks(pybind11 and cython). I need to transfer an object between them (both ways) using python capsules. Since cython and pybind11 use the python capsules different ways, is it even possible to make it work?
I have library A that defines a class Foo and exposes it with pybind11 to python. Library B exposes its API using cython. LibB owns a shared_ptr<Foo> which is a member of one of LibB's classes, say - Bar.
Bar returns the shared_ptr<Foo> member as a PyCapsule which I capture in pybind11 of the Foo class. I'm unpacking the shared_ptr<Foo> from the capsule, returning it to python and the user can operate on this object in python using pybind11 bindings for Foo.
Then I need to put it back in a capsule in pybind11 and return back to Bar.
Bar's python API operates on PyObject and PyCapsule because cython allows that. pybind11 and thus Foo's API does not accept those types and I'm forced to use pybind11::object and pybind11::capsule.
Everything works fine until the moment when I'm trying to use the pybind11::capsule created in pybind11, inside a cython method of class Bar which expects a PyCapsule*.
The shared_ptr<Foo> inside the pybind11::capsule is corrupted and my app crashes.
Has anyone tried to make those 2 libs talk to each other?
libA -> class Foo
namespace foo{
class Foo {
public:
void foo() {...}
}
}
libB -> class Bar
namespace bar {
class Bar {
public:
PyObject* get_foo() {
const char * capsule_name = "foo_in_capsule";
return PyCapsule_New(&m_foo, capsule_name, nullptr);
}
static Bar fooToBar(PyObject * capsule) {
void * foo_ptr = PyCapsule_GetPointer(capsule, "foo_in_capsule");
auto foo = static_cast<std::shared_ptr<foo::Foo>*>(foo_ptr);
// here the shared_ptr is corrupted (garbage numbers returned for use_count() and get() )
std::cout << "checking the capsule: " << foo->use_count() << " " << foo->get() << std::endl
Bar b;
b.m_foo = *foo; //this is what I would like to get
return b;
}
std::shared_ptr<Foo> m_foo;
};
}
pybind11 for Foo
void regclass_foo_Foo(py::module m)
{
py::class_<foo::Foo, std::shared_ptr<foo::Foo>> foo(m, "Foo");
foo.def("foo", &foo::Foo::foo);
foo.def_static("from_capsule", [](py::object* capsule) {
auto* pycapsule_ptr = capsule->ptr();
auto* foo_ptr = reinterpret_cast<std::shared_ptr<foo::Foo>*>(PyCapsule_GetPointer(pycapsule_ptr, "foo_in_capsule"));
return *foo_ptr;
});
foo.def_static("to_capsule", [](std::shared_ptr<foo::Foo>& foo_from_python) {
auto pybind_capsule = py::capsule(&foo_from_python, "foo_in_capsule", nullptr);
return pybind_capsule;
});
}
cython for Bar
cdef extern from "bar.hpp" namespace "bar":
cdef cppclass Bar:
object get_foo() except +
def foo_to_bar(capsule):
b = C.fooToBar(capsule)
return b
putting it all together in python
from bar import Bar, foo_to_bar
from foo import Foo
bar = Bar(... some arguments ...)
capsule1 = bar.get_foo()
foo_from_capsule = Foo.from_capsule(capsule1)
// this is the important part - need to operate on foo using its python api
print("checking if foo works", foo_from_capsule.foo())
// and use it to create another bar object with a (possibly) modified foo object
capsule2 = Foo.to_capsule(foo_from_capsule)
bar2 = foo_to_bar(capsule2)
There's too many unfinished details in your code for me to even test your PyCapsule version. My view is that the issue is with the lifetime of the shared pointers - your capsule points to a shared pointer that's lifetime is tied to the Bar it's in. However, the capsule may outlive that. You should probably be creating a new shared_ptr<Foo>* (with new), pointing to that in your capsule, and defining a destructor (for the capsule) to delete it.
An outline of an alternative approach that I think should work better is as follows:
Write your classes purely in terms of C++ types, so get_foo and foo_to_bar just take/return shared_ptr<Foo>.
Define PyBar as a proper Cython class, rather than using capsules:
cdef public class PyBar [object PyBarStruct, type PyBarType]:
cdef shared_ptr[Bar] ptr
cdef public PyBar PyBar_from_shared_ptr(shared_ptr[Bar] b):
cdef PyBar x = PyBar()
x.ptr = b
return x
This generates a header file containing definitions of PyBarStruct and PyBarType (you probably don't need the latter). I also define a basic module-level function to create a PyBar from a shared pointer (and make that public as well, so it appears in the header too).
Then use PyBind11 to define a custom type-caster to/from shared_ptr<Bar>. load would be something like:
bool load(handle src, bool) {
auto bar_mod = py::import("bar");
auto bar_type = py::getattr(bar_mod,"Bar");
if (!py::isinstance(src,bar_type)) {
return false;
}
// now cast to my PyBarStruct
auto ptr = reinterpret_cast<PyBarStruct*>(src.ptr());
value = ptr->ptr; // access the shared_ptr of the struct
}
while the C++ to Python caster would be something like
static handle cast(std::shared_ptr<Bar> src, return_value_policy /* policy */, handle /* parent */) {
auto bar_mod = py::import("bar"); // See note...
return PyBar_from_shared_ptr(src);
}
I've ensured to include py::import("bar") in both functions because I don't think it's safe to use the Cython-defined functions until the module has been imported somewhere and importing it in the casters does ensure that.
This code is untested so almost certainly has errors, but should give a cleaner approach than PyCapsule.
Related
I have include a minimal working example below - it can be compiled using the typical pybind11 instructions (I use cmake).
I have an abstract base class, Abstract, which is pure virtual. I can easily wrap this in pybind11 using a "trampoline class" (this is well documented by pybind11).
Further, I have a concrete implementation of Abstract, ToBeWrapped, that is also wrapped using pybind11.
My issue is that I have some client code which accepts an arbitrary PyObject* (or, in the case of this example, pybind11's wrapper py::object) and expects to cast this to Abstract*.
However, as illustrated in my example, I am unable to cast the py::object to Abstract*.
I have no problem casting to ToBeWrapped* and then storing that as an Abstract*', however this would require my client code to know ahead of time what kind ofAbstract*` the python interpreter is sending, which defeats the purpose of the abstract base class.
TL;DR
Is it possible to modify this code such that the client accessMethod is able to arbitrarily handle an Abstract* passed from the python interpreter?
#include <pybind11/pybind11.h>
#include <iostream>
namespace py = pybind11;
// abstract base class - cannot be instantiated on its own
class Abstract
{
public:
virtual ~Abstract() = 0;
virtual std::string print() const = 0;
};
Abstract::~Abstract(){}
// concrete implementation of Abstract
class ToBeWrapped : public Abstract
{
public:
ToBeWrapped(const std::string& msg = "heh?")
: myMessage(msg){};
std::string print() const override
{
return myMessage;
}
private:
const std::string myMessage;
};
// We need a trampoline class in order to wrap this with pybind11
class AbstractPy : public Abstract
{
public:
using Abstract::Abstract;
std::string print() const override
{
PYBIND11_OVERLOAD_PURE(
std::string, // return type
Abstract, // parent class
print, // name of the function
// arguments (if any)
);
}
};
// I have client code that accepts a raw PyObject* - this client code base implements its
// own python interpreter, and calls this "accessMethod" expecting to convert the python
// object to its c++ type.
//
// Rather than mocking up the raw PyObject* method (which would be trivial) I elected to
// keep this minimal example 100% pybind11
void accessMethod(py::object obj)
{
// runtime error: py::cast_error
//Abstract* casted = obj.cast<Abstract*>();
// this works
Abstract* casted = obj.cast<ToBeWrapped*>();
}
PYBIND11_MODULE(PyMod, m)
{
m.doc() = R"pbdoc(
This is a python module
)pbdoc";
py::class_<Abstract, AbstractPy>(m, "Abstract")
.def("print", &Abstract::print)
;
py::class_<ToBeWrapped>(m, "WrappedClass")
.def(py::init<const std::string&>())
;
m.def("access", &accessMethod, "This method will attempt to access the wrapped type");
}
You need to declare the hierarchy relationship, so this:
py::class_<ToBeWrapped>(m, "WrappedClass")
should be:
py::class_<ToBeWrapped, Abstract>(m, "WrappedClass")
I am exposing classes from C++ to Python using Boost.Python, for example:
class_<Point>("Point").
def(init<double, double, double>()).
add_property("X", &Point::GetX, &Point::SetX).
add_property("Y", &Point::GetY, &Point::SetY).
add_property("Z", &Point::GetZ, &Point::SetZ).
def("SetXYZ", &Point::SetPnt);
I am also exposing some variables as attributes of my main module:
MainModule.attr("Window") = object(ptr(mainWindow));
Is it possible to list all exposed classes and/or all attributes of a module (in C++)?
I expect to get a list (vector<string>) of all exposed classes: in this case just "Point". The same for exposed variables, in this case just "Window".
Using Python reflection capabilities:
Executing some code from C++:
PyRun_SimpleString("import inspect");
PyRun_SimpleString("import MyModule");
string getClassesCode =
"def GetClasses():\n"
" for name, obj in inspect.getmembers(MyModule):\n"
" if inspect.isclass(obj):\n"
" yield obj.__name__\n"
"_classes = list(GetClasses())\n";
typedef boost::python::list pylist;
object main_namespace = MainModule.attr("__dict__");
try
{
exec(getClassesCode.c_str(), main_namespace);
}
catch (error_already_set const &)
{
PyErr_Print();
}
pylist _classes = extract<pylist>(main_namespace["_classes"]);
for (int i = 0; i < len(_classes); ++i)
{
string className = extract<string>(_classes[i]);
//do whatever you need with the class name
}
Is it possible to list all exposed classes and/or all attributes of a
module (in C++)? I expect to get a list (vector<string>) of all exposed classes ...
I'm not aware of such a feature of Boost.Python, but one alternative, carrying an overhead though, might be to expose a list (you can have it std::vector<string>, why not?) that you prepare "manually" on the C++ side and maintain it as you change your exposure composition. That way, you only have to know the name such list or lists and import them initially to get such a functionality.
I want to set a Python variable from C++ so that the C++ program can create an object Game* game = new Game(); in order for the Python code to be able to reference this instance (and call functions, etc). How can I achieve this?
I feel like I have some core misunderstanding of the way Python or Boost-Python works.
The line main_module.attr("game") = game is in a try catch statement, and the error (using PyErr_Fetch) is "No to_python (by-value) converter found for C++ type: class Game".
E.g.
class_<Game>("Game")
.def("add", &Game::add)
;
object main_module = import("__main__");
Game* game = new Game();
main_module.attr("game") = game; //This does not work
From Python:
import testmodule
testmodule.game.foo(7)
When dealing with language bindings, one often has to be pedantic in the details. By default, when a C++ object transgresses the language boundary, Boost.Python will create a copy, as this is the safest course of action to prevent dangling references. If a copy should not be made, then one needs to be explicit as to the ownership of the C++ object:
To pass a reference to a C++ object to Python while maintaining ownership in C++, use boost::python::ptr() or boost::ref(). The C++ code should guarantee that the C++ object's lifetime is at least as long as the Python object. When using ptr(), if the pointer is null, then the resulting Python object will be None.
To transfer ownership of a C++ object to Python, one can apply the manage_new_object ResultConverterGenerator, allowing ownership to be transferred to Python. C++ code should not attempt to access the pointer once the Python object's lifetime ends.
For shared ownership, one would need to expose the class with a HeldType of a smart pointer supporting shared semantics, such as boost::shared_ptr.
Once the Python object has been created, it would need to be inserted into a Python namespace to be generally accessible:
From within the module definition, use boost::python::scope to obtain a handle to the current scope. For example, the following would insert x into the example module:
BOOST_PYTHON_MODULE(example)
{
boost::python::scope().attr("x") = ...; // example.x
}
To insert into the __main__ module, one can import __main__. For example, the following would insert x into the __main__ module:
boost::python::import("__main__").attr("x") = ...;
Here is an example demonstrating how to directly construct the Python object from C++, transfer ownership of a C++ object to Python, and construct a Python object that references a C++ object:
#include <iostream>
#include <boost/python.hpp>
// Mockup model.
struct spam
{
spam(int id)
: id_(id)
{
std::cout << "spam(" << id_ << "): " << this << std::endl;
}
~spam()
{
std::cout << "~spam(" << id_ << "): " << this << std::endl;
}
// Explicitly disable copying.
spam(const spam&) = delete;
spam& operator=(const spam&) = delete;
int id_;
};
/// #brief Transfer ownership to a Python object. If the transfer fails,
/// then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
// Transfer ownership to a smart pointer, allowing for proper cleanup
// incase Boost.Python throws.
std::unique_ptr<T> ptr(t);
// Use the manage_new_object generator to transfer ownership to Python.
namespace python = boost::python;
typename python::manage_new_object::apply<T*>::type converter;
// Transfer ownership to the Python handler and release ownership
// from C++.
python::handle<> handle(converter(*ptr));
ptr.release();
return python::object(handle);
}
namespace {
spam* global_spam;
} // namespace
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose spam.
auto py_spam_type = python::class_<spam, boost::noncopyable>(
"Spam", python::init<int>())
.def_readonly("id", &spam::id_)
;
// Directly create an instance of Python Spam and insert it into this
// module's namespace.
python::scope().attr("spam1") = py_spam_type(1);
// Construct of an instance of Python Spam from C++ spam, transfering
// ownership to Python. The Python Spam instance will be inserted into
// this module's namespace.
python::scope().attr("spam2") = transfer_to_python(new spam(2));
// Construct an instance of Python Spam from C++, but retain ownership of
// spam in C++. The Python Spam instance will be inserted into the
// __main__ scope.
global_spam = new spam(3);
python::import("__main__").attr("spam3") = python::ptr(global_spam);
}
Interactive usage:
>>> import example
spam(1): 0x1884d40
spam(2): 0x1831750
spam(3): 0x183bd00
>>> assert(1 == example.spam1.id)
>>> assert(2 == example.spam2.id)
>>> assert(3 == spam3.id)
~spam(1): 0x1884d40
~spam(2): 0x1831750
In the example usage, note how Python did not destroy spam(3) upon exit, as it was not granted ownership of the underlying object.
I am creating Python interfaces to some C++ code that I cannot change, using SWIG. One of the C++ classes has a constructor that creates a partially initialized object that cannot be used as yet, an initialization function has to be called on it first. I want to remedy this in Python by providing an alternative constructor that takes care of doing both things (acquisition and initialization) at the same time. Let's say in C++ I have
class X {
public:
X() {...}
void init(T a) {...}
...
};
In C++ I have to instantiate X as
X x;
x.init(a);
In Python I would like to do
x = X(a)
My solution is a hack that depends on the target language and the specific way SWIG generates the wrapper code: in my .i file I have
%inline %{
X* new_X(T a) {
X* ret = new X();
ret->init(a);
return ret;
}
%nodefaultctor X;
class X {
public:
...
%extend {
%pythoncode {
def __init__(self, *args):
this = _modulename.new_X(*args)
try:
self.this.append(this)
except:
self.this = this
}
}
};
This works fine, but it is not very satisfactory:
it depends on how SWIG wraps constructors internally
it is totally target language dependent
This seems to be a relatively common use case, so does anyone know if there is a standard way?
The current answer by V-master does not work as is. But it can be made to work:
%ignore X::X();
// declaration of class X, e.g. %include X.h
%extend X {
X(T a) {
X* newX = new X();
newX->init(a);
return newX;
}
};
Admittedly, this looks a little dubious, but it works, and is essentially an example from the SWIG documentation here.
It is important to note that:
%extend works with both C and C++ code. It does not modify the underlying object in any way---the extensions only show up in the Python interface.
So what this really does is to create a method (not even a class method, actually) that creates a new instance of X, calls init(a) on it and returns it. Because the syntax somewhat resembles a constructor, SWIG will wrap it as such.
you can try something like this:
%ignore X::X();
%extend X {
X(T a) {
init(a);
}
};
this will hide default no-parameter constructor and add new that takes T
Downside is that if ignored one were doing something, then you would need to copy that into this new one as you cannot call other constructor from within same class constructor (unless you are using C++11)
I try to expose two different classes to python, but I don't get it to compile. I tried to follow the boost::python example, which works quite well. But if I try to write the wrapper classes for my classes it doesn't work. I have provided two minimal examples below:
struct Base
{
virtual ~Base() {}
virtual std::unique_ptr<std::string> f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
std::unique_ptr<std::string> f()
{
return this->get_override("f")();
}
};
and
struct Base
{
virtual ~Base() {}
virtual void f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
void f()
{
return this->get_override("f")();
}
};
The first one does not compile because of the unique pointer(I think boost::python does not use unique pointers?) and the second example complains about the return statement inside the void function. Can someone help me how to solve this problems?
The examples are failing to compile because:
The first example attempts to convert an unspecified type (the return type of override::operator()) to an incompatible type. In particular, Boost.Python does not currently support std::unique_ptr, and hence will not convert to it.
The second example attempts to return the unspecified type mentioned above when the calling function declares that it returns void.
From a Python perspective, strings are immutable, and attempting to transferring ownership of a string from Python to C++ violates semantics. However, one could create a copy of a string within C++, and pass ownership of the copied string to C++. For example:
std::unique_ptr<std::string> BaseWrap::f()
{
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("f")();
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
The object returned from this->get_override("f")() has an unspecified type, but can be used to convert to C++ types. The invocation of the override will throw if Python throws, and the conversion to the C++ type will throw if the object returned from Python is not convertible to the C++ type.
Here is a complete example demonstrating two ways to adapt the returned Python object to a C++ object. As mentioned above, the override conversion can be used. Alternatively, one can use boost::python::extract<>, allowing one to check if the conversion will fail before performing the conversion:
#include <memory> // std::unique_ptr
#include <boost/algorithm/string.hpp> // boost::to_upper_copy
#include <boost/python.hpp>
struct base
{
virtual ~base() {}
virtual std::unique_ptr<std::string> perform() = 0;
};
struct base_wrap : base, boost::python::wrapper<base>
{
std::unique_ptr<std::string> perform()
{
namespace python = boost::python;
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("perform")();
// Alternatively, an extract could be used to defer extracting the
// result.
python::object method(this->get_override("perform"));
python::extract<std::string> extractor(method());
// Check that extractor contains a std::string without throwing.
assert(extractor.check());
// extractor() would throw if it did not contain a std::string.
assert(result == extractor());
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<base_wrap, boost::noncopyable>("Base", python::init<>())
.def("perform", python::pure_virtual(&base::perform))
;
python::def("make_upper", +[](base* object) {
auto result = object->perform(); // Force dispatch through base_wrap.
assert(result);
return boost::to_upper_copy(*result);
});
}
Interactive usage:
>>> import example
>>> class Derived(example.Base):
... def perform(self):
... return "abc"
...
>>> derived = Derived()
>>> assert("ABC" == example.make_upper(derived))