Develop Reference

Transfer a c++ object between cython and pybind11 using python capsules

I have two c++ libraries that expose python API but using two different frameworks(pybind11 and cython). I need to transfer an object between them (both ways) using python capsules. Since cython and pybind11 use the python capsules different ways, is it even possible to make it work?
I have library A that defines a class Foo and exposes it with pybind11 to python. Library B exposes its API using cython. LibB owns a shared_ptr<Foo> which is a member of one of LibB's classes, say - Bar.
Bar returns the shared_ptr<Foo> member as a PyCapsule which I capture in pybind11 of the Foo class. I'm unpacking the shared_ptr<Foo> from the capsule, returning it to python and the user can operate on this object in python using pybind11 bindings for Foo.
Then I need to put it back in a capsule in pybind11 and return back to Bar.
Bar's python API operates on PyObject and PyCapsule because cython allows that. pybind11 and thus Foo's API does not accept those types and I'm forced to use pybind11::object and pybind11::capsule.
Everything works fine until the moment when I'm trying to use the pybind11::capsule created in pybind11, inside a cython method of class Bar which expects a PyCapsule*.
The shared_ptr<Foo> inside the pybind11::capsule is corrupted and my app crashes.
Has anyone tried to make those 2 libs talk to each other?
libA -> class Foo
namespace foo{
class Foo {
public:
void foo() {...}
}
}
libB -> class Bar
namespace bar {
class Bar {
public:
PyObject* get_foo() {
const char * capsule_name = "foo_in_capsule";
return PyCapsule_New(&m_foo, capsule_name, nullptr);
}
static Bar fooToBar(PyObject * capsule) {
void * foo_ptr = PyCapsule_GetPointer(capsule, "foo_in_capsule");
auto foo = static_cast<std::shared_ptr<foo::Foo>*>(foo_ptr);
// here the shared_ptr is corrupted (garbage numbers returned for use_count() and get() )
std::cout << "checking the capsule: " << foo->use_count() << " " << foo->get() << std::endl
Bar b;
b.m_foo = *foo; //this is what I would like to get
return b;
}
std::shared_ptr<Foo> m_foo;
};
}
pybind11 for Foo
void regclass_foo_Foo(py::module m)
{
py::class_<foo::Foo, std::shared_ptr<foo::Foo>> foo(m, "Foo");
foo.def("foo", &foo::Foo::foo);
foo.def_static("from_capsule", [](py::object* capsule) {
auto* pycapsule_ptr = capsule->ptr();
auto* foo_ptr = reinterpret_cast<std::shared_ptr<foo::Foo>*>(PyCapsule_GetPointer(pycapsule_ptr, "foo_in_capsule"));
return *foo_ptr;
});
foo.def_static("to_capsule", [](std::shared_ptr<foo::Foo>& foo_from_python) {
auto pybind_capsule = py::capsule(&foo_from_python, "foo_in_capsule", nullptr);
return pybind_capsule;
});
}
cython for Bar
cdef extern from "bar.hpp" namespace "bar":
cdef cppclass Bar:
object get_foo() except +
def foo_to_bar(capsule):
b = C.fooToBar(capsule)
return b
putting it all together in python
from bar import Bar, foo_to_bar
from foo import Foo
bar = Bar(... some arguments ...)
capsule1 = bar.get_foo()
foo_from_capsule = Foo.from_capsule(capsule1)
// this is the important part - need to operate on foo using its python api
print("checking if foo works", foo_from_capsule.foo())
// and use it to create another bar object with a (possibly) modified foo object
capsule2 = Foo.to_capsule(foo_from_capsule)
bar2 = foo_to_bar(capsule2)

There's too many unfinished details in your code for me to even test your PyCapsule version. My view is that the issue is with the lifetime of the shared pointers - your capsule points to a shared pointer that's lifetime is tied to the Bar it's in. However, the capsule may outlive that. You should probably be creating a new shared_ptr<Foo>* (with new), pointing to that in your capsule, and defining a destructor (for the capsule) to delete it.
An outline of an alternative approach that I think should work better is as follows:
Write your classes purely in terms of C++ types, so get_foo and foo_to_bar just take/return shared_ptr<Foo>.
Define PyBar as a proper Cython class, rather than using capsules:
cdef public class PyBar [object PyBarStruct, type PyBarType]:
cdef shared_ptr[Bar] ptr
cdef public PyBar PyBar_from_shared_ptr(shared_ptr[Bar] b):
cdef PyBar x = PyBar()
x.ptr = b
return x
This generates a header file containing definitions of PyBarStruct and PyBarType (you probably don't need the latter). I also define a basic module-level function to create a PyBar from a shared pointer (and make that public as well, so it appears in the header too).
Then use PyBind11 to define a custom type-caster to/from shared_ptr<Bar>. load would be something like:
bool load(handle src, bool) {
auto bar_mod = py::import("bar");
auto bar_type = py::getattr(bar_mod,"Bar");
if (!py::isinstance(src,bar_type)) {
return false;
}
// now cast to my PyBarStruct
auto ptr = reinterpret_cast<PyBarStruct*>(src.ptr());
value = ptr->ptr; // access the shared_ptr of the struct
}
while the C++ to Python caster would be something like
static handle cast(std::shared_ptr<Bar> src, return_value_policy /* policy */, handle /* parent */) {
auto bar_mod = py::import("bar"); // See note...
return PyBar_from_shared_ptr(src);
}
I've ensured to include py::import("bar") in both functions because I don't think it's safe to use the Cython-defined functions until the module has been imported somewhere and importing it in the casters does ensure that.
This code is untested so almost certainly has errors, but should give a cleaner approach than PyCapsule.

Reusing SWIG mappings in custom typemap

I'm currently working on a Python wrapper for a C++ library for which I want to use SWIG. In my C++ library I have a method with the following signature:
std::vector<SomeClass> getMembers();
Now I know that SWIG has built-in std::vector support but I want to explicitly convert std::vectors to Python Lists (I just think it is cleaner). For that I have the following typemap:
template<typename T>
PyObject* toList(vector<T> vec){
size_t size = vec.size();
PyObject *o = PyList_New(size);
for(size_t i = 0; i < size; i++){
PyList_SetItem(o, i, toPythonInstance<T>(vec[i]));
}
return o;
}
%define OUTPUT_VEC_TO_LIST(type)
%typemap (out) std::vector<type> {
$result = toList<type>($1);
}
%enddef
Now the template method:
template<T>
PyObject* toPythonInstance(T& val){}
Can be specialized to add support for the necessary datatypes. The problem I'm facing now is the following:
SomeClass is wrapped automatically by SWIG. So what I like to do is to reuse this wrapper in my vector type map, i.e. have the following:
template<>
PyObject* toPythonInstance<SomeClass>(SomeClass& val){
//call some SWIG macro to automatically wrap the given instance to
//a Python object
}
Inspecting the code generated by SWIG, I already found the following functions
SWIG_NewPointerObj(...);
SWIG_ConvertPtr(...);
which seem to be responsible for doing exactly what I want. However, I do not want to interfere with any internals of SWIG. So if somebody knows how to achieve what I want with the "public" SWIG interface, I would be very glad!

SWIG actually makes a whole bunch of runtime information part of an external interface, see http://www.swig.org/Doc3.0/Modules.html#Modules_external_run_time for details. This includes the functions you're likely to want for your efforts.
I disagree with your assessment that mapping std::vector to list is cleaner in Python - you always end up copying and visiting every member of that vector to do this. In effect you make a copy of the original container and end up with two containers, so changes to the Python list won't be reflected back on the underlying C++ container. The Python supplied std::vector wrapping should implement the protocols you care about to enable pythonic syntax by default, and can support the ABCs correctly too.. (And if they don't I'm up for writing patches!)

Set a python variable to a C++ object pointer with boost-python

I want to set a Python variable from C++ so that the C++ program can create an object Game* game = new Game(); in order for the Python code to be able to reference this instance (and call functions, etc). How can I achieve this?
I feel like I have some core misunderstanding of the way Python or Boost-Python works.
The line main_module.attr("game") = game is in a try catch statement, and the error (using PyErr_Fetch) is "No to_python (by-value) converter found for C++ type: class Game".
E.g.
class_<Game>("Game")
.def("add", &Game::add)
;
object main_module = import("__main__");
Game* game = new Game();
main_module.attr("game") = game; //This does not work
From Python:
import testmodule
testmodule.game.foo(7)

When dealing with language bindings, one often has to be pedantic in the details. By default, when a C++ object transgresses the language boundary, Boost.Python will create a copy, as this is the safest course of action to prevent dangling references. If a copy should not be made, then one needs to be explicit as to the ownership of the C++ object:
To pass a reference to a C++ object to Python while maintaining ownership in C++, use boost::python::ptr() or boost::ref(). The C++ code should guarantee that the C++ object's lifetime is at least as long as the Python object. When using ptr(), if the pointer is null, then the resulting Python object will be None.
To transfer ownership of a C++ object to Python, one can apply the manage_new_object ResultConverterGenerator, allowing ownership to be transferred to Python. C++ code should not attempt to access the pointer once the Python object's lifetime ends.
For shared ownership, one would need to expose the class with a HeldType of a smart pointer supporting shared semantics, such as boost::shared_ptr.
Once the Python object has been created, it would need to be inserted into a Python namespace to be generally accessible:
From within the module definition, use boost::python::scope to obtain a handle to the current scope. For example, the following would insert x into the example module:
BOOST_PYTHON_MODULE(example)
{
boost::python::scope().attr("x") = ...; // example.x
}
To insert into the __main__ module, one can import __main__. For example, the following would insert x into the __main__ module:
boost::python::import("__main__").attr("x") = ...;
Here is an example demonstrating how to directly construct the Python object from C++, transfer ownership of a C++ object to Python, and construct a Python object that references a C++ object:
#include <iostream>
#include <boost/python.hpp>
// Mockup model.
struct spam
{
spam(int id)
: id_(id)
{
std::cout << "spam(" << id_ << "): " << this << std::endl;
}
~spam()
{
std::cout << "~spam(" << id_ << "): " << this << std::endl;
}
// Explicitly disable copying.
spam(const spam&) = delete;
spam& operator=(const spam&) = delete;
int id_;
};
/// #brief Transfer ownership to a Python object. If the transfer fails,
/// then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
// Transfer ownership to a smart pointer, allowing for proper cleanup
// incase Boost.Python throws.
std::unique_ptr<T> ptr(t);
// Use the manage_new_object generator to transfer ownership to Python.
namespace python = boost::python;
typename python::manage_new_object::apply<T*>::type converter;
// Transfer ownership to the Python handler and release ownership
// from C++.
python::handle<> handle(converter(*ptr));
ptr.release();
return python::object(handle);
}
namespace {
spam* global_spam;
} // namespace
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose spam.
auto py_spam_type = python::class_<spam, boost::noncopyable>(
"Spam", python::init<int>())
.def_readonly("id", &spam::id_)
;
// Directly create an instance of Python Spam and insert it into this
// module's namespace.
python::scope().attr("spam1") = py_spam_type(1);
// Construct of an instance of Python Spam from C++ spam, transfering
// ownership to Python. The Python Spam instance will be inserted into
// this module's namespace.
python::scope().attr("spam2") = transfer_to_python(new spam(2));
// Construct an instance of Python Spam from C++, but retain ownership of
// spam in C++. The Python Spam instance will be inserted into the
// __main__ scope.
global_spam = new spam(3);
python::import("__main__").attr("spam3") = python::ptr(global_spam);
}
Interactive usage:
>>> import example
spam(1): 0x1884d40
spam(2): 0x1831750
spam(3): 0x183bd00
>>> assert(1 == example.spam1.id)
>>> assert(2 == example.spam2.id)
>>> assert(3 == spam3.id)
~spam(1): 0x1884d40
~spam(2): 0x1831750
In the example usage, note how Python did not destroy spam(3) upon exit, as it was not granted ownership of the underlying object.

how to include shared object in python [duplicate]

I'm just getting started with ctypes and would like to use a C++ class that I have exported in a dll file from within python using ctypes.
So lets say my C++ code looks something like this:
class MyClass {
public:
int test();
...
I would know create a .dll file that contains this class and then load the .dll file in python using ctypes.
Now how would I create an Object of type MyClass and call its test function? Is that even possible with ctypes? Alternatively I would consider using SWIG or Boost.Python but ctypes seems like the easiest option for small projects.

Besides Boost.Python(which is probably a more friendly solution for larger projects that require one-to-one mapping of C++ classes to python classes), you could provide on the C++ side a C interface. It's one solution of many so it has its own trade offs, but I will present it for the benefit of those who aren't familiar with the technique. For full disclosure, with this approach one wouldn't be interfacing C++ to python, but C++ to C to Python. Below I included an example that meets your requirements to show you the general idea of the extern "c" facility of C++ compilers.
//YourFile.cpp (compiled into a .dll or .so file)
#include <new> //For std::nothrow
//Either include a header defining your class, or define it here.
extern "C" //Tells the compile to use C-linkage for the next scope.
{
//Note: The interface this linkage region needs to use C only.
void * CreateInstanceOfClass( void )
{
// Note: Inside the function body, I can use C++.
return new(std::nothrow) MyClass;
}
//Thanks Chris.
void DeleteInstanceOfClass (void *ptr)
{
delete(std::nothrow) ptr;
}
int CallMemberTest(void *ptr)
{
// Note: A downside here is the lack of type safety.
// You could always internally(in the C++ library) save a reference to all
// pointers created of type MyClass and verify it is an element in that
//structure.
//
// Per comments with Andre, we should avoid throwing exceptions.
try
{
MyClass * ref = reinterpret_cast<MyClass *>(ptr);
return ref->Test();
}
catch(...)
{
return -1; //assuming -1 is an error condition.
}
}
} //End C linkage scope.
You can compile this code with
gcc -shared -o test.so test.cpp
#creates test.so in your current working directory.
In your python code you could do something like this (interactive prompt from 2.7 shown):
>>> from ctypes import cdll
>>> stdc=cdll.LoadLibrary("libc.so.6") # or similar to load c library
>>> stdcpp=cdll.LoadLibrary("libstdc++.so.6") # or similar to load c++ library
>>> myLib=cdll.LoadLibrary("/path/to/test.so")
>>> spam = myLib.CreateInstanceOfClass()
>>> spam
[outputs the pointer address of the element]
>>> value=CallMemberTest(spam)
[does whatever Test does to the spam reference of the object]
I'm sure Boost.Python does something similar under the hood, but perhaps understanding the lower levels concepts is helpful. I would be more excited about this method if you were attempting to access functionality of a C++ library and a one-to-one mapping was not required.
For more information on C/C++ interaction check out this page from Sun: http://dsc.sun.com/solaris/articles/mixing.html#cpp_from_c

The short story is that there is no standard binary interface for C++ in the way that there is for C. Different compilers output different binaries for the same C++ dynamic libraries, due to name mangling and different ways to handle the stack between library function calls.
So, unfortunately, there really isn't a portable way to access C++ libraries in general. But, for one compiler at a time, it's no problem.
This blog post also has a short overview of why this currently won't work. Maybe after C++0x comes out, we'll have a standard ABI for C++? Until then, you're probably not going to have any way to access C++ classes through Python's ctypes.

The answer by AudaAero is very good but not complete (at least for me).
On my system (Debian Stretch x64 with GCC and G++ 6.3.0, Python 3.5.3) I have segfaults as soon has I call a member function that access a member value of the class.
I diagnosticated by printing pointer values to stdout that the void* pointer coded on 64 bits in wrappers is being represented on 32 bits in Python. Thus big problems occurs when it is passed back to a member function wrapper.
The solution I found is to change:
spam = myLib.CreateInstanceOfClass()
Into
Class_ctor_wrapper = myLib.CreateInstanceOfClass
Class_ctor_wrapper.restype = c_void_p
spam = c_void_p(Class_ctor_wrapper())
So two things were missing: setting the return type to c_void_p (the default is int) and then creating a c_void_p object (not just an integer).
I wish I could have written a comment but I still lack 27 rep points.

Extending AudaAero's and Gabriel Devillers answer I would complete the class object instance creation by:
stdc=c_void_p(cdll.LoadLibrary("libc.so.6"))
using ctypes c_void_p data type ensures the proper representation of the class object pointer within python.
Also make sure that the dll's memory management be handled by the dll (allocated memory in the dll should be deallocated also in the dll, and not in python)!

I ran into the same problem. From trial and error and some internet research (not necessarily from knowing the g++ compiler or C++ very well), I came across this particular solution that seems to be working quite well for me.
//model.hpp
class Model{
public:
static Model* CreateModel(char* model_name) asm("CreateModel"); // static method, creates an instance of the class
double GetValue(uint32_t index) asm("GetValue"); // object method
}
#model.py
from ctypes import ...
if __name__ == '__main__':
# load dll as model_dll
# Static Method Signature
fCreateModel = getattr(model_dll, 'CreateModel') # or model_dll.CreateModel
fCreateModel.argtypes = [c_char_p]
fCreateModel.restype = c_void_p
# Object Method Signature
fGetValue = getattr(model_dll, 'GetValue') # or model_dll.GetValue
fGetValue.argtypes = [c_void_p, c_uint32] # Notice two Params
fGetValue.restype = c_double
# Calling the Methods
obj_ptr = fCreateModel(c_char_p(b"new_model"))
val = fGetValue(obj_ptr, c_int32(0)) # pass in obj_ptr as first param of obj method
>>> nm -Dg libmodel.so
U cbrt#GLIBC_2.2.5
U close#GLIBC_2.2.5
00000000000033a0 T CreateModel # <----- Static Method
U __cxa_atexit#GLIBC_2.2.5
w __cxa_finalize#GLIBC_2.2.5
U fprintf#GLIBC_2.2.5
0000000000002b40 T GetValue # <----- Object Method
w __gmon_start__
...
...
... # Mangled Symbol Names Below
0000000000002430 T _ZN12SHMEMWrapper4HashEPKc
0000000000006120 B _ZN12SHMEMWrapper8info_mapE
00000000000033f0 T _ZN5Model12DestroyModelEPKc
0000000000002b20 T _ZN5Model14GetLinearIndexElll
First, I was able to avoid the extern "C" directive completely by instead using the asm keyword which, to my knowledge, asks the compiler to use a given name instead of the generated one when exporting the function to the shared object lib's symbol table. This allowed me to avoid the weird symbol names that the C++ compiler generates automatically. They look something like the _ZN1... pattern you see above. Then in a program using Python ctypes, I was able to access the class functions directly using the custom name I gave them. The program looks like fhandle = mydll.myfunc or fhandler = getattr(mydll, 'myfunc') instead of fhandle = getattr(mydll, '_ZN12...myfunc...'). Of course, you could just use the long name; it would make no difference, but I figure the shorter name is a little cleaner and doesn't require using nm to read the symbol table and extract the names in the first place.
Second, in the spirit of Python's style of object oriented programming, I decided to try passing in my class' object pointer as the first argument of the class object method, just like when we pass self in as the first method in Python object methods. To my surprise, it worked! See the Python section above. Apparently, if you set the first argument in the fhandle.argtypes argument to c_void_ptr and pass in the ptr you get from your class' static factory method, the program should execute cleanly. Class static methods seem to work as one would expect like in Python; just use the original function signature.
I'm using g++ 12.1.1, python 3.10.5 on Arch Linux. I hope this helps someone.

Can I export c++ template class to C and therefore to python with ctypes?

For a non template class I would write something like that
But I don't know what should I do if my class is a template class.
I've tried something like that and it's not working.
extern "C" {
Demodulator<double>* Foo_new_double(){ return new Demodulator<double>(); }
Demodulator<float>* Foo_new_float(){ return new Demodulator<float>(); }
void demodulateDoubleMatrix(Demodulator<double>* demodulator, double * input, int rows, int columns){ demodulator->demodulateMatrixPy(input, rows, columns) }
}

Note: Your question contradicts the code partially, so I'm ignoring the code for now.
C++ templates are an elaborated macro mechanism that gets resolved at compile time. In other words, the binary only contains the code from template instantiations (which is what you get when you apply parameters, typically types, to the the template), and those are all that you can export from a binary to other languages. Exporting them is like exporting any regular type, see for example std::string.
Since the templates themselves don't survive compilation, there is no way that you can export them from a binary, not to C, not to Python, not even to C++! For the latter, you can provide the templates themselves though, but that doesn't include them in a binary.
Two assumptions:
Exporting/importing works via binaries. Of course, you could write an import that parses C++.
C++ specifies (or specified?) export templates, but as far as I know, this isn't really implemented in the wild, so I left that option out.

The C++ language started as a superset of C: That is, it contains new keywords, syntax and capabilities that C does not provide. C does not have the concept of a class, has no concept of a member function and does not support the concept of access restrictions. C also does not support inheritance. The really big difference, however, is templates. C has macros, and that's it.
Therefore no, you can't directly expose C++ code to C in any fashion, you will have to use C-style code in your C++ to expose the C++ layer.
template<T> T foo(T i) { /* ... */ }
extern "C" int fooInt(int i) { return foo(i); }
However C++ was originally basically a C code generator, and C++ can still interop (one way) with the C ABI: member functions are actually implemented by turning this->function(int arg); into ThisClass0int1(this, arg); or something like that. In theory, you could write something to do this to your code, perhaps leveraging clang.
But that's a non-trivial task, something that's already well-tackled by SWIG, Boost::Python and Cython.
The problem with templates, however, is that the compiler ignores them until you "instantiate" (use) them. std::vector<> is not a concrete thing until you specify std::vector<int> or something. And now the only concrete implementation of that is std::vector<int>. Until you've specified it somewhere, std::vector<string> doesn't exist in your binary.
You probably want to start by looking at something like this http://kos.gd/2013/01/5-ways-to-use-python-with-native-code/, select a tool, e.g. SWIG, and then start building an interface to expose what you want/need to C. This is a lot less work than building the wrappers yourself. Depending which tool you use, it may be as simple as writing a line saying using std::vector<int> or typedef std::vector<int> IntVector or something.
---- EDIT ----
The problem with a template class is that you are creating an entire type that C can't understand, consider:
template<typename T>
class Foo {
T a;
int b;
T c;
public:
Foo(T a_) : a(a_) {}
void DoThing();
T GetA() { return a; }
int GetB() { return b; }
T GetC() { return c; }
};
The C language doesn't support the class keyword, never mind understand that members a, b and c are private, or what a constructor is, and C doesn't understand member functions.
Again it doesn't understand templates so you'll need to do what C++ does automatically and generate an instantiation, by hand:
struct FooDouble {
double a;
int b;
double c;
};
Except, all of those variables are private. So do you really want to be exposing them? If not, you probably just need to typedef "FooDouble" to something the same size as Foo and make a macro to do that.
Then you need to write replacements for the member functions. C doesn't understand constructors, so you will need to write a
extern "C" FooDouble* FooDouble_construct(double a);
FooDouble* FooDouble_construct(double a) {
Foo* foo = new Foo(a);
return reinterept_cast<FooDouble*>(foo);
}
and a destructor
extern "C" void FooDouble_destruct(FooDouble* foo);
void FooDouble_destruct(FooDouble* foo) {
delete reinterpret_cast<Foo*>(foo);
}
and a similar pass-thru for the accessors.