Develop Reference

Can the Cython's pyx file take only a small part of the header's defenitions [duplicate]

Say I have a .h file with the following code:
class MyClass
{
public:
int Attribute1;
int Attribute2;
MyClass(){};
virtual ~MyClass(){};
virtual void Method1(string var1);
virtual float Method2(float var2);
};
And a related .pyx file with:
cdef class PyClass:
cdef MyClass *classptr
[standard __cinit__ and __dealloc__ declarations ]
cdef int Attribute1;
def Method1(self, var1):
self.classptr.Method1(var1)
... and pxd file with:
cdef extern from "mycode.h":
cdef cppclass MyClass:
MyClass() except +
int Attribute1
void Method1(string)
And I use a setup.py to create a .so library to import in a python module.
Note that even though MyClass::Method2 and MyClass::Attribute2 exist in c++, I don't mention them in Cython, so they won't be visible in the .so library. Maybe I did so because they're indirectly used by MyClass::Method1(), or just because I don't plan to invoke them in the python code.
Is this a bad practice likely to cause a problem/weird behavior?
If so, why?

cdef class PyClass:
cdef MyClass *classptr
# ...
cdef int Attribute1;
Attribute1 doesn't do what you think. It's a separate value stored as part of PyClass and has nothing to do with the Attribute1 in classptr. You probably want to write a property instead.
However, to answer your question:
Yes, it's fine for you to only wrap the specific functions you're interested in. Cython doesn't need to know all the details of your C++ classes - it only needs to know enough details to generate valid C++ code using them. A few quick examples of things it's useful to omit:
Templates. For example std::string is really a template typedef, but it may not be necessarily for a Cython wrapper to know this, or for optional allocator template types, or for numeric template types which Cython doesn't really support.
Complicated inheritance hierarchies: doesn't matter if the functions you care about actually come from a base-class. Just wrap the derived class you're using.
Interfaces that return other classes - because then you'd need to wrap the second class (which might expose a third class...).
There really is no consequence beyond not being able to call the code you haven't wrapped from python. Cython's C++ support is (and will likely always be) somewhat incomplete and it's often necessary to give it a simplified C++ interface to get anything done.

Initializing Cython objects with existing C Objects

C++ Model
Say I have the following C++ data structures I wish to expose to Python.
#include <memory>
#include <vector>
struct mystruct
{
int a, b, c, d, e, f, g, h, i, j, k, l, m;
};
typedef std::vector<std::shared_ptr<mystruct>> mystruct_list;
Boost Python
I can wrap these fairly effectively using boost::python with the following code, easily allowing me to use the existing mystruct (copying the shared_ptr) rather than recreating an existing object.
#include "mystruct.h"
#include <boost/python.hpp>
using namespace boost::python;
BOOST_PYTHON_MODULE(example)
{
class_<mystruct, std::shared_ptr<mystruct>>("MyStruct", init<>())
.def_readwrite("a", &mystruct::a);
// add the rest of the member variables
class_<mystruct_list>("MyStructList", init<>())
.def("at", &mystruct_list::at, return_value_policy<copy_const_reference>());
// add the rest of the member functions
}
Cython
In Cython, I have no idea how to extract an item from mystruct_list, without copying the underlying data. I have no idea how I could initialize MyStruct from the existing shared_ptr<mystruct>, without copying all the data over in one of various forms.
from libcpp.memory cimport shared_ptr
from cython.operator cimport dereference
cdef extern from "mystruct.h" nogil:
cdef cppclass mystruct:
int a, b, c, d, e, f, g, h, i, j, k, l, m
ctypedef vector[v] mystruct_list
cdef class MyStruct:
cdef shared_ptr[mystruct] ptr
def __cinit__(MyStruct self):
self.ptr.reset(new mystruct)
property a:
def __get__(MyStruct self):
return dereference(self.ptr).a
def __set__(MyStruct self, int value):
dereference(self.ptr).a = value
cdef class MyStructList:
cdef mystruct_list c
cdef mystruct_list.iterator it
def __cinit__(MyStructList self):
pass
def __getitem__(MyStructList self, int index):
# How do return MyStruct without copying the underlying `mystruct`
pass
I see many possible workarounds, and none of them are very satisfactory:
I could initialize an empty MyStruct, and in Cython assign over the shared_ptr. However, this would result in wasting an initalized struct for absolutely no reason.
MyStruct value
value.ptr = self.c.at(index)
return value
I also could copy the data from the existing mystruct to the new mystruct. However, this suffers from similar bloat.
MyStruct value
dereference(value.ptr).a = dereference(self.c.at(index)).a
return value
I could also expose a init=True flag for each __cinit__ method, which would prevent reconstructing the object internally if the C-object exists already (when init is False). However, this could cause catastrophic issues, since it would be exposed to the Python API and would allow dereferencing a null or uninitialized pointer.
def __cinit__(MyStruct self, bint init=True):
if init:
self.ptr.reset(new mystruct)
I could also overload __init__ with the Python-exposed constructor (which would reset self.ptr), but this would have risky memory safety if __new__ was used from the Python layer.
Bottom-Line
I would love to use Cython, for compilation speed, syntactical sugar, and numerous other reasons, as opposed to the fairly clunky boost::python. I'm looking at pybind11 right now, and it may solve the compilation speed issues, but I would still prefer to use Cython.
Is there any way I can do such a simple task idiomatically in Cython? Thanks.

The way this works in Cython is by having a factory class to create Python objects out of the shared pointer. This gives you access to the underlying C/C++ structure without copying.
Example Cython code:
<..>
cdef class MyStruct:
cdef shared_ptr[mystruct] ptr
def __cinit__(self):
# Do not create new ref here, we will
# pass one in from Cython code
self.ptr = NULL
def __dealloc__(self):
# Do de-allocation here, important!
if self.ptr is not NULL:
<de-alloc>
<rest per MyStruct code above>
cdef object PyStruct(shared_ptr[mystruct] MyStruct_ptr):
"""Python object factory class taking Cpp mystruct pointer
as argument
"""
# Create new MyStruct object. This does not create
# new structure but does allocate a null pointer
cdef MyStruct _mystruct = MyStruct()
# Set pointer of cdef class to existing struct ptr
_mystruct.ptr = MyStruct_ptr
# Return the wrapped MyStruct object with MyStruct_ptr
return _mystruct
def make_structure():
"""Function to create new Cpp mystruct and return
python object representation of it
"""
cdef MyStruct mypystruct = PyStruct(new mystruct)
return mypystruct
Note the type for the argument of PyStruct is a pointer to the Cpp struct.
mypystruct then is a python object of class MyStruct, as returned by the factory class, which refers to the
Cpp mystruct without copying. mypystruct can be safely returned in def cython functions and used in python space, per make_structure code.
To return a Python object of an existing Cpp mystruct pointer just wrap it with PyStruct like
return PyStruct(my_cpp_struct_ptr)
anywhere in your Cython code.
Obviously only def functions are visible there so the Cpp function calls would need to be wrapped as well inside MyStruct if they are to be used in Python space, at least if you want the Cpp function calls inside the Cython class to let go of the GiL (probably worth doing for obvious reasons).
For a real-world example see this Cython extension code and the underlying C code bindings in Cython. Also see this code for Python function wrapping of C function calls that let go of GIL. Not Cpp but same applies.
See also official Cython documentation on when a factory class/function is needed (Note that all constructor arguments will be passed as Python objects). For built in types, Cython does this conversion for you but for custom structures or objects a factory class/function is needed.
The Cpp structure initialisation could be handled in __new__ of PyStruct if needed, per suggestion above, if you want the factory class to actually create the C++ structure for you (depends on the use case really).
The benefit of a factory class with pointer arguments is it allows you to use existing pointers of C/C++ structures and wrap them in a Python extension class, rather than always having to create new ones. It would be perfectly safe to, for example, have multiple Python objects referring to the same underlying C struct. Python's ref counting ensures they won't be de-allocated prematurely. You should still check for null when deallocating though as the shared pointer could already had been de-allocated explicitly (eg, by del).
Note that there is, however, some overhead in creating new python objects even if they do point to the same C++ structure. Not a lot, but still.
IMO this auto de-allocation and ref counting of C/C++ pointers is one of the greatest features of Python's C extension API. As all that acts on Python objects (alone), the C/C++ structures need to be wrapped in a compatible Python object class definition.
Note - My experience is mostly in C, the above may need adjusting as I'm more familiar with regular C pointers than C++'s shared pointers.

Cython extension class: How do I expose methods in the auto-generated C struct?

I have existing C++ code that defines some classes I need to use, but I need to be able to send those classes to Python code. Specifically, I need to create class instances in C++, create Python objects to serve as wrappers for these C++ objects, then pass these Python objects to Python code for processing. This is just one piece of a larger C++ program, so it needs to be done ultimately in C++ using the C/Python API.
To make my life easier, I have used Cython to define extension classes (cdef classes) that serve as the Python wrappers for my C++ objects. I am using the typical format where the cdef class contains a pointer to the C++ class, which is then initialized when the cdef class instance is created. Since I also want to be able to replace the pointer if I have an existing C++ object to wrap, I have added methods to my cdef classes to accept() the C++ object and take its pointer. My other cdef classes successfully use the accept() method in Cython, for example when one object owns another.
Here is a sample of my Cython code:
MyCPlus.pxd
cdef extern from "MyCPlus.h" namespace "mynamespace":
cdef cppclass MyCPlus_Class:
MyCPlus_Class() except +
PyModule.pyx
cimport MyCPlus
from libcpp cimport bool
cdef class Py_Class [object Py_Class, type PyType_Class]:
cdef MyCPlus.MyCPlus_Class* thisptr
cdef bool owned
cdef void accept(self, MyCPlus.MyCPlus_Class &indata):
if self.owned:
del self.thisptr
self.thisptr = &indata
self.owned = False
def __cinit__(self):
self.thisptr = new MyCPlus.MyCPlus_Class()
self.owned = True
def __dealloc__(self):
if self.owned:
del self.thisptr
The problem comes when I try to access the accept() method from C++. I tried using the public and api keywords on my cdef class and on the accept() method, but I cannot figure out how to expose this method in the C struct in Cython's auto-generated .h file. No matter what I try, the C struct looks like this:
PyModule.h (auto-generated by Cython)
struct Py_Class {
PyObject_HEAD
struct __pyx_vtabstruct_11PyModule_Py_Class *__pyx_vtab;
mynamespace::MyCPlus_Class *thisptr;
bool owned;
};
I also tried typing the self input as a Py_Class, and I even tried forward-declaring Py_Class with the public and api keywords. I also experimented with making accept() a static method. Nothing I've tried works to expose the accept() method so that I can use it from C++. I did try accessing it through __pyx_vtab, but I got a compiler error, "invalid use of incomplete type". I have searched quite a bit, but haven't seen a solution to this. Can anyone help me? Please and thank you!

As you pointed in your comment, it does seem that the __pyx_vtab member is for Cython use only, since it doesn't even define the struct type for it in the exported header(s).
Adding to your response, one approach could also be:
cdef api class Py_Class [object Py_Class, type Py_ClassType]:
...
cdef void accept(self, MyCPlus.MyCPlus_Class &indata):
... # do stuff here
...
cdef api void (*Py_Class_accept)(Py_Class self, MyCPlus.MyCPlus_Class &indata)
Py_Class_accept = &Py_Class.accept
Basically, we define a function pointer and set it to the extension's method we want to expose. This is not that much different to your response's cdef'd function; the main difference would be that we can define our methods as usual in the class definition without having to duplicate functionality or method/function calls to another function to expose it. One caveat is that we would've to define our function pointer's signature almost verbatim to the method's one in addition to the self's extension type (in this case) and etc; then again this also applies for regular functions.
Do note that I tried this up on a C-level Cython .pyx file, I haven't and do not intent to test it on a CPP implementation file. But hopefully this might work just as fine, I guess.

This is not really a solution, but I came up with a workaround for my problem. I am still hoping for a solution that allows me to tell Cython to expose the accept() method to C++.
My workaround is that I wrote a separate function for my Python class (not a method). I then gave the api keyword both to my Python class and to the new function:
cdef api class Py_Class [object Py_Class, type PyType_Class]:
(etc.)
cdef api Py_Class wrap_MyCPlusClass(MyCPlus.MyCPlus_Class &indata):
wrapper = Py_Class()
del wrapper.thisptr
wrapper.thisptr = &indata
wrapper.owned = False
return wrapper
This gets a little unwieldy with the number of different classes I need to wrap, but at least Cython puts the function in the API where it is easy to use:
struct Py_Class* wrap_MyCPlusClass(mynamespace::MyCPlusClass &);

You probably want to use cpdef instead of cdef when declaring accept. See the docs:
Callable from Python and C
* Are declared with the cpdef statement.
* Can be called from anywhere, because it uses a little Cython magic.
* Uses the faster C calling conventions when being called from other Cython code.
Try that!

Cython - error when trying to access contents of pointer to struct

I have a cdefed class in Cython which looks very similar to this:
cdef class AprilTagDetector:
cdef capriltag.apriltag_detector_t* _apriltag_detector
def __cinit__(self):
self._apriltag_detector = capriltag.apriltag_detector_create();
# standard null checks
# standard __dealloc__(self) here
property quad_decimate:
def __get__(self):
return self._apriltag_detector.quad_decimate
The corresponding .pxd file looks like this:
cdef extern from "apriltag.h":
# The detector itself
ctypedef struct apriltag_detector_t:
pass
# Detector constructor and destructor
apriltag_detector_t* apriltag_detector_create()
void apriltag_detector_destroy(apriltag_detector_t* td);
The problem is, when I go to compile this code, it spits out this error:
property quad_decimate:
def __get__(self):
return self._apriltag_detector.quad_decimate ^
------------------------------------------------------------
apriltags.pyx:47:14: Cannot convert 'apriltag_detector_t *' to Python object
What's going on here? I haven't been able to figure it out from the Cython docs.

I, thankfully, figured out the problem when working on this project with a friend at a hackerspace.
The problem is in the ctypedef struct apriltag_detector_t block.
When I wrote pass in the block, I thought that Cython would automatically work out the internal contents of the struct, and let me access the element(s) I needed - here, quad_decimate.
Not so.
To get Cython to understand the contents of a struct, you will have to tell it what's in the struct as so:
ctypedef struct apriltag_detector_t:
float quad_decimate

Passing a bounded method in Cython as argument

I am trying to wrap some C++ code into Cython and I came up with some trouble trying to pass a method from a class as an argument to a function.
I do not know if it makes it more clear, but class A represents a statistical model (so myAMethod uses not only the arguments passed but many instance variables) and B has different methods for minimizing the function passed.
In C++ I have something of this style:
class A
{
public:
double myAMethod(double*)
};
class B
{
public:
double myBMethod(A&, double (A::*f) (double*)
}
So what I am trying to do is to use instances of both A and B in Cython code. I had no trouble wrapping the classes, but when I try to use myBMethod, I don't know how to pass a pointer of the kind A::*myAMethod
If I do this:
myBMethod(ptrToAObj[0], &ptrToAObj.myAMethod),
then Cython compiles this code to [...] &ptrToAObj->myAMethod [...], and I get the message one would expect from g++:
"ISO C++ forbids taking the address of a bound member function to form a pointer to member function."
But if I try to point straight to the class method, and do myBMethod(ptrToAObj[0], A.myAMethod), then Cython won't compile and say that
myAMethod is not a static member from A.
And that's pretty much all I was able to advance. I could work at C++ level and avoid any of these anoyances, but if I were able to use instances of A and B in Python (via Cython) interactively, it would help me in speedig my development pace.
Any help will be really appreciated, and I apologize if this question as been already answered and/or is available in a referece - I search SO, Cython reference and Smith's "Cython" book and I did not found this theme adressed.
Thanks in advance!

I have a partial (if horrendous) solution. I'm prepared to believe there's a better way, but I don't know it.
In cpp_bit.hpp:
class A {
public:
double myAMethod(double*) { return 0.0; }
};
typedef double (A::*A_f_ptr)(double *);
class B {
public:
double myBMethod(A& a, A_f_ptr f) {
double x = 0.1;
return (a.*f)(&x);
}
};
A_f_ptr getAMethod() {
return &A::myAMethod;
}
I've given the functions very basic implementations, just so I can check for really obvious crashes. I've also created a function pointer which returns a pointer to myAMethod. You'll need to do this for every method you want to wrap.
In py_bit.pyx
# distutils: language = c++
from cython.operator import dereference
cdef extern from "cpp_bit.hpp":
cdef cppclass A:
double myAMethod(double*)
cdef cppclass A_f_ptr:
pass
cdef cppclass B:
double myBMethod(A&, A_f_ptr)
cdef A_f_ptr getAMethod()
cdef class PyA:
cdef A* thisptr
def __cinit__(self):
self.thisptr = new A()
def __dealloc__(self):
del self.thisptr
cpdef myAMethod(self,double[:] o):
return self.thisptr.myAMethod(&o[0])
cdef class PyB:
cdef B* thisptr
def __cinit__(self):
self.thisptr = new B()
def __dealloc__(self):
del self.thisptr
cpdef myBMethod(self,PyA a):
return self.thisptr.myBMethod(dereference(a.thisptr),getAMethod())
I couldn't figure out how to typedef a member function pointer in Cython, so instead I created an empty cppclass with the same name. This works because cython just seems to use it for type-checking and nothing more, and since it includes cpp_bit.hpp (where it's defined) you can use it no problem.
All I've done is left the task of getting the member function pointer to c++ (in getAMethod, which I call). I don't think it's entirely satisfactory, but it looks workable, and is only a short extra c++ function for every member function you want to access. You could play with where you put it to encapsulate it more cleanly.
An alternative, untested approach: (Edit: further thought suggests this might be very tricky! Attempt this at your own risk!)
Personally, I'd be tempted to change the c++ interface so that myBMethod is defined as
double myBMethod(std::function<double (double*)>)
(since presumably you always call it with the A instance it's passed with). Then use lambda functions in c++(11!) to wrap the A instance and function together
b.myBMethod([&](double* d){ return a.myAMethod(d) };
It may then take a bit of hugely complicated Cython wrapping that I haven't yet considered, but it should be possible to convert a simple double (double*) function pointer to the c++ function object, and so use it more directly.
It's also possible that your actual design is more complicated in ways I haven't considered, and this approach isn't flexible enough anyway.