Develop Reference

How to get a C object and pass it in argument of a function in cython

I am trying to use a C library "myClib" from python.
This library has a function "myCfunction" that return a pointer struct "myCStruct" and then had other functions that take a pointer of this struct in argument.
I don't have the definition of this struct I only get this in the .h file :
typedef struct myCStruct myCStruct
The library is provided with as a static library with a .a file. Some .so files are also provided they contains some dependencies used by the library. The .h files are also provided but the full definition of "myCStruct" seems not provided in any of the .h.
I don't have to read or process any information that would be contained in this object but I need to pass it in argument of an other C functions of the library.
Right now to execute the C code I am using cython to generate a .so file that I can import in my python code.
The definition of the function in my pyx file looks like this :
cdef extern from "myClib.h":
void myCfunction()
def py_myCfunction():
return myCfunction()
I didn't declare the return type of py_myCfunction as I didn't define any class matching the myCStruct since I don't have any idea about the members.
My problem is that when I call py_myCfunction in my python program I am getting an object of type NoneType.
So 2 questions :
Could I get a Nonetype object because I miss the declaration in the pyx ? Or does it necessary means that myCfunction return nothing (it's supposed to return something).
If the problem is there because I miss the declaration, which return type could my py_myCfunction has since I don't know the member of the C struct equivalent ?

You need to declare the C API so cython knows what it is and how to call it
You should wrap the C struct * in a Python class, and add methods
to the class that call the other C API functions that reference the struct
Here's example cython code based on your question:
# declare external stuff for cython
cdef extern from "myClib.h":
struct myCStruct:
pass
struct myCStruct *myCFunction();
int otherFunction(struct myCStruct *cstruct, int parameter);
# define python glue for python
cdef class MyPythonClass:
cdef myCStruct *cstruct
def __cinit__(self):
self.cstruct = myCFunction()
# plus methods that call other functions in myClib using the self.cstruct
def other(self, parameter):
return otherFunction(self.cstruct, parameter)
# if myCLib has "myCStruct_Free"
def __dealloc__(self):
if self.cstruct != NULL:
myCStruct_Free(self.cstruct)
self.cstruct = NULL

Converting C++ object to python object in cython?

I am having a hard time with my first cython program, so please don't discourage me if I have a stupid question.
This is my example:
c_foo.h:
// c_foo.h
class foo:
{
public:
....
std::unique_ptr<2DVector> get_cFoo_unique_ptr(); // Vec2D<T> is a vector of vectors of T
// std::vector<std::vector<T>>
private:
std::unique_ptr<Vec2D> cFoo_unique_ptr; // Vec2D<T> is a vector of vectors of T
}
foo.pyx:
# foo.pyx
import cython
... # other import
from cython.operator cimport unique_ptr as cpp_unique_ptr
cdef extern from c_foo.h:
cdef cppclass c_foo:
...
cpp_unique_ptr get_cFoo_unique_ptr()
cdef class py_foo:
cdef c_foo* cFoo
...
def get_Vec2D(self):
return self.cFoo.get_cFoo_unique_ptr().get()
So, what I am trying to do here is getting data from C++ unique_ptr and return them in python.
And cython complains about it.
def get_Vec2D(self):
return self.cFoo.get_cFoo_unique_ptr().get()
----------------------------------------------------------
foo.pyx:xx:xx: Cannot convert 'T *' to Python object
Another attempt is that I try other way to declare cython unique_ptr of vectors:
# foo.pyx
import cython
... # other import
from cython.memory cimport unique_ptr as cpp_unique_ptr
from cython.vector cimport vector as cpp_vector
cdef extern from c_foo.h:
cdef cppclass c_foo:
...
cpp_unique_ptr[cpp_vector[cpp_vector[T]]] get_cFoo_unique_ptr()
cdef class py_foo:
cdef c_foo* cFoo
...
def get_Vec2D(self):
return self.cFoo.get_cFoo_unique_ptr().get()
And still the same error:
Cannot convert 'vector[vector[T]] *' to Python object
So, my question is how can I access the data of C++ unique_ptr in cython?
And is there any good practice that I should know for passing data between C++ and python via cython?
Thanks

This kind of thing confused me for a long time. I have only used Cython in a C environment, not C++, so I don't know about other options that might exist in your case. But I'll explain how I do it, and hope that will be helpful.
You must create Python objects at the end of your Cython function, since those are the only valid return values. That means, among other things, that you can't return a pointer or a C array. For example, suppose I have a Cython function f that calculates a double array:
def f():
cdef double aa[1000]
cdef int i
# ....
# some calculation that populates aa
# ....
# Now I can't just return aa because aa isn't a Python object.
# I have to convert it to a Python list:
a = []
for i in range(1000):
a.append(aa[i]) # Cython knows how convert one C float to one Python float
return a

Passing a bounded method in Cython as argument

I am trying to wrap some C++ code into Cython and I came up with some trouble trying to pass a method from a class as an argument to a function.
I do not know if it makes it more clear, but class A represents a statistical model (so myAMethod uses not only the arguments passed but many instance variables) and B has different methods for minimizing the function passed.
In C++ I have something of this style:
class A
{
public:
double myAMethod(double*)
};
class B
{
public:
double myBMethod(A&, double (A::*f) (double*)
}
So what I am trying to do is to use instances of both A and B in Cython code. I had no trouble wrapping the classes, but when I try to use myBMethod, I don't know how to pass a pointer of the kind A::*myAMethod
If I do this:
myBMethod(ptrToAObj[0], &ptrToAObj.myAMethod),
then Cython compiles this code to [...] &ptrToAObj->myAMethod [...], and I get the message one would expect from g++:
"ISO C++ forbids taking the address of a bound member function to form a pointer to member function."
But if I try to point straight to the class method, and do myBMethod(ptrToAObj[0], A.myAMethod), then Cython won't compile and say that
myAMethod is not a static member from A.
And that's pretty much all I was able to advance. I could work at C++ level and avoid any of these anoyances, but if I were able to use instances of A and B in Python (via Cython) interactively, it would help me in speedig my development pace.
Any help will be really appreciated, and I apologize if this question as been already answered and/or is available in a referece - I search SO, Cython reference and Smith's "Cython" book and I did not found this theme adressed.
Thanks in advance!

I have a partial (if horrendous) solution. I'm prepared to believe there's a better way, but I don't know it.
In cpp_bit.hpp:
class A {
public:
double myAMethod(double*) { return 0.0; }
};
typedef double (A::*A_f_ptr)(double *);
class B {
public:
double myBMethod(A& a, A_f_ptr f) {
double x = 0.1;
return (a.*f)(&x);
}
};
A_f_ptr getAMethod() {
return &A::myAMethod;
}
I've given the functions very basic implementations, just so I can check for really obvious crashes. I've also created a function pointer which returns a pointer to myAMethod. You'll need to do this for every method you want to wrap.
In py_bit.pyx
# distutils: language = c++
from cython.operator import dereference
cdef extern from "cpp_bit.hpp":
cdef cppclass A:
double myAMethod(double*)
cdef cppclass A_f_ptr:
pass
cdef cppclass B:
double myBMethod(A&, A_f_ptr)
cdef A_f_ptr getAMethod()
cdef class PyA:
cdef A* thisptr
def __cinit__(self):
self.thisptr = new A()
def __dealloc__(self):
del self.thisptr
cpdef myAMethod(self,double[:] o):
return self.thisptr.myAMethod(&o[0])
cdef class PyB:
cdef B* thisptr
def __cinit__(self):
self.thisptr = new B()
def __dealloc__(self):
del self.thisptr
cpdef myBMethod(self,PyA a):
return self.thisptr.myBMethod(dereference(a.thisptr),getAMethod())
I couldn't figure out how to typedef a member function pointer in Cython, so instead I created an empty cppclass with the same name. This works because cython just seems to use it for type-checking and nothing more, and since it includes cpp_bit.hpp (where it's defined) you can use it no problem.
All I've done is left the task of getting the member function pointer to c++ (in getAMethod, which I call). I don't think it's entirely satisfactory, but it looks workable, and is only a short extra c++ function for every member function you want to access. You could play with where you put it to encapsulate it more cleanly.
An alternative, untested approach: (Edit: further thought suggests this might be very tricky! Attempt this at your own risk!)
Personally, I'd be tempted to change the c++ interface so that myBMethod is defined as
double myBMethod(std::function<double (double*)>)
(since presumably you always call it with the A instance it's passed with). Then use lambda functions in c++(11!) to wrap the A instance and function together
b.myBMethod([&](double* d){ return a.myAMethod(d) };
It may then take a bit of hugely complicated Cython wrapping that I haven't yet considered, but it should be possible to convert a simple double (double*) function pointer to the c++ function object, and so use it more directly.
It's also possible that your actual design is more complicated in ways I haven't considered, and this approach isn't flexible enough anyway.

How to wrap a C++ functor in Cython

I'm trying to wrap a C++ library in which the logic is implemented as templatized functors in .hpp files, and I'm struggling to find the right way to expose the C++ functors as Cython/Python functions. How are functors like the one below supposed to be wrapped in Cython?
I believe this should be possible, at least for template classes and functions, according to the Cython 0.20 docs.
Note: I think I've figured out how to wrap normal C++ functions—the problem occurs when I'm trying to wrap a templatized functor, i.e. a template struct that overloads the () operator (making it act like a function when a data type is fixed).
Disclaimer: I'm a total novice in C++ and very new to Cython so apologies if I'm making obvious mistakes here.
The functor I'm trying to wrap:
#include <vector>
#include "EMD_DEFS.hpp"
#include "flow_utils.hpp"
template<typename NUM_T, FLOW_TYPE_T FLOW_TYPE= NO_FLOW>
struct emd_hat_gd_metric {
NUM_T operator()(const std::vector<NUM_T>& P, const std::vector<NUM_T>& Q,
const std::vector< std::vector<NUM_T> >& C,
NUM_T extra_mass_penalty= -1,
std::vector< std::vector<NUM_T> >* F= NULL);
};
My wrapper.pyx file:
# distutils: language = c++
from libcpp.vector cimport vector
cdef extern from "lib/emd_hat.hpp":
# Apparently `cppclass` is necessary here even though
# `emd_hat_gd_metric` is not a class...?
cdef cppclass emd_hat_gd_metric[NUM_T]:
NUM_T operator()(vector[NUM_T]& P,
vector[NUM_T]& Q,
vector[vector[NUM_T]]& C) except +
cdef class EMD:
cdef emd_hat_gd_metric *__thisptr
def __cinit__(self):
self.__thisptr = new emd_hat_gd_metric()
def __dealloc__(self):
del self.__thisptr
def calculate(self, P, Q, C):
# What goes here? How do I call the functor as a function?
return self.__thisptr(P, Q, C)
The above just gives a Calling non-function type 'emd_hat_gd_metric[NUM_T]' error when I try to compile it with cython --cplus wrapper.pyx.
Here's the full library I'm trying to wrap.
End goal: to be able to call emd_hat_gd_metric as a Cython/Python function, with arguments being NumPy arrays.

I couldn't find a real solution, but here's a workaround (that requires modifying the C++ code): just instantiate the template function with the data type you need in the C++ header, then declare that function normally in your .pyx file.
It's a little unwieldy if you need many different data types, but I only needed double. It would also be nicer if it wasn't necessary to modify the external library… but it works.
In the C++ some_library.hpp file:
Instantiate the functor with the data type you need (say, double):
template<typename T>
struct some_template_functor {
T operator()(T x);
};
// Add this:
some_template_functor<double> some_template_functor_double;
In the Cython .pyx file:
Declare the function normally (no need for cppclass):
cdef extern from "path/to/some_library.hpp":
cdef double some_template_functor_double(double x)
Then you can call some_template_functor_double from within Cython.

Wrapping a pre-initialized pointer in a cython class

I'm trying to use a C library which uses a callback function (callback_function) to provide a pointer to a struct I'd like to wrap (glp_tree).
What is the correct way to initialize an instance with a pointer not created in __cinit__? I can't find an example of this pattern in the cython documentation.
I have some working code (see below), which casts the pointer to an integer and back, but I'm not sure this is good practice / sane.
cdef extern from "stdint.h":
ctypedef unsigned long long uint64_t
cdef extern from "glpk.h":
ctypedef struct glp_tree:
pass
cdef void callback_func(glp_tree* tree, void *info):
treeobj = Tree(<uint64_t>tree) // cast to an integer...
cdef class Tree:
cdef glp_tree* ptr
def __init__(self, uint64_t ptr):
self.ptr = <glp_tree*>ptr // ... and back to a pointer
Passing the glp_tree object directly seems to work (although it's not what I want to do), but trying to pass the pointer results in a compiler error:
Cannot convert 'glp_tree *' to Python object

Instead of using __init__/__cinit__ (which always expects Python objects as arguments), you can use a custom #staticmethod cdef to create instances:
cdef class Tree:
cdef glp_tree* ptr
def __init__(self, *args):
raise TypeError('Cannot create instance from Python')
#staticmethod
cdef Tree create(glp_tree* ptr):
obj = <Tree>Tree.__new__(Tree) # create instance without calling __init__
obj.ptr = ptr
return obj

Casting a pointer to an integer is an option, but then the correct type to use is uintptr_t, not uint64_t (it's self-documenting and always has the right width for the platform).
The problem is that constructing a Tree is a Python operation, as you can clearly see in the cython -a output. The input to the constructor has to be converted to Python data structures, and pointers have no obvious conversion.

This will work
cdef class Tree:
cdef glp_tree* ptr
def __init__(self, long ptr):
self.ptr = <glp_tree*>PyLong_AsVoidPtr(ptr)