I want to wrap this simple function:
double foo(int a, int* b)
{
double div = ((double) a) / ((double) *b);
*b = a + (*b);
return div;
}
what i'm trying to do is this:
cdef pyfoo(int c, int d):
res = foo(c, &d);
return (res, d)
I want to return a list of values, but using cdef it does not work
it gives me the erroe: AttributeError no attribute pyfoo
If instead of cdef i use def or cpdef it works.
Is there some way in cython to do this using cdef?
cdef functions can't be accessed by Python code. That's why they're fast: calling functions from Python is always slow.
The solution is using cpdef instead, so that it's both accessible from Python and fast when called from C. Or, don't call it from Python code.
The problem seems that cdef creates a function with a C signature and C call semantics. They cannot be called by Python. cpdef should be used to provide a Python wrapper around a cdef function.
Look it that way: cython is providing a bridge between C and python. cdef is the on the C end of the bridge. cpdef is on the Python end of the bridge.
There is a nice talk about cython.
Related
I'd like to know the difference between def, cdef and cpdef when I declare a function.
The difference between def and the others it's more or less clear.
And I've also seen that sometimes it's added the return type in the declaration (cdef void/double/int... name) and sometimes not.
I'd also like to know how to declare a string variable in cython, as I didn't know it, I declared it as object.
The key difference is in where the function can be called from: def functions can be called from Python and Cython while cdef function can be called from Cython and C.
Both types of functions can be declared with any mixture of typed and untyped arguments, and in both cases the internals are compiled to C by Cython (and the compiled code should be very, very similar):
# A Cython class for illustrative purposes
cdef class C:
pass
def f(int arg1, C arg2, arg3):
# takes an integer, a "C" and an untyped generic python object
pass
cdef g(int arg1, C arg2, arg3):
pass
In the example above, f will be visible to Python (once it has imported the Cython module) and g will not be and cannot be called from Python. g will translate into a C signature of:
PyObject* some_name(int, struct __pyx_obj_11name_of_module_C *, PyObject*)
(where struct __pyx_obj_11name_of_module_C * is just the C struct that our class C is translated into). This allows it to be passed to C functions as a function pointer for example. In contrast f cannot (easily) be called from C.
Restrictions on cdef functions:
cdef functions cannot be defined inside other functions - this is because there is no way of storing any captured variables in a C function pointer. E.g. the following code is illegal:
# WON'T WORK!
def g(a):
cdef (int b):
return a+b
cdef functions cannot take *args and **kwargs type arguments. This is because they cannot easily be translated into a C signature.
Advantages of cdef functions
cdef functions can take any type of argument, including those that have no Python equivalent (for example pointers). def functions cannot have these, since they must be callable from Python.
cdef functions can also specify a return type (if it is not specified then they return a Python object, PyObject* in C). def functions always return a Python object, so cannot specify a return type:
cdef int h(int* a):
# specify a return type and take a non-Python compatible argument
return a[0]
cdef functions are quicker to call than def functions because they translate to a simple C function call.
cpdef functions
cpdef functions cause Cython to generate a cdef function (that allows a quick function call from Cython) and a def function (which allows you to call it from Python). Interally the def function just calls the cdef function. In terms of the types of arguments allowed, cpdef functions have all the restrictions of both cdef and def functions.
When to use a cdef function
Once the function has been called there is no difference in the speed that the code inside a cdef and a def function runs at. Therefore, only use a cdef function if:
You need to pass non-Python types in or out, or
You need to pass it to C as a function pointer, or
You are calling it often (so the sped-up function call is important) and you don't need to call it from Python.
Use a cpdef function when you are calling it often (so the sped-up function call is important) but you do need to call it from Python.
def declares a function in Python. Since Cython is based on C runtime, it allows you to use cdef and cpdef.
cdef declares function in the layer of C language. As you know (or not?) in C language you have to define type of returning value for each function. Sometimes function returns with void, and this is equal for just return in Python.
Python is an object-oriented language. So you also can define class method in layer of C++ language, and override this methods in subclasses:
cdef class A:
cdef foo(self):
print "A"
cdef class B(A)
cdef foo(self, x=None)
print "B", x
cdef class C(B):
cpdef foo(self, x=True, int k=3)
print "C", x, k
Summary, why do we need to use def, cdef and cpdef? Because if you use Cython, your Python code will be converted into C code before compile. So with this things you can control the resulting C-code listing.
For more information I suggest you to read the official documentation: http://docs.cython.org/src/reference/language_basics.html
We would need to create a PyCapsule from a method of a class in Cython. We managed to write a code which compiles and even runs without error but the results are wrong.
A simple example is here: https://github.com/paugier/cython_capi/tree/master/using_cpython_pycapsule_class
The capsules are executed by Pythran (one needs to use the version on github https://github.com/serge-sans-paille/pythran).
The .pyx file:
from cpython.pycapsule cimport PyCapsule_New
cdef int twice_func(int c):
return 2*c
cdef class Twice:
cdef public dict __pyx_capi__
def __init__(self):
self.__pyx_capi__ = self.get_capi()
cpdef get_capi(self):
return {
'twice_func': PyCapsule_New(
<void *>twice_func, 'int (int)', NULL),
'twice_cpdef': PyCapsule_New(
<void *>self.twice_cpdef, 'int (int)', NULL),
'twice_cdef': PyCapsule_New(
<void *>self.twice_cdef, 'int (int)', NULL),
'twice_static': PyCapsule_New(
<void *>self.twice_static, 'int (int)', NULL)}
cpdef int twice_cpdef(self, int c):
return 2*c
cdef int twice_cdef(self, int c):
return 2*c
#staticmethod
cdef int twice_static(int c):
return 2*c
The file compiled by pythran (call_capsule_pythran.py).
# pythran export call_capsule(int(int), int)
def call_capsule(capsule, n):
r = capsule(n)
return r
Once again it is a new feature of Pythran so one needs the version on github...
And the test file:
try:
import faulthandler
faulthandler.enable()
except ImportError:
pass
import unittest
from twice import Twice
from call_capsule_pythran import call_capsule
class TestAll(unittest.TestCase):
def setUp(self):
self.obj = Twice()
self.capi = self.obj.__pyx_capi__
def test_pythran(self):
value = 41
print('\n')
for name, capsule in self.capi.items():
print('capsule', name)
result = call_capsule(capsule, value)
if name.startswith('twice'):
if result != 2*value:
how = 'wrong'
else:
how = 'good'
print(how, f'result ({result})\n')
if __name__ == '__main__':
unittest.main()
It is buggy and gives:
capsule twice_func
good result (82)
capsule twice_cpdef
wrong result (4006664390)
capsule twice_cdef
wrong result (4006664390)
capsule twice_static
good result (82)
It shows that it works fine for the standard function and for the static function but that there is a problem for the methods.
Note that the fact that it works for two capsules seems to indicate that the problem does not come from Pythran.
Edit
After DavidW's comments, I understand that we would have to create at run time (for example in get_capi) a C function with the signature int(int) from the bound method twice_cdef whose signature is actually int(Twice, int).
I don't know if this is really impossible to do with Cython...
To follow up/expand on my comments:
The basic issue is that the Pythran is expecting a C function pointer with the signature int f(int) to be contained within the PyCapsule. However, the signature of your methods is int(PyObject* self, int c). The 2 gets passed as self (not causing disaster since it isn't actually used...) and some arbitrary bit of memory is used in place of the int c. Unfortunately it isn't possible to use pure C code to create a C function pointer with "bound arguments" so Cython can't (and realistically won't be able to) do it.
Modification 1 is to get better compile-time type checking of what you're passing to your PyCapsules by creating a function that accepts the correct types and casting in there, rather than just casting to <void*> blindly. This doesn't solve your problem but warns you at compile-time when it isn't going to work:
ctypedef int(*f_ptr_type)(int)
cdef make_PyCapsule(f_ptr_type f, string):
return PyCapsule_New(
<void *>f, string, NULL)
# then in get_capi:
'twice_func': make_PyCapsule(twice_func, b'int (int)'), # etc
It is actually possible to create C function from arbitrary Python callables using ctypes (or cffi) - see Using function pointers to methods of classes without the gil (bottom of answer). This adds an extra layer of Python calls so isn't terribly quick, and the code is a bit messy. ctypes achieves this by using runtime code generation (which isn't that portable or something you can do in pure C) to build a function on the fly and then create a pointer to that.
Although you claim in the comments that you don't think you can use the Python interpreter, I don't think this is true - Pythran generates Python extension modules (so is pretty bound to the Python interpreter) and it seems to work in your test case shown here:
_func_cache = []
cdef f_ptr_type py_to_fptr(f):
import ctypes
functype = ctypes.CFUNCTYPE(ctypes.c_int,ctypes.c_int)
ctypes_f = functype(f)
_func_cache.append(ctypes_f) # ensure references are kept
return (<f_ptr_type*><size_t>ctypes.addressof(ctypes_f))[0]
# then in make_capi:
'twice_cpdef': make_PyCapsule(py_to_fptr(self.twice_cpdef), b'int (int)')
Unfortunately it only works for cpdef and not cdef functions since it does rely on having a Python callable. cdef functions can be made to work with a lambda (provided you change get_capi to def instead of cpdef):
'twice_cdef': make_PyCapsule(py_to_fptr(lambda x: self.twice_cdef(x)), b'int (int)'),
It's all a little messy but can be made to work.
I am trying to wrap some C++ code into Cython and I came up with some trouble trying to pass a method from a class as an argument to a function.
I do not know if it makes it more clear, but class A represents a statistical model (so myAMethod uses not only the arguments passed but many instance variables) and B has different methods for minimizing the function passed.
In C++ I have something of this style:
class A
{
public:
double myAMethod(double*)
};
class B
{
public:
double myBMethod(A&, double (A::*f) (double*)
}
So what I am trying to do is to use instances of both A and B in Cython code. I had no trouble wrapping the classes, but when I try to use myBMethod, I don't know how to pass a pointer of the kind A::*myAMethod
If I do this:
myBMethod(ptrToAObj[0], &ptrToAObj.myAMethod),
then Cython compiles this code to [...] &ptrToAObj->myAMethod [...], and I get the message one would expect from g++:
"ISO C++ forbids taking the address of a bound member function to form a pointer to member function."
But if I try to point straight to the class method, and do myBMethod(ptrToAObj[0], A.myAMethod), then Cython won't compile and say that
myAMethod is not a static member from A.
And that's pretty much all I was able to advance. I could work at C++ level and avoid any of these anoyances, but if I were able to use instances of A and B in Python (via Cython) interactively, it would help me in speedig my development pace.
Any help will be really appreciated, and I apologize if this question as been already answered and/or is available in a referece - I search SO, Cython reference and Smith's "Cython" book and I did not found this theme adressed.
Thanks in advance!
I have a partial (if horrendous) solution. I'm prepared to believe there's a better way, but I don't know it.
In cpp_bit.hpp:
class A {
public:
double myAMethod(double*) { return 0.0; }
};
typedef double (A::*A_f_ptr)(double *);
class B {
public:
double myBMethod(A& a, A_f_ptr f) {
double x = 0.1;
return (a.*f)(&x);
}
};
A_f_ptr getAMethod() {
return &A::myAMethod;
}
I've given the functions very basic implementations, just so I can check for really obvious crashes. I've also created a function pointer which returns a pointer to myAMethod. You'll need to do this for every method you want to wrap.
In py_bit.pyx
# distutils: language = c++
from cython.operator import dereference
cdef extern from "cpp_bit.hpp":
cdef cppclass A:
double myAMethod(double*)
cdef cppclass A_f_ptr:
pass
cdef cppclass B:
double myBMethod(A&, A_f_ptr)
cdef A_f_ptr getAMethod()
cdef class PyA:
cdef A* thisptr
def __cinit__(self):
self.thisptr = new A()
def __dealloc__(self):
del self.thisptr
cpdef myAMethod(self,double[:] o):
return self.thisptr.myAMethod(&o[0])
cdef class PyB:
cdef B* thisptr
def __cinit__(self):
self.thisptr = new B()
def __dealloc__(self):
del self.thisptr
cpdef myBMethod(self,PyA a):
return self.thisptr.myBMethod(dereference(a.thisptr),getAMethod())
I couldn't figure out how to typedef a member function pointer in Cython, so instead I created an empty cppclass with the same name. This works because cython just seems to use it for type-checking and nothing more, and since it includes cpp_bit.hpp (where it's defined) you can use it no problem.
All I've done is left the task of getting the member function pointer to c++ (in getAMethod, which I call). I don't think it's entirely satisfactory, but it looks workable, and is only a short extra c++ function for every member function you want to access. You could play with where you put it to encapsulate it more cleanly.
An alternative, untested approach: (Edit: further thought suggests this might be very tricky! Attempt this at your own risk!)
Personally, I'd be tempted to change the c++ interface so that myBMethod is defined as
double myBMethod(std::function<double (double*)>)
(since presumably you always call it with the A instance it's passed with). Then use lambda functions in c++(11!) to wrap the A instance and function together
b.myBMethod([&](double* d){ return a.myAMethod(d) };
It may then take a bit of hugely complicated Cython wrapping that I haven't yet considered, but it should be possible to convert a simple double (double*) function pointer to the c++ function object, and so use it more directly.
It's also possible that your actual design is more complicated in ways I haven't considered, and this approach isn't flexible enough anyway.
I'm trying to wrap a C++ library in which the logic is implemented as templatized functors in .hpp files, and I'm struggling to find the right way to expose the C++ functors as Cython/Python functions. How are functors like the one below supposed to be wrapped in Cython?
I believe this should be possible, at least for template classes and functions, according to the Cython 0.20 docs.
Note: I think I've figured out how to wrap normal C++ functions—the problem occurs when I'm trying to wrap a templatized functor, i.e. a template struct that overloads the () operator (making it act like a function when a data type is fixed).
Disclaimer: I'm a total novice in C++ and very new to Cython so apologies if I'm making obvious mistakes here.
The functor I'm trying to wrap:
#include <vector>
#include "EMD_DEFS.hpp"
#include "flow_utils.hpp"
template<typename NUM_T, FLOW_TYPE_T FLOW_TYPE= NO_FLOW>
struct emd_hat_gd_metric {
NUM_T operator()(const std::vector<NUM_T>& P, const std::vector<NUM_T>& Q,
const std::vector< std::vector<NUM_T> >& C,
NUM_T extra_mass_penalty= -1,
std::vector< std::vector<NUM_T> >* F= NULL);
};
My wrapper.pyx file:
# distutils: language = c++
from libcpp.vector cimport vector
cdef extern from "lib/emd_hat.hpp":
# Apparently `cppclass` is necessary here even though
# `emd_hat_gd_metric` is not a class...?
cdef cppclass emd_hat_gd_metric[NUM_T]:
NUM_T operator()(vector[NUM_T]& P,
vector[NUM_T]& Q,
vector[vector[NUM_T]]& C) except +
cdef class EMD:
cdef emd_hat_gd_metric *__thisptr
def __cinit__(self):
self.__thisptr = new emd_hat_gd_metric()
def __dealloc__(self):
del self.__thisptr
def calculate(self, P, Q, C):
# What goes here? How do I call the functor as a function?
return self.__thisptr(P, Q, C)
The above just gives a Calling non-function type 'emd_hat_gd_metric[NUM_T]' error when I try to compile it with cython --cplus wrapper.pyx.
Here's the full library I'm trying to wrap.
End goal: to be able to call emd_hat_gd_metric as a Cython/Python function, with arguments being NumPy arrays.
I couldn't find a real solution, but here's a workaround (that requires modifying the C++ code): just instantiate the template function with the data type you need in the C++ header, then declare that function normally in your .pyx file.
It's a little unwieldy if you need many different data types, but I only needed double. It would also be nicer if it wasn't necessary to modify the external library… but it works.
In the C++ some_library.hpp file:
Instantiate the functor with the data type you need (say, double):
template<typename T>
struct some_template_functor {
T operator()(T x);
};
// Add this:
some_template_functor<double> some_template_functor_double;
In the Cython .pyx file:
Declare the function normally (no need for cppclass):
cdef extern from "path/to/some_library.hpp":
cdef double some_template_functor_double(double x)
Then you can call some_template_functor_double from within Cython.
In cython the marshalling of the types like int or char is done automatically,
but if I use a
cdef struct MyClass_Tag:
pass
ctypedef MyClass_Tag* MyClass_ptr
....
cdef class MyClass:
cdef MyClass_ptr obj
....
Now for wrapping any function like for example in c is some function foo that takes
foo(char* , MyClass_ptr self)
#return stuff
To wrap this function I do:
def py_foo(char* n, self):
return foo(n,self.obj)
So from py to c:
char is done automatically
but self is of type MyClass so to call the foo i have to write self.obj ,
so i pass the same obj
in here is where happen the marshalling from py to c
The problem is that I don't understand where happen the marshalling from c to py,
I mean at which point does it happen?
Even in this case or even if you give me some other example it will be ok.
Thank you!
I didn't completly understood your question, but you might want to use the command:
cython -a <yourfile.pyx>
Then, open the generated with your webbrowser. You'll see how cython have translated your code into C, and then, found your answers :)
As the parameter name self suggests the C foo() function might make more sense as a method in Python:
cdef extern void c_foo "foo" (char* n, MyClass_ptr)
cdef class MyClass:
cdef MyClass_ptr this
def foo(self, unicode n not None):
b = n.encode('ascii')
cdef char* p = b
c_foo(p, self.this)
See also Wrapping C library.
Cython translates your code to C (or C++). At this point you can forget it exists. Generated C code is an ordinary C extension for Python. This code contains ordinary Python C API calls such as PyBytes_AsString() (convert Python bytes to char*) or PyInt_FromLong() (convert C int to Python integer).
MyClass_ptr is used as is:
struct __pyx_obj_3foo_MyClass {
PyObject_HEAD
MyClass_ptr this; /* <-- using MyClass_ptr as is */
};
i.e., pure Python code never sees it so there is no conversion from/to Python objects.