In cython the marshalling of the types like int or char is done automatically,
but if I use a
cdef struct MyClass_Tag:
pass
ctypedef MyClass_Tag* MyClass_ptr
....
cdef class MyClass:
cdef MyClass_ptr obj
....
Now for wrapping any function like for example in c is some function foo that takes
foo(char* , MyClass_ptr self)
#return stuff
To wrap this function I do:
def py_foo(char* n, self):
return foo(n,self.obj)
So from py to c:
char is done automatically
but self is of type MyClass so to call the foo i have to write self.obj ,
so i pass the same obj
in here is where happen the marshalling from py to c
The problem is that I don't understand where happen the marshalling from c to py,
I mean at which point does it happen?
Even in this case or even if you give me some other example it will be ok.
Thank you!
I didn't completly understood your question, but you might want to use the command:
cython -a <yourfile.pyx>
Then, open the generated with your webbrowser. You'll see how cython have translated your code into C, and then, found your answers :)
As the parameter name self suggests the C foo() function might make more sense as a method in Python:
cdef extern void c_foo "foo" (char* n, MyClass_ptr)
cdef class MyClass:
cdef MyClass_ptr this
def foo(self, unicode n not None):
b = n.encode('ascii')
cdef char* p = b
c_foo(p, self.this)
See also Wrapping C library.
Cython translates your code to C (or C++). At this point you can forget it exists. Generated C code is an ordinary C extension for Python. This code contains ordinary Python C API calls such as PyBytes_AsString() (convert Python bytes to char*) or PyInt_FromLong() (convert C int to Python integer).
MyClass_ptr is used as is:
struct __pyx_obj_3foo_MyClass {
PyObject_HEAD
MyClass_ptr this; /* <-- using MyClass_ptr as is */
};
i.e., pure Python code never sees it so there is no conversion from/to Python objects.
Related
I am trying to use a C library "myClib" from python.
This library has a function "myCfunction" that return a pointer struct "myCStruct" and then had other functions that take a pointer of this struct in argument.
I don't have the definition of this struct I only get this in the .h file :
typedef struct myCStruct myCStruct
The library is provided with as a static library with a .a file. Some .so files are also provided they contains some dependencies used by the library. The .h files are also provided but the full definition of "myCStruct" seems not provided in any of the .h.
I don't have to read or process any information that would be contained in this object but I need to pass it in argument of an other C functions of the library.
Right now to execute the C code I am using cython to generate a .so file that I can import in my python code.
The definition of the function in my pyx file looks like this :
cdef extern from "myClib.h":
void myCfunction()
def py_myCfunction():
return myCfunction()
I didn't declare the return type of py_myCfunction as I didn't define any class matching the myCStruct since I don't have any idea about the members.
My problem is that when I call py_myCfunction in my python program I am getting an object of type NoneType.
So 2 questions :
Could I get a Nonetype object because I miss the declaration in the pyx ? Or does it necessary means that myCfunction return nothing (it's supposed to return something).
If the problem is there because I miss the declaration, which return type could my py_myCfunction has since I don't know the member of the C struct equivalent ?
You need to declare the C API so cython knows what it is and how to call it
You should wrap the C struct * in a Python class, and add methods
to the class that call the other C API functions that reference the struct
Here's example cython code based on your question:
# declare external stuff for cython
cdef extern from "myClib.h":
struct myCStruct:
pass
struct myCStruct *myCFunction();
int otherFunction(struct myCStruct *cstruct, int parameter);
# define python glue for python
cdef class MyPythonClass:
cdef myCStruct *cstruct
def __cinit__(self):
self.cstruct = myCFunction()
# plus methods that call other functions in myClib using the self.cstruct
def other(self, parameter):
return otherFunction(self.cstruct, parameter)
# if myCLib has "myCStruct_Free"
def __dealloc__(self):
if self.cstruct != NULL:
myCStruct_Free(self.cstruct)
self.cstruct = NULL
I'd like to know the difference between def, cdef and cpdef when I declare a function.
The difference between def and the others it's more or less clear.
And I've also seen that sometimes it's added the return type in the declaration (cdef void/double/int... name) and sometimes not.
I'd also like to know how to declare a string variable in cython, as I didn't know it, I declared it as object.
The key difference is in where the function can be called from: def functions can be called from Python and Cython while cdef function can be called from Cython and C.
Both types of functions can be declared with any mixture of typed and untyped arguments, and in both cases the internals are compiled to C by Cython (and the compiled code should be very, very similar):
# A Cython class for illustrative purposes
cdef class C:
pass
def f(int arg1, C arg2, arg3):
# takes an integer, a "C" and an untyped generic python object
pass
cdef g(int arg1, C arg2, arg3):
pass
In the example above, f will be visible to Python (once it has imported the Cython module) and g will not be and cannot be called from Python. g will translate into a C signature of:
PyObject* some_name(int, struct __pyx_obj_11name_of_module_C *, PyObject*)
(where struct __pyx_obj_11name_of_module_C * is just the C struct that our class C is translated into). This allows it to be passed to C functions as a function pointer for example. In contrast f cannot (easily) be called from C.
Restrictions on cdef functions:
cdef functions cannot be defined inside other functions - this is because there is no way of storing any captured variables in a C function pointer. E.g. the following code is illegal:
# WON'T WORK!
def g(a):
cdef (int b):
return a+b
cdef functions cannot take *args and **kwargs type arguments. This is because they cannot easily be translated into a C signature.
Advantages of cdef functions
cdef functions can take any type of argument, including those that have no Python equivalent (for example pointers). def functions cannot have these, since they must be callable from Python.
cdef functions can also specify a return type (if it is not specified then they return a Python object, PyObject* in C). def functions always return a Python object, so cannot specify a return type:
cdef int h(int* a):
# specify a return type and take a non-Python compatible argument
return a[0]
cdef functions are quicker to call than def functions because they translate to a simple C function call.
cpdef functions
cpdef functions cause Cython to generate a cdef function (that allows a quick function call from Cython) and a def function (which allows you to call it from Python). Interally the def function just calls the cdef function. In terms of the types of arguments allowed, cpdef functions have all the restrictions of both cdef and def functions.
When to use a cdef function
Once the function has been called there is no difference in the speed that the code inside a cdef and a def function runs at. Therefore, only use a cdef function if:
You need to pass non-Python types in or out, or
You need to pass it to C as a function pointer, or
You are calling it often (so the sped-up function call is important) and you don't need to call it from Python.
Use a cpdef function when you are calling it often (so the sped-up function call is important) but you do need to call it from Python.
def declares a function in Python. Since Cython is based on C runtime, it allows you to use cdef and cpdef.
cdef declares function in the layer of C language. As you know (or not?) in C language you have to define type of returning value for each function. Sometimes function returns with void, and this is equal for just return in Python.
Python is an object-oriented language. So you also can define class method in layer of C++ language, and override this methods in subclasses:
cdef class A:
cdef foo(self):
print "A"
cdef class B(A)
cdef foo(self, x=None)
print "B", x
cdef class C(B):
cpdef foo(self, x=True, int k=3)
print "C", x, k
Summary, why do we need to use def, cdef and cpdef? Because if you use Cython, your Python code will be converted into C code before compile. So with this things you can control the resulting C-code listing.
For more information I suggest you to read the official documentation: http://docs.cython.org/src/reference/language_basics.html
We would need to create a PyCapsule from a method of a class in Cython. We managed to write a code which compiles and even runs without error but the results are wrong.
A simple example is here: https://github.com/paugier/cython_capi/tree/master/using_cpython_pycapsule_class
The capsules are executed by Pythran (one needs to use the version on github https://github.com/serge-sans-paille/pythran).
The .pyx file:
from cpython.pycapsule cimport PyCapsule_New
cdef int twice_func(int c):
return 2*c
cdef class Twice:
cdef public dict __pyx_capi__
def __init__(self):
self.__pyx_capi__ = self.get_capi()
cpdef get_capi(self):
return {
'twice_func': PyCapsule_New(
<void *>twice_func, 'int (int)', NULL),
'twice_cpdef': PyCapsule_New(
<void *>self.twice_cpdef, 'int (int)', NULL),
'twice_cdef': PyCapsule_New(
<void *>self.twice_cdef, 'int (int)', NULL),
'twice_static': PyCapsule_New(
<void *>self.twice_static, 'int (int)', NULL)}
cpdef int twice_cpdef(self, int c):
return 2*c
cdef int twice_cdef(self, int c):
return 2*c
#staticmethod
cdef int twice_static(int c):
return 2*c
The file compiled by pythran (call_capsule_pythran.py).
# pythran export call_capsule(int(int), int)
def call_capsule(capsule, n):
r = capsule(n)
return r
Once again it is a new feature of Pythran so one needs the version on github...
And the test file:
try:
import faulthandler
faulthandler.enable()
except ImportError:
pass
import unittest
from twice import Twice
from call_capsule_pythran import call_capsule
class TestAll(unittest.TestCase):
def setUp(self):
self.obj = Twice()
self.capi = self.obj.__pyx_capi__
def test_pythran(self):
value = 41
print('\n')
for name, capsule in self.capi.items():
print('capsule', name)
result = call_capsule(capsule, value)
if name.startswith('twice'):
if result != 2*value:
how = 'wrong'
else:
how = 'good'
print(how, f'result ({result})\n')
if __name__ == '__main__':
unittest.main()
It is buggy and gives:
capsule twice_func
good result (82)
capsule twice_cpdef
wrong result (4006664390)
capsule twice_cdef
wrong result (4006664390)
capsule twice_static
good result (82)
It shows that it works fine for the standard function and for the static function but that there is a problem for the methods.
Note that the fact that it works for two capsules seems to indicate that the problem does not come from Pythran.
Edit
After DavidW's comments, I understand that we would have to create at run time (for example in get_capi) a C function with the signature int(int) from the bound method twice_cdef whose signature is actually int(Twice, int).
I don't know if this is really impossible to do with Cython...
To follow up/expand on my comments:
The basic issue is that the Pythran is expecting a C function pointer with the signature int f(int) to be contained within the PyCapsule. However, the signature of your methods is int(PyObject* self, int c). The 2 gets passed as self (not causing disaster since it isn't actually used...) and some arbitrary bit of memory is used in place of the int c. Unfortunately it isn't possible to use pure C code to create a C function pointer with "bound arguments" so Cython can't (and realistically won't be able to) do it.
Modification 1 is to get better compile-time type checking of what you're passing to your PyCapsules by creating a function that accepts the correct types and casting in there, rather than just casting to <void*> blindly. This doesn't solve your problem but warns you at compile-time when it isn't going to work:
ctypedef int(*f_ptr_type)(int)
cdef make_PyCapsule(f_ptr_type f, string):
return PyCapsule_New(
<void *>f, string, NULL)
# then in get_capi:
'twice_func': make_PyCapsule(twice_func, b'int (int)'), # etc
It is actually possible to create C function from arbitrary Python callables using ctypes (or cffi) - see Using function pointers to methods of classes without the gil (bottom of answer). This adds an extra layer of Python calls so isn't terribly quick, and the code is a bit messy. ctypes achieves this by using runtime code generation (which isn't that portable or something you can do in pure C) to build a function on the fly and then create a pointer to that.
Although you claim in the comments that you don't think you can use the Python interpreter, I don't think this is true - Pythran generates Python extension modules (so is pretty bound to the Python interpreter) and it seems to work in your test case shown here:
_func_cache = []
cdef f_ptr_type py_to_fptr(f):
import ctypes
functype = ctypes.CFUNCTYPE(ctypes.c_int,ctypes.c_int)
ctypes_f = functype(f)
_func_cache.append(ctypes_f) # ensure references are kept
return (<f_ptr_type*><size_t>ctypes.addressof(ctypes_f))[0]
# then in make_capi:
'twice_cpdef': make_PyCapsule(py_to_fptr(self.twice_cpdef), b'int (int)')
Unfortunately it only works for cpdef and not cdef functions since it does rely on having a Python callable. cdef functions can be made to work with a lambda (provided you change get_capi to def instead of cpdef):
'twice_cdef': make_PyCapsule(py_to_fptr(lambda x: self.twice_cdef(x)), b'int (int)'),
It's all a little messy but can be made to work.
I am trying to wrap some C++ code into Cython and I came up with some trouble trying to pass a method from a class as an argument to a function.
I do not know if it makes it more clear, but class A represents a statistical model (so myAMethod uses not only the arguments passed but many instance variables) and B has different methods for minimizing the function passed.
In C++ I have something of this style:
class A
{
public:
double myAMethod(double*)
};
class B
{
public:
double myBMethod(A&, double (A::*f) (double*)
}
So what I am trying to do is to use instances of both A and B in Cython code. I had no trouble wrapping the classes, but when I try to use myBMethod, I don't know how to pass a pointer of the kind A::*myAMethod
If I do this:
myBMethod(ptrToAObj[0], &ptrToAObj.myAMethod),
then Cython compiles this code to [...] &ptrToAObj->myAMethod [...], and I get the message one would expect from g++:
"ISO C++ forbids taking the address of a bound member function to form a pointer to member function."
But if I try to point straight to the class method, and do myBMethod(ptrToAObj[0], A.myAMethod), then Cython won't compile and say that
myAMethod is not a static member from A.
And that's pretty much all I was able to advance. I could work at C++ level and avoid any of these anoyances, but if I were able to use instances of A and B in Python (via Cython) interactively, it would help me in speedig my development pace.
Any help will be really appreciated, and I apologize if this question as been already answered and/or is available in a referece - I search SO, Cython reference and Smith's "Cython" book and I did not found this theme adressed.
Thanks in advance!
I have a partial (if horrendous) solution. I'm prepared to believe there's a better way, but I don't know it.
In cpp_bit.hpp:
class A {
public:
double myAMethod(double*) { return 0.0; }
};
typedef double (A::*A_f_ptr)(double *);
class B {
public:
double myBMethod(A& a, A_f_ptr f) {
double x = 0.1;
return (a.*f)(&x);
}
};
A_f_ptr getAMethod() {
return &A::myAMethod;
}
I've given the functions very basic implementations, just so I can check for really obvious crashes. I've also created a function pointer which returns a pointer to myAMethod. You'll need to do this for every method you want to wrap.
In py_bit.pyx
# distutils: language = c++
from cython.operator import dereference
cdef extern from "cpp_bit.hpp":
cdef cppclass A:
double myAMethod(double*)
cdef cppclass A_f_ptr:
pass
cdef cppclass B:
double myBMethod(A&, A_f_ptr)
cdef A_f_ptr getAMethod()
cdef class PyA:
cdef A* thisptr
def __cinit__(self):
self.thisptr = new A()
def __dealloc__(self):
del self.thisptr
cpdef myAMethod(self,double[:] o):
return self.thisptr.myAMethod(&o[0])
cdef class PyB:
cdef B* thisptr
def __cinit__(self):
self.thisptr = new B()
def __dealloc__(self):
del self.thisptr
cpdef myBMethod(self,PyA a):
return self.thisptr.myBMethod(dereference(a.thisptr),getAMethod())
I couldn't figure out how to typedef a member function pointer in Cython, so instead I created an empty cppclass with the same name. This works because cython just seems to use it for type-checking and nothing more, and since it includes cpp_bit.hpp (where it's defined) you can use it no problem.
All I've done is left the task of getting the member function pointer to c++ (in getAMethod, which I call). I don't think it's entirely satisfactory, but it looks workable, and is only a short extra c++ function for every member function you want to access. You could play with where you put it to encapsulate it more cleanly.
An alternative, untested approach: (Edit: further thought suggests this might be very tricky! Attempt this at your own risk!)
Personally, I'd be tempted to change the c++ interface so that myBMethod is defined as
double myBMethod(std::function<double (double*)>)
(since presumably you always call it with the A instance it's passed with). Then use lambda functions in c++(11!) to wrap the A instance and function together
b.myBMethod([&](double* d){ return a.myAMethod(d) };
It may then take a bit of hugely complicated Cython wrapping that I haven't yet considered, but it should be possible to convert a simple double (double*) function pointer to the c++ function object, and so use it more directly.
It's also possible that your actual design is more complicated in ways I haven't considered, and this approach isn't flexible enough anyway.
I want to wrap this simple function:
double foo(int a, int* b)
{
double div = ((double) a) / ((double) *b);
*b = a + (*b);
return div;
}
what i'm trying to do is this:
cdef pyfoo(int c, int d):
res = foo(c, &d);
return (res, d)
I want to return a list of values, but using cdef it does not work
it gives me the erroe: AttributeError no attribute pyfoo
If instead of cdef i use def or cpdef it works.
Is there some way in cython to do this using cdef?
cdef functions can't be accessed by Python code. That's why they're fast: calling functions from Python is always slow.
The solution is using cpdef instead, so that it's both accessible from Python and fast when called from C. Or, don't call it from Python code.
The problem seems that cdef creates a function with a C signature and C call semantics. They cannot be called by Python. cpdef should be used to provide a Python wrapper around a cdef function.
Look it that way: cython is providing a bridge between C and python. cdef is the on the C end of the bridge. cpdef is on the Python end of the bridge.
There is a nice talk about cython.