I am having a hard time with my first cython program, so please don't discourage me if I have a stupid question.
This is my example:
c_foo.h:
// c_foo.h
class foo:
{
public:
....
std::unique_ptr<2DVector> get_cFoo_unique_ptr(); // Vec2D<T> is a vector of vectors of T
// std::vector<std::vector<T>>
private:
std::unique_ptr<Vec2D> cFoo_unique_ptr; // Vec2D<T> is a vector of vectors of T
}
foo.pyx:
# foo.pyx
import cython
... # other import
from cython.operator cimport unique_ptr as cpp_unique_ptr
cdef extern from c_foo.h:
cdef cppclass c_foo:
...
cpp_unique_ptr get_cFoo_unique_ptr()
cdef class py_foo:
cdef c_foo* cFoo
...
def get_Vec2D(self):
return self.cFoo.get_cFoo_unique_ptr().get()
So, what I am trying to do here is getting data from C++ unique_ptr and return them in python.
And cython complains about it.
def get_Vec2D(self):
return self.cFoo.get_cFoo_unique_ptr().get()
----------------------------------------------------------
foo.pyx:xx:xx: Cannot convert 'T *' to Python object
Another attempt is that I try other way to declare cython unique_ptr of vectors:
# foo.pyx
import cython
... # other import
from cython.memory cimport unique_ptr as cpp_unique_ptr
from cython.vector cimport vector as cpp_vector
cdef extern from c_foo.h:
cdef cppclass c_foo:
...
cpp_unique_ptr[cpp_vector[cpp_vector[T]]] get_cFoo_unique_ptr()
cdef class py_foo:
cdef c_foo* cFoo
...
def get_Vec2D(self):
return self.cFoo.get_cFoo_unique_ptr().get()
And still the same error:
Cannot convert 'vector[vector[T]] *' to Python object
So, my question is how can I access the data of C++ unique_ptr in cython?
And is there any good practice that I should know for passing data between C++ and python via cython?
Thanks
This kind of thing confused me for a long time. I have only used Cython in a C environment, not C++, so I don't know about other options that might exist in your case. But I'll explain how I do it, and hope that will be helpful.
You must create Python objects at the end of your Cython function, since those are the only valid return values. That means, among other things, that you can't return a pointer or a C array. For example, suppose I have a Cython function f that calculates a double array:
def f():
cdef double aa[1000]
cdef int i
# ....
# some calculation that populates aa
# ....
# Now I can't just return aa because aa isn't a Python object.
# I have to convert it to a Python list:
a = []
for i in range(1000):
a.append(aa[i]) # Cython knows how convert one C float to one Python float
return a
Related
I'm trying to run a cython example which is a bit more complex than the one which can be found in many tutorials (e.g. this guide ).
Here are a minimal example (please don't mind it doesn't have much functionality) and steps to reproduce my problem:
There are c++-classesRectangle and Group2 (I put here everything into the .h file to make it shorter):
// Rectangle.h
namespace shapes {
class Rectangle {
public:
Rectangle() {}
};
class Group2 {
public:
Group2(Rectangle rect0, Rectangle rect1) {}
};
}
Then I create a grp2.pyx file (in the same folder as the above header), with wrappers for Rectangle and Group2:
# RECTANGLE
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass Rectangle:
Rectangle() except +
cdef class PyRectangle:
cdef Rectangle c_rect
def __cinit__(self):
self.c_rect = Rectangle()
# GROUP2
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass Group2:
Group2(Rectangle rect0, Rectangle rect1) except +
cdef class PyGroup2:
cdef Group2 c_group2
def __cinit__(self, Rectangle rect0, Rectangle rect1):
self.c_group2 = Group2(rect0, rect1)
The extension is built via a setup.py file that I call from command line (python setup.py build_ext -i):
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules = cythonize(Extension(
name="grp2", # the extension name
sources=["grp2.pyx"], # the Cython source
language="c++", # generate and compile C++ code
)))
At this point I have the error in the _cinint_ of PyGroup2:
Cannot convert Python object argument to type 'Rectangle'
I suppose there is some mistake in my pyx file, but I cannot tell what.
You should use PyRectangle in the signatures of def-functions and PyRectangle.c_rect when passing rectangles to C++-functions.
That means your code should be:
cdef class PyGroup2:
...
def __cinit__(self, PyRectangle rect0, PyRectangle rect1):
self.c_group2 = Group2(rect0.c_rect, rect1.c_rect)
Read on for a more detailed explanation why.
All arguments passed to def-functions are Python-objects (i.e. of type object in Cython-parlance), after all those functions will be called from pure Python, which only knows Python-objects.
However, you can add some syntactic sugar and use "late-binding" in the signature of a def-function, for example, instead of
def do_something(n):
...
use
def do_something(int n):
...
Under the hood, Cython will transform this code to something like:
def do_something(n_):
cdef int n = n_ # conversion to C-int
...
This automatic conversion is possible for builtin-types like int or double, because there is functionality in Python-C-API for these conversions (i.e. PyLong_AsLong, PyFloat_AsDouble). Cython also handles the error checking, so you should not undertake these conversion manually.
However, for user-defined types/classes like your Rectangle-class such automatic conversion is not possible - Cython can only automatically convert to cdef-classes/extensions, i.e. PyRectangle, thus PyRectangle should be used in the signature:
cdef class PyGroup2:
...
def __cinit__(self, PyRectangle rect0, PyRectangle rect1):
...
After Cython took care of conversion from object to PyRectangle, the last step from PyRectangle to Rectangle must be taken manually by utilizing the c_rect - pointer:
...
def __cinit__(self, PyRectangle rect0, PyRectangle rect1):
self.c_group2 = Group2(rect0.c_rect, rect1.c_rect)
The rules are similar for cpdef-function, because they can be called from pure Python. The "early binding" works only for types which Cython can automatically coverted from/to Python objects.
Unsurprisingly, the only function which can habe C++-classes in their signatures are the cdef-functions.
This question already has answers here:
Wrapping a pre-initialized pointer in a cython class
(3 answers)
Closed 1 year ago.
I am trying to implement cdef class with cinit which must consume cdef struct pointer.
C header
typedef struct foo *FOO;
PXD files:
extern ...:
cdef struct foo
ctypedef foo *FOO
PYX file:
cdef class ClassX(object):
def __cinit__(self, FOO arg):
...
And Cython compiler says "Cannot convert Python object argument to type 'FOO'". What is wrong and how to pass struct pointer to cinit?
Looks like I found the similar question: how to cython class function's parameter accept c++ struct. Short answer: it is not possible to pass C-struct to Python (def) function. I have to wrap a struct into Python object.
Here are some ways I managed to solve it for my case, which are very specific.
Method 1 - Convert pointer to a uint64 number (kind of dirty)
You have to make sure the reference isn't deleted before a copy, but first convert your struct pointer into an unsigned 64bit number and then cast and possibly deref back into your object. Like so
from cython.operator cimport dereference as deref
ctypedef unsigned long long uint64_t
...
cdef class SomeClass(object):
cdef FooStruct foo
def __cinit__(self, uint64_t arg):
self.foo = deref(<FooStruct*> arg) # preferably copy over keeping pointer
Then to construct it
cdef FooStruct yourfoo = ... # your struct
cdef SomeClass res
res = SomeClass(<uint64_t> &yourfoo)
It's not the most pretty solution, but it'll work for specific situations.
Method 2 - Same PYX file assignment function (preferable)
This method I found to only work with object references in the same PYX file. So if you define your cdef class with a setter function that copies the data over, which is not exposed to python, then you can update the internal struct.
cdef class SomeClass(object):
cdef FooStruct foo
def __cinit__(self):
...
cdef void _set(self, FooStruct &data):
self.foo = data # copy
Now to construct it's two lines
cdef FooStruct yourfoo = ... # your struct
cdef SomeClass res
res = SomeClass()
res._set(yourfoo)
I have a cdefed class in Cython which looks very similar to this:
cdef class AprilTagDetector:
cdef capriltag.apriltag_detector_t* _apriltag_detector
def __cinit__(self):
self._apriltag_detector = capriltag.apriltag_detector_create();
# standard null checks
# standard __dealloc__(self) here
property quad_decimate:
def __get__(self):
return self._apriltag_detector.quad_decimate
The corresponding .pxd file looks like this:
cdef extern from "apriltag.h":
# The detector itself
ctypedef struct apriltag_detector_t:
pass
# Detector constructor and destructor
apriltag_detector_t* apriltag_detector_create()
void apriltag_detector_destroy(apriltag_detector_t* td);
The problem is, when I go to compile this code, it spits out this error:
property quad_decimate:
def __get__(self):
return self._apriltag_detector.quad_decimate ^
------------------------------------------------------------
apriltags.pyx:47:14: Cannot convert 'apriltag_detector_t *' to Python object
What's going on here? I haven't been able to figure it out from the Cython docs.
I, thankfully, figured out the problem when working on this project with a friend at a hackerspace.
The problem is in the ctypedef struct apriltag_detector_t block.
When I wrote pass in the block, I thought that Cython would automatically work out the internal contents of the struct, and let me access the element(s) I needed - here, quad_decimate.
Not so.
To get Cython to understand the contents of a struct, you will have to tell it what's in the struct as so:
ctypedef struct apriltag_detector_t:
float quad_decimate
The problem was, that I had a C++ vector and needed to convert it to a numpy array. The vector was from an external library. All I could sanely do, was to add an extra function into the source and recompile it again to the original DLL. The idea was, that I import the function with ctypes, then call it with a void pointer pointer and let the C++ function insert the vector pointer.
Here started the first problems. I was only able to send a pointer back to Python by explicitly allocating memory. Not only needed I to use
myVec = new vector<T>
even for an int pointer I needed to use
myIntP = new int
Simply doing
x = 10
int* myIntP = &x
*pythonIntPP = myIntP
will not work. Back in Python, I will still have the original pointer.
**pythonIntPP = x
will work, though.
Why do I have to allocate memory for a pointer?
Anyway, let's get to the solution I found:
First, you will have a C++ function in a DLL that accepts a void**.
In Python do
from ctypes import *
lib = cdll.LoadLibrary(r"C:\path\to.dll")
gvp = lib.GetVectorPointer
gvp.restype = c_int
gvp.argtypes = [POINTER(c_void_p), POINTER(c_int)] # we need the vector pointer and size
voidPP = pointer(c_void_p())
sizeP = pointer(c_int())
gvp(voidPP, sizeP)
Now we have the vector pointer and size in a Python object, but we can't unpack the pointer in Cython. So, we need create a cdef function, export it in a DLL (pyd file) and again import it within Python.
In Cython we do
import numpy as np
cimport numpy as np
from libcpp.vector cimport vector
ctypedef vector[T] Vec
ctypedef myVec* VecP
ctypedef VecP* VecPP
arr = np.empty(1, dtype=np.uint8)
def allocate_arr(size):
arr.resize(size, refcheck=False)
cdef public int vec_to_numpy(void** voidPP):
cdef VecP vecp= <VecP>ppVoid[0]
cdef int size = vecp[0].size()
cdef np.ndarray[T, ndim=1] data = arr # arr needs to be allocated
cdef int i
for i in range(size):
data[i] = vecp[0][i]
return 0
In Python we can now call this function and get the array
lib = windll.LoadLibrary(r"C:\path\to\module.pyd")
v2n = lib.vec_to_numpy
v2n.restype = c_int
v2n.argtypes = [POINTER(c_void_p)]
import module
module.allocate_arr(sizeP) # allocate first
v2n(voidPP) # send the vector pointer to the Cython function
module.arr # this is the numpy array with the vector data
TIP:
You can tell the linker to export your Cython function with one of two hacks.
Write this line for every function to export at the beginning of your code (change names accordingly).
cdef void emit_pragma 'foo ();\n#pragma comment(linker, "/export:vec_to_numpy=vec_to_numpy")\nstatic void unused' ():pass
or use this hack in your function definition (use __stdcall to avoid name mangling).
cdef int vec_to_numpy 'unused;\nextern "C" __declspec(dllexport) public int __stdcall vec_to_numpy' (void** voidPP):
Do you know of better ways to have full flexibility between those three languages?
Is there a simpler way to export symbols, so that they can be imported into Python that doesn't involve hacking? Being able to use cdef function directly in Python is very handy to me.
In cython the marshalling of the types like int or char is done automatically,
but if I use a
cdef struct MyClass_Tag:
pass
ctypedef MyClass_Tag* MyClass_ptr
....
cdef class MyClass:
cdef MyClass_ptr obj
....
Now for wrapping any function like for example in c is some function foo that takes
foo(char* , MyClass_ptr self)
#return stuff
To wrap this function I do:
def py_foo(char* n, self):
return foo(n,self.obj)
So from py to c:
char is done automatically
but self is of type MyClass so to call the foo i have to write self.obj ,
so i pass the same obj
in here is where happen the marshalling from py to c
The problem is that I don't understand where happen the marshalling from c to py,
I mean at which point does it happen?
Even in this case or even if you give me some other example it will be ok.
Thank you!
I didn't completly understood your question, but you might want to use the command:
cython -a <yourfile.pyx>
Then, open the generated with your webbrowser. You'll see how cython have translated your code into C, and then, found your answers :)
As the parameter name self suggests the C foo() function might make more sense as a method in Python:
cdef extern void c_foo "foo" (char* n, MyClass_ptr)
cdef class MyClass:
cdef MyClass_ptr this
def foo(self, unicode n not None):
b = n.encode('ascii')
cdef char* p = b
c_foo(p, self.this)
See also Wrapping C library.
Cython translates your code to C (or C++). At this point you can forget it exists. Generated C code is an ordinary C extension for Python. This code contains ordinary Python C API calls such as PyBytes_AsString() (convert Python bytes to char*) or PyInt_FromLong() (convert C int to Python integer).
MyClass_ptr is used as is:
struct __pyx_obj_3foo_MyClass {
PyObject_HEAD
MyClass_ptr this; /* <-- using MyClass_ptr as is */
};
i.e., pure Python code never sees it so there is no conversion from/to Python objects.