I am trying to pass a struct pointer from one Cython class to another. Here is some example code:
cdef struct MyStruct:
int a
int b
cdef class MyClass:
cdef MyStruct* s
def __init__(self):
self.s = <MyStruct*> malloc(sizeof(MyStruct))
self.s.a = 1
self.s.b = 2
cdef MyStruct* get_my_struct(self):
return self.s
cdef class PrinterClass:
cdef object m
def __init__(self):
self.m = MyClass()
cpdef print_struct(self):
cdef MyStruct* my_struct
my_struct = self.m.get_my_struct()
print(my_struct.a)
When I try to compile this class, I get these 2 errors around the my_struct = self.m.get_my_struct() line:
Cannot convert Python object to 'MyStruct *
and
Storing unsafe C derivative of temporary Python reference
Why is Cython attempting to do conversions here? Can't it just pass the pointer as is?
In PrinterClass, replace cdef object m with cdef MyClass m or explicitly cast self.m to MyClass: my_struct = (<MyClass>self.m).get_my_struct(). (In addition, a __dealloc__ should be added to MyClass).
I guess the difference lies in that object is a python object(in essence,dict), while cdef class is another kind of class(in essence, struct), see Extension types (aka. cdef classes).
Expect further revelations from other experts :)
Related
I am interested in subclassing an existing Cython class (we'll call it A), which has say the following __cinit__(self, int a, int b, *argv) function signature.
My new class B would have the following __cinit__(self, int a, int c, *argv), where b is no longer required, or used.
I want something along the lines of:
cdef class A:
cdef int a
cdef int b
def __cinit__(self, int a, int b, *argv):
self.a = a
self.b = b
cdef class B(A):
cdef double c
def __cinit__(self, int a, double c, *argv):
self.a = a
self.c = c
Is there a way to do this?
No. It's not possible to do what you want.
The point of using __cinit__ rather than __init__ is that __cinit__ is always called. The way that Cython ensures this is that it takes control away from you by arranging the calls to the base classes itself.
If possible it'd be better to use __init__ instead of __cinit__ unless you need to "guarantee to be called" since this gives more flexibility.
Your best option is probably to use a staticmethod factory function for class B. But you do lose the guaranteed call of B.__cinit__.
Trying to learn a little Cython, I've been attempting to write a toy library that just holds a few cstrings (corresponding to the available choices for a factoral/categorical data type). The strings being pointed to within the class are being overwritten, and my C/Cython-foo is too minimal to figure out why.
The result is something like this:
>>> import coupla
>>> ff = coupla.CouplaStrings(["one", "two"])
>>> ff
write, two
>>> ff
, two
>>> ff
two, two
Help is greatly appreciated! I feel like I'm going crazy. Just using the to_cstring_array and to_str_list functions seems to work fine, but within the class it goes kaputt.
cdef extern from "Python.h":
char* PyUnicode_AsUTF8(object unicode)
from libc.stdlib cimport malloc, free
cdef char **to_cstring_array(list_str):
"""Stolen from Stackoverflow:
https://stackoverflow.com/questions/17511309/fast-string-array-cython/17511714#17511714
"""
cdef Py_ssize_t num_strs = len(list_str)
cdef char **ret = <char **>malloc(num_strs * sizeof(char *))
for i in range(num_strs):
ret[i] = PyUnicode_AsUTF8(list_str[i])
return ret
cdef to_str_list(char **cstr_array, Py_ssize_t size):
cdef int i
result = []
for i in range(size):
result.append(bytes(cstr_array[i]).decode("utf-8"))
return result
cdef class CouplaStrings:
cdef char **_strings
cdef Py_ssize_t _num_strings
def __init__(self, strings):
cdef Py_ssize_t num_strings = len(strings)
cdef char **tstrings = <char **> to_cstring_array(strings)
self._num_strings = num_strings
self._strings = tstrings
def __repr__(self):
"""Just for testing."""
return ", ".join(to_str_list(self._strings, self._num_strings))
def __dealloc__(self):
free(self._strings)
Edit:
See the answer below by user2357112. An edited version of CouplaStrings seems to avoid that particular problem, though I wouldn't swear on its overall correctness.
Edit 2: THIS IS WRONG IGNORE ONLY KEPT FOR HISTORICAL PURPOSES
cdef class CouplaStrings:
cdef char **_strings
cdef Py_ssize_t _num_strings
def __init__(self, strings):
cdef Py_ssize_t num_strings = len(strings)
cdef char **ret = <char **> PyMem_Malloc(num_strings * sizeof(char *))
for i in range(num_strings):
ret[i] = <char *> PyMem_Realloc(PyUnicode_AsUTF8(strings[i]),
sizeof(char *))
self._num_strings = num_strings
self._strings = ret
def __repr__(self):
"""Just for testing."""
return ", ".join(to_str_list(self._strings, self._num_strings))
def __dealloc__(self):
PyMem_Free(self._strings)
You've failed to account for ownership and memory management.
The UTF-8 encoding returned by PyUnicode_AsUTF8 is owned by the string object PyUnicode_AsUTF8 was called on, and it is reclaimed when that string dies. To prevent the string object from dying before your object does, your object needs to keep a (Python) reference to the string object. Alternatively, you can copy the UTF-8 encodings into memory you allocate yourself, and take responsibility for freeing that memory yourself.
Otherwise, you'll just have an array of dangling pointers.
I've written a Cython module which wraps a foreign C function, and it's working as expected. However, I'd like to wrap the rest of the functions provided by my C binary, which have identical signatures. In Python, I could just do:
def boilerplate(func):
def wrapped_f(c, d):
# modify x and y, producing mod_c and mod_d
result = func(mod_c, mod_d)
# modify foreign function return values, producing final_c, final_d
return final_c, final_d
return wrapped_f
#boilerplate
def func_a(a, b)
return _foreign_func_a(a, b)
#boilerplate
def func_b(a, b)
return _foreign_func_b(a, b)
Is there a similar pattern I can use in Cython, in order to "cythonise" wrapped_f, assuming _foreign_func_a and its accompanying structs etc. have been cimported?
However, when I move the generic operations into the decorator:
def boilerplate(func):
def wrapped(double[::1] wlon, double[::1] wlat):
cdef _FFIArray x_ffi, y_ffi
x_ffi.data = <void*>&wlon[0]
x_ffi.len = wlon.shape[0]
y_ffi.data = <void*>&wlat[0]
y_ffi.len = wlat.shape[0]
cdef _Result_Tuple result = func(x_ffi, y_ffi)
cdef double* eastings_ptr = <double*>(result.e.data)
cdef double* northings_ptr = <double*>(result.n.data)
cdef double[::1] e = <double[:result.e.len:1]>eastings_ptr
cdef double[::1] n = <double[:result.n.len:1]>northings_ptr
e_numpy = np.copy(e)
n_numpy = np.copy(n)
drop_float_array(result.e, result.n)
return e_numpy, n_numpy
return wrapped
#boilerplate
def convert_bng(double[::1] lons, double[::1] lats):
"""wrapper around threaded conversion function
"""
return convert_to_bng_threaded(lons, lats)
I get errors when
trying to convert x_ffi and y_ffi to _FFIArray to Python objects in wrapped
converting Python object func to _Result_Tuple in wrapped
converting lons and lats to _FFI_Array in convert_to_bng_threaded, and
converting _Result_Tuple back to a Python object in convert_bng_threaded
Your essential problem (based on your updated question) is that you're trying to wrap a function that takes pure C data types (and thus can only be defined as a cdef function, and can be called from Cython but not Python). However, decorators work on Python functions, so it doesn't quite come together.
Fortunately you can do something very similar handling the wrapped function a using C function pointer. You need a slightly different syntax but the idea is very much the same. (For the sake of this answer I'm assuming you are using the definitions of C data types from this previous question, which I think is reasonable)
# pass a function pointer in
cdef boilerplate(_Result_Tuple (*func)(_FFIArray, _FFIArray)):
def wrapped(double[::1] wlon, double[::1] wlat):
cdef _FFIArray x_ffi, y_ffi
x_ffi.data = <void*>&wlon[0]
x_ffi.len = wlon.shape[0]
y_ffi.data = <void*>&wlat[0]
y_ffi.len = wlat.shape[0]
cdef _Result_Tuple result = func(x_ffi, y_ffi)
cdef double* eastings_ptr = <double*>(result.e.data)
cdef double* northings_ptr = <double*>(result.n.data)
cdef double[::1] e = <double[:result.e.len:1]>eastings_ptr
cdef double[::1] n = <double[:result.n.len:1]>northings_ptr
e_numpy = np.copy(e)
n_numpy = np.copy(n)
drop_float_array(result.e, result.n)
return e_numpy, n_numpy
return wrapped
# do this instead of using a decorator syntax
convert_bng = boilerplate(&convert_to_bng_threaded)
I'm trying to define a Cython class that accepts a function as one of the class attribute in the __init__ function.
I followed this question and tried the following
ctypedef int (*f_type)(int, int)
cdef class ClassWFunction:
cdef f_type f
def __init__( self, f_type f):
self.f = f
def doStuff(self, int x, int y):
return self.f(x, y)
cdef int example_f(int a, int b):
return a*b
classInstance = ClassWFunction(example_f)
classInstance.doStuff(2, 4)
which gives me the error:
Cannot convert 'int (int, int)' to Python object
I also read this page from Cython's documentation and tried following that approach:
cdef class f_type:
cpdef int evaluate(self, int a, int b) except *:
return 0
cdef class ClassWFunctionV2:
cdef f_type f
def __init__( self, f_type f):
self.f = f
def doStuff(self, int a, int b):
return self.f.evaluate(a, b)
cdef class example_f(f_type):
cpdef int evaluate(self, int a, int b) except *:
return a*b
classInstance = ClassWFunctionV2(example_f)
classInstance.doStuff(3, 4)
which gives me a TypeError:
TypeError: Argument 'f' has incorrect type (expected _cython_magic_9f51dc40b83b28506fce9fb260a84618.f_type, got type)
Hopefully there's a way to make my first attempt work, since this second approach has a lot of boilerplate I don't quite understand!
Thanks for your time...
-------- Edit ----------
The point of writing the code this way is twofold:
1) to have a flexible way to change the f function when instantiating the class.
2) The f function gets reused in many of the class methods, so I'd like to assign it to self
In my pure python code, I do stuff like
def f1(x): return x**2
def f2(x): return x**3
cl1 = ClassWFunction(f1)
cl2 = ClassWFunction(f2)
and then proceed to do things with those classes. However, I'm not 100% sure this is the best way to go around, so feel free to suggest a different approach.
-------- Edit2 ----------
As a less flexible but (hopefully!) easier alternative, I tried to hard-code the function onto the class. This would fit objective (2) above, even if it doesn't fit objective (1).
Consider (this also gives an error)
ctypedef int (*f_type)(int)
cdef class ClassWFunction:
cdef f_type example_f
cdef double x
def __init__( self, int x):
self.x = x
# Tried with a python function too
cdef int example_f(int z):
return x*z
self.example_f = example_f
def doStuff(self, int x, int y):
return self.example_f(x)*y
classInstance = ClassWFunction(3)
classInstance.doStuff(2, 4)
With C++11 I have been using the following pattern for implementing a graph data structure with parallel iterators. Nodes are just indices, edges are entries in an adjacency data structure. For iterating over all nodes, a function (lambda, closure...) is passed to a parallelForNodes method and called with each node as an argument. Iteration details are nicely encapsulated in the method.
Now I would like to try the same concept with Cython. Cython provides the cython.parallel.prange function which uses OpenMP for parallelizing a loop over a range. For parallelism to work, Python's Global Interpreter Lock needs to be deactivated with the nogil=True parameter. Without the GIL, using Python objects is not allowed, which makes this tricky.
Is it possible to use this approach with Cython?
class Graph:
def __init__(self, n=0):
self.n = n
self.m = 0
self.z = n # max node id
self.adja = [[] for i in range(self.z)]
self.deg = [0 for i in range(self.z)]
def forNodes(self, handle):
for u in range(self.z):
handle(u)
def parallelForNodes(self, handle):
# first attempt which will fail...
for u in prange(self.z, nogil=True):
handle(u)
# usage
def initialize(u):
nonlocal ls
ls[u] = 1
G.parallelForNodes(initialize)
Firstly, things cannot be Python objects without the GIL.
from cython.parallel import prange
cdef class Graph:
cdef int n, m, z
def __cinit__(self, int n=0):
self.z = n # max node id
cdef void parallelForNodes(self, void (*handle)(int) nogil) nogil:
cdef int u
for u in prange(self.z, nogil=True):
handle(u)
The biggest catch there is that our function pointer was also nogil.
parallelForNodes does not have to be nogil itself, but there's no reason for it not to be.
Then we need a C function to call:
cdef int[100] ls
cdef void initialize(int u) nogil:
global ls
ls[u] = 1
and it just works!
Graph(100).parallelForNodes(initialize)
# Print it!
cdef int[:] ls_ = ls
print(list(ls_))