Develop Reference

cython class containing c strings; buffer overrun?

Trying to learn a little Cython, I've been attempting to write a toy library that just holds a few cstrings (corresponding to the available choices for a factoral/categorical data type). The strings being pointed to within the class are being overwritten, and my C/Cython-foo is too minimal to figure out why.
The result is something like this:
>>> import coupla
>>> ff = coupla.CouplaStrings(["one", "two"])
>>> ff
write, two
>>> ff
, two
>>> ff
two, two
Help is greatly appreciated! I feel like I'm going crazy. Just using the to_cstring_array and to_str_list functions seems to work fine, but within the class it goes kaputt.
cdef extern from "Python.h":
char* PyUnicode_AsUTF8(object unicode)
from libc.stdlib cimport malloc, free
cdef char **to_cstring_array(list_str):
"""Stolen from Stackoverflow:
https://stackoverflow.com/questions/17511309/fast-string-array-cython/17511714#17511714
"""
cdef Py_ssize_t num_strs = len(list_str)
cdef char **ret = <char **>malloc(num_strs * sizeof(char *))
for i in range(num_strs):
ret[i] = PyUnicode_AsUTF8(list_str[i])
return ret
cdef to_str_list(char **cstr_array, Py_ssize_t size):
cdef int i
result = []
for i in range(size):
result.append(bytes(cstr_array[i]).decode("utf-8"))
return result
cdef class CouplaStrings:
cdef char **_strings
cdef Py_ssize_t _num_strings
def __init__(self, strings):
cdef Py_ssize_t num_strings = len(strings)
cdef char **tstrings = <char **> to_cstring_array(strings)
self._num_strings = num_strings
self._strings = tstrings
def __repr__(self):
"""Just for testing."""
return ", ".join(to_str_list(self._strings, self._num_strings))
def __dealloc__(self):
free(self._strings)
Edit:
See the answer below by user2357112. An edited version of CouplaStrings seems to avoid that particular problem, though I wouldn't swear on its overall correctness.
Edit 2: THIS IS WRONG IGNORE ONLY KEPT FOR HISTORICAL PURPOSES
cdef class CouplaStrings:
cdef char **_strings
cdef Py_ssize_t _num_strings
def __init__(self, strings):
cdef Py_ssize_t num_strings = len(strings)
cdef char **ret = <char **> PyMem_Malloc(num_strings * sizeof(char *))
for i in range(num_strings):
ret[i] = <char *> PyMem_Realloc(PyUnicode_AsUTF8(strings[i]),
sizeof(char *))
self._num_strings = num_strings
self._strings = ret
def __repr__(self):
"""Just for testing."""
return ", ".join(to_str_list(self._strings, self._num_strings))
def __dealloc__(self):
PyMem_Free(self._strings)

You've failed to account for ownership and memory management.
The UTF-8 encoding returned by PyUnicode_AsUTF8 is owned by the string object PyUnicode_AsUTF8 was called on, and it is reclaimed when that string dies. To prevent the string object from dying before your object does, your object needs to keep a (Python) reference to the string object. Alternatively, you can copy the UTF-8 encodings into memory you allocate yourself, and take responsibility for freeing that memory yourself.
Otherwise, you'll just have an array of dangling pointers.

Passing a struct* from one Cython class to another

I am trying to pass a struct pointer from one Cython class to another. Here is some example code:
cdef struct MyStruct:
int a
int b
cdef class MyClass:
cdef MyStruct* s
def __init__(self):
self.s = <MyStruct*> malloc(sizeof(MyStruct))
self.s.a = 1
self.s.b = 2
cdef MyStruct* get_my_struct(self):
return self.s
cdef class PrinterClass:
cdef object m
def __init__(self):
self.m = MyClass()
cpdef print_struct(self):
cdef MyStruct* my_struct
my_struct = self.m.get_my_struct()
print(my_struct.a)
When I try to compile this class, I get these 2 errors around the my_struct = self.m.get_my_struct() line:
Cannot convert Python object to 'MyStruct *
and
Storing unsafe C derivative of temporary Python reference
Why is Cython attempting to do conversions here? Can't it just pass the pointer as is?

In PrinterClass, replace cdef object m with cdef MyClass m or explicitly cast self.m to MyClass: my_struct = (<MyClass>self.m).get_my_struct(). (In addition, a __dealloc__ should be added to MyClass).
I guess the difference lies in that object is a python object(in essence,dict), while cdef class is another kind of class(in essence, struct), see Extension types (aka. cdef classes).
Expect further revelations from other experts :)

How to structure a cython module with many identical function signatures

I've written a Cython module which wraps a foreign C function, and it's working as expected. However, I'd like to wrap the rest of the functions provided by my C binary, which have identical signatures. In Python, I could just do:
def boilerplate(func):
def wrapped_f(c, d):
# modify x and y, producing mod_c and mod_d
result = func(mod_c, mod_d)
# modify foreign function return values, producing final_c, final_d
return final_c, final_d
return wrapped_f
#boilerplate
def func_a(a, b)
return _foreign_func_a(a, b)
#boilerplate
def func_b(a, b)
return _foreign_func_b(a, b)
Is there a similar pattern I can use in Cython, in order to "cythonise" wrapped_f, assuming _foreign_func_a and its accompanying structs etc. have been cimported?
However, when I move the generic operations into the decorator:
def boilerplate(func):
def wrapped(double[::1] wlon, double[::1] wlat):
cdef _FFIArray x_ffi, y_ffi
x_ffi.data = <void*>&wlon[0]
x_ffi.len = wlon.shape[0]
y_ffi.data = <void*>&wlat[0]
y_ffi.len = wlat.shape[0]
cdef _Result_Tuple result = func(x_ffi, y_ffi)
cdef double* eastings_ptr = <double*>(result.e.data)
cdef double* northings_ptr = <double*>(result.n.data)
cdef double[::1] e = <double[:result.e.len:1]>eastings_ptr
cdef double[::1] n = <double[:result.n.len:1]>northings_ptr
e_numpy = np.copy(e)
n_numpy = np.copy(n)
drop_float_array(result.e, result.n)
return e_numpy, n_numpy
return wrapped
#boilerplate
def convert_bng(double[::1] lons, double[::1] lats):
"""wrapper around threaded conversion function
"""
return convert_to_bng_threaded(lons, lats)
I get errors when
trying to convert x_ffi and y_ffi to _FFIArray to Python objects in wrapped
converting Python object func to _Result_Tuple in wrapped
converting lons and lats to _FFI_Array in convert_to_bng_threaded, and
converting _Result_Tuple back to a Python object in convert_bng_threaded

Your essential problem (based on your updated question) is that you're trying to wrap a function that takes pure C data types (and thus can only be defined as a cdef function, and can be called from Cython but not Python). However, decorators work on Python functions, so it doesn't quite come together.
Fortunately you can do something very similar handling the wrapped function a using C function pointer. You need a slightly different syntax but the idea is very much the same. (For the sake of this answer I'm assuming you are using the definitions of C data types from this previous question, which I think is reasonable)
# pass a function pointer in
cdef boilerplate(_Result_Tuple (*func)(_FFIArray, _FFIArray)):
def wrapped(double[::1] wlon, double[::1] wlat):
cdef _FFIArray x_ffi, y_ffi
x_ffi.data = <void*>&wlon[0]
x_ffi.len = wlon.shape[0]
y_ffi.data = <void*>&wlat[0]
y_ffi.len = wlat.shape[0]
cdef _Result_Tuple result = func(x_ffi, y_ffi)
cdef double* eastings_ptr = <double*>(result.e.data)
cdef double* northings_ptr = <double*>(result.n.data)
cdef double[::1] e = <double[:result.e.len:1]>eastings_ptr
cdef double[::1] n = <double[:result.n.len:1]>northings_ptr
e_numpy = np.copy(e)
n_numpy = np.copy(n)
drop_float_array(result.e, result.n)
return e_numpy, n_numpy
return wrapped
# do this instead of using a decorator syntax
convert_bng = boilerplate(&convert_to_bng_threaded)

Passing a function as a class attribute of a Cython class

I'm trying to define a Cython class that accepts a function as one of the class attribute in the __init__ function.
I followed this question and tried the following
ctypedef int (*f_type)(int, int)
cdef class ClassWFunction:
cdef f_type f
def __init__( self, f_type f):
self.f = f
def doStuff(self, int x, int y):
return self.f(x, y)
cdef int example_f(int a, int b):
return a*b
classInstance = ClassWFunction(example_f)
classInstance.doStuff(2, 4)
which gives me the error:
Cannot convert 'int (int, int)' to Python object
I also read this page from Cython's documentation and tried following that approach:
cdef class f_type:
cpdef int evaluate(self, int a, int b) except *:
return 0
cdef class ClassWFunctionV2:
cdef f_type f
def __init__( self, f_type f):
self.f = f
def doStuff(self, int a, int b):
return self.f.evaluate(a, b)
cdef class example_f(f_type):
cpdef int evaluate(self, int a, int b) except *:
return a*b
classInstance = ClassWFunctionV2(example_f)
classInstance.doStuff(3, 4)
which gives me a TypeError:
TypeError: Argument 'f' has incorrect type (expected _cython_magic_9f51dc40b83b28506fce9fb260a84618.f_type, got type)
Hopefully there's a way to make my first attempt work, since this second approach has a lot of boilerplate I don't quite understand!
Thanks for your time...
-------- Edit ----------
The point of writing the code this way is twofold:
1) to have a flexible way to change the f function when instantiating the class.
2) The f function gets reused in many of the class methods, so I'd like to assign it to self
In my pure python code, I do stuff like
def f1(x): return x**2
def f2(x): return x**3
cl1 = ClassWFunction(f1)
cl2 = ClassWFunction(f2)
and then proceed to do things with those classes. However, I'm not 100% sure this is the best way to go around, so feel free to suggest a different approach.
-------- Edit2 ----------
As a less flexible but (hopefully!) easier alternative, I tried to hard-code the function onto the class. This would fit objective (2) above, even if it doesn't fit objective (1).
Consider (this also gives an error)
ctypedef int (*f_type)(int)
cdef class ClassWFunction:
cdef f_type example_f
cdef double x
def __init__( self, int x):
self.x = x
# Tried with a python function too
cdef int example_f(int z):
return x*z
self.example_f = example_f
def doStuff(self, int x, int y):
return self.example_f(x)*y
classInstance = ClassWFunction(3)
classInstance.doStuff(2, 4)

How do I create a fixed-length, mutable array of Python objects in Cython?

I need to have an array of python objects to be used in creating a trie datastructure. I need a structure that will be fixed-length like a tuple and mutable like a list. I don't want to use a list because I want to be able to ensure that the list is exactly the right size (if it starts allocating extra elements, the memory overhead could add up very quickly as the trie grows larger). Is there a way to do this? I tried creating an array of objects:
cdef class TrieNode:
cdef object members[32]
...but that gave an error:
Error compiling Cython file:
------------------------------------------------------------
...
cdef class TrieNode:
cdef object members[32]
^
------------------------------------------------------------
/Users/jason/src/pysistence/source/pysistence/trie.pyx:2:23: Array element cannot be a Python object
What is the best way to do what I'm trying to do?

I don't know about the best solution, but here's a solution:
from cpython.ref cimport PyObject, Py_XINCREF, Py_XDECREF
DEF SIZE = 32
cdef class TrieNode:
cdef PyObject *members[SIZE]
def __cinit__(self):
cdef object temp_object
for i in range(SIZE):
temp_object = int(i)
# increment its refcount so it's not gc'd.
# We hold a reference to the object and are responsible for
# decref-ing it in __dealloc__.
Py_XINCREF(<PyObject*>temp_object)
self.members[i] = <PyObject*>temp_object
def __init__(self):
# just to show that it works...
for i in range(SIZE):
print <object>self.members[i]
def __dealloc__(self):
# make sure we decref the members elements.
for i in range(SIZE):
Py_XDECREF(self.members[i])
self.members[i] = NULL
A Cython object is an automatically refcounted PyObject *. You can always roll your own arrays of PyObject *'s as long as you take responsibility for refcounting the little buggers. This can be a major headache for non-trivial cases.

If you only need few fixed sizes of such a structure, I'd look at making classes with uniformly named __slots__, including one size slot to store the size. You'll need to declare a separate class for each size (number of slots). Define a cdecl function to access slots by index. Access performance will probably be not as great as with plain address arithmetics of a C array, but you'll be sure that there's only so many slots and none more.

How about this?
class TrieNode():
def __init__(self, length = 32):
self.members = list()
self.length = length
for i in range(length):
self.members.append(None)
def set(self, idx, item):
if idx < self.length and idx >= 0:
self.members[idx] = item
else:
print "ERROR: Specified index out of range."
# Alternately, you could raise an IndexError.
def unset(self, idx):
if idx < self.length and idx >= 0:
self.members[idx] = None
else:
raise IndexError("Specified index out of range (0..%d)." % self.length)