I need to convert a Python list of ints to vector[int] in a cdef function to call another C function. I tried this:
cdef pylist_to_handles(hs):
cdef vector[int] o_vect
for h in hs:
o_vect.push_back(h)
return o_vect
This should work because I only need to call this from other cdef functions, but I get this error:
Cannot convert 'vector<int>' to Python object
What am I doing wrong ?
In Cython 0.17 using libcpp.vector, you can do this:
cdef vector[int] vect = hs
What you really have is this:
cdef object pylist_to_handles(hs):
...
return <object>o_vect
If you do not explicitily set a type, it is assumed to be a python object ("object" in code). As you see in the code, you're trying to cast vector[int] to an object, but Cython does not know how to handle that.
Just add a return type in cdef:
cdef vector[int] pylist_to_handles(hs):
Related
We have cython code built on top of an existing C library -- very simplified example here just to show the problem. The somecapi_t structure is opaque with an allocate function, a free function, and several functions to manipulate the structure.
data.pxd:
cdef extern from "<some_c_api.h>" nogil:
ctypedef struct some_c_api_t:
pass
some_c_api_t *some_c_api_alloc()
void some_c_api_free(some_c_api_t *scap)
int some_c_api_get_count(const some_c_api_t *scap)
Class Data is the python representation of the C structure. Class Data uses a helper class, Helper, to implement _getitem_, _setitem_, and other operations on part of the C type in a more pythonic way than the C API provides. Here, all we show is _len_. For sake of example, we also have an equivalent function, get_count that tries to access the "scap" pointer from Data.
data.pyx
from cpython.exc cimport PyErr_SetFromErrno
cdef class Helper:
cdef some_c_api_t *scap
def __cinit__(self, data, some_c_api_t *scap):
self.data = data
self.scap = scap
def __len__(self):
return some_c_api_get_count(self.scap)
def get_count(self):
return some_c_api_get_count(self.data.scap)
cdef class Data:
cdef some_c_api_t *scap
def __cinit__(self):
self.scap = some_c_api_alloc()
if self.scap == NULL:
PyErr_SetFromErrno(OSError)
def __dealloc__(self):
some_c_api_free(self.scap)
#property
def values(self):
return Helper(self, self.scap)
When compiled, we get two errors. First, the call to Helper's constructor is trying to convert "scap" to a python object even though it's declared as the C type in Helper's constructor.
#property
def values(self):
return Helper(self, self.scap)
^
------------------------------------------------------------
vna/data.pyx:30:32: Cannot convert 'some_c_api_t *' to Python object
Similarly, when get_count tries to get the scap pointer from Data, it thinks it's a python object.
def get_count(self):
return some_c_api_get_count(self.data.scap)
^
------------------------------------------------------------
vna/data.pyx:14:45: Cannot convert Python object to 'const some_c_api_t *'
Another approach we tried was to define a "get_scap" method in Data:
cdef some_c_api_t *scap(self):
return self.scap
and call it from Helper. In every case, however, attempts to pass the C pointer to Helper fails, insisting that the API between these two "cdef" classes must consist only of Python objects.
Is it possible for the Helper class to get access the C pointer?
I'm trying to pass an np.ndarray from Python to instantiate a Cython class. However, I can't work out how to do it for an any-dimensional array. I'd like my .pyx interface to look like:
wrapper.pyx:
import numpy as np
cimport numpy as np
cdef extern from "myClass.h":
cdef cppclass C_myClass "myClass":
void myClass(np.float32_t*, int*, int)
cdef class array:
cdef C_myClass* cython_class
cdef int shape[8] # max number of dimensions = 8
cdef int ndim
def __init__(self, np.ndarray[dtype=np.float32_t] numpy_array):
self.ndim = numpy_array.ndim
for dim in range(self.ndim):
self.shape[dim] = numpy_array.shape[dim]
self.cython_class = new C_myClass(&numpy_array[0], &self.shape[0], self.ndim)
def __del__(self):
del self.cython_class
Such that the class constructor can look like:
myClass.h:
myClass(float* array_, int* shape_, int ndim_);
Do any of you know how to handle an array of any dimensions within Cython, while still being able to get the array shape parameters (I don't want the user to have to flatten the array or pass in the array shapes themselves)?
There isn't a direct Cython type for "numpy array with any number of dimensions" (or indeed "typed memoryview with any number of dimensions").
Therefore, I suggest leaving the array argument to the constructor untyped and using the buffer protocol yourself.
The following code is untested, but should give you the right idea
from cpython.buffer cimport Py_buffer, PyObject_GetBuffer, PyBUF_ND, PyBUF_FORMAT, PyBuffer_Release
...
def __init__(self, array):
cdef Py_buffer buf
# shape and ndim can be passed as before - they don't
# need the array to be typed
self.ndim = numpy_array.ndim
for dim in range(self.ndim):
self.shape[dim] = numpy_array.shape[dim]
# Note that PyBUF_ND requires a C contiguous array. This looks
# to be an implicit requirement of your C++ class that you
# probably don't realize you have
PyObject_GetBuffer(array, buf, PyBUF_ND | PyBUF_FORMAT)
if not buf.format == "f":
PyBuffer_Release(buf)
raise TypeError("Numpy array must have dtype float32_t")
self.cython_class = new C_myClass(<float32_t*>buf.buf, &self.shape[0], self.ndim)
You then need to make sure you release the buffer with ByBuffer_Release.
It isn't clear whether C_myClass copies the data or holds a pointer to it.
if C_myClass copies the data then call PyBuffer_release straight away after constructing self.cython_class
if C_myClass holds a reference to the data then call PyBuffer_release in the destructor after self.cython_class is deleted.
Assume we have a cython class A with a pointer to float like in
# A.pyx
cdef class A:
cdef float * ptr
We also have a cython class B in another module which needs access to the data under ptr:
# B.pyx
cdef class B:
cdef float * f_ptr
cpdef submit(self, ptr_var):
self.f_ptr= get_from( ptr_var ) # ???
The corresponding Python code using A and B might be something like
from A import A
from B import B
a = A()
b = B()
ptr = a.get_ptr()
b.submit(ptr)
How can we define get_ptr() and what would we use for get_from in B?
The solution is to wrap the pointer variable into a Python object. Module libc.stdint offers a type named uintptr_t which is an integer large enough for storing any kind of pointer. With this the solution might look as follows.
from libc.stdint cimport uintptr_t
cdef class A:
cdef float * ptr
def get_ptr(self):
return <uintptr_t>self.ptr
The expression in angle brackets <uintptr_t> corresponds to a cast to uintptr_t. In class B we then have to cast the variable back to a pointer to float.
from libc.stdint cimport uintptr_t
cdef class B:
cdef float * f_ptr
cpdef submit(self, uintptr_t ptr_var):
self.f_ptr= <float *>ptr_var
This works for any kind of pointers not only for pointers to float. One has to make sure that both modules (A and B) deal with the same kind of pointer since that information is lost once the pointer is wrapped in a uintptr_t.
main.h
ifndef MAIN_H
define MAIN_H
ifdef __cplusplus
extern "C" {
endif
typedef struct Pythonout{
int pn;
double *px;
}Pythonout;
struct Pythonout l1tf_main(char *ifile_y, double lambda, int rflag);
ifdef __cplusplus
}
endif
endif /* MAIN_H */
Following is the Cython pyx file using main.h
.pyx
cimport numpy as np
cdef extern from "main.h":
ctypedef struct Pythonout:
int n
double *x
cdef Pythonout l1tf_main(char *ifile_y
,double lambdaval,
int rflag);
cdef class Pyclass:
cdef Pythonout pnx
def __cinit__(self, char *pfilename,
lambdaval, rflag):
self.pnx = l1tf_main(pfilename,
lambdaval, rflag)
#property
def n(self):
return self.pnx.n
#property
def x(self):
cdef np.npy_float64 shape[1]
shape[0] = <np.npy_intp> self.pnx.n
ndarray =
np.PyArray_SimpleNewFromData(1,
&(self.pnx.n),np.NPY_FLOAT64,
<void *> self.pnx.x)
np.PyArray_UpdateFlags(ndarray,
ndarray.flags.num
| np.NPY_OWNDATA)
return ndarray
cpdef filtered_trend(char *pfilename, double
lambdaval, int rflag):
pnx = Pyclass(pfilename, lambdaval, rflag)
return pnx.x
In the class I am getting the following error while compiling:
‘Pythonout {aka struct Pythonout}’ has no member named ‘n’
‘Pythonout {aka struct Pythonout}’ has no member named ‘x’
When calling the object value pnx.n and pnx.x.
There's at least two issues with your code.
The trivial issue causing the compilation errors: in C you call the struct attributes px and pn, while in Cython you call them x and n. This means that the code Cython generates does not match the C header. Make these consistent.
np.PyArray_UpdateFlags(ndarray,
ndarray.flags.num
| np.NPY_OWNDATA)
This tells Numpy that in now owns the data in x and is responsible for deallocating it. However suppose you have the Python code:
x1 = PyClassInstance.x
x2 = PyClassInstance.x
You now have two Numpy arrays that each believe the own the same data and will both try to deallocate it. (Similarly if you never access x then pnx.x is never deallocated) What you should probably do is have the PyClass instance be responsible for deallocating its pnx.x (in a __dealloc__ function). Then in your x property do:
ndarray = PyArray_SimpleNewFromData(...)
Py_INCREF(self) # SetBaseObject doesn't do this so you must do it manually
PyArray_SetBaseObject(ndarray, self) # check return value for errors...
Now the Numpy array treats the PyClass instance as owning the data.
It seems that there is currently a pull request in Cython's repo to wrap c++ std::array but until then, I could use some help. I am currently wrapping the std::array like so:
cdef extern from "<array>" namespace "std" nogil:
cdef cppclass array2 "std::array<double, 2>":
array2() except+
double& operator[](size_t)
This works, but I have to loop over a cython memory view say, double arr[:], and copy the values one by one. Is there an easier way to do this? Essentially I'd like to do the following:
cdef double arr[2]
arr[0] = 1.0
arr[1] = 2.0
cdef array2 array2_arr = arr
#and the reverse
cdef array2 reverse
reverse[0] = 1.0
reverse[1] = 2.0
cdef double reverse_arr[2] = reverse
Is this completely unreasonable? As a result, working with std::array is extremely tedious because I have a to have a for-loop to copy values from cython to c++ and from c++ to cython. Furthermore, since cython doesn't give us the ability to have non-type template parameters, I have to define a wrapper for every variation of std::array in my code. Any suggestions on how to efficiently work with std::array would be greatly appreciated.
edit:
I'm now able to go from a memory view to an array2 type using the following:
def __cinit__(self, double[:] mem):
cdef array2 *arr = <array2 *>(&mem[0])
But it seems that no matter what I do, I cannot get cython to convert an array2 type to a memoryview:
cdef array2 arr = self.thisptr.getArray()
# error: '__pyx_t_1' declared as a pointer to a reference of type 'double &'
cdef double[::1] mview = <double[:2]>(&arr[0])
#OR
# Stop must be provided to indicate shape
cdef double[::1] mview = <double[::2]>(&arr[0])
Please help me figure out how to cast a C++ pointer to a memory view. Every combination I have tried to date has resulted in some kind of casting error.
EDIT:
I found that I am to perform the following syntax with no error using a newer version of Cython (I was using Cythong 0.22) and upgraded to 0.23.5.
cdef double[::1] mview = <double[:4]>(&arr[0])
However, if I attempt to return mview from the function I am using it in, I get garbage memory. Returning the memoryview to the pointer of my array loses scope and therefore destructs my array automatically. As soon as I figure out how to properly return my array, I'll attempt to update the official answer.
After much fiddling, I found the answer to my question.
Definition of array and class that uses array:
cdef extern from "<array>" namespace "std" nogil:
cdef cppclass array4 "std::array<int, 4>":
array4() except+
int& operator[](size_t)
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass ArrayFun:
ArrayFun(array4&)
array4 getArray()
Python implementation
cdef class PyArrayFun:
cdef ArrayFun *thisptr # hold a C++ instance which we're wrapping
def __cinit__(self, int[:] mem):
#
# Conversion from memoryview to std::array<int,4>
#
cdef array4 *arr = <array4 *>(&mem[0])
self.thisptr = new ArrayFun(arr[0])
def getArray(self):
cdef array4 arr = self.thisptr.getArray()
#
# Conversion from std::array<int, 4> to memoryview
#
cdef int[::1] mview = <int[:4]>(&arr[0])
cdef int[::1] new_view = mview.copy()
for i in range(0,4):
print ("arr is ", arr[i])
print("new_view is ", new_view[i])
# A COPY MUST be returned because arr goes out of scope and is
# default destructed when this function exist. Therefore we have to
# copy again. This kinda of sucks because we have to copy the
# internal array out from C++, and then we have to copy the array
# out from Python, therefore 2 copies for one array.
return mview.copy()