I'm trying to pass an np.ndarray from Python to instantiate a Cython class. However, I can't work out how to do it for an any-dimensional array. I'd like my .pyx interface to look like:
wrapper.pyx:
import numpy as np
cimport numpy as np
cdef extern from "myClass.h":
cdef cppclass C_myClass "myClass":
void myClass(np.float32_t*, int*, int)
cdef class array:
cdef C_myClass* cython_class
cdef int shape[8] # max number of dimensions = 8
cdef int ndim
def __init__(self, np.ndarray[dtype=np.float32_t] numpy_array):
self.ndim = numpy_array.ndim
for dim in range(self.ndim):
self.shape[dim] = numpy_array.shape[dim]
self.cython_class = new C_myClass(&numpy_array[0], &self.shape[0], self.ndim)
def __del__(self):
del self.cython_class
Such that the class constructor can look like:
myClass.h:
myClass(float* array_, int* shape_, int ndim_);
Do any of you know how to handle an array of any dimensions within Cython, while still being able to get the array shape parameters (I don't want the user to have to flatten the array or pass in the array shapes themselves)?
There isn't a direct Cython type for "numpy array with any number of dimensions" (or indeed "typed memoryview with any number of dimensions").
Therefore, I suggest leaving the array argument to the constructor untyped and using the buffer protocol yourself.
The following code is untested, but should give you the right idea
from cpython.buffer cimport Py_buffer, PyObject_GetBuffer, PyBUF_ND, PyBUF_FORMAT, PyBuffer_Release
...
def __init__(self, array):
cdef Py_buffer buf
# shape and ndim can be passed as before - they don't
# need the array to be typed
self.ndim = numpy_array.ndim
for dim in range(self.ndim):
self.shape[dim] = numpy_array.shape[dim]
# Note that PyBUF_ND requires a C contiguous array. This looks
# to be an implicit requirement of your C++ class that you
# probably don't realize you have
PyObject_GetBuffer(array, buf, PyBUF_ND | PyBUF_FORMAT)
if not buf.format == "f":
PyBuffer_Release(buf)
raise TypeError("Numpy array must have dtype float32_t")
self.cython_class = new C_myClass(<float32_t*>buf.buf, &self.shape[0], self.ndim)
You then need to make sure you release the buffer with ByBuffer_Release.
It isn't clear whether C_myClass copies the data or holds a pointer to it.
if C_myClass copies the data then call PyBuffer_release straight away after constructing self.cython_class
if C_myClass holds a reference to the data then call PyBuffer_release in the destructor after self.cython_class is deleted.
Related
Assume we have a cython class A with a pointer to float like in
# A.pyx
cdef class A:
cdef float * ptr
We also have a cython class B in another module which needs access to the data under ptr:
# B.pyx
cdef class B:
cdef float * f_ptr
cpdef submit(self, ptr_var):
self.f_ptr= get_from( ptr_var ) # ???
The corresponding Python code using A and B might be something like
from A import A
from B import B
a = A()
b = B()
ptr = a.get_ptr()
b.submit(ptr)
How can we define get_ptr() and what would we use for get_from in B?
The solution is to wrap the pointer variable into a Python object. Module libc.stdint offers a type named uintptr_t which is an integer large enough for storing any kind of pointer. With this the solution might look as follows.
from libc.stdint cimport uintptr_t
cdef class A:
cdef float * ptr
def get_ptr(self):
return <uintptr_t>self.ptr
The expression in angle brackets <uintptr_t> corresponds to a cast to uintptr_t. In class B we then have to cast the variable back to a pointer to float.
from libc.stdint cimport uintptr_t
cdef class B:
cdef float * f_ptr
cpdef submit(self, uintptr_t ptr_var):
self.f_ptr= <float *>ptr_var
This works for any kind of pointers not only for pointers to float. One has to make sure that both modules (A and B) deal with the same kind of pointer since that information is lost once the pointer is wrapped in a uintptr_t.
I'm trying to call to a C function that accept a pointer to pointer and redirect it to a internally-allocated 1D array, something along that line:
typedef myStruct {
const char* name;
int status;
} myStruct;
int foo(const myStruct** array, size_t* size);
I'm trying to wrap it with Python and NumPy, and get a NumPy array that wraps that internal memory (as read-only).
I've got the following:
arr = np.ctypeslib.ndpointer(dtype=np.dtype([('name', np.intp),
('status', np.int)]))()
size = ct.c_size_t()
_call('foo', ct.byref(arr), ct.byref(size))
arr = np.ctypeslib.as_array(arr, shape=(size.value,))
arr.flags.writeable = False # not our memory!
Where _call() is a wrapper that check for the returned value.
I'm getting the following:
ValueError: '<P' is not a valid PEP 3118 buffer format string
over the call to as_array(). What am I doing wrong?
EDIT: My goal here is to read the data as NumPy structured array, as I think it's the best way to describe a C array of structs in python.
This could perhaps be viewed as a bug in ndpointer - the problem is that as_array checks to see if its argument is a pointer, but that test fails for ndpointer, and numpy tries to create an array containing a single pointer (which fails because numpy has no pointer type).
You can get by just fine without using numpy at all:
class myStruct(ctypes.Structure):
_fields_ = [
('name', ctypes.c_char_p),
('status', ctypes.c_int),
]
arr = ctypes.POINTER(myStruct)()
size = ct.c_size_t()
_call('foo', ct.byref(arr), ct.byref(size))
# convert from `myStruct*` to `myStruct[size]`
arr = ctypes.cast(arr, ctypes.POINTER(myStruct * size.value)).contents
arr is now a ctypes array.
It seems that there is currently a pull request in Cython's repo to wrap c++ std::array but until then, I could use some help. I am currently wrapping the std::array like so:
cdef extern from "<array>" namespace "std" nogil:
cdef cppclass array2 "std::array<double, 2>":
array2() except+
double& operator[](size_t)
This works, but I have to loop over a cython memory view say, double arr[:], and copy the values one by one. Is there an easier way to do this? Essentially I'd like to do the following:
cdef double arr[2]
arr[0] = 1.0
arr[1] = 2.0
cdef array2 array2_arr = arr
#and the reverse
cdef array2 reverse
reverse[0] = 1.0
reverse[1] = 2.0
cdef double reverse_arr[2] = reverse
Is this completely unreasonable? As a result, working with std::array is extremely tedious because I have a to have a for-loop to copy values from cython to c++ and from c++ to cython. Furthermore, since cython doesn't give us the ability to have non-type template parameters, I have to define a wrapper for every variation of std::array in my code. Any suggestions on how to efficiently work with std::array would be greatly appreciated.
edit:
I'm now able to go from a memory view to an array2 type using the following:
def __cinit__(self, double[:] mem):
cdef array2 *arr = <array2 *>(&mem[0])
But it seems that no matter what I do, I cannot get cython to convert an array2 type to a memoryview:
cdef array2 arr = self.thisptr.getArray()
# error: '__pyx_t_1' declared as a pointer to a reference of type 'double &'
cdef double[::1] mview = <double[:2]>(&arr[0])
#OR
# Stop must be provided to indicate shape
cdef double[::1] mview = <double[::2]>(&arr[0])
Please help me figure out how to cast a C++ pointer to a memory view. Every combination I have tried to date has resulted in some kind of casting error.
EDIT:
I found that I am to perform the following syntax with no error using a newer version of Cython (I was using Cythong 0.22) and upgraded to 0.23.5.
cdef double[::1] mview = <double[:4]>(&arr[0])
However, if I attempt to return mview from the function I am using it in, I get garbage memory. Returning the memoryview to the pointer of my array loses scope and therefore destructs my array automatically. As soon as I figure out how to properly return my array, I'll attempt to update the official answer.
After much fiddling, I found the answer to my question.
Definition of array and class that uses array:
cdef extern from "<array>" namespace "std" nogil:
cdef cppclass array4 "std::array<int, 4>":
array4() except+
int& operator[](size_t)
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass ArrayFun:
ArrayFun(array4&)
array4 getArray()
Python implementation
cdef class PyArrayFun:
cdef ArrayFun *thisptr # hold a C++ instance which we're wrapping
def __cinit__(self, int[:] mem):
#
# Conversion from memoryview to std::array<int,4>
#
cdef array4 *arr = <array4 *>(&mem[0])
self.thisptr = new ArrayFun(arr[0])
def getArray(self):
cdef array4 arr = self.thisptr.getArray()
#
# Conversion from std::array<int, 4> to memoryview
#
cdef int[::1] mview = <int[:4]>(&arr[0])
cdef int[::1] new_view = mview.copy()
for i in range(0,4):
print ("arr is ", arr[i])
print("new_view is ", new_view[i])
# A COPY MUST be returned because arr goes out of scope and is
# default destructed when this function exist. Therefore we have to
# copy again. This kinda of sucks because we have to copy the
# internal array out from C++, and then we have to copy the array
# out from Python, therefore 2 copies for one array.
return mview.copy()
I have a function in C that takes an array of complex floats and does calculations on them in-place.:
/* foo.c */
void foo(cmplx_float* array, int length) {...}
The complex float struct looks like this:
typedef struct cmplx_float {
float real;
float imag;
} cmplx_float ;
I need to call that function in python using ctypes. In Python I have an Numpy 1-D ndarray of complex64 elements.
I have also made a class derived from ctypes.Structure:
class c_float(Structure):
_fields_ = [('real', c_float),
('imag', c_float)]
I imagine that I might need another python class that implements an array of structs. Overall I'm just having problems with connecting the pieces together. What needs to be done to eventually call my function in Python basically something like this more or less:
some_ctype_array = SomeConversionCode(cmplx_numpy_array)
lib.foo(some_ctype_array, length)
You can use the ndpointer from numpy.ctypeslib to declare the first argument to be a one-dimensional contiguous array of type numpy.complex64:
import numpy as np
from numpy import ctypeslib
# ...code to load the shared library as `lib` not shown...
# Declare the argument types of lib.foo:
lib.foo.argtypes = [ctypeslib.ndpointer(np.complex64, ndim=1, flags='C'), c_int]
Then you can do, for example,
z = np.array([1+2j, -3+4j, 5.0j], dtype=np.complex64)
lib.foo(z, z.size)
You might want to wrap that in a function that doesn't require the second argument:
def foo(z):
# Ensure that we use a contiguous array of complex64. If the
# call to foo(z, z.size) modifies z in place, and that is the
# intended effect of the function, then the following line should
# be removed. (The input z is then *required* to be a contiguous
# array of np.complex64.)
z = np.ascontiguousarray(z, dtype=np.complex64)
# Call the C function.
lib.foo(z, z.size)
I need to convert a Python list of ints to vector[int] in a cdef function to call another C function. I tried this:
cdef pylist_to_handles(hs):
cdef vector[int] o_vect
for h in hs:
o_vect.push_back(h)
return o_vect
This should work because I only need to call this from other cdef functions, but I get this error:
Cannot convert 'vector<int>' to Python object
What am I doing wrong ?
In Cython 0.17 using libcpp.vector, you can do this:
cdef vector[int] vect = hs
What you really have is this:
cdef object pylist_to_handles(hs):
...
return <object>o_vect
If you do not explicitily set a type, it is assumed to be a python object ("object" in code). As you see in the code, you're trying to cast vector[int] to an object, but Cython does not know how to handle that.
Just add a return type in cdef:
cdef vector[int] pylist_to_handles(hs):