I have some classes implemented as cdef class in cython. In client python code, I would like to compose the classes with multiple inheritance, but I'm getting a type error. Here is a minimal reproducible example:
In [1]: %load_ext cython
In [2]: %%cython
...: cdef class A:
...: cdef int x
...: def __init__(self):
...: self.x = 0
...: cdef class B:
...: cdef int y
...: def __init__(self):
...: self.y = 0
...:
In [3]: class C(A, B):
...: pass
...:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-83ef5091d3a6> in <module>()
----> 1 class C(A, B):
2 pass
TypeError: Error when calling the metaclass bases
multiple bases have instance lay-out conflict
Is there any way to get around this?
The docs say that:
A Python class can inherit from multiple extension types provided that the usual Python rules for multiple inheritance are followed (i.e. the C layouts of all the base classes must be compatible).
I'm trying to understand what this could possibly mean given the trivial example above.
It's pretty restricted. As best as I can tell all but one of the classes has to be empty. Empty classes can have def functions, but not cdef functions or cdef attributes.
Take a Cython class:
cdef class A:
cdef int x
This translates to C code:
struct __pyx_obj_2bc_A { // the name might be different
PyObject_HEAD
int x;
};
Essentially just a C struct containing the basic Python object stuff, and an integer.
The restriction is that a derived class must contain only one PyObject_HEAD and that its PyObject* should also be interpretable as a struct __pyx_obj_2bc_A* or a struct __pyx_obj_2bc_B*.
In your case the two integers x and y would attempt to occupy the same memory (so conflict). However, if one of the types was empty then they would share the PyObject_HEAD but not conflict further.
cdef functions cause a struct __pyx_vtabstruct_2bc_A *__pyx_vtab; to be added to the struct (so it's not empty). This contains function pointers which allows inherited classes to override the cdef functions.
Having two cdef classes that inherit from a common third class is OK, event if the common third class is not empty.
cdef class A:
cdef int x
cdef class B(A):
cdef int y
cdef class C(A):
pass
class D(B,C):
pass
The internal Python code that does this check is the function best_base if you really want to investigate the details of the algorithm.
With reference to "is there any way to get round this?" the answer is "not really." Your best option is probably composition rather than inheritance (i.e. have class C hold an A and B object, rather than inherit from A and B)
Related
We have cython code built on top of an existing C library -- very simplified example here just to show the problem. The somecapi_t structure is opaque with an allocate function, a free function, and several functions to manipulate the structure.
data.pxd:
cdef extern from "<some_c_api.h>" nogil:
ctypedef struct some_c_api_t:
pass
some_c_api_t *some_c_api_alloc()
void some_c_api_free(some_c_api_t *scap)
int some_c_api_get_count(const some_c_api_t *scap)
Class Data is the python representation of the C structure. Class Data uses a helper class, Helper, to implement _getitem_, _setitem_, and other operations on part of the C type in a more pythonic way than the C API provides. Here, all we show is _len_. For sake of example, we also have an equivalent function, get_count that tries to access the "scap" pointer from Data.
data.pyx
from cpython.exc cimport PyErr_SetFromErrno
cdef class Helper:
cdef some_c_api_t *scap
def __cinit__(self, data, some_c_api_t *scap):
self.data = data
self.scap = scap
def __len__(self):
return some_c_api_get_count(self.scap)
def get_count(self):
return some_c_api_get_count(self.data.scap)
cdef class Data:
cdef some_c_api_t *scap
def __cinit__(self):
self.scap = some_c_api_alloc()
if self.scap == NULL:
PyErr_SetFromErrno(OSError)
def __dealloc__(self):
some_c_api_free(self.scap)
#property
def values(self):
return Helper(self, self.scap)
When compiled, we get two errors. First, the call to Helper's constructor is trying to convert "scap" to a python object even though it's declared as the C type in Helper's constructor.
#property
def values(self):
return Helper(self, self.scap)
^
------------------------------------------------------------
vna/data.pyx:30:32: Cannot convert 'some_c_api_t *' to Python object
Similarly, when get_count tries to get the scap pointer from Data, it thinks it's a python object.
def get_count(self):
return some_c_api_get_count(self.data.scap)
^
------------------------------------------------------------
vna/data.pyx:14:45: Cannot convert Python object to 'const some_c_api_t *'
Another approach we tried was to define a "get_scap" method in Data:
cdef some_c_api_t *scap(self):
return self.scap
and call it from Helper. In every case, however, attempts to pass the C pointer to Helper fails, insisting that the API between these two "cdef" classes must consist only of Python objects.
Is it possible for the Helper class to get access the C pointer?
I'm building a hierarchy of dataclasses.dataclass (Python 3.10).
E.g.
import dataclasses
#dataclasses.dataclass
class A:
a: int
#dataclasses.dataclass
class B(A):
b: int
I now have a bunch of As, which I want to additionally specify as B without adding all of A's properties to the constructor. Something like this:
a = A(1)
b = B(a, 1)
I know I could use dataclasses.asdict
b = B(**dataclasses.asdict(a), b=1)
Is this an acceptable solution in the sense of best practice? It looks a bit inefficient and less readable than it could be.
I tried overwriting the B.__init__ (and B.__new__) but this seems like too much code is needed. And with overwriting the __init__, it's not possible to make the dataclass frozen.
You can define a classmethod on B to make your approach more ergonomic.
#dataclasses.dataclass
class B(A):
b: int
#classmethod
def from_A(cls, a, **kwargs):
return cls(**dataclasses.asdict(a), **kwargs)
b = B.from_A(a, b=1)
Assume we have a cython class A with a pointer to float like in
# A.pyx
cdef class A:
cdef float * ptr
We also have a cython class B in another module which needs access to the data under ptr:
# B.pyx
cdef class B:
cdef float * f_ptr
cpdef submit(self, ptr_var):
self.f_ptr= get_from( ptr_var ) # ???
The corresponding Python code using A and B might be something like
from A import A
from B import B
a = A()
b = B()
ptr = a.get_ptr()
b.submit(ptr)
How can we define get_ptr() and what would we use for get_from in B?
The solution is to wrap the pointer variable into a Python object. Module libc.stdint offers a type named uintptr_t which is an integer large enough for storing any kind of pointer. With this the solution might look as follows.
from libc.stdint cimport uintptr_t
cdef class A:
cdef float * ptr
def get_ptr(self):
return <uintptr_t>self.ptr
The expression in angle brackets <uintptr_t> corresponds to a cast to uintptr_t. In class B we then have to cast the variable back to a pointer to float.
from libc.stdint cimport uintptr_t
cdef class B:
cdef float * f_ptr
cpdef submit(self, uintptr_t ptr_var):
self.f_ptr= <float *>ptr_var
This works for any kind of pointers not only for pointers to float. One has to make sure that both modules (A and B) deal with the same kind of pointer since that information is lost once the pointer is wrapped in a uintptr_t.
I'm using following cython code:
cimport numpy as np
np.import_array()
cdef class Seen:
cdef bint sint_
def __cinit__(self):
print('INIT seen object')
self.sint_ = 0
cdef saw_int(self, object val):
self.sint_ = 1
def test(object val):
test_type(val)
def test_type(object val, Seen seen=Seen()):
print ('BEFORE:', seen.sint_)
val = int(val)
seen.saw_int(val)
print ('AFTER:', seen.sint_)
Build it and call functions like so:
import test
import numpy as np
test.test(-1)
print('')
test.test(np.iinfo(np.uint64).max)
The output which produces questions:
INIT seen object
BEFORE: False
AFTER: True
BEFORE: True
AFTER: True
As output states - seen object is not instantiated in second test.test call. But at the same time if change test_type declaration like so:
cdef test_type(object val):
cdef Seen seen=Seen()
...
Init happens on each call.
So 2 questions:
Why 2 realizations of test_type is different? As far as I remember from cython docs these two is interchangable.
How should I pass seen object to the test_type with the default as init new one? If (..., Seen seen=Seen()) not working?
The default value of a function is evaluated once, when the function is defined. If you want a new Seen instance each time you call test_type, do the following:
def test_type(object val, Seen seen=None):
if seen is None:
seen = Seen()
print ('BEFORE:', seen.sint_)
val = int(val)
seen.saw_int(val)
print ('AFTER:', seen.sint_)
Caveat: I'm not very familiar with cython, so there might be a subtlety that I am missing. But this would be the issue in ordinary Python code, and I suspect the same issue applies here.
I am trying to pass a struct back into my Python from a c file. Let's say I have a file pointc.c like this:
typedef struct Point {
int x;
int y;
} Point;
struct Point make_and_send_point(int x, int y);
struct Point make_and_send_point(int x, int y) {
struct Point p = {x, y};
return p;
}
Then I set-up a point.pyx file like this:
"# distutils: language = c"
# distutils: sources = pointc.c
cdef struct Point:
int x
int y
cdef extern from "pointc.c":
Point make_and_send_point(int x, int y)
def make_point(int x, int y):
return make_and_send_point(x, y) // This won't work, but compiles without the 'return' in-front of the function call
How do I get the returned struct into my Python? Is this kind of thing only possible by creating a struct in the Cython and sending by reference to a void c function?
As a reference, my setup.py is:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules = cythonize(
"point.pyx",
language="c"
)
)
Most typically you would write some kind of wrapper class that holds the c-level struct, for example:
# point.pyx
cdef extern from "pointc.c":
ctypedef struct Point:
int x
int y
Point make_and_send_point(int x, int y)
cdef class PyPoint:
cdef Point p
def __init__(self, x, y):
self.p = make_and_send_point(x, y)
#property
def x(self):
return self.p.x
#property
def y(self):
return self.p.y
In usage
>>> import point
>>> p = point.PyPoint(10, 10)
>>> p.x
10
Cython's default behaviour given a struct is to convert it to a Python dictionary, which may be good enough for you. (This only works for structs made up of simple types though).
There's a couple of reasons why this isn't working. First you should do cdef extern from from headers, not source files, otherwise you get errors about multiple definitions (I assume this is just a mistake in creating your minimal example). Second you need to put the definition of Point within your cdef extern block:
cdef extern from "pointc.h":
cdef struct Point:
int x
int y
If you don't do that then Cython creates a mangled internal name for your struct (__pyx_t_5point_Point) which doesn't match the C function signature and thus it fails.
With this corrected, you get the correct default behaviour of converting structs to dicts. (This should work both ways - you can convert dicts back to structs). In the event that this isn't what you want, follow #chrisb's answer