In Cython I would like to store an array of functions, but the functions don't all have the same signature. Specifically, some have two parameters and some three.
I defined the following:
ctypedef long (*func2param)(long param1, long param2)
ctypedef long (*funct3param)(long param1, long param2, long param3)
However, even if I had just one such signature, I'm still not sure how to go about making it work. Trying to assign a cdef function to an array of one of the above types gives me:
func_array2[i] = func_list[i][FUNCTION]
Cannot convert Python object to 'func2param'
Trying to cast gives:
func_array2[i] = <func2param>func_list[i][FUNCTION]
Python objects cannot be cast to pointers of primitive types
I can't think of anything useful you can do with an array of function pointers of unknown type - it isn't safe to call them (since you don't know the signature) and there really isn't anything else to do with function pointers. Therefore you at least need to find some way of storing what type of pointer it is.
One option would be to store a struct containing both pointers:
cdef struct FuncPtrStruct:
func2param f2p
func3param f3p
You then set one to NULL, store in the other one, and only call the non-NULL one The array specification would then be similar to cdef FuncPtrStruct array[10].
However, I'd probably use a C Union instead to store both pointers in the same memory location (+ an enum to identify the type). This is a little more effort to set up (you need an enum to define the type of the union, the union itself, and a struct containing the enum and the union); however the advantage is that you can add a lot more different types of function pointers without using more memory (for the "two-type" case the memory is probably equal):
# cpdef if you want to use the enum from Python too
cdef enum FuncType:
twoArg, threeArg
cdef union FuncPtrUnion:
func2param f2p
func3param f3p
cdef struct FuncPtrHolder:
FuncType type_
FuncPtrUnion value
Just to illustrate how you'd use it:
cdef long f(long x1, long x2):
print("f",x1,x2)
return 0
cdef long g(long x1, long x2, long x3):
print("g",x1,x2,x3)
return 1
def example():
cdef FuncPtrHolder fArray[10]
for i in range(10):
if i%2:
fArray[i].type_ = twoArg
fArray[i].value.f2p = &f
else:
fArray[i].type_ = threeArg
fArray[i].value.f3p = &g
# use
for i in range(10):
if fArray[i].type_ == twoArg:
fArray[i].value.f2p(i,i+1)
elif fArray[i].type_ == threeArg:
fArray[i].value.f3p(i,i+1,i+2)
It looks like in your code you have some list of Python objects and this is why you're getting compile errors. Without code it's impossible to know why, but I think Cython can automatically generate Python wrappers for cdef functions so I'd guess you've somehow made a list of those wrappers. If you want to handle Python lists of FuncPtrHolder (or otherwise use it from Python) you'd have to wrap it in a cdef class.
In this case my preferred, simpler solution would probably be to just use func3param and have the "two parameter" functions simply ignore the third argument. This means all the function pointers would have a consistent signature, which would make a lot more sense to me.
Related
I am trying to convert a list of objects (GeoJSON) to shapely objects using cython, but I am running into a error:
This piece of code seems to be the issue: cdef object result[N]. How do I declare a list/array from a given list?
Here is my current code:
def convert_geoms(list array):
cdef int i, N=len(array)
cdef double x, s=0.0
cdef object result[N] # ERROR HERE
for i in range(N):
g = build_geometry_objects2(array[i])
result[i] = g
return result
There's two issues with cdef object result[N]:
It creates a C array of Python objects - this doesn't really work because C arrays aren't easily integrated with the Python object reference counting (in this you'd need to copy the whole array to something else when you returned it anyway, since it's a local variable that's only scoped to the function).
For C arrays of the form sometype result[N], N must be known at compile-time. In this case N is different for each function call, so the variable definition is invalid anyway.
There's multiple solutions - most of them involve just accepting that you're using Python objects so not worrying about specifying the types and just writing valid Python code. I'd probably write it as a list comprehension. I suspect Cython will do surprisingly well at producing optimized code for that
return [ build_geometry_objects2(array[i]) for i in range(len(array)) ]
# or
return [ build_geometry_objects2(a) for a in array ]
The second version is probably better, but if it matters you can time it.
If the performance really matters you can use Python C API calls which you can cimport from cpython.list. See Cythonize list of all splits of a string for an example of something similar where list creation is optimized this way. The advantage of PyList_New is that it creates an appropriately sized list at the start filled with NULL pointers, which you can then fill in.
I am writing a python wrapper for a C library with ctypes and I have a C function that returns a structure. But that structure is defined in another C file and that file is wrapped by swig.
I simplified structure and code.
This structure is wrapped by swig.
struct point {
int x;
int y;
};
This function is wrapped with ctypes.
struct point add_points(struct point a, struct point b) {
struct point c;
c.x = a.x + b.x;
c.y = a.y + b.y;
return c;
}
Python wrapper.
import swigModule # contains class point generated from c structure point
import ctypes
_libc = ctypes.CDLL('./c_file.so')
def add_points(a, b):
add_points = _libc.add_points
add_points.argtypes = [swigModule.point, swigModule.point,]
add_points.restype = swigModule.point,
result = add_points(a, b)
return result
Problem is, I can't use swig generated class point as restype and argtype in ctypes. But I can't write my own structure wrapper like this.
class Point(ctypes.Structure):
_fields_ = [("x", ctypes.c_int),
("y", ctypes.c_int)]
Because the fields of the C structure are hidden, I don't have access to the source code. I only know the name of the structure from swig wrapper.
I have two C files, one is generated with swig and has structure data types. And another has functions and is wrapped with ctypes. I want to use data class from swig in my ctypes functions.
How can I map swig class generated from c structure to ctypes class so that I can use it as a return type?
It's totally possible to make SWIG and ctypes work together, in various different ways, as I'll show with a few examples below. However with that said there is a big caveat: in my experience it's almost never going to get you something that's maintainable, portable, extensible, etc. so I've tended to favour alternatives options where possible. (Those probably include asking the original authors for the source using whatever means of persuasion is appropriate, re-writing the whole thing from scratch, or simply using an alternative library).
Anyway, let's assume you've got a SWIG wrapped library built for a given Python version. And for whatever reason they didn't wrap your favourite function. So you'd like to just use ctypes to monkey patch something into the SWIG generated code. That can actually be pretty easy, provided:
The functions you want to call only take and return objects by pointer, not by value.
The functions you want to call only return existing instances of your types and don't malloc new ones. (There's a workaround for that, but it starts to make stuff difficulty again, really fast).
The structs you care about actually wrapped all the members you cared about and (if it's C++ code) they're POD types.
Your case doesn't really match those constraints, but for a warm-up exercise let's take a look at doing this with the following "easy_mode" function because it does let us introduce one key point:
struct point *easy_mode(struct point *a, struct point *b) {
a->x += b->x;
a->y += b->y;
return a;
}
This function is pretty easy to use given an existing SWIG wrapper that doesn't already wrap it, but does wrap the struct. We can simply use the SWIG code to create instances of the struct and the extract a pointer from them to give (via ctypes) to the exported but not wrapped function, for example:
from ctypes import *
from test import point
p1 = point()
p2 = point()
p1.x = 123
p1.y = 156
p2.x = 123
p2.y = 156
so = CDLL('./_test.so')
so.easy_mode(int(p1.this), int(p2.this))
print(p1.x)
print(p1.y)
Is sufficient to call this function and as we know the return type is really just modifying p1 we can use that knowledge and roll with it. The key thing to take away from this though is that calling int(p1.this) gets you an integer representation of the pointer that the SWIG object is proxying for. And that's all ctypes needs for a pointer.
Let's take this forward though, to the case where we pass and return structs by value. This is much harder because the way the function gets called depends on the size and members of the struct. Their types and ordering matters. And it varies from architecture to architecture. It can even vary within a given architecture based on various things. Luckily ctypes (via libffi, which is an interesting thing in and of itself if you've never seen it before) hides all that from us.
So now our targeted missing function can be something like this:
struct point add_them(struct point a, struct point b) {
struct point ret = { a.x + b.x, a.y + b.y };
return ret;
}
The problem is that in the scenario of having just an existing SWIG module that doesn't call it we know absolutely nothing about the members of struct point. That's critical to being able to call by value. Sure we can make some guesses and if it's simple enough to just guess then you may as well do that and be done with it for ctypes.
Fortunately the existence of a useable SWIG wrapping of the struct gives us (if a few assumptions hold true) enough to make at least a good enough guess as to the types/layout of the struct. Furthermore since we know how to get a pointer to the underlying memory an instance of the struct uses we can construct tests that will show us things about the layout. If all goes well we can use that to construct a ctypes Structure definition of fields that will be compatible.
The crux of this is that we're going to memset an instance to 0 and then try to set each byte to a marker value one at a type, using the SWIG generated setter code for each member. When we inspect that we can make the deductions we need.
Before we can do that however it's helpful to have an upperbound on the size of the struct. We can get that by calling malloc_useable_size() which tells us what the heap allocation got rounded up to. So we can do something like this:
useable_size = CDLL(None).malloc_usable_size
upper_size_bound = useable_size(int(p1.this))
buffer_type = POINTER(c_char * upper_size_bound)
print('Upper size bound is %d' % upper_size_bound)
choose_type = dict([(1, c_uint8), (2, c_uint16), (4, c_uint32)]).get
def generate_members(obj):
for member_name in (x for x in dir(obj) if not x[0] == '_' and x != 'this'):
print('Looking at member: %s' % member_name)
def pos(shift):
test = point()
memset(int(test.this), 0, upper_size_bound)
pattern = 0xff << (8 * shift)
try:
setattr(test, member_name, pattern)
except:
return -1
return bytes(cast(int(test.this), buffer_type).contents).find(b'\xff')
a=[pos(x) for x in range(upper_size_bound)]
offset = min((x for x in a if x >= 0))
size = 1 + max(a) - offset
print('%s is a %d byte type at offset %d' % (member_name, size, offset))
pad = [('pad', c_ubyte * offset)] if (offset > 0) else []
class Member(Structure):
_pack_ = 1
_fields_ = pad + [(member_name, choose_type(size))]
yield Member
This takes every member that SWIG knows about for the given struct and, having computed an array working out the offset of each byte of a given field then computes both the offset and size of that member. (The use of min/max means it should work with both BE and LE hardware). We can take the size and map it onto a type. Given the knowledge we now have we could compute the layout that best matches what we learned. I cheated though and for each member generated a struct that has padding added at the beginning to position the member at exactly the offset we computed. The python code above is a generator that yields a ctypes Structure that has a member of the right size/type at the right offset.
In reality you'll need to deduce a lot more. Floating point numbers would probably be best served with known exactly representable values. We need to consider both signed and unsigned types. Arrays, strings and even nested types could be done. It's all possible on a trial+error basis but that's left as an exercise for the reader.
Finally we need to pull it together. Since I cheated with the one structure per member thing above all we need to do is merge these into a union:
class YourType(Union):
_pack_ = 1
_fields_ = list(zip(string.ascii_lowercase, generate_members()))
_anonymous_ = string.ascii_lowercase[:len(_fields_)]
With that we've now got enough to work with to call our add_them function:
MyType=YourType
y=MyType()
y.x = 1
y.y = 2
add_them = so.add_them
add_them.argtypes = [MyType, MyType]
add_them.restype = MyType
v=add_them(y,y)
print(v)
print('V: %d,%d' % (v.x, v.y))
Which does actually work to call a ctypes function using just information derived from a pre-existing SWIG module.
I'd still recommend not doing it in any real code though!
It's an interesting question for sure! It has been asked before, see question 7886790.
That question does not have an accepted answer, though. Doing research on the internet turns up little useful information.
The general consensus seems to be "use either ctypes or SWIG".
So I'm afraid that the answer to your question is no. :-(
What I gathered from researching this question:
SWIG is better when you have to interface to multiple languages.
ctypes is better when you don't want to compile stuff.
ctypes works well with C code, but only with C++ code that has extern C interfaces.
boost.Python comes up a lot for interfacing C++ to Python.
You might find the answers to question 135834 informative.
The Cython documentation explains very well what they allow for, how you can declare them, and how to use them.
However, it is still not clear to me what they really are. For example, a simple assignment from a numpy array like this:
my_arr = np.empty(10, np.int32)
cdef int [:] new_arr = my_arr
can make the accessing/assignment of my_arr faster.
What is it happening behind the scenes? Numpy should already allocate the elements in memory in a contiguous fashion, so what's the deal with memoryviews? Apparently not that much, in fact the memoryview assignment of the numpy array new_arr should be equivalent to
cdef np.ndarray[np.int32_t, ndim=1] new_arr = np.empty(10, np.int32)
in terms of speed. However, memoryviews are considered more general than numpy array buffer; could you make a simple example in which the added 'generalization' is important/interesting?
Furthermore, if I have already allocated a pointer in order to make things as fast as possible, what is the advantage of casting it to a typed memoryview? (the answer to this question might be the same of the one above)
cdef int *my_arr = <int *> malloc(N * sizeof(int))
cdef int[:] new_arr = <int[:N]>my_arr
What is a memoryview:
When you write in a function:
cdef double[:] a
you end up with a __Pyx_memviewslice object:
typedef struct {
struct __pyx_memoryview_obj *memview;
char *data;
Py_ssize_t shape[8];
Py_ssize_t strides[8];
Py_ssize_t suboffsets[8];
} __Pyx_memviewslice;
The memoryview contains a C pointer some some data which it (usually) doesn't directly own. It also contains a pointer to an underlying Python object (struct __pyx_memoryview_obj *memview;). If the data is owned by a Python object then memview holds a reference to that and ensures the Python object that holds the data is kept alive as long as the memoryview is around.
The combination of the pointer to the raw data, and information of how to index it (shape, strides and suboffsets) allows Cython to do indexing the using the raw data pointers and some simple C maths (which is very efficient). e.g.:
x=a[0]
gives something like:
(*((double *) ( /* dim=0 */ (__pyx_v_a.data + __pyx_t_2 * __pyx_v_a.strides[0]) )));
In contrast, if you work with untyped objects and write something like:
a = np.array([1,2,3]) # note no typedef
x = x[0]
the indexing is done as:
__Pyx_GetItemInt(__pyx_v_a, 0, long, 1, __Pyx_PyInt_From_long, 0, 0, 1);
which itself expands to a whole bunch of Python C-api calls (so is slow). Ultimately it calls a's __getitem__ method.
Compared to typed numpy arrays: there really isn't a huge difference.
If you do something like:
cdef np.ndarray[np.int32_t, ndim=1] new_arr
it works practically very like a memoryview, with access to raw pointers and the speed should be very similar.
The advantage to using memoryviews is that you can use a wider range of array types with them (such as the standard library array), so you're more flexible about the types your functions can be called with. This fits in with the general Python idea of "duck-typing" - that your code should work with any parameter that behaves the right way (rather than checking the type).
A second (small) advantage is that you don't need the numpy headers to build your module.
A third (possibly larger) advantage is that memoryviews can be initialised without the GIL while cdef np.ndarrays can't (http://docs.cython.org/src/userguide/memoryviews.html#comparison-to-the-old-buffer-support)
A slight disadvantage to memoryviews is that they seem to be slightly slower to set up.
Compared to just using malloced int pointers:
You won't get any speed advantage (but neither will you get too much speed loss). The minor advantages of converting using a memoryview are:
You can write functions that can be used either from Python or internally within Cython:
cpdef do_something_useful(double[:] x):
# can be called from Python with any array type or from Cython
# with something that's already a memoryview
....
You can let Cython handle the freeing of memory for this type of array, which could simplify your life for things that have an unknown lifetime. See http://docs.cython.org/src/userguide/memoryviews.html#cython-arrays and especially .callback_free_data.
You can pass your data back to python python code (it'll get the underlying __pyx_memoryview_obj or something similar). Be very careful of memory management here (i.e. see point 2!).
The other thing you can do is handle things like 2D arrays defined as pointer to pointer (e.g. double**). See http://docs.cython.org/src/userguide/memoryviews.html#specifying-more-general-memory-layouts. I generally don't like this type of array, but if you have existing C code that already uses if then you can interface with that (and pass it back to Python so your Python code can also use it).
I have a C++ library and I want to wrap some of its functionality in python.
The function splits the given character array into 5 parts, not actual splitting but the structure we pass a pointer to, contains the information about the parts after the function returns. The 5 structures each contain 2 integers, one denoting the beginning of the part, and the other, the part's length.
The python wrapper should accept a python string and return a dictionary or tuple of the 5 parts(as python strings also).
My current approach of calling the function and then splitting the python string based on the sub-part information using python slicing syntax has not yielded any significant speed gains. I realize that there are many similar questions, but none of those cases have been helpful to me.
The Cython definition code is -
cdef extern from "parse.h" namespace util
ctypedef struct part:
int begin;
int len;
ctypedef struct Parsed:
part part1;
part part2;
part part3;
part part4;
part part5;
void ParseFunc(const char* url, int url_len, Parsed* parsed)
The Cython code is -
cimport parseDef
def parse(url, url_len):
cdef parseDef.Parsed parsed
parseDef.parseFunc(url, url_len, &parsed)
part1 = url[parsed.part1.begin:parsed.part1.begin+parsed.part1.len]
#similar code for other parts
return (part1, part2, part3, part4, part5)
Typical string size for this wrapper will be 10-50 generally.
You can get a small benefit from doing the indexing on const char* instead of the string
cimport parseDef
def parse(url, url_len):
cdef const char* url_as_char_ptr = url # automatic conversion
cdef parseDef.Parsed parsed
parseDef.parseFunc(url, url_len, &parsed)
part1 = url_as_char_ptr[parsed.part1.begin:parsed.part1.begin+parsed.part1.len]
#similar code for other parts
return (part1, part2, part3, part4, part5)
I don't think you can beat this by much is that, mostly because the c-code generated is actually pretty efficient. The indexing line is translated to something like
__pyx_t_2 = __Pyx_PyBytes_FromStringAndSize(__pyx_v_url_as_char_ptr + idx1, idx2 - idx1)
(noting that I've replaced parsed.part1.begin with idx1 just for the sake of readability and because I'm testing this with slightly different code since I don't have parseFunc. You can check your exact code with cython -a yourfile.pyx and looking at the html output).
This is basically just calling the Python c-api string constructor function. That will necessarily make a copy of the string it is passed, but you can't avoid that (the Python string constructor always makes a copy). That doesn't leave a lot of overhead to remove.
I am having a dictionary,
my_dict = {'a':[1,2,3], 'b':[4,5] , 'c':[7,1,2])
I want to use this dictionary inside a Cython nogil function . So , I tried to declare it as
cdef dict cy_dict = my_dict
Up to this stage is fine.
Now I need to iterate over the keys of my_dict and if the values are in list, iterate over it. In Python , it is quite easy as follows:
for key in my_dict:
if isinstance(my_dict[key], (list, tuple)):
###### Iterate over the value of the list or tuple
for value in list:
## Do some over operation.
But, inside Cython, I want to implement the same that too inside nogil . As, python objects are not allowed inside nogil, I am all stuck up here.
with nogil:
#### same implementation of the same in Cython
Can anyone please help me out ?
You can't use Python dict without the GIL because everything you could do with it involves manipulating Python objects. You most sensible option is to accept that you need the GIL. There's a less sensible option too involving C++ maps, but it may be hard to apply for your specific case.
You can use with gil: to reacquire the GIL. There is obvious an overhead here (parts using the GIL can't be executed in parallel, and there may be a delay which it waits for the GIL). However, if the dictionary manipulation is a small chunk of a larger piece of Cython code this may not be too bad:
with nogil:
# some large chunk of computationally intensive code goes here
with gil:
# your dictionary code
# more computationally intensive stuff here
The other less sensible option is to use C++ maps (along side other C++ standard library data types). Cython can wrap these and automatically convert them. To give a trivial example based on your example data:
from libcpp.map cimport map
from libcpp.string cimport string
from libcpp.vector cimport vector
from cython.operator cimport dereference, preincrement
def f():
my_dict = {'a':[1,2,3], 'b':[4,5] , 'c':[7,1,2]}
# the following conversion has an computational cost to it
# and must be done with the GIL. Depending on your design
# you might be able to ensure it's only done once so that the
# cost doesn't matter much
cdef map[string,vector[int]] m = my_dict
# cdef statements can't go inside no gil, but much of the work can
cdef map[string,vector[int]].iterator end = m.end()
cdef map[string,vector[int]].iterator it = m.begin()
cdef int total_length = 0
with nogil: # all this stuff can now go inside nogil
while it != end:
total_length += dereference(it).second.size()
preincrement(it)
print total_length
(you need to compile this with language='c++').
The obvious disadvantage to this is that the data-types inside the dict must be known in advance (it can't be an arbitrary Python object). However, since you can't manipulate arbitrary Python objects inside a nogil block you're pretty restricted anyway.
6-year later addendum: I don't recommend the "use C++ objects everywhere" approach as a general approach. The Cython-C++ interface is a bit clunky and you can spend a lot of time working around it. The Python containers are actually better than you think. Everyone tends to forget about the cost of converting their C++ objects to/from Python objects. People rarely consider if they really need to release the GIL or if they just read an article on the internet somewhere saying that the GIL is bad..
It's good for some tasks, but think carefully before blindly replacing all your list with vector, dict with map etc.. As a rule, if your C++ types live entirely within your function it may be a good move (but think twice...). If they're being converted as input or output arguments then think a third time.