I am writing a Cython wrapper for the NAG C library.
In one of the header files from the NAG C library is the macros:
#define NAG_FREE(x) x04bdc((Pointer *)&(x))
Pointer is void*
x04bdc is:
extern void NAG_CALL x04bdc(Pointer *ptr);
NAG_FREE is the NAG library equivalent of free(), to free up memory.
Here is the extract from my lib_nag_integrate.pxd file:
cdef extern from "<nagx04.h>":
void x04bdc(Pointer *ptr)
x04bdc is a "fancy" free (malloc) routine. I cant access this code.
I then create a cdef function in my .pyx file:
cdef void NAG_FREE(double *x):
x04bdc(<Pointer *>&x)
Here i have type casted x to a double pointer, as that is what I am trying to free from memory, however the NAG library examples seem to use it for any type of pointer.
When running the python script which calls a cpdef function which eventually uses NAG_FREE, I get the following error:
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
If i comment out the NAG_FREE calls then it works fine, however NAG say it is necessary to use NAG_FREE.
The cdef function using NAG_FREE is:
cdef (double,double,Integer,Integer) dim1_fin_gen(lib_nag_integrate.NAG_D01SJC_FUN objfun,double a, double b,double epsabs, double epsrel,
Integer max_num_subint,Nag_User *comm,integration_out *output):
"""
:param f: user function
:type f: function
:param a: lower limit of integration
:type a: real float
:param b: upper limit of integration
:type b: real float
:param epsabs: user requested absolute error
:type epsabs: integer
:param epsrel: user requested relative error
:type epsrel: integer
:param max_num_subint: maximum number of subintervals
:type max_num_subint: real integer
:return: integration value of user function f
:rtype: real float
"""
cdef lib_nag_integrate.Nag_QuadProgress _qp
cdef lib_nag_integrate.NagError _fail
cdef double result
cdef double abserr
_fail.print = True
_fail.code = 0
_fail.errnum = 0
_fail.handler = NULL
lib_nag_integrate.d01sjc(objfun, a, b, epsabs, epsrel,
max_num_subint, &result, &abserr,
&_qp, comm, &_fail)
if _fail.code > 0 :
errorMessage = _fail.message
raise NagException(_fail.code,errorMessage)
print(_fail.message)
else:
output[0].result = result
output[0].abserr = abserr
output[0].fun_count = _qp.fun_count
output[0].num_subint = _qp.num_subint
NAG_FREE(_qp.sub_int_beg_pts)
NAG_FREE(_qp.sub_int_end_pts)
NAG_FREE(_qp.sub_int_result)
NAG_FREE(_qp.sub_int_error)
My libnag_integrate.pxd header file imports the following from the c library:
cdef extern from "<nag_types.h>":
ctypedef bint Nag_Boolean
ctypedef long Integer
ctypedef void* Pointer
ctypedef struct NagError:
int code
bint print "print"
char message[512]
Integer errnum
void (*handler)(char*,int,char*)
ctypedef struct Nag_User:
Pointer p
ctypedef struct Nag_QuadProgress:
Integer num_subint
Integer fun_count
double *sub_int_beg_pts
double *sub_int_end_pts
double *sub_int_result
double *sub_int_error
cdef extern from "<nagx04.h>":
(void*) NAG_ALLOC "x04bjc" (size_t size)
void x04bdc(Pointer *ptr)
cdef extern from "<nagd01.h>":
void d01sjc(NAG_D01SJC_FUN f, double a, double b,
double epsabs, double epsrel, Integer max_num_subint, double *result,
double *abserr, Nag_QuadProgress *qp, Nag_User *comm,
NagError *fail)
d01sjc is an integration routine which I cannot access. It allocates the memory of qp.sub_int_beg_pts etc inside.
I think I have a corrupt pointer causing the error. If so, where is it and how to I fix it?
many thanks
Upon further inspection of the structure '_qp'. The same error occurs when dereferencing e.g:
x = _qp.sub_int_end_pts[0]
so its the dereferencing of _qp which is causing the error.
The struct type Nag_QuadProgress is imported from its NAG header file into my .pxd as follows:
cdef extern from "<nag_types.h>":
ctypedef struct Nag_QuadProgress:
Integer num_subint
Integer fun_count
double *sub_int_beg_pts
double *sub_int_end_pts
double *sub_int_result
double *sub_int_error
Any ideas why dereferencing the pointers in this structure causes the error?
From Cython's point of view you're using NAG_FREE as a function, so that's what you should declare it as. It doesn't really matter that it's a really macro, and it certainly doesn't help to attempt to reimplement it.
cdef extern from "whatever_the_nag_header_is":
void NAG_FREE(Pointer x)
# or
void NAG_FREE(void *x)
# or
void NAG_FREE(...) # accepts anything - Cython doesn't attempt to match types
You may have to play around a bit with the type of the arguments to get it to work - I've suggested three options.
Really all you're aiming to do is to give Cython enough information that it can generate the right C code, and the right C code is NAG_FREE(your_variable), as if it's a function call.
With your code:
(<integration_out*>output)[0] suggests you're doing something very wrong. output is an integration_out pointer so why are you casting it? It either does nothing or introduces a potential error.
Despite claiming to return a C tuple type you actually don't bother to return anything.
Related
I'm trying to make a cython-built slice-sampling library. A generic slice sampling library, where you supply a log-density, a starter value, and get a result. Working on the univariate model now. Based on the response here, I've come up with the following.
So i have a function defined in cSlice.pyx:
cdef double univariate_slice_sample(f_type_1 logd, double starter,
double increment_size = 0.5):
some stuff
return value
I have defined in cSlice.pxd:
cdef ctypedef double (*f_type_1)(double)
cdef double univariate_slice_sample(f_type_1 logd, double starter,
double increment_size = *)
where logd is a generic univariate log-density.
In my distribution file, let's say cDistribution.pyx, I have the following:
from cSlice cimport univariate_slice_sample, f_type_1
cdef double log_distribution(alpha_k, y_k, prior):
some stuff
return value
cdef double _sample_alpha_k_slice(
double starter,
double[:] y_k,
Prior prior,
double increment_size
):
cdef f_type_1 f = lambda alpha_k: log_distribution(alpha_k), y_k, prior)
return univariate_slice_sample(f, starter, increment_size)
cpdef double sample_alpha_k_slice(
double starter,
double[:] y_1,
Prior prior,
double increment_size = 0.5
):
return _sample_alpha_1_slice(starter, y_1, prior, increment_size)
the wrapper because apparently lambda's aren't allowed in cpdef's.
When I try compiling the distribution file, I get the following:
cDistribution.pyx:289:22: Cannot convert Python object to 'f_type_1'
pointing at the cdef f_type_1 f = ... line.
I'm unsure of what else to do. I want this code to maintain C speed, and importantly not hit the GIL. Any ideas?
You can jit a C-callback/wrapper for any Python function (cast to a pointer from a Python-object cannot done implicitly), how for example explained in this SO-post.
However, at its core the function will stay slow pure Python function. Numba gives you possibility to create real C-callbacks via a #cfunc. Here is a simplified example:
from numba import cfunc
#cfunc("float64(float64)")
def id_(x):
return x
and this is how it could be used:
%%cython
ctypedef double(*f_type)(double)
cdef void c_print_double(double x, f_type f):
print(2.0*f(x))
import numba
expected_signature = numba.float64(numba.float64)
def print_double(double x,f):
# check the signature of f:
if not f._sig == expected_signature:
raise TypeError("cfunc has not the right type")
# it is not possible to cast a Python object to a pointer directly,
# so we cast the address first to unsigned long long
c_print_double(x, <f_type><unsigned long long int>(f.address))
And now:
print_double(1.0, id_)
# 2.0
We need to check the signature of the cfunc-object during the run time, otherwise the casting <f_type><unsigned long long int>(f.address) would "work" also for the functions with wrong signature - only to (possible) crash during the call or giving funny hard to debug errors. I'm just not sure that my method is the best though - even if it works:
...
#cfunc("float32(float32)")
def id3_(x):
return x
print_double(1.0, id3_)
# TypeError: cfunc has not the right type
I've gotten some success in trying SIMD intrinsics thru cython. Right now I'm struggling to get the compare function in AVX to work because the compare function needed an argument that should not be converted to python object.
cdef extern from "immintrin.h" nogil: # in this example, we use SSE2
ctypedef float __m256
const int _CMP_GT_OS
__m256 _mm256_loadu_ps (float *__P) nogil
void _mm256_storeu_ps (float *__P, __m256 __A) nogil
__m256 _mm256_set1_ps (__m256 __A) nogil
__m256 _mm256_cmp_ps (__m256 __A, __m256 __B, _CMP_GT_OS) nogil
#cython.boundscheck(False) # turn off bounds-checking for entire function
#cython.wraparound (False) # turn off negative index wrapping for entire function
#cython.cdivision (True )
cdef void Example_v4 (float *A, float *B, float delx) :
### this example for A & B having exactly 8 elements
cdef:
__m256 mA, mB, mdelx, mOut
float *out = <float*> malloc( 8 * sizeof(float))
int i
with nogil:
mdelx = _mm256_set1_ps( delx )
mA = _mm256_loadu_ps( &A[0] )
mB = _mm256_loadu_ps( &B[0] )
mOut = _mm256_cmp_ps ( mA, mB, _CMP_GT_OS )
_mm256_storeu_ps( &out[0], mOut )
print ( " i out " )
for i in range(8):
print ( i, out[i] )
return
The problem is when I compile the cython code, I get this part highlighted as the issue.
mOut = _mm256_cmp_ps ( mA, mB, _CMP_GT_OS )
with ^ symbol pointing at _CMP_GT_OS
and message
Converting to Python object not allowed without gil
I believe the issue isn't gil, the intrinsic function is defined in official Intel documentation as
__m256 _mm256_cmp_ps (__m256 __A, __m256 __B, const int imm8)
imm8 can be of many type of Operations, with _CMP_GT_OS being one of them.
I don't know how to deal with 3rd argument and preventing it from converting to python since the intrinsic only recognize C/C++ const int. Any idea how to workaround this problem ?
I did two changes:
in the cdef extern part, I added a value to it. I am still not 100% certain how to use this function. So this is all trial and error but at least doing this way, I can move forward to check and do more trial and error.
cdef extern from "immintrin.h" nogil: # in this example, we use SSE2
ctypedef float __m256
const int _CMP_GT_OS = 14
## other definition like in the question
__m256 _mm256_cmp_ps (__m256 __A, __m256 __B, const int _CMP_GT_OS) nogil
Changing only the declaration in _mm256_cmp_ps part didn't work. I have to assigned it a value first.
Now it can compile without error and can be called and used in the python side. As for result, still strange, so I don't know if it's actually right or not this way. I'll report back when I understand more. In the meantime, feel free to chime in. Thank you.
I have a C++ function which returns a pointer double** - a high dimensional matrix in particular - and I'd like to wrap it to some python code using Cython. How should I act?
Here an example with a function with a double* pointer for sake of simplicity.
My C++ fib.cpp code:
double add(double a, double b)
{
return a+b;
}
double p[]= {1,2,3,4};
double* mult(double a)
{
p[0]=p[0]*a;
p[1]=p[1]*a;
return p;
}
Then there is the fib.hpp file:
double add(double a,double b);
double* mult(double a);
Then the pxd file fib.pxd:
cdef extern from "fib.hpp":
double add(double a,double b);
double* mult(double a);
In the end the pyx file
# distutils: language = c++
# distutils: sources = fib.cpp
cimport fib
def add(a,b):
return fib.add(a,b)
def mult(a): # dropping these lines
return fib.mult(a) # the code works without the double* function
Everything is compiled with the rather standard setup.py :
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("fib2cpp",
sources=["fib.pyx", "fib.cpp"],
language="c++")
setup(name="fib",
ext_modules=cythonize(ext))
When I compile the code:
setup.py build_ext -if
Cannot convert 'double *' to python object.
When I try with a double** function I get the same error.
What should I do?
I have found a solution for the case double *. First of all, the file fib.pxd is useless.
Then we need a new fib.pyx file:
# distutils: language = c++
# distutils: sources = fib.cpp
import numpy as np
cimport numpy as cnp
cdef extern from "fib.hpp":
double fib(int n)
double add (double a, double b)
double* mult(double a)
def make_mult(double a):
cdef double[:] mv = <double[:4]> mult(a) # the 4 stands for
return np.asarray(mv) # the dimension of the array defined in fib.cpp
In the case of a function which returns a nrows*ncols matrix; such as double* make_mat(int nrows, int ncols) , the second line of the function make_mult has to be rewritten as:
cdef double[:,:] mv=<double[:nrows,:ncols]> make_mat(nrows, ncols)
Regrettably, If I have a function double** make_mat(int nrows, int ncols) which always returns a matrix, then the previous code rises the error :
Pointer base type does not match cython.array base type
I am building a Python wrapper to a C code.
In the .c code I have the following defined function
double myfunction_at_k_and_z(struct background * pba,
struct spectra * psp,
struct perturbs *ppt,
double k,
double z,
double * pk_ic)
{ ...body of the function ...}
and in the .h file I have
struct background ba;
struct perturbs pt;
struct spectra sp;
Now I implement a function in my .pyx file, inside a class myclass
def get_myfunction(self, double k, double z):
cdef double dummy
result = myfunction_at_k_and_z(&self.ba, &self.sp, &self.pt, k, z, &dummy)
return result
and in the .pxd file I write:
cdef struct background: ....
cdef struct spectra: ...
cdef struct perturbs: ...
double myfunction_at_k_and_z (background * pba, spectra * psp, perturbs *ppt, double k, double z, double * pk_ic)
where the dots ... denote the components of the structures, which are not relevant here apart from one case: in struct spectra there are
int ln_k_size
double * ln_k
Now my problem is that despite everything compiling, when I run with Python my wrapper and use get_myfunction through e.g. myclass.get_myfunction(1., 1.), there seems to be a problem with the array ln_k, which apparently seems to have ln_k_size=0. This causes segmentation error.
Am I missing something in terms of allocation of memory?
Any way out?
For a bit of learning experience, I'm trying to wrap a few parts of SDL (1.2.14) in Cython in an extension for Python 3.2.
I am having a problem figuring out how to wrap C structs straight into Python, being able to access its attributes directly like:
struct_name.attribute
For example, I want to take the struct SDL_Surface:
typedef struct SDL_Rect {
Uint32 flags
SDL_PixelFormat * format
int w, h
Uint16 pitch
void * pixels
SDL_Rect clip_rect
int refcount
} SDL_Rect;
And be able to use it like so in python:
import SDL
# initializing stuff
Screen = SDL.SetVideoMode( 320, 480, 32, SDL.DOUBLEBUF )
# accessing SDL_Surface.w and SDL_Surface.h
print( Screen.w, ' ', Screen.h )
For right now, I have wrapped the SDL_SetVideoMode and SDL_Surface like this in
a file called SDL.pyx
cdef extern from 'SDL.h':
# Other stuff
struct SDL_Surface:
unsigned long flags
SDL_PixelFormat * format
int w, h
# like past declaration...
SDL_Surface * SDL_SetVideoMode(int, int, int, unsigned )
cdef class Surface(object):
# not sure how to implement
def SetVideoMode(width, height, bpp, flags):
cdef SDL_Surface * screen = SDL_SetVideoMode
if not screen:
err = SDL_GetError()
raise Exception(err)
# Possible way to return?...
return Surface(screen)
How should I implement SDL.Surface?
In a simple case, if struct is opaque, it's as simple as:
cdef extern from "foo.h":
struct spam:
pass
When you want access to members, there are several options, well presented in the docs:
http://docs.cython.org/src/userguide/external_C_code.html#styles-of-struct-union-and-enum-declaration