I am building a Python wrapper to a C code.
In the .c code I have the following defined function
double myfunction_at_k_and_z(struct background * pba,
struct spectra * psp,
struct perturbs *ppt,
double k,
double z,
double * pk_ic)
{ ...body of the function ...}
and in the .h file I have
struct background ba;
struct perturbs pt;
struct spectra sp;
Now I implement a function in my .pyx file, inside a class myclass
def get_myfunction(self, double k, double z):
cdef double dummy
result = myfunction_at_k_and_z(&self.ba, &self.sp, &self.pt, k, z, &dummy)
return result
and in the .pxd file I write:
cdef struct background: ....
cdef struct spectra: ...
cdef struct perturbs: ...
double myfunction_at_k_and_z (background * pba, spectra * psp, perturbs *ppt, double k, double z, double * pk_ic)
where the dots ... denote the components of the structures, which are not relevant here apart from one case: in struct spectra there are
int ln_k_size
double * ln_k
Now my problem is that despite everything compiling, when I run with Python my wrapper and use get_myfunction through e.g. myclass.get_myfunction(1., 1.), there seems to be a problem with the array ln_k, which apparently seems to have ln_k_size=0. This causes segmentation error.
Am I missing something in terms of allocation of memory?
Any way out?
Related
I'm trying to make a cython-built slice-sampling library. A generic slice sampling library, where you supply a log-density, a starter value, and get a result. Working on the univariate model now. Based on the response here, I've come up with the following.
So i have a function defined in cSlice.pyx:
cdef double univariate_slice_sample(f_type_1 logd, double starter,
double increment_size = 0.5):
some stuff
return value
I have defined in cSlice.pxd:
cdef ctypedef double (*f_type_1)(double)
cdef double univariate_slice_sample(f_type_1 logd, double starter,
double increment_size = *)
where logd is a generic univariate log-density.
In my distribution file, let's say cDistribution.pyx, I have the following:
from cSlice cimport univariate_slice_sample, f_type_1
cdef double log_distribution(alpha_k, y_k, prior):
some stuff
return value
cdef double _sample_alpha_k_slice(
double starter,
double[:] y_k,
Prior prior,
double increment_size
):
cdef f_type_1 f = lambda alpha_k: log_distribution(alpha_k), y_k, prior)
return univariate_slice_sample(f, starter, increment_size)
cpdef double sample_alpha_k_slice(
double starter,
double[:] y_1,
Prior prior,
double increment_size = 0.5
):
return _sample_alpha_1_slice(starter, y_1, prior, increment_size)
the wrapper because apparently lambda's aren't allowed in cpdef's.
When I try compiling the distribution file, I get the following:
cDistribution.pyx:289:22: Cannot convert Python object to 'f_type_1'
pointing at the cdef f_type_1 f = ... line.
I'm unsure of what else to do. I want this code to maintain C speed, and importantly not hit the GIL. Any ideas?
You can jit a C-callback/wrapper for any Python function (cast to a pointer from a Python-object cannot done implicitly), how for example explained in this SO-post.
However, at its core the function will stay slow pure Python function. Numba gives you possibility to create real C-callbacks via a #cfunc. Here is a simplified example:
from numba import cfunc
#cfunc("float64(float64)")
def id_(x):
return x
and this is how it could be used:
%%cython
ctypedef double(*f_type)(double)
cdef void c_print_double(double x, f_type f):
print(2.0*f(x))
import numba
expected_signature = numba.float64(numba.float64)
def print_double(double x,f):
# check the signature of f:
if not f._sig == expected_signature:
raise TypeError("cfunc has not the right type")
# it is not possible to cast a Python object to a pointer directly,
# so we cast the address first to unsigned long long
c_print_double(x, <f_type><unsigned long long int>(f.address))
And now:
print_double(1.0, id_)
# 2.0
We need to check the signature of the cfunc-object during the run time, otherwise the casting <f_type><unsigned long long int>(f.address) would "work" also for the functions with wrong signature - only to (possible) crash during the call or giving funny hard to debug errors. I'm just not sure that my method is the best though - even if it works:
...
#cfunc("float32(float32)")
def id3_(x):
return x
print_double(1.0, id3_)
# TypeError: cfunc has not the right type
I am writing a Cython wrapper for the NAG C library.
In one of the header files from the NAG C library is the macros:
#define NAG_FREE(x) x04bdc((Pointer *)&(x))
Pointer is void*
x04bdc is:
extern void NAG_CALL x04bdc(Pointer *ptr);
NAG_FREE is the NAG library equivalent of free(), to free up memory.
Here is the extract from my lib_nag_integrate.pxd file:
cdef extern from "<nagx04.h>":
void x04bdc(Pointer *ptr)
x04bdc is a "fancy" free (malloc) routine. I cant access this code.
I then create a cdef function in my .pyx file:
cdef void NAG_FREE(double *x):
x04bdc(<Pointer *>&x)
Here i have type casted x to a double pointer, as that is what I am trying to free from memory, however the NAG library examples seem to use it for any type of pointer.
When running the python script which calls a cpdef function which eventually uses NAG_FREE, I get the following error:
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
If i comment out the NAG_FREE calls then it works fine, however NAG say it is necessary to use NAG_FREE.
The cdef function using NAG_FREE is:
cdef (double,double,Integer,Integer) dim1_fin_gen(lib_nag_integrate.NAG_D01SJC_FUN objfun,double a, double b,double epsabs, double epsrel,
Integer max_num_subint,Nag_User *comm,integration_out *output):
"""
:param f: user function
:type f: function
:param a: lower limit of integration
:type a: real float
:param b: upper limit of integration
:type b: real float
:param epsabs: user requested absolute error
:type epsabs: integer
:param epsrel: user requested relative error
:type epsrel: integer
:param max_num_subint: maximum number of subintervals
:type max_num_subint: real integer
:return: integration value of user function f
:rtype: real float
"""
cdef lib_nag_integrate.Nag_QuadProgress _qp
cdef lib_nag_integrate.NagError _fail
cdef double result
cdef double abserr
_fail.print = True
_fail.code = 0
_fail.errnum = 0
_fail.handler = NULL
lib_nag_integrate.d01sjc(objfun, a, b, epsabs, epsrel,
max_num_subint, &result, &abserr,
&_qp, comm, &_fail)
if _fail.code > 0 :
errorMessage = _fail.message
raise NagException(_fail.code,errorMessage)
print(_fail.message)
else:
output[0].result = result
output[0].abserr = abserr
output[0].fun_count = _qp.fun_count
output[0].num_subint = _qp.num_subint
NAG_FREE(_qp.sub_int_beg_pts)
NAG_FREE(_qp.sub_int_end_pts)
NAG_FREE(_qp.sub_int_result)
NAG_FREE(_qp.sub_int_error)
My libnag_integrate.pxd header file imports the following from the c library:
cdef extern from "<nag_types.h>":
ctypedef bint Nag_Boolean
ctypedef long Integer
ctypedef void* Pointer
ctypedef struct NagError:
int code
bint print "print"
char message[512]
Integer errnum
void (*handler)(char*,int,char*)
ctypedef struct Nag_User:
Pointer p
ctypedef struct Nag_QuadProgress:
Integer num_subint
Integer fun_count
double *sub_int_beg_pts
double *sub_int_end_pts
double *sub_int_result
double *sub_int_error
cdef extern from "<nagx04.h>":
(void*) NAG_ALLOC "x04bjc" (size_t size)
void x04bdc(Pointer *ptr)
cdef extern from "<nagd01.h>":
void d01sjc(NAG_D01SJC_FUN f, double a, double b,
double epsabs, double epsrel, Integer max_num_subint, double *result,
double *abserr, Nag_QuadProgress *qp, Nag_User *comm,
NagError *fail)
d01sjc is an integration routine which I cannot access. It allocates the memory of qp.sub_int_beg_pts etc inside.
I think I have a corrupt pointer causing the error. If so, where is it and how to I fix it?
many thanks
Upon further inspection of the structure '_qp'. The same error occurs when dereferencing e.g:
x = _qp.sub_int_end_pts[0]
so its the dereferencing of _qp which is causing the error.
The struct type Nag_QuadProgress is imported from its NAG header file into my .pxd as follows:
cdef extern from "<nag_types.h>":
ctypedef struct Nag_QuadProgress:
Integer num_subint
Integer fun_count
double *sub_int_beg_pts
double *sub_int_end_pts
double *sub_int_result
double *sub_int_error
Any ideas why dereferencing the pointers in this structure causes the error?
From Cython's point of view you're using NAG_FREE as a function, so that's what you should declare it as. It doesn't really matter that it's a really macro, and it certainly doesn't help to attempt to reimplement it.
cdef extern from "whatever_the_nag_header_is":
void NAG_FREE(Pointer x)
# or
void NAG_FREE(void *x)
# or
void NAG_FREE(...) # accepts anything - Cython doesn't attempt to match types
You may have to play around a bit with the type of the arguments to get it to work - I've suggested three options.
Really all you're aiming to do is to give Cython enough information that it can generate the right C code, and the right C code is NAG_FREE(your_variable), as if it's a function call.
With your code:
(<integration_out*>output)[0] suggests you're doing something very wrong. output is an integration_out pointer so why are you casting it? It either does nothing or introduces a potential error.
Despite claiming to return a C tuple type you actually don't bother to return anything.
I have a C++ function which returns a pointer double** - a high dimensional matrix in particular - and I'd like to wrap it to some python code using Cython. How should I act?
Here an example with a function with a double* pointer for sake of simplicity.
My C++ fib.cpp code:
double add(double a, double b)
{
return a+b;
}
double p[]= {1,2,3,4};
double* mult(double a)
{
p[0]=p[0]*a;
p[1]=p[1]*a;
return p;
}
Then there is the fib.hpp file:
double add(double a,double b);
double* mult(double a);
Then the pxd file fib.pxd:
cdef extern from "fib.hpp":
double add(double a,double b);
double* mult(double a);
In the end the pyx file
# distutils: language = c++
# distutils: sources = fib.cpp
cimport fib
def add(a,b):
return fib.add(a,b)
def mult(a): # dropping these lines
return fib.mult(a) # the code works without the double* function
Everything is compiled with the rather standard setup.py :
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("fib2cpp",
sources=["fib.pyx", "fib.cpp"],
language="c++")
setup(name="fib",
ext_modules=cythonize(ext))
When I compile the code:
setup.py build_ext -if
Cannot convert 'double *' to python object.
When I try with a double** function I get the same error.
What should I do?
I have found a solution for the case double *. First of all, the file fib.pxd is useless.
Then we need a new fib.pyx file:
# distutils: language = c++
# distutils: sources = fib.cpp
import numpy as np
cimport numpy as cnp
cdef extern from "fib.hpp":
double fib(int n)
double add (double a, double b)
double* mult(double a)
def make_mult(double a):
cdef double[:] mv = <double[:4]> mult(a) # the 4 stands for
return np.asarray(mv) # the dimension of the array defined in fib.cpp
In the case of a function which returns a nrows*ncols matrix; such as double* make_mat(int nrows, int ncols) , the second line of the function make_mult has to be rewritten as:
cdef double[:,:] mv=<double[:nrows,:ncols]> make_mat(nrows, ncols)
Regrettably, If I have a function double** make_mat(int nrows, int ncols) which always returns a matrix, then the previous code rises the error :
Pointer base type does not match cython.array base type
I am trying to access a C++ dll from python (I am new to Python). I overcame many calling convention issues and finally got it to run without any compile/linking error. However when I print the returning array from C++ dll in python it shows all random initialized values. It looks like the values were not correctly returned.
My C++ code looks like this.
double DLL_EXPORT __cdecl *function1(int arg1, int arg2, double arg3[],int arg4,double arg5,double arg6,double arg7, double arg8)
{
double *Areas = new double[7];
....Calculations
return Areas;
}
My python code looks as follows:
import ctypes
CalcDll = ctypes.CDLL("CalcRoutines.dll")
arg3_py=(ctypes.c_double*15)(1.926,1.0383,0.00008,0.00102435,0.0101,0.0,0.002,0.0254,102,1,0.001046153,0.001046153,0.001046153,0.001046153,20)
dummy = ctypes.c_double(0.0)
CalcDll.function1.restype = ctypes.c_double*7
Areas = CalcDll.function1(ctypes.c_int(1),ctypes.c_int(6),arg3_py,ctypes.c_int(0),dummy,dummy,dummy,dummy)
for num in HxAreas:
print("\t", num)
The output of the print statement is as below:
2.4768722583947873e-306
3.252195577561737e+202
2.559357001198207e-306
5e-324
2.560791130833573e-306
3e-323
2.5621383435212475e-306
Any suggestion on what I am doing wrong is greatly appreciated.
Instead of
CalcDll.function1.restype = ctypes.c_double * 7
there should be
CalcDll.function1.restype = ctypes.POINTER(ctypes.c_double)
and then
Areas = CalcDll.function1(ctypes.c_int(1), ctypes.c_int(6), arg3_py,
ctypes.c_int(0), dummy, dummy, dummy, dummy)
for i in range(7):
print("\t", Areas[i])
I'm not sure what ctypes does in case of 'ctypes.c_double * 7', if it tries to extract seven double from the stack or what.
Tested with
double * function1(int arg1, int arg2, double arg3[],
int arg4, double arg5, double arg6,
double arg7, double arg8)
{
double * areas = malloc(sizeof(double) * 7);
int i;
for(i=0; i<7; i++) {
areas[i] = i;
}
return areas;
}
the values in the array are printed correctly with restype = ctypes.POINTER(ctypes.c_double)
For a bit of learning experience, I'm trying to wrap a few parts of SDL (1.2.14) in Cython in an extension for Python 3.2.
I am having a problem figuring out how to wrap C structs straight into Python, being able to access its attributes directly like:
struct_name.attribute
For example, I want to take the struct SDL_Surface:
typedef struct SDL_Rect {
Uint32 flags
SDL_PixelFormat * format
int w, h
Uint16 pitch
void * pixels
SDL_Rect clip_rect
int refcount
} SDL_Rect;
And be able to use it like so in python:
import SDL
# initializing stuff
Screen = SDL.SetVideoMode( 320, 480, 32, SDL.DOUBLEBUF )
# accessing SDL_Surface.w and SDL_Surface.h
print( Screen.w, ' ', Screen.h )
For right now, I have wrapped the SDL_SetVideoMode and SDL_Surface like this in
a file called SDL.pyx
cdef extern from 'SDL.h':
# Other stuff
struct SDL_Surface:
unsigned long flags
SDL_PixelFormat * format
int w, h
# like past declaration...
SDL_Surface * SDL_SetVideoMode(int, int, int, unsigned )
cdef class Surface(object):
# not sure how to implement
def SetVideoMode(width, height, bpp, flags):
cdef SDL_Surface * screen = SDL_SetVideoMode
if not screen:
err = SDL_GetError()
raise Exception(err)
# Possible way to return?...
return Surface(screen)
How should I implement SDL.Surface?
In a simple case, if struct is opaque, it's as simple as:
cdef extern from "foo.h":
struct spam:
pass
When you want access to members, there are several options, well presented in the docs:
http://docs.cython.org/src/userguide/external_C_code.html#styles-of-struct-union-and-enum-declaration