Develop Reference

Wrapping C header macros with Cython

I am writing a Cython wrapper for the NAG C library.
In one of the header files from the NAG C library is the macros:
#define NAG_FREE(x) x04bdc((Pointer *)&(x))
Pointer is void*
x04bdc is:
extern void NAG_CALL x04bdc(Pointer *ptr);
NAG_FREE is the NAG library equivalent of free(), to free up memory.
Here is the extract from my lib_nag_integrate.pxd file:
cdef extern from "<nagx04.h>":
void x04bdc(Pointer *ptr)
x04bdc is a "fancy" free (malloc) routine. I cant access this code.
I then create a cdef function in my .pyx file:
cdef void NAG_FREE(double *x):
x04bdc(<Pointer *>&x)
Here i have type casted x to a double pointer, as that is what I am trying to free from memory, however the NAG library examples seem to use it for any type of pointer.
When running the python script which calls a cpdef function which eventually uses NAG_FREE, I get the following error:
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
If i comment out the NAG_FREE calls then it works fine, however NAG say it is necessary to use NAG_FREE.
The cdef function using NAG_FREE is:
cdef (double,double,Integer,Integer) dim1_fin_gen(lib_nag_integrate.NAG_D01SJC_FUN objfun,double a, double b,double epsabs, double epsrel,
Integer max_num_subint,Nag_User *comm,integration_out *output):
"""
:param f: user function
:type f: function
:param a: lower limit of integration
:type a: real float
:param b: upper limit of integration
:type b: real float
:param epsabs: user requested absolute error
:type epsabs: integer
:param epsrel: user requested relative error
:type epsrel: integer
:param max_num_subint: maximum number of subintervals
:type max_num_subint: real integer
:return: integration value of user function f
:rtype: real float
"""
cdef lib_nag_integrate.Nag_QuadProgress _qp
cdef lib_nag_integrate.NagError _fail
cdef double result
cdef double abserr
_fail.print = True
_fail.code = 0
_fail.errnum = 0
_fail.handler = NULL
lib_nag_integrate.d01sjc(objfun, a, b, epsabs, epsrel,
max_num_subint, &result, &abserr,
&_qp, comm, &_fail)
if _fail.code > 0 :
errorMessage = _fail.message
raise NagException(_fail.code,errorMessage)
print(_fail.message)
else:
output[0].result = result
output[0].abserr = abserr
output[0].fun_count = _qp.fun_count
output[0].num_subint = _qp.num_subint
NAG_FREE(_qp.sub_int_beg_pts)
NAG_FREE(_qp.sub_int_end_pts)
NAG_FREE(_qp.sub_int_result)
NAG_FREE(_qp.sub_int_error)
My libnag_integrate.pxd header file imports the following from the c library:
cdef extern from "<nag_types.h>":
ctypedef bint Nag_Boolean
ctypedef long Integer
ctypedef void* Pointer
ctypedef struct NagError:
int code
bint print "print"
char message[512]
Integer errnum
void (*handler)(char*,int,char*)
ctypedef struct Nag_User:
Pointer p
ctypedef struct Nag_QuadProgress:
Integer num_subint
Integer fun_count
double *sub_int_beg_pts
double *sub_int_end_pts
double *sub_int_result
double *sub_int_error
cdef extern from "<nagx04.h>":
(void*) NAG_ALLOC "x04bjc" (size_t size)
void x04bdc(Pointer *ptr)
cdef extern from "<nagd01.h>":
void d01sjc(NAG_D01SJC_FUN f, double a, double b,
double epsabs, double epsrel, Integer max_num_subint, double *result,
double *abserr, Nag_QuadProgress *qp, Nag_User *comm,
NagError *fail)
d01sjc is an integration routine which I cannot access. It allocates the memory of qp.sub_int_beg_pts etc inside.
I think I have a corrupt pointer causing the error. If so, where is it and how to I fix it?
many thanks
Upon further inspection of the structure '_qp'. The same error occurs when dereferencing e.g:
x = _qp.sub_int_end_pts[0]
so its the dereferencing of _qp which is causing the error.
The struct type Nag_QuadProgress is imported from its NAG header file into my .pxd as follows:
cdef extern from "<nag_types.h>":
ctypedef struct Nag_QuadProgress:
Integer num_subint
Integer fun_count
double *sub_int_beg_pts
double *sub_int_end_pts
double *sub_int_result
double *sub_int_error
Any ideas why dereferencing the pointers in this structure causes the error?

From Cython's point of view you're using NAG_FREE as a function, so that's what you should declare it as. It doesn't really matter that it's a really macro, and it certainly doesn't help to attempt to reimplement it.
cdef extern from "whatever_the_nag_header_is":
void NAG_FREE(Pointer x)
# or
void NAG_FREE(void *x)
# or
void NAG_FREE(...) # accepts anything - Cython doesn't attempt to match types
You may have to play around a bit with the type of the arguments to get it to work - I've suggested three options.
Really all you're aiming to do is to give Cython enough information that it can generate the right C code, and the right C code is NAG_FREE(your_variable), as if it's a function call.
With your code:
(<integration_out*>output)[0] suggests you're doing something very wrong. output is an integration_out pointer so why are you casting it? It either does nothing or introduces a potential error.
Despite claiming to return a C tuple type you actually don't bother to return anything.

Python 3 ctypes call to a function that needs an indirect reference to a buffer through another structure

I have a C shared library with a function that takes one argument.
This argument is a pointer to a structure with two fields.
typedef struct
{
uint8_t *p_data; // Pointer to a fixed lenth buffer (100 bytes)
uint16_t len; // number of valid bytes in the buffer (range 1-100)
} data_t;
I need to setup a buffer of 100 bytes in my Python 3 script (I am using 3.7.2 / 3.7.3),
load the library and call this function.
int
fn_convert_buffer(data_t *data_p)
{
...
}
My Python 3 ctypes call attempt hits incompatible types.
import ctypes as ct
# load the library, etc...
# lib_cdll = ct.CDLL(mySharedLib)
def c_py_fn_convert_buffer(b_p):
global lib_cdll
val = lib_cdll.fn_convert_buffer(ct.byref(b_p))
return int(val)
data_a = bytearray(100)
# Initialize the buffer with data.
uint8_p = ct.c_uint8 * len(data_a)
class BufferStruct_t (ct.Structure):
_pack_ = 1
_fields_ = [
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
("len", ct.c_uint16)
]
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
# TypeError: incompatible types, c_ubyte_Array_100 instance
# instead of LP_c_ubyte_Array_100 instance
# Call C function in shared-library: int fn_convert_buffer(data_t *data_p);
z = c_py_fn_convert_buffer(data_buf)
I need help in understanding what I've missed in the BufferStruct_t definition above. The from_buffer is supposed to get a pointer to the buffer but it seems to get c_ubyte_ARRAY_100.
A byref() on that does not work either
data_buf = BufferStruct_t(ct.byref(uint8_p.from_buffer(data_a)), ct.c_uint16(8))
# TypeError: expected LP_c_ubyte_Array_100 instance, got CArgObject
To test the basics of my flow, I made a sample case that will send the buffer and length parameters individually.
def c_py_fn_convert_data(d_p,l):
global lib_cdll
val = lib_cdll.fn_convert_data(ct.byref(d_p),ct.c_uint32(l))
return int(val)
test_a = ct.c_uint8 * len(data_a)
# Call C function in shared-library:
# int fn_convert_data(uint8_t *data_p, uint32_t length);
z = c_py_fn_convert_data(test_a.from_buffer(data_a), 8)
This simplified case works.
How do I get about building a Python 3 object that carries a reference to a buffer that the shared-library function expects?
Update with two variations that worked.
Update 1 Tried a cast based on something I read later (I don't cast lightly :-))
Changed,
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
to a pointer that is cast to refer an Array of specific length,
data_buf = BufferStruct_t(cast(uint8_p.from_buffer(data_a),
ct.POINTER(ct.c_uint8 * len(data_a))),
ct.c_uint16(8))
Update 2 based on Mark's answer.
Changed _field_ from,
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
to a simple-pointer form,
("p_data", ct.POINTER(ct.c_uint8)),
Both variations worked.
I am however curious to know which of these two ways is more safe/correct ctypes handling.
Is it better to cast to the Array form? or,
Is it better to use simple pointers and rely on the length sent independently?

Your structure definition declared a pointer to an array, not a simple pointer as in the C structure. Here's a working example with a simple implementation of the DLL where the function sums the data:
test.c
#include <stdint.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
typedef struct {
uint8_t *p_data;
uint16_t len;
} data_t;
API int fn_convert_buffer(data_t *data_p)
{
int i;
int sum = 0;
for(i = 0; i < data_p->len; ++i)
sum += data_p->p_data[i];
return sum;
}
test.py
import ctypes as ct
class BufferStruct_t(ct.Structure):
_pack_ = 1
_fields_ = [("p_data", ct.POINTER(ct.c_uint8)), # just a pointer
("len", ct.c_uint16)]
# Helper to initialize the data
def __init__(self,data):
self.p_data = (ct.c_uint8 * len(data))(*data)
self.len = len(data)
dll = ct.CDLL('test')
dll.fn_convert_buffer.argtypes = ct.POINTER(BufferStruct_t),
dll.fn_convert_buffer.restype = ct.c_int
data_buf = BufferStruct_t([1,2,3,4,5])
print(dll.fn_convert_buffer(data_buf))
Output:
15

Python wrapper to C function producing segmentation error

I am building a Python wrapper to a C code.
In the .c code I have the following defined function
double myfunction_at_k_and_z(struct background * pba,
struct spectra * psp,
struct perturbs *ppt,
double k,
double z,
double * pk_ic)
{ ...body of the function ...}
and in the .h file I have
struct background ba;
struct perturbs pt;
struct spectra sp;
Now I implement a function in my .pyx file, inside a class myclass
def get_myfunction(self, double k, double z):
cdef double dummy
result = myfunction_at_k_and_z(&self.ba, &self.sp, &self.pt, k, z, &dummy)
return result
and in the .pxd file I write:
cdef struct background: ....
cdef struct spectra: ...
cdef struct perturbs: ...
double myfunction_at_k_and_z (background * pba, spectra * psp, perturbs *ppt, double k, double z, double * pk_ic)
where the dots ... denote the components of the structures, which are not relevant here apart from one case: in struct spectra there are
int ln_k_size
double * ln_k
Now my problem is that despite everything compiling, when I run with Python my wrapper and use get_myfunction through e.g. myclass.get_myfunction(1., 1.), there seems to be a problem with the array ln_k, which apparently seems to have ln_k_size=0. This causes segmentation error.
Am I missing something in terms of allocation of memory?
Any way out?

How to pass Numpy PyArray* via FFI

Idea is to be able to modify array from library, like an "output" from a function.
Example:
ffi.cdef("""
//Reads data from a file, and store in the numpy array
void read_image(PyArray* arr);
""")
C = ffi.dlopen('libimage.so')
image = np.array([], dtype=np.float32)
C.read_image(image)
assert image.ndim == 2

You can't pass CPython-specific PyXxx structures via CFFI: you need to pass standard C data. Normally I'd answer that you need to design your cdef()'ed function with a standard C interface, for example something like:
ffi.cdef("""
struct myimage_t {
int width, height;
float *data;
};
int read_image(struct myimage_t *output); // fill in '*output'
void free_image(struct myimage_t *img); // free output->data
""")
myimage = ffi.new("struct myimage_t *")
if lib.read_image(myimage) < 0:
raise IOError
...
lib.free_image(myimage)
Then you need to manually convert the myimage to a numpy array, somewhere in the "..." code above.
One better alternative is to use a Python callback: a callback that makes the numpy array according to spec and returns a C-standard float * pointer. The numpy array itself is saved somewhere in the callback. You could save it as a Python global, or more cleanly use a "handle" you pass via C. Requires the API version, not the ABI. In _example_build.py:
ffi.cdef("""
extern "Python" float *alloc_2d_numpy_array(void *handle, int w, int h);
void read_image(void *handle);
""")
ffi.set_source("_example_cffi", """
void read_image(void *handle)
{
// the C code that eventually invokes
float *p = alloc_2d_numpy_array(handle, w, h);
// and then fill the data at 'p'
}
""")
ffi.compile(verbose=True)
In file example.py:
from _example_cffi import ffi, lib
class Context:
pass
#ffi.def_extern()
def alloc_2d_numpy_array(handle, w, h):
context = ffi.from_handle(handle)
context.image = np.ndarray([w, h], dtype=np.float32)
return ffi.cast("float *", ffi.from_buffer(context.image))
context = Context()
lib.read_image(ffi.new_handle(context))
image = context.image

Python ctypes identifying dll function result

I have some functions according to a DLL's documentation (there are more):
# there are 2 lines in the other example like this
# define CALLTYPE __stdcall
# define pLSenvL void*
pLSenvL CALLTYPE LScreateEnvL()
int LSopenLogFileL( pLSenvL pL, char *pcLogFile)
int CALLTYPE LSsetPointerL(pLSenvL pL, double* pdPointer, int* pnPointersNow)
int LSexecuteScriptL( pLSenvL pL, char* pcScript)
int LScloseLogFileL( pLSenvL pL)
void LSclearPointers( pLSenvL pL)
What I've done so far is this:
from ctypes import *
api = CDLL("PATH_TO_DLL")
pLEnv = api.LScreateEnvL()
script = "STRING FULL OF COMMANDS"
print api.LSexecuteScriptL(pLEnv, script)
and it works, but now I want to replicate an example I found:
void CSDlg::OnSolve()
{
int nError, nPointersNow;
CString csScript, cs;
double dNeeds[1], dStart[1];
dNeeds[ 0] = (double) nNeedsM;
pLSenvL pL;
pL = LScreateEnvL();
nError = LSopenLogFileL( pL, "log_file.log");
nError = LSsetPointerL( pL, dNeeds, &nPointersNow);
nError = LSsetPointerL( pL, dStart, &nPointersNow);
csScript = "SET ECHOIN 1\n";
// Run the script
nError = LSexecuteScriptL( pL, (LPCTSTR) csScript);
// Close the log file
LScloseLogFileL( pL);
csStartM.Format( "%d", (int) dStart[0]);
}
So far I've done this:
nError = c_int
nPointersNow = c_int
dNeeds = c_double()
#I'm just setting a random value
dNeeds = [c_double(10)]
pLEnv = api.LScreateEnvL()
nError = api.LSopenLogFileL(pLEnv, "log_file.log")
# here I got
# Procedure called with not enough arguments (8 bytes missing) or wrong calling convention
nError = api.LSsetPointerL(pLEnv, byref(dNeeds), nPointersNow)
# and here I got
# byref argument must be a ctypes instance, not 'list'
So I've searched and I had to do something like this
#now here comes my problem
#according to documentation it is
# int CALLTYPE LSsetPointerL(pLSenvL pL, double* pdPointer, int* pnPointersNow)
api.LSsetPointerL.restype = c_int
api.LSsetPointerL.argtypes = [ ¿?, c_double, c_int]
What should go as first element in that array of argtypes??
Is there something I have to worry about the CALLTYPE definition?
Thanks in advance

"Procedure called with not enough arguments (8 bytes missing) or wrong calling convention" refers to how you called the DLL. You used CDLL, but it says "# define CALLTYPE __stdcall", so you should used WinDLL instead (see ctypes docs). This sometimes works regardless but seems to be failing on the more complicated calls.
You are not instantiating nPointersNow, you probably mean nPointersNow().
You can't set dNeeds to be a list. Maybe you mean:
somefunc.argtypes = [c_double()]
dNeeds = c_double(10)
nError = api.LSsetPointerL(pLEnv, byref(dNeeds), nPointersNow)
And the type plSenvL should be defined somewhere in your docs. LScreateEnvL returns that type, as shown here:
pLSenvL CALLTYPE LScreateEnvL()
pLSenvL __stdcall LScreateEnvL() (<- what the above line really means)
so you need to know what it is. We can guess that it's a pointer (an integer) to something called LSenvL, which you will possibly need to make a structure for. Not hard, but you need to know how it's defined. That said, you may be able to avoid it since the first function returns it. So if you don't need to use it directly you could try and dodgy it like this:
LScreateEnvL = api.LScreateEnvL
LScreateEnvL.restype = c_int # just assume it's an int
LSopenLogFileL = api.LSopenLogFileL
LSopenLogFileL.restype = c_int
LSopenLogFileL.argtypes = (c_int, # again just assume
c_char_p)
LSsetPointerL = api.LSsetPointerL
LSsetPointerL.restype = c_int
LSsetPointerL.argtypes = (c_int, # assume int
POINTER(c_double),
POINTER(c_int))
pLSenvL = LScreateEnvL()
dPointer = c_double()
nPointersNow = c_int()
nError = LSsetPointerL(pLSenvL,
byref(dPointer),
byref(nPointersNow))