Develop Reference

SWIG TypeError: in method 'my_method', argument 1 of type 'double const []'

I am trying to wrap a c function into python using swig.
The function, defined in my_method.h and implemented in my_method.c, is:
double my_method(const double a[], const double b[], const long int len_a, const long int len_b){
...
}
my swig interface file (my_method.i) is:
/* file : my_method.i */
/* name of the python module*/
%module my_method
%{
#include "my_method.h"
%}
%include "my_method.h"
I use dist-utils to generate the wrapped function
#!/usr/bin/env python
"""
setup.py file for SWIG wrapping
"""
from distutils.core import setup, Extension
my_method_module = Extension('_my_method',
sources=['my_method.c', 'my_method_wrap.c'],
)
setup (name = 'my_method',
version = '0.1',
author = "author",
description = """my method""",
ext_modules = [my_method_module],
py_modules = ["my_method"],
)
and then
swig -python my_method.i
python setup.py build_ext --inplace
however when I run my main script in python:
a = np.ones(4)
b = np.ones(4)
res = my_method(a, b, len(a), len(b))
I get the following error:
return _my_method.my_method(a, b, len_a, len_b)
TypeError: in method 'my_method', argument 1 of type 'double const []'
any suggestion?

SWIG has no idea that double a[] represents one or many doubles or that len_a indicates the length. NumPy has a SWIG interface file that defines the typemaps needed to associate a pointer and size with a numpy array or Python list called numpy.i (docs) (download).
Here's a standalone example. You'll either need to reorder the parameters so pointer/size are together or add an adapter function. I've demonstrated the latter in case reordering is not possible:
%module test
%{
#define SWIG_FILE_WITH_INIT
// sample implementation of OP function.
// Just sum all the values in both arrays.
double my_method(const double a[], const double b[], const long int len_a, const long int len_b) {
double sum = 0.0;
for(long int aa = 0; aa < len_a; ++aa)
sum += a[aa];
for(long int bb = 0; bb < len_b; ++bb)
sum += b[bb];
return sum;
}
// adapter function in case re-ordering parameters is not possible
double my_method2(const double a[], int len_a, const double b[], int len_b) {
return my_method(a, b, len_a, len_b);
}
%}
%include "numpy.i"
%init %{
import_array();
%}
// apply the typemaps for numpy arrays to the two pairs of parameters representing the arrays
%apply (double* IN_ARRAY1, int DIM1) {(double* a, int len_a)};
%apply (double* IN_ARRAY1, int DIM1) {(double* b, int len_b)};
// process the adapter function and rename
%rename(my_method) my_method2;
double my_method2(double* a, int len_a, double* b, int len_b);
I built using MSVC on Windows. Note that include paths are needed to Python and its NumPy package headers:
swig -python test.i
cl /LD /W3 /Ic:\python310\include /Ic:\Python310\Lib\site-packages\numpy\core\include /Fe_test.pyd test_wrap.c -link /libpath:c:\python310\libs
Demo:
>>> import test
>>> test.my_method([1, 2, 3], [4, 5, 6])
21.0
>>> import numpy as np
>>> a = np.ones(4)
>>> b = np.ones(4)
>>> test.my_method(a, b)
8.0

Numpy.i typemap for multiple arrays of different types

I have a C function which takes, among other arguments, 3 arrays of differents types :
void create_signal(double ztab[], ..., int *pdata, ..., char ranging[], ...); and I want to use this function in a Python module using SWIG.
I have already figured out how to use C functions which take in an array with numpy.i typemaps :
%apply (double* INPLACE_ARRAY1, int DIM1) {(double xtab[], int length)} for instance,
or even 2 arrays of same type :
apply (double* INPLACE_ARRAY1, int DIM1) {(double xtab[], int length_x), (double ytab[], int length_y)}
but I can't figure out if it is possible to do so if the arrays are of different types (in my case, int, double, and char), or if I should rewrite my own typemap.
Thanks for your help !

Anyway, here goes an example
test.h
#pragma once
void multipleInputs(const float* data, const int nData,
const float* pos, const int nPositions, const int nDim);
test.cpp
#include "test.h"
#include <cstdio>
void multipleInputs(const float* data, const int nData,
const float* pos, const int nPositions, const int nDim)
{
if (nData > 0) {
printf("%d'th element is: %f\n", (nData-1), data[nData-1]);
}
if ((nDim > 0) && (nPositions > 0)){
printf("%d'th dimension of the %d'th element is: %f\n",
(nDim-1), (nPositions-1), data[nData-1]);
}
}
test.i
%module example
%{
#define SWIG_FILE_WITH_INIT
#include "test.h"
%}
%include "numpy.i"
%init {
import_array();
}
%apply (float* IN_ARRAY1, int DIM1) \
{(const float* data, const int nData)}
%apply (float* IN_ARRAY2, int DIM1, int DIM2) \
{(const float* pos, const int nPositions, const int nDim)};
%include "test.h"
setup.py
import numpy
from setuptools import setup, Extension
setup(name="example",
py_modules=["example"],
ext_modules=[Extension("_example",
["test.i","test.cpp"],
include_dirs = [numpy.get_include(), '.'],
swig_opts=['-c++', '-I.'],
)]
)
test.py
import example
example.multipleInputs(np.ones((10),dtype=np.float32),
np.ones((10,10),dtype=np.float32))

Passing C struct to Cython and converting to Python numpy array

main.h
ifndef MAIN_H
define MAIN_H
ifdef __cplusplus
extern "C" {
endif
typedef struct Pythonout{
int pn;
double *px;
}Pythonout;
struct Pythonout l1tf_main(char *ifile_y, double lambda, int rflag);
ifdef __cplusplus
}
endif
endif /* MAIN_H */
Following is the Cython pyx file using main.h
.pyx
cimport numpy as np
cdef extern from "main.h":
ctypedef struct Pythonout:
int n
double *x
cdef Pythonout l1tf_main(char *ifile_y
,double lambdaval,
int rflag);
cdef class Pyclass:
cdef Pythonout pnx
def __cinit__(self, char *pfilename,
lambdaval, rflag):
self.pnx = l1tf_main(pfilename,
lambdaval, rflag)
#property
def n(self):
return self.pnx.n
#property
def x(self):
cdef np.npy_float64 shape[1]
shape[0] = <np.npy_intp> self.pnx.n
ndarray =
np.PyArray_SimpleNewFromData(1,
&(self.pnx.n),np.NPY_FLOAT64,
<void *> self.pnx.x)
np.PyArray_UpdateFlags(ndarray,
ndarray.flags.num
| np.NPY_OWNDATA)
return ndarray
cpdef filtered_trend(char *pfilename, double
lambdaval, int rflag):
pnx = Pyclass(pfilename, lambdaval, rflag)
return pnx.x
In the class I am getting the following error while compiling:
‘Pythonout {aka struct Pythonout}’ has no member named ‘n’
‘Pythonout {aka struct Pythonout}’ has no member named ‘x’
When calling the object value pnx.n and pnx.x.

There's at least two issues with your code.
The trivial issue causing the compilation errors: in C you call the struct attributes px and pn, while in Cython you call them x and n. This means that the code Cython generates does not match the C header. Make these consistent.
np.PyArray_UpdateFlags(ndarray,
ndarray.flags.num
| np.NPY_OWNDATA)
This tells Numpy that in now owns the data in x and is responsible for deallocating it. However suppose you have the Python code:
x1 = PyClassInstance.x
x2 = PyClassInstance.x
You now have two Numpy arrays that each believe the own the same data and will both try to deallocate it. (Similarly if you never access x then pnx.x is never deallocated) What you should probably do is have the PyClass instance be responsible for deallocating its pnx.x (in a __dealloc__ function). Then in your x property do:
ndarray = PyArray_SimpleNewFromData(...)
Py_INCREF(self) # SetBaseObject doesn't do this so you must do it manually
PyArray_SetBaseObject(ndarray, self) # check return value for errors...
Now the Numpy array treats the PyClass instance as owning the data.

Return a struct from C to Python using Cython

I am trying to pass a struct back into my Python from a c file. Let's say I have a file pointc.c like this:
typedef struct Point {
int x;
int y;
} Point;
struct Point make_and_send_point(int x, int y);
struct Point make_and_send_point(int x, int y) {
struct Point p = {x, y};
return p;
}
Then I set-up a point.pyx file like this:
"# distutils: language = c"
# distutils: sources = pointc.c
cdef struct Point:
int x
int y
cdef extern from "pointc.c":
Point make_and_send_point(int x, int y)
def make_point(int x, int y):
return make_and_send_point(x, y) // This won't work, but compiles without the 'return' in-front of the function call
How do I get the returned struct into my Python? Is this kind of thing only possible by creating a struct in the Cython and sending by reference to a void c function?
As a reference, my setup.py is:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules = cythonize(
"point.pyx",
language="c"
)
)

Most typically you would write some kind of wrapper class that holds the c-level struct, for example:
# point.pyx
cdef extern from "pointc.c":
ctypedef struct Point:
int x
int y
Point make_and_send_point(int x, int y)
cdef class PyPoint:
cdef Point p
def __init__(self, x, y):
self.p = make_and_send_point(x, y)
#property
def x(self):
return self.p.x
#property
def y(self):
return self.p.y
In usage
>>> import point
>>> p = point.PyPoint(10, 10)
>>> p.x
10

Cython's default behaviour given a struct is to convert it to a Python dictionary, which may be good enough for you. (This only works for structs made up of simple types though).
There's a couple of reasons why this isn't working. First you should do cdef extern from from headers, not source files, otherwise you get errors about multiple definitions (I assume this is just a mistake in creating your minimal example). Second you need to put the definition of Point within your cdef extern block:
cdef extern from "pointc.h":
cdef struct Point:
int x
int y
If you don't do that then Cython creates a mangled internal name for your struct (__pyx_t_5point_Point) which doesn't match the C function signature and thus it fails.
With this corrected, you get the correct default behaviour of converting structs to dicts. (This should work both ways - you can convert dicts back to structs). In the event that this isn't what you want, follow #chrisb's answer

How do I pass a pre-populated "unsigned char*" buffer to a C++ method using boost.python?

I have a C++ class with a member function that takes an unsigned char* buffer and an unsigned int length as arguments and operates on them. I've wrapped this class with Boost::Python and would like to pass a pre-populated buffer to the class from a Python script. The Python-side buffer is created with struct.pack. I can't figure out how to make the argument type match and keep getting Boost.Python.ArgumentError.
include/Example.h
#ifndef EXAMPLECLASS_H_
#define EXAMPLECLASS_H_
#include <cstdio>
class ExampleClass
{
public:
ExampleClass() {}
virtual ~ExampleClass() {}
void printBuffer(unsigned char* buffer, unsigned int length)
{
for (unsigned int i = 0; i < length; ++i)
{
printf("%c", buffer[i]);
}
printf("\n");
}
};
#endif
src/example.cpp
#include "Example.h"
int main(int argc, char** argv)
{
unsigned char buf[4];
buf[0] = 0x41;
buf[1] = 0x42;
buf[2] = 0x43;
buf[3] = 0x44;
ExampleClass e;
e.printBuffer(buf, 4);
return 0;
}
src/Example_py.cpp
#include <boost/python.hpp>
#include "Example.h"
using namespace boost::python;
BOOST_PYTHON_MODULE(example_py)
{
class_<ExampleClass>("ExampleClass")
.def("printBuffer", &ExampleClass::printBuffer)
;
}
scripts/example.py
#!/usr/bin/env python
import example_py
import struct
import ctypes
buf = struct.pack('BBBB', 0x41, 0x42, 0x43, 0x44)
print 'python:'
print buf
e = example_py.ExampleClass()
print 'c++:'
print e.printBuffer(ctypes.cast(ctypes.c_char_p(buf), ctypes.POINTER(ctypes.c_ubyte)), len(buf))
CMakeLists.txt (incomplete)
include_directories(
include
${Boost_INCLUDE_DIRS}
${PYTHON_INCLUDE_DIRS}
)
add_library(example_py
src/Example_py.cpp
)
target_link_libraries(example_py ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
set_target_properties(example_py PROPERTIES PREFIX "")
add_executable(example src/example.cpp)
target_link_libraries(example example_py)
Output
$ ./example
ABCD
$ ./scripts/example.py
python: ABCD
c++:
Traceback (most recent call last):
File "/home/dustingooding/example/scripts/example.py", line 13, in <module>
print 'c++:', e.printBuffer(ctypes.cast(ctypes.c_char_p(buf), ctypes.POINTER(ctypes.c_ubyte)), len(buf))
Boost.Python.ArgumentError: Python argument types in
ExampleClass.printBuffer(ExampleClass, LP_c_ubyte, int)
did not match C++ signature:
printBuffer(ExampleClass {lvalue}, unsigned char*, unsigned int)
I've tried a number of different approaches (passing 'buf' directly, passing 'buf' as a ctypes.c_char_p, creating a ctypes.ubyte array and populating it with the contents of 'buf' and passing it), but none seem to work.
I don't understand why 'LP_c_ubyte' and 'unsigned char*' don't match.
EDIT
Here's a Github project with a ready-to-go codebase. Feel free to use this. I've added #Tanner's fix. https://github.com/dustingooding/boost_python_ucharp_example

It may be worth considering exposing a Pythonic auxiliary function as the ExampleClass.printBuffer method to Python, that delegates to the c-ish ExampleClass::printBuffer member function. For instance, this would allow the Python users to invoke:
import example
import struct
buf = struct.pack('BBBB', 0x41, 0x42, 0x43, 0x44)
e.printBuffer(buf)
Rather than requiring the user to perform the correct ctypes cast and sizing.
The struct.pack() method returns a str object in Python2 and a bytes object in Python3, so the auxiliary C++ function would need to populate a continuous block of memory with the elements of from either str or bytes. The boost::python::stl_input_iterator can provide a convenient way to construct C++ containers, such as std::vector<char>, from a Python object, such as str or bytes. The only oddity is that stl_input_iterator expects the Python type to support the iterable protocol, which str does not do. However, the builtin iter() Python method can be used to create an iterable object.
/// #brief Auxiliary function used to allow a Python iterable object with char
/// elements to be passed to ExampleClass.printBuffer().
void example_class_print_buffer_wrap(
ExampleClass& self,
boost::python::object py_buffer)
{
namespace python = boost::python;
// `str` objects do not implement the iterator protcol (__iter__),
// but do implement the sequence protocol (__getitem__). Use the
// `iter()` builtin to create an iterator for the buffer.
// >>> __builtins__.iter(py_buffer)
python::object locals(python::borrowed(PyEval_GetLocals()));
python::object py_iter = locals["__builtins__"].attr("iter");
python::stl_input_iterator<char> begin(
py_iter(py_buffer)), end;
// Copy the py_buffer into a local buffer with known continguous memory.
std::vector<char> buffer(begin, end);
// Cast and delegate to the printBuffer member function.
self.printBuffer(
reinterpret_cast<unsigned char*>(&buffer[0]),
buffer.size());
}
With the auxiliary function created, one just needs to expose it as the ExampleClass.printBuffer method:
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<ExampleClass>("ExampleClass")
.def("printBuffer", &example_class_print_buffer_wrap)
;
}
Here is a complete example demonstrating this approach:
#include <cstdio>
#include <vector>
#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
// Mocks...
/// #brief Legacy class that cannot be changed.
class ExampleClass
{
public:
void printBuffer(unsigned char* buffer, unsigned int length)
{
for (unsigned int i = 0; i < length; ++i)
{
printf("%c", buffer[i]);
}
printf("\n");
}
};
/// #brief Auxiliary function used to allow a Python iterable object with char
/// elements to be passed to ExampleClass.printBuffer().
void example_class_print_buffer_wrap(
ExampleClass& self,
boost::python::object py_buffer)
{
namespace python = boost::python;
// `str` objects do not implement the iterator protcol (__iter__),
// but do implement the sequence protocol (__getitem__). Use the
// `iter()` builtin to create an iterator for the buffer.
// >>> __builtins__.iter(py_buffer)
python::object locals(python::borrowed(PyEval_GetLocals()));
python::object py_iter = locals["__builtins__"].attr("iter");
python::stl_input_iterator<char> begin(
py_iter(py_buffer)), end;
// Copy the py_buffer into a local buffer with known continguous memory.
std::vector<char> buffer(begin, end);
// Cast and delegate to the printBuffer member function.
self.printBuffer(
reinterpret_cast<unsigned char*>(&buffer[0]),
buffer.size());
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<ExampleClass>("ExampleClass")
.def("printBuffer", &example_class_print_buffer_wrap)
;
}
Interactive usage:
>>> import example
>>> import struct
>>> buf = struct.pack('BBBB', 0x41, 0x42, 0x43, 0x44)
>>> print 'python:', buf
python: ABCD
>>> e = example.ExampleClass()
>>> e.printBuffer(buf)
ABCD

The python documentation lists the following in the chapter Fundamental Data Types:
class ctypes.c_char_p
Represents the C char * datatype when it points to a zero-terminated string. For a general character pointer
that may also point to binary data, POINTER(c_char) must be used. The
constructor accepts an integer address, or a string.
shows that you should probably use a c_char_p type. If you use the POINTER() functions this will be a LP_c_char_p.
The type
LP_c_ubyte /* corresponds to */ unsigned char;
you should probably use
LP_c_char_p /* which corresponds to */ char *;
Update:
I've corrected the types above. Also: I'm not a python expert, so I might have it wrong. There is also this answer.