I'm trying to use pybind11 to bind a struct that looks like this
struct myStruct {
int na;
int nb;
double* a;
double* b;
}
I'm not sure the right way to go about it. The examples in the pybind11 documentation show how to attach buffer protocol semantics to an object, but not to a class member.
I don't have the luxury of changing the interface of myStruct to contain std::vectors either which would allow me to use the usual .def_readwrite().
I've tried doing something like this
py::class_<myStruct>(m, "myStruct")
.def_property("a",
[](myStruct &s) {return py::array<double>({s.na}, {sizeof(double), s.a};)},
[](myStruct &s, py::array_t<double> val) {std::copy((double*) val.request().ptr, (double*) val.request().ptr + s.na, s.a);)}
)
Which compiles, but in python I don't see changes persist in the underlying data
print(my_struct.a[0]) # prints 0.0
my_struct.a[0] = 123.0
print(my_struct.a[0]) # still prints 0.0
Hey most likely not the most elegant answer, but maybe it gives you a starting point and temporary solution. I think what you need to do is use shared pointers.
Under https://github.com/pybind/pybind11/issues/1150 someone asked something similar but I was not able to adapt it to your example and only got the same result to yours with no changes to the data.
What worked for me in your specific example was using the shared_ptr and defining setter and getter functions for the pointers with a simple def_property for the pybin11 class.
class class_DATA{
public:
int na;
std::shared_ptr<double> a;
void set_a(double a){*class_DATA::a = a; };
double get_a(void){return *class_DATA::a; };
};
PYBIND11_MODULE(TEST,m){
m.doc() = "pybind11 example plugin";
//the costum class
py::class_<class_DATA>(m, "class_DATA", py::dynamic_attr())
.def(py::init<>()) //needed to define constructor
.def_readwrite("na", &class_DATA::na)
.def_property("a", &class_DATA::get_a, &class_DATA::set_a, py::return_value_policy::copy);
}
2017/06/13 EDIT:
I tried using boost as was suggested, but after spending more than 3 days trying to get it to compile and link, and failing, I decided that the stupid painful way was probably the fastest and less painfull.... so now my code just saves a mess of gigantic text files (splitting arrays and the complex/ imaginary parts of the numbers across files) that C++ then reads. Elegant... no.... effective... yes.
I have some scientific code, currently written in Python, that is being slowed down by a numerical 3d integration step within a loop. To overcome this I am re-writing this particular step in C++. (Cython etc is not an option).
Long story short: I want to transfer several very large arrays of complex numbers from the python code to the C++ integrator as conveniently and painlessly as possible. I could do this manually and painfully using text or binary files - but before I embark on this, I was wondering if I have any better options?
I'm using visual studio for C++ and anaconda for python (not my choice!)
Is there any file format or method that would make it quick and convenient to save an array of complex numbers from python and then recreate it in C++?
Many thanks,
Ben
An easy solution that I used many times is to build your "C++ side" as a dll (=shared object on Linux/OS X), provide a simple, C-like entrypoint (straight integers, pointers & co., no STL stuff) and pass the data through ctypes.
This avoids boost/SIP/Swig/... build nightmares, can be kept zero-copy (with ctypes you can pass a straight pointer to your numpy data) and allow you to do whatever you want (especially on the build-side - no friggin' distutils, no boost, no nothing - build it with whatever can build a C-like dll) on the C++ side. It has also the nice side-effect of having your C++ algorithm callable from other languages (virtually any language has some way to interface with C libraries).
Here's a quick artificial example. The C++ side is just:
extern "C" {
double sum_it(double *array, int size) {
double ret = 0.;
for(int i=0; i<size; ++i) {
ret += array[i];
}
return ret;
}
}
This has to be compiled to a dll (on Windows) or a .so (on Linux), making sure to export the sum_it function (automatic with gcc, requires a .def file with VC++).
On the Python side, we can have a wrapper like
import ctypes
import os
import sys
import numpy as np
path = os.path.dirname(__file__)
cdll = ctypes.CDLL(os.path.join(path, "summer.dll" if sys.platform.startswith("win") else "summer.so"))
_sum_it = cdll.sum_it
_sum_it.restype = ctypes.c_double
def sum_it(l):
if isinstance(l, np.ndarray) and l.dtype == np.float64 and len(l.shape)==1:
# it's already a numpy array with the right features - go zero-copy
a = l.ctypes.data
else:
# it's a list or something else - try to create a copy
arr_t = ctypes.c_double * len(l)
a = arr_t(*l)
return _sum_it(a, len(l))
which makes sure that the data is marshaled correctly; then invoking the function is as trivial as
import summer
import numpy as np
# from a list (with copy)
print summer.sum_it([1, 2, 3, 4.5])
# from a numpy array of the right type - zero-copy
print summer.sum_it(np.array([3., 4., 5.]))
See the ctypes documentation for more information on how to use it. See also the relevant documentation in numpy.
For complex numbers, the situation is slightly more complicated, as there's no builtin for it in ctypes; if we want to use std::complex<double> on the C++ side (which is pretty much guaranteed to work fine with the numpy complex layout, namely a sequence of two doubles), we can write the C++ side as:
extern "C" {
std::complex<double> sum_it_cplx(std::complex<double> *array, int size) {
std::complex<double> ret(0., 0.);
for(int i=0; i<size; ++i) {
ret += array[i];
}
return ret;
}
}
Then, on the Python side, we have to replicate the c_complex layout to retrieve the return value (or to be able to build complex arrays without numpy):
class c_complex(ctypes.Structure):
# Complex number, compatible with std::complex layout
_fields_ = [("real", ctypes.c_double), ("imag", ctypes.c_double)]
def __init__(self, pycomplex):
# Init from Python complex
self.real = pycomplex.real
self.imag = pycomplex.imag
def to_complex(self):
# Convert to Python complex
return self.real + (1.j) * self.imag
Inheriting from ctypes.Structure enables the ctypes marshalling magic, which is performed according to the _fields_ member; the constructor and extra methods are just for ease of use on the Python side.
Then, we have to tell ctypes the return type
_sum_it_cplx = cdll.sum_it_cplx
_sum_it_cplx.restype = c_complex
and finally write our wrapper, in a similar fashion to the previous one:
def sum_it_cplx(l):
if isinstance(l, np.ndarray) and l.dtype == np.complex and len(l.shape)==1:
# the numpy array layout for complexes (sequence of two double) is already
# compatible with std::complex (see https://stackoverflow.com/a/5020268/214671)
a = l.ctypes.data
else:
# otherwise, try to build our c_complex
arr_t = c_complex * len(l)
a = arr_t(*(c_complex(r) for r in l))
ret = _sum_it_cplx(a, len(l))
return ret.to_complex()
Testing it as above
# from a complex list (with copy)
print summer.sum_it_cplx([1. + 0.j, 0 + 1.j, 2 + 2.j])
# from a numpy array of the right type - zero-copy
print summer.sum_it_cplx(np.array([1. + 0.j, 0 + 1.j, 2 + 2.j]))
yields the expected results:
(3+3j)
(3+3j)
I see the OP is over a year old now, but I recently addressed a similar problem using the native Python-C/C++ API and its Numpy-C/C++ extension, and since I personally don't enjoy using ctypes for various reasons (e.g., complex number workarounds, messy code), nor Boost, wanted to post my answer for future searchers.
Documentation for the Python-C API and Numpy-C API are both quite extensive (albeit a little overwhelming at first). But after one or two successes, writing native C/C++ extensions becomes very easy.
Here is an example C++ function that can be called from Python. It integrates a 3D numpy array of either real or complex (numpy.double or numpy.cdouble) type. The function will be imported through a DLL (.so) via the module cintegrate.so.
#include "Python.h"
#include "numpy/arrayobject.h"
#include <math.h>
static PyObject * integrate3(PyObject * module, PyObject * args)
{
PyObject * argy=NULL; // Regular Python/C API
PyArrayObject * yarr=NULL; // Extended Numpy/C API
double dx,dy,dz;
// "O" format -> read argument as a PyObject type into argy (Python/C API)
if (!PyArg_ParseTuple(args, "Oddd", &argy,&dx,&dy,&dz)
{
PyErr_SetString(PyExc_ValueError, "Error parsing arguments.");
return NULL;
}
// Determine if it's a complex number array (Numpy/C API)
int DTYPE = PyArray_ObjectType(argy, NPY_FLOAT);
int iscomplex = PyTypeNum_ISCOMPLEX(DTYPE);
// parse python object into numpy array (Numpy/C API)
yarr = (PyArrayObject *)PyArray_FROM_OTF(argy, DTYPE, NPY_ARRAY_IN_ARRAY);
if (yarr==NULL) {
Py_INCREF(Py_None);
return Py_None;
}
//just assume this for 3 dimensional array...you can generalize to N dims
if (PyArray_NDIM(yarr) != 3) {
Py_CLEAR(yarr);
PyErr_SetString(PyExc_ValueError, "Expected 3 dimensional integrand");
return NULL;
}
npy_intp * dims = PyArray_DIMS(yarr);
npy_intp i,j,k,m;
double * p;
//initialize variable to hold result
Py_complex result = {.real = 0, .imag = 0};
if (iscomplex) {
for (i=0;i<dims[0];i++)
for (j=0;j<dims[1];j++)
for (k=0;k<dims[1];k++) {
p = (double*)PyArray_GETPTR3(yarr, i,j,k);
result.real += *p;
result.imag += *(p+1);
}
} else {
for (i=0;i<dims[0];i++)
for (j=0;j<dims[1];j++)
for (k=0;k<dims[1];k++) {
p = (double*)PyArray_GETPTR3(yarr, i,j,k);
result.real += *p;
}
}
//multiply by step size
result.real *= (dx*dy*dz);
result.imag *= (dx*dy*dz);
Py_CLEAR(yarr);
//copy result into returnable type with new reference
if (iscomplex) {
return Py_BuildValue("D", &result);
} else {
return Py_BuildValue("d", result.real);
}
};
Simply put that into a source file (we'll call it cintegrate.cxx along with the module definition stuff, inserted at the bottom:
static PyMethodDef cintegrate_Methods[] = {
{"integrate3", integrate3, METH_VARARGS,
"Pass 3D numpy array (double or complex) and dx,dy,dz step size. Returns Reimman integral"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
static struct PyModuleDef module = {
PyModuleDef_HEAD_INIT,
"cintegrate", /* name of module */
NULL, /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module,
or -1 if the module keeps state in global variables. */
cintegrate_Methods
};
Then compile that via setup.py much like Walter's boost example with just a couple obvious changes- replacing file.cc there with our file cintegrate.cxx, removing boost dependencies, and making sure the path to "numpy/arrayobject.h" is included.
In python then you can use it like:
import cintegrate
import numpy as np
arr = np.random.randn(4,8,16) + 1j*np.random.randn(4,8,16)
# arbitrary step size dx = 1., y=0.5, dz = 0.25
ans = cintegrate.integrate3(arr, 1.0, 0.5, .25)
This specific code hasn't been tested but is mostly copied from working code.
Note added in edit.
As mentioned in the comments, python itself, being an interpreted language, has little potential for computational efficiency. So in order to make python scripts efficient, one must use modules which aren't all interpreted, but under the hood call compiled (and optimized) code written in, say, C/C++. This is exactly what numpy does for you, in particular for operations on whole arrays.
Therefore, the first step towards efficient python scripts is the usage of numpy. Only the second step is to try to use your own compiled (and optimized) code. Therefore, I have assumed in my example below that you were using numpy to store the array of complex numbers. Everything else would be ill-advised.
There are various ways in which you can access python's original data from within a C/C++ program. I personally have done this with boost.Python, but must warn you that the documentation and support are lousy at best: you're pretty much on your own (and stack overflow, of course).
For example your C++ file may look like this
// file.cc
#include <boost/python.hpp>
#include <boost/python/numpy.hpp>
namespace p = boost::python;
namespace n = p::numpy;
n::ndarray func(const n::ndarray&input, double control_variable)
{
/*
your code here, see documentation for boost python
you pass almost any python variable, doesn't have to be numpy stuff
*/
}
BOOST_PYTHON_MODULE(module_name)
{
Py_Initialize();
n::initialize(); // only needed if you use numpy in the interface
p::def("function", func, "doc-string");
}
to compile this, you may use a python script such as
# setup.py
from distutils.core import setup
from distutils.extension import Extension
module_name = Extension(
'module_name',
extra_compile_args=['-std=c++11','-stdlib=libc++','-I/some/path/','-march=native'],
extra_link_args=['-stdlib=libc++'],
sources=['file.cc'],
libraries=['boost_python','boost_numpy'])
setup(
name='module_name',
version='0.1',
ext_modules=[module_name])
and run it as python setup.py build, which will create an appropriate .so file in a sub-directory of build, which you can import from python.
I have an existing c++ code library that uses a struct with std::vector, which should be exposed to python.
in the header:
struct sFOO
{
unsigned int start = 3 ;
double foo = 20.0 ;
};
in the cpp:
namespace myName
{
myfoo::myfoo(){
sFOO singlefoo;
std::vector<sFOO> foos;
}
sFOO singlefoo;
std::vector<sFOO>* myfoo::get_vector(){
return &foos;
}
}
and for boost::python snippet:
using namespace boost::python;
class dummy3{};
BOOST_PYTHON_MODULE(vStr)
{
scope myName = class_<dummy3>("myName");
class_<myName::sFOO>("sFOO")
.add_property("start",&myName::sFOO::start)
.add_property("foo",&myName::sFOO::foo)
;
class_<myName::myfoo>("myfoo", no_init)
.def(init<>())
.def("checkfoo",&myName::myfoo::checkfoo)
.add_property("foos",&myName::myfoo::foos)
.add_property("singlefoo",&myName::myfoo::singlefoo)
}
(FYI, fictitious class dummy3 is used to simulate namespace, and using scope is therefore not an option.)
A compilation and importing processes are OK, and I can access singlefoo, but whenever I try to access vector foos, I encounter the error message below.
Python class registered for C++ class std::vector<myName::sFOO,
std::allocator<myName::sFOO> >
To circumvent this issue,
I've firstly tried vector_indexing_suite, but it didn't help exposing pre-defined vector of struct.
I also assumed that there should be a solution related to exposing pointer to python, so I have tried to get a pointer by following:
.def("get_vector",&myName::myfoo::get_vector)
which produces compile Error.
Since I am quite a novice to both C++ and Boost, any comments including solution, tips for search, and a suggestion to suitable reference would be greatly appreciated.
Thanks in advance!
Method .def("get_vector",&myName::myfoo::get_vector) is not working because it returns a pointer to a vector, so it's necessary to inform the policy that defines how object ownership should be managed:
class_<myName::myfoo>("myfoo", no_init)
// current code
.def("get_vector", &myfoo::get_vector, return_value_policy<reference_existing_object>())
;
In order to use vector_indexing_suite, it is necessary to implement the equal to operator to the class that it holds:
struct sFOO
{
unsigned int start = 3 ;
double foo = 20.0 ;
bool operator==(const sFOO& rhs)
{
return this == &rhs; //< implement your own rules.
}
};
then you can export the vector:
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
class_<std::vector<sFOO>>("vector_sFOO_")
.def(vector_indexing_suite<std::vector<sFOO>>())
;
Swig's manual is kinda confusing to me. I am wrapping my C library to python so that I can test my C code in python. Now I want to know how I can access the C pointer address in Python,
for example, this is the code I have
typedef struct _buffer_t {
char buf[1024];
char *next_ptr;
} buffer_t;
void parse_buffer(buffer_t * buf_p) {
buf_p -> next_ptr ++;
}
What I wanted to do is below, in C code
buffer_t my_buf;
my_buf.next_ptr = my_buf.buf;
parse_buffer(&my_buf);
expect_equal(&(my_buf.buf)+1, my_buf.next_ptr);
How do I do the same thing in Python?
After import the SWIG wrapped module, I have the buffer_t class in python.
The problem is that SWIG is trying to wrap the buffer as a String in Python, which is not just a pointer. The synthesised assignment for next_ptr will allocate memory and make a string copy, not just do a pointer assignment. You can work around this in several ways.
The simplest is to use %extend to add a "reset buffer" method in Python:
%extend {
void resetPtr() {
$self->next_ptr=$self->buf;
}
}
which can then be called in Python to make the assignment you want.
Alternatively if you want to force the buffer to be treated as just another pointer you could force SWIG to treat both members as void* instead of char* and char[]. I had hoped that would be as simple as a %apply per type, but it seems not to work correctly for memberin and memberout in my testing:
%apply void * { char *next };
%apply void * { char buf[ANY] };
Given that the memberin/memberout typemaps are critical to making this work I think %extend is by far the cleanest solution.
I'd like to use some existing C++ code, NvTriStrip, in a Python tool.
SWIG easily handles the functions with simple parameters, but the main function, GenerateStrips, is much more complicated.
What do I need to put in the SWIG interface file to indicate that primGroups is really an output parameter and that it must be cleaned up with delete[]?
///////////////////////////////////////////////////////////////////////////
// GenerateStrips()
//
// in_indices: input index list, the indices you would use to render
// in_numIndices: number of entries in in_indices
// primGroups: array of optimized/stripified PrimitiveGroups
// numGroups: number of groups returned
//
// Be sure to call delete[] on the returned primGroups to avoid leaking mem
//
bool GenerateStrips( const unsigned short* in_indices,
const unsigned int in_numIndices,
PrimitiveGroup** primGroups,
unsigned short* numGroups,
bool validateEnabled = false );
FYI, here is the PrimitiveGroup declaration:
enum PrimType
{
PT_LIST,
PT_STRIP,
PT_FAN
};
struct PrimitiveGroup
{
PrimType type;
unsigned int numIndices;
unsigned short* indices;
PrimitiveGroup() : type(PT_STRIP), numIndices(0), indices(NULL) {}
~PrimitiveGroup()
{
if(indices)
delete[] indices;
indices = NULL;
}
};
Have you looked at the documentation of SWIG regarding their "cpointer.i" and "carray.i" libraries? They're found here. That's how you have to manipulate things unless you want to create your own utility libraries to accompany the wrapped code. Here's the link to the Python handling of pointers with SWIG.
Onto your question on getting it to recognize input versus output. They've got another section in the documentation here, that describes exactly that. You lable things OUTPUT in the *.i file. So in your case you'd write:
%inline{
extern bool GenerateStrips( const unsigned short* in_dices,
const unsigned short* in_numIndices,
PrimitiveGroup** OUTPUT,
unsigned short* numGroups,
bool validated );
%}
which gives you a function that returns both the bool and the PrimitiveGroup* array as a tuple.
Does that help?
It's actually so easy to make python bindings for things directly that I don't know why people bother with confusing wrapper stuff like SWIG.
Just use Py_BuildValue once per element of the outer array, producing one tuple per row. Store those tuples in a C array. Then Call PyList_New and PyList_SetSlice to generate a list of tuples, and return the list pointer from your C function.
I don't know how to do it with SWIG, but you might want to consider moving to a more modern binding system like Pyrex or Cython.
For example, Pyrex gives you access to C++ delete for cases like this. Here's an excerpt from the documentation:
Disposal
The del statement can be applied to a pointer to a C++ struct
to deallocate it. This is equivalent to delete in C++.
cdef Shrubbery *big_sh
big_sh = new Shrubbery(42.0)
display_in_garden_show(big_sh)
del big_sh
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/using_with_c++.html