Develop Reference

Returning and passing around raw POD pointers (arrays) with Python, C++, and pybind11

I have a C++ function which returns a raw float pointer, and another C++ function which accepts a raw float pointer as an argument. Something like:
float* ptr = something;
float* get_ptr(void) { return ptr; }
void use_ptr(float* ptr) { do_work(ptr); }
I want to be able to pass around pointers using Python. Something like this:
import my_native_functions as native
ptr = native.get_ptr()
native.use_ptr(ptr)
I am using pybind11 to create my native python module but I don't know how to create the bindings for the get_ptr() function. If I just do the following:
PYBIND11_MODULE(my_native_functions, m)
{
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
the get_ptr() function returns a Python Float object. I guess this makes sense because there are no pointer types in python. However, because this is now a simple Float, when I call the use_ptr() function and iterate over the pointer in C/C++, only the first element of the array is correct. The rest are garbage. To fix this, in C++, I have to cast my pointer to/from std::size_t. By doing this, everything works just fine.
However, I would like to ask: Is there a "right way" of achieving the above without the casting to/from std::size_t with pybind11?
In case you are curious why I do that:
I do understand that what I am doing is not type-safe. Also, I never touch the pointer/integer on the Python side. I just retrieve it from one native module and pass it to another. Also, I cannot cast the pointer to some kind of numpy view because the pointer is not always on the CPU. Sometimes I want to pass around CUDA pointers. Creating a py::array_t from a CUDA pointer is not possible unless I copy the data (and I do not want to do that).
Thank you.

Wrap the raw pointer in a custom "smart" pointer class (only pretending to be smart really) as described here. You can add some additional information to this class while you're at it, such as the size of the array element and the number of elements. This would make it a generalised array descriptor on the C++ side (but not on the Python side because you are not exposing the raw pointer to Python).
For a simpler option, just wrap your pointer in any old class in order to hide it from Python. No need to expose it to Python as a custom smart pointer. Here's an example that does just that:
#include <pybind11/pybind11.h>
#include <memory>
#include <iostream>
namespace py = pybind11;
template <class T> class ptr_wrapper
{
public:
ptr_wrapper() : ptr(nullptr) {}
ptr_wrapper(T* ptr) : ptr(ptr) {}
ptr_wrapper(const ptr_wrapper& other) : ptr(other.ptr) {}
T& operator* () const { return *ptr; }
T* operator->() const { return ptr; }
T* get() const { return ptr; }
void destroy() { delete ptr; }
T& operator[](std::size_t idx) const { return ptr[idx]; }
private:
T* ptr;
};
float array[3] = { 3.14, 2.18, -1 };
ptr_wrapper<float> get_ptr(void) { return array; }
void use_ptr(ptr_wrapper<float> ptr) {
for (int i = 0; i < 3; ++i)
std::cout << ptr[i] << " ";
std::cout << "\n";
}
PYBIND11_MODULE(Ptr,m)
{
py::class_<ptr_wrapper<float>>(m,"pfloat");
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}

Crash when calling PyArg_ParseTuple on a Numpy array

I wrote a simple Python extension function in C that just reads a Numpy array and it crashes.
static PyObject *test(PyObject *self, PyObject *args)
{
PyArrayObject *array = NULL;
if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) // Crash
return NULL;
return Py_BuildValue("d", 0);
}
Here is how it is called:
l = np.array([1,2,3,1,2,2,1,3])
print("%d" % extension.test(l))
What's wrong with my code?

I think the error is in the code you did not include in your example: have you remembered to call import_array() in your module init function:
... this subroutine must also contain calls to import_array() and/or import_ufunc() depending on which C-API is needed. Forgetting to place these commands will show itself as an ugly segmentation fault (crash) as soon as any C-API subroutine is actually called.
http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.how-to-extend.html#required-subroutine
I copied your example verbatim and added (using python 3)
PyMODINIT_FUNC
PyInit_numpytest(void)
{
import_array();
return PyModule_Create(&numpytest);
}
and the example ran without issues. Removing the call on the other hand produces a crash.

A difference between MacOS and Linux
The following works fine for me on Linux:
PyMODINIT_FUNC PyInit_numpytest(void) {
import_array();
return PyModule_Create(&numpytest);
}
However, when using MacOS, I get the following error when trying to import the library:
SystemError: лy () method: bad call flags
The following works fine on both MacOS and Linux:
PyMODINIT_FUNC PyInit_numpytest(void) {
PyObject* mod = PyModule_Create(&numpytest);
import_array();
return mod;
}
Notice that all I did was to change the order of PyModule_Create and import_array.

boost python won't auto-convert char* data members

I'm trying to wrap a C++ api and I'm hitting a roadblock on some char* class members. It seems that boost-python will auto convert char const * and std::string types into python objects (based on this answer), but it balks at char* types. This is the error I get (in python):
TypeError: No to_python (by-value) converter found for C++ type: char*
It turns out that these particular char * members probably should have been declared as char const * since the strings are never altered.
I'm new to boost-python so maybe there is an obvious answer, but I'm not having much luck googling this one.
Is there an easy way to tell boost-python to auto convert these char* members?
(Unfortunately I can't change the declarations of char * to char const * since the API I am wrapping is not under my control.)
UPDATE:
Ok so I think that I need to add a custom converter to handle the char* members. I started writing one:
/** to-python convert for char* */
struct c_char_p_to_python_str
{
static PyObject* convert(char* s) {
return incref(object(const_cast<const char*>(s)).ptr());
}
};
// register the QString-to-python converter
to_python_converter<char*, c_char_p_to_python_str>();
Unfortunately this does not work. This is the error:
error: expected unqualified-id
to_python_converter<char*, c_char_p_to_python_str>();
^
Looking at the docs I can see that the template args have this signature:
template <class T, class Conversion, bool convertion_has_get_pytype_member=false>
Since char* isn't a class I'm guessing that's why this didn't work. Anyone have some insight?
UPDATE2:
Nope. Turns out to_python_converter needs to get called inside of the BOOST_PYTHON_MODULE call.
I got the to_python_converter working (with some modifications). I also wrote a function to convert form python and registered it with converter::registry::push_back. I can see my to_python code running, but the from_python code never seems to run.

Let's assume we're wrapping some third-party API, and set aside the awfulness of having those pointers exposed and mucking with them from the outside.
Here's a short proof of concept:
#include <boost/python.hpp>
namespace bp = boost::python;
class example
{
public:
example()
{
text = new char[1];
text[0] = '\0';
}
~example()
{
delete[] text;
}
public:
char* text;
};
char const* get_example_text(example* e)
{
return e->text;
}
void set_example_text(example* e, char const* new_text)
{
delete[] e->text;
size_t n(strlen(new_text));
e->text = new char[n+1];
strncpy(e->text, new_text, n);
e->text[n] = '\0';
}
BOOST_PYTHON_MODULE(so02)
{
bp::class_<example>("example")
.add_property("text", &get_example_text, &set_example_text)
;
}
Class example owns text, and is responsible for managing the memory.
We provide an external getter and setter function. The getter is simple, it just provides read access to the string. The setter frees the old string, allocates new memory of appropriate size, and copies the data.
Here's a simple test in python interpreter:
>>> import so02
>>> e = so02.example()
>>> e.text
''
>>> e.text = "foobar"
>>> e.text
'foobar'
Notes:
set_example_text() could perhaps take std::string or bp::object so that we have the lenght easily available, and potentially allow assignment from more than just strings.
If there are many member variables to wrap and the getter/setter pattern is similar, generate the code using templates, or even just few macros.
There may be a way to do this with the converters, I'll have a look into that tomorrow. However, as we're dealing with memory management here, i'd personally prefer to handle it this way, as it's much more obvious what's happening.

This expands on Dan's answer. I wrote some macro definitions which generate lambda expressions. The benefits of this approach are that it is not tied to a particular type or member name.
In the API I am wrapping, I have a few hundred classes to wrap. This allows me to make a single macro call for every char* class member.
Here is a modified version of Dan's example code:
#include <boost/python.hpp>
namespace bp = boost::python;
#define ADD_PROPERTY(TYPE, ATTR) add_property(#ATTR, SET_CHAR_P(TYPE, ATTR), \
GET_CHAR_P(TYPE, ATTR))
#define SET_CHAR_P(TYPE, ATTR) +[](const TYPE& e){ \
if (!e.ATTR) return ""; \
return (const char*)e.ATTR; \
}
#define GET_CHAR_P(TYPE, ATTR) +[](TYPE& e, char const* new_text){ \
delete[] e.ATTR; \
size_t n(strlen(new_text)); \
e.ATTR = new char[n+1]; \
strncpy(e.ATTR, new_text, n); \
e.ATTR[n] = '\0'; \
}
class example
{
public:
example()
{
text = new char[1];
text[0] = '\0';
}
~example()
{
delete[] text;
}
public:
char* text;
};
BOOST_PYTHON_MODULE(topics)
{
bp::class_<example>("example")
.ADD_PROPERTY(example, text);
}

how to execute a python (2.7) script from a c program in Windows

I have been looking for this solution for some time and cannot find a solution. Some solutions have suggested using the "system" function in the . However, this suggestion is then always followed by 10 people saying never to use system as it lacks thread safety. Does anyone know how to do this?
Just some info: my python script gets and parses JSON from the internet, then creates a textfile, which the c program then uses. So, instead of having to run one, then the other, I want to run them both in succession from the C exe.
EDIT: Grammer

You can Use system function in c,c++
like this
system("python ashouri.py");
or
use this code
static PyObject *my_callback = NULL;
static PyObject *
my_set_callback(PyObject *dummy, PyObject *args)
{
PyObject *result = NULL;
PyObject *temp;
if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(my_callback); /* Dispose of previous callback */
my_callback = temp; /* Remember new callback */
/* Boilerplate to return "None" */
Py_INCREF(Py_None);
result = Py_None;
}
return result;
}
Be Successfull

SWIG interfacing C library to Python (Creating 'iterable' Python data type from C 'sequence' struct)

I have written a Python extension for a C library. I have a data structure that looks like this:
typedef struct _mystruct{
double * clientdata;
size_t len;
} MyStruct;
The purpose of this datatype maps directly to the list data type in Python. I therefore, want to create 'list-like' behavior for the exported struct, so that code written using my C extension is more 'Pythonic'.
In particular, this is what I want to be able to do (from python code)
Note: py_ctsruct is a ctsruct datatype being accessed in python.
My requirements can be sumarized as:
list(py_ctsruct) returns a python list with all contents copied out from the c struct
py_cstruct[i] returns ith element (preferably throws IndexError on invalid index)
for elem in py_ctsruct: ability to enumerate
According to PEP234, An object can be iterated over with "for" if it implements
_iter_() or _getitem_(). Using that logic then, I think that by adding the following attributes (via rename) to my SWIG interface file, I will have the desired behavior (apart from req. #1 above - which I still dont know how to achieve):
__len__
__getitem__
__setitem__
I am now able to index the C object in python. I have not yet implemented the Python exception throwing, however if array bounds are exceeded, are return a magic number (error code).
The interesting thing is that when I attempt to iterate over the struct using 'for x in' syntax for example:
for i in py_cstruct:
print i
Python enters into an infinite loop that simply prints the magic (error) number mentioned above, on the console. which suggests to me that there is something wrong with the indexing.
last but not the least, how can I implement requirement 1? this involves (as I understand it):
handling' the function call list() from python
Returning a Python (list) data type from C code
[[Update]]
I would be interested in seeing a little code snippet on what (if any) declarations I need to put in my interface file, so that I can iterate over the elements of the c struct, from Python.

The simplest solution to this is to implement __getitem__ and throw an IndexError exception for an invalid index.
I put together an example of this, using %extend and %exception in SWIG to implement __getitem__ and raise an exception respectively:
%module test
%include "exception.i"
%{
#include <assert.h>
#include "test.h"
static int myErr = 0; // flag to save error state
%}
%exception MyStruct::__getitem__ {
assert(!myErr);
$action
if (myErr) {
myErr = 0; // clear flag for next time
// You could also check the value in $result, but it's a PyObject here
SWIG_exception(SWIG_IndexError, "Index out of bounds");
}
}
%include "test.h"
%extend MyStruct {
double __getitem__(size_t i) {
if (i >= $self->len) {
myErr = 1;
return 0;
}
return $self->clientdata[i];
}
}
I tested it by adding to test.h:
static MyStruct *test() {
static MyStruct inst = {0,0};
if (!inst.clientdata) {
inst.len = 10;
inst.clientdata = malloc(sizeof(double)*inst.len);
for (size_t i = 0; i < inst.len; ++i) {
inst.clientdata[i] = i;
}
}
return &inst;
}
And running the following Python:
import test
for i in test.test():
print i
Which prints:
python run.py
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
and then finishes.
An alternative approach, using a typemap to map MyStruct onto a PyList directly is possible too:
%module test
%{
#include "test.h"
%}
%typemap(out) (MyStruct *) {
PyObject *list = PyList_New($1->len);
for (size_t i = 0; i < $1->len; ++i) {
PyList_SetItem(list, i, PyFloat_FromDouble($1->clientdata[i]));
}
$result = list;
}
%include "test.h"
This will create a PyList with the return value from any function that returns a MyStruct *. I tested this %typemap(out) with the exact same function as the previous method.
You can also write a corresponding %typemap(in) and %typemap(freearg) for the reverse, something like this untested code:
%typemap(in) (MyStruct *) {
if (!PyList_Check($input)) {
SWIG_exception(SWIG_TypeError, "Expecting a PyList");
return NULL;
}
MyStruct *tmp = malloc(sizeof(MyStruct));
tmp->len = PyList_Size($input);
tmp->clientdata = malloc(sizeof(double) * tmp->len);
for (size_t i = 0; i < tmp->len; ++i) {
tmp->clientdata[i] = PyFloat_AsDouble(PyList_GetItem($input, i));
if (PyErr_Occured()) {
free(tmp->clientdata);
free(tmp);
SWIG_exception(SWIG_TypeError, "Expecting a double");
return NULL;
}
}
$1 = tmp;
}
%typemap(freearg) (MyStruct *) {
free($1->clientdata);
free($1);
}
Using an iterator would make more sense for containers like linked lists, but for completeness sake here's how you might go about doing it for MyStruct with __iter__. The key bit is that you get SWIG to wrap another type for you, which provides the __iter__() and next() needed, in this case MyStructIter which is defined and wrapped at the same time using %inline since it's not part of the normal C API:
%module test
%include "exception.i"
%{
#include <assert.h>
#include "test.h"
static int myErr = 0;
%}
%exception MyStructIter::next {
assert(!myErr);
$action
if (myErr) {
myErr = 0; // clear flag for next time
PyErr_SetString(PyExc_StopIteration, "End of iterator");
return NULL;
}
}
%inline %{
struct MyStructIter {
double *ptr;
size_t len;
};
%}
%include "test.h"
%extend MyStructIter {
struct MyStructIter *__iter__() {
return $self;
}
double next() {
if ($self->len--) {
return *$self->ptr++;
}
myErr = 1;
return 0;
}
}
%extend MyStruct {
struct MyStructIter __iter__() {
struct MyStructIter ret = { $self->clientdata, $self->len };
return ret;
}
}
The requirements for iteration over containers are such that the container needs to implement __iter__() and return a new iterator, but in addition to next() which returns the next item and increments the iterator the iterator itself must also supply a __iter__() method. This means that either the container or an iterator can be used identically.
MyStructIter needs to keep track of the current state of iteration - where we are and how much we have left. In this example I did that by keeping a pointer to the next item and a counter that we use to tell when we hit the end. You could also have kept track of the sate by keeping a pointer to the MyStruct the iterator is using and a counter for the position within that, something like:
%inline %{
struct MyStructIter {
MyStruct *list;
size_t pos;
};
%}
%include "test.h"
%extend MyStructIter {
struct MyStructIter *__iter__() {
return $self;
}
double next() {
if ($self->pos < $self->list->len) {
return $self->list->clientdata[$self->pos++];
}
myErr = 1;
return 0;
}
}
%extend MyStruct {
struct MyStructIter __iter__() {
struct MyStructIter ret = { $self, 0 };
return ret;
}
}
(In this instance we could actually have just used the container itself as the iterator as an iterator, by supplying an __iter__() that returned a copy of the container and a next() similar to the first type. I didn't do that in my original answer because I thought that would be less clear than have two distinct types - a container and an iterator for that container)

Look up using the %typemap swig command. http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps
http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps_nn25
The memberin typemap might do what you want.
http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps_nn35
I have a typemap that I found in the Python section that allows me to transfer char** data into the C++ as a list of Python strings. I would guess there would be similar functionality.
Also, you can define %pythoncode in your interface inside the struct inside the swig "i" file. This will allow you to add python methods in the object that gets created for the struct. There is another command %addmethod (I think) that allows you to add methods to the struct or a class as well. Then you can create methods for indexing the objects in C++ or C if you want. There are a lot of ways to solve this.
For an interface I am working on I used a class object that has some methods for accessing the data in my code. Those methods are written in C++. Then I used the %pythoncode directive inside the class inside of the "i" file and created "getitem" and "setitem" methods in Python code that uses the expose C++ methods to make it look like a dictionary style access.

You say you have yet to implement Python exception throwing - that's the problem. From PEP 234:
A new exception is defined, StopIteration, which can be used to signal the end of an iteration.
You must set this exception at the end of your iteration. Since your code doesn't do this, you're running into the situation you've described:
The interpreter loops through your list's custom iternext function
Your function gets to the end of the array, and rather than correctly setting the StopIteration exception, simply returns your 'magic number'.
The interpreter, seeing no good reason to stop iterating, simply continues to print the value returned by iternext... your magic number. To the interpreter, it's just yet another list member.
Fortunately, this is a pretty simple fix, though may not seem as straightforward, because C has no exception facility. The Python C API simply uses a global error indicator that you set when an exception situation is raised, and then the API standards dictate you return NULL all the way up the stack to the interpreter, which then looks at the output of PyErr_Occurred() to see if an error is set, and if it is, prints the relevant exception and traceback.
So in your function, when you reach the end of the array, you just need this:
PyErr_SetString(PyExc_StopIteration,"End of list");
return NULL;
Here's another great answer for further reading on this issue: How to create a generator/iterator with the Python C API?

I encountered the very same problem with Python 2.6, and solved it thank to #aphex reply.
But I wanted to avoid any magic value, or extra boolean to pass end-of-list condition. Sure enough, my iterator have an atEnd() methods that tells me I am past the end of the list.
So in fact, it is fairly easy with SWIG exception handling. I just had to add the following magic:
%ignore MyStructIter::atEnd();
%exception MyStructIter::next {
if( $self->list->atEnd() ) {
PyErr_SetString(PyExc_StopIteration,"End of list");
SWIG_fail;
}
$action
}
The point is this snipet skips the next() calls completly once you are past the end of list.
If you stick to your idioms, it should look like:
%exception MyStructIter::next {
if( $self->pos >= $self->list->len ) {
PyErr_SetString(PyExc_StopIteration,"End of list");
SWIG_fail;
}
$action
}
NOTE FOR PYTHON 3.x:
You shall name your next() function with the magic "__ " prefix&postfix name. One option is simply to add:
%rename(__next__) MyStructIter::next;