I see a problem from one Stackoverflow question. It is described below:
I have a C++ function which returns a raw float pointer, and another C++ function which accepts a raw float pointer as an argument. Something like:
float* ptr = something;
float* get_ptr(void) { return ptr; }
void use_ptr(float* ptr) { do_work(ptr); }
I want to be able to pass around pointers using Python. Something like this:
import my_native_functions as native
ptr = native.get_ptr()
native.use_ptr(ptr)
I am using pybind11 to create my native python module but I don't know how to create the bindings for the get_ptr() function. If I just do the following:
PYBIND11_MODULE(my_native_functions, m)
{
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
the get_ptr() function returns a Python Float object. I guess this makes sense because there are no pointer types in python. However, because this is now a simple Float, when I call the use_ptr() function and iterate over the pointer in C/C++, only the first element of the array is correct. The rest are garbage. To fix this, in C++, I have to cast my pointer to/from std::size_t. By doing this, everything works just fine.
However, I would like to ask: Is there a "right way" of achieving the above without the casting to/from std::size_t with pybind11?
The above is the whole problem description that comes from AstrOne's question:Returning and passing around raw POD pointers (arrays) with Python, C++, and pybind11.
My question is "To fix this, in C++, I have to cast my pointer to/from std::size_t", how should this cast be done? Can here give me a example or more detailed explanation? I don't quite understand.
Replace the float* type with std::uintptr_t. See below
#include <cstdint>
float* ptr = something;
std::uintptr_t get_ptr() { return reinterpret_cast<std::uintptr_t>(ptr); }
void use_ptr(std::uintptr_t ptr) { do_work(reinterpret_cast<float*>(ptr)); }
No pointer types will be exposed to pybind11 now. What is doing here is just cast a pointer type to a integral type, which has no difference in binary representation.
Aside: std::size_t would be Ok in most platforms. As per cppref:
On many platforms (an exception is systems with segmented addressing) std::size_t can safely store the value of any non-member pointer, in which case it is synonymous with std::uintptr_t.
Related
everyone,
now I am using the model of ctypes in python to introduce a .dll file. In my source code I have a function in c++ as followings
bool getFrameFromStream(char* streamName, const void* framebuffer, int frame_number, int overwriteMaxPixelVal=-1)
where framebuffer is a pointer pointing to an array of pixel data of flexible types, e.g. char, Uint16, etc. Since I am a newbee for python, I used ctypes.create_stream_buffer, namely
framebuffer = ctypes.create_string_buffer(1000000)
ctypes.cast(framebuffer, ctypes.c_void_p)
My questions is if the usage promises all the elements in the buffer can be of any type? Or just first one element is flexible while the others should still be char? If so, what should I do if I want to transfer a suitable variable.
Many Thanks
If you have a buffer and you want to convert to different types, you should use a struct. For example, if you have:
struct FrameBuffer {
int a;
int b;
double d;
float g;
}
Then if you have the string that represents this stack in a variable my_data in Python, you could unpack it with the call:
import struct
# ...
mvars = struct.unpack("iidf",my_data);
myvars will be a tuple of the data unpacked from there.
Check the examples in the link I gave you.
I would like to pass as argument of a function in my C module an array of uint8_t's.
I couldn't find a method to directly parse this array, so I'm parsing it to a PyObject_t and then iterating as a PyTuple_t object. This way, I need to cast each element PyObject_t of this tuple to uint8_t.
How can I do that, once that there is no PyInt_FromUINT8_t function or anything like it?
You can usually just get away with B using unsigned char. According to Parsing Arguments you should just be able to do:
uint8_t b;
if (!PyArg_ParseTuple("b", &b)) {
return NULL;
}
If not directly using arguments (e.g. you are dealing with a PyObject, simply use one of the PyInt_*, PyLong_* or PyNumber_* functions (https://docs.python.org/3/c-api/number.html?highlight=pynumber#c.PyNumber_AsSsize_t).
Converting from a uin8_t to a PyObject is simple as well, you can use PyInt_FromLong or PyLong_FromLong
I have a variable number of numpy arrays, which I'd like to pass to a C function. I managed to pass each individual array (using <ndarray>.ctypes.data_as(c_void_p)), but the number of array may vary a lot.
I thought I could pass all of these "pointers" in a list and use the PyList_GetItem() function in the C code. It works like a charm, except that the values of all elements are not the pointers I usually get when they are passed as function arguments.
Though, if I have :
from numpy import array
from ctypes import py_object
a1 = array([1., 2., 3.8])
a2 = array([222.3, 33.5])
values = [a1, a2]
my_cfunc(py_object(values), c_long(len(values)))
And my C code looks like :
void my_cfunc(PyObject *values)
{
int i, n;
n = PyObject_Length(values)
for(i = 0; i < n; i++)
{
unsigned long long *pointer;
pointer = (unsigned long long *)(PyList_GetItem(values, i);
printf("value 0 : %f\n", *pointer);
}
}
The printed value are all 0.0000
I have tried a lot of different solutions, using ctypes.byref(), ctypes.pointer(), etc. But I can't seem to be able to retrieve the real pointer values. I even have the impression the values converted by c_void_p() are truncated to 32 bits...
While there are many documentations about passing numpy pointers to C, I haven't seen anything about c_types within Python list (I admit this may seem strange...).
Any clue ?
After a few hours spent reading many pages of documentation and digging in numpy include files, I've finally managed to understand exactly how it works. Since I've spent a great amount of time searching for these exact explanations, I'm providing the following text as a way to avoid anyone to waste its time.
I repeat the question :
How to transfer a list of numpy arrays, from Python to C
(I also assume you know how to compile, link and import your C module in Python)
Passing a Numpy array from Python to C is rather simple, as long as it's going to be passed as an argument in a C function. You just need to do something like this in Python
from numpy import array
from ctypes import c_long
values = array([1.0, 2.2, 3.3, 4.4, 5.5])
my_c_func(values.ctypes.data_as(c_void_p), c_long(values.size))
And the C code could look like :
void my_c_func(double *value, long size)
{
int i;
for (i = 0; i < size; i++)
printf("%ld : %.10f\n", i, values[i]);
}
That's simple... but what if I have a variable number of arrays ? Of course, I could use the techniques which parses the function's argument list (many examples in Stackoverflow), but I'd like to do something different.
I'd like to store all my arrays in a list and pass this list to the C function, and let the C code handle all the arrays.
In fact, it's extremely simple, easy et coherent... once you understand how it's done ! There is simply one very simple fact to remember :
Any member of a list/tuple/dictionary is a Python object... on the C side of the code !
You can't expect to directly pass a pointer as I initially, and wrongly, thought. Once said, it sounds very simple :-) Though, let's write some Python code :
from numpy import array
my_list = (array([1.0, 2.2, 3.3, 4.4, 5.5]),
array([2.9, 3.8. 4.7, 5.6]))
my_c_func(py_object(my_list))
Well, you don't need to change anything in the list, but you need to specify that you are passing the list as a PyObject argument.
And here is the how all this is being accessed in C.
void my_c_func(PyObject *list)
{
int i, n_arrays;
// Get the number of elements in the list
n_arrays = PyObject_Length(list);
for (i = 0; i LT n_arrays; i++)
{
PyArrayObject *elem;
double *pd;
elem = PyList_GetItem(list,
i);
pd = PyArray_DATA(elem);
printf("Value 0 : %.10f\n", *pd);
}
}
Explanation :
The list is received as a pointer to a PyObject
We get the number of array from the list by using the PyObject_Length() function.
PyList_GetItem() always return a PyObject (in fact a void *)
We retrieve the pointer to the array of data by using the PyArray_DATA() macro.
Normally, PyList_GetItem() returns a PyObject *, but, if you look in the Python.h and ndarraytypes.h, you'll find that they are both defined as (I've expanded the macros !):
typedef struct _object {
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
} PyObject;
And the PyArrayObject... is exactly the same. Though, it's perfectly interchangeable at this level. The content of ob_type is accessible for both objects and contain everything which is needed to manipulate any generic Python object. I admit that I've used one of its member during my investigations. The struct member tp_name is the string containing the name of the object... in clear text; and believe me, it helped ! This is how I discovered what each list element was containing.
While these structures don't contain anything else, how is it that we can access the pointer of this ndarray object ? Simply using object macros... which use an extended structure, allowing the compiler to know how to access the additional object's elements, behind the ob_type pointer. The PyArray_DATA() macro is defined as :
#define PyArray_DATA(obj) ((void *)((PyArrayObject_fields *)(obj))->data)
There, it's casting the PyArayObject * as a PyArrayObject_fields * and this latest structure is simply (simplified and macros expanded !) :
typedef struct tagPyArrayObject_fields {
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
char *data;
int nd;
npy_intp *dimensions;
npy_intp *strides;
PyObject *base;
PyArray_Descr *descr;
int flags;
PyObject *weakreflist;
} PyArrayObject_fields;
As you can see, the first two element of the structure are the same as a PyObject and PyArrayObject, but additional elements can be addressed using this definition. It is tempting to directly access these elements, but it's a very bad and dangerous practice which is more than strongly discouraged. You must rather use the macros and don't bother with the details and elements in all these structures. I just thought you might be interested by some internals.
Note that all PyArrayObject macros are documented in http://docs.scipy.org/doc/numpy/reference/c-api.array.html
For instance, the size of a PyArrayObject can be obtained using the macro PyArray_SIZE(PyArrayObject *)
Finally, it's very simple and logical, once you know it :-)
I am writing a Python app that makes use of PulseAudio API. The implementation is heavily using callbacks written in Python and invoked by PulseAudio's C code.
The most information is passed into the callback by a specific structure, for instance pa_sink_info, which is defined in C as follows:
typedef struct pa_sink_info {
const char *name;
uint32_t index;
const char *description;
pa_sample_spec sample_spec;
pa_channel_map channel_map;
uint32_t owner_module;
pa_cvolume volume;
int mute;
uint32_t monitor_source;
const char *monitor_source_name;
pa_usec_t latency;
const char *driver;
pa_sink_flags_t flags;
pa_proplist *proplist;
pa_usec_t configured_latency;
pa_volume_t base_volume;
pa_sink_state_t state;
uint32_t n_volume_steps;
uint32_t card;
uint32_t n_ports;
pa_sink_port_info** ports;
pa_sink_port_info* active_port;
uint8_t n_formats;
pa_format_info **formats;
} pa_sink_info;
From this structure it's very easy to get scalar values, eg.:
self.some_proc(
struct.contents.index,
struct.contents.name,
struct.contents.description)
But I have a difficulty dealing with ports and active_port, which in Python are described as:
('n_ports', uint32_t),
('ports', POINTER(POINTER(pa_sink_port_info))),
('active_port', POINTER(pa_sink_port_info)),
Here n_ports specifies number of elements in ports, which is a pointer to array of pointers to structures of type pa_sink_port_info. Actually, I don't even know how I can convert these to Python types at all.
What is the most efficient way of converting ports into Python dictionary containing pa_sink_port_info's?
Solving this problem required careful reading of Python's ctypes reference. Once the mechanism of ctypes type translation implementation was clear, it's not so difficult to get to the desired values.
The main idea about pointers is that you use their contents attribute to get to the data the pointer points to. Another useful thing to know is that pointers can be indexed like arrays (it's not validated by the interpreter, so it's your own responsibility to make sure it is indeed an array).
For this particular PulseAudio example, we can process the ports structure member (which is a pointer to array of pointers) as follows:
port_list = []
if struct.contents.ports:
i = 0
while True:
port_ptr = struct.contents.ports[i]
# NULL pointer terminates the array
if port_ptr:
port_struct = port_ptr.contents
port_list.append(port_struct.name)
i += 1
else:
break
I'd like to use some existing C++ code, NvTriStrip, in a Python tool.
SWIG easily handles the functions with simple parameters, but the main function, GenerateStrips, is much more complicated.
What do I need to put in the SWIG interface file to indicate that primGroups is really an output parameter and that it must be cleaned up with delete[]?
///////////////////////////////////////////////////////////////////////////
// GenerateStrips()
//
// in_indices: input index list, the indices you would use to render
// in_numIndices: number of entries in in_indices
// primGroups: array of optimized/stripified PrimitiveGroups
// numGroups: number of groups returned
//
// Be sure to call delete[] on the returned primGroups to avoid leaking mem
//
bool GenerateStrips( const unsigned short* in_indices,
const unsigned int in_numIndices,
PrimitiveGroup** primGroups,
unsigned short* numGroups,
bool validateEnabled = false );
FYI, here is the PrimitiveGroup declaration:
enum PrimType
{
PT_LIST,
PT_STRIP,
PT_FAN
};
struct PrimitiveGroup
{
PrimType type;
unsigned int numIndices;
unsigned short* indices;
PrimitiveGroup() : type(PT_STRIP), numIndices(0), indices(NULL) {}
~PrimitiveGroup()
{
if(indices)
delete[] indices;
indices = NULL;
}
};
Have you looked at the documentation of SWIG regarding their "cpointer.i" and "carray.i" libraries? They're found here. That's how you have to manipulate things unless you want to create your own utility libraries to accompany the wrapped code. Here's the link to the Python handling of pointers with SWIG.
Onto your question on getting it to recognize input versus output. They've got another section in the documentation here, that describes exactly that. You lable things OUTPUT in the *.i file. So in your case you'd write:
%inline{
extern bool GenerateStrips( const unsigned short* in_dices,
const unsigned short* in_numIndices,
PrimitiveGroup** OUTPUT,
unsigned short* numGroups,
bool validated );
%}
which gives you a function that returns both the bool and the PrimitiveGroup* array as a tuple.
Does that help?
It's actually so easy to make python bindings for things directly that I don't know why people bother with confusing wrapper stuff like SWIG.
Just use Py_BuildValue once per element of the outer array, producing one tuple per row. Store those tuples in a C array. Then Call PyList_New and PyList_SetSlice to generate a list of tuples, and return the list pointer from your C function.
I don't know how to do it with SWIG, but you might want to consider moving to a more modern binding system like Pyrex or Cython.
For example, Pyrex gives you access to C++ delete for cases like this. Here's an excerpt from the documentation:
Disposal
The del statement can be applied to a pointer to a C++ struct
to deallocate it. This is equivalent to delete in C++.
cdef Shrubbery *big_sh
big_sh = new Shrubbery(42.0)
display_in_garden_show(big_sh)
del big_sh
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/using_with_c++.html