I have scenario where I need pass around opaque void* pointers through my C++ <-> Python interface implemented based on SWIG (ver 1.3). I am able to return and accept void* in regular functions like this:
void* foo();
void goo( void* );
The problems begins when I try to do the same using generic functions (and this is what I actually need to do). I was able to deal with "foo" scenario above with the function doing essentially something like this (stolen from code generated by SWIG itself for function foo above):
PyObject*
foo(...)
{
if( need to return void* )
return SWIG_NewPointerObj(SWIG_as_voidptr(ptr), SWIGTYPE_p_void, 0 | 0 );
}
I am not convinced this is the best way to do this, but it works. The "goo" scenario is much more troublesome. Here I need to process some generic input and while I can implement conversion logic following the example in the code generated by the SWIG itself for function goo above:
void
goo( PyObject* o )
{
...
if( o is actually void* ) { <==== <1>
void* ptr;
int res = SWIG_ConvertPtr( o, SWIG_as_voidptrptr(&ptr), 0, 0);
if( SWIG_IsOK(res) ) do_goo( ptr );
}
...
}
I do not have any way to implement condition in the line <1>. While for other types I can use functions like: PyInt_Check, PyString_Check etc, for void* I do not have the option to do so. On Python side it is represented by object with type PySwigObject and while it definitely knows that it wraps void* (I can see it if I print the value) I do not know how to access this information from PyObject*.
I would appreciate any help figuring out any way to deal with this problem.
Thank you,
Am I right in thinking you essentially want some way of testing whether the incoming python object is a 'void*'? Presumably this is the same as testing if it is a pointer to anything?
If so, what about using a boost::shared_ptr (or other pointer container) to hold your generic object? This would require some C++ code rewriting though, so might not be feasible.
Related
Question:
After about a week of experimenting, I'm at a fork in the road where I'll either keep trying to get my ctype implementation working, or see if there's a less complicated approach. I am mainly looking for feedback or opinions on if ctypes is the right tool for the job, or maybe there's an easier path, such as one of the options mentioned in this post. Given that I only need structure definitions and will not actually call any of the C functions exposed in the header, maybe one of the options mentioned above would be the path of least resistance. I'm hoping someone with more experience in this area can nudge me in the right direction.
Background
I have some nested structures defined in a c header file which I am trying to consume from Python. My focus is only on the structures themselves; I do not need to call any c functions defined in that header. I just need the typedefs, associated data structures, and any #defines which those structures depend on, i.e a #define indicating the size of an array, etc.
Goal
The main purpose of this Python script is to parse a C-header file and serialize it into a Python equivalent object, which I can then modify for my own use. For example, if the header defined a struct like this:
#ifndef DIAG_H
#define DIAG_H
#include <stdint.h>
typedef struct
{
uint32_t CounterA;
uint32_t CounterB;
uint32_t CounterC;
uint32_t CounterD;
uint32_t CounterE;
uint32_t WriteCount;
} CounterCfg_t;
typedef struct
{
uint16_t Id;
uint16_t Count;
uint32_t stamp;
float voltage;
CounterCfg_t countCfg;
} Entry_t;
typedef struct
{
uint8_t major;
uint8_t minor;
uint8_t patch;
uint8_t beta;
} Cfg_Version_t;
typedef struct
{
float offsetB;
float offsetA;
} Offsets_t;
typedef struct
{
Offsets_t offsets[10];
} Cal_t;
typedef struct
{
float alpha;
uint32_t crc;
Cal_t cal;
Cfg_Version_t version;
Entry_t logs[40];
} Sample_t;
#endif
I would like to be able to parse this header on the Python side and use the equivalent object like this(pseudo code):
foo = Sample_t()
foo.logs[1].countCfg.WriteCount = 50
foo.cal.offsets[2].offsetB = 0.012345
The header file will change frequently as other devs update it, so writing an object equivalent by hand is not an option, which is why I pursued the parsing option. i.e, if someone updates the header, I simply run this script instead of hand editing.
What I have tried:
I went down the path of using ctypesgen to generate ctypes based wrappers and then importing them into my script. However, things have become pretty complicated. For example, ctypesgen is amazing, and does all the hard work for me, however, I now have a 6K line wrapper that gets generated and pulled in. Looking at this auto-generated file, I just wonder if this is overkill for what I am trying to do.
To demonstrate where I'm at right now, I made a quick barebones example where I try traversing some nested structures. The example includes the ctypesgen generated wrappers, based on a simple header I provided, along with the source to walk the structure. As you can see from the code, this requires me to access protected members such as _fields_, _type_, _length_, _obj, etc, and because of that, it feels like I am doing something wrong in general. That, coupled with my lack of ctype knowledge, has brought me here, to see if there's an easier way. Or maybe a different library for my use case.
My Attempt at walking the nested structures
Thanks!
I see a problem from one Stackoverflow question. It is described below:
I have a C++ function which returns a raw float pointer, and another C++ function which accepts a raw float pointer as an argument. Something like:
float* ptr = something;
float* get_ptr(void) { return ptr; }
void use_ptr(float* ptr) { do_work(ptr); }
I want to be able to pass around pointers using Python. Something like this:
import my_native_functions as native
ptr = native.get_ptr()
native.use_ptr(ptr)
I am using pybind11 to create my native python module but I don't know how to create the bindings for the get_ptr() function. If I just do the following:
PYBIND11_MODULE(my_native_functions, m)
{
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
the get_ptr() function returns a Python Float object. I guess this makes sense because there are no pointer types in python. However, because this is now a simple Float, when I call the use_ptr() function and iterate over the pointer in C/C++, only the first element of the array is correct. The rest are garbage. To fix this, in C++, I have to cast my pointer to/from std::size_t. By doing this, everything works just fine.
However, I would like to ask: Is there a "right way" of achieving the above without the casting to/from std::size_t with pybind11?
The above is the whole problem description that comes from AstrOne's question:Returning and passing around raw POD pointers (arrays) with Python, C++, and pybind11.
My question is "To fix this, in C++, I have to cast my pointer to/from std::size_t", how should this cast be done? Can here give me a example or more detailed explanation? I don't quite understand.
Replace the float* type with std::uintptr_t. See below
#include <cstdint>
float* ptr = something;
std::uintptr_t get_ptr() { return reinterpret_cast<std::uintptr_t>(ptr); }
void use_ptr(std::uintptr_t ptr) { do_work(reinterpret_cast<float*>(ptr)); }
No pointer types will be exposed to pybind11 now. What is doing here is just cast a pointer type to a integral type, which has no difference in binary representation.
Aside: std::size_t would be Ok in most platforms. As per cppref:
On many platforms (an exception is systems with segmented addressing) std::size_t can safely store the value of any non-member pointer, in which case it is synonymous with std::uintptr_t.
I'm trying to use pybind11 to bind a struct that looks like this
struct myStruct {
int na;
int nb;
double* a;
double* b;
}
I'm not sure the right way to go about it. The examples in the pybind11 documentation show how to attach buffer protocol semantics to an object, but not to a class member.
I don't have the luxury of changing the interface of myStruct to contain std::vectors either which would allow me to use the usual .def_readwrite().
I've tried doing something like this
py::class_<myStruct>(m, "myStruct")
.def_property("a",
[](myStruct &s) {return py::array<double>({s.na}, {sizeof(double), s.a};)},
[](myStruct &s, py::array_t<double> val) {std::copy((double*) val.request().ptr, (double*) val.request().ptr + s.na, s.a);)}
)
Which compiles, but in python I don't see changes persist in the underlying data
print(my_struct.a[0]) # prints 0.0
my_struct.a[0] = 123.0
print(my_struct.a[0]) # still prints 0.0
Hey most likely not the most elegant answer, but maybe it gives you a starting point and temporary solution. I think what you need to do is use shared pointers.
Under https://github.com/pybind/pybind11/issues/1150 someone asked something similar but I was not able to adapt it to your example and only got the same result to yours with no changes to the data.
What worked for me in your specific example was using the shared_ptr and defining setter and getter functions for the pointers with a simple def_property for the pybin11 class.
class class_DATA{
public:
int na;
std::shared_ptr<double> a;
void set_a(double a){*class_DATA::a = a; };
double get_a(void){return *class_DATA::a; };
};
PYBIND11_MODULE(TEST,m){
m.doc() = "pybind11 example plugin";
//the costum class
py::class_<class_DATA>(m, "class_DATA", py::dynamic_attr())
.def(py::init<>()) //needed to define constructor
.def_readwrite("na", &class_DATA::na)
.def_property("a", &class_DATA::get_a, &class_DATA::set_a, py::return_value_policy::copy);
}
For a non template class I would write something like that
But I don't know what should I do if my class is a template class.
I've tried something like that and it's not working.
extern "C" {
Demodulator<double>* Foo_new_double(){ return new Demodulator<double>(); }
Demodulator<float>* Foo_new_float(){ return new Demodulator<float>(); }
void demodulateDoubleMatrix(Demodulator<double>* demodulator, double * input, int rows, int columns){ demodulator->demodulateMatrixPy(input, rows, columns) }
}
Note: Your question contradicts the code partially, so I'm ignoring the code for now.
C++ templates are an elaborated macro mechanism that gets resolved at compile time. In other words, the binary only contains the code from template instantiations (which is what you get when you apply parameters, typically types, to the the template), and those are all that you can export from a binary to other languages. Exporting them is like exporting any regular type, see for example std::string.
Since the templates themselves don't survive compilation, there is no way that you can export them from a binary, not to C, not to Python, not even to C++! For the latter, you can provide the templates themselves though, but that doesn't include them in a binary.
Two assumptions:
Exporting/importing works via binaries. Of course, you could write an import that parses C++.
C++ specifies (or specified?) export templates, but as far as I know, this isn't really implemented in the wild, so I left that option out.
The C++ language started as a superset of C: That is, it contains new keywords, syntax and capabilities that C does not provide. C does not have the concept of a class, has no concept of a member function and does not support the concept of access restrictions. C also does not support inheritance. The really big difference, however, is templates. C has macros, and that's it.
Therefore no, you can't directly expose C++ code to C in any fashion, you will have to use C-style code in your C++ to expose the C++ layer.
template<T> T foo(T i) { /* ... */ }
extern "C" int fooInt(int i) { return foo(i); }
However C++ was originally basically a C code generator, and C++ can still interop (one way) with the C ABI: member functions are actually implemented by turning this->function(int arg); into ThisClass0int1(this, arg); or something like that. In theory, you could write something to do this to your code, perhaps leveraging clang.
But that's a non-trivial task, something that's already well-tackled by SWIG, Boost::Python and Cython.
The problem with templates, however, is that the compiler ignores them until you "instantiate" (use) them. std::vector<> is not a concrete thing until you specify std::vector<int> or something. And now the only concrete implementation of that is std::vector<int>. Until you've specified it somewhere, std::vector<string> doesn't exist in your binary.
You probably want to start by looking at something like this http://kos.gd/2013/01/5-ways-to-use-python-with-native-code/, select a tool, e.g. SWIG, and then start building an interface to expose what you want/need to C. This is a lot less work than building the wrappers yourself. Depending which tool you use, it may be as simple as writing a line saying using std::vector<int> or typedef std::vector<int> IntVector or something.
---- EDIT ----
The problem with a template class is that you are creating an entire type that C can't understand, consider:
template<typename T>
class Foo {
T a;
int b;
T c;
public:
Foo(T a_) : a(a_) {}
void DoThing();
T GetA() { return a; }
int GetB() { return b; }
T GetC() { return c; }
};
The C language doesn't support the class keyword, never mind understand that members a, b and c are private, or what a constructor is, and C doesn't understand member functions.
Again it doesn't understand templates so you'll need to do what C++ does automatically and generate an instantiation, by hand:
struct FooDouble {
double a;
int b;
double c;
};
Except, all of those variables are private. So do you really want to be exposing them? If not, you probably just need to typedef "FooDouble" to something the same size as Foo and make a macro to do that.
Then you need to write replacements for the member functions. C doesn't understand constructors, so you will need to write a
extern "C" FooDouble* FooDouble_construct(double a);
FooDouble* FooDouble_construct(double a) {
Foo* foo = new Foo(a);
return reinterept_cast<FooDouble*>(foo);
}
and a destructor
extern "C" void FooDouble_destruct(FooDouble* foo);
void FooDouble_destruct(FooDouble* foo) {
delete reinterpret_cast<Foo*>(foo);
}
and a similar pass-thru for the accessors.
I'd like to use some existing C++ code, NvTriStrip, in a Python tool.
SWIG easily handles the functions with simple parameters, but the main function, GenerateStrips, is much more complicated.
What do I need to put in the SWIG interface file to indicate that primGroups is really an output parameter and that it must be cleaned up with delete[]?
///////////////////////////////////////////////////////////////////////////
// GenerateStrips()
//
// in_indices: input index list, the indices you would use to render
// in_numIndices: number of entries in in_indices
// primGroups: array of optimized/stripified PrimitiveGroups
// numGroups: number of groups returned
//
// Be sure to call delete[] on the returned primGroups to avoid leaking mem
//
bool GenerateStrips( const unsigned short* in_indices,
const unsigned int in_numIndices,
PrimitiveGroup** primGroups,
unsigned short* numGroups,
bool validateEnabled = false );
FYI, here is the PrimitiveGroup declaration:
enum PrimType
{
PT_LIST,
PT_STRIP,
PT_FAN
};
struct PrimitiveGroup
{
PrimType type;
unsigned int numIndices;
unsigned short* indices;
PrimitiveGroup() : type(PT_STRIP), numIndices(0), indices(NULL) {}
~PrimitiveGroup()
{
if(indices)
delete[] indices;
indices = NULL;
}
};
Have you looked at the documentation of SWIG regarding their "cpointer.i" and "carray.i" libraries? They're found here. That's how you have to manipulate things unless you want to create your own utility libraries to accompany the wrapped code. Here's the link to the Python handling of pointers with SWIG.
Onto your question on getting it to recognize input versus output. They've got another section in the documentation here, that describes exactly that. You lable things OUTPUT in the *.i file. So in your case you'd write:
%inline{
extern bool GenerateStrips( const unsigned short* in_dices,
const unsigned short* in_numIndices,
PrimitiveGroup** OUTPUT,
unsigned short* numGroups,
bool validated );
%}
which gives you a function that returns both the bool and the PrimitiveGroup* array as a tuple.
Does that help?
It's actually so easy to make python bindings for things directly that I don't know why people bother with confusing wrapper stuff like SWIG.
Just use Py_BuildValue once per element of the outer array, producing one tuple per row. Store those tuples in a C array. Then Call PyList_New and PyList_SetSlice to generate a list of tuples, and return the list pointer from your C function.
I don't know how to do it with SWIG, but you might want to consider moving to a more modern binding system like Pyrex or Cython.
For example, Pyrex gives you access to C++ delete for cases like this. Here's an excerpt from the documentation:
Disposal
The del statement can be applied to a pointer to a C++ struct
to deallocate it. This is equivalent to delete in C++.
cdef Shrubbery *big_sh
big_sh = new Shrubbery(42.0)
display_in_garden_show(big_sh)
del big_sh
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/using_with_c++.html