Question:
After about a week of experimenting, I'm at a fork in the road where I'll either keep trying to get my ctype implementation working, or see if there's a less complicated approach. I am mainly looking for feedback or opinions on if ctypes is the right tool for the job, or maybe there's an easier path, such as one of the options mentioned in this post. Given that I only need structure definitions and will not actually call any of the C functions exposed in the header, maybe one of the options mentioned above would be the path of least resistance. I'm hoping someone with more experience in this area can nudge me in the right direction.
Background
I have some nested structures defined in a c header file which I am trying to consume from Python. My focus is only on the structures themselves; I do not need to call any c functions defined in that header. I just need the typedefs, associated data structures, and any #defines which those structures depend on, i.e a #define indicating the size of an array, etc.
Goal
The main purpose of this Python script is to parse a C-header file and serialize it into a Python equivalent object, which I can then modify for my own use. For example, if the header defined a struct like this:
#ifndef DIAG_H
#define DIAG_H
#include <stdint.h>
typedef struct
{
uint32_t CounterA;
uint32_t CounterB;
uint32_t CounterC;
uint32_t CounterD;
uint32_t CounterE;
uint32_t WriteCount;
} CounterCfg_t;
typedef struct
{
uint16_t Id;
uint16_t Count;
uint32_t stamp;
float voltage;
CounterCfg_t countCfg;
} Entry_t;
typedef struct
{
uint8_t major;
uint8_t minor;
uint8_t patch;
uint8_t beta;
} Cfg_Version_t;
typedef struct
{
float offsetB;
float offsetA;
} Offsets_t;
typedef struct
{
Offsets_t offsets[10];
} Cal_t;
typedef struct
{
float alpha;
uint32_t crc;
Cal_t cal;
Cfg_Version_t version;
Entry_t logs[40];
} Sample_t;
#endif
I would like to be able to parse this header on the Python side and use the equivalent object like this(pseudo code):
foo = Sample_t()
foo.logs[1].countCfg.WriteCount = 50
foo.cal.offsets[2].offsetB = 0.012345
The header file will change frequently as other devs update it, so writing an object equivalent by hand is not an option, which is why I pursued the parsing option. i.e, if someone updates the header, I simply run this script instead of hand editing.
What I have tried:
I went down the path of using ctypesgen to generate ctypes based wrappers and then importing them into my script. However, things have become pretty complicated. For example, ctypesgen is amazing, and does all the hard work for me, however, I now have a 6K line wrapper that gets generated and pulled in. Looking at this auto-generated file, I just wonder if this is overkill for what I am trying to do.
To demonstrate where I'm at right now, I made a quick barebones example where I try traversing some nested structures. The example includes the ctypesgen generated wrappers, based on a simple header I provided, along with the source to walk the structure. As you can see from the code, this requires me to access protected members such as _fields_, _type_, _length_, _obj, etc, and because of that, it feels like I am doing something wrong in general. That, coupled with my lack of ctype knowledge, has brought me here, to see if there's an easier way. Or maybe a different library for my use case.
My Attempt at walking the nested structures
Thanks!
Related
I'm working on building a library in rust that I think would be extremely useful in other languages. I would like to expose this functionality with idiomatic bindings to as many languages as possible with as little effort as I can get away with. Obviously SWIG is a great choice for this project.
I'm using a fantastic project called safer_ffi to produce the C interface to the rust library. It removes a lot of the error prone boiler plate on the rust side but also limits my options on exactly what the C interface looks like. Currently it represents strings with this C type:
typedef struct {
uint8_t * ptr;
size_t len;
} slice_boxed_uint8_t;
I can't for the life of me set the ptr member of the struct without causing a TypeError in python. My interface file is simply:
%module swig_example
%{
/* Includes the header in the wrapper code */
#include "swig_example.h"
%}
%include "stdint.i"
%include "cstring.i"
/* Parse the header file to generate wrappers */
%include "swig_example.h"
and I try and set up the struct with the following python:
def _str_to_slice(input: str) -> slice_boxed_uint8_t:
slice = slice_boxed_uint8_t()
slice.ptr = input
slice.len = len(input)
return slice
which produces the following error "TypeError: in method 'slice_boxed_uint8_t_ptr_set', argument 2 of type 'uint8_t '". I have tried all sorts of combinations of how to invoke it and how to generate the bindings. I've been walking through the generated C code but haven't found the issue yet. It looks like it understands that this pointer is a char but isn't making the connection that it is okay to use as a uint8_t*. I might have misunderstood some of the generated C code, I'm still not very deep on walking through that yet.
I did my best to include all relevant info but I know I might be missing some important context in this post so the code can be found here. The README.md points out here to find all relevant files, reasoning on how things are set up and I checked in the SWIG generated c and python files. This project is the smallest subset of my original project I could make to it easier for others to troubleshoot
Huge thank you to any help that anyone can provide!
For a non template class I would write something like that
But I don't know what should I do if my class is a template class.
I've tried something like that and it's not working.
extern "C" {
Demodulator<double>* Foo_new_double(){ return new Demodulator<double>(); }
Demodulator<float>* Foo_new_float(){ return new Demodulator<float>(); }
void demodulateDoubleMatrix(Demodulator<double>* demodulator, double * input, int rows, int columns){ demodulator->demodulateMatrixPy(input, rows, columns) }
}
Note: Your question contradicts the code partially, so I'm ignoring the code for now.
C++ templates are an elaborated macro mechanism that gets resolved at compile time. In other words, the binary only contains the code from template instantiations (which is what you get when you apply parameters, typically types, to the the template), and those are all that you can export from a binary to other languages. Exporting them is like exporting any regular type, see for example std::string.
Since the templates themselves don't survive compilation, there is no way that you can export them from a binary, not to C, not to Python, not even to C++! For the latter, you can provide the templates themselves though, but that doesn't include them in a binary.
Two assumptions:
Exporting/importing works via binaries. Of course, you could write an import that parses C++.
C++ specifies (or specified?) export templates, but as far as I know, this isn't really implemented in the wild, so I left that option out.
The C++ language started as a superset of C: That is, it contains new keywords, syntax and capabilities that C does not provide. C does not have the concept of a class, has no concept of a member function and does not support the concept of access restrictions. C also does not support inheritance. The really big difference, however, is templates. C has macros, and that's it.
Therefore no, you can't directly expose C++ code to C in any fashion, you will have to use C-style code in your C++ to expose the C++ layer.
template<T> T foo(T i) { /* ... */ }
extern "C" int fooInt(int i) { return foo(i); }
However C++ was originally basically a C code generator, and C++ can still interop (one way) with the C ABI: member functions are actually implemented by turning this->function(int arg); into ThisClass0int1(this, arg); or something like that. In theory, you could write something to do this to your code, perhaps leveraging clang.
But that's a non-trivial task, something that's already well-tackled by SWIG, Boost::Python and Cython.
The problem with templates, however, is that the compiler ignores them until you "instantiate" (use) them. std::vector<> is not a concrete thing until you specify std::vector<int> or something. And now the only concrete implementation of that is std::vector<int>. Until you've specified it somewhere, std::vector<string> doesn't exist in your binary.
You probably want to start by looking at something like this http://kos.gd/2013/01/5-ways-to-use-python-with-native-code/, select a tool, e.g. SWIG, and then start building an interface to expose what you want/need to C. This is a lot less work than building the wrappers yourself. Depending which tool you use, it may be as simple as writing a line saying using std::vector<int> or typedef std::vector<int> IntVector or something.
---- EDIT ----
The problem with a template class is that you are creating an entire type that C can't understand, consider:
template<typename T>
class Foo {
T a;
int b;
T c;
public:
Foo(T a_) : a(a_) {}
void DoThing();
T GetA() { return a; }
int GetB() { return b; }
T GetC() { return c; }
};
The C language doesn't support the class keyword, never mind understand that members a, b and c are private, or what a constructor is, and C doesn't understand member functions.
Again it doesn't understand templates so you'll need to do what C++ does automatically and generate an instantiation, by hand:
struct FooDouble {
double a;
int b;
double c;
};
Except, all of those variables are private. So do you really want to be exposing them? If not, you probably just need to typedef "FooDouble" to something the same size as Foo and make a macro to do that.
Then you need to write replacements for the member functions. C doesn't understand constructors, so you will need to write a
extern "C" FooDouble* FooDouble_construct(double a);
FooDouble* FooDouble_construct(double a) {
Foo* foo = new Foo(a);
return reinterept_cast<FooDouble*>(foo);
}
and a destructor
extern "C" void FooDouble_destruct(FooDouble* foo);
void FooDouble_destruct(FooDouble* foo) {
delete reinterpret_cast<Foo*>(foo);
}
and a similar pass-thru for the accessors.
I am writing a Python app that makes use of PulseAudio API. The implementation is heavily using callbacks written in Python and invoked by PulseAudio's C code.
The most information is passed into the callback by a specific structure, for instance pa_sink_info, which is defined in C as follows:
typedef struct pa_sink_info {
const char *name;
uint32_t index;
const char *description;
pa_sample_spec sample_spec;
pa_channel_map channel_map;
uint32_t owner_module;
pa_cvolume volume;
int mute;
uint32_t monitor_source;
const char *monitor_source_name;
pa_usec_t latency;
const char *driver;
pa_sink_flags_t flags;
pa_proplist *proplist;
pa_usec_t configured_latency;
pa_volume_t base_volume;
pa_sink_state_t state;
uint32_t n_volume_steps;
uint32_t card;
uint32_t n_ports;
pa_sink_port_info** ports;
pa_sink_port_info* active_port;
uint8_t n_formats;
pa_format_info **formats;
} pa_sink_info;
From this structure it's very easy to get scalar values, eg.:
self.some_proc(
struct.contents.index,
struct.contents.name,
struct.contents.description)
But I have a difficulty dealing with ports and active_port, which in Python are described as:
('n_ports', uint32_t),
('ports', POINTER(POINTER(pa_sink_port_info))),
('active_port', POINTER(pa_sink_port_info)),
Here n_ports specifies number of elements in ports, which is a pointer to array of pointers to structures of type pa_sink_port_info. Actually, I don't even know how I can convert these to Python types at all.
What is the most efficient way of converting ports into Python dictionary containing pa_sink_port_info's?
Solving this problem required careful reading of Python's ctypes reference. Once the mechanism of ctypes type translation implementation was clear, it's not so difficult to get to the desired values.
The main idea about pointers is that you use their contents attribute to get to the data the pointer points to. Another useful thing to know is that pointers can be indexed like arrays (it's not validated by the interpreter, so it's your own responsibility to make sure it is indeed an array).
For this particular PulseAudio example, we can process the ports structure member (which is a pointer to array of pointers) as follows:
port_list = []
if struct.contents.ports:
i = 0
while True:
port_ptr = struct.contents.ports[i]
# NULL pointer terminates the array
if port_ptr:
port_struct = port_ptr.contents
port_list.append(port_struct.name)
i += 1
else:
break
I've written a high level motor controller in Python, and have got to a point where I want to go a little lower level to get some speed, so I'm interested in coding those bits in C.
I don't have much experience with C, but the math I'm working on is pretty straightforward, so I'm sure I can implement with a minimal amount of banging my head against the wall. What I'm not sure about is how best to invoke this compiled C program in order to pipe it's outputs back into my high-level python controller.
I've used a little bit of ctypes, but only to pull some functions from a manufacfturer-supplied DLL...not sure if that is an appropriate path to go down in this case.
Any thoughts?
You can take a look at this tutorial here.
Also, a more reliable example on the official python website, here.
For example,
sum.h function
int sum(int a, int b)
A file named, module.c,
#include <Python.h>
#include "sum.h"
static PyObject* mod_sum(PyObject *self, PyObject *args)
{
int a;
int b;
int s;
if (!PyArg_ParseTuple(args,"ii",&a,&b))
return NULL;
s = sum(a,b);
return Py_BuildValue("i",s);
}
static PyMethodDef ModMethods[] = {
{"sum", mod_sum, METH_VARARGS, "Description.."},
{NULL,NULL,0,NULL}
};
PyMODINIT_FUNC initmod(void)
{
PyObject *m;
m = Py_InitModule("module",ModMethods);
if (m == NULL)
return;
}
Python
import module
s = module.sum(3,5)
Another option: try numba.
It gives you C-like speed for free: just import numba and #autojit your functions, for a wonderful speed increase.
Won't work if you have complicated data types, but if you're looping and jumping around array indices, it might be just what you're looking for.
you can use SWIG, it is very simple to use
You can use Cython for setting the necessary c types and compile your python syntax code.
I have scenario where I need pass around opaque void* pointers through my C++ <-> Python interface implemented based on SWIG (ver 1.3). I am able to return and accept void* in regular functions like this:
void* foo();
void goo( void* );
The problems begins when I try to do the same using generic functions (and this is what I actually need to do). I was able to deal with "foo" scenario above with the function doing essentially something like this (stolen from code generated by SWIG itself for function foo above):
PyObject*
foo(...)
{
if( need to return void* )
return SWIG_NewPointerObj(SWIG_as_voidptr(ptr), SWIGTYPE_p_void, 0 | 0 );
}
I am not convinced this is the best way to do this, but it works. The "goo" scenario is much more troublesome. Here I need to process some generic input and while I can implement conversion logic following the example in the code generated by the SWIG itself for function goo above:
void
goo( PyObject* o )
{
...
if( o is actually void* ) { <==== <1>
void* ptr;
int res = SWIG_ConvertPtr( o, SWIG_as_voidptrptr(&ptr), 0, 0);
if( SWIG_IsOK(res) ) do_goo( ptr );
}
...
}
I do not have any way to implement condition in the line <1>. While for other types I can use functions like: PyInt_Check, PyString_Check etc, for void* I do not have the option to do so. On Python side it is represented by object with type PySwigObject and while it definitely knows that it wraps void* (I can see it if I print the value) I do not know how to access this information from PyObject*.
I would appreciate any help figuring out any way to deal with this problem.
Thank you,
Am I right in thinking you essentially want some way of testing whether the incoming python object is a 'void*'? Presumably this is the same as testing if it is a pointer to anything?
If so, what about using a boost::shared_ptr (or other pointer container) to hold your generic object? This would require some C++ code rewriting though, so might not be feasible.