Develop Reference

Trouble with parsing structures from a C-header

Question:
After about a week of experimenting, I'm at a fork in the road where I'll either keep trying to get my ctype implementation working, or see if there's a less complicated approach. I am mainly looking for feedback or opinions on if ctypes is the right tool for the job, or maybe there's an easier path, such as one of the options mentioned in this post. Given that I only need structure definitions and will not actually call any of the C functions exposed in the header, maybe one of the options mentioned above would be the path of least resistance. I'm hoping someone with more experience in this area can nudge me in the right direction.
Background
I have some nested structures defined in a c header file which I am trying to consume from Python. My focus is only on the structures themselves; I do not need to call any c functions defined in that header. I just need the typedefs, associated data structures, and any #defines which those structures depend on, i.e a #define indicating the size of an array, etc.
Goal
The main purpose of this Python script is to parse a C-header file and serialize it into a Python equivalent object, which I can then modify for my own use. For example, if the header defined a struct like this:
#ifndef DIAG_H
#define DIAG_H
#include <stdint.h>
typedef struct
{
uint32_t CounterA;
uint32_t CounterB;
uint32_t CounterC;
uint32_t CounterD;
uint32_t CounterE;
uint32_t WriteCount;
} CounterCfg_t;
typedef struct
{
uint16_t Id;
uint16_t Count;
uint32_t stamp;
float voltage;
CounterCfg_t countCfg;
} Entry_t;
typedef struct
{
uint8_t major;
uint8_t minor;
uint8_t patch;
uint8_t beta;
} Cfg_Version_t;
typedef struct
{
float offsetB;
float offsetA;
} Offsets_t;
typedef struct
{
Offsets_t offsets[10];
} Cal_t;
typedef struct
{
float alpha;
uint32_t crc;
Cal_t cal;
Cfg_Version_t version;
Entry_t logs[40];
} Sample_t;
#endif
I would like to be able to parse this header on the Python side and use the equivalent object like this(pseudo code):
foo = Sample_t()
foo.logs[1].countCfg.WriteCount = 50
foo.cal.offsets[2].offsetB = 0.012345
The header file will change frequently as other devs update it, so writing an object equivalent by hand is not an option, which is why I pursued the parsing option. i.e, if someone updates the header, I simply run this script instead of hand editing.
What I have tried:
I went down the path of using ctypesgen to generate ctypes based wrappers and then importing them into my script. However, things have become pretty complicated. For example, ctypesgen is amazing, and does all the hard work for me, however, I now have a 6K line wrapper that gets generated and pulled in. Looking at this auto-generated file, I just wonder if this is overkill for what I am trying to do.
To demonstrate where I'm at right now, I made a quick barebones example where I try traversing some nested structures. The example includes the ctypesgen generated wrappers, based on a simple header I provided, along with the source to walk the structure. As you can see from the code, this requires me to access protected members such as _fields_, _type_, _length_, _obj, etc, and because of that, it feels like I am doing something wrong in general. That, coupled with my lack of ctype knowledge, has brought me here, to see if there's an easier way. Or maybe a different library for my use case.
My Attempt at walking the nested structures
Thanks!

How to make use of existing C structures in Python?

I have a couple of header files that are already defined in C (C++ if we're being technical, but the header is C compatible) that I use to define a bunch of data types (structs). I would like to make use of these in a python script that I am going use to test the corresponding C++ application. This is mostly to avoid having to redefine them in python as some of the structs are unwieldy, but also it would be nice to have them defined in one place so if changes happen down the road it will be easier to adapt.
When I started looking into this I thought that this was certainly doable but none of the examples I have come across get me quite there. The closest I got was using cffi. I got a simple example working how I want it to:
Data types header:
// mylib.h
struct Point2D
{
float x;
float y;
};
struct Point3D
{
float x;
float y;
float z;
};
Python code:
from cffi import FFI
with open("./mylib.h", "r") as fo:
header_text = fo.read()
ffi = FFI()
ffi.cdef(header_text)
point = ffi.new("struct Point2D*")
But this fails if I have #includes or #ifdefs in the header file, per the cffi documentation:
The declarations can contain types, functions, constants and global
variables. What you pass to the cdef() must not contain more than
that; in particular, #ifdef or #include directives are not supported.
Are there any tricks I can do to make this work?

You cannot directly access C structs in Python. You will need to 'bind' C functions to Python functions. This only allows you to access C functions from Python - not a C struct.
Testing C++ is generally done using Google Test. If you require using Python to test C++ functionality then you will need to create bindings in Python to access the C++ functions (as C functions using extern "C").
You can only bind to a C/C++ library. Google "Call C functions in Python" for more.

How to include multiple language development header in C++

I want to include the Python.h and the ruby.h header in the same C/C++ file, as I want to work with both at the same time. Which would be the best way to include both and prevent the compiler/preprocessor to warn about multiple redefinitions of the same variables or there is another way to use those languages from C?
MWE:
// file.cpp
#include <iostream>
#include <Python.h>
#include <ruby.h>
int main() {
return 0;
}
I warnings like this from the preprocessor:
In file included from /path/to/Python/include/Python.h:8,
from /path/to/file.cpp:4:
/path/to/Python/include/pyconfig.h:61: warning: "HAVE_HYPOT" redefined
#define HAVE_HYPOT
In file included from /path/to/Ruby/include/ruby-2.7.0/ruby/ruby.h:24,
from /path/to/Ruby/include/ruby-2.7.0/ruby.h:33,
from /path/to/file.cpp:5:
/path/to/Ruby/include/ruby-2.7.0/x64-mingw32/ruby/config.h:211: note: this is the location of the previous definition
#define HAVE_HYPOT 1

An example to demonstrate not getting complaint for redefinitions in headers.
Though you may have to re-declare every functions you need in a wrapper header.
P.h
#define AAA 2
R.h
#define AAA 1
Pwrapper.h
int get(void);
Pwrapper.cc
#inclide "P.h" //include here, so Pwrapper.h won't conflict definition with R.h
int get(void){
return AAA;
}
In your actual source file for operations
#include "Pwrapper.h"
#include "R.h"

Not sure whats in python.h but since python is an interpreted language, Id assume python.h references object code that interprets python code which is held as a text data in your c++ code as in
char * pythonScript = "print \"Hello, World!\"";
pythonExec(pythonScript);
Python is an interpreted language (at least on linux) and is also a foreign languge to a c++ compiler.
The only time Ive seen c++ directly support a foreign language is the implementation dependent asm keyword where some compilers allow you to write assembly language code blocks directly into the C++ source code. Its not supported by all compilers and asm is the only time Ive seen this style of support.
Debateably, things like opengl are languages in their own right and there is a way of foreign language support in the sense that each opengl function and variable is replicated with a c++ function or variable to the extent that the full language is mapped into c++.
Sorry I dont have a full answer but given that I think what youre trying to do isnt really supported, I hope it provides you some pointers to where you should be looking.
Maybe someone has a better answer.

Edit:
This does not work (as alluded in a comment) for preprocessor macros, which are not #undef'd, so does not answer the question exactly.
Original answer
Not sure about linking / global variables within implementation files, or about what your files contain, but for header files, you can put the include macro within a namespace:
namespace a {
#include Python.h
}
namespace b {
#include ruby.h
}
Then you reference the proper namespace to use a variable:
a::SAME_NAME
b::SAME_NAME

Reusing SWIG mappings in custom typemap

I'm currently working on a Python wrapper for a C++ library for which I want to use SWIG. In my C++ library I have a method with the following signature:
std::vector<SomeClass> getMembers();
Now I know that SWIG has built-in std::vector support but I want to explicitly convert std::vectors to Python Lists (I just think it is cleaner). For that I have the following typemap:
template<typename T>
PyObject* toList(vector<T> vec){
size_t size = vec.size();
PyObject *o = PyList_New(size);
for(size_t i = 0; i < size; i++){
PyList_SetItem(o, i, toPythonInstance<T>(vec[i]));
}
return o;
}
%define OUTPUT_VEC_TO_LIST(type)
%typemap (out) std::vector<type> {
$result = toList<type>($1);
}
%enddef
Now the template method:
template<T>
PyObject* toPythonInstance(T& val){}
Can be specialized to add support for the necessary datatypes. The problem I'm facing now is the following:
SomeClass is wrapped automatically by SWIG. So what I like to do is to reuse this wrapper in my vector type map, i.e. have the following:
template<>
PyObject* toPythonInstance<SomeClass>(SomeClass& val){
//call some SWIG macro to automatically wrap the given instance to
//a Python object
}
Inspecting the code generated by SWIG, I already found the following functions
SWIG_NewPointerObj(...);
SWIG_ConvertPtr(...);
which seem to be responsible for doing exactly what I want. However, I do not want to interfere with any internals of SWIG. So if somebody knows how to achieve what I want with the "public" SWIG interface, I would be very glad!

SWIG actually makes a whole bunch of runtime information part of an external interface, see http://www.swig.org/Doc3.0/Modules.html#Modules_external_run_time for details. This includes the functions you're likely to want for your efforts.
I disagree with your assessment that mapping std::vector to list is cleaner in Python - you always end up copying and visiting every member of that vector to do this. In effect you make a copy of the original container and end up with two containers, so changes to the Python list won't be reflected back on the underlying C++ container. The Python supplied std::vector wrapping should implement the protocols you care about to enable pythonic syntax by default, and can support the ABCs correctly too.. (And if they don't I'm up for writing patches!)

Calling C functions in Python

I have a bunch of functions that I've written in C and I'd like some code I've written in Python to be able to access those functions.
I've read several questions on here that deal with a similar problem (here and here for example) but I'm confused about which approach I need to take.
One question recommends ctypes and another recommends cython. I've read a bit of the documentation for both, and I'm completely unclear about which one will work better for me.
Basically I've written some python code to do some two dimensional FFTs and I'd like the C code to be able to see that result and then process it through the various C functions I've written. I don't know if it will be easier for me to call the Python from C or vice versa.

You should call C from Python by writing a ctypes wrapper. Cython is for making python-like code run faster, ctypes is for making C functions callable from python. What you need to do is the following:
Write the C functions you want to use. (You probably did this already)
Create a shared object (.so, for linux, os x, etc) or dynamically loaded library (.dll, for windows) for those functions. (Maybe you already did this, too)
Write the ctypes wrapper (It's easier than it sounds, I wrote a how-to for that)
Call a function from that wrapper in Python. (This is just as simple as calling any other python function)

If I understand well, you have no preference for dialoging as c => python or like python => c.
In that case I would recommend Cython. It is quite open to many kinds of manipulation, specially, in your case, calling a function that has been written in Python from C.
Here is how it works (public api) :
The following example assumes that you have a Python Class (self is an instance of it), and that this class has a method (name method) you want to call on this class and deal with the result (here, a double) from C. This function, written in a Cython extension would help you to do this call.
cdef public api double cy_call_func_double(object self, char* method, bint *error):
if (hasattr(self, method)):
error[0] = 0
return getattr(self, method)();
else:
error[0] = 1
On the C side, you'll then be able to perform the call like so :
PyObject *py_obj = ....
...
if (py_obj) {
int error;
double result;
result = cy_call_func_double(py_obj, (char*)"initSimulation", &error);
cout << "Do something with the result : " << result << endl;
}
Where PyObject is a struct provided by Python/C API
After having caught the py_obj (by casting a regular python object, in your cython extension like this : <PyObject *>my_python_object), you would finally be able to call the initSimulation method on it and do something with the result.
(Here a double, but Cython can deal easily with vectors, sets, ...)
Well, I am aware that what I just wrote can be confusing if you never wrote anything using Cython, but it aims to be a short demonstration of the numerous things it can do for you in term of merging.
By another hand, this approach can take more time than recoding your Python code into C, depending on the complexity of your algorithms.
In my opinion, investing time into learning Cython is pertinent only if you plan to have this kind of needs quite often...
Hope this was at least informative...

Well, here you are referring to two below things.
How to call c function within from python (Extending python)
How to call python function/script from C program (Embedding Python)
For #2 that is 'Embedding Python'
You may use below code segment:
#include "python.h"
int main(int argc, char *argv[]) {
Py_SetProgramName(argv[0]); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print 'Today is',ctime(time())\n");
/*Or if you want to run python file within from the C code*/
//pyRun_SimpleFile("Filename");
Py_Finalize();
return 0; }
For #1 that is 'Extending Python'
Then best bet would be to use Ctypes (btw portable across all variant of python).
>> from ctypes import *
>> libc = cdll.msvcrt
>> print libc.time(None)
>> 1438069008
>> printf = libc.printf
>> printf("Hello, %s\n", "World!")
>> Hello, World!
14
>> printf("%d bottles of beer\n", 42)
>> 42 bottles of beer
19
For detailed guide you may want to refer to my blog article:

It'll be easier to call C from python. Your scenario sounds weird - normally people write most of the code in python except for the processor-intensive portion, which is written in C. Is the two-dimensional FFT the computationally-intensive part of your code?

There's a nice and brief tutorial on this from Digital Ocean here. Short version:
1. Write C Code
You've already done this, so super short example:
#include <stdio.h>
int addFive(int i) {
return i + 5;
}
2. Create Shared Library File
Assuming the above C file is saved as c_functions.c, then to generate the .so file to call from python type in your terminal:
cc -fPIC -shared -o c_functions.so c_functions.c
3. Use Your C Code in Python!
Within your python module:
# Access your C code
from ctypes import *
so_file = "./c_functions.so"
c_functions = CDLL(so_file)
# Use your C code
c_functions.addFive(10)
That last line will output 15. You're done!

The answer from BLimitless quoting Digital Ocean is fine for simple cases, but it defaults to allowing int type arguments only. If need a different type for your input argument, for example to a float type, you need to add this:
c_functions.addFive.argtypes=[ctypes.c_float]
And if you change the output argument, for example to a float type you need this:
c_functions.addFive.restype=ctypes.c_float