I use SWIG for generating wrappers. Therefore I need a function which looks like
%inline %{
// Serializes into a string
void* SerCmd(Class *v, int *length, char *str)
{
QByteArray ba;
QDataStream out(&ba, QIODevice::WriteOnly);
out << *v;
*length = ba.size();
str = new char[ba.size()];
memcpy(str, ba.constData(), ba.size());
return str;
}
%}
This function is called from python then but who is deleting the memory I allocate with new? Is python doing that for me or how can that be achieved?
Thanks!
If this doesn't answer your question, I will remove it. But according to the SWIG information found here:
http://www.swig.org/Doc1.3/Library.html#Library_stl_cpp_library
a std::string can be used instead of manually allocating memory. Given this information, this more than likely can be used.
%inline %{
// Serializes into a string
void SerCmd(Class *v, int *length, std::string& str)
{
QByteArray ba;
QDataStream out(&ba, QIODevice::WriteOnly);
out << *v;
*length = ba.size();
str.clear();
str.append(ba.constData(), ba.size());
}
%}
Since you noted that the std::string can contain NULLs, then the proper way to handle it is to use the string::append() function.
http://en.cppreference.com/w/cpp/string/basic_string/append
Please note item 4) in the link above (null characters are perfectly ok). Note that std::string does not determine its size by a null character, unlike C-strings.
Now, to get to this data, use the string::data() function, along with the string::size() function to tell you how much data you have in the string.
I don't know SWIG either, but since you asked, "Is python doing that for me or how can that be achieved?"
Python garbage collection deletes stuff when it is no longer in scope i.e. no longer has anything pointing to it. However, it cannot delete stuff it is not aware of. These docs may help.
Here is the doc on how memory management works:
https://docs.python.org/2/c-api/memory.html
And here are the docs to the gc module to help give you a more control of the process.
https://docs.python.org/2/library/gc.html
Related
Question:
After about a week of experimenting, I'm at a fork in the road where I'll either keep trying to get my ctype implementation working, or see if there's a less complicated approach. I am mainly looking for feedback or opinions on if ctypes is the right tool for the job, or maybe there's an easier path, such as one of the options mentioned in this post. Given that I only need structure definitions and will not actually call any of the C functions exposed in the header, maybe one of the options mentioned above would be the path of least resistance. I'm hoping someone with more experience in this area can nudge me in the right direction.
Background
I have some nested structures defined in a c header file which I am trying to consume from Python. My focus is only on the structures themselves; I do not need to call any c functions defined in that header. I just need the typedefs, associated data structures, and any #defines which those structures depend on, i.e a #define indicating the size of an array, etc.
Goal
The main purpose of this Python script is to parse a C-header file and serialize it into a Python equivalent object, which I can then modify for my own use. For example, if the header defined a struct like this:
#ifndef DIAG_H
#define DIAG_H
#include <stdint.h>
typedef struct
{
uint32_t CounterA;
uint32_t CounterB;
uint32_t CounterC;
uint32_t CounterD;
uint32_t CounterE;
uint32_t WriteCount;
} CounterCfg_t;
typedef struct
{
uint16_t Id;
uint16_t Count;
uint32_t stamp;
float voltage;
CounterCfg_t countCfg;
} Entry_t;
typedef struct
{
uint8_t major;
uint8_t minor;
uint8_t patch;
uint8_t beta;
} Cfg_Version_t;
typedef struct
{
float offsetB;
float offsetA;
} Offsets_t;
typedef struct
{
Offsets_t offsets[10];
} Cal_t;
typedef struct
{
float alpha;
uint32_t crc;
Cal_t cal;
Cfg_Version_t version;
Entry_t logs[40];
} Sample_t;
#endif
I would like to be able to parse this header on the Python side and use the equivalent object like this(pseudo code):
foo = Sample_t()
foo.logs[1].countCfg.WriteCount = 50
foo.cal.offsets[2].offsetB = 0.012345
The header file will change frequently as other devs update it, so writing an object equivalent by hand is not an option, which is why I pursued the parsing option. i.e, if someone updates the header, I simply run this script instead of hand editing.
What I have tried:
I went down the path of using ctypesgen to generate ctypes based wrappers and then importing them into my script. However, things have become pretty complicated. For example, ctypesgen is amazing, and does all the hard work for me, however, I now have a 6K line wrapper that gets generated and pulled in. Looking at this auto-generated file, I just wonder if this is overkill for what I am trying to do.
To demonstrate where I'm at right now, I made a quick barebones example where I try traversing some nested structures. The example includes the ctypesgen generated wrappers, based on a simple header I provided, along with the source to walk the structure. As you can see from the code, this requires me to access protected members such as _fields_, _type_, _length_, _obj, etc, and because of that, it feels like I am doing something wrong in general. That, coupled with my lack of ctype knowledge, has brought me here, to see if there's an easier way. Or maybe a different library for my use case.
My Attempt at walking the nested structures
Thanks!
This (I hope) is quite a simple issue, but despite doing some reading (I'm v. new to SWIG, and fairly green C-wise) I'm just not able to make the "connection" in my head.
I have a function from a library (legacy code, keen not to edit):
extern int myfunction(char *infile, char *maskfile, int check, float *median, char *msg)
My aim is to create a wrapper for this in Python using SWIG.
The values of the median and msg variables are changed by the C function. When the return int != 0 then there will be some error information in the msg arg. Where the return int == 0, then median variable will contain a float with value assigned from myfunction.
This generally runs OK where the return value is 0. I use %array_functions and %pointer_functions to create the pointers needing to be passed, as per this .i file:
%module test
%include "cpointer.i"
%include "carrays.i"
%{
#include <stdint.h>
%}
extern int myfunction(char *infile, char *maskfile, int check, float *median, char *msg)
%pointer_functions(float, floatp);
%pointer_functions(char, charp);
%array_functions(char, charArray);
After swig-ing, compiling and linking, I can call the function in python:
import test
errmsg_buffer = 1024
_infile = 'test2.dat'
infile = imstat.new_charArray(len(_infile))
for i in xrange(len(_infile)):
imstat.charArray_setitem(infile,i,_infile[i])
maskfile = imstat.new_charArray(1)
imstat.charArray_setitem(maskfile,0,'')
check = 0
med = imstat.new_floatp()
errmsg = imstat.new_charArray(errmsg_buffer)
out = test.myfunction(infile,maskfile,check,med,errmsg)
median = test.floatp_value(med)
This works sometimes, but often not - I get a lot of segfaults which are generally fixed by changing the errmsg_buffer length (clearly not a useful fix!). The C code that changes the msg string is:
(void)sprintf(errmsg,"file not found");
My main issue is in proper handling of msg string, which I suspect is causing the segfaults (and might be due to incorrect implementation via new_charArray?).
What is the best way to do this?
Can I add something to the .i that converts the char *msg into a python str?
Can this be done without "pre-initialising" with new_CharArray? I'd presumably get a buffer overflow if errmsg_buffer is too small.
I hope this is clear - happy to add comments for further discussion.
Your wrapper can be much simplified using SWIG. Try this SWIG interface file (details below):
%module test
%include "typemaps.i"
%include "cstring.i"
%apply float *OUTPUT { float *median };
%cstring_bounded_output(char *msg, 1024);
extern int myfunction(char *infile, char *maskfile, int check, float *median, char *msg);
Then, from python, use the module in the following way:
import test
infile = 'test2.dat'
maskfile = ''
check = 0
out, median, errmsg = test.myfunction(infile,maskfile,check)
if out == 0: print(errmsg)
...
However, from what you write, it is not quite clear to me why your approach segfaults.
Details
The typemaps.i file contains the float *OUTPUT typemap, which is then applied to the float *median argument and turns this from an argument into a float output value. See the SWIG docs on argument handling for details.
The cstrings.i file contains SWIG macros to deal with C strings. Here, I used the %cstring_bounded_output macro. This creates a char * buffer of the given size 1024 and passes this as the argument for char *msg automatically. Then, the contents after the function complete are converted into a python string and appended to the output. See here for details.
SWIG handles the first two char * arguments by default, that is converting python strings to appropriate char * and passing these. Note that the passed char * for these arguments are immutable, i.e., if your myfunction attempts to modify these, bad things will happen. Read about how SWIG handles C strings here.
So, your wrapped myfunction then is used as shown above and has the following signature in python:
myfunction(infile, maskfile, check) -> (out, median, msg)
EDIT:
The SWIG docs about carrays.i state:
Note: %array_functions() and %array_class() should not be used with types of char or char *.
I think your code is not creating correctly NULL-terminated C char *, so perhaps this could be causing the segfaults.
I am not learn SWIG very deeply.But I try give you some suggestions.
1.
If your program modifies the input parameter or uses it to return data, consider using the cstring.i library file described in the SWIG Library chapter.
Data is copied into a new Python string and returned.
If your program needs to work with binary data, you can use a typemap to expand a Python string into a pointer/length argument pair. As luck would have it, just such a typemap is already defined. Just do this:
%apply (char *STRING, int LENGTH) { (char *data, int size) };
...
int parity(char *data, int size, int initial);
Python:
parity("e\x09ffss\x00\x00\x01\nx", 0)
If you need to return binary data, you might use the cstring.i library file. The cdata.i library can also be used to extra binary data from arbitrary pointers.
2.I think "pre-initialising" maybe necessary.
I'm using SWIG and my function returns a char *, which was malloc'ed.
SWIG returns PyString_FromStringAndSize(my-char-str, len).
Is there a way to free this my-char-str without editing the C wrapper code?
Use the %newobject directive in your .i file. From the SWIG 2.0 documentation:
If you have a function that allocates memory like this,
char *foo() {
char *result = (char *) malloc(...);
...
return result;
}
then the SWIG generated wrappers will have a memory leak--the returned data will be copied into a string object and the old contents ignored.
To fix the memory leak, use the %newobject directive.
%newobject foo;
...
char *foo();
I was wondering what is the correct way to wrap an array of strings in C to a Python list using SWIG.
The array is inside a struct :
typedef struct {
char** my_array;
char* some_string;
}Foo;
SWIG automatically wraps some_string to a python string.
What should I put in the SWIG interface file so that I can access my_array in Python as a regular Python string list ['string1', 'string2' ] ?
I have used typemap as sugested :
%typemap(python,out) char** {
int len,i;
len = 0;
while ($1[len]) len++;
$result = PyList_New(len);
for (i = 0; i < len; i++) {
PyList_SetItem($result,i,PyString_FromString($1[i]));
}
}
But that still didn't work. In Python, the my_array variable appears as SwigPyObject: _20afba0100000000_p_p_char.
I wonder if that is because the char** is inside a struct? Maybe I need to inform SWIG that?
Any ideas?
I don't think there is a option to handle this conversion automatically in SWIG. You need use Typemap feature of SWIG and write type converter manually. Here you can find a conversion from Python list to char** http://www.swig.org/Doc1.3/Python.html#Python_nn59 so half of job is done. What you need to do right now is to check rest of documentation of Typemap and write converter from char** to Python list.
I am not an expert on this but I think:
%typemap(python,out) char** {
applies to a function that returns char **. Your char ** is inside a structure.. have a look at the code generated by swig to confirm the map got applied or not.
You might have to use something like:
%typemap(python,out) struct Foo {
To have a map that works on a structure Foo that gets returned.
Background: I used the same typemap definition as you used, but then for a char ** successfully.
I am sorry for being slightly off-topic, but if it is an option for you I would strongly recommend using ctypes instead of swig. Here is a related question I asked previously in ctypes context: Passing a list of strings to from python/ctypes to C function expecting char **
I'd like to use some existing C++ code, NvTriStrip, in a Python tool.
SWIG easily handles the functions with simple parameters, but the main function, GenerateStrips, is much more complicated.
What do I need to put in the SWIG interface file to indicate that primGroups is really an output parameter and that it must be cleaned up with delete[]?
///////////////////////////////////////////////////////////////////////////
// GenerateStrips()
//
// in_indices: input index list, the indices you would use to render
// in_numIndices: number of entries in in_indices
// primGroups: array of optimized/stripified PrimitiveGroups
// numGroups: number of groups returned
//
// Be sure to call delete[] on the returned primGroups to avoid leaking mem
//
bool GenerateStrips( const unsigned short* in_indices,
const unsigned int in_numIndices,
PrimitiveGroup** primGroups,
unsigned short* numGroups,
bool validateEnabled = false );
FYI, here is the PrimitiveGroup declaration:
enum PrimType
{
PT_LIST,
PT_STRIP,
PT_FAN
};
struct PrimitiveGroup
{
PrimType type;
unsigned int numIndices;
unsigned short* indices;
PrimitiveGroup() : type(PT_STRIP), numIndices(0), indices(NULL) {}
~PrimitiveGroup()
{
if(indices)
delete[] indices;
indices = NULL;
}
};
Have you looked at the documentation of SWIG regarding their "cpointer.i" and "carray.i" libraries? They're found here. That's how you have to manipulate things unless you want to create your own utility libraries to accompany the wrapped code. Here's the link to the Python handling of pointers with SWIG.
Onto your question on getting it to recognize input versus output. They've got another section in the documentation here, that describes exactly that. You lable things OUTPUT in the *.i file. So in your case you'd write:
%inline{
extern bool GenerateStrips( const unsigned short* in_dices,
const unsigned short* in_numIndices,
PrimitiveGroup** OUTPUT,
unsigned short* numGroups,
bool validated );
%}
which gives you a function that returns both the bool and the PrimitiveGroup* array as a tuple.
Does that help?
It's actually so easy to make python bindings for things directly that I don't know why people bother with confusing wrapper stuff like SWIG.
Just use Py_BuildValue once per element of the outer array, producing one tuple per row. Store those tuples in a C array. Then Call PyList_New and PyList_SetSlice to generate a list of tuples, and return the list pointer from your C function.
I don't know how to do it with SWIG, but you might want to consider moving to a more modern binding system like Pyrex or Cython.
For example, Pyrex gives you access to C++ delete for cases like this. Here's an excerpt from the documentation:
Disposal
The del statement can be applied to a pointer to a C++ struct
to deallocate it. This is equivalent to delete in C++.
cdef Shrubbery *big_sh
big_sh = new Shrubbery(42.0)
display_in_garden_show(big_sh)
del big_sh
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/using_with_c++.html