Our code base currently supports a single SWIG interface file (for Python) that has grown over the years to include roughly 300 C++ classes (technically interfaces), all of which inherit from a single base class, and all of which exist in a single global namespace. This allows us, with a minimal amount of SWIG code, to implement dynamic casting among the C++ classes that the SWIG classes represent while at the same time simplifying by keeping the C++ inheritance structure out of SWIG.
As long as we compiled our SWIG interface in a single module, this mechanism worked well -- but as the SWIG interface file has grown it has become difficult to manage, and compile/link times have grown. To address this I split the interface file up into separate modules by the names of the derived classes -- one module for class names beginning with "A" to "G", one for names beginning with "H" to "N", etc., resulting in four derived-class modules and a base class module. I was able to get these modules to compile and link, and exhibit expected behavior for the dynamic casting, following the method outlined here: (http://www.swig.org/Doc3.0/SWIGDocumentation.html#Modules_nn1)
However, breaking the single module into four parts (five parts counting the base class) causes problems with the namespace when containers come into play. Consider the following function, from a class in my v-to-z interface file:
void RemoveIsolated(const std::vector<global::IFoo*> spRemoveIsolated) {
…
}
That takes a vector of one of the derived classes that exist in the global namespace. This worked without issue when I had only one module but now class IFoo lives in the a-to-g module -- so if I cast something to an IFoo*, it's an a-to-g.IFoo*. However, the function demands a global::IFoo*.
This seems to be a situation that could be addressed by the SWIG template mechanism. I've seen discussions in which people have had success by means of at one point (possibly in the interface file for the base class??) declaring
%template(FooVector) std::vector<global::Foo*>;
And at another point (possibly in the interface file for the derived class??):
%template () std::vector<global::Foo*>;
But my attempts to implement this have not been successful. The discussions are somewhat ambiguous, it's possible that I'm doing something wrong. Can anyone provide clarification, ideally with an example?
The piece of information it looks like you're missing is the %import directive, which lets modules cooperate with the definition of types, without repeating them and still ending up with a single wrapped type. The documentation suggests using this to reduce module size even.
Probably all you need to do is have your v-to-z module %import the a-to-g module to get this working for you. (Personally I'd have tried to divide them up by functionality rather than alphabetically though, so the dependency between then wouldn't be an issue)
Thanks for your suggestion Flexo. Importing the a-to-g module did not work; the C++ compiler complained that all of the classes (interfaces) declared there were not part of the global namespace when it tried to compile to v-to-z wrapper file. However, going through the exercise led me to question why we had been having success previously when we were compiling a single module. It turned out that we were using a typemapping macro in the interface file for the single module that would take a
const std::vector<global::IFoo*>
and map it thusly:
TYPEMAPMACRO(global::IFoo, SWIGTYPE_p_global__IFoo)
for vector containers. The macro itself, for anyone who's interested, is:
%define TYPEMAPMACRO(type, name)
%typemap(in) const std::vector {
/*Check if is a list */
std::vector vec;
void *pobj = 0;
if(PyTuple_Check($input))
{
size_t size = PyTuple_Size($input);
for (size_t j = 0; j < size; j++) {
PyObject *o = PyTuple_GetItem($input, j);
void *argp1 = 0 ;
int res1 = SWIG_ConvertPtr(o, &argp1, name, 0 | 0 );
if (!SWIG_IsOK(res1)) {
SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "Typemap of std::vector" "', argument " "1"" of type '" """'");
}
vec.push_back(reinterpret_cast< type * >(argp1));
}
$1 = vec;
}
else if (SWIG_IsOK(SWIG_ConvertPtr($input, &pobj, name, 0 | 0 ))) {
PyObject *o = $input;
void *argp1 = 0 ;
int res1 = SWIG_ConvertPtr(o, &argp1, name, 0 | 0 );
if (!SWIG_IsOK(res1)) {
SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "Typemap of std::vector" "', argument " "1"" of type '" """'");
}
vec.push_back(reinterpret_cast< type * >(argp1));
$1 = vec;
}
else {
PyErr_SetString(PyExc_TypeError, "not a list");
return NULL;
}
}
%typecheck(SWIG_TYPECHECK_POINTER) std::vector {
void *pobj = 0;
if(!PyTuple_Check($input) && !SWIG_IsOK(SWIG_ConvertPtr($input, &pobj, name, 0 | 0 ))) {
$1 = 0;
PyErr_Clear();
} else {
$1 = 1;
}
}
%enddef
My sense is that this is standard boilerplate stuff, I don't claim to understand it well as it's someone else's code, but what I do understand now that I did not before is that I needed to place the macro for the typemap before the function that uses the typemap (e.g the "RemoveIsolated" example above). That ordering had been broken when I divided my big module up into smaller ones.
Related
I'm exposing an API to Python, written in C++ that I have no access to change, using Boost Python.
I have successfully exposed methods returning references to a std:map where the key,value pairs are value types - eg:
class_< std::map<std::string, std::string> >("StringMap")
.def(map_indexing_suite< std::map<std::string, std::string>, true >());
This works seamlessly. But when trying to achieve a similar result where the map values are pointers to classes I've exposed within the API doesn't work:
struct X_wrap : X, wrapper<X>
{
X_wrap(int i): X(i) {}
// virtual methods here, omitted for brevity - as unlikely to be the issue
}
BOOST_PYTHON_MODULE(my_py_extension)
{
class_< std::map<std::string, X*> >("XPtrMap")
.def(map_indexing_suite< std::map<std::string, X*> >());
class_<X_wrap, boost::noncopyable, bases<XBase> >("X", init<int>())
// other definitions omitted
}
Error seen in g++ 7.3.0:
/usr/include/boost/python/detail/caller.hpp:100:98: error: ‘struct boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<X*>’ has no member named ‘get_pytype’
I understand why the compiler is complaining - the X* in the map needs to be wrapped in a call policy so that it can be returned to Python, just like with a basic method that returns a raw pointer.
My question is what is the best way to do this?
From Googling it strikes that I can perhaps specify a DerivedPolicies child class of map_indexing_suite that will overload the necessary parts to wrap the X* in an appropriate return_value_policy. However so far I've be unsuccessful in putting anything together that the compiler doesn't bawk at!
I also suspect I can literally copy-and-paste the whole map_indexing_suite and rename it, and make the changes therein to produce a new indexing_suite with the right return_value_policy, but this seems ugly compared to the solution using DerviedPolicies - assuming I'm right that DeriviedPolicies can be used at all!
Any help, pointers, or examples gratefully received!
EDIT
I have proved that the cut-and-paste option works with a single trivial change of is_class to is_pointer. It's curious that is_pointer is not allowed in the original as the target policy can handle pointers. I'm yet to convince myself that it's an object lifetime restriction that means pointers are not allowed in the original?
The whole class is public so I suspect it's possible to avoid the full cut-and-paste by simply inheriting from map_indexing_suite or perhaps by using the mysterious DerivedPolicies parameter?
extension_def(Class& cl)
{
// Wrap the map's element (value_type)
std::string elem_name = "mapptr_indexing_suite_";
object class_name(cl.attr("__name__"));
extract<std::string> class_name_extractor(class_name);
elem_name += class_name_extractor();
elem_name += "_entry";
typedef typename mpl::if_<
mpl::and_<is_pointer<data_type>, mpl::bool_<!NoProxy> >
, return_internal_reference<>
, default_call_policies
>::type get_data_return_policy;
class_<value_type>(elem_name.c_str())
.def("__repr__", &DerivedPolicies::print_elem)
.def("data", &DerivedPolicies::get_data, get_data_return_policy())
.def("key", &DerivedPolicies::get_key)
;
}
EDIT 2
Now see answer
Slightly cleaner implementation from the cut-and-paste is to inherit map_indexing_suite - a few tweaks are needed to make this work.
This seems reasonably sensible - if someone chimes in with a neater solution or can better explain DerivedPolicies then great, otherwise I'll accept the below as the answer in a few days or so...
using namespace boost;
using namespace boost::python;
//Forward declaration
template <class Container, bool NoProxy, class DerivedPolicies>
class mapptr_indexing_suite;
template <class Container, bool NoProxy>
class final_mapptr_derived_policies
: public mapptr_indexing_suite<Container,
NoProxy, final_mapptr_derived_policies<Container, NoProxy> > {};
template <
class Container,
bool NoProxy = false,
class DerivedPolicies
= final_mapptr_derived_policies<Container, NoProxy> >
class mapptr_indexing_suite
: public map_indexing_suite<
Container,
NoProxy,
DerivedPolicies
>
{
public:
// Must be explicit if the compiler is
// going to take from the base class
using typename map_indexing_suite<
Container,NoProxy,DerivedPolicies>::data_type;
using typename map_indexing_suite<
Container,NoProxy,DerivedPolicies>::value_type;
// Only one class needs to be overridden from the base
template <class Class>
static void
extension_def(Class& cl)
{
// Wrap the map's element (value_type)
std::string elem_name = "mapptr_indexing_suite_";
object class_name(cl.attr("__name__"));
extract<std::string> class_name_extractor(class_name);
elem_name += class_name_extractor();
elem_name += "_entry";
// use of is_pointer here is the only
// difference to the base map_indexing_suite
typedef typename mpl::if_<
mpl::and_<std::is_pointer<data_type>, mpl::bool_<!NoProxy> >
, return_internal_reference<>
, default_call_policies
>::type get_data_return_policy;
class_<value_type>(elem_name.c_str())
.def("__repr__", &DerivedPolicies::print_elem)
.def("data", &DerivedPolicies::get_data, get_data_return_policy())
.def("key", &DerivedPolicies::get_key)
;
}
};
I am exposing classes from C++ to Python using Boost.Python, for example:
class_<Point>("Point").
def(init<double, double, double>()).
add_property("X", &Point::GetX, &Point::SetX).
add_property("Y", &Point::GetY, &Point::SetY).
add_property("Z", &Point::GetZ, &Point::SetZ).
def("SetXYZ", &Point::SetPnt);
I am also exposing some variables as attributes of my main module:
MainModule.attr("Window") = object(ptr(mainWindow));
Is it possible to list all exposed classes and/or all attributes of a module (in C++)?
I expect to get a list (vector<string>) of all exposed classes: in this case just "Point". The same for exposed variables, in this case just "Window".
Using Python reflection capabilities:
Executing some code from C++:
PyRun_SimpleString("import inspect");
PyRun_SimpleString("import MyModule");
string getClassesCode =
"def GetClasses():\n"
" for name, obj in inspect.getmembers(MyModule):\n"
" if inspect.isclass(obj):\n"
" yield obj.__name__\n"
"_classes = list(GetClasses())\n";
typedef boost::python::list pylist;
object main_namespace = MainModule.attr("__dict__");
try
{
exec(getClassesCode.c_str(), main_namespace);
}
catch (error_already_set const &)
{
PyErr_Print();
}
pylist _classes = extract<pylist>(main_namespace["_classes"]);
for (int i = 0; i < len(_classes); ++i)
{
string className = extract<string>(_classes[i]);
//do whatever you need with the class name
}
Is it possible to list all exposed classes and/or all attributes of a
module (in C++)? I expect to get a list (vector<string>) of all exposed classes ...
I'm not aware of such a feature of Boost.Python, but one alternative, carrying an overhead though, might be to expose a list (you can have it std::vector<string>, why not?) that you prepare "manually" on the C++ side and maintain it as you change your exposure composition. That way, you only have to know the name such list or lists and import them initially to get such a functionality.
For a non template class I would write something like that
But I don't know what should I do if my class is a template class.
I've tried something like that and it's not working.
extern "C" {
Demodulator<double>* Foo_new_double(){ return new Demodulator<double>(); }
Demodulator<float>* Foo_new_float(){ return new Demodulator<float>(); }
void demodulateDoubleMatrix(Demodulator<double>* demodulator, double * input, int rows, int columns){ demodulator->demodulateMatrixPy(input, rows, columns) }
}
Note: Your question contradicts the code partially, so I'm ignoring the code for now.
C++ templates are an elaborated macro mechanism that gets resolved at compile time. In other words, the binary only contains the code from template instantiations (which is what you get when you apply parameters, typically types, to the the template), and those are all that you can export from a binary to other languages. Exporting them is like exporting any regular type, see for example std::string.
Since the templates themselves don't survive compilation, there is no way that you can export them from a binary, not to C, not to Python, not even to C++! For the latter, you can provide the templates themselves though, but that doesn't include them in a binary.
Two assumptions:
Exporting/importing works via binaries. Of course, you could write an import that parses C++.
C++ specifies (or specified?) export templates, but as far as I know, this isn't really implemented in the wild, so I left that option out.
The C++ language started as a superset of C: That is, it contains new keywords, syntax and capabilities that C does not provide. C does not have the concept of a class, has no concept of a member function and does not support the concept of access restrictions. C also does not support inheritance. The really big difference, however, is templates. C has macros, and that's it.
Therefore no, you can't directly expose C++ code to C in any fashion, you will have to use C-style code in your C++ to expose the C++ layer.
template<T> T foo(T i) { /* ... */ }
extern "C" int fooInt(int i) { return foo(i); }
However C++ was originally basically a C code generator, and C++ can still interop (one way) with the C ABI: member functions are actually implemented by turning this->function(int arg); into ThisClass0int1(this, arg); or something like that. In theory, you could write something to do this to your code, perhaps leveraging clang.
But that's a non-trivial task, something that's already well-tackled by SWIG, Boost::Python and Cython.
The problem with templates, however, is that the compiler ignores them until you "instantiate" (use) them. std::vector<> is not a concrete thing until you specify std::vector<int> or something. And now the only concrete implementation of that is std::vector<int>. Until you've specified it somewhere, std::vector<string> doesn't exist in your binary.
You probably want to start by looking at something like this http://kos.gd/2013/01/5-ways-to-use-python-with-native-code/, select a tool, e.g. SWIG, and then start building an interface to expose what you want/need to C. This is a lot less work than building the wrappers yourself. Depending which tool you use, it may be as simple as writing a line saying using std::vector<int> or typedef std::vector<int> IntVector or something.
---- EDIT ----
The problem with a template class is that you are creating an entire type that C can't understand, consider:
template<typename T>
class Foo {
T a;
int b;
T c;
public:
Foo(T a_) : a(a_) {}
void DoThing();
T GetA() { return a; }
int GetB() { return b; }
T GetC() { return c; }
};
The C language doesn't support the class keyword, never mind understand that members a, b and c are private, or what a constructor is, and C doesn't understand member functions.
Again it doesn't understand templates so you'll need to do what C++ does automatically and generate an instantiation, by hand:
struct FooDouble {
double a;
int b;
double c;
};
Except, all of those variables are private. So do you really want to be exposing them? If not, you probably just need to typedef "FooDouble" to something the same size as Foo and make a macro to do that.
Then you need to write replacements for the member functions. C doesn't understand constructors, so you will need to write a
extern "C" FooDouble* FooDouble_construct(double a);
FooDouble* FooDouble_construct(double a) {
Foo* foo = new Foo(a);
return reinterept_cast<FooDouble*>(foo);
}
and a destructor
extern "C" void FooDouble_destruct(FooDouble* foo);
void FooDouble_destruct(FooDouble* foo) {
delete reinterpret_cast<Foo*>(foo);
}
and a similar pass-thru for the accessors.
We can extract a PyObject pointing to a python method using
PyObject *method = PyDict_GetItemString(methodsDictionary,methodName.c_str());
I want to know how many arguments the method takes. So if the function is
def f(x,y):
return x+y
how do I find out it needs 2 arguments?
Followed through the link provided by Jon. Assuming you don't want to (or can't) use Boost in your application, the following should get you the number (easily adapted from How to find the number of parameters to a Python function from C?):
PyObject *key, *value;
int pos = 0;
while(PyDict_Next(methodsDictionary, &pos, &key, &value)) {
if(PyCallable_Check(value)) {
PyObject* fc = PyObject_GetAttrString(value, "func_code");
if(fc) {
PyObject* ac = PyObject_GetAttrString(fc, "co_argcount");
if(ac) {
const int count = PyInt_AsLong(ac);
// we now have the argument count, do something with this function
Py_DECREF(ac);
}
Py_DECREF(fc);
}
}
}
If you're in Python 2.x the above definitely works. In Python 3.0+, you seem to need to use "__code__" instead of "func_code" in the snippet above.
I appreciate the inability to use Boost (my company won't allow it for the projects I've been working on lately), but in general, I would try to knuckle down and use that if you can, as I've found that the Python C API can in general get a bit fiddly as you try to do complicated stuff like this.
In regex or Script (e.g. one written in python) how can I add
printf("TRACING: %s is called\n", __PRETTY_FUNCTION__);
at entry of all the function definitions, e.g.,
INT4
FunctionNameCouldBeAny (UINT4 ui)
{
// insert here, at entry
printf("TRACING: %s is called\n", __PRETTY_FUNCTION__);
}
for one file xxx.c?
for all c files *.c under directory /work_space/test/src?
Please note, functions defined in the same file may share the same prefix, but not always.
COMMENT 1: -finstrument-functions does not work for my gcc; or else I have to provide __cyg_profile_func_enter()/exit() functions, and find a way to print out the name from binary address. I wonder if there is more efficient way from regex.
Doing the job thoroughly is going to be hard. The problem is recognizing when a function is defined; there can be many possible layouts, and it is hard to recognize all possibilities. For example:
int x() { return 1; } int y(int z) { return z + 13; }
Most ad hoc systems won't detect y, even if they detect and handle x. But that's lousy code layout, and you probably don't indulge in such code layouts.
How do you start your functions?
static void function(int arg1) {
static void
function(int arg1) {
static void
function(int arg1)
{
static void
function(
int arg1
)
{
static void
function
(
int arg1
)
{
Etc. Depending on the notation(s) used, you need to write multiple different regular expressions. Note that if you can't apply heuristics such as '{ at start of line marks start of function - or structure/union definition, or data initialization' (because you use a { at the end of the line containing the function), it gets rather tricky.
You may need to tokenize the input, and keep track of whether you're inside a function definition. Although my example used keywords, you can have functions using user-defined types only:
Xyz *pqr(Abc def)
{
Then, of course, you might have old code written without prototypes:
Xyz *pqr(def)
Abc def;
{
All this is before you get involved in preprocessor stuff, which can really confuse things:
#define BEGIN {
#define END }
Xyz *pqr(Abc def)
BEGIN
...
END
(The original Bourne shell source was, reputedly, written using macros akin to those.)
So, normally, you develop an ad hoc system for recognizing the functions laid out in the style used by your project. One hopes that your project is systematic enough to have a limited variety of choices, but if this is old code that has been maintained over many years by many people, there are likely to be special cases all over the place.
I have a simple solution, that is, mighty Macro! if you only want to apply these to a small set of functions, and they are not called by function pointer
here is the simple solution:
1 locate call functions' definitions and declearations
2 rename them to something else, for example: myfunc to _myfunc, only names changed in these functions so it's fairly easy to change back
3 define a macro, with the name myfunc of which do this: 1) print your log 2) call the original function
you can grab all these functions' declarations to a single macro definition file, do some simple text work(maybe with script lang help, or text editor macro help), but this solution is not good for vast amount of functions, and don't deal with function pointer.
for example, you have function like this:
int myfunc(int arg)
int myfunc(int arg){
return 0;
}
rename myfunc to _myfunc and add a macro named myfunc
#define myfunc(arg)\
print("print whatever you want!");\
myfunc(arg);\
int _myfunc(int arg);
int _myfunc(int arg){
return 0;
}
if macro not good enough, wrap it with another function
int myfunc(arg){
print("print whatever you want!");
return myfunc(arg);
}
int _myfunc(int arg);
int _myfunc(int arg){
return 0;
}
If you want to debug linux kernel with printk and you want to get the call flow at run time, I have a very basic local solution for that.
Open your .c file in notepad++ --> ctrl+H --> Find what: \n{ -> Replace with: \n{ printk \("Your_Name: %s:%d \\n", __func__, __LINE__ \); --> Replace All
Comment kernel parent dir Makefile as like below:
+#KBUILD_CFLAGS += $(call cc-option,-Wdeclaration-after-statement,)
Hope it will solve your problem.