I am trying to compile a C++ module to use in scipy.weave that is composed of several headers and source C++ files. These files contain classes and methods that extensively use the Numpy/C-API interface. But I am failing to figure out how to include import_array() successfully. I have been struggling on this for the past week and I am going nuts. I hope you could help me with it because the weave help is not very explanatory.
In practice I have first a module called pycapi_utils that contains some routines to interface C objects with Python objects. It consists of a header file pycapi_utils.h and a source file pycapi_utils.cpp such as:
//pycapi_utils.h
#if ! defined _PYCAPI_UTILS_H
#define _PYCAPI_UTILS_H 1
#include <stdlib.h>
#include <Python.h>
#include <numpy/arrayobject.h>
#include <tuple>
#include <list>
typedef std::tuple<const char*,PyObject*> pykeyval; //Tuple type (string,Pyobj*) as dictionary entry (key,val)
typedef std::list<pykeyval> kvlist;
//Declaration of methods
PyObject* array_double_to_pyobj(double* v_c, long int NUMEL); //Convert from array to Python list (double)
...
...
#endif
and
//pycapi_utils.cpp
#include "pycapi_utils.h"
PyObject* array_double_to_pyobj(double* v_c, long int NUMEL){
//Convert a double array to a Numpy array
PyObject* out_array = PyArray_SimpleNew(1, &NUMEL, NPY_DOUBLE);
double* v_b = (double*) ((PyArrayObject*) out_array)->data;
for (int i=0;i<NUMEL;i++) v_b[i] = v_c[i];
free(v_c);
return out_array;
}
Then I have a further module model that contains classes and routines dealing with some mathematical model. Again it consists of a header and source file like:
//model.h
#if ! defined _MODEL_H
#define _MODEL_H 1
//model class
class my_model{
int i,j;
public:
my_model();
~my_model();
double* update(double*);
}
//Simulator
PyObject* simulate(double* input);
#endif
and
//model.cpp
#include "pycapi_utils.h"
#include "model.h"
//Define class and methods
model::model{
...
...
}
...
...
double* model::update(double* input){
double* x = (double*)calloc(N,sizeof(double));
...
...
// Do something
...
...
return x;
}
PyObject* simulate(double* input){
//Initialize Python interface
Py_Initialize;
import_array();
model random_network;
double* output;
output = random_network.update(input);
return array_double_to_pyobj(output); // from pycapi_utils.h
}
The above code is included in a scipy.weave module in Python with
def model_py(input):
support_code="""
#include "model.h"
"""
code = """
return_val = simulate(input.data());
"""
libs=['gsl','gslcblas','m']
vars = ['input']
out = weave.inline(code,
vars,
support_code=support_code,
sources = source_files,
libraries=libs
type_converters=converters.blitz,
compiler='gcc',
extra_compile_args=['-std=c++11'],
force=1)
It fails to compile giving:
error: int _import_array() was not declared in this scope
Noteworthy is that if I lump into pycapi_utils.h also the source pycapi_utils.cpp, everything works fine. But I don't want to use this solution, as in practice my modules here need to be included in several other modules that also use PyObjects and need call import_array().
I was looking to this post on stack exchange, but I cannot figure out if and how to properly define the #define directives in my case. Also the example in that post is not exactly my case as there, import_array() is called within the global scope of main() whereas in my case import_array() is called within my simulate routine which is invoked by main() build by scipy.weave.
I had a similar problem, as the link you've posted points out, the root of all evil is that the PyArray_API is defined static, which means that each translation unit has it's own PyArray_API which is initialized with PyArray_API = NULL by default. Thus import_array() must be called once for every *.cpp file. In your case it should be sufficient to call it in pycapi_utils.cpp and also once in model.cpp. You can also test if array_import is necessary before actualy calling it with:
if(PyArray_API == NULL)
{
import_array();
}
So apparently if I include in the pycapi_utils module a simple initialization routine such as:
//pycapi_utils.h
...
...
void init_numpy();
//pycapi_utils.cpp
...
...
void init_numpy(){
Py_Initialize;
import_array();
}
and then I invoke this routine at the beginning of any function / method that uses Numpy objects in my C code, it works. That is, the above code is edited as:
//pycapi_utils.cpp
...
...
PyObject* array_double_to_pyobj(...){
init_numpy();
...
...
}
//model.cpp
...
...
PyObject* simulate(...){
init_numpy();
...
...
}
My only concern at this point is whether there is a way to minimize number of calls to init_numpy(), or regardless I have to call it from any function that I define within my CPP modules that uses Numpy objects...
Related
Sorry for the long title, here is what I'm trying to achieve: I have a small C++ class with a bool template parameter which, when true, disables its setter methods using std::enable_if. Here is a simplified example:
template< bool IS_CONST >
class Handle
{
public:
Handle(void) : m_value(0) { }
Handle(int value) : m_value(value) { }
template < bool T = IS_CONST, typename COMPILED = typename std::enable_if< T == false >::type >
void set(int value)
{
m_value = value;
}
int get(void) const
{
return m_value;
}
private:
int m_value;
};
This code compiles an work as expected: Handle< true > doesn't have the set method, Handle< false > has it.
Now I'm trying to bind this to Python using SWIG. I'm using the following file to generate the binding:
%module core
%include "Handle.h"
%template(NonConstHandle) Handle< false >;
%template(ConstHandle) Handle< false >;
%{
#include "Test.h"
%}
SWIG generates the module without complaining, it compiles fine, but the set method is never bound, even in the specialized NonConstHandle. e.g. the following Python test fails with AttributeError: 'NonConstHandle' object has no attribute 'set' :
import core
handle = core.NonConstHandle()
assert(handle.get() == 0)
handle.set(1)
assert(handle.get() == 1)
const_handle = core.ConstHandle()
assert(const_handle .get() == 0)
try:
const_handle .set(1)
print("this should not print")
except:
pass
print("all good")
When I searched on the subject, I found lots of things related to enable_if and SWIG which leads me to think that it's supported, but I can't figure out why set is not generated, although there's no error / warning emitted by SWIG...
Any help appreciated !
Regards
The problem here is that every time you create a template in C++ you need at least one %template directive in your SWIG interface in order to make it have any effect on the generated wrapper.
When people vaguely hint that std::enable_if works they typically mean two things. Firstly that it parses ok and secondly that %template works for them. Both things are true here.
Since you've used SFINAE inside your template class with a template function you need one %template for each. Otherwise the set member is ignored totally, as you've seen. Side stepping the SFINAE/enable_if bit of your question an example of template functions inside template classes is a good place to start.
So we can change your .i file to look something like this:
%module test
%{
#include "test.h"
%}
%include "test.h"
// Be explicit about what 'versions' of set to instantiate in our wrapper
%template(set) Handle::set<false, void>;
%template(NonConstHandle) Handle<false>;
%template(ConstHandle) Handle<true>;
The problem is that (having fixed a few minor bugs in it) your test python now hits "this should not print", because we've generated a (totally legal) set() function even in the const case by explicitly spelling out the template parameters instead of getting them deduced.
So we've generated code to call:
Handle<true>::set<false, void>(int);
Which uh works in this instance, because it can compile just not in the intuitive way.
I'm not aware of a way of making the deduction happen here (which is a shame, because they're defaulted so it should be possible right? - Maybe one for a patch into SWIG trunk, although doing both the defaulting and the SFINAE is going to be tricky)
Fortunately there is a simple workaround using %ignore to drop the version we don't want too:
%module test
%{
#include "test.h"
%}
%include "test.h"
%template(set) Handle::set<false, void>;
%ignore Handle<true>::set;
%template(NonConstHandle) Handle<false>;
%template(ConstHandle) Handle<true>;
Which does then generate the code you expected.
It's worth noting that often it's simpler to explicitly spell out the way you want complex templated code to work when generating wrappers - you typically need extra helpers or tweaks to the interface to get it to work in Python in the way you hope for. So you could also solve your example by doing something like this:
%module test
%{
#include "test.h"
%}
template <bool>
class Handle {
public:
Handle(void);
Handle(int value);
int get(void) const;
};
template<>
class Handle<false>
{
public:
Handle(void);
Handle(int value);
void set(int value);
int get(void) const;
};
%template(NonConstHandle) Handle<false>;
%template(ConstHandle) Handle<true>;
Or a similar trick:
%module test
%{
#include "test.h"
typedef Handle<true> ConstHandle;
typedef Handle<false> NonConstHandle;
%}
struct ConstHandle {
ConstHandle(void);
ConstHandle(int value);
int get(void) const;
};
struct NonConstHandle
{
NonConstHandle(void);
NonConstHandle(int value);
void set(int value);
int get(void) const;
};
Although note that in this last case you'll need to use %apply as well if you want to use the templates as arguments in/out of functions.
This question is about how to pass a C++ object to a python function that is called in a (C++) embedded Python interpreter.
The following C++ class (MyClass.h) is designed for testing:
#ifndef MyClassH
#define MyClassH
#include <string>
using std::string;
class MyClass
{
public:
MyClass(const string& lbl): label(lbl) {}
~MyClass(){}
string getLabel() {return label;}
private:
string label;
};
#endif
A python module, exposing the C++ class, can be generated by the following Swig interface file:
%module passmetopython
%{ #include "MyClass.h" %}
%include "std_string.i"
//Expose to Python
%include "MyClass.h"
Below is a Python script using the python module
import passmetopython as pmtp
def execute(obj):
#This function is to be called from C/C++, with a
#MyClass object as an argument
print ("Entering execute function")
lbl = obj.getLabel();
print ("Printing from within python execute function. Object label is: " + lbl)
return True
def main():
c = pmtp.MyClass("Test 1")
retValue = execute(c)
print("Return value: " + str(retValue))
#Test function from within python
if __name__ == '__main__':
main()
This question is about how to get the python execute() function working, when called from c++, with a C++ object as an argument.
The following C++ program was written to test the functions (minimum amount of error checking):
#include "Python.h"
#include <iostream>
#include <sstream>
#include "MyClass.h"
using namespace std;
int main()
{
MyClass obj("In C++");
cout << "Object label: \"" << obj.getLabel() << "\"" << endl;
//Setup the Python interpreter and eventually call the execute function in the
//demo python script
Py_Initialize();
//Load python Demo script, "passmetopythonDemo.py"
string PyModule("passmetopythonDemo");
PyObject* pm = PyUnicode_DecodeFSDefault(PyModule.c_str());
PyRun_SimpleString("import sys");
stringstream cmd;
cmd << "sys.path.append(\"" << "." << "\")";
PyRun_SimpleString(cmd.str().c_str());
PyObject* PyModuleP = PyImport_Import(pm);
Py_DECREF(pm);
//Now create PyObjects for the Python functions that we want to call
PyObject* pFunc = PyObject_GetAttrString(PyModuleP, "execute");
if(pFunc)
{
//Setup argument
PyObject* pArgs = PyTuple_New(1);
//Construct a PyObject* from long
PyObject* pObj(NULL);
/* My current attempt to create avalid argument to Python */
pObj = PyLong_FromLong((long) &obj);
PyTuple_SetItem(pArgs, 0, pObj);
/***** Calling python here *****/
cout<<endl<<"Calling function with an MyClass argument\n\n";
PyObject* res = PyObject_CallObject(pFunc, pArgs);
if(!res)
{
cerr << "Failed calling function..";
}
}
return 0;
}
When running the above code, the execute() python function, with a MyClass object as an argument, fails and returns NULL. However, the Python function is entered, as I can see the output (Entering execute function) in the console output, indicating that the object passed is not, indeed, a valid MyClass object.
There are a lot of examples on how to pass simple types, like ints, doubles or string types to Python from C/C++. But there are very few example showing how to pass a C/C++ object/ pointer, which is kind of puzzling.
The above code, with a CMake file, can be checked out from github:
https://github.com/TotteKarlsson/miniprojects/tree/master/passMeToPython
This code is not to use any boost python or other API's. Cython sounds interesting though, and if it can be used to simplify on the C++ side, it could be acceptable.
This is a partial answer to my own question. I'm saying partial, because I do believe there is a better way.
Building on this post http://swig.10945.n7.nabble.com/Pass-a-Swig-wrapped-C-class-to-embedded-Python-code-td8812.html
I generated the swig runtime header, as described here, section 15.4: http://www.swig.org/Doc2.0/Modules.html#Modules_external_run_time
Including the generated header in the C++ code above, allow the following code to be written:
PyObject* pObj = SWIG_NewPointerObj((void*)&obj, SWIG_TypeQuery("_p_MyClass"), 0 );
This code is using information from the Swig python wrap source files, namely the "swig" name of the type MyClass, i.e. _p_MyClass.
With the above PyObject* as an argument to the PyObject_CallObject function, the python execute() function in the code above executes fine, and the Python code, using the generated python module, do have proper access to the MyClass objects internal data. This is great.
Although the above code illustrate how to pass, and retrieve data between C++ and Python in a quite simple fashion, its not ideal, in my opinion.
The usage of the swig header file in the C++ code is really not that pretty, and in addition, it requires a user to "manually" look into swig generated wrapper code in order to find the "_p_MyClass" code.
There must be a better way!? Perhaps something should be added to the swig interface file in order to get this looking nicer?
PyObject *pValue;
pValue = PyObject_CallMethod(pInstance, "add","(i)",x);
if (pValue)
Py_DECREF(pValue);
else
PyErr_Print();
I need help in writing a wrapper to call some functions and get JSON responce (a unicode string) from a third-party shared library. The library's headers file is shown below:
#include <string>
#include <ExportLib.h>
// some code ignored here
typedef std::string UString;
using namespace std;
namespace ns1{
class DLL_PUBLIC S_API {
public:
static UString function1();
static UString function2();
// some code ignored here
};
}
The problem is I'm not so good in C/C++, so I have no idea what to do with Cython. I would be very grateful if someone would point me in right direction. I wrote a .pyx file like so:
from libcpp.string cimport string
cdef extern from "libName.h" namespace "ns1":
cdef cppclass S_API:
string function1;
string function2;
This compiles fine and I do get a .so file, which I can import in Pyhton. But I am still unable to access function1() or any other function inside the module.
I have a C++ function that accepts a callback, like this:
void func(std::function<void(A, B)> callback) { ... }
I want to call this function from Cython by giving it a closure, i.e. something I would have done with a lambda if I was calling it from C++. If this was a C function, it would have some extra void* arguments:
typedef void(*callback_t)(int, int, void*);
void func(callback_t callback, void *user_data) {
callback(1, 2, user_data);
}
and then I would just pass PyObject* as user_data (there is a more detailed example here).
Is there way to do this more in C++ way, without having to resort to explicit user_data?
What I believe you're aiming to do is pass a callable Python object to something accepting a std::function. You need to do create a bit of C++ code to make it happen, but it's reasonably straightforward.
Starting by defining "accepts_std_function.hpp" as simply as possible to provide an illustrative example:
#include <functional>
#include <string>
inline void call_some_std_func(std::function<void(int,const std::string&)> callback) {
callback(5,std::string("hello"));
}
The trick is then to create a wrapper class that holds a PyObject* and defines operator(). Defining operator() allows it to be converted to a std::function. Most of the class is just refcounting. "py_obj_wrapper.hpp":
#include <Python.h>
#include <string>
#include "call_obj.h" // cython helper file
class PyObjWrapper {
public:
// constructors and destructors mostly do reference counting
PyObjWrapper(PyObject* o): held(o) {
Py_XINCREF(o);
}
PyObjWrapper(const PyObjWrapper& rhs): PyObjWrapper(rhs.held) { // C++11 onwards only
}
PyObjWrapper(PyObjWrapper&& rhs): held(rhs.held) {
rhs.held = 0;
}
// need no-arg constructor to stack allocate in Cython
PyObjWrapper(): PyObjWrapper(nullptr) {
}
~PyObjWrapper() {
Py_XDECREF(held);
}
PyObjWrapper& operator=(const PyObjWrapper& rhs) {
PyObjWrapper tmp = rhs;
return (*this = std::move(tmp));
}
PyObjWrapper& operator=(PyObjWrapper&& rhs) {
held = rhs.held;
rhs.held = 0;
return *this;
}
void operator()(int a, const std::string& b) {
if (held) { // nullptr check
call_obj(held,a,b); // note, no way of checking for errors until you return to Python
}
}
private:
PyObject* held;
};
This file uses a very short Cython file to do the conversions from C++ types to Python types. "call_obj.pyx":
from libcpp.string cimport string
cdef public void call_obj(obj, int a, const string& b):
obj(a,b)
You then just need to create the Cython code wraps these types. Compile this module and call test_func to run this. ("simple_version.pyx":)
cdef extern from "py_obj_wrapper.hpp":
cdef cppclass PyObjWrapper:
PyObjWrapper()
PyObjWrapper(object) # define a constructor that takes a Python object
# note - doesn't match c++ signature - that's fine!
cdef extern from "accepts_std_func.hpp":
void call_some_std_func(PyObjWrapper) except +
# here I lie about the signature
# because C++ does an automatic conversion to function pointer
# for classes that define operator(), but Cython doesn't know that
def example(a,b):
print(a,b)
def test_call():
cdef PyObjWrapper f = PyObjWrapper(example)
call_some_std_func(f)
The above version works but is somewhat limited in that if you want to do this with a different std::function specialization you need to rewrite some of it (and the conversion from C++ to Python types doesn't naturally lend itself to a template implementation). One easy way round this is to use the Boost Python library object class, which has a templated operator(). This comes at the cost of introducing an extra library dependency.
First defining the header "boost_wrapper.hpp" to simplify the conversion from PyObject* to boost::python::object
#include <boost/python/object.hpp>
inline boost::python::object get_as_bpo(PyObject* o) {
return boost::python::object(boost::python::handle<>(boost::python::borrowed(o)));
}
You then just need to Cython code to wrap this class ("boost_version.pyx"). Again, call test_func
cdef extern from "boost_wrapper.hpp":
cdef cppclass bpo "boost::python::object":
# manually set name (it'll conflict with "object" otherwise
bpo()
bpo get_as_bpo(object)
cdef extern from "accepts_std_func.hpp":
void call_some_std_func(bpo) except + # again, lie about signature
def example(a,b):
print(a,b)
def test_call():
cdef bpo f = get_as_bpo(example)
call_some_std_func(f)
A "setup.py"
from distutils.core import setup, Extension
from Cython.Build import cythonize
extensions = [
Extension(
"simple_version", # the extension name
sources=["simple_version.pyx", "call_obj.pyx" ],
language="c++", # generate and compile C++ code
),
Extension(
"boost_version", # the extension name
sources=["boost_version.pyx"],
libraries=['boost_python'],
language="c++", # generate and compile C++ code
)
]
setup(ext_modules = cythonize(extensions))
(A final option is to use ctypes to generate a C function pointer from a Python callable. See Using function pointers to methods of classes without the gil (bottom half of answer) and http://osdir.com/ml/python-cython-devel/2009-10/msg00202.html. I'm not going to go into detail about this here.)
I have some C code that runs on an embedded system, generating a stream of data which my python code will read on the other side of a bluetooth/usb line. The protocol of the stream is still under heavy development, and changes frequently, but is defined in a single .h file. I would like to use SWIG to keep the python side of things up to date, specifically by providing access to the stream data layouts (structs)
Here is a sample .h file describing a number of structures and a few constants (as #defines), obviously a very small subset of the whole protocol, for brevity.
//datalayouts.h
#ifdef SWIG
#define __attribute__(x)
#endif
#define TOKEN_TYPE_SYNC_VALUE 1
#define TOKEN_TYPE_DELTA 2
typedef struct __attribute__((packed))
{
uint8_t token_type;
uint32_t timestamp;
uint32_t value;
} struct_token_type_sync_value;
typedef struct __attribute__((packed))
{
uint8_t token_type;
int16_t delta;
} struct_token_type_delta;
Coupled with this is the basic interface file
%module datalayouts
%{
#include "datalayouts.h"
%}
%include "datalayouts.h"
It all compiles and imports just fine. In python I can create a variable of type token_type_sync_value, but what I want to do is cast a portion of the data I'm reading from the stream (as a string), to impose the correct structure on it.
For example:
>>> from datalayouts token_type_sync_value
>>> data = stream.read() #returns 100+ bytes
>>> if ord(data[0]) == TOKEN_TYPE_SYNC_VALUE:
... #here I want to access data[0:9] as a token_type_sync_value
Is this possible, if so how?
You can do this with SWIG, the simplest solution is to use %extend to supply an extra constructor from within Python that takes a PyObect to use as a buffer:
%module test
%include <stdint.i>
%inline %{
#ifdef SWIG
#define __attribute__(x)
#endif
#define TOKEN_TYPE_SYNC_VALUE 1
#define TOKEN_TYPE_DELTA 2
typedef struct __attribute__((packed))
{
uint8_t token_type;
int16_t delta;
} struct_token_type_delta;
%}
%extend struct_token_type_delta {
struct_token_type_delta(PyObject *in) {
assert(PyObject_CheckBuffer(in));
Py_buffer view;
const int ret = PyObject_GetBuffer(in, &view, PyBUF_SIMPLE);
assert(0==ret);
assert(view.len >= sizeof(struct_token_type_delta));
struct_token_type_delta *result = new struct_token_type_delta(*static_cast<const struct_token_type_delta*>(view.buf));
PyBuffer_Release(&view); // Note you could/should retain view.obj for the life of this object to prevent use after free
return result;
}
}
You'd need to do this for each type you wanted to construct from a buffer, but the actual code for the constructor of each remains the same so could be wrapped as a macro (using %define) quite simply. You would also want to do something to prevent the use after free error, by retaining the reference to the underlying buffer for longer.
Personally if it were me doing this though I'd look for a different solution, because there are nicer ways of getting the same result and writing code that creates and maintains thin POD/bean like objects is tedious and dull in any language let alone 2 or more. Assuming protbuf is too heavyweight to use in your embedded system I'd look to solve this in reverse, using ctypes for Python and then having your Python code also generate the header for your C build tools as well. So something like:
import ctypes
class ProtocolStructure(type(ctypes.Structure)):
def __str__(self):
s='''
typedef struct __attribute__((packed)) {
\t%s
}'''
return s % '\n\t'.join(('%s %s;' % (ty.__name__[2:], name) for name,ty in self._fields_))
class struct_token_type_delta(ctypes.Structure, metaclass=ProtocolStructure):
_fields_ = (('token_type', ctypes.c_uint8),
('delta', ctypes.c_int16))
if __name__ == '__main__':
# when this file is run instead of imported print the header file to stdout
h='''
#ifndef PROTO_H
#define PROTO_H
%s
#endif
'''
print(h % ';\n'.join('%s %s;\n' % (ty, name) for name,ty in globals().items() if issubclass(type(ty), ProtocolStructure)))
Which then lets you write:
import proto
proto.struct_token_type_delta.from_buffer(bytearray(b'\xff\x11\x22'))