This question is about how to pass a C++ object to a python function that is called in a (C++) embedded Python interpreter.
The following C++ class (MyClass.h) is designed for testing:
#ifndef MyClassH
#define MyClassH
#include <string>
using std::string;
class MyClass
{
public:
MyClass(const string& lbl): label(lbl) {}
~MyClass(){}
string getLabel() {return label;}
private:
string label;
};
#endif
A python module, exposing the C++ class, can be generated by the following Swig interface file:
%module passmetopython
%{ #include "MyClass.h" %}
%include "std_string.i"
//Expose to Python
%include "MyClass.h"
Below is a Python script using the python module
import passmetopython as pmtp
def execute(obj):
#This function is to be called from C/C++, with a
#MyClass object as an argument
print ("Entering execute function")
lbl = obj.getLabel();
print ("Printing from within python execute function. Object label is: " + lbl)
return True
def main():
c = pmtp.MyClass("Test 1")
retValue = execute(c)
print("Return value: " + str(retValue))
#Test function from within python
if __name__ == '__main__':
main()
This question is about how to get the python execute() function working, when called from c++, with a C++ object as an argument.
The following C++ program was written to test the functions (minimum amount of error checking):
#include "Python.h"
#include <iostream>
#include <sstream>
#include "MyClass.h"
using namespace std;
int main()
{
MyClass obj("In C++");
cout << "Object label: \"" << obj.getLabel() << "\"" << endl;
//Setup the Python interpreter and eventually call the execute function in the
//demo python script
Py_Initialize();
//Load python Demo script, "passmetopythonDemo.py"
string PyModule("passmetopythonDemo");
PyObject* pm = PyUnicode_DecodeFSDefault(PyModule.c_str());
PyRun_SimpleString("import sys");
stringstream cmd;
cmd << "sys.path.append(\"" << "." << "\")";
PyRun_SimpleString(cmd.str().c_str());
PyObject* PyModuleP = PyImport_Import(pm);
Py_DECREF(pm);
//Now create PyObjects for the Python functions that we want to call
PyObject* pFunc = PyObject_GetAttrString(PyModuleP, "execute");
if(pFunc)
{
//Setup argument
PyObject* pArgs = PyTuple_New(1);
//Construct a PyObject* from long
PyObject* pObj(NULL);
/* My current attempt to create avalid argument to Python */
pObj = PyLong_FromLong((long) &obj);
PyTuple_SetItem(pArgs, 0, pObj);
/***** Calling python here *****/
cout<<endl<<"Calling function with an MyClass argument\n\n";
PyObject* res = PyObject_CallObject(pFunc, pArgs);
if(!res)
{
cerr << "Failed calling function..";
}
}
return 0;
}
When running the above code, the execute() python function, with a MyClass object as an argument, fails and returns NULL. However, the Python function is entered, as I can see the output (Entering execute function) in the console output, indicating that the object passed is not, indeed, a valid MyClass object.
There are a lot of examples on how to pass simple types, like ints, doubles or string types to Python from C/C++. But there are very few example showing how to pass a C/C++ object/ pointer, which is kind of puzzling.
The above code, with a CMake file, can be checked out from github:
https://github.com/TotteKarlsson/miniprojects/tree/master/passMeToPython
This code is not to use any boost python or other API's. Cython sounds interesting though, and if it can be used to simplify on the C++ side, it could be acceptable.
This is a partial answer to my own question. I'm saying partial, because I do believe there is a better way.
Building on this post http://swig.10945.n7.nabble.com/Pass-a-Swig-wrapped-C-class-to-embedded-Python-code-td8812.html
I generated the swig runtime header, as described here, section 15.4: http://www.swig.org/Doc2.0/Modules.html#Modules_external_run_time
Including the generated header in the C++ code above, allow the following code to be written:
PyObject* pObj = SWIG_NewPointerObj((void*)&obj, SWIG_TypeQuery("_p_MyClass"), 0 );
This code is using information from the Swig python wrap source files, namely the "swig" name of the type MyClass, i.e. _p_MyClass.
With the above PyObject* as an argument to the PyObject_CallObject function, the python execute() function in the code above executes fine, and the Python code, using the generated python module, do have proper access to the MyClass objects internal data. This is great.
Although the above code illustrate how to pass, and retrieve data between C++ and Python in a quite simple fashion, its not ideal, in my opinion.
The usage of the swig header file in the C++ code is really not that pretty, and in addition, it requires a user to "manually" look into swig generated wrapper code in order to find the "_p_MyClass" code.
There must be a better way!? Perhaps something should be added to the swig interface file in order to get this looking nicer?
PyObject *pValue;
pValue = PyObject_CallMethod(pInstance, "add","(i)",x);
if (pValue)
Py_DECREF(pValue);
else
PyErr_Print();
Related
I am using Python code within C++ code and trying to pass a list argument to a function written in Python. I tried the normal way of executing the Python code without passing any argument and it was all working fine but when I pass a list as an argument, I get a segmentation fault.
Here is my code:
#define PY_SSIZE_T_CLEAN
#include</usr/include/python3.6/Python.h>
#include <bits/stdc++.h>
using namespace std;
int callModuleFunc(int array[], size_t size) {
PyObject *mymodule = PyImport_ImportModule("test");
PyObject *myfunc = PyObject_GetAttrString(mymodule, "get_lists");
cout<<"Imported"<<endl;
PyObject *mylist = PyList_New(size);
cout<<"Imported3"<<endl;
for (size_t i = 0; i != size; ++i) {
PyList_SET_ITEM(mylist, i, PyLong_FromLong(array[i]));
}
PyObject *arglist = Py_BuildValue("(O)", mylist);
cout<<"Imported1"<<endl;
PyObject *result = PyObject_CallObject(myfunc, arglist); // getting segmentation fault here
cout<<"Imported5"<<endl;
int retval = (int)PyLong_AsLong(result);
Py_DECREF(result);
Py_DECREF(arglist);
Py_DECREF(mylist);
Py_DECREF(myfunc);
Py_DECREF(mymodule);
return retval;
}
int main(int argc, char const *argv[])
{
wchar_t * program = Py_DecodeLocale(argv[0], NULL);
if(!program){
cout<<"***Error***"<<endl;
exit(1);
}
Py_SetProgramName(program);
Py_Initialize();
PyObject *module = NULL, *result = NULL;
PyRun_SimpleString("print('Hello from python')\n"
"print('Hiii')");
int arr[5] = {1,3,4,5,6};
callModuleFunc(arr, 5);
if(Py_FinalizeEx() < 0){
cout<<"***Error***"<<endl;
exit(120);
}
PyMem_RawFree(program);
return 0;
}
When I call PyObject_CallObject(myfunc, arglist), I get a segmentation fault.
I am totally new to it so I'm just trying stuff from the internet.
I'm using Python version 3.6 with g++ compiler 7.5.
Here is my test.py:
def get_lists(l1):
print("Lists: ", l1)
return 10
Please let me know how I can resolve this.
Thanks
The Python function get_lists is not known during execution.
Note: The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.
see https://docs.python.org/3/library/test.html
If the name of the imported Python file is renamed to another name (e.g. list.py) the Python function can be found.
Additional Hints
For the environment of the OP the problem is then solved. For my environment (Python 3.9.5) I need additionally replace the PyRun_SimpleString with:
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\".\")");
Otherwise the module can't be imported.
Finally, it is advisable to check each function call for errors (e.g. whether it returns NULL) and to use PyErr_Print so that the actual cause of the error is printed instead of a crash due to a segmentation fault.
For example an error message could look like this:
AttributeError: module 'test' has no attribute 'get_lists'
Test
The output of the line Lists: [1, 3, 4, 5, 6] on the console shows that the Python function is called correctly when the above points are taken into account.
I'm learning C++, and in particular C interface to Python. Right now, my focus is on calling or importing python objects from C++ main program.
I've been studying the following link but couldn't understand some concepts. (https://www.codeproject.com/Articles/820116/Embedding-Python-program-in-a-C-Cplusplus-code)
Following is the sections of the tutorial that I can't understand fully.
My questions are:
calling of module:
Is it correct for me to assume "CPyObject pModule = PyImport_Import(pName)" is doing this job?
importing of objects:
i. Is it correct for me to assume "CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger")" is doing this job?
ii.If I want to import a dataframe from python to C++ as a CPyObject, how can I manipulate this object in C++. I'm asking because there is no equivalent object to dataframe in C++.
3) Is there anything else I need to do to make sure my Python module file is visible and callable to C++ ? Such as saving them in the same folder?
Consider the following Python program, stored in pyemb3.py:
def getInteger():
print('Python function getInteger() called')
c = 100*50/30
return c
Now we want to call the function getInteger() from the following C++ code and print the value returned this function. This is the client C++ code:
#include <stdio.h>
#include <Python.h>
#include <pyhelper.hpp>
int main()
{
CPyInstance hInstance;
CPyObject pName = PyUnicode_FromString("pyemb3");
CPyObject pModule = PyImport_Import(pName);
if(pModule)
{
CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger");
if(pFunc && PyCallable_Check(pFunc))
{
CPyObject pValue = PyObject_CallObject(pFunc, NULL);
printf_s("C: getInteger() = %ld\n", PyLong_AsLong(pValue));
}
else
{
printf("ERROR: function getInteger()\n");
}
}
else
{
printf_s("ERROR: Module not imported\n");
}
return 0;
}
The problem is that 100*50/30 is not an integer, it is a float.
to get an integer use integer division: 100*50//30
If you are not sure about the returned type, you can use the Py_TYPE macro on pValue or just simply check for the type you are looking for with: PyLong_Check or PyLong_CheckExact
1: if PyImport_Import does not return null then the import was successful and the module was already executed by the time the function returned.
2: The PyObject_GetAttrString or the PyObject_GetAttr is the right way to get the imported module's objects.
3: Use these flags to ensure Python is embedded. Use Py_SetPath before Py_Initialize to add your module's path to sys.path.
Introduction
I'm trying to do a simple thing - in theory, but so far not in practise. I have C++ console application, with embedded python script. It has to pass some variables inside python, do calculation, and send it back.
Example code
my cpp file look like this (based on THIS answer):
#include <iostream>
#include <boost/python.hpp>
using namespace boost::python;
int main()
{
int a = 2;
int b = 3;
Py_Initialize();
try
{
object module = import("__main__");
object name_space = module.attr("__dict__");
exec_file("Script.py", name_space, name_space);
object MyFunc = name_space["MyFunc"];
object result = MyFunc(a,b);
int sum = extract<int>(result["sum"]);
std::cout << sum;
}
catch (error_already_set)
{
PyErr_Print();
}
Py_Finalize();
}
My Script.py file is very simple
def MyFunc(a,b):
result = a+b
return result
The Issue
I expected, that python will accept two variables, sum it, and let extract it back do my C++. Then C++ print the results. However it ends with following error:
TypeError: 'int' object has no attribute '__getitem__'
I think that MyFunc(a,b) is not correct call, however I can't figure why. Should I pass variables in other way? If yes, pleas explain me how to do this.
In general what is the recommended way to pass a Python function through Boost-Python for use later in C++ code (i.e. as a callback in a C++ object)?
More specifically, I have a C++ class FooCPP that I've successfully exposed to Python via Boost-Python; the user interacts with the Python class Foo that runs the C++ counterpart under the hood. Contrived example:
# Foo.py
from foo_base import FooBase
class Foo(FooBase):
...
def callback(val=42.):
return val
foo = Foo()
foo.run(callback)
And the Boost Python bindings:
// foo_bindings.cpp
#include "foo.hpp"
#include <boost/python.hpp>
namespace bp = boost::python;
FooPython::Run(const bp::object& py_callback)
// TODO: Do something with the python callback to make it a C++ function!
std::function<double(double)> cpp_callback;
FooCPP::Run(cpp_callback);
)
BOOST_PYTHON_MODULE(foo_base){
bp::class_<FooPython>("FooBase")
.def("run", &FooPython::Run)
;
}
So how can I address the TODO comment in foo_bindings.cpp?
I've gone through a number of related SO questions -- e.g. pass python function to boost c and sending py function as boost function arg -- and I'm familiar with the Boost-Python docs, but have not found a good solution/explanation. Thanks in advance!
Notes: C++11, boost v1.58.0, ubuntu 16.04
Update
I may have just found a solution, where I can implement a functor in foo_bindings.cpp, e.g.,
struct PythonCallback {
public:
PythonCallback(bp::object cb_func) : cb_func_(cb_func) {}
double operator() (const double& val) {
// Call the callback function in python
return cb_func_(val);
}
private:
bp::object cb_func_;
};
But then what should the FooCPP::Run signature be? I.e. what type is defined for the cpp_callback passed in?
And does the BOOST_PYTHON_MODULE code need to change for this callback functor?
Implement a functor in foo_bindings.cpp, where the callback is invoked with call:
#include <boost/python.hpp>
#include <boost/python/call.hpp>
struct PythonCallback : {
public:
PythonCallback(PyObject* func) : cb_(func) {}
double operator() (const double& value) {
// Call the callback function in python
return boost::python::call<double>(cb_, value);
}
private:
PyObject* cb_;
};
I have my existing working code that depends on boost::uuids::uuid. Now I an trying to generate a python module out of it. SWIG is successfully generating all important classes and functions. But I am facing problem with the functions that takes or returns boost uuid.
I want to convert between boost uuid and python uuid. Is there any uuid.i that I can use ? I see there is an uuid python module.I understand I can import that module from an uuid.i with PyImport_ImportModule("uuid").
But how to instantiate and use the python's uuid class inside typemap ?
This is fairly straight forward to do any you're on the right lines with the PyImport_ImportModule call. What you need to do is figure out how you're going to marshal the UUIDs between the two types and then write one typemap for each direction. I ended up going via string representations since that's the simplest portable way to do it in my view. That's using uuid_io.hpp's IO operators in both directions.
When we wrap it like this Python code never see's the boost type at all and vice-versa.
I've assumed you're targeting Python 3.4 here, but everything I've done should be simple adjust to target older Python with too.
First I put together a C++ header file to demonstrate the wrapping I implemented:
#include <boost/uuid/uuid.hpp>
#include <boost/uuid/random_generator.hpp>
#include <boost/uuid/uuid_io.hpp>
#include <iostream>
inline boost::uuids::uuid testout() {
static boost::uuids::random_generator gen;
return gen();
}
inline void testin(const boost::uuids::uuid& in) {
std::cout << in << "\n";
}
Nothing clever there, just one function each for each direction of passing the objects.
Next I wrote some Python to exercise the interface I wanted to produce:
import test
import uuid
a=test.testout()
print(type(a))
print(a)
b=uuid.uuid4()
print(type(b))
print(b)
test.testin(a)
test.testin(b)
And finally a SWIG interface that would generate this module, once I've written the actual UUID wrapping.
%module test
%{
#include "test.hh"
%}
%include "boost_uuid.i"
%include "test.hh"
With all this in place we can now write an implementation boost_uuid.i that works for our scenario:
%{
#include <boost/uuid/uuid_io.hpp>
#include <boost/uuid/uuid.hpp>
#include <sstream>
namespace {
PyObject *py_uuid = nullptr;
}
%}
%init %{
py_uuid = PyImport_ImportModule("uuid"); // Handle error
%}
%typemap(in) const boost::uuids::uuid& (boost::uuids::uuid tmp) {
PyObject *str = PyObject_Str($input);
assert(str); // TODO: check properly
const char *uuid_str = PyUnicode_AsUTF8(str); // Note: Python 3.x, adjust as needed
assert(uuid_str); // TODO: check me
std::istringstream in(uuid_str);
Py_DECREF(str);
in >> tmp; // TODO: Check return!
$1 = &tmp;
}
%typemap(out) boost::uuids::uuid {
// Check this actually works!
static PyObject *uuid_ctor = PyObject_GetAttrString(py_uuid, "UUID");
std::ostringstream out;
out << $1;
PyObject *str = PyUnicode_DecodeUTF8(out.str().c_str(), out.str().size(), NULL);
// Theoretically this string conversion could have just failed
$result = PyObject_CallFunctionObjArgs(uuid_ctor, str, NULL);
Py_DECREF(str);
}
I did contemplate using boost::python to simplify some of this code a little since it's boost we're wrapping. In the end I didn't bother with that though.
You could also use boost's lexical_cast to avoid the stringstream that I've used. That's largely a matter of taste.
None of this is going to be super high performance code, but then when you're passing across language boundaries it also isn't likely to be the bottleneck in your system anyway. You could use the byte-by-byte access that both boost and Python's UUID module give as well and make it just be a direct copy operation, but I avoided that because of the requirement to consider endianness which increased the complexity.
Note:
No proper error handling
No typemap for pass by value on boost UUID input types.
No typemap for pass by non-const reference for input.
All of those should be fairly easy to add as required using this as a starting point.
This compiles and runs as expected:
swig3.0 -c++ -python -py3 -Wall test.i
g++ -g -std=c++1y -shared test_wrap.cxx -o _test.so -I/usr/include/python3.4 -Wall -Wextra -lpython3.4m
python3.4 run.py
<class 'uuid.UUID'>
b37c9285-f055-4f08-b4e9-4a238be3b09d
<class 'uuid.UUID'>
cf1071e6-2e7f-45af-8920-a68290ee61d4
b37c9285-f055-4f08-b4e9-4a238be3b09d
cf1071e6-2e7f-45af-8920-a68290ee61d4