Develop Reference

Using python within C code and passing list as an argument

I am using Python code within C++ code and trying to pass a list argument to a function written in Python. I tried the normal way of executing the Python code without passing any argument and it was all working fine but when I pass a list as an argument, I get a segmentation fault.
Here is my code:
#define PY_SSIZE_T_CLEAN
#include</usr/include/python3.6/Python.h>
#include <bits/stdc++.h>
using namespace std;
int callModuleFunc(int array[], size_t size) {
PyObject *mymodule = PyImport_ImportModule("test");
PyObject *myfunc = PyObject_GetAttrString(mymodule, "get_lists");
cout<<"Imported"<<endl;
PyObject *mylist = PyList_New(size);
cout<<"Imported3"<<endl;
for (size_t i = 0; i != size; ++i) {
PyList_SET_ITEM(mylist, i, PyLong_FromLong(array[i]));
}
PyObject *arglist = Py_BuildValue("(O)", mylist);
cout<<"Imported1"<<endl;
PyObject *result = PyObject_CallObject(myfunc, arglist); // getting segmentation fault here
cout<<"Imported5"<<endl;
int retval = (int)PyLong_AsLong(result);
Py_DECREF(result);
Py_DECREF(arglist);
Py_DECREF(mylist);
Py_DECREF(myfunc);
Py_DECREF(mymodule);
return retval;
}
int main(int argc, char const *argv[])
{
wchar_t * program = Py_DecodeLocale(argv[0], NULL);
if(!program){
cout<<"***Error***"<<endl;
exit(1);
}
Py_SetProgramName(program);
Py_Initialize();
PyObject *module = NULL, *result = NULL;
PyRun_SimpleString("print('Hello from python')\n"
"print('Hiii')");
int arr[5] = {1,3,4,5,6};
callModuleFunc(arr, 5);
if(Py_FinalizeEx() < 0){
cout<<"***Error***"<<endl;
exit(120);
}
PyMem_RawFree(program);
return 0;
}
When I call PyObject_CallObject(myfunc, arglist), I get a segmentation fault.
I am totally new to it so I'm just trying stuff from the internet.
I'm using Python version 3.6 with g++ compiler 7.5.
Here is my test.py:
def get_lists(l1):
print("Lists: ", l1)
return 10
Please let me know how I can resolve this.
Thanks

The Python function get_lists is not known during execution.
Note: The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.
see https://docs.python.org/3/library/test.html
If the name of the imported Python file is renamed to another name (e.g. list.py) the Python function can be found.
Additional Hints
For the environment of the OP the problem is then solved. For my environment (Python 3.9.5) I need additionally replace the PyRun_SimpleString with:
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\".\")");
Otherwise the module can't be imported.
Finally, it is advisable to check each function call for errors (e.g. whether it returns NULL) and to use PyErr_Print so that the actual cause of the error is printed instead of a crash due to a segmentation fault.
For example an error message could look like this:
AttributeError: module 'test' has no attribute 'get_lists'
Test
The output of the line Lists: [1, 3, 4, 5, 6] on the console shows that the Python function is called correctly when the above points are taken into account.

Why am I unable to access my Python function after importing its module in C++?

I am attempting to call a function print_stuff which is a member of the class Spam from a user-defined Python module Spam in C++
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int main()
{
Py_Initialize();
//modify python module search path
auto modulePath = /* hardcoded search path for module */
PyObject* sysPath = PySys_GetObject("path");
PyList_Append(sysPath, PyUnicode_FromString(modulePath));
PyObject* pModule = PyImport_ImportModule("Spam");
PyObject* pDict = PyModule_GetDict(pModule);
PyObject* pFunc = PyObject_GetAttrString(pDict, "print_stuff");
if (pFunc != NULL)
PyObject_CallObject(pFunc, NULL);
else
PyErr_Print();
return 1;
}
My Python module consists of 2 files; Spam.py and __init__.py both located in same directory /Spam which is a subfolder of the directory containing my VS solution and C++ code.
Spam.py
class Spam:
def __init__(self):
pass
def print_stuff(self):
print((1,2,3))
__init__.py
import Spam as Spam
When I run my C++ code I get the following error: AttributeError: 'dict' object has no attribute 'print_stuff'
I have tried getting the contents of the dictionary returned from PyModule_GetDict(pModule) as a string and confirmed that it doesn't mention the function.
I looked up the docs for PyModule_GetDict which says that it returns the same dictionary as you would get calling __dict__ on the module in Python. When I tried calling Spam.__dict__ inside of __init__.py I got the same result, my function was missing, but when I tried Spam.Spam.__dict__ I got a dictionary containing the function.
From this I decided to change the import statement to from Spam import Spam so that Spam.__dict__ would now include my function, which it did. However, nothing changes when I run my C++ code. I still get the same error from my dictionary and am still unable to call my function.
I did wonder if it was an issue of not having an instance of the class in the my C++ code, so I also tried removing the surrounding class and having Spam.py just contain the function definition and then importing that but, again, I got the same error.
I have a hunch this is some kind of namespace issue, otherwise I don't see how PyModule_GetDict and __dict__ could return different dictionaries for the same module, but honestly I don't know where to go from here.
Is my class Spam not considered part of the module I am trying to import from C++? Is my __init__.py missing something or perhaps I need to include another statement in C++ to import the class after I have imported the module or have I not correctly imported the module in the first place?
I have seen similar questions before but they tended to be in cases where exceptions were raised, the search path was incorrect, or where one the prerequisite values had returned NULL. I am fairly experienced with C++ but am new to Python and newer still to the Python/C API, although I have read the docs covering the basics of the API and how modules work in Python.

First, I don't think you need the last two lines in spam.py. I reduced it to this:
class Spam:
def __init__(self):
pass
def print_stuff(self):
print((1,2,3))
Now, let's test the module:
>>> import spam
>>> s = spam.Spam()
>>> s.print_stuff()
(1, 2, 3)
So far, so good. Let's see if we can use it from C++. Here's a working version of your program:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <stdlib.h>
int main()
{
Py_Initialize();
//modify python module search path
auto modulePath = ".";
PyObject* sysPath = PySys_GetObject("path");
PyList_Append(sysPath, PyUnicode_FromString(modulePath));
PyObject* pModule = PyImport_ImportModule("spam");
PyObject* spam_class = PyObject_GetAttrString(pModule, "Spam");
if( !spam_class ) { PyErr_Print(); return EXIT_FAILURE; }
PyObject* spam = PyObject_CallObject(spam_class, NULL);
if( !spam ) { PyErr_Print(); return EXIT_FAILURE; }
PyObject* pFunc = PyObject_GetAttrString(spam, "print_stuff");
if (pFunc != NULL)
PyObject_CallObject(pFunc, NULL);
else
PyErr_Print();
return 1;
}
I named the module spam.py, lowercase. The module object has a Spam class as an attribute. We get that class and call it to create an object, just like we do in Python:
>>> s = spam.Spam()
Now we have an object. It has methods, which are attributes, which we can get with PyObject_GetAttrString, as usual. You know the rest.
It's been a few years since I worked with this stuff, and my experience was from the other side, making C modules accessible to Python. I worked my way to the above example by using dir in the Python interpreter until I got what I needed. If you "think like the interpreter" you may find it all makes more sense.

boost python - embedding in c++ and passing variables

Introduction
I'm trying to do a simple thing - in theory, but so far not in practise. I have C++ console application, with embedded python script. It has to pass some variables inside python, do calculation, and send it back.
Example code
my cpp file look like this (based on THIS answer):
#include <iostream>
#include <boost/python.hpp>
using namespace boost::python;
int main()
{
int a = 2;
int b = 3;
Py_Initialize();
try
{
object module = import("__main__");
object name_space = module.attr("__dict__");
exec_file("Script.py", name_space, name_space);
object MyFunc = name_space["MyFunc"];
object result = MyFunc(a,b);
int sum = extract<int>(result["sum"]);
std::cout << sum;
}
catch (error_already_set)
{
PyErr_Print();
}
Py_Finalize();
}
My Script.py file is very simple
def MyFunc(a,b):
result = a+b
return result
The Issue
I expected, that python will accept two variables, sum it, and let extract it back do my C++. Then C++ print the results. However it ends with following error:
TypeError: 'int' object has no attribute '__getitem__'
I think that MyFunc(a,b) is not correct call, however I can't figure why. Should I pass variables in other way? If yes, pleas explain me how to do this.

Passing a C++ object to Python

This question is about how to pass a C++ object to a python function that is called in a (C++) embedded Python interpreter.
The following C++ class (MyClass.h) is designed for testing:
#ifndef MyClassH
#define MyClassH
#include <string>
using std::string;
class MyClass
{
public:
MyClass(const string& lbl): label(lbl) {}
~MyClass(){}
string getLabel() {return label;}
private:
string label;
};
#endif
A python module, exposing the C++ class, can be generated by the following Swig interface file:
%module passmetopython
%{ #include "MyClass.h" %}
%include "std_string.i"
//Expose to Python
%include "MyClass.h"
Below is a Python script using the python module
import passmetopython as pmtp
def execute(obj):
#This function is to be called from C/C++, with a
#MyClass object as an argument
print ("Entering execute function")
lbl = obj.getLabel();
print ("Printing from within python execute function. Object label is: " + lbl)
return True
def main():
c = pmtp.MyClass("Test 1")
retValue = execute(c)
print("Return value: " + str(retValue))
#Test function from within python
if __name__ == '__main__':
main()
This question is about how to get the python execute() function working, when called from c++, with a C++ object as an argument.
The following C++ program was written to test the functions (minimum amount of error checking):
#include "Python.h"
#include <iostream>
#include <sstream>
#include "MyClass.h"
using namespace std;
int main()
{
MyClass obj("In C++");
cout << "Object label: \"" << obj.getLabel() << "\"" << endl;
//Setup the Python interpreter and eventually call the execute function in the
//demo python script
Py_Initialize();
//Load python Demo script, "passmetopythonDemo.py"
string PyModule("passmetopythonDemo");
PyObject* pm = PyUnicode_DecodeFSDefault(PyModule.c_str());
PyRun_SimpleString("import sys");
stringstream cmd;
cmd << "sys.path.append(\"" << "." << "\")";
PyRun_SimpleString(cmd.str().c_str());
PyObject* PyModuleP = PyImport_Import(pm);
Py_DECREF(pm);
//Now create PyObjects for the Python functions that we want to call
PyObject* pFunc = PyObject_GetAttrString(PyModuleP, "execute");
if(pFunc)
{
//Setup argument
PyObject* pArgs = PyTuple_New(1);
//Construct a PyObject* from long
PyObject* pObj(NULL);
/* My current attempt to create avalid argument to Python */
pObj = PyLong_FromLong((long) &obj);
PyTuple_SetItem(pArgs, 0, pObj);
/***** Calling python here *****/
cout<<endl<<"Calling function with an MyClass argument\n\n";
PyObject* res = PyObject_CallObject(pFunc, pArgs);
if(!res)
{
cerr << "Failed calling function..";
}
}
return 0;
}
When running the above code, the execute() python function, with a MyClass object as an argument, fails and returns NULL. However, the Python function is entered, as I can see the output (Entering execute function) in the console output, indicating that the object passed is not, indeed, a valid MyClass object.
There are a lot of examples on how to pass simple types, like ints, doubles or string types to Python from C/C++. But there are very few example showing how to pass a C/C++ object/ pointer, which is kind of puzzling.
The above code, with a CMake file, can be checked out from github:
https://github.com/TotteKarlsson/miniprojects/tree/master/passMeToPython
This code is not to use any boost python or other API's. Cython sounds interesting though, and if it can be used to simplify on the C++ side, it could be acceptable.

This is a partial answer to my own question. I'm saying partial, because I do believe there is a better way.
Building on this post http://swig.10945.n7.nabble.com/Pass-a-Swig-wrapped-C-class-to-embedded-Python-code-td8812.html
I generated the swig runtime header, as described here, section 15.4: http://www.swig.org/Doc2.0/Modules.html#Modules_external_run_time
Including the generated header in the C++ code above, allow the following code to be written:
PyObject* pObj = SWIG_NewPointerObj((void*)&obj, SWIG_TypeQuery("_p_MyClass"), 0 );
This code is using information from the Swig python wrap source files, namely the "swig" name of the type MyClass, i.e. _p_MyClass.
With the above PyObject* as an argument to the PyObject_CallObject function, the python execute() function in the code above executes fine, and the Python code, using the generated python module, do have proper access to the MyClass objects internal data. This is great.
Although the above code illustrate how to pass, and retrieve data between C++ and Python in a quite simple fashion, its not ideal, in my opinion.
The usage of the swig header file in the C++ code is really not that pretty, and in addition, it requires a user to "manually" look into swig generated wrapper code in order to find the "_p_MyClass" code.
There must be a better way!? Perhaps something should be added to the swig interface file in order to get this looking nicer?

PyObject *pValue;
pValue = PyObject_CallMethod(pInstance, "add","(i)",x);
if (pValue)
Py_DECREF(pValue);
else
PyErr_Print();

Performance - extending C++ with Python function

I have a main application written in C++. To give user some more dynamic control via scripting, I chose to embed some python in the code. FYI, my user are not keen on learning and writing any C++, so shared lib is out of the question. The C++ code will call user defined function in python at runtime.
Ex: user would write the following python function. Of cause, in practice user would supply a bit more complex function definition than a simple add.
def userAdd(a, b):
return a+b
The C++ code uses the following function to call userAdd. I need to run evalUserFunc many times with various sets of (a,b). How slow would this method be compared to calling a hard coded C++ function. I'm seeing a .pyc generated upon execution. Is this some what analogues to a .so in C++? Thank you in advance.
float myClass::evalUserFunc(float a, float b)
{
PyObject *pFunc, *pArgs;
PyObject *pa, *pb, *pValue;
int res = 0;
if (_pModule != NULL) {
pFunc = PyObject_GetAttrString(_pModule, 'userAdd');
pArgs = PyTuple_New(2);
pa = PyFloat_FromDouble(a);
pb = PyFloat_FromDouble(b);
PyTuple_SetItem(pArgs, 0, pa);
PyTuple_SetItem(pArgs, 1, pb);
pValue = PyObject_CallObject(pFunc, pArgs);
res = PyFloat_AsDouble(pValue);
}
return res;
}