I have a main application written in C++. To give user some more dynamic control via scripting, I chose to embed some python in the code. FYI, my user are not keen on learning and writing any C++, so shared lib is out of the question. The C++ code will call user defined function in python at runtime.
Ex: user would write the following python function. Of cause, in practice user would supply a bit more complex function definition than a simple add.
def userAdd(a, b):
return a+b
The C++ code uses the following function to call userAdd. I need to run evalUserFunc many times with various sets of (a,b). How slow would this method be compared to calling a hard coded C++ function. I'm seeing a .pyc generated upon execution. Is this some what analogues to a .so in C++? Thank you in advance.
float myClass::evalUserFunc(float a, float b)
{
PyObject *pFunc, *pArgs;
PyObject *pa, *pb, *pValue;
int res = 0;
if (_pModule != NULL) {
pFunc = PyObject_GetAttrString(_pModule, 'userAdd');
pArgs = PyTuple_New(2);
pa = PyFloat_FromDouble(a);
pb = PyFloat_FromDouble(b);
PyTuple_SetItem(pArgs, 0, pa);
PyTuple_SetItem(pArgs, 1, pb);
pValue = PyObject_CallObject(pFunc, pArgs);
res = PyFloat_AsDouble(pValue);
}
return res;
}
Related
I am using Python code within C++ code and trying to pass a list argument to a function written in Python. I tried the normal way of executing the Python code without passing any argument and it was all working fine but when I pass a list as an argument, I get a segmentation fault.
Here is my code:
#define PY_SSIZE_T_CLEAN
#include</usr/include/python3.6/Python.h>
#include <bits/stdc++.h>
using namespace std;
int callModuleFunc(int array[], size_t size) {
PyObject *mymodule = PyImport_ImportModule("test");
PyObject *myfunc = PyObject_GetAttrString(mymodule, "get_lists");
cout<<"Imported"<<endl;
PyObject *mylist = PyList_New(size);
cout<<"Imported3"<<endl;
for (size_t i = 0; i != size; ++i) {
PyList_SET_ITEM(mylist, i, PyLong_FromLong(array[i]));
}
PyObject *arglist = Py_BuildValue("(O)", mylist);
cout<<"Imported1"<<endl;
PyObject *result = PyObject_CallObject(myfunc, arglist); // getting segmentation fault here
cout<<"Imported5"<<endl;
int retval = (int)PyLong_AsLong(result);
Py_DECREF(result);
Py_DECREF(arglist);
Py_DECREF(mylist);
Py_DECREF(myfunc);
Py_DECREF(mymodule);
return retval;
}
int main(int argc, char const *argv[])
{
wchar_t * program = Py_DecodeLocale(argv[0], NULL);
if(!program){
cout<<"***Error***"<<endl;
exit(1);
}
Py_SetProgramName(program);
Py_Initialize();
PyObject *module = NULL, *result = NULL;
PyRun_SimpleString("print('Hello from python')\n"
"print('Hiii')");
int arr[5] = {1,3,4,5,6};
callModuleFunc(arr, 5);
if(Py_FinalizeEx() < 0){
cout<<"***Error***"<<endl;
exit(120);
}
PyMem_RawFree(program);
return 0;
}
When I call PyObject_CallObject(myfunc, arglist), I get a segmentation fault.
I am totally new to it so I'm just trying stuff from the internet.
I'm using Python version 3.6 with g++ compiler 7.5.
Here is my test.py:
def get_lists(l1):
print("Lists: ", l1)
return 10
Please let me know how I can resolve this.
Thanks
The Python function get_lists is not known during execution.
Note: The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.
see https://docs.python.org/3/library/test.html
If the name of the imported Python file is renamed to another name (e.g. list.py) the Python function can be found.
Additional Hints
For the environment of the OP the problem is then solved. For my environment (Python 3.9.5) I need additionally replace the PyRun_SimpleString with:
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\".\")");
Otherwise the module can't be imported.
Finally, it is advisable to check each function call for errors (e.g. whether it returns NULL) and to use PyErr_Print so that the actual cause of the error is printed instead of a crash due to a segmentation fault.
For example an error message could look like this:
AttributeError: module 'test' has no attribute 'get_lists'
Test
The output of the line Lists: [1, 3, 4, 5, 6] on the console shows that the Python function is called correctly when the above points are taken into account.
I'm learning C++, and in particular C interface to Python. Right now, my focus is on calling or importing python objects from C++ main program.
I've been studying the following link but couldn't understand some concepts. (https://www.codeproject.com/Articles/820116/Embedding-Python-program-in-a-C-Cplusplus-code)
Following is the sections of the tutorial that I can't understand fully.
My questions are:
calling of module:
Is it correct for me to assume "CPyObject pModule = PyImport_Import(pName)" is doing this job?
importing of objects:
i. Is it correct for me to assume "CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger")" is doing this job?
ii.If I want to import a dataframe from python to C++ as a CPyObject, how can I manipulate this object in C++. I'm asking because there is no equivalent object to dataframe in C++.
3) Is there anything else I need to do to make sure my Python module file is visible and callable to C++ ? Such as saving them in the same folder?
Consider the following Python program, stored in pyemb3.py:
def getInteger():
print('Python function getInteger() called')
c = 100*50/30
return c
Now we want to call the function getInteger() from the following C++ code and print the value returned this function. This is the client C++ code:
#include <stdio.h>
#include <Python.h>
#include <pyhelper.hpp>
int main()
{
CPyInstance hInstance;
CPyObject pName = PyUnicode_FromString("pyemb3");
CPyObject pModule = PyImport_Import(pName);
if(pModule)
{
CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger");
if(pFunc && PyCallable_Check(pFunc))
{
CPyObject pValue = PyObject_CallObject(pFunc, NULL);
printf_s("C: getInteger() = %ld\n", PyLong_AsLong(pValue));
}
else
{
printf("ERROR: function getInteger()\n");
}
}
else
{
printf_s("ERROR: Module not imported\n");
}
return 0;
}
The problem is that 100*50/30 is not an integer, it is a float.
to get an integer use integer division: 100*50//30
If you are not sure about the returned type, you can use the Py_TYPE macro on pValue or just simply check for the type you are looking for with: PyLong_Check or PyLong_CheckExact
1: if PyImport_Import does not return null then the import was successful and the module was already executed by the time the function returned.
2: The PyObject_GetAttrString or the PyObject_GetAttr is the right way to get the imported module's objects.
3: Use these flags to ensure Python is embedded. Use Py_SetPath before Py_Initialize to add your module's path to sys.path.
I am new to the business of writing custom Python modules and I am a bit confused how Capsules work. I use Python 2.7.6 from the system OSX installation and try to use Capsules (as recommended for Python > 2.7) for passing pointers around (before they used PyCObject for that). My code does not work at the moment and I would like to get some insights how things should be handled in principle here. The code should define a class LuscherClm and I want be able to do the following:
>>> c40=Luscher(4,0)
>>>
>>> c40(0.12)
>>> <print the result of the evaluation>
First question: at the moment I would have to do something like:
>>> c40=Luscher.init(4,0)
>>>
>>> c40.eval(0.12)
Segfault
My first question is therefore: how do I have to modify the method table to have more operator-style casts instead of the member functions init and eval.
However, my code has other problems and here is the relevant part (the underlying C++ class works smoothly, I use it in production a lot):
The destructor:
//destructor
static void clm_destruct(PyObject* capsule){
void* ptr=PyCapsule_GetPointer(capsule,"zetfunc");
Zetafunc* zetptr=static_cast<Zetafunc*>(ptr);
delete zetptr;
return;
}
The constructor: it returns the pointer to the capsule. I do not know whether this is correct. Because in this case when I call, clm=LuscherClm.init(l,m), the clm object is a PyCapsule and has no attribute eval so that I cannot call clm.eval(x) on that. How should this be handled?
//constructor
static PyObject* clm_init(PyObject* self, PyObject *args){
//return value
PyObject* result=NULL;
//parse variables
unsigned int lval=0;
int mval=0;
if(!PyArg_ParseTuple(args,"li",&lval,&mval)){
::std::cout << "Please specify l and m!" << ::std::endl;
return result;
}
//class instance:
Zetafunc* zetfunc=new Zetafunc(lval,mval);
instanceCapsule=PyCapsule_New(static_cast<void*> (zetfunc),"zetfunc",&clm_destruct);
return instanceCapsule;
}
So how is the capsule passed to the evaluate function? the code below is not correct since I have not updated it after moving from CObjects to Capsules. Shall the capsule be a global variable (I do not like that) or how can I pass it to the evaluation function? Or shall I call it on self, but what is self at the moment?
//evaluate the function
static PyObject* clm_evaluate(PyObject* self, PyObject* args){
//get the PyCObject from the capsule:
void* tmpzetfunc=PyCapsule_GetPointer(instanceCapsule,"zetfunc");
if (PyErr_Occurred()){
std::cerr << "Some Error occured!" << std::endl;
return NULL;
}
Zetafunc* zetfunc=static_cast< Zetafunc* >(tmpzetfunc);
//parse value:
double x;
if(!PyArg_ParseTuple(args,"d",&x)){
std::cerr << "Specify a number at which you want to evaluate the function" << std::endl;
return NULL;
}
double result=(*zetfunc)(x).re();
//return the result as a packed function:
return Py_BuildValue("d",result);
}
//methods
static PyMethodDef LuscherClmMethods[] = {
{"init", clm_init, METH_VARARGS, "Initialize clm class!"},
{"eval", clm_evaluate, METH_VARARGS, "Evaluate the Zeta-Function!"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
Python < 3 initialisation function:
PyMODINIT_FUNC
initLuscherClm(void)
{
PyObject *m = Py_InitModule("LuscherClm", LuscherClmMethods);
return;
}
Can you explain to me what is wrong and why? I would like to stay away from SWIG or boost if possible, since this module should be easily portable and I want to avoid having to install additional packages every time I want to use it somewhere else.
Further: what is the overhead produced by the C/API in calling the function? I need to call it an order of O(10^6) times and I would still like it to be fast.
Ok, I am using boost.python now but I get a segfault when I run object.eval(). That is my procedure now:
BOOST_PYTHON_MODULE(threevecd)
{
class_< threevec<double> >("threevecd",init<double,double,double>());
}
BOOST_PYTHON_MODULE(LuscherClm)
{
class_<Zetafunc>("LuscherClm",init<int,int, optional<double,threevec<double>,double,int> >())
.def("eval",&Zetafunc::operator(),return_value_policy<return_by_value>());
boost::python::to_python_converter<dcomplex,dcomplex_to_python_object>();
}
dcomplex is my own complex number implementation. So I had to write a converter:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
if(fabs(comp.im())<std::numeric_limits<double>::epsilon()){
boost::python::object result=boost::python::object(complex<double>(comp.re(),comp.im()));
return boost::python::incref(result.ptr());
}
else{
return Py_BuildValue("d",comp.re());
}
}
};
Complex128 is a numpy extension which is not understood by boost. So my questions are:
1) how can I return a complex number as a python datatype (is complex a standard python type?)
2) Why do I get a segfault. My result in my testcase is real so it should default to the else statement. I guess that the pointer runs out of scope and thats it. But even in the if-case (where I take care about ref-increments), it segfaults. Can someone help me with the type conversion issue?
Thanks
Thorsten
Ok, I got it. The following converter does the job:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
PyObject* result;
if(std::abs(comp.im())<=std::numeric_limits<double>::epsilon()){
result=PyFloat_FromDouble(comp.re());
}
else{
result=PyComplex_FromDoubles(comp.re(),comp.im());
}
Py_INCREF(result);
return result;
}
};
Using this converter and the post by Wouter, I suppose my question is answered. Thanks
I am hoping that this is a simple SWIG issue. I am using the Google OR-Tools optimization library. It is a C++ library that is wrapped in SWIG (which I know little about). I am having great difficult getting a Python callback function to work. There is a C++ function
DecisionBuilder* MakePhase(const std::vector<IntVar*>& vars,
IndexEvaluator1* var_evaluator,
IntValueStrategy val_str);
along with
typedef ResultCallback1<int64, int64> IndexEvaluator1;
and the relevant SWIG (I believe) is
DecisionBuilder* VarEvalValStrPhase(
const std::vector<IntVar*>& vars,
ResultCallback1<int64, int64>* var_evaluator,
operations_research::Solver::IntValueStrategy val_str) {
return self->MakePhase(vars, var_evaluator, val_str);
}
and in another SWIG file we have
%{
static int64 PyCallback1Int64Int64(PyObject* pyfunc, int64 i) {
// () needed to force creation of one-element tuple
PyObject* pyresult = PyEval_CallFunction(pyfunc, "(l)", static_cast<long>(i));
int64 result = 0;
if (!pyresult) {
PyErr_SetString(PyExc_RuntimeError,
"ResultCallback1<int64, int64> invocation failed.");
} else {
result = PyInt_AsLong(pyresult);
Py_DECREF(pyresult);
}
return result;
}
%}
%typemap(in) ResultCallback1<int64, int64>* {
if (!PyCallable_Check($input)) {
PyErr_SetString(PyExc_TypeError, "Need a callable object!");
SWIG_fail;
}
$1 = NewPermanentCallback(&PyCallback1Int64Int64, $input);
}
In my Python module I have defined a function, Run1, as follows (and here is where part of me thinks there should be some type casts, but I gather that is not the Python way):
def Run1(index1):
return index1
and set
selector_callback = Run1
solver = pywrapcp.Solver("graph-coloring")
Finally, I call
solver.Phase(nodes,
selector_callback,
solver.INT_VALUE_DEFAULT)
and here alas is where things go kablooie, I always get the following error:
File "C:\dev\Python27\lib\site-packages\ortools-1.3853-py2.7-win-amd64.egg\ortools\constraint_solver\pywrapcp.py", line 457, in Phase
def Phase(self, *args): return _pywrapcp.Solver_Phase(self, *args)
NotImplementedError: Wrong number or type of arguments for overloaded function 'Solver_Phase'.
Possible C/C++ prototypes are:
operations_research::Solver::MakePhase(std::vector< operations_research::IntVar *,std::allocator< operations_research::IntVar * > > const &,operations_research::Solver::IntVarStrategy,operations_research::Solver::IntValueStrategy)
operations_research::Solver::MakePhase(std::vector< operations_research::IntervalVar *,std::allocator< operations_research::IntervalVar * > > const &,operations_research::Solver::IntervalStrategy)
operations_research::Solver::MakePhase(std::vector< operations_research::SequenceVar *,std::allocator< operations_research::SequenceVar * > > const &,operations_research::Solver::SequenceStrategy)
The difficulty is with the callback function in the second argument; if I use one of the built-in values instead of the callback the operation is successful. But, I do need to have my own function in there.
I am not importing any SWIG files in my module. Do I need to do this?
So after days, I found the answer. If I call
solver.VarEvalValStrPhase(nodes,
selector_callback,
solver.INT_VALUE_DEFAULT)
instead of the standard function name Phase referenced throughout the manuals, it works. If I were to use another argument combination I'd have to use another function name I believe. It appears that overloading fails in this case. Which is fine, but a warning from the developers would have been nice.
I've written a high level motor controller in Python, and have got to a point where I want to go a little lower level to get some speed, so I'm interested in coding those bits in C.
I don't have much experience with C, but the math I'm working on is pretty straightforward, so I'm sure I can implement with a minimal amount of banging my head against the wall. What I'm not sure about is how best to invoke this compiled C program in order to pipe it's outputs back into my high-level python controller.
I've used a little bit of ctypes, but only to pull some functions from a manufacfturer-supplied DLL...not sure if that is an appropriate path to go down in this case.
Any thoughts?
You can take a look at this tutorial here.
Also, a more reliable example on the official python website, here.
For example,
sum.h function
int sum(int a, int b)
A file named, module.c,
#include <Python.h>
#include "sum.h"
static PyObject* mod_sum(PyObject *self, PyObject *args)
{
int a;
int b;
int s;
if (!PyArg_ParseTuple(args,"ii",&a,&b))
return NULL;
s = sum(a,b);
return Py_BuildValue("i",s);
}
static PyMethodDef ModMethods[] = {
{"sum", mod_sum, METH_VARARGS, "Description.."},
{NULL,NULL,0,NULL}
};
PyMODINIT_FUNC initmod(void)
{
PyObject *m;
m = Py_InitModule("module",ModMethods);
if (m == NULL)
return;
}
Python
import module
s = module.sum(3,5)
Another option: try numba.
It gives you C-like speed for free: just import numba and #autojit your functions, for a wonderful speed increase.
Won't work if you have complicated data types, but if you're looping and jumping around array indices, it might be just what you're looking for.
you can use SWIG, it is very simple to use
You can use Cython for setting the necessary c types and compile your python syntax code.