I'm trying to use SWIG directors to call Python code from C++. However if the code is running in another thread, I must acquire the GIL. My code looks roughly like this (minus the thread stuff):
struct Callback {
virtual ~Callback() {}
virtual void call() {} // will be overwritten in Python
};
struct Foo {
Callback *c;
Foo(Callback* c) : c(c) {}
void doSomething() { c->call(); } // being called from another thread
};
Python:
from demo import *
class MyCallback(Callback):
def call(*args):
print("python callback")
c = MyCallback()
f = Foo(c)
f.doSomething()
Swig:
%module(directors="1", threads="1") demo
%feature("director");
%{
#include <test.h>
%}
%thread;
%include <test.h>
How can I acquire the GIL before calling the Python callback?
To elaborate, the threads feature of SWIG creates code that releases the GIL when C++ code is called, but when Python code is called from C++ it isn't reacquired, so that's what I want to do.
The call stack looks like this: Python -> Foo.doSomething -> C++ -> Callback.call -> Python.
Acquiring the GUIL is explained clearly in Non-Python created threads subsection of Thread State and the Global Interpreter Lock section in Python manual. But you say the issue is how to do this from SWIG generated code. How about this, added to your SWIG .i file:
struct PythonCallback: public Callback
{
virtual void call() {
getGIL();
lockedCall();
releaseGIL();
}
virtual void lockedCall() {} // will be overwritten in Python
};
You only need to expose PythonCallback (not Callback) and its lockedCall method, which you can rename to call if you wish via SWIG's %rename directive. Then in Python you would derive from PythonCallback, and override the call method (which in reality is the lockedCall method if you %rename'd it).
Related
I want to enable the writing of user python-plugins for an existing C++ application. The main application will emit signals using boost::signals2 that trigger the execution of the users python code. I set up a working sample where an initialize method in the users python module is being called. The user can subscribe to signals calling a "connect"-method:
import ctypes
CALLBACK_VOID_FUNC = CFUNCTYPE(None)
def myVoidHandler():
print("myVoidHandler (python)")
def initialize(myCHandle):
global myCDll, callback1
myCDll = ctypes.CDLL("CppEventsd", handle = myCHandle)
[...]
callback1 = CALLBACK_VOID_FUNC(myVoidHandler)
myCDll.connect(b"readyToGo", callback1)
[...]
The signature of the connect method:
extern "C" __declspec(dllexport) bool connect(char* signalName, void(*pF)())
Now the python function myVoidHandler could be called from cpp just like this:
pF();
But more importantly, pF can be connected to a boost signal.
However I want to spare the users going through this (and spare myself the support requests because they did not get it quite right), so I want to connect the python functions from the cpp side by using a naming convention for the functions/slots.
I can easily retrieve a PyObjet pointer to the python callback:
PyObject *pModule, *pDict, *pVoidHandlerF;
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pVoidHandlerF = PyDict_GetItemString(pDict, "myVoidHandler");
BUT I cannot figure out how to get the actual function pointer (equivalent to pF) from the cpp side! Some transformation seems to be going on in the python to Cpp transition.
Any ideas?
Instead of connecting directly to the python function pointer, you could create your own C++ function that uses the python C-API to invoke the python function. Here is an example:
boost::signals2::signal<void(void)>& sig = [...];
PyObject* pVoidHandlerF = [...];
sig.connect([=]() {
PyObject_CallObject(pVoidHandlerF, NULL);
});
This allows you to do other stuff on the C++ side, for example, passing arguments, returning values, checking for errors, etc.
In general what is the recommended way to pass a Python function through Boost-Python for use later in C++ code (i.e. as a callback in a C++ object)?
More specifically, I have a C++ class FooCPP that I've successfully exposed to Python via Boost-Python; the user interacts with the Python class Foo that runs the C++ counterpart under the hood. Contrived example:
# Foo.py
from foo_base import FooBase
class Foo(FooBase):
...
def callback(val=42.):
return val
foo = Foo()
foo.run(callback)
And the Boost Python bindings:
// foo_bindings.cpp
#include "foo.hpp"
#include <boost/python.hpp>
namespace bp = boost::python;
FooPython::Run(const bp::object& py_callback)
// TODO: Do something with the python callback to make it a C++ function!
std::function<double(double)> cpp_callback;
FooCPP::Run(cpp_callback);
)
BOOST_PYTHON_MODULE(foo_base){
bp::class_<FooPython>("FooBase")
.def("run", &FooPython::Run)
;
}
So how can I address the TODO comment in foo_bindings.cpp?
I've gone through a number of related SO questions -- e.g. pass python function to boost c and sending py function as boost function arg -- and I'm familiar with the Boost-Python docs, but have not found a good solution/explanation. Thanks in advance!
Notes: C++11, boost v1.58.0, ubuntu 16.04
Update
I may have just found a solution, where I can implement a functor in foo_bindings.cpp, e.g.,
struct PythonCallback {
public:
PythonCallback(bp::object cb_func) : cb_func_(cb_func) {}
double operator() (const double& val) {
// Call the callback function in python
return cb_func_(val);
}
private:
bp::object cb_func_;
};
But then what should the FooCPP::Run signature be? I.e. what type is defined for the cpp_callback passed in?
And does the BOOST_PYTHON_MODULE code need to change for this callback functor?
Implement a functor in foo_bindings.cpp, where the callback is invoked with call:
#include <boost/python.hpp>
#include <boost/python/call.hpp>
struct PythonCallback : {
public:
PythonCallback(PyObject* func) : cb_(func) {}
double operator() (const double& value) {
// Call the callback function in python
return boost::python::call<double>(cb_, value);
}
private:
PyObject* cb_;
};
This question is about how to pass a C++ object to a python function that is called in a (C++) embedded Python interpreter.
The following C++ class (MyClass.h) is designed for testing:
#ifndef MyClassH
#define MyClassH
#include <string>
using std::string;
class MyClass
{
public:
MyClass(const string& lbl): label(lbl) {}
~MyClass(){}
string getLabel() {return label;}
private:
string label;
};
#endif
A python module, exposing the C++ class, can be generated by the following Swig interface file:
%module passmetopython
%{ #include "MyClass.h" %}
%include "std_string.i"
//Expose to Python
%include "MyClass.h"
Below is a Python script using the python module
import passmetopython as pmtp
def execute(obj):
#This function is to be called from C/C++, with a
#MyClass object as an argument
print ("Entering execute function")
lbl = obj.getLabel();
print ("Printing from within python execute function. Object label is: " + lbl)
return True
def main():
c = pmtp.MyClass("Test 1")
retValue = execute(c)
print("Return value: " + str(retValue))
#Test function from within python
if __name__ == '__main__':
main()
This question is about how to get the python execute() function working, when called from c++, with a C++ object as an argument.
The following C++ program was written to test the functions (minimum amount of error checking):
#include "Python.h"
#include <iostream>
#include <sstream>
#include "MyClass.h"
using namespace std;
int main()
{
MyClass obj("In C++");
cout << "Object label: \"" << obj.getLabel() << "\"" << endl;
//Setup the Python interpreter and eventually call the execute function in the
//demo python script
Py_Initialize();
//Load python Demo script, "passmetopythonDemo.py"
string PyModule("passmetopythonDemo");
PyObject* pm = PyUnicode_DecodeFSDefault(PyModule.c_str());
PyRun_SimpleString("import sys");
stringstream cmd;
cmd << "sys.path.append(\"" << "." << "\")";
PyRun_SimpleString(cmd.str().c_str());
PyObject* PyModuleP = PyImport_Import(pm);
Py_DECREF(pm);
//Now create PyObjects for the Python functions that we want to call
PyObject* pFunc = PyObject_GetAttrString(PyModuleP, "execute");
if(pFunc)
{
//Setup argument
PyObject* pArgs = PyTuple_New(1);
//Construct a PyObject* from long
PyObject* pObj(NULL);
/* My current attempt to create avalid argument to Python */
pObj = PyLong_FromLong((long) &obj);
PyTuple_SetItem(pArgs, 0, pObj);
/***** Calling python here *****/
cout<<endl<<"Calling function with an MyClass argument\n\n";
PyObject* res = PyObject_CallObject(pFunc, pArgs);
if(!res)
{
cerr << "Failed calling function..";
}
}
return 0;
}
When running the above code, the execute() python function, with a MyClass object as an argument, fails and returns NULL. However, the Python function is entered, as I can see the output (Entering execute function) in the console output, indicating that the object passed is not, indeed, a valid MyClass object.
There are a lot of examples on how to pass simple types, like ints, doubles or string types to Python from C/C++. But there are very few example showing how to pass a C/C++ object/ pointer, which is kind of puzzling.
The above code, with a CMake file, can be checked out from github:
https://github.com/TotteKarlsson/miniprojects/tree/master/passMeToPython
This code is not to use any boost python or other API's. Cython sounds interesting though, and if it can be used to simplify on the C++ side, it could be acceptable.
This is a partial answer to my own question. I'm saying partial, because I do believe there is a better way.
Building on this post http://swig.10945.n7.nabble.com/Pass-a-Swig-wrapped-C-class-to-embedded-Python-code-td8812.html
I generated the swig runtime header, as described here, section 15.4: http://www.swig.org/Doc2.0/Modules.html#Modules_external_run_time
Including the generated header in the C++ code above, allow the following code to be written:
PyObject* pObj = SWIG_NewPointerObj((void*)&obj, SWIG_TypeQuery("_p_MyClass"), 0 );
This code is using information from the Swig python wrap source files, namely the "swig" name of the type MyClass, i.e. _p_MyClass.
With the above PyObject* as an argument to the PyObject_CallObject function, the python execute() function in the code above executes fine, and the Python code, using the generated python module, do have proper access to the MyClass objects internal data. This is great.
Although the above code illustrate how to pass, and retrieve data between C++ and Python in a quite simple fashion, its not ideal, in my opinion.
The usage of the swig header file in the C++ code is really not that pretty, and in addition, it requires a user to "manually" look into swig generated wrapper code in order to find the "_p_MyClass" code.
There must be a better way!? Perhaps something should be added to the swig interface file in order to get this looking nicer?
PyObject *pValue;
pValue = PyObject_CallMethod(pInstance, "add","(i)",x);
if (pValue)
Py_DECREF(pValue);
else
PyErr_Print();
As a learning process of Python C API I am trying to call a Python function inside of functor passed to tbb parallel_for.
Operation of calling the function crashes instance of Python process.
I am not doing anything not thread safe. I get the item from a list and I then call a Python function with the item passed as an argument to the function. In the end I set the item back to the list. Any hints what have I done wrong ?
It is most likely that you forgot to grab Global Interpreter Lock (GIL) when you call Python function back from C++. For example, TBB module for Python implements this using swig:
class PyCaller : public swig::SwigPtr_PyObject {
public:
using swig::SwigPtr_PyObject::SwigPtr_PyObject; // gets constructors
void operator()() const {
SWIG_PYTHON_THREAD_BEGIN_BLOCK;
PyObject* r = PyObject_CallFunctionObjArgs((PyObject*)*this, NULL);
if(r) Py_DECREF(r);
SWIG_PYTHON_THREAD_END_BLOCK;
}
};
// Usage:
tbb::task_group tg;
void enqueue( PyObject *c ) {
tg.run( PyCaller(c) );
}
And you can see how SWIG implements it - here.
Other options to consider include using Numba's #cfunc(nopython=True) decorator and Cython's nogil attribute which both make function faster and enable Python's function to run in parallel as they remove GIL from the compiled function.
I'm trying to register a Python function for callback in a DLL written in C++ and I can't figure out how to do it.
My C++ code header looks like:
extern "C" {
__declspec(dllexport) void RegisterFunc((void (*)());
}
And the implementation is:
void (*funcCallback)() = NULL;
__declspec(dllimport) void RegisterFunc(void(*func)())
{
funcCallback = func;
}
My Cython pyx file looks like:
cdef extern void RegisterFunc(void (*)())
def PyRegisterFunc(func):
RegisterFunc(func)
When compiled this gives the error message:
Cannot convert Python object to 'void (*)(void)'
If I change the code to:
def PyRegisterFunc(func):
RegisterFunc(<void (*)()>func)
I get the error message:
Python objects cannot be cast to pointers of primitive types
I've also tried &func and get the error message: Cannot take address of Python variable.
What is the correct way of passing this Python function as a pointer to my C code?
UPDATE: I should mention my intended purpose for this Python function. The Python application is written in PyQt and the function will emit a signal. The DLL itself has no knowledge of Qt and is compiled without the Qt libraries. A method in the DLL will call this function indirectly which will emit a signal to the Python application.
See Cython docs: Using Cython Declarations from C