As a learning process of Python C API I am trying to call a Python function inside of functor passed to tbb parallel_for.
Operation of calling the function crashes instance of Python process.
I am not doing anything not thread safe. I get the item from a list and I then call a Python function with the item passed as an argument to the function. In the end I set the item back to the list. Any hints what have I done wrong ?
It is most likely that you forgot to grab Global Interpreter Lock (GIL) when you call Python function back from C++. For example, TBB module for Python implements this using swig:
class PyCaller : public swig::SwigPtr_PyObject {
public:
using swig::SwigPtr_PyObject::SwigPtr_PyObject; // gets constructors
void operator()() const {
SWIG_PYTHON_THREAD_BEGIN_BLOCK;
PyObject* r = PyObject_CallFunctionObjArgs((PyObject*)*this, NULL);
if(r) Py_DECREF(r);
SWIG_PYTHON_THREAD_END_BLOCK;
}
};
// Usage:
tbb::task_group tg;
void enqueue( PyObject *c ) {
tg.run( PyCaller(c) );
}
And you can see how SWIG implements it - here.
Other options to consider include using Numba's #cfunc(nopython=True) decorator and Cython's nogil attribute which both make function faster and enable Python's function to run in parallel as they remove GIL from the compiled function.
Related
I want to create a function in python, pass it's function pointer to c and execute it there.
So my python file:
import ctypes
import example
def tester_print():
print("Hello")
my_function_ptr = ctypes.CFUNCTYPE(None)(tester_print)
example.pass_func(my_function_ptr)
And here is what my function in c looks like:
typedef void (*MyFunctionType)(void);
PyObject* pass_func(PyObject *self, PyObject* args)
{
PyObject* callable_object;
if (!PyArg_ParseTuple(args, "O", &callable_object))
return NULL;
if (!PyCallable_Check(callable_object))
{
PyErr_SetString(PyExc_TypeError, "The object is not a callable function.");
return NULL;
}
PyObject* function_pointer = PyCapsule_New(callable_object, "my_function_capsule", NULL);
if (function_pointer == NULL) return NULL;
MyFunctionType my_function = (MyFunctionType) PyCapsule_GetPointer(function_pointer, "my_function_capsule");
if (my_function == NULL) return NULL;
my_function(); // Or (*my_function)() Both same result.
// PyCapsule_Free(function_pointer);
Py_RETURN_NONE;
}
Doing this causes a seg fault on my_function() call. How can I do this?
If you're just trying to pass a Python function to a C extension, pass it directly (don't use ctypes) and use PyObject_Call to call it:
example.pass_func(tester_print)
and
PyObject_CallNoArgs(callable_object);
If you need a real C function pointer for whatever reason, the usual approach is to write a C wrapper that takes the callable as an argument:
void callable_wrapper(PyObject *func) {
PyObject_CallNoArgs(func);
// plus whatever other code you need (e.g. reference counting, return value handling)
}
Most reasonable C APIs that take a callback function also provide a way to add an arbitrary argument to the callable ("user data"); for example, with pthreads:
result = pthread_create(&tid, &attr, callable_wrapper, callable_object);
Make sure to handle reference counting correctly: increment the reference on your callable object before passing it to the C API, and decrement the reference when it is no longer needed (e.g. if the callback is only called once, the callable_wrapper could DECREF before returning).
When using threads, you additionally need to ensure that you hold the GIL when calling any Python code; see https://docs.python.org/3/c-api/init.html#non-python-created-threads for more details and a code sample.
What your current code is doing is receiving a pointer to a ctypes CFUNCTYPE object as callable_object, placing that pointer in a capsule, taking it back out again, and calling it as if it was a C function pointer. This doesn't work, since it effectively attempts to call the CFUNCTYPE object as if it were a C function (the capsule stuff winds up being useless). When you're using the Python C API, there's almost never any need for ctypes in Python, because the C API can directly interact with Python objects.
I want to enable the writing of user python-plugins for an existing C++ application. The main application will emit signals using boost::signals2 that trigger the execution of the users python code. I set up a working sample where an initialize method in the users python module is being called. The user can subscribe to signals calling a "connect"-method:
import ctypes
CALLBACK_VOID_FUNC = CFUNCTYPE(None)
def myVoidHandler():
print("myVoidHandler (python)")
def initialize(myCHandle):
global myCDll, callback1
myCDll = ctypes.CDLL("CppEventsd", handle = myCHandle)
[...]
callback1 = CALLBACK_VOID_FUNC(myVoidHandler)
myCDll.connect(b"readyToGo", callback1)
[...]
The signature of the connect method:
extern "C" __declspec(dllexport) bool connect(char* signalName, void(*pF)())
Now the python function myVoidHandler could be called from cpp just like this:
pF();
But more importantly, pF can be connected to a boost signal.
However I want to spare the users going through this (and spare myself the support requests because they did not get it quite right), so I want to connect the python functions from the cpp side by using a naming convention for the functions/slots.
I can easily retrieve a PyObjet pointer to the python callback:
PyObject *pModule, *pDict, *pVoidHandlerF;
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pVoidHandlerF = PyDict_GetItemString(pDict, "myVoidHandler");
BUT I cannot figure out how to get the actual function pointer (equivalent to pF) from the cpp side! Some transformation seems to be going on in the python to Cpp transition.
Any ideas?
Instead of connecting directly to the python function pointer, you could create your own C++ function that uses the python C-API to invoke the python function. Here is an example:
boost::signals2::signal<void(void)>& sig = [...];
PyObject* pVoidHandlerF = [...];
sig.connect([=]() {
PyObject_CallObject(pVoidHandlerF, NULL);
});
This allows you to do other stuff on the C++ side, for example, passing arguments, returning values, checking for errors, etc.
I am new to python and C-extensions. I am writing a python code where I have created two threads and have defined a callback function py_cb().
In one thread I am appending to a global list after certain time intervals, whereas from the other thread, I am calling a C-extension library api. The C-extension api spawns a thread that calls the python callback function (py_cb) defined in my original file. In the callback function, I am trying to display the global list, but it seems it has a different id. I checked it using id(listName). Is there a way I can use the right global variable in the C-extension python callback function? I was assuming that global values will be shared across all the threads, is that not the case? Also, please suggest better solutions.
Following is the C code:
char *MODULE_NAME = "simple";
---
PyMODINIT_FUNC init_cmodule(void)
{
PyObject *m = Py_InitModule3("_cmodule",module_methods,module_docstring);
if (m == NULL)
return;
}
static PyObject *cmodule_cmodule(PyObject *self, PyObject *args)
{
int value = register_cb();
PyObject *ret = Py_BuildValue("i",value);
return ret;
}
void notifyFooHandle()
{
printf("inside the notify_foo function\n");
pModule = PyImport_ImportModule(MODULE_NAME);
if (!pModule) {
printf ("Failed to load the module\n");
return;
}
PyObject *pFunc;
pFunc = PyObject_GetAttrString(pModule, "py_cb");
if (pFunc && PyCallable_Check(pFunc)) {
PyObject_CallObject(pFunc,NULL);
}
else {
Py_DECREF(pFunc);
PyErr_Print();
printf("Failed to send notification\n");
return;
}
return;
}
void notify_foo(void)
{
int t = 5;
while (t < 10) {
notifyFooHandle();
sleep(5);
t++;
}
return;
}
int register_cb(void)
{
pthread_t notify_t;
int rc = pthread_create(¬ify_t, NULL, (void *) ¬ify_foo,NULL);
---
}
Following is the python callback API in simple.py file:
def py_cb():
print "Received a call back from cmodule"
global myList
print "Global myList is ", id(myList)
print myList
return
Can you show us the code in your C library? If you're initializing a python VM there and then calling py_cb, it's easy to understand why the lists are different: you have two different python VM instances.
EDIT:
I think your problem is you're using two different python instances. First, you have your main program in python. In that instance you have a global "myList" so every function invoked from there will be accessing that particular instance of "myList". Then, you load a C module. When that C module opens you original python module in order to load py_cb, you are using a different python instance, you'll have a second "myList". In short, you have TWO different python instances running, the one you create when you run your main python script and one you create inside your C library to call py_cb.
If you want to share a common python instance, you'll have to create you instance in C, make it globally accessible in your C module, insert there you C functions and then run your main python function. When python calls to a C function, you'll be inside your original address space and not a new one, and when you make a call back to python you'll be using always the same python instance.
I'm trying to use SWIG directors to call Python code from C++. However if the code is running in another thread, I must acquire the GIL. My code looks roughly like this (minus the thread stuff):
struct Callback {
virtual ~Callback() {}
virtual void call() {} // will be overwritten in Python
};
struct Foo {
Callback *c;
Foo(Callback* c) : c(c) {}
void doSomething() { c->call(); } // being called from another thread
};
Python:
from demo import *
class MyCallback(Callback):
def call(*args):
print("python callback")
c = MyCallback()
f = Foo(c)
f.doSomething()
Swig:
%module(directors="1", threads="1") demo
%feature("director");
%{
#include <test.h>
%}
%thread;
%include <test.h>
How can I acquire the GIL before calling the Python callback?
To elaborate, the threads feature of SWIG creates code that releases the GIL when C++ code is called, but when Python code is called from C++ it isn't reacquired, so that's what I want to do.
The call stack looks like this: Python -> Foo.doSomething -> C++ -> Callback.call -> Python.
Acquiring the GUIL is explained clearly in Non-Python created threads subsection of Thread State and the Global Interpreter Lock section in Python manual. But you say the issue is how to do this from SWIG generated code. How about this, added to your SWIG .i file:
struct PythonCallback: public Callback
{
virtual void call() {
getGIL();
lockedCall();
releaseGIL();
}
virtual void lockedCall() {} // will be overwritten in Python
};
You only need to expose PythonCallback (not Callback) and its lockedCall method, which you can rename to call if you wish via SWIG's %rename directive. Then in Python you would derive from PythonCallback, and override the call method (which in reality is the lockedCall method if you %rename'd it).
I am trying to embed python within an exiting C++ application. However I am stuck at trying to call an instance member function from the python script.
I have an existing class in which I have a private member function that wraps the native function and meets the PyCFunction interface.
class MY_CLASS {
public:
void MY_CLASS();
void submit(TypeA a, int b);
private:
PyObject* PythonMethod_submit(PyObject*, PyObject*);
}
When I create the PyMethodDef I use the method name to identify the function I want however I get a compliation error indicating I have to the wrong type signature.
MY_CLASS::MY_CLASS() {
// Python C/API Intilization stuff
PyMethodDef ModuleMethods[] = {
{ "submit",
PythonMethod_submit,
METH_VARARGS,
"docstring" }
};
Py_InitModule("myclass", ModuleMethods);
// Further interaction with the embeded interpreter
}
If I use a static function as described in the Python C API this process works, when I use the class member I recieve an error asserting PyObject* (MY_CLASS::)(PyObject*, PyObject*) does not match PyObject* (*)(PyObject*, PyObject*).
Is it possible to pass a member function as function pointer as stated in the error?
No. A non-static member function has a MY_CLASS * implicitly passed in (as this), whereas a static member function, which behaves like a C function, does not. See this page for the different type signatures: http://www.parashift.com/c++-faq/fnptr-vs-memfnptr-types.html
What you can do as an alternative is have a member in your Python object that holds a pointer (MY_CLASS *) to your C++ object.