I have a Python app that calls a C++ boost python library and it all works. However, I have a callback C++ to Python scenario where C++ from a boost thread calls python and I get an access violation on the C++ side. If I do exactly the same callback using the python thread it works perfectly. Therefore I suspect that I can not simply callback Python from C++ using a boost thread but need to do something extra for it to work?
The most likely culprit is that the Global Interpreter Lock (GIL) is not being held by a thread when it is invoking Python code, resulting in undefined behavior. Verify all paths that make direct or indirect Python calls, acquire the GIL before invoking Python code.
The GIL is a mutex around the CPython interpreter. This mutex prevents parallel operations to be performed on Python objects. Thus, at any point in time, a max of one thread, the one that has acquired the GIL, is allowed to perform operations on Python objects. When multiple threads are present, invoking Python code whilst not holding the GIL results in undefined behavior.
C or C++ threads are sometimes referred to as alien threads in the Python documentation. The Python interpreter has no ability to control the alien thread. Therefore, alien threads are responsible for managing the GIL to permit concurrent or parallel execution with Python threads. One must meticulously consider:
The stack unwinding, as Boost.Python may throw an exception.
Indirect calls to Python, such as copy-constructors or destructors
One solution is to wrap Python callbacks with a custom type that is aware of GIL management.
Using a RAII-style class to manage the GIL provides an elegant exception-safe solution. For example, with the following with_gil class, when a with_gil object is created, the calling thread acquires the GIL. When the with_gil object is destructed, it restores the GIL state.
/// #brief Guard that will acquire the GIL upon construction, and
/// restore its state upon destruction.
class with_gil
{
public:
with_gil() { state_ = PyGILState_Ensure(); }
~with_gil() { PyGILState_Release(state_); }
with_gil(const with_gil&) = delete;
with_gil& operator=(const with_gil&) = delete;
private:
PyGILState_STATE state_;
};
And its usage:
{
with_gil gil; // Acquire GIL.
// perform Python calls, may throw
} // Restore GIL.
With being able to manage the GIL via with_gil, the next step is to create a functor that properly manages the GIL. The following py_callable class will wrap a boost::python::object and acquire the GIL for all paths in which Python code is invoked:
/// #brief Helper type that will manage the GIL for a python callback.
///
/// #detail GIL management:
/// * Acquire the GIL when copying the `boost::python` object
/// * The newly constructed `python::object` will be managed
/// by a `shared_ptr`. Thus, it may be copied without owning
/// the GIL. However, a custom deleter will acquire the
/// GIL during deletion
/// * When `py_callable` is invoked (operator()), it will acquire
/// the GIL then delegate to the managed `python::object`
class py_callable
{
public:
/// #brief Constructor that assumes the caller has the GIL locked.
py_callable(const boost::python::object& object)
{
with_gil gil;
object_.reset(
// GIL locked, so it is safe to copy.
new boost::python::object{object},
// Use a custom deleter to hold GIL when the object is deleted.
[](boost::python::object* object)
{
with_gil gil;
delete object;
});
}
// Use default copy-constructor and assignment-operator.
py_callable(const py_callable&) = default;
py_callable& operator=(const py_callable&) = default;
template <typename ...Args>
void operator()(Args... args)
{
// Lock the GIL as the python object is going to be invoked.
with_gil gil;
(*object_)(std::forward<Args>(args)...);
}
private:
std::shared_ptr<boost::python::object> object_;
};
By managing the boost::python::object on the free-space, one can freely copy the shared_ptr without having to hold the GIL. This allows for us to safely use the default generated copy-constructor, assignment operator, destructor, etc.
One would use the py_callable as follows:
// thread 1
boost::python::object object = ...; // GIL must be held.
py_callable callback(object); // GIL no longer required.
work_queue.post(callback);
// thread 2
auto callback = work_queue.pop(); // GIL not required.
// Invoke the callback. If callback is `py_callable`, then it will
// acquire the GIL, invoke the wrapped `object`, then release the GIL.
callback(...);
Here is a complete example demonstrating having a Python extension invoke a Python object as a callback from a C++ thread:
#include <memory> // std::shared_ptr
#include <thread> // std::this_thread, std::thread
#include <utility> // std::forward
#include <boost/python.hpp>
/// #brief Guard that will acquire the GIL upon construction, and
/// restore its state upon destruction.
class with_gil
{
public:
with_gil() { state_ = PyGILState_Ensure(); }
~with_gil() { PyGILState_Release(state_); }
with_gil(const with_gil&) = delete;
with_gil& operator=(const with_gil&) = delete;
private:
PyGILState_STATE state_;
};
/// #brief Helper type that will manage the GIL for a python callback.
///
/// #detail GIL management:
/// * Acquire the GIL when copying the `boost::python` object
/// * The newly constructed `python::object` will be managed
/// by a `shared_ptr`. Thus, it may be copied without owning
/// the GIL. However, a custom deleter will acquire the
/// GIL during deletion
/// * When `py_callable` is invoked (operator()), it will acquire
/// the GIL then delegate to the managed `python::object`
class py_callable
{
public:
/// #brief Constructor that assumes the caller has the GIL locked.
py_callable(const boost::python::object& object)
{
with_gil gil;
object_.reset(
// GIL locked, so it is safe to copy.
new boost::python::object{object},
// Use a custom deleter to hold GIL when the object is deleted.
[](boost::python::object* object)
{
with_gil gil;
delete object;
});
}
// Use default copy-constructor and assignment-operator.
py_callable(const py_callable&) = default;
py_callable& operator=(const py_callable&) = default;
template <typename ...Args>
void operator()(Args... args)
{
// Lock the GIL as the python object is going to be invoked.
with_gil gil;
(*object_)(std::forward<Args>(args)...);
}
private:
std::shared_ptr<boost::python::object> object_;
};
BOOST_PYTHON_MODULE(example)
{
// Force the GIL to be created and initialized. The current caller will
// own the GIL.
PyEval_InitThreads();
namespace python = boost::python;
python::def("call_later",
+[](int delay, python::object object) {
// Create a thread that will invoke the callback.
std::thread thread(+[](int delay, py_callable callback) {
std::this_thread::sleep_for(std::chrono::seconds(delay));
callback("spam");
}, delay, py_callable{object});
// Detach from the thread, allowing caller to return.
thread.detach();
});
}
Interactive usage:
>>> import time
>>> import example
>>> def shout(message):
... print message.upper()
...
>>> example.call_later(1, shout)
>>> print "sleeping"; time.sleep(3); print "done sleeping"
sleeping
SPAM
done sleeping
Related
PyCapsule_New accepts a destructor function, which is called when the capsule is destroyed:
PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
I am trying to use this mechanism to pass ownership of an object created by C++ code to Python. Specifically, the destructor simply calls "delete" for the object.
auto ptr = make_unique<ObjType>(arg);
PyObject * ret = PyCapsule_New(ptr.release(), nullptr, Destroyer);
void Destroyer(PyObject *capsule)
{
auto rawPtr = static_cast<ObjType*>(PyCapsule_GetPointer(capsule, nullptr));
delete rawPtr;
}
It seems to me there is a potential memory leak here: If the PyCapsule_New fails, the released raw pointer becomes dangling. I tried to get confirmation from Python C API document. However, it only mentions that upon failure, an exception is set, and a NULL is returned. It doesn't talk about the ownership.
It seems reasonable to assume the pointer will be dangling, because if the capsule is not generated in the first place, there is no handler to be passed to the destructor.
However, I am not sure if PyCapsule_New internally calls the destructor or not, specifically:
Inside PyCapsule_New, the capsule construction is almost complete.
A failure happens just before it returns.
PyCapsule_New sets an exception, returns NULL, after calling the destructor (???)
If the highlighted part is never going to happen, it seems to me the above code will have to be re-written as
auto ptr = make_unique<ObjType>(arg);
PyObject * ret = PyCapsule_New(ptr.get(), nullptr, Destroyer);
if (ret != nullptr)
ptr.release();
Could someone please help confirm if that's the case?
Changing comment to answer, as suggested.
Short answer: No, when PyCapsule_New fails, it does not call the destroyer.
See implementation in https://github.com/python/cpython/blob/master/Objects/capsule.c#L44
PyObject *
PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
{
PyCapsule *capsule;
if (!pointer) {
PyErr_SetString(PyExc_ValueError, "PyCapsule_New called with null pointer");
return NULL;
}
capsule = PyObject_NEW(PyCapsule, &PyCapsule_Type);
if (capsule == NULL) {
return NULL;
}
capsule->pointer = pointer;
capsule->name = name;
capsule->context = NULL;
capsule->destructor = destructor;
return (PyObject *)capsule;
}
As a result, the first implementation does contain potential memory leak. "release()" should only be called when PyCapsule_New is successful.
As a learning process of Python C API I am trying to call a Python function inside of functor passed to tbb parallel_for.
Operation of calling the function crashes instance of Python process.
I am not doing anything not thread safe. I get the item from a list and I then call a Python function with the item passed as an argument to the function. In the end I set the item back to the list. Any hints what have I done wrong ?
It is most likely that you forgot to grab Global Interpreter Lock (GIL) when you call Python function back from C++. For example, TBB module for Python implements this using swig:
class PyCaller : public swig::SwigPtr_PyObject {
public:
using swig::SwigPtr_PyObject::SwigPtr_PyObject; // gets constructors
void operator()() const {
SWIG_PYTHON_THREAD_BEGIN_BLOCK;
PyObject* r = PyObject_CallFunctionObjArgs((PyObject*)*this, NULL);
if(r) Py_DECREF(r);
SWIG_PYTHON_THREAD_END_BLOCK;
}
};
// Usage:
tbb::task_group tg;
void enqueue( PyObject *c ) {
tg.run( PyCaller(c) );
}
And you can see how SWIG implements it - here.
Other options to consider include using Numba's #cfunc(nopython=True) decorator and Cython's nogil attribute which both make function faster and enable Python's function to run in parallel as they remove GIL from the compiled function.
Does boost::python provide any guarantee when the C++ destructor of a
wrapped object is called considering the moment of reaching the zero
reference count of the corresponding python object?
I am concerned about a C++ object that opens a file for writing and performs the file closing in its destructor. Is it guaranteed that the file is written when all python references to the object are deleted or out of scope?
I mean:
A=MyBoostPythonObject()
del A # Is the C++ destructor of MyBoostPythonObject called here?
My experience suggests that the destructor is always called at this point but could not find any guarantee for this.
Boost.Python makes the guarantee that if the Python object has ownership of the wrapped C++ object, then when the Python object is deleted, the wrapped C++ object will be deleted. The Python object's lifetime is dictated by Python, wherein when an object’s reference count reaches zero, the object may be immediately destroyed. For non-simplistic cases, such as cyclic references, the objects will be managed by the garbage collector, and may be destroyed before the program exits.
One Pythonic solution may be to expose a type that implements the context manager protocol. The content manager protocol is made up of a pair of methods: one which will be invoked when entering a runtime context, and one which will be invoked when exiting a runtime context. By using a context manager, one could control the scope in which a file is open.
>>> with MyBoostPythonObject() as A: # opens file.
... A.write(...) # file remains open while in scope.
... # A destroyed once context's scope is exited.
Here is an example demonstrating exposing a RAII-type class to Python as a context manager:
#include <boost/python.hpp>
#include <iostream>
// Legacy API.
struct spam
{
spam(int x) { std::cout << "spam(): " << x << std::endl; }
~spam() { std::cout << "~spam()" << std::endl; }
void perform() { std::cout << "spam::perform()" << std::endl; }
};
/// #brief Python Context Manager for the Spam class.
class spam_context_manager
{
public:
spam_context_manager(int x): x_(x) {}
void perform() { return impl_->perform(); }
// context manager protocol
public:
// Use a static member function to get a handle to the self Python
// object.
static boost::python::object enter(boost::python::object self)
{
namespace python = boost::python;
spam_context_manager& myself =
python::extract<spam_context_manager&>(self);
// Construct the RAII object.
myself.impl_ = std::make_shared<spam>(myself.x_);
// Return this object, allowing caller to invoke other
// methods exposed on this class.
return self;
}
bool exit(boost::python::object type,
boost::python::object value,
boost::python::object traceback)
{
// Destroy the RAII object.
impl_.reset();
return false; // Do not suppress the exception.
}
private:
std::shared_ptr<spam> impl_;
int x_;
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<spam_context_manager>("Spam", python::init<int>())
.def("perform", &spam_context_manager::perform)
.def("__enter__", &spam_context_manager::enter)
.def("__exit__", &spam_context_manager::exit)
;
}
Interactive usage:
>>> import example
>>> with example.Spam(42) as spam:
... spam.perform()
...
spam(): 42
spam::perform()
~spam()
I want to set a Python variable from C++ so that the C++ program can create an object Game* game = new Game(); in order for the Python code to be able to reference this instance (and call functions, etc). How can I achieve this?
I feel like I have some core misunderstanding of the way Python or Boost-Python works.
The line main_module.attr("game") = game is in a try catch statement, and the error (using PyErr_Fetch) is "No to_python (by-value) converter found for C++ type: class Game".
E.g.
class_<Game>("Game")
.def("add", &Game::add)
;
object main_module = import("__main__");
Game* game = new Game();
main_module.attr("game") = game; //This does not work
From Python:
import testmodule
testmodule.game.foo(7)
When dealing with language bindings, one often has to be pedantic in the details. By default, when a C++ object transgresses the language boundary, Boost.Python will create a copy, as this is the safest course of action to prevent dangling references. If a copy should not be made, then one needs to be explicit as to the ownership of the C++ object:
To pass a reference to a C++ object to Python while maintaining ownership in C++, use boost::python::ptr() or boost::ref(). The C++ code should guarantee that the C++ object's lifetime is at least as long as the Python object. When using ptr(), if the pointer is null, then the resulting Python object will be None.
To transfer ownership of a C++ object to Python, one can apply the manage_new_object ResultConverterGenerator, allowing ownership to be transferred to Python. C++ code should not attempt to access the pointer once the Python object's lifetime ends.
For shared ownership, one would need to expose the class with a HeldType of a smart pointer supporting shared semantics, such as boost::shared_ptr.
Once the Python object has been created, it would need to be inserted into a Python namespace to be generally accessible:
From within the module definition, use boost::python::scope to obtain a handle to the current scope. For example, the following would insert x into the example module:
BOOST_PYTHON_MODULE(example)
{
boost::python::scope().attr("x") = ...; // example.x
}
To insert into the __main__ module, one can import __main__. For example, the following would insert x into the __main__ module:
boost::python::import("__main__").attr("x") = ...;
Here is an example demonstrating how to directly construct the Python object from C++, transfer ownership of a C++ object to Python, and construct a Python object that references a C++ object:
#include <iostream>
#include <boost/python.hpp>
// Mockup model.
struct spam
{
spam(int id)
: id_(id)
{
std::cout << "spam(" << id_ << "): " << this << std::endl;
}
~spam()
{
std::cout << "~spam(" << id_ << "): " << this << std::endl;
}
// Explicitly disable copying.
spam(const spam&) = delete;
spam& operator=(const spam&) = delete;
int id_;
};
/// #brief Transfer ownership to a Python object. If the transfer fails,
/// then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
// Transfer ownership to a smart pointer, allowing for proper cleanup
// incase Boost.Python throws.
std::unique_ptr<T> ptr(t);
// Use the manage_new_object generator to transfer ownership to Python.
namespace python = boost::python;
typename python::manage_new_object::apply<T*>::type converter;
// Transfer ownership to the Python handler and release ownership
// from C++.
python::handle<> handle(converter(*ptr));
ptr.release();
return python::object(handle);
}
namespace {
spam* global_spam;
} // namespace
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose spam.
auto py_spam_type = python::class_<spam, boost::noncopyable>(
"Spam", python::init<int>())
.def_readonly("id", &spam::id_)
;
// Directly create an instance of Python Spam and insert it into this
// module's namespace.
python::scope().attr("spam1") = py_spam_type(1);
// Construct of an instance of Python Spam from C++ spam, transfering
// ownership to Python. The Python Spam instance will be inserted into
// this module's namespace.
python::scope().attr("spam2") = transfer_to_python(new spam(2));
// Construct an instance of Python Spam from C++, but retain ownership of
// spam in C++. The Python Spam instance will be inserted into the
// __main__ scope.
global_spam = new spam(3);
python::import("__main__").attr("spam3") = python::ptr(global_spam);
}
Interactive usage:
>>> import example
spam(1): 0x1884d40
spam(2): 0x1831750
spam(3): 0x183bd00
>>> assert(1 == example.spam1.id)
>>> assert(2 == example.spam2.id)
>>> assert(3 == spam3.id)
~spam(1): 0x1884d40
~spam(2): 0x1831750
In the example usage, note how Python did not destroy spam(3) upon exit, as it was not granted ownership of the underlying object.
I'm trying to use SWIG directors to call Python code from C++. However if the code is running in another thread, I must acquire the GIL. My code looks roughly like this (minus the thread stuff):
struct Callback {
virtual ~Callback() {}
virtual void call() {} // will be overwritten in Python
};
struct Foo {
Callback *c;
Foo(Callback* c) : c(c) {}
void doSomething() { c->call(); } // being called from another thread
};
Python:
from demo import *
class MyCallback(Callback):
def call(*args):
print("python callback")
c = MyCallback()
f = Foo(c)
f.doSomething()
Swig:
%module(directors="1", threads="1") demo
%feature("director");
%{
#include <test.h>
%}
%thread;
%include <test.h>
How can I acquire the GIL before calling the Python callback?
To elaborate, the threads feature of SWIG creates code that releases the GIL when C++ code is called, but when Python code is called from C++ it isn't reacquired, so that's what I want to do.
The call stack looks like this: Python -> Foo.doSomething -> C++ -> Callback.call -> Python.
Acquiring the GUIL is explained clearly in Non-Python created threads subsection of Thread State and the Global Interpreter Lock section in Python manual. But you say the issue is how to do this from SWIG generated code. How about this, added to your SWIG .i file:
struct PythonCallback: public Callback
{
virtual void call() {
getGIL();
lockedCall();
releaseGIL();
}
virtual void lockedCall() {} // will be overwritten in Python
};
You only need to expose PythonCallback (not Callback) and its lockedCall method, which you can rename to call if you wish via SWIG's %rename directive. Then in Python you would derive from PythonCallback, and override the call method (which in reality is the lockedCall method if you %rename'd it).