In the following MCVE two different scripts get executed. Both don't do anything special, the first one has an empty function, the second script does literally nothing. But depending on the script PyEval_EvalCode increfs the passed global/local dictionary or not. The documentation doesn't say anything. But depending on the script I have a dangling dictionary after it got executed. When do I have to decref them afterwards or when not? Or what am I missing here? The following C snippet outputs the refcount of the passed dictionaries.
I tried this on Windows with the standard Python-2.7 interpreter.
#pragma comment(lib, "C:\\Python27\\libs\\python27.lib")
#include "C:\\Python27\\include\\Python.h"
static int execute(const char* script)
{
PyObject* globals = PyDict_New();
PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins());
PyObject* code = Py_CompileString(script, "test", Py_file_input);
if (!code)
{
PyErr_Print();
return 0;
}
PyObject* result = PyEval_EvalCode((PyCodeObject*)code, globals, nullptr);
Py_DECREF(code);
if (!result)
{
PyErr_Print();
return 0;
}
Py_DECREF(result);
printf("Refcount of globals: %zd\n", globals->ob_refcnt);
Py_DECREF(globals); // missing decref spotted by user2357112
return 0;
}
int main()
{
const char* script = nullptr;
Py_Initialize();
// First script contains a function
script =
"def main():\n" \
" pass\n" \
"main()\n";
execute(script);
// second script does nothing
script = "12345";
execute(script);
Py_Finalize();
return 0;
}
The extra reference is the function's own reference to its global variable dict. This reference is not your concern, and you do not need to decref that reference.
You only need to perform decrefs to account for clearing references you own.
After tracing everything down I realized there is a circular reference. The function gets a reference to the globals dictionary, and the dictionary keeps the function alive. Calling the garbage collector right after the script got executed fixes the problem and the circular reference gets resolved.
execute(script);
PyGC_Collect();
P.S. Credit goes also to user2357112 who spotted a missing decref in my MCVE.
Related
I am trying to create a Python (2.7.12) extension in C that does the following:
Provide a read only nested dictionary with module level scope for the Python programmer.
A background thread invisible to the Python programmer will add, delete, and modify entries in the dictionary.
The extension will be built directly into the Python interpreter.
I created a simplified version of this extension that adds one entry to the dictionary and then constantly modifies it with new values. Below is the C file containing comments about what it is doing along with my understanding how the reference counts are being handled.
#include <Python.h>
#include <pthread.h>
static PyObject *module;
static PyObject *pyitem_error;
static PyObject *item;
static PyObject *item_handle;
static pthread_t thread;
void *stuff(void *param)
{
int garbage = 0;
PyObject *size;
PyObject *value;
while(1)
{
// Build a dictionary called size containg two integer objects
// Py_BuildValue will pass ownership of its reference to size to this thread
size = NULL;
size = Py_BuildValue("{s:i,s:i}", "l", garbage, "w", garbage);
if(size == NULL)
{
goto error;
}
// Build a dictionary containing an integer object and the size dictionary
// Py_BuildValue will create and own a reference to the size dictionary but not steal it
// Py_BuildValue will pass ownership of its reference to value to this thread
value = NULL;
value = Py_BuildValue("{s:i,s:O}", "h", garbage, "base", size);
if(value == NULL)
{
goto error;
}
// Add the new data to the dictionary
// PyDict_SetItemString will borrow a reference to value
PyDict_SetItemString(item, "dim", value);
error:
Py_XDECREF(size);
Py_XDECREF(value);
garbage++;
}
return NULL;
}
// There will be methods for this module in the future
static PyMethodDef pyitem_methods[] =
{
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC initpyitem(void)
{
// Create a module object
// Own a reference to it since Py_InitModule returns a borrowed reference
module = Py_InitModule("pyitem", pyitem_methods);
Py_INCREF(module);
// Create an exception object for future use
// Own a second reference to it since PyModule_AddObject will steal a reference
pyitem_error = PyErr_NewException("pyitem.error", NULL, NULL);
Py_INCREF(pyitem_error);
PyModule_AddObject(module, "error", pyitem_error);
// Create a dictionary object and a proxy object that makes it read only
// Own a second reference to the proxy object since PyModule_AddObject will steal a reference
item = PyDict_New();
item_handle = PyDictProxy_New(item);
Py_INCREF(item_handle);
PyModule_AddObject(module, "item", item_handle);
// Start the background thread that modifies the dictionary
pthread_create(&thread, NULL, stuff, NULL);
}
Below is a Python program using this extension. All it does is print out what is in the dictionary.
import pyitem
while True:
print pyitem.item
print
This extension seems to work for a while and then crashes with a segmentation fault. An examination of the core dump reveals the following:
Core was generated by `python pyitem_test.py'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 PyObject_Malloc (nbytes=nbytes#entry=42) at Objects/obmalloc.c:831
831 if ((pool->freeblock = *(block **)bp) != NULL) {
[Current thread is 1 (Thread 0x7f144a824700 (LWP 3931))]
This core dump leads me to believe the issue might have to do with my handling of the object reference counts. I believe this might be one cause since problems posed by others with the same core dump resolved the issue by properly handling reference counts. However, I do not see anything wrong with my handling of the object reference counts.
Another thing that comes to mind is that the print function in Python is likely only borrowing a references to the contents of the dictionary. When it is trying to print the dictionary (or access its contents in any other way), the background thread comes along and replaces the old entry with a new one. This causes the reference count of the old entry to decrease and the object is then removed by the garbage collector. However, the print function is still trying to use the old reference which causes an error.
Something that I found interesting is that I can change how quickly or slowly the extension has a segmentation fault by only changing the names of the keys in the dictionaries.
Does anyone have any insights as to what the issue may be? Is there a better way to create the extension and still have the properties that I want?
I believe I have found the cause of the segmentation fault. The background thread is modifying the state of the interpreter without obtaining the Global Interpreter Lock (GIL). This would indeed cause the interpreter to behave in unexpected ways.
To fix this, I first call the function PyEval_InitThreads() in the module initialization function. The next thing to do is enclose any instructions in the background thread that make use of Python C API with the functions PyGILState_Ensure() and PyGILState_Release(). Below is the modified source code with this fix.
#include <Python.h>
#include <pthread.h>
static PyObject *module;
static PyObject *pyitem_error;
static PyObject *item;
static PyObject *item_handle;
static pthread_t thread;
void *stuff(void *param)
{
int garbage = 0;
PyObject *size;
PyObject *value;
PyGILState_STATE state; // Needed for PyGILState_Ensure() and PyGILState_Release()
while(1)
{
// Obtain the GIL
state = PyGILState_Ensure();
size = NULL;
size = Py_BuildValue("{s:i,s:i}", "l", garbage, "w", garbage);
if(size == NULL)
{
goto error;
}
value = NULL;
value = Py_BuildValue("{s:i,s:O}", "h", garbage, "base", size);
if(value == NULL)
{
goto error;
}
PyDict_SetItemString(item, "dim", value);
error:
Py_XDECREF(size);
Py_XDECREF(value);
// Release the GIL
PyGILState_Release(state);
garbage++;
}
return NULL;
}
static PyMethodDef pyitem_methods[] =
{
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC initpyitem(void)
{
module = Py_InitModule("pyitem", pyitem_methods);
Py_INCREF(module);
pyitem_error = PyErr_NewException("pyitem.error", NULL, NULL);
Py_INCREF(pyitem_error);
PyModule_AddObject(module, "error", pyitem_error);
item = PyDict_New();
item_handle = PyDictProxy_New(item);
Py_INCREF(item_handle);
PyModule_AddObject(module, "item", item_handle);
// Initialize Global Interpreter Lock (GIL)
PyEval_InitThreads();
pthread_create(&thread, NULL, stuff, NULL);
}
The extension now runs without any segmentation faults.
PyCapsule_New accepts a destructor function, which is called when the capsule is destroyed:
PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
I am trying to use this mechanism to pass ownership of an object created by C++ code to Python. Specifically, the destructor simply calls "delete" for the object.
auto ptr = make_unique<ObjType>(arg);
PyObject * ret = PyCapsule_New(ptr.release(), nullptr, Destroyer);
void Destroyer(PyObject *capsule)
{
auto rawPtr = static_cast<ObjType*>(PyCapsule_GetPointer(capsule, nullptr));
delete rawPtr;
}
It seems to me there is a potential memory leak here: If the PyCapsule_New fails, the released raw pointer becomes dangling. I tried to get confirmation from Python C API document. However, it only mentions that upon failure, an exception is set, and a NULL is returned. It doesn't talk about the ownership.
It seems reasonable to assume the pointer will be dangling, because if the capsule is not generated in the first place, there is no handler to be passed to the destructor.
However, I am not sure if PyCapsule_New internally calls the destructor or not, specifically:
Inside PyCapsule_New, the capsule construction is almost complete.
A failure happens just before it returns.
PyCapsule_New sets an exception, returns NULL, after calling the destructor (???)
If the highlighted part is never going to happen, it seems to me the above code will have to be re-written as
auto ptr = make_unique<ObjType>(arg);
PyObject * ret = PyCapsule_New(ptr.get(), nullptr, Destroyer);
if (ret != nullptr)
ptr.release();
Could someone please help confirm if that's the case?
Changing comment to answer, as suggested.
Short answer: No, when PyCapsule_New fails, it does not call the destroyer.
See implementation in https://github.com/python/cpython/blob/master/Objects/capsule.c#L44
PyObject *
PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
{
PyCapsule *capsule;
if (!pointer) {
PyErr_SetString(PyExc_ValueError, "PyCapsule_New called with null pointer");
return NULL;
}
capsule = PyObject_NEW(PyCapsule, &PyCapsule_Type);
if (capsule == NULL) {
return NULL;
}
capsule->pointer = pointer;
capsule->name = name;
capsule->context = NULL;
capsule->destructor = destructor;
return (PyObject *)capsule;
}
As a result, the first implementation does contain potential memory leak. "release()" should only be called when PyCapsule_New is successful.
I am new to the business of writing custom Python modules and I am a bit confused how Capsules work. I use Python 2.7.6 from the system OSX installation and try to use Capsules (as recommended for Python > 2.7) for passing pointers around (before they used PyCObject for that). My code does not work at the moment and I would like to get some insights how things should be handled in principle here. The code should define a class LuscherClm and I want be able to do the following:
>>> c40=Luscher(4,0)
>>>
>>> c40(0.12)
>>> <print the result of the evaluation>
First question: at the moment I would have to do something like:
>>> c40=Luscher.init(4,0)
>>>
>>> c40.eval(0.12)
Segfault
My first question is therefore: how do I have to modify the method table to have more operator-style casts instead of the member functions init and eval.
However, my code has other problems and here is the relevant part (the underlying C++ class works smoothly, I use it in production a lot):
The destructor:
//destructor
static void clm_destruct(PyObject* capsule){
void* ptr=PyCapsule_GetPointer(capsule,"zetfunc");
Zetafunc* zetptr=static_cast<Zetafunc*>(ptr);
delete zetptr;
return;
}
The constructor: it returns the pointer to the capsule. I do not know whether this is correct. Because in this case when I call, clm=LuscherClm.init(l,m), the clm object is a PyCapsule and has no attribute eval so that I cannot call clm.eval(x) on that. How should this be handled?
//constructor
static PyObject* clm_init(PyObject* self, PyObject *args){
//return value
PyObject* result=NULL;
//parse variables
unsigned int lval=0;
int mval=0;
if(!PyArg_ParseTuple(args,"li",&lval,&mval)){
::std::cout << "Please specify l and m!" << ::std::endl;
return result;
}
//class instance:
Zetafunc* zetfunc=new Zetafunc(lval,mval);
instanceCapsule=PyCapsule_New(static_cast<void*> (zetfunc),"zetfunc",&clm_destruct);
return instanceCapsule;
}
So how is the capsule passed to the evaluate function? the code below is not correct since I have not updated it after moving from CObjects to Capsules. Shall the capsule be a global variable (I do not like that) or how can I pass it to the evaluation function? Or shall I call it on self, but what is self at the moment?
//evaluate the function
static PyObject* clm_evaluate(PyObject* self, PyObject* args){
//get the PyCObject from the capsule:
void* tmpzetfunc=PyCapsule_GetPointer(instanceCapsule,"zetfunc");
if (PyErr_Occurred()){
std::cerr << "Some Error occured!" << std::endl;
return NULL;
}
Zetafunc* zetfunc=static_cast< Zetafunc* >(tmpzetfunc);
//parse value:
double x;
if(!PyArg_ParseTuple(args,"d",&x)){
std::cerr << "Specify a number at which you want to evaluate the function" << std::endl;
return NULL;
}
double result=(*zetfunc)(x).re();
//return the result as a packed function:
return Py_BuildValue("d",result);
}
//methods
static PyMethodDef LuscherClmMethods[] = {
{"init", clm_init, METH_VARARGS, "Initialize clm class!"},
{"eval", clm_evaluate, METH_VARARGS, "Evaluate the Zeta-Function!"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
Python < 3 initialisation function:
PyMODINIT_FUNC
initLuscherClm(void)
{
PyObject *m = Py_InitModule("LuscherClm", LuscherClmMethods);
return;
}
Can you explain to me what is wrong and why? I would like to stay away from SWIG or boost if possible, since this module should be easily portable and I want to avoid having to install additional packages every time I want to use it somewhere else.
Further: what is the overhead produced by the C/API in calling the function? I need to call it an order of O(10^6) times and I would still like it to be fast.
Ok, I am using boost.python now but I get a segfault when I run object.eval(). That is my procedure now:
BOOST_PYTHON_MODULE(threevecd)
{
class_< threevec<double> >("threevecd",init<double,double,double>());
}
BOOST_PYTHON_MODULE(LuscherClm)
{
class_<Zetafunc>("LuscherClm",init<int,int, optional<double,threevec<double>,double,int> >())
.def("eval",&Zetafunc::operator(),return_value_policy<return_by_value>());
boost::python::to_python_converter<dcomplex,dcomplex_to_python_object>();
}
dcomplex is my own complex number implementation. So I had to write a converter:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
if(fabs(comp.im())<std::numeric_limits<double>::epsilon()){
boost::python::object result=boost::python::object(complex<double>(comp.re(),comp.im()));
return boost::python::incref(result.ptr());
}
else{
return Py_BuildValue("d",comp.re());
}
}
};
Complex128 is a numpy extension which is not understood by boost. So my questions are:
1) how can I return a complex number as a python datatype (is complex a standard python type?)
2) Why do I get a segfault. My result in my testcase is real so it should default to the else statement. I guess that the pointer runs out of scope and thats it. But even in the if-case (where I take care about ref-increments), it segfaults. Can someone help me with the type conversion issue?
Thanks
Thorsten
Ok, I got it. The following converter does the job:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
PyObject* result;
if(std::abs(comp.im())<=std::numeric_limits<double>::epsilon()){
result=PyFloat_FromDouble(comp.re());
}
else{
result=PyComplex_FromDoubles(comp.re(),comp.im());
}
Py_INCREF(result);
return result;
}
};
Using this converter and the post by Wouter, I suppose my question is answered. Thanks
I am new to python and C-extensions. I am writing a python code where I have created two threads and have defined a callback function py_cb().
In one thread I am appending to a global list after certain time intervals, whereas from the other thread, I am calling a C-extension library api. The C-extension api spawns a thread that calls the python callback function (py_cb) defined in my original file. In the callback function, I am trying to display the global list, but it seems it has a different id. I checked it using id(listName). Is there a way I can use the right global variable in the C-extension python callback function? I was assuming that global values will be shared across all the threads, is that not the case? Also, please suggest better solutions.
Following is the C code:
char *MODULE_NAME = "simple";
---
PyMODINIT_FUNC init_cmodule(void)
{
PyObject *m = Py_InitModule3("_cmodule",module_methods,module_docstring);
if (m == NULL)
return;
}
static PyObject *cmodule_cmodule(PyObject *self, PyObject *args)
{
int value = register_cb();
PyObject *ret = Py_BuildValue("i",value);
return ret;
}
void notifyFooHandle()
{
printf("inside the notify_foo function\n");
pModule = PyImport_ImportModule(MODULE_NAME);
if (!pModule) {
printf ("Failed to load the module\n");
return;
}
PyObject *pFunc;
pFunc = PyObject_GetAttrString(pModule, "py_cb");
if (pFunc && PyCallable_Check(pFunc)) {
PyObject_CallObject(pFunc,NULL);
}
else {
Py_DECREF(pFunc);
PyErr_Print();
printf("Failed to send notification\n");
return;
}
return;
}
void notify_foo(void)
{
int t = 5;
while (t < 10) {
notifyFooHandle();
sleep(5);
t++;
}
return;
}
int register_cb(void)
{
pthread_t notify_t;
int rc = pthread_create(¬ify_t, NULL, (void *) ¬ify_foo,NULL);
---
}
Following is the python callback API in simple.py file:
def py_cb():
print "Received a call back from cmodule"
global myList
print "Global myList is ", id(myList)
print myList
return
Can you show us the code in your C library? If you're initializing a python VM there and then calling py_cb, it's easy to understand why the lists are different: you have two different python VM instances.
EDIT:
I think your problem is you're using two different python instances. First, you have your main program in python. In that instance you have a global "myList" so every function invoked from there will be accessing that particular instance of "myList". Then, you load a C module. When that C module opens you original python module in order to load py_cb, you are using a different python instance, you'll have a second "myList". In short, you have TWO different python instances running, the one you create when you run your main python script and one you create inside your C library to call py_cb.
If you want to share a common python instance, you'll have to create you instance in C, make it globally accessible in your C module, insert there you C functions and then run your main python function. When python calls to a C function, you'll be inside your original address space and not a new one, and when you make a call back to python you'll be using always the same python instance.
Hi I am trying to return a value (119599936) from the embedded python to C++
Py_Initialize();
PyObject* myModuleString = PyString_FromString((char*)"memory_leak_test");
PyObject* myModule = PyImport_Import(myModuleString);
PyObject* myFunction = PyObject_GetAttrString(myModule,(char*)"begin");
PyObject* args = PyTuple_Pack(1,PyString_FromString("CacheSetup"));
PyObject* myResult = PyObject_CallObject(myFunction, args);
double result = PyFloat_AsDouble(myResult);// <<<<<<<<< I guess thts where i'm goin wrong
std::cout << result << std::endl;
Py_Finalize();
I tried
unsigned long result = PyLong_AsUnsignedLong(myResult);
unsigned long long result = PyLong_AsUnsignedLongLongMask(myResult);
double result = PyLong_AsDouble(myResult2);
None of the above seemed to work for the when I try to return the number 119599936, the error i'm getting is
Exception SystemError: '..\\Objects\\longobject.c:739: bad argument to internal
function' in 'garbage collection' ignored
Fatal Python error: unexpected exception during garbage collection
Since your code performs no error checking whatsoever, and you don't provide the source to the memory_leak_test module, it's anyone's guess what went wrong. My guess would be that memory_leak_test failed to import, or that memory_leak_test.begin("CacheSetup") raised an exception or returned None instead of returning a number like your code expects.
To debug this problem, rewrite with the following guidelines in mind:
Check errors from Python/C functions that may return them. Whenever an error you can't handle appears, call PyErr_Print() and terminate current execution (unless you are expecting the error and want to handle it properly - in that case, call PyErr_Clear() instead.)
Invoke Py_DECREF on objects returned as new references once you no longer need them. Examples of these are the return values of PyString_FromString, PyTuple_Pack, and PyObject_CallOject.
Learn of and use convenience functions such as PyObject_CallFunction, PyObject_CallMethod, and PyImport_ImportModule which significantly reduce boilerplate code for creating temporary strings and tuples. The best way to find these is by perusing existing Python/C sources, such as those that come with Python.
Use PyObject_Print() to print intermediate results for debugging.
Here is a possible modification of your code according to the above:
Py_Initialize();
PyObject *module = PyImport_ImportModule("memory_leak_test");
if (!module) {
PyErr_Print();
return;
}
PyObject *result = PyObject_CallMethod(module, "begin", "s", "CacheSetup");
Py_DECREF(module);
if (!result) {
PyErr_Print();
return;
}
printf("got result of type %s\n", Py_TYPE(result)->tp_name);
PyObject_Print(result, stdout, 0);
putchar('\n');
Py_DECREF(result);
Py_Finalize();