I'm new to the Python C-API and browsing through some source code to pick parts of it up.
Here is a minimal version of a function that I found, in the C source of a package that contains extension modules:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
static PyObject *
modulename_myfunc(PyObject *self, PyObject *args) {
// Call PyArg_ParseTuple, etc ...
// Dummy values; in the real function they are calculated
int is_found = 1;
Py_ssize_t n_bytes_found = 1024;
PyObject *result;
result = Py_BuildValue("(Oi)",
is_found ? Py_True : Py_False, // Py_INCREF?
n_bytes_found);
return result;
}
Does this introduce a small memory leak by failing to use Py_INCREF on either Py_True or Py_False? The C-API docs for Boolean object seem pretty explicit about always needing to incref/decref Py_True and Py_False.
If a Py_INCREF does need to be introduced, how can it most properly be used here, assuming that Py_RETURN_TRUE/Py_RETURN_FALSE aren't really applicable because a tuple is being returned?
The reason a Py_INCREF is not used here is because Py_BuildValue, when being passed an object with "O" will increment the reference count for you:
O (object) [PyObject *]
Pass a Python object untouched (except for its reference count, which is incremented by one). If the object passed in is a NULL pointer, it is assumed that this was caused because the call producing the argument found an error and set an exception. Therefore, Py_BuildValue() will return NULL but won’t raise an exception. If no exception has been raised yet, SystemError is set.
You'll see a similar usage here in CPython itself for example.
Related
So I have this c extension function which loads a python module and uses a list of c++ strings to get a specific global attribute from that module:
PyObject* get_global_constant(const char* module_name, std::vector<std::string> constant_names) {
/* Gets a global variable from a Python module */
PyObject *temp_module = PyImport_ImportModule(module_name);
PyObject *global_var = PyImport_AddModule(module_name);
for (std::string constant : constant_names) {
global_var = PyObject_GetAttrString(global_var, constant.c_str());
}
Py_DECREF(temp_module);
return global_var;
}
Does this leak?
Every call to PyObject_GetAttrString leaks a reference (excluding the final call, for which you return the reference). It returns a new reference so you need to call Py_DECREF on that reference.
You should also be error-checking your return values (most Python C API functions return NULL if they raise an exception). If you're getting a segmentation fault it's most likely because you're not error checking.
PyImport_AddModule seems pointless given that you just got a reference to the module when you imported it.
I am implementing a C function as an extension for Python. Inside abstract.h, I found the following:
/* ==== Iterators ================================================ */
/* Takes an object and returns an iterator for it.
This is typically a new iterator but if the argument is an iterator, this
returns itself. */
PyAPI_FUNC(PyObject *) PyObject_GetIter(PyObject *);
/* Returns 1 if the object 'obj' provides iterator protocols, and 0 otherwise.
This function always succeeds. */
PyAPI_FUNC(int) PyIter_Check(PyObject *);
When I try to get iterators using PyObject_GetIter on obviously non-iterable objects like a number, I get the error SystemError: <built-in function xxx> returned a result with an error set.
static PyObject *my_method(PyObject *self, PyObject *args)
{
PyObject *obj;
PyArg_ParseTuple(args, "O", &obj)
// printf("\ncheck %d",PyIter_Check(obj)); // always 0
PyObject *iter = PyObject_GetIter(obj); // throws error
return PyLong_FromLong(0);
}
I would like to handle errors on my own. So I tried to use the PyIter_Check to test if the object has an iterator. However, this function returned 0 for all objects I provided, including the iterable ones.
I thought that it might be caused by the PyAPI_FUNC() macro but I found it in pyport.h and it seems to be only adding __declspec.
Why is the function PyIter_Check returning zero for all objects?
PyIter_Check is for checking whether an object is an iterator, not whether it can provide one. There doesn’t appear to be a PyIterable_Check.
Moreover, Python pretty much enforces EAFP: since anything could provide an __iter__ that raises an exception, you have to check for an error from PyObject_GetIter anyway, so the only point of the *_Check functions is to provide early sanity checks (sometimes with better error messages).
I am trying to write some python function in crystal-lang through the C Python API.
My code follows:
METH_VARARGS = 0x0001
#[Link("python3.5m")]
lib Python
alias PyObject = Void*
struct PyMethodDef
name : UInt8*
func : Void*
flags : LibC::Int
doc : UInt8*
end
fun Py_Initialize
fun Py_Finalize
fun PyObject_CallObject(func : PyObject, args : PyObject) : PyObject
fun PyCFunction_NewEx(method : PyMethodDef*, __self__ : PyObject, ) : PyObject
fun PyLong_AsLong(n : PyObject) : Int64
fun PyLong_FromLong(n : Int64) : PyObject
end
def new_method_def(name : String, function, flags : LibC::Int)
x = Pointer(Python::PyMethodDef).malloc(1)
x.value.name = name.to_unsafe
x.value.func = function
x.value.flags = flags
x.value.doc = nil
x
end
Python.Py_Initialize
a = ->(args : Void*) {
puts Python.PyLong_AsLong(args)
Pointer(Void).null
}
name = "num"
number = Python.PyLong_FromLong(1)
Python.Py_IncRef(number)
method = Python.PyCFunction_NewEx(new_method_def(name,a.pointer,METH_VARARGS),number)
Python.PyObject_CallObject(method,Pointer(Void).null)
Python.Py_Finalize
Everything works if I set nil instead of number when in PyCFunction_NewEx, but as the code is, it throws an invalid acces memory exception when Py_Finalize is called.
I can't understand what's causing it.
Can someone help me?
The root problem here is that you're calling a C function of three parameters with only two arguments.
Regrettably, PyCFunction_NewEx is missing from the documentation, despite being a public API function. But all of the examples using it pass three arguments. And if you go to the source:
PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)
That's 3.7, but this is the same in 3.5 and in 2.7, and in every other version since the function was added to the API in 2.3. The whole point of NewEx is to allow you to pass a module.
Presumably, the function is expecting that third argument either in a register or on the stack, and you haven't put anything there, so it's completely arbitrary what you're passing. Slightly different code will leave completely different values in those places, so it's not surprising that you get different results:
If the value happens to be 0, that's fine; you're allowed to pass NULL as the module value.
If the value happens to be something that points to unmapped memory, like, say, 1 (as in the raw C long/long long, not a PyLongObject), you should get a segfault from the attempt to incref the module.
If the value happens to be a pointer to some random thing in memory, the incref will work, but will corrupt that random thing. Which could do just about anything, but a mysterious segfault at some arbitrary later point is almost the least surprising thing it could do.
Meanwhile, from a comment:
I am calling PyCFunction_NewEx because PyCFunction_New is a marco in the source code.
If you're using Python 2.3-2.6 or 3.0-3.2, then sure. But in later versions, including the 3.5 you say you're using, CPython goes out of its way to define PyCFunction_New as a function specifically so that it will be present in the API (and even the stable API, for 3.x). See 3.5 for example:
/* undefine macro trampoline to PyCFunction_NewEx */
#undef PyCFunction_New
PyAPI_FUNC(PyObject *)
PyCFunction_New(PyMethodDef *ml, PyObject *self)
{
return PyCFunction_NewEx(ml, self, NULL);
}
So, you really can just call PyCFunction_New.
PyCapsule_New accepts a destructor function, which is called when the capsule is destroyed:
PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
I am trying to use this mechanism to pass ownership of an object created by C++ code to Python. Specifically, the destructor simply calls "delete" for the object.
auto ptr = make_unique<ObjType>(arg);
PyObject * ret = PyCapsule_New(ptr.release(), nullptr, Destroyer);
void Destroyer(PyObject *capsule)
{
auto rawPtr = static_cast<ObjType*>(PyCapsule_GetPointer(capsule, nullptr));
delete rawPtr;
}
It seems to me there is a potential memory leak here: If the PyCapsule_New fails, the released raw pointer becomes dangling. I tried to get confirmation from Python C API document. However, it only mentions that upon failure, an exception is set, and a NULL is returned. It doesn't talk about the ownership.
It seems reasonable to assume the pointer will be dangling, because if the capsule is not generated in the first place, there is no handler to be passed to the destructor.
However, I am not sure if PyCapsule_New internally calls the destructor or not, specifically:
Inside PyCapsule_New, the capsule construction is almost complete.
A failure happens just before it returns.
PyCapsule_New sets an exception, returns NULL, after calling the destructor (???)
If the highlighted part is never going to happen, it seems to me the above code will have to be re-written as
auto ptr = make_unique<ObjType>(arg);
PyObject * ret = PyCapsule_New(ptr.get(), nullptr, Destroyer);
if (ret != nullptr)
ptr.release();
Could someone please help confirm if that's the case?
Changing comment to answer, as suggested.
Short answer: No, when PyCapsule_New fails, it does not call the destroyer.
See implementation in https://github.com/python/cpython/blob/master/Objects/capsule.c#L44
PyObject *
PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
{
PyCapsule *capsule;
if (!pointer) {
PyErr_SetString(PyExc_ValueError, "PyCapsule_New called with null pointer");
return NULL;
}
capsule = PyObject_NEW(PyCapsule, &PyCapsule_Type);
if (capsule == NULL) {
return NULL;
}
capsule->pointer = pointer;
capsule->name = name;
capsule->context = NULL;
capsule->destructor = destructor;
return (PyObject *)capsule;
}
As a result, the first implementation does contain potential memory leak. "release()" should only be called when PyCapsule_New is successful.
I've am learning about Python-C extensions and am puzzled as to why methods that use keyword arguments must be cast to PyCFunctions.
My understanding of a PyCFunction is that it takes two pointers to PyObjects and returns a single pointer to a PyObject - e.g.
PyObject* myFunc(PyObject* self, PyObject* args)
If I'm going to use a function that uses keyword arguments, then this function will take three pointers to PyObjects and returns a single pointer to a PyObject - e.g.
PyObject* myFunc(PyObject* self, PyObject* args, PyObject* keywordArgs)
However, when I create the module functions array (for a function called 'adder'):
{ "adder", (PyCFunction)adder, METH_VARARGS | METH_KEYWORDS, "adder method" }
works fine. It feels like I cast a float to an int and still got to use the non-integer parts of the float. If I didn't see this work, I would have thought it wouldn't work. What am I not understanding here?
Also, I saw some references to a PyCFunctionWithKeywords, which seems to have the function signature I thought I needed, but my compiler complained (gave warnings) about 'incompatible pointer types'.
Was PyCFunctionWithKeywords deprecated? If not, is there a time when I should/must use it?
If your function handles keyword arguments, then it must correspond to a PyCFunctionWithKeywords. However, C doesn’t do overloading, and the structure built by PyMethodDef is defined to expect a PyCFunction, rather than, say, a completely unchecked void *. So you must cast your PyCFunctionWithKeywords to a PyCFunction to stop the compiler complaining, that’s all.
Remember that you must also pass METH_KEYWORDS in the flags to tell Python that your function has the signature of a PyCFunctionWithKeywords, not a PyCFunction.
D'Olveiro is 100% correct, but (depending on your compiler) you may still get a "cast-function-type" warning, which you can safely ignore. Or (if using a gcc varient) surround with pragma to temporarily turn off that warning:
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wcast-function-type"
static PyMethodDef MyPythonMethods[] = {
{"myMethod", (PyCFunction)MyMethodFunction, METH_VARARGS, "doc string"},
...
{NULL, NULL, 0, NULL}
};
#pragma GCC diagnostic pop