So I have this c extension function which loads a python module and uses a list of c++ strings to get a specific global attribute from that module:
PyObject* get_global_constant(const char* module_name, std::vector<std::string> constant_names) {
/* Gets a global variable from a Python module */
PyObject *temp_module = PyImport_ImportModule(module_name);
PyObject *global_var = PyImport_AddModule(module_name);
for (std::string constant : constant_names) {
global_var = PyObject_GetAttrString(global_var, constant.c_str());
}
Py_DECREF(temp_module);
return global_var;
}
Does this leak?
Every call to PyObject_GetAttrString leaks a reference (excluding the final call, for which you return the reference). It returns a new reference so you need to call Py_DECREF on that reference.
You should also be error-checking your return values (most Python C API functions return NULL if they raise an exception). If you're getting a segmentation fault it's most likely because you're not error checking.
PyImport_AddModule seems pointless given that you just got a reference to the module when you imported it.
Related
I want to create a function in python, pass it's function pointer to c and execute it there.
So my python file:
import ctypes
import example
def tester_print():
print("Hello")
my_function_ptr = ctypes.CFUNCTYPE(None)(tester_print)
example.pass_func(my_function_ptr)
And here is what my function in c looks like:
typedef void (*MyFunctionType)(void);
PyObject* pass_func(PyObject *self, PyObject* args)
{
PyObject* callable_object;
if (!PyArg_ParseTuple(args, "O", &callable_object))
return NULL;
if (!PyCallable_Check(callable_object))
{
PyErr_SetString(PyExc_TypeError, "The object is not a callable function.");
return NULL;
}
PyObject* function_pointer = PyCapsule_New(callable_object, "my_function_capsule", NULL);
if (function_pointer == NULL) return NULL;
MyFunctionType my_function = (MyFunctionType) PyCapsule_GetPointer(function_pointer, "my_function_capsule");
if (my_function == NULL) return NULL;
my_function(); // Or (*my_function)() Both same result.
// PyCapsule_Free(function_pointer);
Py_RETURN_NONE;
}
Doing this causes a seg fault on my_function() call. How can I do this?
If you're just trying to pass a Python function to a C extension, pass it directly (don't use ctypes) and use PyObject_Call to call it:
example.pass_func(tester_print)
and
PyObject_CallNoArgs(callable_object);
If you need a real C function pointer for whatever reason, the usual approach is to write a C wrapper that takes the callable as an argument:
void callable_wrapper(PyObject *func) {
PyObject_CallNoArgs(func);
// plus whatever other code you need (e.g. reference counting, return value handling)
}
Most reasonable C APIs that take a callback function also provide a way to add an arbitrary argument to the callable ("user data"); for example, with pthreads:
result = pthread_create(&tid, &attr, callable_wrapper, callable_object);
Make sure to handle reference counting correctly: increment the reference on your callable object before passing it to the C API, and decrement the reference when it is no longer needed (e.g. if the callback is only called once, the callable_wrapper could DECREF before returning).
When using threads, you additionally need to ensure that you hold the GIL when calling any Python code; see https://docs.python.org/3/c-api/init.html#non-python-created-threads for more details and a code sample.
What your current code is doing is receiving a pointer to a ctypes CFUNCTYPE object as callable_object, placing that pointer in a capsule, taking it back out again, and calling it as if it was a C function pointer. This doesn't work, since it effectively attempts to call the CFUNCTYPE object as if it were a C function (the capsule stuff winds up being useless). When you're using the Python C API, there's almost never any need for ctypes in Python, because the C API can directly interact with Python objects.
I'm new to the Python C-API and browsing through some source code to pick parts of it up.
Here is a minimal version of a function that I found, in the C source of a package that contains extension modules:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
static PyObject *
modulename_myfunc(PyObject *self, PyObject *args) {
// Call PyArg_ParseTuple, etc ...
// Dummy values; in the real function they are calculated
int is_found = 1;
Py_ssize_t n_bytes_found = 1024;
PyObject *result;
result = Py_BuildValue("(Oi)",
is_found ? Py_True : Py_False, // Py_INCREF?
n_bytes_found);
return result;
}
Does this introduce a small memory leak by failing to use Py_INCREF on either Py_True or Py_False? The C-API docs for Boolean object seem pretty explicit about always needing to incref/decref Py_True and Py_False.
If a Py_INCREF does need to be introduced, how can it most properly be used here, assuming that Py_RETURN_TRUE/Py_RETURN_FALSE aren't really applicable because a tuple is being returned?
The reason a Py_INCREF is not used here is because Py_BuildValue, when being passed an object with "O" will increment the reference count for you:
O (object) [PyObject *]
Pass a Python object untouched (except for its reference count, which is incremented by one). If the object passed in is a NULL pointer, it is assumed that this was caused because the call producing the argument found an error and set an exception. Therefore, Py_BuildValue() will return NULL but won’t raise an exception. If no exception has been raised yet, SystemError is set.
You'll see a similar usage here in CPython itself for example.
I am trying to write some python function in crystal-lang through the C Python API.
My code follows:
METH_VARARGS = 0x0001
#[Link("python3.5m")]
lib Python
alias PyObject = Void*
struct PyMethodDef
name : UInt8*
func : Void*
flags : LibC::Int
doc : UInt8*
end
fun Py_Initialize
fun Py_Finalize
fun PyObject_CallObject(func : PyObject, args : PyObject) : PyObject
fun PyCFunction_NewEx(method : PyMethodDef*, __self__ : PyObject, ) : PyObject
fun PyLong_AsLong(n : PyObject) : Int64
fun PyLong_FromLong(n : Int64) : PyObject
end
def new_method_def(name : String, function, flags : LibC::Int)
x = Pointer(Python::PyMethodDef).malloc(1)
x.value.name = name.to_unsafe
x.value.func = function
x.value.flags = flags
x.value.doc = nil
x
end
Python.Py_Initialize
a = ->(args : Void*) {
puts Python.PyLong_AsLong(args)
Pointer(Void).null
}
name = "num"
number = Python.PyLong_FromLong(1)
Python.Py_IncRef(number)
method = Python.PyCFunction_NewEx(new_method_def(name,a.pointer,METH_VARARGS),number)
Python.PyObject_CallObject(method,Pointer(Void).null)
Python.Py_Finalize
Everything works if I set nil instead of number when in PyCFunction_NewEx, but as the code is, it throws an invalid acces memory exception when Py_Finalize is called.
I can't understand what's causing it.
Can someone help me?
The root problem here is that you're calling a C function of three parameters with only two arguments.
Regrettably, PyCFunction_NewEx is missing from the documentation, despite being a public API function. But all of the examples using it pass three arguments. And if you go to the source:
PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)
That's 3.7, but this is the same in 3.5 and in 2.7, and in every other version since the function was added to the API in 2.3. The whole point of NewEx is to allow you to pass a module.
Presumably, the function is expecting that third argument either in a register or on the stack, and you haven't put anything there, so it's completely arbitrary what you're passing. Slightly different code will leave completely different values in those places, so it's not surprising that you get different results:
If the value happens to be 0, that's fine; you're allowed to pass NULL as the module value.
If the value happens to be something that points to unmapped memory, like, say, 1 (as in the raw C long/long long, not a PyLongObject), you should get a segfault from the attempt to incref the module.
If the value happens to be a pointer to some random thing in memory, the incref will work, but will corrupt that random thing. Which could do just about anything, but a mysterious segfault at some arbitrary later point is almost the least surprising thing it could do.
Meanwhile, from a comment:
I am calling PyCFunction_NewEx because PyCFunction_New is a marco in the source code.
If you're using Python 2.3-2.6 or 3.0-3.2, then sure. But in later versions, including the 3.5 you say you're using, CPython goes out of its way to define PyCFunction_New as a function specifically so that it will be present in the API (and even the stable API, for 3.x). See 3.5 for example:
/* undefine macro trampoline to PyCFunction_NewEx */
#undef PyCFunction_New
PyAPI_FUNC(PyObject *)
PyCFunction_New(PyMethodDef *ml, PyObject *self)
{
return PyCFunction_NewEx(ml, self, NULL);
}
So, you really can just call PyCFunction_New.
I have a python extension module written in C++, which contains multiple functions. One of these generates an instance of a custom structure, which I then want to use with other functions of my module in Python as follows
import MyModule
var = MyModule.genFunc()
MyModule.readFunc(var)
To do this, I've tried using PyCapsule objects to pass a pointer to these objects between Python and C, but this produces errors when attempting to read them in the second C function ("PyCapsule_GetPointer called with invalid PyCapsule object"). Python, however, if asked to print the PyCapsule object (var) correctly identifies it as a "'capsule object "testcapsule"'. My C code appears as follows:
struct MyStruct {
int value;
};
static PyObject* genFunc(PyObject* self, PyObject *args) {
MyStruct var;
PyObject *capsuleTest;
var.value = 1;
capsuleTest = PyCapsule_New(&var, "testcapsule", NULL);
return capsuleTest;
}
static PyObject* readFunc(PyObject* self, PyObject *args) {
PyCapsule_GetPointer(args, "testcapsule");
return 0;
}
Thank you for your help.
Like stated in a comment to your question, you'll run into an issue when reading data from the local variable MyStruct var. For this you can use the third destructor to PyCapsule_New.
But that's not the reason for your problem just now. You're using PyCapsule_GetPointer(args, "testcapsule") on the args parameter. And since it's not a capsule, even though var is one, you might have defined the signature of the function as METH_VARARGS. Instead you need to unpack the tuple or use METH_O.
The use case is the following:
Given a (fixed, not changeable) DLL implemented in C
Wanted: a wrapper to this DLL implemented in python (chosen method: ctypes)
Some of the functions in the DLL need synchronization primitives. To aim for maximum flexibility, the designers of the DLL completely rely on client-provided callbacks. More precisely this DLL shall have:
a callback function to create a synchronizaton object
callback functions to acquire/release a lock on the synchronizaton object
and one callback function to destroy the synchronizaton object
Because from the viewpoint of the DLL, the synchronizaton object is opaque, it will be repesented by a void * entity. For example if one of the DLL functions wants to acquire a lock it shall do:
void* mutex;
/* get the mutex object via the create_mutex callback */
create_mutex(&mutex);
/* acquire a lock */
lock_mutex(mutex);
... etc
It can be seen, that the callback create_mutex input parameter has output semantics. This is achieved with void ** signature.
This callback (and the other three) must be implemented in python. I've failed :-) For simplicity, let's focus on only the creating callback, and also for simplicity, let the opaque object be an int.
The toy-DLL, which emulates the use of callbacks, is the following (ct_test.c):
#include <stdio.h>
#include <stdlib.h>
typedef int (* callback_t)(int**);
callback_t func;
int* global_i_p = NULL;
int mock_callback(int** ipp)
{
int* dynamic_int_p = (int *) malloc(sizeof(int));
/* dynamic int value from C */
*dynamic_int_p = 2;
*ipp = dynamic_int_p;
return 0;
}
void set_py_callback(callback_t f)
{
func = f;
}
void set_c_callback()
{
func = mock_callback;
}
void test_callback(void)
{
printf("global_i_p before: %p\n", global_i_p);
func(&global_i_p);
printf("global_i_p after: %p, pointed value:%d\n", global_i_p, *global_i_p);
/* to be nice */
if (func == mock_callback)
free(global_i_p);
}
The python code, which would like to provide the callback, and use the DLL is the following:
from ctypes import *
lib = CDLL("ct_test.so")
# "dynamic" int value from python
int = c_int(1)
int_p = pointer(int)
def pyfunc(p_p_i):
p_p_i.contents = int_p
# create callback type and instance
CALLBACK = CFUNCTYPE(c_int, POINTER (POINTER(c_int)))
c_pyfunc = CALLBACK(pyfunc)
# functions from .so
set_py_callback = lib.set_py_callback
set_c_callback = lib.set_c_callback
test_callback = lib.test_callback
# set one of the callbacks
set_py_callback(c_pyfunc)
#set_c_callback()
# test it
test_callback()
When using the in-DLL provided callback (set via set_c_callback()), this works as expected:
~/dev/test$ python ct_test.py
global_i_p before: (nil)
global_i_p after: 0x97eb008, pointed value:2
However, in the other case - with the python callback - fails:
~/dev/test$ python ct_test.py
global_i_p before: (nil)
Traceback (most recent call last):
File "/home/packages/python/2.5/python2.5-2.5.2/Modules/_ctypes/callbacks.c", line 284, in 'converting callback result'
TypeError: an integer is required
Exception in <function pyfunc at 0xa14079c> ignored
Segmentation fault
Where am I wrong?
You appear to be incorrectly defining the return type. It looks like your C callback returns an int, while the Python one you are declaring as return c_int, yet not explicitly returning anything (thus actually returning None). If you "return 0" it might stop crashing. You should do that or change the callback signature to CFUNCTYPE(None, ...etc) in any case.
Also, although it's not a current problem here, you're shadowing the "int" builtin name. This might lead to problems later.
Edited: to correctly refer to the C return type as "int", not "void".
The segfault is due to incorrect pointer handling in your Python callback. You have more levels of pointer indirection than strict necessary, which is probably the source of your confusion. In the Python callback you set p_p_i.contents, but that only changes what the Python ctypes object points at, not the underlying pointer. To do that, do pointer derefrence via array access syntax. A distilled example:
ip = ctypes.POINTER(ctypes.c_int)()
i = ctypes.c_int(99)
# Wrong way
ipp = ctypes.pointer(ip)
ipp.contents = ctypes.pointer(i)
print bool(ip) # False --> still NULL
# Right way
ipp = ctypes.pointer(ip)
ipp[0] = ctypes.pointer(i)
print ip[0] # 99 --> success!
The type error is due to a type incompatibility as described in Peter Hansen's answer.