Develop Reference

Python C API, send a python function pointer to c and execute it

I want to create a function in python, pass it's function pointer to c and execute it there.
So my python file:
import ctypes
import example
def tester_print():
print("Hello")
my_function_ptr = ctypes.CFUNCTYPE(None)(tester_print)
example.pass_func(my_function_ptr)
And here is what my function in c looks like:
typedef void (*MyFunctionType)(void);
PyObject* pass_func(PyObject *self, PyObject* args)
{
PyObject* callable_object;
if (!PyArg_ParseTuple(args, "O", &callable_object))
return NULL;
if (!PyCallable_Check(callable_object))
{
PyErr_SetString(PyExc_TypeError, "The object is not a callable function.");
return NULL;
}
PyObject* function_pointer = PyCapsule_New(callable_object, "my_function_capsule", NULL);
if (function_pointer == NULL) return NULL;
MyFunctionType my_function = (MyFunctionType) PyCapsule_GetPointer(function_pointer, "my_function_capsule");
if (my_function == NULL) return NULL;
my_function(); // Or (*my_function)() Both same result.
// PyCapsule_Free(function_pointer);
Py_RETURN_NONE;
}
Doing this causes a seg fault on my_function() call. How can I do this?

If you're just trying to pass a Python function to a C extension, pass it directly (don't use ctypes) and use PyObject_Call to call it:
example.pass_func(tester_print)
and
PyObject_CallNoArgs(callable_object);
If you need a real C function pointer for whatever reason, the usual approach is to write a C wrapper that takes the callable as an argument:
void callable_wrapper(PyObject *func) {
PyObject_CallNoArgs(func);
// plus whatever other code you need (e.g. reference counting, return value handling)
}
Most reasonable C APIs that take a callback function also provide a way to add an arbitrary argument to the callable ("user data"); for example, with pthreads:
result = pthread_create(&tid, &attr, callable_wrapper, callable_object);
Make sure to handle reference counting correctly: increment the reference on your callable object before passing it to the C API, and decrement the reference when it is no longer needed (e.g. if the callback is only called once, the callable_wrapper could DECREF before returning).
When using threads, you additionally need to ensure that you hold the GIL when calling any Python code; see https://docs.python.org/3/c-api/init.html#non-python-created-threads for more details and a code sample.
What your current code is doing is receiving a pointer to a ctypes CFUNCTYPE object as callable_object, placing that pointer in a capsule, taking it back out again, and calling it as if it was a C function pointer. This doesn't work, since it effectively attempts to call the CFUNCTYPE object as if it were a C function (the capsule stuff winds up being useless). When you're using the Python C API, there's almost never any need for ctypes in Python, because the C API can directly interact with Python objects.

Is incrementing Py_True/Py_False refcount always necessary?

I'm new to the Python C-API and browsing through some source code to pick parts of it up.
Here is a minimal version of a function that I found, in the C source of a package that contains extension modules:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
static PyObject *
modulename_myfunc(PyObject *self, PyObject *args) {
// Call PyArg_ParseTuple, etc ...
// Dummy values; in the real function they are calculated
int is_found = 1;
Py_ssize_t n_bytes_found = 1024;
PyObject *result;
result = Py_BuildValue("(Oi)",
is_found ? Py_True : Py_False, // Py_INCREF?
n_bytes_found);
return result;
}
Does this introduce a small memory leak by failing to use Py_INCREF on either Py_True or Py_False? The C-API docs for Boolean object seem pretty explicit about always needing to incref/decref Py_True and Py_False.
If a Py_INCREF does need to be introduced, how can it most properly be used here, assuming that Py_RETURN_TRUE/Py_RETURN_FALSE aren't really applicable because a tuple is being returned?

The reason a Py_INCREF is not used here is because Py_BuildValue, when being passed an object with "O" will increment the reference count for you:
O (object) [PyObject *]
Pass a Python object untouched (except for its reference count, which is incremented by one). If the object passed in is a NULL pointer, it is assumed that this was caused because the call producing the argument found an error and set an exception. Therefore, Py_BuildValue() will return NULL but won’t raise an exception. If no exception has been raised yet, SystemError is set.
You'll see a similar usage here in CPython itself for example.

Error in writing Python functions in Crystal-lang

I am trying to write some python function in crystal-lang through the C Python API.
My code follows:
METH_VARARGS = 0x0001
#[Link("python3.5m")]
lib Python
alias PyObject = Void*
struct PyMethodDef
name : UInt8*
func : Void*
flags : LibC::Int
doc : UInt8*
end
fun Py_Initialize
fun Py_Finalize
fun PyObject_CallObject(func : PyObject, args : PyObject) : PyObject
fun PyCFunction_NewEx(method : PyMethodDef*, __self__ : PyObject, ) : PyObject
fun PyLong_AsLong(n : PyObject) : Int64
fun PyLong_FromLong(n : Int64) : PyObject
end
def new_method_def(name : String, function, flags : LibC::Int)
x = Pointer(Python::PyMethodDef).malloc(1)
x.value.name = name.to_unsafe
x.value.func = function
x.value.flags = flags
x.value.doc = nil
x
end
Python.Py_Initialize
a = ->(args : Void*) {
puts Python.PyLong_AsLong(args)
Pointer(Void).null
}
name = "num"
number = Python.PyLong_FromLong(1)
Python.Py_IncRef(number)
method = Python.PyCFunction_NewEx(new_method_def(name,a.pointer,METH_VARARGS),number)
Python.PyObject_CallObject(method,Pointer(Void).null)
Python.Py_Finalize
Everything works if I set nil instead of number when in PyCFunction_NewEx, but as the code is, it throws an invalid acces memory exception when Py_Finalize is called.
I can't understand what's causing it.
Can someone help me?

The root problem here is that you're calling a C function of three parameters with only two arguments.
Regrettably, PyCFunction_NewEx is missing from the documentation, despite being a public API function. But all of the examples using it pass three arguments. And if you go to the source:
PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)
That's 3.7, but this is the same in 3.5 and in 2.7, and in every other version since the function was added to the API in 2.3. The whole point of NewEx is to allow you to pass a module.
Presumably, the function is expecting that third argument either in a register or on the stack, and you haven't put anything there, so it's completely arbitrary what you're passing. Slightly different code will leave completely different values in those places, so it's not surprising that you get different results:
If the value happens to be 0, that's fine; you're allowed to pass NULL as the module value.
If the value happens to be something that points to unmapped memory, like, say, 1 (as in the raw C long/long long, not a PyLongObject), you should get a segfault from the attempt to incref the module.
If the value happens to be a pointer to some random thing in memory, the incref will work, but will corrupt that random thing. Which could do just about anything, but a mysterious segfault at some arbitrary later point is almost the least surprising thing it could do.
Meanwhile, from a comment:
I am calling PyCFunction_NewEx because PyCFunction_New is a marco in the source code.
If you're using Python 2.3-2.6 or 3.0-3.2, then sure. But in later versions, including the 3.5 you say you're using, CPython goes out of its way to define PyCFunction_New as a function specifically so that it will be present in the API (and even the stable API, for 3.x). See 3.5 for example:
/* undefine macro trampoline to PyCFunction_NewEx */
#undef PyCFunction_New
PyAPI_FUNC(PyObject *)
PyCFunction_New(PyMethodDef *ml, PyObject *self)
{
return PyCFunction_NewEx(ml, self, NULL);
}
So, you really can just call PyCFunction_New.

Parsing User Defined Types Using PyArg_ParseTuple

How to parse userdefined types (or types from an existing non-standard library) using PyArg_ParseTuple?

Instead of using the plain O format, as Martijn suggested, I normally prefer using the O& format. It allows you to pass a function that will be called to convert any PyObject* to an arbitrary C (double) pointer. Here is some example usage, in which I'm converting a passed value to a pointer to my own object type:
/**
* This method should return 0 if it does not work or 1 in case it does
* PyArg_*() functions will handle the rest from there or let your program
* continue in case things are fine.
*/
int MyConverter(PyObject* o, MyOwnType** mine) {
//write the converter here.
}
Then, at the point you need to parse your object:
/**
* Simple example
*/
static PyObject* py_do_something(PyObject*, PyObject* args, PyObject* kwds) {
/* Parses input arguments in a single shot */
static char* kwlist[] = {"o", 0};
MyOwnType* obj = 0; ///< if things go OK, obj will be there after the next call
if (!PyArg_ParseTupleAndKeywords(args, kwds, "O&", kwlist, &MyConverter, &obj))
return 0; ///< we have failed, let Python check exceptions.
/* if you get to this point, your converter worked, just use */
/* your newly allocated object and free it, when done. */
}
The advantage of this approach is that you can encapsulate your MyConverter on a C-API and then re-use it in other functions for the the same job.

Custom python classes can be parsed using the O format:
O (object) [PyObject *]
Store a Python object (without any conversion) in a C object pointer. The C program thus receives the actual object that was passed. The object’s reference count is not increased. The pointer stored is not NULL.

Python C Extension - Why are methods that use keyword arguments cast to PyCFunction

I've am learning about Python-C extensions and am puzzled as to why methods that use keyword arguments must be cast to PyCFunctions.
My understanding of a PyCFunction is that it takes two pointers to PyObjects and returns a single pointer to a PyObject - e.g.
PyObject* myFunc(PyObject* self, PyObject* args)
If I'm going to use a function that uses keyword arguments, then this function will take three pointers to PyObjects and returns a single pointer to a PyObject - e.g.
PyObject* myFunc(PyObject* self, PyObject* args, PyObject* keywordArgs)
However, when I create the module functions array (for a function called 'adder'):
{ "adder", (PyCFunction)adder, METH_VARARGS | METH_KEYWORDS, "adder method" }
works fine. It feels like I cast a float to an int and still got to use the non-integer parts of the float. If I didn't see this work, I would have thought it wouldn't work. What am I not understanding here?
Also, I saw some references to a PyCFunctionWithKeywords, which seems to have the function signature I thought I needed, but my compiler complained (gave warnings) about 'incompatible pointer types'.
Was PyCFunctionWithKeywords deprecated? If not, is there a time when I should/must use it?

If your function handles keyword arguments, then it must correspond to a PyCFunctionWithKeywords. However, C doesn’t do overloading, and the structure built by PyMethodDef is defined to expect a PyCFunction, rather than, say, a completely unchecked void *. So you must cast your PyCFunctionWithKeywords to a PyCFunction to stop the compiler complaining, that’s all.
Remember that you must also pass METH_KEYWORDS in the flags to tell Python that your function has the signature of a PyCFunctionWithKeywords, not a PyCFunction.

D'Olveiro is 100% correct, but (depending on your compiler) you may still get a "cast-function-type" warning, which you can safely ignore. Or (if using a gcc varient) surround with pragma to temporarily turn off that warning:
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wcast-function-type"
static PyMethodDef MyPythonMethods[] = {
{"myMethod", (PyCFunction)MyMethodFunction, METH_VARARGS, "doc string"},
...
{NULL, NULL, 0, NULL}
};
#pragma GCC diagnostic pop