We're working on some Python/C-API code, and we've encountered a method that would like to be passed a callback. The method will ship periodic updates to the callback as a form a feedback. As it turns out, we're not that interested in periodic feedback. The only way to disable the method's default feedback mechanism is to pass it some kind of callback.
The technique we've employed is to declare a module level function that just returns None, ie:
static PyObject*
donothing(PyObject* self, PyObject* args) {
return Py_None;
}
But of course, this function also needs to be registered with the modules method table, ie:
static PyMethodDef methods[] = {
{"donothing", donothing, METH_VARARGS, "do nothing"},
...
{NULL}
};
Then, when we go to call the method, we need to grab a reference to this method, ie: PyObject_GetAttrString(module_reference, "donothing").
All of this feels like we're spending too much time spinning our wheels just to do nothing. Then it ocurred to me..hey, seems like a perfect use for lambda x: None. But after spending an hour with the Python/C-API docs, I can't figure out how one create's lambdas.
I see there are references to closures on the page http://docs.python.org/2/c-api/function.html, but I can't sort out the details on how one creates them.
Any pointers (or references to RTFM) would be greatly appreciated.
A lambda expressions is used to create simple anonymous functions. These have a PyFunction_Type wrapping an object of PyCode_Type, which is a chunk of executable code. But you're already on the C side, so creating a Python function would be a little too much. Instead you should create an object of PyCFunction_Type. This is similar to what you've tried to do with the module methods.
The boilerplate in C wouldn't be too big either, but only a few lines:
static PyObject *
donothing(PyObject *self, PyObject *args) {
Py_RETURN_NONE;
}
static PyMethodDef donothing_ml = {"donothing", donothing, METH_VARARGS, "doc"};
The object then is created with PyCFunction_New(&donothing_ml, NULL) which yields a <built-in function donothing>. This function is independent of your module and can be used like any other PyObject.
It's not exactly a high level lambda, but rather a low level implementation of lambda *args: None.
However if you'd really like to create a high level lambda you can do this with a single statement like dastrobu proposed
l = PyRun_String("lambda *args: None", Py_eval_input, PyEval_GetGlobals(), NULL);
or if you'd like to assemble it yourself you could do
PyCodeObject *c = (PyCodeObject *) Py_CompileString("None", "fn", Py_eval_input);
#if PY_MAJOR_VERSION >= 3
c->co_name = PyUnicode_FromString("<c-lambda>"); // function name
#else
c->co_name = PyString_FromString("<c-lambda>"); // function name
#endif
c->co_flags |= CO_VARARGS; // accept *args
c->co_nlocals = 1; // needed in Python 3
l = PyFunction_New((PyObject *) c, PyEval_GetGlobals());
In both cases you'll get a function with dissasembled code dis(l) equivalent to a lambda:
1 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
Related
I want to create a function in python, pass it's function pointer to c and execute it there.
So my python file:
import ctypes
import example
def tester_print():
print("Hello")
my_function_ptr = ctypes.CFUNCTYPE(None)(tester_print)
example.pass_func(my_function_ptr)
And here is what my function in c looks like:
typedef void (*MyFunctionType)(void);
PyObject* pass_func(PyObject *self, PyObject* args)
{
PyObject* callable_object;
if (!PyArg_ParseTuple(args, "O", &callable_object))
return NULL;
if (!PyCallable_Check(callable_object))
{
PyErr_SetString(PyExc_TypeError, "The object is not a callable function.");
return NULL;
}
PyObject* function_pointer = PyCapsule_New(callable_object, "my_function_capsule", NULL);
if (function_pointer == NULL) return NULL;
MyFunctionType my_function = (MyFunctionType) PyCapsule_GetPointer(function_pointer, "my_function_capsule");
if (my_function == NULL) return NULL;
my_function(); // Or (*my_function)() Both same result.
// PyCapsule_Free(function_pointer);
Py_RETURN_NONE;
}
Doing this causes a seg fault on my_function() call. How can I do this?
If you're just trying to pass a Python function to a C extension, pass it directly (don't use ctypes) and use PyObject_Call to call it:
example.pass_func(tester_print)
and
PyObject_CallNoArgs(callable_object);
If you need a real C function pointer for whatever reason, the usual approach is to write a C wrapper that takes the callable as an argument:
void callable_wrapper(PyObject *func) {
PyObject_CallNoArgs(func);
// plus whatever other code you need (e.g. reference counting, return value handling)
}
Most reasonable C APIs that take a callback function also provide a way to add an arbitrary argument to the callable ("user data"); for example, with pthreads:
result = pthread_create(&tid, &attr, callable_wrapper, callable_object);
Make sure to handle reference counting correctly: increment the reference on your callable object before passing it to the C API, and decrement the reference when it is no longer needed (e.g. if the callback is only called once, the callable_wrapper could DECREF before returning).
When using threads, you additionally need to ensure that you hold the GIL when calling any Python code; see https://docs.python.org/3/c-api/init.html#non-python-created-threads for more details and a code sample.
What your current code is doing is receiving a pointer to a ctypes CFUNCTYPE object as callable_object, placing that pointer in a capsule, taking it back out again, and calling it as if it was a C function pointer. This doesn't work, since it effectively attempts to call the CFUNCTYPE object as if it were a C function (the capsule stuff winds up being useless). When you're using the Python C API, there's almost never any need for ctypes in Python, because the C API can directly interact with Python objects.
in python I can write:
def test(a, b=None):
if b is None:
return
else:
print(123)
in cpp, it's better to avoid pointers, so I use reference instead,
so how to do the same thing?
#include "stdio.h"
void test(int a, const int &b) {
// how to check ?? since b should not be nullptr
printf("123\n"); };
int main() { test(); }
in cpp, it's better to avoid pointers, so I use reference instead
References can't refer to NULL, so pointers are the traditional way to do this, e.g. void test(int a, const int *b=NULL). Much of the reason references are encouraged over pointers is because it saves you from handling NULL arguments; if you need NULL arguments, references aren't saving you from anything.
std::optional is sometimes used for similar scenarios, but it's much newer (C++17), and I don't think there is strong consensus on whether it's preferable; that said, using it with std::nullopt as the default is close to what you've got already, and a reasonable way to handle the problem.
The alternative (mentioned in the answers to the above linked question) is just to overload the function twice, once with the argument, and once without; this could be combined with the std::optional approach to allow simpler calling for users only passing one argument (with a default, the generated code inlines the creation of the default argument at each call site that relies on it), but still implementing the function via common code (the single argument function just turning around and calling the two-argument function).
C++ references cannot be NULL.They always point to valid object,initialised during their declaration.
As mentioned in the other answers, NULL references aren't allowed in C++, so you can't use NULL as a default value for a by-reference argument, and std::optional would be a good choice instead.
You can define your own sentinel-object though, to perform the same function as NULL without actually being a NULL-reference, like this:
#include "stdio.h"
const int & get_sentinel_ref()
{
static int sentinel = 0; // must be declared static
return sentinel; // in order to have a fixed address
}
void test(int a, const int &b = get_sentinel_ref())
{
// Check if b is referring to our sentinel-value or not
// Note that I'm comparing memory-addresses here, not values
// otherwise the code would do the wrong thing if the user
// passed in zero (or whatever dummy-value sentinel is set
// to in the get_sentinel_ref() function above)
if (&b == &get_sentinel_ref())
{
printf("a is %i, but you didn't supply a second argument!\n", a);
}
else
{
printf("a is %i, b is %i\n", a, b);
}
}
int main(int, char **)
{
test(5);
test(6,7);
return 0;
}
... When run, the above program prints:
a is 5, but you didn't supply a second argument!
a is 6, b is 7
I am trying to write some python function in crystal-lang through the C Python API.
My code follows:
METH_VARARGS = 0x0001
#[Link("python3.5m")]
lib Python
alias PyObject = Void*
struct PyMethodDef
name : UInt8*
func : Void*
flags : LibC::Int
doc : UInt8*
end
fun Py_Initialize
fun Py_Finalize
fun PyObject_CallObject(func : PyObject, args : PyObject) : PyObject
fun PyCFunction_NewEx(method : PyMethodDef*, __self__ : PyObject, ) : PyObject
fun PyLong_AsLong(n : PyObject) : Int64
fun PyLong_FromLong(n : Int64) : PyObject
end
def new_method_def(name : String, function, flags : LibC::Int)
x = Pointer(Python::PyMethodDef).malloc(1)
x.value.name = name.to_unsafe
x.value.func = function
x.value.flags = flags
x.value.doc = nil
x
end
Python.Py_Initialize
a = ->(args : Void*) {
puts Python.PyLong_AsLong(args)
Pointer(Void).null
}
name = "num"
number = Python.PyLong_FromLong(1)
Python.Py_IncRef(number)
method = Python.PyCFunction_NewEx(new_method_def(name,a.pointer,METH_VARARGS),number)
Python.PyObject_CallObject(method,Pointer(Void).null)
Python.Py_Finalize
Everything works if I set nil instead of number when in PyCFunction_NewEx, but as the code is, it throws an invalid acces memory exception when Py_Finalize is called.
I can't understand what's causing it.
Can someone help me?
The root problem here is that you're calling a C function of three parameters with only two arguments.
Regrettably, PyCFunction_NewEx is missing from the documentation, despite being a public API function. But all of the examples using it pass three arguments. And if you go to the source:
PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)
That's 3.7, but this is the same in 3.5 and in 2.7, and in every other version since the function was added to the API in 2.3. The whole point of NewEx is to allow you to pass a module.
Presumably, the function is expecting that third argument either in a register or on the stack, and you haven't put anything there, so it's completely arbitrary what you're passing. Slightly different code will leave completely different values in those places, so it's not surprising that you get different results:
If the value happens to be 0, that's fine; you're allowed to pass NULL as the module value.
If the value happens to be something that points to unmapped memory, like, say, 1 (as in the raw C long/long long, not a PyLongObject), you should get a segfault from the attempt to incref the module.
If the value happens to be a pointer to some random thing in memory, the incref will work, but will corrupt that random thing. Which could do just about anything, but a mysterious segfault at some arbitrary later point is almost the least surprising thing it could do.
Meanwhile, from a comment:
I am calling PyCFunction_NewEx because PyCFunction_New is a marco in the source code.
If you're using Python 2.3-2.6 or 3.0-3.2, then sure. But in later versions, including the 3.5 you say you're using, CPython goes out of its way to define PyCFunction_New as a function specifically so that it will be present in the API (and even the stable API, for 3.x). See 3.5 for example:
/* undefine macro trampoline to PyCFunction_NewEx */
#undef PyCFunction_New
PyAPI_FUNC(PyObject *)
PyCFunction_New(PyMethodDef *ml, PyObject *self)
{
return PyCFunction_NewEx(ml, self, NULL);
}
So, you really can just call PyCFunction_New.
I have a python extension module written in C++, which contains multiple functions. One of these generates an instance of a custom structure, which I then want to use with other functions of my module in Python as follows
import MyModule
var = MyModule.genFunc()
MyModule.readFunc(var)
To do this, I've tried using PyCapsule objects to pass a pointer to these objects between Python and C, but this produces errors when attempting to read them in the second C function ("PyCapsule_GetPointer called with invalid PyCapsule object"). Python, however, if asked to print the PyCapsule object (var) correctly identifies it as a "'capsule object "testcapsule"'. My C code appears as follows:
struct MyStruct {
int value;
};
static PyObject* genFunc(PyObject* self, PyObject *args) {
MyStruct var;
PyObject *capsuleTest;
var.value = 1;
capsuleTest = PyCapsule_New(&var, "testcapsule", NULL);
return capsuleTest;
}
static PyObject* readFunc(PyObject* self, PyObject *args) {
PyCapsule_GetPointer(args, "testcapsule");
return 0;
}
Thank you for your help.
Like stated in a comment to your question, you'll run into an issue when reading data from the local variable MyStruct var. For this you can use the third destructor to PyCapsule_New.
But that's not the reason for your problem just now. You're using PyCapsule_GetPointer(args, "testcapsule") on the args parameter. And since it's not a capsule, even though var is one, you might have defined the signature of the function as METH_VARARGS. Instead you need to unpack the tuple or use METH_O.
Sorry if this is too vague. I was recently reading about python's list.sort() method and read that it was written in C for performance reasons.
I'm assuming that the python code just passes a list to the C code and the C code passes a list back, but how does the python code know where to pass it or that C gave it the correct data type, and how does the C code know what data type it was given?
Python can be extended in C/C++ (more info here)
It basically means that you can wrap a C module like this
#include "Python.h"
// Static function returning a PyObject pointer
static PyObject *
keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
// takes self, args and kwargs.
{
int voltage;
// No such thing as strings here. Its a tough life.
char *state = "a stiff";
char *action = "voom";
char *type = "Norwegian Blue";
// Possible keywords
static char *kwlist[] = {"voltage", "state", "action", "type", NULL};
// unpack arguments
if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
&voltage, &state, &action, &type))
return NULL;
// print to stdout
printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
action, voltage);
printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);
// Reference count some None.
Py_INCREF(Py_None);
// return some none.
return Py_None;
}
// Static PyMethodDef
static PyMethodDef keywdarg_methods[] = {
/* The cast of the function is necessary since PyCFunction values
* only take two PyObject* parameters, and keywdarg_parrot() takes
* three.
*/
// Declare the parrot function, say what it takes and give it a doc string.
{"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
"Print a lovely skit to standard output."},
{NULL, NULL, 0, NULL} /* sentinel */
};
And using the Python header files it will define and understand entry points and return locations in the C/C++ code.
I can't speak to Python/C interaction directly, but I can give some background to how these sorts of things work in general.
On a particular platform or implementation, there is a calling convention that specifies how parameters are passed to subroutines and how values are returned to the caller. Compilers and interpreters that target that platform or implementation generate code to conform to that convention, so that subroutines/modules/whatever written in different languages can communicate with each other.
In my assembly class, we had an assignment where we had to write a program using VAX assembler, C, and Pascal (this was in the mid-Cretaceous1980s). The driver was in one of C or Pascal (can't remember which anymore), which called the assembly routine, which called the other routine (which was written in whichever language the driver wasn't). Our assembly code had to pop and push parameters from the stack based on the VMS calling convention.
Each computing platform has (or should have) an application binary interface (ABI). This is a specification of how parameters are passed between routines, how values are returned, what state the machine should be in and so on.
The ABI will specify things such as (for example):
The first integer argument (up to some number of bits, say 32) will be passed in a certain register (such as %EAX or R3). The second will be passed in another specific, register, and so on.
After the list of register is used, additional integer arguments will be passed on the stack, starting at a certain offset from the value of the stack pointer when the call is made.
Pointer arguments will be treated the same as integer arguments.
Floating-point arguments will be passed in floating-point registers F1, F2, and so on, until those registers are used up, and then on the stack.
Compound arguments (such as structures) will be passed as integer arguments if they are very small (e.g., four char objects in one structure) or on the stack if they are large.
Each compiler or other language implementation will generate code that conforms to the ABI, at least where its routines call or are called from other routines that might be outside the language.