Sorry if this is too vague. I was recently reading about python's list.sort() method and read that it was written in C for performance reasons.
I'm assuming that the python code just passes a list to the C code and the C code passes a list back, but how does the python code know where to pass it or that C gave it the correct data type, and how does the C code know what data type it was given?
Python can be extended in C/C++ (more info here)
It basically means that you can wrap a C module like this
#include "Python.h"
// Static function returning a PyObject pointer
static PyObject *
keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
// takes self, args and kwargs.
{
int voltage;
// No such thing as strings here. Its a tough life.
char *state = "a stiff";
char *action = "voom";
char *type = "Norwegian Blue";
// Possible keywords
static char *kwlist[] = {"voltage", "state", "action", "type", NULL};
// unpack arguments
if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
&voltage, &state, &action, &type))
return NULL;
// print to stdout
printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
action, voltage);
printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);
// Reference count some None.
Py_INCREF(Py_None);
// return some none.
return Py_None;
}
// Static PyMethodDef
static PyMethodDef keywdarg_methods[] = {
/* The cast of the function is necessary since PyCFunction values
* only take two PyObject* parameters, and keywdarg_parrot() takes
* three.
*/
// Declare the parrot function, say what it takes and give it a doc string.
{"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
"Print a lovely skit to standard output."},
{NULL, NULL, 0, NULL} /* sentinel */
};
And using the Python header files it will define and understand entry points and return locations in the C/C++ code.
I can't speak to Python/C interaction directly, but I can give some background to how these sorts of things work in general.
On a particular platform or implementation, there is a calling convention that specifies how parameters are passed to subroutines and how values are returned to the caller. Compilers and interpreters that target that platform or implementation generate code to conform to that convention, so that subroutines/modules/whatever written in different languages can communicate with each other.
In my assembly class, we had an assignment where we had to write a program using VAX assembler, C, and Pascal (this was in the mid-Cretaceous1980s). The driver was in one of C or Pascal (can't remember which anymore), which called the assembly routine, which called the other routine (which was written in whichever language the driver wasn't). Our assembly code had to pop and push parameters from the stack based on the VMS calling convention.
Each computing platform has (or should have) an application binary interface (ABI). This is a specification of how parameters are passed between routines, how values are returned, what state the machine should be in and so on.
The ABI will specify things such as (for example):
The first integer argument (up to some number of bits, say 32) will be passed in a certain register (such as %EAX or R3). The second will be passed in another specific, register, and so on.
After the list of register is used, additional integer arguments will be passed on the stack, starting at a certain offset from the value of the stack pointer when the call is made.
Pointer arguments will be treated the same as integer arguments.
Floating-point arguments will be passed in floating-point registers F1, F2, and so on, until those registers are used up, and then on the stack.
Compound arguments (such as structures) will be passed as integer arguments if they are very small (e.g., four char objects in one structure) or on the stack if they are large.
Each compiler or other language implementation will generate code that conforms to the ABI, at least where its routines call or are called from other routines that might be outside the language.
Related
In some documents, I learned that we shall use 'restrict' to decorate function parameters or memory allocation statements. Like this:
void(int*restrict paraA){}
int*restrict A = (int*)malloc(10 * sizeof(int);
And I'm confused about whether a 'restricted' pointer can be returned by any function.
Here I have a function funcA:
int* funcA(paraA, paraB){
int*restrict result = (int*)calloc(10,sizeof(int));
......
return result;
}
Could I just use the return pointer like this? Will it be safe?
void funcB(){
int*restrict pA = funcA(0, 0);
pA[0] = 1;
}
In msvc, I learn about __declspec(restrict).
Should I use __declspec(restrict) int* funcA() or int*restrict funcA() instead in gcc?
And if I want to use funcA in Python, will __declspec(dllexport,restrict) int* funcA() effect the result in Python compared with declspec(dllexport) int* funcA()?
This usage of restrict doesn't make much sense. The keyword only serves a purpose when you have two or more pointers to compatible types and the compiler can't know if they potentially point at the same object. Or if they possibly point to an object with external linkage ("global") in the same translation unit. More details here.
In funcA the pointer has no relation to any other pointer or object in the program, since it is assigned to dynamic memory inside the function. restrict fills no obvious purpose at all in that function.
I am trying to write some python function in crystal-lang through the C Python API.
My code follows:
METH_VARARGS = 0x0001
#[Link("python3.5m")]
lib Python
alias PyObject = Void*
struct PyMethodDef
name : UInt8*
func : Void*
flags : LibC::Int
doc : UInt8*
end
fun Py_Initialize
fun Py_Finalize
fun PyObject_CallObject(func : PyObject, args : PyObject) : PyObject
fun PyCFunction_NewEx(method : PyMethodDef*, __self__ : PyObject, ) : PyObject
fun PyLong_AsLong(n : PyObject) : Int64
fun PyLong_FromLong(n : Int64) : PyObject
end
def new_method_def(name : String, function, flags : LibC::Int)
x = Pointer(Python::PyMethodDef).malloc(1)
x.value.name = name.to_unsafe
x.value.func = function
x.value.flags = flags
x.value.doc = nil
x
end
Python.Py_Initialize
a = ->(args : Void*) {
puts Python.PyLong_AsLong(args)
Pointer(Void).null
}
name = "num"
number = Python.PyLong_FromLong(1)
Python.Py_IncRef(number)
method = Python.PyCFunction_NewEx(new_method_def(name,a.pointer,METH_VARARGS),number)
Python.PyObject_CallObject(method,Pointer(Void).null)
Python.Py_Finalize
Everything works if I set nil instead of number when in PyCFunction_NewEx, but as the code is, it throws an invalid acces memory exception when Py_Finalize is called.
I can't understand what's causing it.
Can someone help me?
The root problem here is that you're calling a C function of three parameters with only two arguments.
Regrettably, PyCFunction_NewEx is missing from the documentation, despite being a public API function. But all of the examples using it pass three arguments. And if you go to the source:
PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)
That's 3.7, but this is the same in 3.5 and in 2.7, and in every other version since the function was added to the API in 2.3. The whole point of NewEx is to allow you to pass a module.
Presumably, the function is expecting that third argument either in a register or on the stack, and you haven't put anything there, so it's completely arbitrary what you're passing. Slightly different code will leave completely different values in those places, so it's not surprising that you get different results:
If the value happens to be 0, that's fine; you're allowed to pass NULL as the module value.
If the value happens to be something that points to unmapped memory, like, say, 1 (as in the raw C long/long long, not a PyLongObject), you should get a segfault from the attempt to incref the module.
If the value happens to be a pointer to some random thing in memory, the incref will work, but will corrupt that random thing. Which could do just about anything, but a mysterious segfault at some arbitrary later point is almost the least surprising thing it could do.
Meanwhile, from a comment:
I am calling PyCFunction_NewEx because PyCFunction_New is a marco in the source code.
If you're using Python 2.3-2.6 or 3.0-3.2, then sure. But in later versions, including the 3.5 you say you're using, CPython goes out of its way to define PyCFunction_New as a function specifically so that it will be present in the API (and even the stable API, for 3.x). See 3.5 for example:
/* undefine macro trampoline to PyCFunction_NewEx */
#undef PyCFunction_New
PyAPI_FUNC(PyObject *)
PyCFunction_New(PyMethodDef *ml, PyObject *self)
{
return PyCFunction_NewEx(ml, self, NULL);
}
So, you really can just call PyCFunction_New.
We're working on some Python/C-API code, and we've encountered a method that would like to be passed a callback. The method will ship periodic updates to the callback as a form a feedback. As it turns out, we're not that interested in periodic feedback. The only way to disable the method's default feedback mechanism is to pass it some kind of callback.
The technique we've employed is to declare a module level function that just returns None, ie:
static PyObject*
donothing(PyObject* self, PyObject* args) {
return Py_None;
}
But of course, this function also needs to be registered with the modules method table, ie:
static PyMethodDef methods[] = {
{"donothing", donothing, METH_VARARGS, "do nothing"},
...
{NULL}
};
Then, when we go to call the method, we need to grab a reference to this method, ie: PyObject_GetAttrString(module_reference, "donothing").
All of this feels like we're spending too much time spinning our wheels just to do nothing. Then it ocurred to me..hey, seems like a perfect use for lambda x: None. But after spending an hour with the Python/C-API docs, I can't figure out how one create's lambdas.
I see there are references to closures on the page http://docs.python.org/2/c-api/function.html, but I can't sort out the details on how one creates them.
Any pointers (or references to RTFM) would be greatly appreciated.
A lambda expressions is used to create simple anonymous functions. These have a PyFunction_Type wrapping an object of PyCode_Type, which is a chunk of executable code. But you're already on the C side, so creating a Python function would be a little too much. Instead you should create an object of PyCFunction_Type. This is similar to what you've tried to do with the module methods.
The boilerplate in C wouldn't be too big either, but only a few lines:
static PyObject *
donothing(PyObject *self, PyObject *args) {
Py_RETURN_NONE;
}
static PyMethodDef donothing_ml = {"donothing", donothing, METH_VARARGS, "doc"};
The object then is created with PyCFunction_New(&donothing_ml, NULL) which yields a <built-in function donothing>. This function is independent of your module and can be used like any other PyObject.
It's not exactly a high level lambda, but rather a low level implementation of lambda *args: None.
However if you'd really like to create a high level lambda you can do this with a single statement like dastrobu proposed
l = PyRun_String("lambda *args: None", Py_eval_input, PyEval_GetGlobals(), NULL);
or if you'd like to assemble it yourself you could do
PyCodeObject *c = (PyCodeObject *) Py_CompileString("None", "fn", Py_eval_input);
#if PY_MAJOR_VERSION >= 3
c->co_name = PyUnicode_FromString("<c-lambda>"); // function name
#else
c->co_name = PyString_FromString("<c-lambda>"); // function name
#endif
c->co_flags |= CO_VARARGS; // accept *args
c->co_nlocals = 1; // needed in Python 3
l = PyFunction_New((PyObject *) c, PyEval_GetGlobals());
In both cases you'll get a function with dissasembled code dis(l) equivalent to a lambda:
1 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
I've am learning about Python-C extensions and am puzzled as to why methods that use keyword arguments must be cast to PyCFunctions.
My understanding of a PyCFunction is that it takes two pointers to PyObjects and returns a single pointer to a PyObject - e.g.
PyObject* myFunc(PyObject* self, PyObject* args)
If I'm going to use a function that uses keyword arguments, then this function will take three pointers to PyObjects and returns a single pointer to a PyObject - e.g.
PyObject* myFunc(PyObject* self, PyObject* args, PyObject* keywordArgs)
However, when I create the module functions array (for a function called 'adder'):
{ "adder", (PyCFunction)adder, METH_VARARGS | METH_KEYWORDS, "adder method" }
works fine. It feels like I cast a float to an int and still got to use the non-integer parts of the float. If I didn't see this work, I would have thought it wouldn't work. What am I not understanding here?
Also, I saw some references to a PyCFunctionWithKeywords, which seems to have the function signature I thought I needed, but my compiler complained (gave warnings) about 'incompatible pointer types'.
Was PyCFunctionWithKeywords deprecated? If not, is there a time when I should/must use it?
If your function handles keyword arguments, then it must correspond to a PyCFunctionWithKeywords. However, C doesn’t do overloading, and the structure built by PyMethodDef is defined to expect a PyCFunction, rather than, say, a completely unchecked void *. So you must cast your PyCFunctionWithKeywords to a PyCFunction to stop the compiler complaining, that’s all.
Remember that you must also pass METH_KEYWORDS in the flags to tell Python that your function has the signature of a PyCFunctionWithKeywords, not a PyCFunction.
D'Olveiro is 100% correct, but (depending on your compiler) you may still get a "cast-function-type" warning, which you can safely ignore. Or (if using a gcc varient) surround with pragma to temporarily turn off that warning:
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wcast-function-type"
static PyMethodDef MyPythonMethods[] = {
{"myMethod", (PyCFunction)MyMethodFunction, METH_VARARGS, "doc string"},
...
{NULL, NULL, 0, NULL}
};
#pragma GCC diagnostic pop
In regex or Script (e.g. one written in python) how can I add
printf("TRACING: %s is called\n", __PRETTY_FUNCTION__);
at entry of all the function definitions, e.g.,
INT4
FunctionNameCouldBeAny (UINT4 ui)
{
// insert here, at entry
printf("TRACING: %s is called\n", __PRETTY_FUNCTION__);
}
for one file xxx.c?
for all c files *.c under directory /work_space/test/src?
Please note, functions defined in the same file may share the same prefix, but not always.
COMMENT 1: -finstrument-functions does not work for my gcc; or else I have to provide __cyg_profile_func_enter()/exit() functions, and find a way to print out the name from binary address. I wonder if there is more efficient way from regex.
Doing the job thoroughly is going to be hard. The problem is recognizing when a function is defined; there can be many possible layouts, and it is hard to recognize all possibilities. For example:
int x() { return 1; } int y(int z) { return z + 13; }
Most ad hoc systems won't detect y, even if they detect and handle x. But that's lousy code layout, and you probably don't indulge in such code layouts.
How do you start your functions?
static void function(int arg1) {
static void
function(int arg1) {
static void
function(int arg1)
{
static void
function(
int arg1
)
{
static void
function
(
int arg1
)
{
Etc. Depending on the notation(s) used, you need to write multiple different regular expressions. Note that if you can't apply heuristics such as '{ at start of line marks start of function - or structure/union definition, or data initialization' (because you use a { at the end of the line containing the function), it gets rather tricky.
You may need to tokenize the input, and keep track of whether you're inside a function definition. Although my example used keywords, you can have functions using user-defined types only:
Xyz *pqr(Abc def)
{
Then, of course, you might have old code written without prototypes:
Xyz *pqr(def)
Abc def;
{
All this is before you get involved in preprocessor stuff, which can really confuse things:
#define BEGIN {
#define END }
Xyz *pqr(Abc def)
BEGIN
...
END
(The original Bourne shell source was, reputedly, written using macros akin to those.)
So, normally, you develop an ad hoc system for recognizing the functions laid out in the style used by your project. One hopes that your project is systematic enough to have a limited variety of choices, but if this is old code that has been maintained over many years by many people, there are likely to be special cases all over the place.
I have a simple solution, that is, mighty Macro! if you only want to apply these to a small set of functions, and they are not called by function pointer
here is the simple solution:
1 locate call functions' definitions and declearations
2 rename them to something else, for example: myfunc to _myfunc, only names changed in these functions so it's fairly easy to change back
3 define a macro, with the name myfunc of which do this: 1) print your log 2) call the original function
you can grab all these functions' declarations to a single macro definition file, do some simple text work(maybe with script lang help, or text editor macro help), but this solution is not good for vast amount of functions, and don't deal with function pointer.
for example, you have function like this:
int myfunc(int arg)
int myfunc(int arg){
return 0;
}
rename myfunc to _myfunc and add a macro named myfunc
#define myfunc(arg)\
print("print whatever you want!");\
myfunc(arg);\
int _myfunc(int arg);
int _myfunc(int arg){
return 0;
}
if macro not good enough, wrap it with another function
int myfunc(arg){
print("print whatever you want!");
return myfunc(arg);
}
int _myfunc(int arg);
int _myfunc(int arg){
return 0;
}
If you want to debug linux kernel with printk and you want to get the call flow at run time, I have a very basic local solution for that.
Open your .c file in notepad++ --> ctrl+H --> Find what: \n{ -> Replace with: \n{ printk \("Your_Name: %s:%d \\n", __func__, __LINE__ \); --> Replace All
Comment kernel parent dir Makefile as like below:
+#KBUILD_CFLAGS += $(call cc-option,-Wdeclaration-after-statement,)
Hope it will solve your problem.