I'm venturing into C extensions for the first time, and am somewhat new to C as well. I've got a working C extension, however, if i repeatedly call the utility in python, I eventually get a segmentation fault: 11.
#include <Python.h>
static PyObject *getasof(PyObject *self, PyObject *args) {
PyObject *fmap;
long dt;
if (!PyArg_ParseTuple(args, "Ol", &fmap, &dt))
return NULL;
long length = PyList_Size(fmap);
for (int i = 0; i < length; i++) {
PyObject *event = PyList_GetItem(fmap, i);
long dti = PyInt_AsLong(PyList_GetItem(event, 0));
if (dti > dt) {
PyObject *output = PyList_GetItem(event, 1);
return output;
}
}
Py_RETURN_NONE;
};
The function args are
a time series (list of lists): ex [[1, 'a'], [5, 'b']]
a time (long): ex 4
And it's supposed to iterate over the list of lists til it finds a value greater than the time given. Then return that value. As I mentioned, it correctly returns the answer, but if I call it enough times, it segfaults.
My gut feeling is that this has to do with reference counting, but I'm not familiar enough with the concept to know if this is the direct cause.
Any help would be appreciated.
"My gut feeling is that this has to do with reference counting..." Your instincts are correct.
PyList_GetItem returns a borrowed reference, which means your function doesn't "own" a reference to the item. So there is a problem here:
PyObject *output = PyList_GetItem(event, 1);
return output;
You don't own a reference to the item, but you return it to the caller, so the caller doesn't own a reference either. The caller will run into a problem if the item is garbage collected while the caller is still trying to use it. So you'll need to increase the reference count of the item before you return it:
PyObject *output = PyList_GetItem(event, 1);
Py_INCREF(output);
return output;
That assumes that PyList_GetItem(event, 1) doesn't fail! Except for PyArg_ParseTuple, you aren't checking the return values of the C API functions, which means you are assuming the input argument always has the exact structure that you expect. That's fine while you're testing code and figuring out how this works, but eventually you should be checking the return values of the C API functions for failure, and handling it appropriately.
Related
I have a module method which takes in a python list, and then outputs the same list with all items multiplied by 100.
I've attemped to follow the C intro here as close as possible but still running into issues.
static PyObject *
test_update_list(PyObject *self, PyObject *args)
{
PyObject *listObj = NULL;
PyObject *item = NULL;
PyObject *mult = PyLong_FromLong(100);
PyObject *incremented_item = NULL;
if (!PyArg_ParseTuple(args, "O", &listObj))
{
return NULL;
}
/* get the number of lines passed to us */
Py_ssize_t numLines = PyList_Size(listObj);
/* should raise an error here. */
if (numLines < 0) return NULL; /* Not a list */
for (Py_ssize_t i=0; i<numLines; i++) {
// pick the item
item = PyList_GetItem(listObj, i);
if (mult == NULL)
goto error;
// increment it
incremented_item = PyNumber_Add(item, mult);
if (incremented_item == NULL)
goto error;
// update the list item
if (PyObject_SetItem(listObj, i, incremented_item) < 0)
goto error;
}
error:
Py_XDECREF(item);
Py_XDECREF(mult);
Py_XDECREF(incremented_item);
return listObj;
};
The above complies fine, however when I run in ipython, I get the below error.
If I take away the error handling I get a seg fault.
---------------------------------------------------------------------------
SystemError Traceback (most recent call last)
SystemError: null argument to internal routine
The above exception was the direct cause of the following exception:
SystemError Traceback (most recent call last)
<ipython-input-3-da275aa3369f> in <module>()
----> 1 testadd.test_update_list([1,2,3])
SystemError: <built-in function ulist> returned a result with an error set
Any help is appreciated.
So you have a number of issues that all need to be corrected. I've listed them all under separate headings so you can go through them one at a time.
Always returning listObj
When you get an error in your for loop, you would goto the error label, which was still returning the list. By returning this list you hide that there was an error in your function. You must always return NULL when you expect your function to raise an exception.
Does not increment listObj ref count on return
When your function is invoked you are given a borrowed reference to your arguments. When you return one of those arguments you are creating a new reference to your list, and so must increment its reference count. Otherwise the interpreter will have a reference count that is one lower than the number of actual references to the object. This will end up with a bug where the interpreter deallocates your list when there is only 1 reference rather than 0! This could result in a seg fault, or it could in the worst case scenario result in random parts of the program access the that has since been deallocated and allocated for some other object.
Uses PyObject_SetItem with primitive
PyObject_SetItem can be used with dicts and other class that implements obj[key] = val. So you cannot supply it with an argument of type Py_ssize_t. Instead, use PyList_SetItem which only accepts Py_ssize_t as its index argument.
Bad memory handling of item and incremented_item
PyObject_SetItem and PyList_SetItem both handle decreasing the reference count of the object that was already at the position that was being set. So we don't need to worry about managing the reference count of item as we are only working with a reference borrowed from the list. These pair of functions also steal a reference to incremented_item, and so we don't need to worry about managing its reference count either.
Memory leak on incorrect arguments
For example, when you call your function with an int. You will create a new reference to the 100 int object, but because you return NULL rather than goto error, this reference will be lost. As such you need to handle such scenarios differently. In my solution, I move the PyLong_FromLong call to after the arg and type checking. In this way we are only create this new* object once we are guaranteed it will be used.
Working code
Side note: I removed the goto statements as there was only one left, and so it made more sense to do the error handling at that point rather than later.
static PyObject *
testadd_update_list(PyObject *self, PyObject *args)
{
PyObject *listObj = NULL;
PyObject *item = NULL;
PyObject *mult = NULL;
PyObject *incremented_item = NULL;
Py_ssize_t numLines;
if (!PyArg_ParseTuple(args, "O:update_list", &listObj))
{
return NULL;
}
if (!PyList_Check(listObj)) {
PyErr_BadArgument();
return NULL;
}
/* get the number of lines passed to us */
// Don't want to rely on the error checking of this function as it gives a weird stack trace.
// Instead, we use Py_ListCheck() and PyErr_BadArgument() as above. Since list is definitely
// a list now, then PyList_Size will never throw an error, and so we could use
// PyList_GET_SIZE(listObj) instead.
numLines = PyList_Size(listObj);
// only initialise mult here, otherwise the above returns would create a memory leak
mult = PyLong_FromLong(100);
if (mult == NULL) {
return NULL;
}
for (Py_ssize_t i=0; i<numLines; i++) {
// pick the item
// It is possible for this line to raise an error, but our invariants should
// ensure no error is ever raised. `list` is always of type list and `i` is always
// in bounds.
item = PyList_GetItem(listObj, i);
// increment it, and check for type errors or memory errors
incremented_item = PyNumber_Add(item, mult);
if (incremented_item == NULL) {
// ERROR!
Py_DECREF(mult);
return NULL;
}
// update the list item
// We definitely have a list, and our index is in bounds, so we should never see an error
// here.
PyList_SetItem(listObj, i, incremented_item);
// PyList_SetItem steals our reference to incremented_item, and so we must be careful in
// how we handle incremented_item now. Either incremented_item will not be our
// responsibility any more or it is NULL. As such, we can just remove our Py_XDECREF call
}
// success!
// We are returning a *new reference* to listObj. We must increment its ref count as a result!
Py_INCREF(listObj);
Py_DECREF(mult);
return listObj;
}
Footnote:
* PyLong_FromLong(100) doesn't actually create a new object, but rather returns a new reference to an existing object. Integers with low values (0 <= i < 128 I think) are all cached and this same object is returned when needed. This is an implementation detail that is meant to avoid high levels of allocating and deallocating integers for small values, and so improve the performance of Python.
I have a question on how to properly use Py_DECREF() on PyList in C. So let say I have a function called build_list() which accepts a string linked list as its input argument and returns a Python list if everything goes well, or NULL if there is an error.
Below is the minimalistic example:
struct strlist {
char *str;
size_t len;
struct strlist *next;
};
PyObject *build_list (struct strlist *inlist) {
struct strlist *node = NULL;
PyObject *tmp_obj = NULL;
int success;
PyObject *ret_obj = PyList_New(0);
if (ret_obj == NULL) {
return NULL;
}
node = inlist;
while (node != NULL) {
tmp_obj = PyString_FromStringAndSize(node->str, node->len);
if (tmp_obj == NULL) {
Py_DECREF(ret_obj);
return NULL;
}
else {
success = PyList_Append(ret_obj, tmp_obj);
Py_DECREF(tmp_obj);
if (success != 0) {
Py_DECREF(ret_obj);
return NULL;
}
}
node = node->next;
}
return ret_obj;
}
Have I used Py_DECREF() correctly in this case?
My particular question is:
If a few elements have been appended to the list before an error occur, my code will decrement the reference to the list directly (inside success != 0) while the elements inside the list technically still have refcount of 1. Should I instead decrement each element's reference first before I finally decrement the reference to the list?
Thank you.
After creating the list, it has a refcount of 1. Each string is born with a refcount of 1 and appending it to the list increases that to 2 (because the list and your function reference it). So it's correct to DECREF after the Append(), as your function no longer uses the string itself.
Inside both error paths (tmp_obj == NULL, success != 0), DECREFing the list-object will free the object (because it's refcount is now 0). The list-object will walk it's members and DECREF every one of them, reducing the refcount of each string to 0, freeing them.
Long story short: Your code is correct. The elements inside the list must have a refcount of (at least) 1, because the list is referring to them. It's the list's exclusive responsibility to DECREF it's members.
As an exercise, you may try reducing the strings' refcounts yourself. The interpreter will most likely crash (maybe at exit()), because when the list is freed, the strings' refcounts go to -1, triggering an assertion.
I've written a Python C++ extension, however I have a problem with one of its functions.
The function provided by this extension takes 2 arrays as inputs and produces one as an output.
I've only left the relevant part of function's code
float* forward(float* input, float* kernels, npy_intp* input_dims, npy_intp* kernels_dims){
float* output = new float[output_size];
//some irrelevant matrix operation code
return output;
}
And the wrapper:
static PyObject *module_forward(PyObject *self, PyObject *args)
{
PyObject *input_obj, *kernels_obj;
if (!PyArg_ParseTuple(args, "OO", &input_obj, &kernels_obj))
return NULL;
PyObject *input_array = PyArray_FROM_OTF(input_obj, NPY_FLOAT, NPY_IN_ARRAY);
PyObject *kernels_array = PyArray_FROM_OTF(kernels_obj, NPY_FLOAT, NPY_IN_ARRAY);
if (input_array == NULL || kernels_array == NULL) {
Py_XDECREF(input_array);
Py_XDECREF(kernels_array);
return NULL;
}
float *input = (float*)PyArray_DATA(input_array);
float *kernels = (float*)PyArray_DATA(kernels_array);
npy_intp *input_dims = PyArray_DIMS(input_array);
npy_intp *kernels_dims = PyArray_DIMS(kernels_array);
/////////THE ACTUAL FUNCTION
float* output = forward(input, kernels, input_dims, kernels_dims);
Py_DECREF(input_array);
Py_DECREF(kernels_array);
npy_intp output_dims[4] = {input_dims[0], input_dims[1]-kernels_dims[0]+1, input_dims[2]-kernels_dims[1]+1, kernels_dims[3]};
PyObject* ret_output = PyArray_SimpleNewFromData(4, output_dims, NPY_FLOAT, output);
delete output;//<-----THE PROBLEMATIC LINE////////////////////////////
PyObject *ret = Py_BuildValue("O", ret_output);
Py_DECREF(ret_output);
return ret;
}
The delete operator that I highlighted is where the magic happens: without it this function leaks memory, with it it crashes because of memory access violation.
The fun thing is I wrote another method, that returns two arrays. So the function returns a float** pointing to two float* elements:
float** gradients = backward(input, kernels, grads, input_dims, kernel_dims, PyArray_DIMS(grads_array));
Py_DECREF(input_array);
Py_DECREF(kernels_array);
Py_DECREF(grads_array);
PyObject* ret_g_input = PyArray_SimpleNewFromData(4, input_dims, NPY_FLOAT, gradients[0]);
PyObject* ret_g_kernels = PyArray_SimpleNewFromData(4, kernel_dims, NPY_FLOAT, gradients[1]);
delete gradients[0];
delete gradients[1];
delete gradients;
PyObject* ret_list = PyList_New(0);
PyList_Append(ret_list, ret_g_input);
PyList_Append(ret_list, ret_g_kernels);
PyObject *ret = Py_BuildValue("O", ret_list);
Py_DECREF(ret_g_input);
Py_DECREF(ret_g_kernels);
return ret;
Notice that the second example works flawlessly, no crashes or memory leaks, while still calling delete on arrays after they have been built into PyArray objects.
Could someone enlighten me about what's going on in here?
From the PyArray_SimpleNewFromData docs:
Create an array wrapper around data pointed to by the given pointer.
If you create an array with PyArray_SimpleNewFromData, it's going to create a wrapper around the data you give it, rather than making a copy. That means the data it wraps has to outlive the array. delete-ing the data violates that.
You have several options:
You could create the array differently so you don't just make a wrapper around the original data.
You could carefully control access to the array and make sure its lifetime ends before you delete the data.
You could create a Python object that owns the data and will delete the data when the object's lifetime ends, and set the array's base to that object with PyArray_SetBaseObject, so the array keeps the owner object alive until the array itself dies.
Refering to http://mail.python.org/pipermail/python-dev/2009-June/090210.html
AND http://dan.iel.fm/posts/python-c-extensions/
and here is other places i searched regarding my question:
http://article.gmane.org/gmane.comp.python.general/424736
http://joyrex.spc.uchicago.edu/bookshelves/python/cookbook/pythoncook-CHP-16-SECT-3.html
http://docs.python.org/2/c-api/sequence.html#PySequence_Check
Python extension module with variable number of arguments
I am inexperienced in Python/C API.
I have the following code:
sm_int_list = (1,20,3)
c_int_array = (ctypes.c_int * len(sm_int_list))(*sm_int_list)
sm_str_tuple = ('some','text', 'here')
On the C extension side, i have done something like this:
static PyObject* stuff_here(PyObject *self, PyObject *args)
{
char* input;
int *i1, *i2;
char *s1, *s2;
// args = (('some','text', 'here'), [1,20,3], ('some','text', 'here'), [1,20,3])
**PyArg_ParseTuple(args, "(s#:):#(i:)#(s#:):#(i:)#", &s1, &i1, &s2, &i2)**;
/*stuff*/
}
such that:
stuff.here(('some','text', 'here'), [1,20,3], ('some','text', 'here'), [1,20,3])
returns data in the same form as args after some computation.
I would like to know the PyArg_ParseTuple expression, is it the proper way to parse
an array of varying string
an array of integers
UPDATE NEW
Is this the correct way?:
static PyObject* stuff_here(PyObject *self, PyObject *args)
unsigned int tint[], cint[];
ttotal=0, ctotal=0;
char *tstr, *cstr;
int *t_counts, *c_counts;
Py_ssize_t size;
PyObject *t_str1, *t_int1, *c_str2, *c_int2; //the C var that takes in the py variable value
PyObject *tseq, cseq;
int t_seqlen=0, c_seqlen=0;
if (!PyArg_ParseTuple(args, "OOiOOi", &t_str1, &t_int1, &ttotal, &c_str2, &c_int2, &ctotal))
{
return NULL;
}
if (!PySequence_Check(tag_str1) && !PySequence_Check(cat_str2)) return NULL;
else:
{
//All things t
tseq = PySequence_Fast(t_str1, "iterable");
t_seqlen = PySequence_Fast_GET_SIZE(tseq);
t_counts = PySequence_Fast(t_int1);
//All things c
cseq = PySequence_Fast(c_str2);
c_seqlen = PySequence_Fast_GET_SIZE(cseq);
c_counts = PySequence_Fast(c_int2);
//Make c arrays of all things tag and cat
for (i=0; i<t_seqlen; i++)
{
tstr[i] = PySequence_Fast_GET_ITEM(tseq, i);
tcounts[i] = PySequence_Fast_GET_ITEM(t_counts, i);
}
for (i=0; i<c_seqlen; i++)
{
cstr[i] = PySequence_Fast_GET_ITEM(cseq, i);
ccounts[i] = PySequence_Fast_GET_ITEM(c_counts, i);
}
}
OR
PyArg_ParseTuple(args, "(s:)(i:)(s:)(i:)", &s1, &i1, &s2, &i2)
And then again while returning,
Py_BuildValue("sisi", arr_str1,arr_int1,arr_str2,arr_int2) ??
Infact if someone could in detail clarify the various PyArg_ParseTuple function that would be of great benefit. the Python C API, as i find it in the documentation, is not exactly a tutorial on things to do.
You can use PyArg_ParseTuple to parse a real tuple, that has a fixed structure. Especially the number of items in the subtuples cannot change.
As the 2.7.5 documentation says, your format "(s#:):#(i:)#(s#:):#(i:)#" is wrong since : cannot occur in nested parenthesis. The format "(sss)(iii)(sss)(iii)", along with total of 12 pointer arguments should match your arguments. Likewise for Py_BuildValue you can use the same format string (which creates 4 tuples within 1 tuple), or "(sss)[iii](sss)[iii]" if the type matters (this makes the integers to be in lists instead of tuples).
I am using what seems to be the exact usgae of PyArg_ParseTuple, yet the code is still failing to work. I am using python 2.7
This is my C code for the Python Extension I am writing:
static PyObject* tpp(PyObject* self, PyObject* args)
{
PyObject* obj;
PyObject* seq;
int i, len;
PyObject* item;
int arrayValue, temp;
if (!PyArg_ParseTuple(args, "O", &obj)){
printf("Item is not a list\n");
return NULL;
}
seq = PySequence_Fast(obj, "expected a sequence");
len = PySequence_Size(obj);
arrayValue = -5;
printf("[\n");
for (i = 0; i < len; i++) {
item = PySequence_Fast_GET_ITEM(seq, i);
// printf("%d : %d, PyArg: ", item, *item);
// PyArg_ParseTuple(item, "I", &temp);
PyObject* objectsRepresentation = PyObject_Repr(item);
const char* s = PyString_AsString(objectsRepresentation);
printf("%s\n", s);
PyObject* objType = PyObject_Type(item);
PyObject* objTypeString = PyObject_Repr(objType);
const char* sType = PyString_AsString(objTypeString);
printf("%s\n", sType);
if (PyArg_ParseTuple(item, "i", &arrayValue) != 0){
printf("%d\n", arrayValue);
printf("horray!\n");
}
}
Py_DECREF(seq);
printf("]\n");
printf("Item is a list!\n");
Py_RETURN_NONE;
}
Then I just build the extension and go to the terminal
import et
and then
et.tpp([1,2])
fails to print the line
if (PyArg_ParseTuple(item, "i", &arrayValue) != 0){
printf("%d\n", arrayValue);
printf("horray!\n");
}
I checked the type, as you can see in the code, of the elements in the list, and it prints 'int'. Yet for some reason PyArg_ParseTuple is having errors.
I need to be able to access information from lists in python to copy some data, pass it to my C code elsewhere, and then return the result to python.
Thank you so much!
The answer is to use long PyInt_AsLong(PyObject *io)
"long PyInt_AsLong(PyObject *io) Will first attempt to cast the object to a PyIntObject, if it is not already one, and then return its value. If there is an error, -1 is returned, and the caller should check PyErr_Occurred() to find out whether there was an error, or whether the value just happened to be -1."
This is from http://docs.python.org/2/c-api/int.html That is the official c python int objects documentation which has all relevant methods.
Unfortunately this returns only a long value. However, a simple cast should suffice if the expected values will be small.
PyArg_ParseTuple() is about parsing tuples only, as the name suggests. In your code, item is an int, not a tuple. In order to convert an int object to a C value, you need to use arrayValue = PyInt_AsLong(item). Note that it returns a C long, not an int, so you should declare arrayValue as a long.
(EDIT: previously I mentioned PyInt_FromLong by mistake.)