Develop Reference

Replacing error with default value while coercing in numpy array

I'm trying to coerce the values in my NumPy array to float. However, in my array, there might be some values which might not coerce successfully and I want to replace those values with a default value. I do want the speed of NumPy though. I do not want to do the python loop. What's the best route to achieve this behavior?
For instance:
import numpy as np
my_array = np.array(["1", "2", "3", "NA"])
new_array = magic_coerce(my_array, float, -1.0) # I want to implement this
print (new_array) # should print [1., 2., 3. -1.]
I'm trying to write my own ufunc in c, and I have the following:
int is_float(const char* c)
{
regex_t regex;
regcomp(&regex, "^[+-]?([0-9]*[.])?[0-9]+$", REG_EXTENDED);
return regexec(&regex, c, 0, NULL, 0) == 0;
}
float to_float(const char *c, float default_value)
{
float result = default_value;
if (is_float(c))
{
result = atof(c);
}
return result;
}
static PyMethodDef LogitMethods[] = {
{NULL, NULL, 0, NULL}
};
/* The loop definition must precede the PyMODINIT_FUNC. */
static void double_logitprod(char **args, npy_intp *dimensions,
npy_intp* steps, void* data)
{
npy_intp i;
npy_intp n = dimensions[0];
char *in1 = args[0], *in2 = args[1];
char *out = args[2];
npy_intp in1_step = steps[0];
npy_intp out_step = steps[2];
double tmp;
for (i = 0; i < n; i++) {
/*BEGIN main ufunc computation*/
char *tmp1 = (char *) in1;
tmp = *((double *)in2);
*((double *) out) = to_float(tmp1, tmp);
/*END main ufunc computation*/
in1 += in1_step;
out += out_step;
}
}
/*This a pointer to the above function*/
PyUFuncGenericFunction funcs[1] = {&double_logitprod};
/* These are the input and return dtypes of logit.*/
static char types[3] = {NPY_OBJECT, NPY_DOUBLE,
NPY_DOUBLE};
But it looks like it's not working correctly. What's the type for UNICODE in numpy? NPY_UNICODE gives an error, so I coerced it to NPY_OBJECT, but this does not seem to play with it.

How is python's float.__eq__ implemented in the language?

I know that the best way to compare two floats for equality is usually to use math.isclose(float_a, float_b). But I was curious to know how python does it if you simply do float_a == float_b.
I suppose it's implemented in C, but what is the logic behind it ?

Here is the source code for float object comparisons
Essentially. It looks super complex, but that complexity is mostly in handling the case where a float is compared to an int (int objects in Python are arbitrarily sized, they aren't C-int's wrapped in a Python object).
But for the simple case of float and float:
static PyObject*
float_richcompare(PyObject *v, PyObject *w, int op)
{
double i, j;
int r = 0;
assert(PyFloat_Check(v));
i = PyFloat_AS_DOUBLE(v);
/* Switch on the type of w. Set i and j to doubles to be compared,
* and op to the richcomp to use.
*/
if (PyFloat_Check(w))
j = PyFloat_AS_DOUBLE(w);
So it just creates two C doubles from the float objects, then (skipping all the int handling stuff):
Compare:
switch (op) {
case Py_EQ:
r = i == j;
break;
case Py_NE:
r = i != j;
break;
case Py_LE:
r = i <= j;
break;
case Py_GE:
r = i >= j;
break;
case Py_LT:
r = i < j;
break;
case Py_GT:
r = i > j;
break;
}
return PyBool_FromLong(r);
It just does a C-level == comparison, ultimately. So it does not do math.isclose(float_a, float_b). underneath the hood.

Python C binding - get array from python to C++

As the title says: I wold like to make a python binding in C++ that does some algebraic operations on some array. For this, I have to parse the python "array object" into C++ as a vector of double or integer or whatever the case may be.
I tried to do this but I face some issues. I've created a new python type and a class with the name Typer where I have this method that tries to get the elements of a python array, then compute the sum (as a first step).
tatic PyObject *Typer_vectorsum(Typer *self, PyObject *args)
{
PyObject *retval;
PyObject *list;
if (!PyArg_ParseTuple(args, "O", &list))
return NULL;
double *arr;
arr = (double *)malloc(sizeof(double) * PyTuple_Size(list));
int length;
length = PyTuple_Size(list);
PyObject *item = NULL;
for (int i = 0; i < length; ++i)
{
item = PyTuple_GetItem(list, i);
if (!PyFloat_Check(item))
{
exit(1);
}
arr[i] = PyFloat_AsDouble(item);
}
double result = 0.0;
for (int i = 0; i < length; ++i)
{
result += arr[i];
}
retval = PyFloat_FromDouble(result);
free(arr);
return retval;
}
In this method I parse the python array object into a C array (allocating the memory of the array with malloc). Then I add every element from the object to my C array and just compute the sum in the last for-loop.
If I build the project and then create a python test file, nothing happens (the file compiles without any issues but it is not printing anything).
y = example.Typer() . #typer is the init
tuple = (1, 2, 3)
print(y.vectorsum(tuple))
Am I missing something? And also, Is there a nice and easy way of getting a python array object into C++, but as a std::vector instead of a classic C array?
Thank you in advance!

The tuple contains ints, not floats, so your PyFloat_Check fails. And no, there is no direct way from Python tuple to C array or C++ std::vector. The reason being that the tuple is an array of Python objects, not an array of C values such as doubles.
Here's your example with improved error checking, after which it should work:
PyObject *retval;
PyObject *list;
if (!PyArg_ParseTuple(args, "O!", &PyTuple_Type, &list))
return NULL;
double *arr =
arr = (double *)malloc(sizeof(double) * PyTuple_GET_SIZE(list));
int length;
length = PyTuple_GET_SIZE(list);
PyObject *item = NULL;
for (int i = 0; i < length; ++i)
{
item = PyTuple_GET_ITEM(list, i);
arr[i] = PyFloat_AsDouble(item);
if (arr[i] == -1. && PyErr_Occurred())
{
exit(1);
}
}
double result = 0.0;
for (int i = 0; i < length; ++i)
{
result += arr[i];
}
retval = PyFloat_FromDouble(result);
free(arr);
return retval;

Pass a numpy array from Python to C

I'm having successfully embedded a Python script into a C module. The Python script produces a multi-dimensional Numpy array. Whereas the entire calculation in python takes 9 ms, the final tolist() conversion in order to return it to C takes 4 ms alone. I would like to change that by passing the Numpy array as reference and do the iterations in C again. But I can't currently figure out, how this can be done.
There are a lot of samples around, which use the other way around: Passing a Numpy array to a C function which is called from Python, but this is not my use case.
Any pointer welcome.

Ok, it's a while ago but I solved it like so:
My python process delivers an array, containing one array, containing one array, containing N arrays of M floats each. The input is a JPEG image.
Unwrapping it like so:
int predict(PyObject *pyFunction, unsigned char *image_pointer, unsigned long image_len) {
int result = -1;
PyObject *pImage = NULL;
PyObject *pList = NULL;
pImage = PyBytes_FromStringAndSize((const char *)image_pointer, image_len);
if (!pImage) {
fprintf(stderr, "Cannot provide image to python 'predict'\n");
return result;
}
pList = PyObject_CallFunctionObjArgs(pyFunction, pImage, NULL);
Py_DECREF(pImage);
PyArrayObject *pPrediction = reinterpret_cast<PyArrayObject *>(pList);
if (!pPrediction) {
fprintf(stderr, "Cannot predict, for whatever reason\n");
return result;
}
if (PyArray_NDIM(pPrediction) != 4) {
fprintf(stderr, "Prediction failed, returned array with wrong dimensions\n");
} else {
RESULTPTR pResult = reinterpret_cast<RESULTPTR>(PyArray_DATA(pPrediction));
int len0 = PyArray_SHAPE(pPrediction)[0];
int len1 = PyArray_SHAPE(pPrediction)[1];
int len2 = PyArray_SHAPE(pPrediction)[2];
int len3 = PyArray_SHAPE(pPrediction)[3];
for (int i = 0; i < len0; i++) {
int offs1 = i * len1;
for (int j = 0; j < len1; j++) {
int offs2 = j * len2;
for (int k = 0; k < len2; k++) {
int offs3 = k * len3;
for (int l = 0; l < len3; l++) {
float f = (*pResult)[offs1 + offs2 + offs3 + l];
//printf("data: %.8f\n", f);
}
}
}
}
result = 0;
}
Py_XDECREF(pList);
return result;
}
HTH

How does python extend work?

If I have a list:
a = [1,2,3,4]
and then add 4 elements using extend
a.extend(range(5,10))
I get
a = [1, 2, 3, 4, 5, 6, 7, 8, 9]
How does python do this? does it create a new list and copy the elements across or does it make 'a' bigger? just concerned that using extend will gobble up memory. I'am also asking as there is a comment in some code I'm revising that extending by 10000 x 100 is quicker than doing it in one block of 1000000.

Python's documentation on it says:
Extend the list by appending all the
items in the given list; equivalent to
a[len(a):] = L.
As to "how" it does it behind the scene, you really needn't concern yourself about it.

L.extend(M) is amortized O(n) where n=len(m), so excessive copying is not usually a problem. The times it can be a problem is when there is not enough space to extend into, so a copy is performed. This is a problem when the list is large and you have limits on how much time is acceptable for an individual extend call.
That is the point when you should look for a more efficient datastructure for your problem. I find it is rarely a problem in practice.
Here is the relevant code from CPython, you can see that extra space is allocated when the list is extended to avoid excessive copying
static PyObject *
listextend(PyListObject *self, PyObject *b)
{
PyObject *it; /* iter(v) */
Py_ssize_t m; /* size of self */
Py_ssize_t n; /* guess for size of b */
Py_ssize_t mn; /* m + n */
Py_ssize_t i;
PyObject *(*iternext)(PyObject *);
/* Special cases:
1) lists and tuples which can use PySequence_Fast ops
2) extending self to self requires making a copy first
*/
if (PyList_CheckExact(b) || PyTuple_CheckExact(b) || (PyObject *)self == b) {
PyObject **src, **dest;
b = PySequence_Fast(b, "argument must be iterable");
if (!b)
return NULL;
n = PySequence_Fast_GET_SIZE(b);
if (n == 0) {
/* short circuit when b is empty */
Py_DECREF(b);
Py_RETURN_NONE;
}
m = Py_SIZE(self);
if (list_resize(self, m + n) == -1) {
Py_DECREF(b);
return NULL;
}
/* note that we may still have self == b here for the
* situation a.extend(a), but the following code works
* in that case too. Just make sure to resize self
* before calling PySequence_Fast_ITEMS.
*/
/* populate the end of self with b's items */
src = PySequence_Fast_ITEMS(b);
dest = self->ob_item + m;
for (i = 0; i < n; i++) {
PyObject *o = src[i];
Py_INCREF(o);
dest[i] = o;
}
Py_DECREF(b);
Py_RETURN_NONE;
}
it = PyObject_GetIter(b);
if (it == NULL)
return NULL;
iternext = *it->ob_type->tp_iternext;
/* Guess a result list size. */
n = _PyObject_LengthHint(b, 8);
if (n == -1) {
Py_DECREF(it);
return NULL;
}
m = Py_SIZE(self);
mn = m + n;
if (mn >= m) {
/* Make room. */
if (list_resize(self, mn) == -1)
goto error;
/* Make the list sane again. */
Py_SIZE(self) = m;
}
/* Else m + n overflowed; on the chance that n lied, and there really
* is enough room, ignore it. If n was telling the truth, we'll
* eventually run out of memory during the loop.
*/
/* Run iterator to exhaustion. */
for (;;) {
PyObject *item = iternext(it);
if (item == NULL) {
if (PyErr_Occurred()) {
if (PyErr_ExceptionMatches(PyExc_StopIteration))
PyErr_Clear();
else
goto error;
}
break;
}
if (Py_SIZE(self) < self->allocated) {
/* steals ref */
PyList_SET_ITEM(self, Py_SIZE(self), item);
++Py_SIZE(self);
}
else {
int status = app1(self, item);
Py_DECREF(item); /* append creates a new ref */
if (status < 0)
goto error;
}
}
/* Cut back result list if initial guess was too large. */
if (Py_SIZE(self) < self->allocated)
list_resize(self, Py_SIZE(self)); /* shrinking can't fail */
Py_DECREF(it);
Py_RETURN_NONE;
error:
Py_DECREF(it);
return NULL;
}
PyObject *
_PyList_Extend(PyListObject *self, PyObject *b)
{
return listextend(self, b);
}

It works as if it were defined like this
def extend(lst, iterable):
for x in iterable:
lst.append(x)
This mutates the list, it does not create a copy of it.
Depending on the underlying implementation, append and extend may trigger the list to copy its own data structures but this is normal and nothing to worry about. For example array-based implementations typically grow the underlying array exponentially and need to copy the list of elements when they do so.

How does python do this? does it create a new list and copy the elements across or does it make 'a' bigger?
>>> a = ['apples', 'bananas']
>>> b = a
>>> a is b
True
>>> c = ['apples', 'bananas']
>>> a is c
False
>>> a.extend(b)
>>> a
['apples', 'bananas', 'apples', 'bananas']
>>> b
['apples', 'bananas', 'apples', 'bananas']
>>> a is b
True
>>>

It does not create a new list object, it extends a. This is self-evident from the fact that you don't make an assigment. Python will not magically replace your objects with other objects. :-)
How the memory allocation happens inside the list object is implementation dependent.