Let's say I have the following function written in C:
/* C-Code for calculating the average for a given dataset */
#define NOE 16
int calcAvg(float *data, float *avg)
{
float sum; sum = 0;
int i;
for (i = 0; i < NOE; i++)
{
data[i] = i;
sum += data[i];
}
avg = sum/n;
return 0;
}
Now I want to pass the data from an np.array to that C function "calAvg". Also, I want the result to be stored in "result" which I defined in python.
# Python Code
result = float(0)
a = np.array([1, 2, 3, 4])
myCfuncion.calc(a, result)
I have already created a C module and imported it into python. The problem I have is that I do not know how to pass pointers in such a way I showed.
Anybody does have an idea?
Related
I'm having successfully embedded a Python script into a C module. The Python script produces a multi-dimensional Numpy array. Whereas the entire calculation in python takes 9 ms, the final tolist() conversion in order to return it to C takes 4 ms alone. I would like to change that by passing the Numpy array as reference and do the iterations in C again. But I can't currently figure out, how this can be done.
There are a lot of samples around, which use the other way around: Passing a Numpy array to a C function which is called from Python, but this is not my use case.
Any pointer welcome.
Ok, it's a while ago but I solved it like so:
My python process delivers an array, containing one array, containing one array, containing N arrays of M floats each. The input is a JPEG image.
Unwrapping it like so:
int predict(PyObject *pyFunction, unsigned char *image_pointer, unsigned long image_len) {
int result = -1;
PyObject *pImage = NULL;
PyObject *pList = NULL;
pImage = PyBytes_FromStringAndSize((const char *)image_pointer, image_len);
if (!pImage) {
fprintf(stderr, "Cannot provide image to python 'predict'\n");
return result;
}
pList = PyObject_CallFunctionObjArgs(pyFunction, pImage, NULL);
Py_DECREF(pImage);
PyArrayObject *pPrediction = reinterpret_cast<PyArrayObject *>(pList);
if (!pPrediction) {
fprintf(stderr, "Cannot predict, for whatever reason\n");
return result;
}
if (PyArray_NDIM(pPrediction) != 4) {
fprintf(stderr, "Prediction failed, returned array with wrong dimensions\n");
} else {
RESULTPTR pResult = reinterpret_cast<RESULTPTR>(PyArray_DATA(pPrediction));
int len0 = PyArray_SHAPE(pPrediction)[0];
int len1 = PyArray_SHAPE(pPrediction)[1];
int len2 = PyArray_SHAPE(pPrediction)[2];
int len3 = PyArray_SHAPE(pPrediction)[3];
for (int i = 0; i < len0; i++) {
int offs1 = i * len1;
for (int j = 0; j < len1; j++) {
int offs2 = j * len2;
for (int k = 0; k < len2; k++) {
int offs3 = k * len3;
for (int l = 0; l < len3; l++) {
float f = (*pResult)[offs1 + offs2 + offs3 + l];
//printf("data: %.8f\n", f);
}
}
}
}
result = 0;
}
Py_XDECREF(pList);
return result;
}
HTH
I am working on a project involving object detection through deep learning, with the underlying detection code written in C. Due to the requirements of the project, this code has a Python wrapper around it, which interfaces with the required C functions through ctypes. Images are read from Python, and then transferred into C to be processed as a batch.
In its current state, the code is very unoptimized: the images (640x360x3 each) are read using cv2.imread then stacked into a numpy array. For example, for a batch size of 16, the dimensions of this array are (16,360,640,3). Once this is done, a pointer to this array is passed through ctypes into C where the array is parsed, pixel values are normalized and rearranged into a 2D array. The dimensions of the 2D array are 16x691200 (16x(640*360*3)), arranged as follows.
row [0]: Image 0: (B)r0(B)r1(B)r2.... (G)r0(G)r1(G)r2.... (R)r0(R)r1(R)r2....
row [1]: Image 1: (B)r0(B)r1(B)r2.... (G)r0(G)r1(G)r2.... (R)r0(R)r1(R)r2....
.
.
row [15]: Image 15: (B)r0(B)r1(B)r2.... (G)r0(G)r1(G)r2.... (R)r0(R)r1(R)r2....
`
The C code for doing this currently looks like this, where the pixel values are accessed through strides and arranged sequentially per image. nb is the total number of images in the batch (usually 16); h, w, c are 360,640 and 3 respectively.
matrix ndarray_to_matrix(unsigned char* src, long* shape, long* strides)
{
int nb = shape[0];
int h = shape[1];
int w = shape[2];
int c = shape[3];
matrix X = make_matrix(nb, h*w*c);
int step_b = strides[0];
int step_h = strides[1];
int step_w = strides[2];
int step_c = strides[3];
int b, i, j, k;
int index1, index2 = 0;
for(b = 0; b < nb ; ++b) {
for(i = 0; i < h; ++i) {
for(k= 0; k < c; ++k) {
for(j = 0; j < w; ++j) {
index1 = k*w*h + i*w + j;
index2 = step_b*b + step_h*i + step_w*j + step_c*k;
X.vals[b][index1] = src[index2]/255.;
}
}
}
}
return X;
}
And the corresponding Python code that calls this function: (array is the original numpy array)
for i in range(start, end):
imgName = imgDir + '/' + allImageName[i]
img = cv2.imread(imgName, 1)
batchImageData[i-start,:,:] = img[:,:]
data = batchImageData.ctypes.data_as(POINTER(c_ubyte))
resmatrix = self.ndarray_to_matrix(data, batchImageData.ctypes.shape, batchImageData.ctypes.strides)
As of now, this ctypes implementation takes about 35 ms for a batch of 16 images. I'm working on a very FPS critical image processing pipeline, so is there a more efficient way of doing these operations? Specifically:
Can I read the image directly as a 'strided' one dimensional array in Python from disk, thus avoiding the iterative access and copying?
I have looked into numpy operations such as:
np.ascontiguousarray(img.transpose(2,0,1).flat, dtype=float)/255. which should achieve something similar, but this is actually taking more time possibly because of it being called in Python.
Would Cython help anywhere during the read operation?
Regarding the ascontiguousarray method, I'm assuming that it's pretty slow as python has to do some memory works to return a C-like contiguous array.
EDIT 1:
I saw this answer, apparently openCV's imread function should already return a contiguous array.
I am not very familiar with ctypes, but happen to use the PyBind library and can only recommend using it. It implements Python's buffer protocol hence allowing you to interact with python data with almost no overhead.
I've answered a question explaining how to pass a numpy array from Python to C/C++, do something dummy to it in C++ and return a dynamically created array back to Python.
EDIT 2 : I've added a simple example that receives a Numpy array, send it to C and prints it from C. You can find it here. Hope it helps!
EDIT 3 :
To answer your last comment, yes you can definitely do that.
You could modify your code to (1) instantiate a 2D numpy array in C++, (2) pass its reference to the data to your C function that will modify it instead of declaring a Matrix and (3) return that instance to Python by reference.
Your function would become:
void ndarray_to_matrix(unsigned char* src, double * x, long* shape, long* strides)
{
int nb = shape[0];
int h = shape[1];
int w = shape[2];
int c = shape[3];
int step_b = strides[0];
int step_h = strides[1];
int step_w = strides[2];
int step_c = strides[3];
int b, i, j, k;
int index1, index2 = 0;
for(b = 0; b < nb ; ++b) {
for(i = 0; i < h; ++i) {
for(k= 0; k < c; ++k) {
for(j = 0; j < w; ++j) {
index1 = k*w*h + i*w + j;
index2 = step_b*b + step_h*i + step_w*j + step_c*k;
X.vals[b][index1] = src[index2]/255.;
}
}
}
}
}
And you'd add, in your C++ wrapper code
// Instantiate the output array, assuming we know b, h, c,w
py::array_t<double> x = py::array_t<double>(b*h*c*w);
py::buffer_info bufx = x.request();
double*ptrx = (double *) bufx.ptr;
// Call to your C function with ptrx as input
ndarray_to_matrix(src, ptrx, shape, strides);
// now reshape x
x.reshape({b, h*c*w});
Do not forget to modify the prototype of the C++ wrapper function to return a numpy array like:
py::array_t<double> read_matrix(...){}...
This should work, I didn't test it though :)
I have written a good bit of code in python and it works great. But now I'm scaling up the size of the problems that I'm analyzing and python is dreadfully slow. The slow part of the python code is
for i in range(0,H,1):
x1 = i - length
x2 = i + length
for j in range(0,W,1):
#print i, ',', j # check the limits
y1 = j - length
y2 = j + length
IntRed[i,j] = np.mean(RawRed[x1:x2,y1:y2])
With H and W equal to 1024 the function takes around 5 minutes to excute. I've written a simple c++ program/function that performs the same computation and it excutes in less than a second with the same data size.
double summ = 0;
double total_num = 0;
double tmp_num = 0 ;
int avesize = 2;
for( i = 0+avesize; i <X-avesize ;i++)
for(j = 0+avesize;j<Y-avesize;j++)
{
// loop through sub region of the matrix
// if the value is not zero add it to the sum
// and increment the counter.
for( int ii = -2; ii < 2; ii ++)
{
int iii = i + ii;
for( int jj = -2; jj < 2 ; jj ++ )
{
int jjj = j + jj;
tmp_num = gsl_matrix_get(m,iii,jjj);
if(tmp_num != 0 )
{
summ = summ + tmp_num;
total_num++;
}
}
}
gsl_matrix_set(Matrix_mean,i,j,summ/total_num);
summ = 0;
total_num = 0;
}
I have some other methods to perform on the 2D array. The one listed is a simple examples.
What I want to do is pass a python 2D array to my c++ function and return a 2D array back to python.
I've read a bit about swig, and have sereached pervious questions, and it seems like it's a possible solution. But I can't seem to figure out what I actually need to do.
Can I get any help? Thanks
You can use arrays as it is described here: Doc - 5.4.5 Arrays, the carray.i or std_vector.i from the SWIG library.
I find it easier to work with std::vector from the SWIG library std_vector.i to send a python list to a C++ SWIG extension. Though in your case where optimization matters, it may not be the optimal.
In your case you can define:
test.i
%module test
%{
#include "test.h"
%}
%include "std_vector.i"
namespace std {
%template(Line) vector < int >;
%template(Array) vector < vector < int> >;
}
void print_array(std::vector< std::vector < int > > myarray);
test.h
#ifndef TEST_H__
#define TEST_H__
#include <stdio.h>
#include <vector>
void print_array(std::vector< std::vector < int > > myarray);
#endif /* TEST_H__ */
test.cpp
#include "test.h"
void print_array(std::vector< std::vector < int > > myarray)
{
for (int i=0; i<2; i++)
for (int j=0; j<2; j++)
printf("[%d][%d] = [%d]\n", i, j, myarray[i][j]);
}
If you run the following python code (I used python 2.6.5), you can see that the C++ function can access the python list:
>>> import test
>>> a = test.Array()
>>> a = [[0, 1], [2, 3]]
>>> test.print_array(a)
[0][0] = [0]
[0][1] = [1]
[1][0] = [2]
[1][1] = [3]
I am using scipy's weave.inline to perform computationally expensive tasks. I have problems returning an one-dimensional array back into the python scope. Weave.inline uses a special argument called "return_val" for the purpose of returning values back into the python scope.
The following example returning an integer value works well:
>>> from scipy.weave import inline
>>> print inline(r'''int N = 10; return_val = N;''')
10
However the following example, which indeed compiles without prompting an error, does not return the array i would expect:
>>> from scipy.weave import inline
>>> code =\
r'''
int* pairs;
int lenght = 0;
for (int i=0;i<N;i++){
lenght += 1;
pairs = (int *)malloc(sizeof(int)*lenght);
pairs[i] = i;
std::cout << pairs[i] << std::endl;
}
return_val = pairs;
'''
>>> N = 5
>>> R = inline(code,['N'])
>>> print "RETURN_VAL:",R
0
1
2
3
4
RETURN_VAL: 1
I need to reallocate the size of the array "pairs" dynamically which is why I can't pass a numpy.array or python list per se.
All you need to do is use the raw python c-api calls, or if you're looking for something a bit more convenient, the built in scipy weave wrappers.
No guarantees about leaks or efficiency, but it should look something a bit like this:
from scipy.weave import inline
code = r'''
py::list ret;
for(int i = 0; i < N; i++) {
py::list item;
for(int j = 0; j < i; j++) {
item.append(j);
}
ret.append(item);
}
return_val = ret;
'''
N = 5
R = inline(code,['N'])
print R
If you absolutely don't know the size of the output array in advance, you must create it in your inline code. I'm pretty sure that your array allocated by using malloc will result in leaked memory since you have no way of controlling when this memory is to be freed.
The solution is to create a numpy array, fill it with your function's results and return it.
import scipy.weave
code = r"""
npy_intp dims[1] = {n};
PyObject* out_array = PyArray_SimpleNew(1, dims, NPY_DOUBLE);
double* data = (double*) ((PyArrayObject*) out_array)->data;
for (int i=0; i<n; ++i) data[i] = i;
return_val = out_array;
Py_XDECREF(out_array);
"""
n = 5
out_array = scipy.weave.inline(code, ["n"])
print "Array:", out_array
I have to do a program that gives all permutations of n numbers {1,2,3..n} using backtracking. I managed to do it in C, and it works very well, here is the code:
int st[25], n=4;
int valid(int k)
{
int i;
for (i = 1; i <= k - 1; i++)
if (st[k] == st[i])
return 0;
return 1;
}
void bktr(int k)
{
int i;
if (k == n + 1)
{
for (i = 1; i <= n; i++)
printf("%d ", st[i]);
printf("\n");
}
else
for (i = 1; i <= n; i++)
{
st[k] = i;
if (valid(k))
bktr(k + 1);
}
}
int main()
{
bktr(1);
return 0;
}
Now I have to write it in Python. Here is what I did:
st=[]
n=4
def bktr(k):
if k==n+1:
for i in range(1,n):
print (st[i])
else:
for i in range(1,n):
st[k]=i
if valid(k):
bktr(k+1)
def valid(k):
for i in range(1,k-1):
if st[k]==st[i]:
return 0
return 1
bktr(1)
I get this error:
list assignment index out of range
at st[k]==st[i].
Python has a "permutations" functions in the itertools module:
import itertools
itertools.permutations([1,2,3])
If you need to write the code yourself (for example if this is homework), here is the issue:
Python lists do not have a predetermined size, so you can't just set e.g. the 10th element to 3. You can only change existing elements or add to the end.
Python lists (and C arrays) also start at 0. This means you have to access the first element with st[0], not st[1].
When you start your program, st has a length of 0; this means you can not assign to st[1], as it is not the end.
If this is confusing, I recommend you use the st.append(element) method instead, which always adds to the end.
If the code is done and works, I recommend you head over to code review stack exchange because there are a lot more things that could be improved.