I have a C++ function returning a std::vector and I want to use it in python, so I'm using the C numpy api:
static PyObject *
py_integrate(PyObject *self, PyObject *args){
...
std::vector<double> integral;
cpp_function(integral); // This changes integral
npy_intp size = {integral.size()};
PyObject *out = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, &(integral[0]));
return out;
}
Here's how I call it from python:
import matplotlib.pyplot as plt
a = py_integrate(parameters)
print a
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(a)
print a
What happens is: The first print is ok, the values are correct. But when I plot a they are not; in the second print I see very strange values like 1E-308 1E-308 ... or 0 0 0 ... as an uninitialized memory. I don't understand why the first print is ok.
Partial solution (not working):
static void DeleteVector(void *ptr)
{
std::cout << "Delete" << std::endl;
vector * v = static_cast<std::vector<double> * >(ptr);
delete v;
return;
}
static PyObject *
cppfunction(PyObject *self, PyObject *args)
{
std::vector<double> *vector = new std::vector<double>();
vector->push_back(1.);
PyObject *py_integral = PyCObject_FromVoidPtr(vector, DeleteVector);
npy_intp size = {vector->size()};
PyArrayObject *out;
((PyArrayObject*) out)->base = py_integral;
return (PyObject*)(out);
}
Your std::vector object appears to be local to that function. PyArray_SimpleNewFromData does not make a copy of the data you pass it. It just keeps a pointer. So once your py_integrate function returns, the vector is deallocated. The print works the first time because nothing has written over the freed memory yet, but by the time you get to the next print, something else has used that memory, causing the values to be different.
You need to make a NumPy array that owns its own storage space and then copy the data into it.
Alternatively, allocate your vector on the heap. Then store a pointer to it in a CObject. Provide a destructor that deletes the vector. Then, take a look at the C-level PyArrayObject type. It has a PyObject * member called base. Store your CObject there. Then when the NumPy array is garbage collected, the reference count on this base object will be decremented, and assuming you haven't taken a copy of it elsewhere, your vector will be deleted thanks to the destructor you provided.
Fixer-upper
You forgot to actually create the PyArray. Try this:
(You didn't post DeleteVector, so I can only hope that it's right)
std::vector<double> *vector = new std::vector<double>();
vector->push_back(1.);
PyObject *py_integral = PyCObject_FromVoidPtr(vector, DeleteVector);
npy_intp size = {vector->size()};
PyObject *out = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, &((*vector)[0]));
((PyArrayObject*) out)->base = py_integral;
return out;
Note: I'm not a C++ programmer, so I can only assume that &((*vector)[0]) works as intended with a pointer to a vector. I do know that the vector reallocate its storage area if you grow it, so don't increase its size after getting that pointer or it won't be valid anymore.
You will need to make a copy of the vector, since the vector will go out of scope and the memory will no longer be usable by the time you need it in Python (as stated by kwatford).
One way to make the Numpy array you need (by copying the data) is:
PyObject *out = nullptr;
std::vector<double> *vector = new std::vector<double>();
vector->push_back(1.);
npy_intp size = {vector.size()};
out = PyArray_SimpleNew(1, &size, NPY_DOUBLE);
memcpy(PyArray_DATA((PyArrayObject *) out), vector.data(), vector.size());
Related
I have a C shared library with a function that takes one argument.
This argument is a pointer to a structure with two fields.
typedef struct
{
uint8_t *p_data; // Pointer to a fixed lenth buffer (100 bytes)
uint16_t len; // number of valid bytes in the buffer (range 1-100)
} data_t;
I need to setup a buffer of 100 bytes in my Python 3 script (I am using 3.7.2 / 3.7.3),
load the library and call this function.
int
fn_convert_buffer(data_t *data_p)
{
...
}
My Python 3 ctypes call attempt hits incompatible types.
import ctypes as ct
# load the library, etc...
# lib_cdll = ct.CDLL(mySharedLib)
def c_py_fn_convert_buffer(b_p):
global lib_cdll
val = lib_cdll.fn_convert_buffer(ct.byref(b_p))
return int(val)
data_a = bytearray(100)
# Initialize the buffer with data.
uint8_p = ct.c_uint8 * len(data_a)
class BufferStruct_t (ct.Structure):
_pack_ = 1
_fields_ = [
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
("len", ct.c_uint16)
]
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
# TypeError: incompatible types, c_ubyte_Array_100 instance
# instead of LP_c_ubyte_Array_100 instance
# Call C function in shared-library: int fn_convert_buffer(data_t *data_p);
z = c_py_fn_convert_buffer(data_buf)
I need help in understanding what I've missed in the BufferStruct_t definition above. The from_buffer is supposed to get a pointer to the buffer but it seems to get c_ubyte_ARRAY_100.
A byref() on that does not work either
data_buf = BufferStruct_t(ct.byref(uint8_p.from_buffer(data_a)), ct.c_uint16(8))
# TypeError: expected LP_c_ubyte_Array_100 instance, got CArgObject
To test the basics of my flow, I made a sample case that will send the buffer and length parameters individually.
def c_py_fn_convert_data(d_p,l):
global lib_cdll
val = lib_cdll.fn_convert_data(ct.byref(d_p),ct.c_uint32(l))
return int(val)
test_a = ct.c_uint8 * len(data_a)
# Call C function in shared-library:
# int fn_convert_data(uint8_t *data_p, uint32_t length);
z = c_py_fn_convert_data(test_a.from_buffer(data_a), 8)
This simplified case works.
How do I get about building a Python 3 object that carries a reference to a buffer that the shared-library function expects?
Update with two variations that worked.
Update 1 Tried a cast based on something I read later (I don't cast lightly :-))
Changed,
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
to a pointer that is cast to refer an Array of specific length,
data_buf = BufferStruct_t(cast(uint8_p.from_buffer(data_a),
ct.POINTER(ct.c_uint8 * len(data_a))),
ct.c_uint16(8))
Update 2 based on Mark's answer.
Changed _field_ from,
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
to a simple-pointer form,
("p_data", ct.POINTER(ct.c_uint8)),
Both variations worked.
I am however curious to know which of these two ways is more safe/correct ctypes handling.
Is it better to cast to the Array form? or,
Is it better to use simple pointers and rely on the length sent independently?
Your structure definition declared a pointer to an array, not a simple pointer as in the C structure. Here's a working example with a simple implementation of the DLL where the function sums the data:
test.c
#include <stdint.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
typedef struct {
uint8_t *p_data;
uint16_t len;
} data_t;
API int fn_convert_buffer(data_t *data_p)
{
int i;
int sum = 0;
for(i = 0; i < data_p->len; ++i)
sum += data_p->p_data[i];
return sum;
}
test.py
import ctypes as ct
class BufferStruct_t(ct.Structure):
_pack_ = 1
_fields_ = [("p_data", ct.POINTER(ct.c_uint8)), # just a pointer
("len", ct.c_uint16)]
# Helper to initialize the data
def __init__(self,data):
self.p_data = (ct.c_uint8 * len(data))(*data)
self.len = len(data)
dll = ct.CDLL('test')
dll.fn_convert_buffer.argtypes = ct.POINTER(BufferStruct_t),
dll.fn_convert_buffer.restype = ct.c_int
data_buf = BufferStruct_t([1,2,3,4,5])
print(dll.fn_convert_buffer(data_buf))
Output:
15
MNIST is the hello world of machine learning and I've practiced it with TensorFlow and with pure python and numpy.
For more practice I am trying to write it in C on my own with only the standard library because I am relatively new to C and it's a great way to learn.
It's taken three weeks, and a lot of SEGFAULTS but I get 81% accuracy. Not very good but it's for learning.
The most troubling stuff was of course malloc/free for the data in the matrix struct as below:
typedef struct matrix{
int rows, cols;
float *data;
} matrix;
The forward and backward passes have things like:
1) matrix dot product
2) matrix add
3) matrix subtract
4) activation function (sigmoid in this case)
To avoid memory leaks I pass in three structs like so:
void matrix_add(matrix *a, matrix *b, matrix *res);
If res requires a dimensions change from a previous layer, then I free it and do a new malloc like so:
void zero_out_data(matrix *res, int rows, int cols)
{
if (res->rows != rows || res->cols != cols)
{
if ((res->rows*res->cols) != (rows*cols))
{
free(res->data);
res->data = NULL;
free(res);
res = NULL;
res = malloc(sizeof(matrix));
// make_matrix will calloc the data based on rows*cols
// any other init stuff that could be needed
make_matrix(res, rows, cols);
}
res->rows = rows;
res->cols = cols;
}
else {
res->rows = rows;
res->cols = cols;
for (int i =0; i < (rows*cols); i++)
{
res->data[i] = 0.0;
}
}
}
Then I can use that like so:
void sigmoid(matrix *z, matrix *res)
{
zero_out_data(res, z->rows, z->cols);
for (int i = 0; i < (z->rows*z->cols); i++)
{
res->data[i] = 1.0/(1.0+exp(-z->data[i]));
}
}
This gets very messy because a single forward pass has the following:
/* forward pass */
for (int k=0; k < (network->num_layers-1); k++)
{
matrix_dot(network->weights[k], activation, dot);
matrix_add(dot, network->biases[k], zs[k]);
sigmoid(zs[k], activation);
sigmoid(zs[k], activations[k+1]);
}
/* end forward pass */
As you can imagine the backprop gets alot messier. I have to pre-create 8 different matrices, plus many more of those pointers to pointers of matrices like the activations and zs above, for the gradient descent.
What I would like to be able to do is return a matrix from a function like matrix_dot so that I can do:
sigmoid(matrix_add(matrix_dot(network->weights[k], activation), network->biases[k]));
That's kind of in the style of python/numpy.
Of course I can't return a local variable from a function because it's taken off the stack once the function returns.
If I return a pointer, then the above style will cause sever memory leaks.
Please note: I am not trying to write my own library/framework. I am simply trying to learn neural networks and coding in C. I have been a python developer for 7 years or so, and my C skills need improvement.
Memory leak in void zero_out_data(matrix *res, int rows, int cols)
matrix *res malloc out of the function and pass to zero_out_data. In zero_out_data, res is free and malloc again. If you want to change pointer res's value, then you need parameter like matrix **res.
If you want zero out data, no need malloc new matrix, just malloc the data part. I think your make_matrix function can malloc memory for data.
void zero_out_data(matrix *res, int rows, int col) {
if (res->data == NULL) {
make_matrix(res, rows, cols);
} else if (res->rows != rows || res->cols != cols) {
if ((res->rows*res->cols) != (rows*cols))
{
free(res->data);
res->data = NULL;
make_matrix(res, rows, cols);
}
}
res->rows = rows;
res->cols = cols;
for (int i =0; i < (rows*cols); i++)
{
res->data[i] = 0.0;
}
}
How to implement this: sigmoid(matrix_add(matrix_dot(network->weights[k], activation), network->biases[k])); ?
You can use static or global variables to implement what you want. This will not be thread safe and reentrant. Examples in below:
matrix *matrix_dot(matrix *in_a, matrix *in_b)
{
static matrix res = {0, 0, NULL}; // static variable
// calculate the res's cols and rows number
zero_out_data(&res, res_cols, res_rows); // malloc new data
// do some math.
return &res;
}
// matrix_add will be just like matrix_dot
// I was wrong about sigmod no need new matrix. sigmod can also do it like matrix_dot.
You can use global variable replace static variable.
If you want thread-safe or reentrant, then just use local variable, then you can do it like this.
matrix *matrix_dot(matrix *in_a, matrix *in_b, matrix *res)
{
zero_out_data(res, xxx, xxx);
// do some math
return res;
}
// matrix_add will be the same.
// define local variables.
matrix add_res, dot_res, sig_res;
add_res->data = NULL;
dot_res->data = NULL;
sig_res->data = NULL;
sigmod(matrix_add(matrix_dot(network->weights[k], activation, &dot_res), network->biases[k], &add_res), &sig_res)
// Now remember to free data in matrix
free(add_res->data);
free(dot_res->data);
free(sig_res->data);
I've written an extension in C++ for Python, and I'm currently debugging it.
The extension takes 3 numpy matrices and produces 2 as a result. To the inner C++ function that does the actualy calculation I pass 3 float C arrays (just flattened and converted from input numpy arrays), and return a C float array of arrays. Everything works as intended but ONLY if I print this output array of arrays before returning it.
What the hell is going on in here?
float** gradient(float* inputs, float* kernels, float* grads, npy_intp* input_dims, npy_intp* kernels_dims, npy_intp* output_dims){
float* g_inputs = new float[batch*h*w*ch_in];
for (int i = 0; i < batch*h*w*ch_in; i++) g_inputs[i] = 0;
float* g_kernels = new float[size*ch_out];
for (int i = 0; i < size*ch_out; i++) g_kernels[i] = 0;
float* ret[2] = {{g_inputs}, {g_kernels}};
std::cout<<ret<<std::endl; //<---without this it doesn't work
return ret;
}
I've omitted irrelevant code for clarity.
You are returning a pointer to an object with automatic lifetime. In other words, your function returns a dangling pointer, which is Undefined Behaviour.
Although aerostatic lizards are an uncommon result of UB, anything can happen and the symptom you observe, unlike the lizards, is common.
I have a C++ function computing a large tensor which I would like to return to Python as a NumPy array via pybind11.
From the documentation of pybind11, it seems like using STL unique_ptr is desirable.
In the following example, the commented out version works, whereas the given one compiles but fails at runtime ("Unable to convert function return value to a Python type!").
Why is the smartpointer version failing? What is the canonical way to create and return a NumPy array?
PS: Due to program structure and size of the array, it is desirable to not copy memory but create the array from a given pointer. Memory ownership should be taken by Python.
typedef typename py::array_t<double, py::array::c_style | py::array::forcecast> py_cdarray_t;
// py_cd_array_t _test()
std::unique_ptr<py_cdarray_t> _test()
{
double * memory = new double[3]; memory[0] = 11; memory[1] = 12; memory[2] = 13;
py::buffer_info bufinfo (
memory, // pointer to memory buffer
sizeof(double), // size of underlying scalar type
py::format_descriptor<double>::format(), // python struct-style format descriptor
1, // number of dimensions
{ 3 }, // buffer dimensions
{ sizeof(double) } // strides (in bytes) for each index
);
//return py_cdarray_t(bufinfo);
return std::unique_ptr<py_cdarray_t>( new py_cdarray_t(bufinfo) );
}
A few comments (then a working implementation).
pybind11's C++ object wrappers around Python types (like pybind11::object, pybind11::list, and, in this case, pybind11::array_t<T>) are really just wrappers around an underlying Python object pointer. In this respect there are already taking on the role of a shared pointer wrapper, and so there's no point in wrapping that in a unique_ptr: returning the py::array_t<T> object directly is already essentially just returning a glorified pointer.
pybind11::array_t can be constructed directly from a data pointer, so you can skip the py::buffer_info intermediate step and just give the shape and strides directly to the pybind11::array_t constructor. A numpy array constructed this way won't own its own data, it'll just reference it (that is, the numpy owndata flag will be set to false).
Memory ownership can be tied to the life of a Python object, but you're still on the hook for doing the deallocation properly. Pybind11 provides a py::capsule class to help you do exactly this. What you want to do is make the numpy array depend on this capsule as its parent class by specifying it as the base argument to array_t. That will make the numpy array reference it, keeping it alive as long as the array itself is alive, and invoke the cleanup function when it is no longer referenced.
The c_style flag in the older (pre-2.2) releases only had an effect on new arrays, i.e. when not passing a value pointer. That was fixed in the 2.2 release to also affect the automatic strides if you specify only shapes but not strides. It has no effect at all if you specify the strides directly yourself (as I do in the example below).
So, putting the pieces together, this code is a complete pybind11 module that demonstrates how you can accomplish what you're looking for (and includes some C++ output to demonstrate that is indeed working correctly):
#include <iostream>
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
namespace py = pybind11;
PYBIND11_PLUGIN(numpywrap) {
py::module m("numpywrap");
m.def("f", []() {
// Allocate and initialize some data; make this big so
// we can see the impact on the process memory use:
constexpr size_t size = 100*1000*1000;
double *foo = new double[size];
for (size_t i = 0; i < size; i++) {
foo[i] = (double) i;
}
// Create a Python object that will free the allocated
// memory when destroyed:
py::capsule free_when_done(foo, [](void *f) {
double *foo = reinterpret_cast<double *>(f);
std::cerr << "Element [0] = " << foo[0] << "\n";
std::cerr << "freeing memory # " << f << "\n";
delete[] foo;
});
return py::array_t<double>(
{100, 1000, 1000}, // shape
{1000*1000*8, 1000*8, 8}, // C-style contiguous strides for double
foo, // the data pointer
free_when_done); // numpy array references this parent
});
return m.ptr();
}
Compiling that and invoking it from Python shows it working:
>>> import numpywrap
>>> z = numpywrap.f()
>>> # the python process is now taking up a bit more than 800MB memory
>>> z[1,1,1]
1001001.0
>>> z[0,0,100]
100.0
>>> z[99,999,999]
99999999.0
>>> z[0,0,0] = 3.141592
>>> del z
Element [0] = 3.14159
freeing memory # 0x7fd769f12010
>>> # python process memory size has dropped back down
I recommend using ndarray. A foundational principle is that the underlying data is never copied unless explicitly requested (or you quickly end up with huge inefficiencies). Below is an example of it in use, but there are other features I haven't shown, including conversion to Eigen arrays (ndarray::asEigen(array)), which makes it pretty powerful.
Header:
#ifndef MYTENSORCODE_H
#define MYTENSORCODE_H
#include "ndarray_fwd.h"
namespace myTensorNamespace {
ndarray::Array<double, 2, 1> myTensorFunction(int param1, double param2);
} // namespace myTensorNamespace
#endif // include guard
Lib:
#include "ndarray.h"
#include "myTensorCode.h"
namespace myTensorNamespace {
ndarray::Array<double, 2, 1> myTensorFunction(int param1, double param2) {
std::size_t const size = calculateSize();
ndarray::Array<double, 2, 1> array = ndarray::allocate(size, size);
array.deep() = 0; // initialise
for (std::size_t ii = 0; ii < size; ++ii) {
array[ii][ndarray::view(ii, ii + 1)] = 1.0;
}
return array;
}
} // namespace myTensorNamespace
Wrapper:
#include "pybind11/pybind11.h"
#include "ndarray.h"
#include "ndarray/pybind11.h"
#include "myTensorCode.h"
namespace py = pybind11;
using namespace pybind11::literals;
namespace myTensorNamespace {
namespace {
PYBIND11_MODULE(myTensorModule, mod) {
mod.def("myTensorFunction", &myTensorFunction, "param1"_a, "param2"_a);
}
} // anonymous namespace
} // namespace myTensorNamespace
I have the following python code:
r = range(1,10)
r_squared = []
for item in r:
print item
r_squared.append(item*item)
How would I convert this code to C? Is there something like a mutable array in C or how would I do the equivalent of the python append?
simple array in c.Arrays in the C are Homogenous
int arr[10];
int i = 0;
for(i=0;i<sizeof(arr);i++)
{
arr[i] = i; // Initializing each element seperately
}
Try using vectors in C go through this link
/ vector-usage.c
#include <stdio.h>
#include "vector.h"
int main() {
// declare and initialize a new vector
Vector vector;
vector_init(&vector);
// fill it up with 150 arbitrary values
// this should expand capacity up to 200
int i;
for (i = 200; i > -50; i--) {
vector_append(&vector, i);
}
// set a value at an arbitrary index
// this will expand and zero-fill the vector to fit
vector_set(&vector, 4452, 21312984);
// print out an arbitrary value in the vector
printf("Heres the value at 27: %d\n", vector_get(&vector, 27));
// we're all done playing with our vector,
// so free its underlying data array
vector_free(&vector);
}
Arrays in C are mutable by default, in that you can write a[i] = 3, just like Python lists.
However, they're fixed-length, unlike Python lists.
For your problem, that should actually be fine. You know the final size you want; just create an array of that size, and assign to the members.
But of course there are problems for which you do need append.
Writing a simple library for appendable arrays (just like Python lists) is a pretty good learning project for C. You can also find plenty of ready-made implementations if that's what you want, but not in the standard library.
The key is to not use a stack array, but rather memory allocated on the heap with malloc. Keep track of the pointer to that memory, the capacity, and the used size. When the used size reaches the capacity, multiply it by some number (play with different numbers to get an idea of how they affect performance), then realloc. That's just about all there is to it. (And if you look at the CPython source for the list type, that's basically the same thing it's doing.)
Here's an example. You'll want to add some error handling (malloc and realloc can return NULL) and of course the rest of the API beyond append (especially a delete function, which will call free on the allocated memory), but this should be enough to show you the idea:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct {
int *i;
size_t len;
size_t capacity;
} IntArray;
IntArray int_array_make() {
IntArray a = {
.i = malloc(10 * sizeof(int)),
.len = 0,
.capacity = 10
};
return a;
}
void int_array_append(IntArray *a, int value) {
if (a->len+1 == a->capacity) {
size_t new_capacity = (int)(a->capacity * 1.6);
a->i = realloc(a->i, new_capacity * sizeof(int));
a->capacity = new_capacity;
}
a->i[a->len++] = value;
}
int main(int argc, char *argv[]) {
IntArray a = int_array_make();
for (int i = 0; i != 50; i++)
int_array_append(&a, i);
for (int i = 0; i != a.len; ++i)
printf("%d ", a.i[i]);
printf("\n");
}
c doesnt have any way of dynamically increasing the size of the array like in python. arrays here are of fixed length
if you know the size of the array that you will be using, u can use this kind of declaration, like this
int arr[10];
or if you would want to add memery on the fly (in runtime), use malloc call along with structure (linked lists)