I am writing Python C-extensions to a library and wish to return data as an Numpy Array. The library has a function that returns data from a sensor into a C structure. I would like to take the data from that structure and return it as a Numpy Array.
The structure definition in the library:
typedef struct rs_extrinsics
{
float rotation[9]; /* column-major 3x3 rotation matrix */
float translation[3]; /* 3 element translation vector, in meters */
} rs_extrinsics;
The function prototype:
void rs_get_device_extrinsics(const rs_device * device, rs_stream from_stream, rs_stream to_stream, rs_extrinsics * extrin, rs_error ** error);
Here is my code that is just trying to return the first value for now:
static PyObject *get_device_extrinsics(PyObject *self, PyObject *args)
{
PyArrayObject *result;
int dimensions = 12;
rs_stream from_stream;
rs_stream to_stream;
rs_extrinsics extrin;
if (!PyArg_ParseTuple(args, "iiffffffffffff", &from_stream, &to_stream, &extrin)) {
return NULL;
}
result = (PyArrayObject *) PyArray_FromDims(1, &dimensions, PyArray_DOUBLE);
if (result == NULL) {
return NULL;
}
rs_get_device_extrinsics(dev, from_stream, to_stream, &extrin, &e);
check_error();
result[0] = extrin.rotation[0];
return PyArray_Return(result);
}
I get the following error on compile:
error: assigning to 'PyArrayObject' (aka 'struct tagPyArrayObject_fields') from incompatible type 'float'
result[0] = extrin.rotation[0];
^ ~~~~~~~~~~~~~~~~~~
PyArrayObject, apart from data, have multiple fields:
typedef struct PyArrayObject {
PyObject_HEAD
char *data;
int nd;
npy_intp *dimensions;
npy_intp *strides;
PyObject *base;
PyArray_Descr *descr;
int flags;
PyObject *weakreflist;
} PyArrayObject;
You should get your data field data from your PyArrayObjects. Like this: result->data[index]; Also you need to cast your data to proper type indicated by result->descr->type character code. Also dims, you are passing to PyArray constructor should be of type npy_intp *, not int. Type of array in your case should be NPY_DOUBLE.
If you are calling your function from python (are you?), you better just passing list object from Python and use PyList C API to manage float sequence.
PyObject*list_of_floats;
PyArg_ParseTuple(args, "iiO", &from_stream, &to_stream, &list_of_floats);
Related
I am using PyArg_Parsetuple to parse a bytearray sent from Python with the Y format specifier.
Y (bytearray) [PyByteArrayObject *]
Requires that the Python object is a bytearray object, without attempting any conversion.
Raises TypeError if the object is not a bytearray object.
In C code I am doing:
static PyObject* py_write(PyObject* self, PyObject* args)
{
PyByteArrayObject* obj;
PyArg_ParseTuple(args, "Y", &obj);
.
.
.
The Python script is sending the following data:
arr = bytearray()
arr.append(0x2)
arr.append(0x0)
How do I loop over the PyByteArrayObject* in C? To print 2 and 0?
Rather than poking implementation details, you should go through the documented API, particularly, accessing the data buffer through PyByteArray_AS_STRING or PyByteArray_AsString rather than through direct struct member access:
char *data = PyByteArray_AS_STRING(bytearray);
Py_ssize_t len = PyByteArray_GET_SIZE(bytearray);
for (Py_ssize_t i = 0; i < len; i++) {
do_whatever_with(data[i]);
}
Note that everything in the public API takes the bytearray as a PyObject *, not a PyByteArrayObject *.
With the help of the comment section, I found the definition for PyByteArrayObject
/* Object layout */
typedef struct {
PyObject_VAR_HEAD
Py_ssize_t ob_alloc; /* How many bytes allocated in ob_bytes */
char *ob_bytes; /* Physical backing buffer */
char *ob_start; /* Logical start inside ob_bytes */
Py_ssize_t ob_exports; /* How many buffer exports */
} PyByteArrayObject;
And the actual code to loop
PyByteArrayObject* obj;
PyArg_ParseTuple(args, "Y", &obj);
Py_ssize_t i = 0;
for (i = 0; i < PyByteArray_GET_SIZE(obj); i++)
printf("%u\n", obj->ob_bytes[i]);
And I got the expected output.
Even better, simply use the Direct API
char* s = PyByteArray_AsString(obj);
int i = 0;
for (i = 0; i < PyByteArray_GET_SIZE(obj); i++)
printf("%u\n", s[i]);
Pardon any syntax errors. I have C++ code that is setup similar to this:
template<typename T>
void addytox(T *x, T *y, int n)
{
for(int i = 0; i < n; ++i) {
x[i] += y[i];
}
return;
}
void my_func(void *x, void *y, int n, int dtype)
{
/* Here is where I am unsure of how to do a simple static cast using
the dtype identifier. I want to avoid long code using a switch or
if/else that would check all possible conditions, for example having
code that looks like this:
if (dtype == 0) {
addytox((int*)x, (int*)y, n);
}
else if (dtype == 1) {
addytox((float*)x, (float*)y, n);
}
else if (dtype == 2) {
addytox((double*)x, (double*)y, n);
}
else {
//Print/log some error...
exit;
}
return;
*/
}
The reason the code it setup like this is because my_func is pointing to a NumPy array which could of any type (int, float32, float64, etc), and I am calling my_func from Python via ctypes. I know the C++ will not know what type the NumPy array is, but I can easily get the data type in Python, and pass that into my_func (in this case, integer dtype). What I'd like to know is if I could use that identifier an be able to call function addytox only once, with the proper type cast.
for example:
addytox((cast_type*)x, (cast_type*)y, n));
Is it possible to do something like this in C++, and if so how would I go about doing it?
Thank you.
Unfortunately as I understand the issue, compile time type determination with templates is not going to help you at run time. You are pretty much stuck with a switch-type mechanism to determine the type you need to invoke at runtime.
HOWEVER, there are some brilliant template metaprogramming techniques that I can share. These help bridge the gap between compile and run-time type determination.
// Generic Declaration. Note the default value.
// For any of the TypeId's not specialized, the compiler will give errors.
template<int TypeId = 0>
struct DispatchAddYToX;
// Specialize for typeId = 0, which let's say is int
template<>
struct DispatchAddYToX<0> // Let's say TypeId 0 = int
{
enum { MyId = 0 };
typedef int MyType;
void dispatch(void* x, void* y, int n, int dType)
{
// Expanded version, for clarity.
if(dType == MyId)
{
// Awriiite! We have the correct type ID.
// ADL should take care of lookup.
addYToX((MyType*)x, (MyType*)y, n);
}
else
{
// If not the correct ID for int, try the next one.
DispatchAddYToX<MyId + 1>::dispatch(x, y, n, dType);
}
}
};
// Specialize for typeId = 1, which let's say is float
template<>
struct DispatchAddYToX<1> // Let's say TypeId 1 = float
{
enum { MyId = 1 };
typedef float MyType;
void dispatch(void* x, void* y, int n, int dType)
{
// Nice and compact version
(dType == MyId) ? addYToX((MyType*)x, (MyType*)y, n) :
DispatchAddYToX<MyId + 1>::dispatch(x, y, n, dType);
}
};
...
// And so on for the rest of the type id's.
// Now for a C-style wrapper.
// Use this with your python hook
void addYToXWrapper(void* x, void*y, int n, int dType)
{
// Defaults to start at 0 (int)
// Will replace the switch statement.
DispatchAddYToX::dispatch(x, y, n, dType);
}
So in the end, it's a fancy switch table which does almost the same thing. The interface is much cleaner though, in my opinion :)
I am moderately experienced in python and C but new to writing python modules as wrappers on C functions. For a project I needed one function named "score" to run much faster than I was able to get in python so I coded it in C and literally just want to be able to call it from python. It takes in a python list of integers and I want the C function to get an array of integers, the length of that array, and then return an integer back to python. Here is my current (working) solution.
static PyObject *module_score(PyObject *self, PyObject *args) {
int i, size, value, *gene;
PyObject *seq, *data;
/* Parse the input tuple */
if (!PyArg_ParseTuple(args, "O", &data))
return NULL;
seq = PySequence_Fast(data, "expected a sequence");
size = PySequence_Size(seq);
gene = (int*) PyMem_Malloc(size * sizeof(int));
for (i = 0; i < size; i++)
gene[i] = PyInt_AsLong(PySequence_Fast_GET_ITEM(seq, i));
/* Call the external C function*/
value = score(gene, size);
PyMem_Free(gene);
/* Build the output tuple */
PyObject *ret = Py_BuildValue("i", value);
return ret;
}
This works but seems to leak memory and at a rate I can't ignore. I made sure that the leak is happening in the shown function by temporarily making the score function just return 0 and still saw the leaking behavior. I had thought that the call to PyMem_Free should take care of the PyMem_Malloc'ed storage but my current guess is that something in this function is getting allocated and retained on each call since the leaking behavior is proportional to the number of calls to this function. Am I not doing the sequence to array conversion correctly or am I possibly returning the ending value inefficiently? Any help is appreciated.
seq is a new Python object so you will need delete that object. You should check if seq is NULL, too.
Something like (untested):
static PyObject *module_score(PyObject *self, PyObject *args) {
int i, size, value, *gene;
long temp;
PyObject *seq, *data;
/* Parse the input tuple */
if (!PyArg_ParseTuple(args, "O", &data))
return NULL;
if (!(seq = PySequence_Fast(data, "expected a sequence")))
return NULL;
size = PySequence_Size(seq);
gene = (int*) PyMem_Malloc(size * sizeof(int));
for (i = 0; i < size; i++) {
temp = PyInt_AsLong(PySequence_Fast_GET_ITEM(seq, i));
if (temp == -1 && PyErr_Occurred()) {
Py_DECREF(seq);
PyErr_SetString(PyExc_ValueError, "an integer value is required");
return NULL;
}
/* Do whatever you need to verify temp will fit in an int */
gene[i] = (int*)temp;
}
/* Call the external C function*/
value = score(gene, size);
PyMem_Free(gene);
Py_DECREF(seq):
/* Build the output tuple */
PyObject *ret = Py_BuildValue("i", value);
return ret;
}
I use python as an interface to operate the image, but when I need to write some custom functions to operate the matrix, I find out that numpy.ndarray is too slow when I iterate. I want to transfer the array to cv::Mat so that I can handle it easily because I used to write C++ code for image processing based on cv::Mat structure.
my test.cpp:
#include <Python.h>
#include <iostream>
using namespace std;
static PyObject *func(PyObject *self, PyObject *args) {
printf("What should I write here?\n");
// How to parse the args to get an np.ndarray?
// cv::Mat m = whateverFunction(theParsedArray);
return Py_BuildValue("s", "Any help?");
}
static PyMethodDef My_methods[] = {
{ "func", (PyCFunction) func, METH_VARARGS, NULL },
{ NULL, NULL, 0, NULL }
};
PyMODINIT_FUNC initydw_cvpy(void) {
PyObject *m=Py_InitModule("ydw_cvpy", My_methods);
if (m == NULL)
return;
}
main.py:
if __name__ == '__main__':
print ydw_cvpy.func()
Result:
What should I write here?
Any help?
It's been a while since I've played with raw C python bindings (I usually use boost::python) but the key is PyArray_FromAny. Some untested sample code would look like
PyObject* source = /* somehow get this from the argument list */
PyArrayObject* contig = (PyArrayObject*)PyArray_FromAny(source,
PyArray_DescrFromType(NPY_UINT8),
2, 2, NPY_ARRAY_CARRAY, NULL);
if (contig == nullptr) {
// Throw an exception
return;
}
cv::Mat mat(PyArray_DIM(contig, 0), PyArray_DIM(contig, 1), CV_8UC1,
PyArray_DATA(contig));
/* Use mat here */
PyDECREF(contig); // You can't use mat after this line
Note that this assumes that you have a an CV_8UC1 array. If you have a CV_8UC3 you need to ask for a 3 dimensional array
PyArrayObject* contig = (PyArrayObject*)PyArray_FromAny(source,
PyArray_DescrFromType(NPY_UINT8),
3, 3, NPY_ARRAY_CARRAY, NULL);
assert(contig && PyArray_DIM(contig, 2) == 3);
If you need to handle any random type of array, you probably also want to look at this answer.
I am trying to output an array of values from a C function wrapped using SWIG for Python. The way I am trying to do is using the following typemap.
Pseudo code:
int oldmain() {
float *output = {0,1};
return output;
}
Typemap:
%typemap(out) float* {
int i;
$result = PyList_New($1_dim0);
for (i = 0; i < $1_dim0; i++) {
PyObject *o = PyFloat_FromDouble((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
My code compiles well, but it hangs when I run access this function (with no more ways to debug it).
Any suggestions on where I am going wrong?
Thanks.
The easiest way to allow the length to vary is to add another output parameter that tells you the size of the array too:
%module test
%include <stdint.i>
%typemap(in,numinputs=0,noblock=1) size_t *len {
size_t templen;
$1 = &templen;
}
%typemap(out) float* oldmain {
int i;
$result = PyList_New(templen);
for (i = 0; i < templen; i++) {
PyObject *o = PyFloat_FromDouble((double)$1[i]);
PyList_SetItem($result,i,o);
}
}
%inline %{
float *oldmain(size_t *len) {
static float output[] = {0.f, 1.f, 2, 3, 4};
*len = sizeof output/sizeof *output;
return output;
}
%}
This is modified from this answer to add size_t *len which can be used to return the length of the array at run time. The typemap completely hides that output from the Python wrapper though and instead uses it in the %typemap(out) instead of a fixed size to control the length of the returned list.
This should get you going:
/* example.c */
float * oldmain() {
static float output[] = {0.,1.};
return output;
}
You are returning a pointer here, and swig has no idea about the size of it. Plain $1_dim0 would not work, so you would have to hard code or do some other magic. Something like this:
/* example.i */
%module example
%{
/* Put header files here or function declarations like below */
extern float * oldmain();
%}
%typemap(out) float* oldmain {
int i;
//$1, $1_dim0, $1_dim1
$result = PyList_New(2);
for (i = 0; i < 2; i++) {
PyObject *o = PyFloat_FromDouble((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
%include "example.c"
Then in python you should get:
>> import example
>> example.oldmain()
[0.0, 1.0]
When adding typemaps you may find -debug-tmsearch very handy, i.e.
swig -python -debug-tmsearch example.i
Should clearly indicate that your typemap is used when looking for a suitable 'out' typemap for float *oldmain. Also if you just like to access c global variable array you can do the same trick using typemap for varout instead of just out.