Pardon any syntax errors. I have C++ code that is setup similar to this:
template<typename T>
void addytox(T *x, T *y, int n)
{
for(int i = 0; i < n; ++i) {
x[i] += y[i];
}
return;
}
void my_func(void *x, void *y, int n, int dtype)
{
/* Here is where I am unsure of how to do a simple static cast using
the dtype identifier. I want to avoid long code using a switch or
if/else that would check all possible conditions, for example having
code that looks like this:
if (dtype == 0) {
addytox((int*)x, (int*)y, n);
}
else if (dtype == 1) {
addytox((float*)x, (float*)y, n);
}
else if (dtype == 2) {
addytox((double*)x, (double*)y, n);
}
else {
//Print/log some error...
exit;
}
return;
*/
}
The reason the code it setup like this is because my_func is pointing to a NumPy array which could of any type (int, float32, float64, etc), and I am calling my_func from Python via ctypes. I know the C++ will not know what type the NumPy array is, but I can easily get the data type in Python, and pass that into my_func (in this case, integer dtype). What I'd like to know is if I could use that identifier an be able to call function addytox only once, with the proper type cast.
for example:
addytox((cast_type*)x, (cast_type*)y, n));
Is it possible to do something like this in C++, and if so how would I go about doing it?
Thank you.
Unfortunately as I understand the issue, compile time type determination with templates is not going to help you at run time. You are pretty much stuck with a switch-type mechanism to determine the type you need to invoke at runtime.
HOWEVER, there are some brilliant template metaprogramming techniques that I can share. These help bridge the gap between compile and run-time type determination.
// Generic Declaration. Note the default value.
// For any of the TypeId's not specialized, the compiler will give errors.
template<int TypeId = 0>
struct DispatchAddYToX;
// Specialize for typeId = 0, which let's say is int
template<>
struct DispatchAddYToX<0> // Let's say TypeId 0 = int
{
enum { MyId = 0 };
typedef int MyType;
void dispatch(void* x, void* y, int n, int dType)
{
// Expanded version, for clarity.
if(dType == MyId)
{
// Awriiite! We have the correct type ID.
// ADL should take care of lookup.
addYToX((MyType*)x, (MyType*)y, n);
}
else
{
// If not the correct ID for int, try the next one.
DispatchAddYToX<MyId + 1>::dispatch(x, y, n, dType);
}
}
};
// Specialize for typeId = 1, which let's say is float
template<>
struct DispatchAddYToX<1> // Let's say TypeId 1 = float
{
enum { MyId = 1 };
typedef float MyType;
void dispatch(void* x, void* y, int n, int dType)
{
// Nice and compact version
(dType == MyId) ? addYToX((MyType*)x, (MyType*)y, n) :
DispatchAddYToX<MyId + 1>::dispatch(x, y, n, dType);
}
};
...
// And so on for the rest of the type id's.
// Now for a C-style wrapper.
// Use this with your python hook
void addYToXWrapper(void* x, void*y, int n, int dType)
{
// Defaults to start at 0 (int)
// Will replace the switch statement.
DispatchAddYToX::dispatch(x, y, n, dType);
}
So in the end, it's a fancy switch table which does almost the same thing. The interface is much cleaner though, in my opinion :)
Related
I am writing Python C-extensions to a library and wish to return data as an Numpy Array. The library has a function that returns data from a sensor into a C structure. I would like to take the data from that structure and return it as a Numpy Array.
The structure definition in the library:
typedef struct rs_extrinsics
{
float rotation[9]; /* column-major 3x3 rotation matrix */
float translation[3]; /* 3 element translation vector, in meters */
} rs_extrinsics;
The function prototype:
void rs_get_device_extrinsics(const rs_device * device, rs_stream from_stream, rs_stream to_stream, rs_extrinsics * extrin, rs_error ** error);
Here is my code that is just trying to return the first value for now:
static PyObject *get_device_extrinsics(PyObject *self, PyObject *args)
{
PyArrayObject *result;
int dimensions = 12;
rs_stream from_stream;
rs_stream to_stream;
rs_extrinsics extrin;
if (!PyArg_ParseTuple(args, "iiffffffffffff", &from_stream, &to_stream, &extrin)) {
return NULL;
}
result = (PyArrayObject *) PyArray_FromDims(1, &dimensions, PyArray_DOUBLE);
if (result == NULL) {
return NULL;
}
rs_get_device_extrinsics(dev, from_stream, to_stream, &extrin, &e);
check_error();
result[0] = extrin.rotation[0];
return PyArray_Return(result);
}
I get the following error on compile:
error: assigning to 'PyArrayObject' (aka 'struct tagPyArrayObject_fields') from incompatible type 'float'
result[0] = extrin.rotation[0];
^ ~~~~~~~~~~~~~~~~~~
PyArrayObject, apart from data, have multiple fields:
typedef struct PyArrayObject {
PyObject_HEAD
char *data;
int nd;
npy_intp *dimensions;
npy_intp *strides;
PyObject *base;
PyArray_Descr *descr;
int flags;
PyObject *weakreflist;
} PyArrayObject;
You should get your data field data from your PyArrayObjects. Like this: result->data[index]; Also you need to cast your data to proper type indicated by result->descr->type character code. Also dims, you are passing to PyArray constructor should be of type npy_intp *, not int. Type of array in your case should be NPY_DOUBLE.
If you are calling your function from python (are you?), you better just passing list object from Python and use PyList C API to manage float sequence.
PyObject*list_of_floats;
PyArg_ParseTuple(args, "iiO", &from_stream, &to_stream, &list_of_floats);
Following these answers, I've currently defined a Rust 1.0 function as follows, in order to be callable from Python using ctypes:
use std::vec;
extern crate libc;
use libc::{c_int, c_float, size_t};
use std::slice;
#[no_mangle]
pub extern fn convert_vec(input_lon: *const c_float,
lon_size: size_t,
input_lat: *const c_float,
lat_size: size_t) -> Vec<(i32, i32)> {
let input_lon = unsafe {
slice::from_raw_parts(input_lon, lon_size as usize)
};
let input_lat = unsafe {
slice::from_raw_parts(input_lat, lat_size as usize)
};
let combined: Vec<(i32, i32)> = input_lon
.iter()
.zip(input_lat.iter())
.map(|each| convert(*each.0, *each.1))
.collect();
return combined
}
And I'm setting up the Python part like so:
from ctypes import *
class Int32_2(Structure):
_fields_ = [("array", c_int32 * 2)]
rust_bng_vec = lib.convert_vec_py
rust_bng_vec.argtypes = [POINTER(c_float), c_size_t,
POINTER(c_float), c_size_t]
rust_bng_vec.restype = POINTER(Int32_2)
This seems to be OK, but I'm:
Not sure how to transform combined (a Vec<(i32, i32)>) to a C-compatible structure, so it can be returned to my Python script.
Not sure whether I should be returning a reference (return &combined?) and how I would have to annotate the function with the appropriate lifetime specifier if I did
The most important thing to note is that there is no such thing as a tuple in C. C is the lingua franca of library interoperability, and you will be required to restrict yourself to abilities of this language. It doesn't matter if you are talking between Rust and another high-level language; you have to speak C.
There may not be tuples in C, but there are structs. A two-element tuple is just a struct with two members!
Let's start with the C code that we would write:
#include <stdio.h>
#include <stdint.h>
typedef struct {
uint32_t a;
uint32_t b;
} tuple_t;
typedef struct {
void *data;
size_t len;
} array_t;
extern array_t convert_vec(array_t lat, array_t lon);
int main() {
uint32_t lats[3] = {0, 1, 2};
uint32_t lons[3] = {9, 8, 7};
array_t lat = { .data = lats, .len = 3 };
array_t lon = { .data = lons, .len = 3 };
array_t fixed = convert_vec(lat, lon);
tuple_t *real = fixed.data;
for (int i = 0; i < fixed.len; i++) {
printf("%d, %d\n", real[i].a, real[i].b);
}
return 0;
}
We've defined two structs — one to represent our tuple, and another to represent an array, as we will be passing those back and forth a bit.
We will follow this up by defining the exact same structs in Rust and define them to have the exact same members (types, ordering, names). Importantly, we use #[repr(C)] to let the Rust compiler know to not do anything funky with reordering the data.
extern crate libc;
use std::slice;
use std::mem;
#[repr(C)]
pub struct Tuple {
a: libc::uint32_t,
b: libc::uint32_t,
}
#[repr(C)]
pub struct Array {
data: *const libc::c_void,
len: libc::size_t,
}
impl Array {
unsafe fn as_u32_slice(&self) -> &[u32] {
assert!(!self.data.is_null());
slice::from_raw_parts(self.data as *const u32, self.len as usize)
}
fn from_vec<T>(mut vec: Vec<T>) -> Array {
// Important to make length and capacity match
// A better solution is to track both length and capacity
vec.shrink_to_fit();
let array = Array { data: vec.as_ptr() as *const libc::c_void, len: vec.len() as libc::size_t };
// Whee! Leak the memory, and now the raw pointer (and
// eventually C) is the owner.
mem::forget(vec);
array
}
}
#[no_mangle]
pub extern fn convert_vec(lon: Array, lat: Array) -> Array {
let lon = unsafe { lon.as_u32_slice() };
let lat = unsafe { lat.as_u32_slice() };
let vec =
lat.iter().zip(lon.iter())
.map(|(&lat, &lon)| Tuple { a: lat, b: lon })
.collect();
Array::from_vec(vec)
}
We must never accept or return non-repr(C) types across the FFI boundary, so we pass across our Array. Note that there's a good amount of unsafe code, as we have to convert an unknown pointer to data (c_void) to a specific type. That's the price of being generic in C world.
Let's turn our eye to Python now. Basically, we just have to mimic what the C code did:
import ctypes
class FFITuple(ctypes.Structure):
_fields_ = [("a", ctypes.c_uint32),
("b", ctypes.c_uint32)]
class FFIArray(ctypes.Structure):
_fields_ = [("data", ctypes.c_void_p),
("len", ctypes.c_size_t)]
# Allow implicit conversions from a sequence of 32-bit unsigned
# integers.
#classmethod
def from_param(cls, seq):
return cls(seq)
# Wrap sequence of values. You can specify another type besides a
# 32-bit unsigned integer.
def __init__(self, seq, data_type = ctypes.c_uint32):
array_type = data_type * len(seq)
raw_seq = array_type(*seq)
self.data = ctypes.cast(raw_seq, ctypes.c_void_p)
self.len = len(seq)
# A conversion function that cleans up the result value to make it
# nicer to consume.
def void_array_to_tuple_list(array, _func, _args):
tuple_array = ctypes.cast(array.data, ctypes.POINTER(FFITuple))
return [tuple_array[i] for i in range(0, array.len)]
lib = ctypes.cdll.LoadLibrary("./target/debug/libtupleffi.dylib")
lib.convert_vec.argtypes = (FFIArray, FFIArray)
lib.convert_vec.restype = FFIArray
lib.convert_vec.errcheck = void_array_to_tuple_list
for tupl in lib.convert_vec([1,2,3], [9,8,7]):
print tupl.a, tupl.b
Forgive my rudimentary Python. I'm sure an experienced Pythonista could make this look a lot prettier! Thanks to #eryksun for some nice advice on how to make the consumer side of calling the method much nicer.
A word about ownership and memory leaks
In this example code, we've leaked the memory allocated by the Vec. Theoretically, the FFI code now owns the memory, but realistically, it can't do anything useful with it. To have a fully correct example, you'd need to add another method that would accept the pointer back from the callee, transform it back into a Vec, then allow Rust to drop the value. This is the only safe way, as Rust is almost guaranteed to use a different memory allocator than the one your FFI language is using.
Not sure whether I should be returning a reference and how I would have to annotate the function with the appropriate lifetime specifier if I did
No, you don't want to (read: can't) return a reference. If you could, then the ownership of the item would end with the function call, and the reference would point to nothing. This is why we need to do the two-step dance with mem::forget and returning a raw pointer.
I am moderately experienced in python and C but new to writing python modules as wrappers on C functions. For a project I needed one function named "score" to run much faster than I was able to get in python so I coded it in C and literally just want to be able to call it from python. It takes in a python list of integers and I want the C function to get an array of integers, the length of that array, and then return an integer back to python. Here is my current (working) solution.
static PyObject *module_score(PyObject *self, PyObject *args) {
int i, size, value, *gene;
PyObject *seq, *data;
/* Parse the input tuple */
if (!PyArg_ParseTuple(args, "O", &data))
return NULL;
seq = PySequence_Fast(data, "expected a sequence");
size = PySequence_Size(seq);
gene = (int*) PyMem_Malloc(size * sizeof(int));
for (i = 0; i < size; i++)
gene[i] = PyInt_AsLong(PySequence_Fast_GET_ITEM(seq, i));
/* Call the external C function*/
value = score(gene, size);
PyMem_Free(gene);
/* Build the output tuple */
PyObject *ret = Py_BuildValue("i", value);
return ret;
}
This works but seems to leak memory and at a rate I can't ignore. I made sure that the leak is happening in the shown function by temporarily making the score function just return 0 and still saw the leaking behavior. I had thought that the call to PyMem_Free should take care of the PyMem_Malloc'ed storage but my current guess is that something in this function is getting allocated and retained on each call since the leaking behavior is proportional to the number of calls to this function. Am I not doing the sequence to array conversion correctly or am I possibly returning the ending value inefficiently? Any help is appreciated.
seq is a new Python object so you will need delete that object. You should check if seq is NULL, too.
Something like (untested):
static PyObject *module_score(PyObject *self, PyObject *args) {
int i, size, value, *gene;
long temp;
PyObject *seq, *data;
/* Parse the input tuple */
if (!PyArg_ParseTuple(args, "O", &data))
return NULL;
if (!(seq = PySequence_Fast(data, "expected a sequence")))
return NULL;
size = PySequence_Size(seq);
gene = (int*) PyMem_Malloc(size * sizeof(int));
for (i = 0; i < size; i++) {
temp = PyInt_AsLong(PySequence_Fast_GET_ITEM(seq, i));
if (temp == -1 && PyErr_Occurred()) {
Py_DECREF(seq);
PyErr_SetString(PyExc_ValueError, "an integer value is required");
return NULL;
}
/* Do whatever you need to verify temp will fit in an int */
gene[i] = (int*)temp;
}
/* Call the external C function*/
value = score(gene, size);
PyMem_Free(gene);
Py_DECREF(seq):
/* Build the output tuple */
PyObject *ret = Py_BuildValue("i", value);
return ret;
}
Refering to http://mail.python.org/pipermail/python-dev/2009-June/090210.html
AND http://dan.iel.fm/posts/python-c-extensions/
and here is other places i searched regarding my question:
http://article.gmane.org/gmane.comp.python.general/424736
http://joyrex.spc.uchicago.edu/bookshelves/python/cookbook/pythoncook-CHP-16-SECT-3.html
http://docs.python.org/2/c-api/sequence.html#PySequence_Check
Python extension module with variable number of arguments
I am inexperienced in Python/C API.
I have the following code:
sm_int_list = (1,20,3)
c_int_array = (ctypes.c_int * len(sm_int_list))(*sm_int_list)
sm_str_tuple = ('some','text', 'here')
On the C extension side, i have done something like this:
static PyObject* stuff_here(PyObject *self, PyObject *args)
{
char* input;
int *i1, *i2;
char *s1, *s2;
// args = (('some','text', 'here'), [1,20,3], ('some','text', 'here'), [1,20,3])
**PyArg_ParseTuple(args, "(s#:):#(i:)#(s#:):#(i:)#", &s1, &i1, &s2, &i2)**;
/*stuff*/
}
such that:
stuff.here(('some','text', 'here'), [1,20,3], ('some','text', 'here'), [1,20,3])
returns data in the same form as args after some computation.
I would like to know the PyArg_ParseTuple expression, is it the proper way to parse
an array of varying string
an array of integers
UPDATE NEW
Is this the correct way?:
static PyObject* stuff_here(PyObject *self, PyObject *args)
unsigned int tint[], cint[];
ttotal=0, ctotal=0;
char *tstr, *cstr;
int *t_counts, *c_counts;
Py_ssize_t size;
PyObject *t_str1, *t_int1, *c_str2, *c_int2; //the C var that takes in the py variable value
PyObject *tseq, cseq;
int t_seqlen=0, c_seqlen=0;
if (!PyArg_ParseTuple(args, "OOiOOi", &t_str1, &t_int1, &ttotal, &c_str2, &c_int2, &ctotal))
{
return NULL;
}
if (!PySequence_Check(tag_str1) && !PySequence_Check(cat_str2)) return NULL;
else:
{
//All things t
tseq = PySequence_Fast(t_str1, "iterable");
t_seqlen = PySequence_Fast_GET_SIZE(tseq);
t_counts = PySequence_Fast(t_int1);
//All things c
cseq = PySequence_Fast(c_str2);
c_seqlen = PySequence_Fast_GET_SIZE(cseq);
c_counts = PySequence_Fast(c_int2);
//Make c arrays of all things tag and cat
for (i=0; i<t_seqlen; i++)
{
tstr[i] = PySequence_Fast_GET_ITEM(tseq, i);
tcounts[i] = PySequence_Fast_GET_ITEM(t_counts, i);
}
for (i=0; i<c_seqlen; i++)
{
cstr[i] = PySequence_Fast_GET_ITEM(cseq, i);
ccounts[i] = PySequence_Fast_GET_ITEM(c_counts, i);
}
}
OR
PyArg_ParseTuple(args, "(s:)(i:)(s:)(i:)", &s1, &i1, &s2, &i2)
And then again while returning,
Py_BuildValue("sisi", arr_str1,arr_int1,arr_str2,arr_int2) ??
Infact if someone could in detail clarify the various PyArg_ParseTuple function that would be of great benefit. the Python C API, as i find it in the documentation, is not exactly a tutorial on things to do.
You can use PyArg_ParseTuple to parse a real tuple, that has a fixed structure. Especially the number of items in the subtuples cannot change.
As the 2.7.5 documentation says, your format "(s#:):#(i:)#(s#:):#(i:)#" is wrong since : cannot occur in nested parenthesis. The format "(sss)(iii)(sss)(iii)", along with total of 12 pointer arguments should match your arguments. Likewise for Py_BuildValue you can use the same format string (which creates 4 tuples within 1 tuple), or "(sss)[iii](sss)[iii]" if the type matters (this makes the integers to be in lists instead of tuples).
I am using what seems to be the exact usgae of PyArg_ParseTuple, yet the code is still failing to work. I am using python 2.7
This is my C code for the Python Extension I am writing:
static PyObject* tpp(PyObject* self, PyObject* args)
{
PyObject* obj;
PyObject* seq;
int i, len;
PyObject* item;
int arrayValue, temp;
if (!PyArg_ParseTuple(args, "O", &obj)){
printf("Item is not a list\n");
return NULL;
}
seq = PySequence_Fast(obj, "expected a sequence");
len = PySequence_Size(obj);
arrayValue = -5;
printf("[\n");
for (i = 0; i < len; i++) {
item = PySequence_Fast_GET_ITEM(seq, i);
// printf("%d : %d, PyArg: ", item, *item);
// PyArg_ParseTuple(item, "I", &temp);
PyObject* objectsRepresentation = PyObject_Repr(item);
const char* s = PyString_AsString(objectsRepresentation);
printf("%s\n", s);
PyObject* objType = PyObject_Type(item);
PyObject* objTypeString = PyObject_Repr(objType);
const char* sType = PyString_AsString(objTypeString);
printf("%s\n", sType);
if (PyArg_ParseTuple(item, "i", &arrayValue) != 0){
printf("%d\n", arrayValue);
printf("horray!\n");
}
}
Py_DECREF(seq);
printf("]\n");
printf("Item is a list!\n");
Py_RETURN_NONE;
}
Then I just build the extension and go to the terminal
import et
and then
et.tpp([1,2])
fails to print the line
if (PyArg_ParseTuple(item, "i", &arrayValue) != 0){
printf("%d\n", arrayValue);
printf("horray!\n");
}
I checked the type, as you can see in the code, of the elements in the list, and it prints 'int'. Yet for some reason PyArg_ParseTuple is having errors.
I need to be able to access information from lists in python to copy some data, pass it to my C code elsewhere, and then return the result to python.
Thank you so much!
The answer is to use long PyInt_AsLong(PyObject *io)
"long PyInt_AsLong(PyObject *io) Will first attempt to cast the object to a PyIntObject, if it is not already one, and then return its value. If there is an error, -1 is returned, and the caller should check PyErr_Occurred() to find out whether there was an error, or whether the value just happened to be -1."
This is from http://docs.python.org/2/c-api/int.html That is the official c python int objects documentation which has all relevant methods.
Unfortunately this returns only a long value. However, a simple cast should suffice if the expected values will be small.
PyArg_ParseTuple() is about parsing tuples only, as the name suggests. In your code, item is an int, not a tuple. In order to convert an int object to a C value, you need to use arrayValue = PyInt_AsLong(item). Note that it returns a C long, not an int, so you should declare arrayValue as a long.
(EDIT: previously I mentioned PyInt_FromLong by mistake.)