Idea is to be able to modify array from library, like an "output" from a function.
Example:
ffi.cdef("""
//Reads data from a file, and store in the numpy array
void read_image(PyArray* arr);
""")
C = ffi.dlopen('libimage.so')
image = np.array([], dtype=np.float32)
C.read_image(image)
assert image.ndim == 2
You can't pass CPython-specific PyXxx structures via CFFI: you need to pass standard C data. Normally I'd answer that you need to design your cdef()'ed function with a standard C interface, for example something like:
ffi.cdef("""
struct myimage_t {
int width, height;
float *data;
};
int read_image(struct myimage_t *output); // fill in '*output'
void free_image(struct myimage_t *img); // free output->data
""")
myimage = ffi.new("struct myimage_t *")
if lib.read_image(myimage) < 0:
raise IOError
...
lib.free_image(myimage)
Then you need to manually convert the myimage to a numpy array, somewhere in the "..." code above.
One better alternative is to use a Python callback: a callback that makes the numpy array according to spec and returns a C-standard float * pointer. The numpy array itself is saved somewhere in the callback. You could save it as a Python global, or more cleanly use a "handle" you pass via C. Requires the API version, not the ABI. In _example_build.py:
ffi.cdef("""
extern "Python" float *alloc_2d_numpy_array(void *handle, int w, int h);
void read_image(void *handle);
""")
ffi.set_source("_example_cffi", """
void read_image(void *handle)
{
// the C code that eventually invokes
float *p = alloc_2d_numpy_array(handle, w, h);
// and then fill the data at 'p'
}
""")
ffi.compile(verbose=True)
In file example.py:
from _example_cffi import ffi, lib
class Context:
pass
#ffi.def_extern()
def alloc_2d_numpy_array(handle, w, h):
context = ffi.from_handle(handle)
context.image = np.ndarray([w, h], dtype=np.float32)
return ffi.cast("float *", ffi.from_buffer(context.image))
context = Context()
lib.read_image(ffi.new_handle(context))
image = context.image
Related
cuda1.cu
#include <iostream>
using namespace std ;
# define DELLEXPORT extern "C" __declspec(dllexport)
__global__ void kernel(long* answer = 0){
*answer = threadIdx.x + (blockIdx.x * blockDim.x);
}
DELLEXPORT void resoult(long* h_answer){
long* d_answer = 0;
cudaMalloc(&d_answer, sizeof(long));
kernel<<<10,1000>>>(d_answer);
cudaMemcpy(&h_answer, d_answer, sizeof(long), cudaMemcpyDeviceToHost);
cudaFree(d_answer);
}
main.py
import ctypes
import numpy as np
add_lib = ctypes.CDLL(".\\a.dll")
resoult= add_lib.resoult
resoult.argtypes = [ctypes.POINTER(ctypes.c_long)]
x = ctypes.c_long()
print("R:",resoult(x))
print("RV: ",x.value)
print("RB: ",resoult(ctypes.byref(x)))
output in python:0
output in cuda: 2096
I implemented based on c language without any problems but in cuda mode I have a problem how can I have the correct output value
Thanks
cudaMemcpy is expecting pointers for dst and src.
In your function resoult, h_answer is a pointer to a long allocated by the caller.
Since it's already the pointer where the data should be copied to, you should use it as is and not take it's address by using &h_answer.
Therefore you need to change your cudaMemcpy from:
cudaMemcpy(&h_answer, d_answer, sizeof(long), cudaMemcpyDeviceToHost);
To:
cudaMemcpy(h_answer, d_answer, sizeof(long), cudaMemcpyDeviceToHost);
I have a C function that reads a binary file and returns a dynamically sized array of unsigned integers (the size is based off metadata from the binary file):
//example.c
#include <stdio.h>
#include <stdlib.h>
__declspec(dllexport)unsigned int *read_data(char *filename, size_t* array_size){
FILE *f = fopen(filename, "rb");
fread(array_size, sizeof(size_t), 1, f);
unsigned int *array = (unsigned int *)malloc(*array_size * sizeof(unsigned int));
fread(array, sizeof(unsigned int), *array_size, f);
fclose(f);
return array;
}
This answer appears to be saying that the correct way to pass the created array from C to Python is something like this:
# example_wrap.py
from ctypes import *
import os
os.add_dll_directory(os.getcwd())
indexer_dll = CDLL("example.dll")
def read_data(filename):
filename = bytes(filename, 'utf-8')
size = c_size_t()
ptr = indexer_dll.read_data(filename, byref(size))
return ptr[:size]
However, when I run the python wrapper, the code silently fails at ptr[:size] as if I'm trying to access an array out of bounds, and I probably am, but what is the correct way to pass this dynamically size array?
A few considerations:
First, you need to properly set the prototype of the C function so that ctypes can properly convert between the C and Python types.
Second, since size is actually a ctypes.c_size_t object, you actually need to use size.value to access the numeric value of the array size.
Third, since ptr[:size.value] actually copies the array contents to a Python list, you'll want to make sure you also free() the allocated C array since you're not going to use it anymore.
(Perhaps copying the array to a Python list is not ideal here, but I'll assume it's ok here, since otherwise you have more complexity in handling the C array in Python.)
This should work:
from ctypes import *
import os
os.add_dll_directory(os.getcwd())
indexer_dll = CDLL("example.dll")
indexer_dll.read_data.argtypes = [c_char_p, POINTER(c_size_t)
indexer_dll.read_data.restype = POINTER(c_int)
libc = cdll.msvcrt
def read_data(filename):
filename = bytes(filename, 'utf-8')
size = c_size_t()
ptr = indexer_dll.read_data(filename, byref(size))
result = ptr[:size.value]
libc.free(ptr)
return result
I'm trying to figure out, how I could achieve this:
I'm having a Python script, which in the end produces a Numpy array, an array of arrays of floats, to be more specific. I have all properly set: I can pass parameters from C to Python, launch Py functions from C, and process returned scalar values in C.
What I'm currently not being able to do, is to return such a Numpy array as a result of a Py function to C.
Could somebody probably provide me a pointer, how to achieve this?
TIA
What you need to look at is Inter Process communication (IPC). There are several ways to perform it.
You can use one of:
Files (Easy to use)
Shared memory (really fast)
Named pipes
Sockets (slow)
See Wikipedia's IPC page and find the best approach for your needs.
Here's a small working demo example (1D, not 2D array! it's not perfect, adjust to your needs).
# file ./pyscript.py
import numpy as np
# print inline
np.set_printoptions(linewidth=np.inf)
x = np.random.random(10)
print(x)
# [0.52523722 0.29053534 0.95293405 0.7966214 0.77120688 0.22154705 0.29398872 0.47186567 0.3364234 0.38107864]
~ demo.c
// file ./demo.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
int main()
{
int fd[2];
pipe(fd); // create pipes
char buf[4096];
pid_t pid=fork();
if (pid==0) { // child process
dup2(fd[1],1);
close(fd[0]);
close(fd[1]);
char *pyscript = "./pyscript.py";
char *args[] = {"python3", pyscript, (char*)NULL};
execv("/usr/bin/python3",args);
}
else {
int status;
close(fd[1]);
int bytes = read(fd[0], buf, sizeof(buf));
printf("Python script output: %.*s\n", bytes, buf);
char* values[10];
int count = 0;
values[count++] = &buf[1]; // ignore the '[' coming from numpy array output
char* p = buf;
while (*p) {
if (*p == ' ') {
*p = 0;
values[count++] = p + 1;
}
p++;
}
float a[10];
float f;
for (int i = 0; i < 10; i++) {
printf("%f\n", f = atof(values[i]) ); // float values
a[i] = f;
}
waitpid(pid, &status, 0);
}
return 0;
}
Sample output
# cc demo.c
# ./a.out
Python script output: [0.23286839 0.54437959 0.37798547 0.17190732 0.49473837 0.48112695 0.93113395 0.20877592 0.96032973 0.30025713]
0.23286839
0.54437959
0.232868
0.544380
0.377985
0.171907
0.494738
0.481127
0.931134
0.208776
0.960330
0.300257
a will be your desired result, an array of float.
One has to use the PyList API for decoding list objects from Python to C
https://docs.python.org/3.3/c-api/list.html?highlight=m
Solved.
I have a C shared library with a function that takes one argument.
This argument is a pointer to a structure with two fields.
typedef struct
{
uint8_t *p_data; // Pointer to a fixed lenth buffer (100 bytes)
uint16_t len; // number of valid bytes in the buffer (range 1-100)
} data_t;
I need to setup a buffer of 100 bytes in my Python 3 script (I am using 3.7.2 / 3.7.3),
load the library and call this function.
int
fn_convert_buffer(data_t *data_p)
{
...
}
My Python 3 ctypes call attempt hits incompatible types.
import ctypes as ct
# load the library, etc...
# lib_cdll = ct.CDLL(mySharedLib)
def c_py_fn_convert_buffer(b_p):
global lib_cdll
val = lib_cdll.fn_convert_buffer(ct.byref(b_p))
return int(val)
data_a = bytearray(100)
# Initialize the buffer with data.
uint8_p = ct.c_uint8 * len(data_a)
class BufferStruct_t (ct.Structure):
_pack_ = 1
_fields_ = [
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
("len", ct.c_uint16)
]
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
# TypeError: incompatible types, c_ubyte_Array_100 instance
# instead of LP_c_ubyte_Array_100 instance
# Call C function in shared-library: int fn_convert_buffer(data_t *data_p);
z = c_py_fn_convert_buffer(data_buf)
I need help in understanding what I've missed in the BufferStruct_t definition above. The from_buffer is supposed to get a pointer to the buffer but it seems to get c_ubyte_ARRAY_100.
A byref() on that does not work either
data_buf = BufferStruct_t(ct.byref(uint8_p.from_buffer(data_a)), ct.c_uint16(8))
# TypeError: expected LP_c_ubyte_Array_100 instance, got CArgObject
To test the basics of my flow, I made a sample case that will send the buffer and length parameters individually.
def c_py_fn_convert_data(d_p,l):
global lib_cdll
val = lib_cdll.fn_convert_data(ct.byref(d_p),ct.c_uint32(l))
return int(val)
test_a = ct.c_uint8 * len(data_a)
# Call C function in shared-library:
# int fn_convert_data(uint8_t *data_p, uint32_t length);
z = c_py_fn_convert_data(test_a.from_buffer(data_a), 8)
This simplified case works.
How do I get about building a Python 3 object that carries a reference to a buffer that the shared-library function expects?
Update with two variations that worked.
Update 1 Tried a cast based on something I read later (I don't cast lightly :-))
Changed,
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
to a pointer that is cast to refer an Array of specific length,
data_buf = BufferStruct_t(cast(uint8_p.from_buffer(data_a),
ct.POINTER(ct.c_uint8 * len(data_a))),
ct.c_uint16(8))
Update 2 based on Mark's answer.
Changed _field_ from,
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
to a simple-pointer form,
("p_data", ct.POINTER(ct.c_uint8)),
Both variations worked.
I am however curious to know which of these two ways is more safe/correct ctypes handling.
Is it better to cast to the Array form? or,
Is it better to use simple pointers and rely on the length sent independently?
Your structure definition declared a pointer to an array, not a simple pointer as in the C structure. Here's a working example with a simple implementation of the DLL where the function sums the data:
test.c
#include <stdint.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
typedef struct {
uint8_t *p_data;
uint16_t len;
} data_t;
API int fn_convert_buffer(data_t *data_p)
{
int i;
int sum = 0;
for(i = 0; i < data_p->len; ++i)
sum += data_p->p_data[i];
return sum;
}
test.py
import ctypes as ct
class BufferStruct_t(ct.Structure):
_pack_ = 1
_fields_ = [("p_data", ct.POINTER(ct.c_uint8)), # just a pointer
("len", ct.c_uint16)]
# Helper to initialize the data
def __init__(self,data):
self.p_data = (ct.c_uint8 * len(data))(*data)
self.len = len(data)
dll = ct.CDLL('test')
dll.fn_convert_buffer.argtypes = ct.POINTER(BufferStruct_t),
dll.fn_convert_buffer.restype = ct.c_int
data_buf = BufferStruct_t([1,2,3,4,5])
print(dll.fn_convert_buffer(data_buf))
Output:
15
For a bit of learning experience, I'm trying to wrap a few parts of SDL (1.2.14) in Cython in an extension for Python 3.2.
I am having a problem figuring out how to wrap C structs straight into Python, being able to access its attributes directly like:
struct_name.attribute
For example, I want to take the struct SDL_Surface:
typedef struct SDL_Rect {
Uint32 flags
SDL_PixelFormat * format
int w, h
Uint16 pitch
void * pixels
SDL_Rect clip_rect
int refcount
} SDL_Rect;
And be able to use it like so in python:
import SDL
# initializing stuff
Screen = SDL.SetVideoMode( 320, 480, 32, SDL.DOUBLEBUF )
# accessing SDL_Surface.w and SDL_Surface.h
print( Screen.w, ' ', Screen.h )
For right now, I have wrapped the SDL_SetVideoMode and SDL_Surface like this in
a file called SDL.pyx
cdef extern from 'SDL.h':
# Other stuff
struct SDL_Surface:
unsigned long flags
SDL_PixelFormat * format
int w, h
# like past declaration...
SDL_Surface * SDL_SetVideoMode(int, int, int, unsigned )
cdef class Surface(object):
# not sure how to implement
def SetVideoMode(width, height, bpp, flags):
cdef SDL_Surface * screen = SDL_SetVideoMode
if not screen:
err = SDL_GetError()
raise Exception(err)
# Possible way to return?...
return Surface(screen)
How should I implement SDL.Surface?
In a simple case, if struct is opaque, it's as simple as:
cdef extern from "foo.h":
struct spam:
pass
When you want access to members, there are several options, well presented in the docs:
http://docs.cython.org/src/userguide/external_C_code.html#styles-of-struct-union-and-enum-declaration