Develop Reference

Pointer argument passing in python ctypes

I have the following c function.
/* returns (uint8_t *)outbuf */
uint8_t *func(uint8_t *inbuf, uint32_t inbuf_len, uint32_t *outbuf_len);
This function returns outbuf, the output length is unknown before calling the function so the function receives a pointer to the length as an argument outbuf_len, also the caller is responsible to free outbuf.
I want to get the result of this function from python, so I started writing the following code:
import ctypes as ct
libb = ct.cdll.LoadLibrary('./a.so')
libb.func.restype = ct.c_char_p
inbuf = bytearray(inbuf_len)
inbuf = python_data
arr = ct.c_ubyte * inbuf_len
outbuf_len = ct.c_uint # there is no ct.c_uint_p...
outbuf = libb.func(arr.from_buffer_copy(inbuf), inbuf_len, outbuf_len)
print hexlify(outbuf) #prints only the first 4 bytes of outbuf
The problems i have is:
I didn't find pointer to uint in ctypes types, so how can I pass the outbuf_len pointer to the C function?
when printing the outbuf, only the first 4 bytes that are pointed by the pointer are printed.
How do I free() the outbuf buffer from python?
I have the source of the C function so it is possible to change how arguments are passed the the C function.
Thanks.

If you'll be passing Python byte strings as the input buffer, here's a way to do it. I made a minimal example of the C call:
test.c
#include <stdint.h>
#include <stdlib.h>
#define API __declspec(dllexport)
// double every character in the input buffer as an example
API uint8_t *func(uint8_t *inbuf, uint32_t inbuf_len, uint32_t *outbuf_len) {
*outbuf_len = inbuf_len * 2;
uint8_t* outbuf = malloc(*outbuf_len);
for(uint32_t i = 0; i < inbuf_len; ++i) {
outbuf[i*2] = inbuf[i];
outbuf[i*2+1] = inbuf[i];
}
return outbuf;
}
API void freebuf(uint8_t* buf) {
free(buf);
}
test.py
import ctypes as ct
dll = ct.CDLL('./test')
# c_char_p accepts Python byte strings and is compatible with C uint8_t*
# but don't use it for the output buffer because ctypes converts the pointer
# back to a Python byte string and you would lose access to the pointer for
# later freeing it. Use POINTER(ct.c_char) to get the actual pointer back.
dll.func.argtypes = ct.c_char_p, ct.c_uint32, ct.POINTER(ct.c_uint32)
dll.func.restype = ct.POINTER(ct.c_char)
dll.freebuf.argtypes = ct.POINTER(ct.c_char),
dll.freebuf.restype = None
def func(inbuf):
outlen = ct.c_uint32() # create storage for the output length and pass by reference
outbuf = dll.func(inbuf, len(inbuf), ct.byref(outlen))
# Slicing the returned POINTER(c_char) returns a Python byte string.
# If you used POINTER(c_uint8) for the return value instead,
# you'd get a list of integer byte values.
data = outbuf[:outlen.value]
# Can free the pointer now if you want, or return it for freeing later
dll.freebuf(outbuf)
return data
print(func(b'ABC'))
Output:
b'AABBCC'

cTypes - passing a string as a pointer from python to c

I'm trying to pass a string (as a pointer) from python to a C function using cTypes.
The c function needs to get a pointer to a string (array of chars) but I haven't successfully got this to work with multiple chars, but my only success(kind of successful, look at the output) is with 1 char and would love for some help!
I need to send a pointer of a string to char - (unsigned char * input)
My Python Code:
def printmeP(CHAR):
print("In Print function")
print(CHAR)
c_sends = pointer(c_char_p(CHAR.encode('utf-8')))
print("c_sends: ")
print(c_sends[0])
python_p_printme(c_sends)
print("DONE function - Python")
print(c_sends[0])
return
from ctypes import c_double, c_int,c_char, c_wchar,c_char_p, c_wchar_p, pointer, POINTER,create_string_buffer, byref, CDLL
import sys
lib_path = '/root/mbedtls/programs/test/mylib_linux.so' .format(sys.platform)
CHAR = "d"
try:
mylib_lib = CDLL(lib_path)
except:
print('STARTING...' )
python_p_printme = mylib_lib.printme
python_p_printme.restype = None
printmeP(CHAR)
My C Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void printme(char * param) {
printf("\nStart c Function!\n");
printf("%s\n Printing param\n ", param);
char add = '!';
strncat(param, &add, 1);
printf("%s\n Printing param\n ", param);
printf("Function Done - c\n");
}
My Output:
In Print function
d <--- this is what i am sending
c_sends:
b'd' <--- this is after encoding
��[� <-------------=|
Printing param |
��[� | This is the c code print
Printing param | **Question:** Why does it print '�' and no what's supposed to be printed
Function Done - c <--=|
DONE function - Python
b'd!' <--------------------- this is the last print that prints after the change.
Would love for some help, thanks to everyone that participated :)
sincerely,
Roe

There are a number of problems:
Define .argtypes for your functions. It will catch errors passing incorrect parameters. Add the line below and note it is plural and is a tuple of argument types. The comma makes a 1-tuple:
python_p_printme.argtypes = c_char_p,
Once you make that change, you'll get an error because this code:
c_sends = pointer(c_char_p(CHAR.encode('utf-8')
is actually sending a C char** (a pointer to a c_char_p). Once you've set the argtypes properly, you can just call the function with a byte string and it will work. Your function becomes:
def printmeP(CHAR):
print("In Print function")
print(CHAR)
python_p_printme(CHAR.encode())
print("DONE function - Python")
There is one more subtle problem. While the program may appear to work at this point Python strings are immutable so if the function being called requires a mutable string, you must allocate a mutable buffer using either create_unicode_buffer (for w_char_p) or create_string_buffer (for c_char_p); otherwise, the strcat in your C code is going to corrupt the Python string.
Here's a full example:
test.cpp
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
// For Windows compatibility
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
// For C++ compiler compatibility
#ifdef __cplusplus
extern "C" {
#endif
API void alter_me(char* param, size_t length) {
// truncates if additional info exceeds length
strncat_s(param, length, " adding some info", length - 1);
}
#ifdef __cplusplus
}
#endif
test.py
from ctypes import *
lib = CDLL('./test')
alter_me = lib.alter_me
alter_me.argtypes = c_char_p,c_size_t
alter_me.restype = None
data = create_string_buffer(b'test',size=10)
alter_me(data,sizeof(data))
print(data.value)
data = create_string_buffer(b'test',size=50)
alter_me(data,sizeof(data))
print(data.value)
Output:
b'test addi'
b'test adding some info'
Note you do not need to use create_string_buffer if the C function does not alter the buffer, such as if the C parameter is const char*. Then you could just call printme(b'test string').

You could use create_string_buffer.
The documentation can be found here:
https://docs.python.org/3/library/ctypes.html#ctypes.create_string_buffer
ctypes.create_string_buffer(init_or_size, size=None)
This function creates a mutable character buffer. The returned object is a ctypes array of c_char.
init_or_size must be an integer which specifies the size of the array, or a bytes object which will be used to initialize the array items.
With buf.value.decode("utf-8") you can convert the buffer back to a UTF-8 python string.
A small example with your C code library might look like this:
from ctypes import *
mylib_lib = cdll.LoadLibrary("<your-lib>")
mylib_lib.printme.argtypes = c_char_p,
buf = create_string_buffer(128)
buf.value = b'Hello World'
mylib_lib.printme(buf)
print("print on python side:", buf.value.decode("utf-8"))
It would output:
Start c Function!
Hello World
...
print on python side: Hello World!
on the console.

Call a Python function from C and consume a 2D Numpy array in C

I'm trying to figure out, how I could achieve this:
I'm having a Python script, which in the end produces a Numpy array, an array of arrays of floats, to be more specific. I have all properly set: I can pass parameters from C to Python, launch Py functions from C, and process returned scalar values in C.
What I'm currently not being able to do, is to return such a Numpy array as a result of a Py function to C.
Could somebody probably provide me a pointer, how to achieve this?
TIA

What you need to look at is Inter Process communication (IPC). There are several ways to perform it.
You can use one of:
Files (Easy to use)
Shared memory (really fast)
Named pipes
Sockets (slow)
See Wikipedia's IPC page and find the best approach for your needs.

Here's a small working demo example (1D, not 2D array! it's not perfect, adjust to your needs).
# file ./pyscript.py
import numpy as np
# print inline
np.set_printoptions(linewidth=np.inf)
x = np.random.random(10)
print(x)
# [0.52523722 0.29053534 0.95293405 0.7966214 0.77120688 0.22154705 0.29398872 0.47186567 0.3364234 0.38107864]
~ demo.c
// file ./demo.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
int main()
{
int fd[2];
pipe(fd); // create pipes
char buf[4096];
pid_t pid=fork();
if (pid==0) { // child process
dup2(fd[1],1);
close(fd[0]);
close(fd[1]);
char *pyscript = "./pyscript.py";
char *args[] = {"python3", pyscript, (char*)NULL};
execv("/usr/bin/python3",args);
}
else {
int status;
close(fd[1]);
int bytes = read(fd[0], buf, sizeof(buf));
printf("Python script output: %.*s\n", bytes, buf);
char* values[10];
int count = 0;
values[count++] = &buf[1]; // ignore the '[' coming from numpy array output
char* p = buf;
while (*p) {
if (*p == ' ') {
*p = 0;
values[count++] = p + 1;
}
p++;
}
float a[10];
float f;
for (int i = 0; i < 10; i++) {
printf("%f\n", f = atof(values[i]) ); // float values
a[i] = f;
}
waitpid(pid, &status, 0);
}
return 0;
}
Sample output
# cc demo.c
# ./a.out
Python script output: [0.23286839 0.54437959 0.37798547 0.17190732 0.49473837 0.48112695 0.93113395 0.20877592 0.96032973 0.30025713]
0.23286839
0.54437959
0.232868
0.544380
0.377985
0.171907
0.494738
0.481127
0.931134
0.208776
0.960330
0.300257
a will be your desired result, an array of float.

One has to use the PyList API for decoding list objects from Python to C
https://docs.python.org/3.3/c-api/list.html?highlight=m
Solved.

Python 3 ctypes call to a function that needs an indirect reference to a buffer through another structure

I have a C shared library with a function that takes one argument.
This argument is a pointer to a structure with two fields.
typedef struct
{
uint8_t *p_data; // Pointer to a fixed lenth buffer (100 bytes)
uint16_t len; // number of valid bytes in the buffer (range 1-100)
} data_t;
I need to setup a buffer of 100 bytes in my Python 3 script (I am using 3.7.2 / 3.7.3),
load the library and call this function.
int
fn_convert_buffer(data_t *data_p)
{
...
}
My Python 3 ctypes call attempt hits incompatible types.
import ctypes as ct
# load the library, etc...
# lib_cdll = ct.CDLL(mySharedLib)
def c_py_fn_convert_buffer(b_p):
global lib_cdll
val = lib_cdll.fn_convert_buffer(ct.byref(b_p))
return int(val)
data_a = bytearray(100)
# Initialize the buffer with data.
uint8_p = ct.c_uint8 * len(data_a)
class BufferStruct_t (ct.Structure):
_pack_ = 1
_fields_ = [
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
("len", ct.c_uint16)
]
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
# TypeError: incompatible types, c_ubyte_Array_100 instance
# instead of LP_c_ubyte_Array_100 instance
# Call C function in shared-library: int fn_convert_buffer(data_t *data_p);
z = c_py_fn_convert_buffer(data_buf)
I need help in understanding what I've missed in the BufferStruct_t definition above. The from_buffer is supposed to get a pointer to the buffer but it seems to get c_ubyte_ARRAY_100.
A byref() on that does not work either
data_buf = BufferStruct_t(ct.byref(uint8_p.from_buffer(data_a)), ct.c_uint16(8))
# TypeError: expected LP_c_ubyte_Array_100 instance, got CArgObject
To test the basics of my flow, I made a sample case that will send the buffer and length parameters individually.
def c_py_fn_convert_data(d_p,l):
global lib_cdll
val = lib_cdll.fn_convert_data(ct.byref(d_p),ct.c_uint32(l))
return int(val)
test_a = ct.c_uint8 * len(data_a)
# Call C function in shared-library:
# int fn_convert_data(uint8_t *data_p, uint32_t length);
z = c_py_fn_convert_data(test_a.from_buffer(data_a), 8)
This simplified case works.
How do I get about building a Python 3 object that carries a reference to a buffer that the shared-library function expects?
Update with two variations that worked.
Update 1 Tried a cast based on something I read later (I don't cast lightly :-))
Changed,
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
to a pointer that is cast to refer an Array of specific length,
data_buf = BufferStruct_t(cast(uint8_p.from_buffer(data_a),
ct.POINTER(ct.c_uint8 * len(data_a))),
ct.c_uint16(8))
Update 2 based on Mark's answer.
Changed _field_ from,
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
to a simple-pointer form,
("p_data", ct.POINTER(ct.c_uint8)),
Both variations worked.
I am however curious to know which of these two ways is more safe/correct ctypes handling.
Is it better to cast to the Array form? or,
Is it better to use simple pointers and rely on the length sent independently?

Your structure definition declared a pointer to an array, not a simple pointer as in the C structure. Here's a working example with a simple implementation of the DLL where the function sums the data:
test.c
#include <stdint.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
typedef struct {
uint8_t *p_data;
uint16_t len;
} data_t;
API int fn_convert_buffer(data_t *data_p)
{
int i;
int sum = 0;
for(i = 0; i < data_p->len; ++i)
sum += data_p->p_data[i];
return sum;
}
test.py
import ctypes as ct
class BufferStruct_t(ct.Structure):
_pack_ = 1
_fields_ = [("p_data", ct.POINTER(ct.c_uint8)), # just a pointer
("len", ct.c_uint16)]
# Helper to initialize the data
def __init__(self,data):
self.p_data = (ct.c_uint8 * len(data))(*data)
self.len = len(data)
dll = ct.CDLL('test')
dll.fn_convert_buffer.argtypes = ct.POINTER(BufferStruct_t),
dll.fn_convert_buffer.restype = ct.c_int
data_buf = BufferStruct_t([1,2,3,4,5])
print(dll.fn_convert_buffer(data_buf))
Output:
15

Marshaling a Python PIL Image using SWIG

I've got a library that takes in a very simple C image structure:
// Represents a one-channel 8-bit image
typedef struct simple_image_t {
uint32 rows;
uint32 cols;
uint8 *imgdata;
} simple_image;
I didn't create this library, nor this structure, so I can't change it. I'm responsible for wrapping this library for python using SWIG. The Python wrapper needs to be able to take in a PIL Image and convert it into this structure. Here's how I'm doing it right now (using a SWIG %inline%):
// Allows python to easily create and initialize this structure
simple_image* py_make_simple_image(uint32 width, uint32 height)
{
simple_image* img = new simple_image();
img->rows = height;
img->cols = width;
img->imgdata = new uint8[height * width];
return img;
}
// Allows python to set a particular pixel value
void py_set_simple_image(simple_image* img, uint32 pos, uint8 val)
{
img->imgdata[pos] = val;
}
And then on the python wrapper side here's how things look right now:
# Make sure it's an 8-bit image
if pil_image.mode != "L":
pil_image = pil_image.convert("L")
# Create the simple image structure
(width, height) = pil_image.size
img = swig_wrapper.py_make_simple_image(width, height)
try:
# Copy the image data into the simple image structure
pos = 0
for pixel in pil_image.getdata():
swig_wrapper.py_set_simple_image(img, pos, pixel)
pos += 1
# Call some library method that accepts a simple_image*
return swig_wrapper.some_image_method(img)
finally:
# Clean up the simple image structure
swig_wrapper.py_destroy_simple_image(img)
Amazingly this works, however as you may have guessed it's incredibly slow when working with even moderately large images. I know with SWIG the proper way to do things is to use a typemap, however that would mean digging in to the C API of PIL, and I just didn't have time to do that at the moment.
What are my options in terms of speed? Are there quicker ways of marshaling the pixel data from a PIL image to this simple image structure? Has someone already done this and my Google skills are just that bad? Am I just boned and soon will need to learn the internals of PIL?
Thanks.

PIL's Image.tostring() returns a string of the exact data you need for imgdata. The typemap I used is fairly simple, but not perfect, which I'll note below. Here is the sample code I created on Windows that worked for me:
sample.h
typedef unsigned int uint32;
typedef unsigned char uint8;
typedef struct simple_image_t {
uint32 rows;
uint32 cols;
uint8 *imgdata;
} simple_image;
#ifdef SAMPLE_EXPORT
# define SAMPLE_API __declspec(dllexport)
#else
# define SAMPLE_API __declspec(dllimport)
#endif
SAMPLE_API void some_func(const simple_image* si);
sample.c
#include <stdio.h>
#define SAMPLE_EXPORT
#include "sample.h"
void some_func(const simple_image* si)
{
uint32 i,j;
printf(
"rows = %d\n"
"cols = %d\n",
si->rows,si->cols);
/* Dump a simple map of the image data */
for(i = 0; i < si->rows; i++)
{
for(j = 0; j < si->cols; j++)
{
if(si->imgdata[i * si->rows + j] < 0x80)
printf(" ");
else
printf("*");
}
printf("\n");
}
}
sample.i
%module sample
%begin %{
#pragma warning(disable:4100 4127 4706)
%}
%{
#include "sample.h"
%}
%include <windows.i>
%typemap(in) uint8* (char* buffer, Py_ssize_t length) {
PyString_AsStringAndSize($input,&buffer,&length);
$1 = (uint8*)buffer;
}
%include "sample.h"
makefile
all: _sample.pyd
sample.dll: sample.c sample.h
cl /nologo /W4 /LD /MD sample.c
sample_wrap.c: sample.i
#echo sample.i
swig -python sample.i
_sample.pyd: sample_wrap.c sample.dll
cl /nologo /W4 /LD /MD /Fe_sample.pyd sample_wrap.c /Ic:\Python27\include -link /LIBPATH:c:\Python27\libs python27.lib sample.lib
example.py
from PIL import Image
import sample
im = Image.open('sample.gif')
im = im.convert('L')
si = sample.simple_image()
si.rows,si.cols = im.size
s = im.tostring() # Must keep a reference
si.imgdata = s
sample.some_func(si)
With this quick example I haven't determined how the typemap should correctly increment the reference count of the string object. Note that the above code could crash if the following code were used:
si.imgdata = im.tostring()
The current typemap's PyString_AsStringAndSize returns a direct pointer to the PyString object's buffer, but doesn't increment the reference count for the object. It can be garbage collected before some_func executes (and was for me, crashing Python). Assigning to s keeps a reference to the string and prevents problems. The typemap should copy the buffer, but you were looking for speed so this hack may be what you want.

May be you could convert the image to a char array using the array module, and then, from swig, memcpy the data to your C array.
import array
imagar = array.array('B', pil_image.getdata())
(mem, length) = imagar.buffer_info()
swig_wrapper.py_copy(img, mem, length)
being py_copy something like:
void py_copy(simple_image* img, uint32 mem, uint32 length) {
memcpy((void*)img->imgdata ,(void*)mem, length );
}

How about using ctypes? It allows you to have direct access to c structures, so no need to create a Python equivalent of the struct, and you should also be able to do a memcpy (which would be faster than copying pixel by pixel).