Develop Reference

Python Ctypes - Writing to memory created by malloc

I have something like this:
import ctypes
dll = ctypes.CDLL('libc.so.6')
dll.malloc.argtypes = ctypes.c_size_t
dll.malloc.restype = ctypes.c_void_p
ptr: int = dll.malloc(100) # how do i write to this memory?
print(ptr) # gives a memory address
How do I set a value to that memory? Preferably a ctypes.py_object.
I'm aware that doing this in Python is a bad idea, but I'm just experimenting with breaking the rules.

You can write to the pointer similar to C. Cast the void pointer to something that can be dereferenced and write to it with Python or use ctypes to call a C function that can use the pointer. A couple of examples:
import ctypes as ct
dll = ct.CDLL('msvcrt') # Windows C runtime
# void* malloc(size_t size);
dll.malloc.argtypes = ct.c_size_t,
dll.malloc.restype = ct.c_void_p
# void free(void* ptr);
dll.free.argtypes = ct.c_void_p,
dll.free.restype = None
# char* strcpy(char* dest, const char* src)
dll.strcpy.argtypes = ct.c_char_p, ct.c_char_p
dll.strcpy.restype = ct.c_char_p
# casting to pointer to specific-sized array enables
# use of ".raw" and checks that array bounds aren't exceeded.
ptr = ct.cast(dll.malloc(10), ct.POINTER(ct.c_char * 10))
print(ptr.contents.raw)
# Note use of [:] to replace the entire array contents with a same-sized bytes object
ptr.contents[:] = b'abcdefghij'
print(ptr.contents.raw)
# char (*ptr)(10) isn't correct for strcpy, so cast to char*
dll.strcpy(ct.cast(ptr, ct.POINTER(ct.c_char)), b'ABCDEFGHIJ')
print(ptr.contents.raw)
dll.free(ptr)
Output:
b'\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00'
b'abcdefghij'
b'ABCDEFGHIJ'

Pointer argument passing in python ctypes

I have the following c function.
/* returns (uint8_t *)outbuf */
uint8_t *func(uint8_t *inbuf, uint32_t inbuf_len, uint32_t *outbuf_len);
This function returns outbuf, the output length is unknown before calling the function so the function receives a pointer to the length as an argument outbuf_len, also the caller is responsible to free outbuf.
I want to get the result of this function from python, so I started writing the following code:
import ctypes as ct
libb = ct.cdll.LoadLibrary('./a.so')
libb.func.restype = ct.c_char_p
inbuf = bytearray(inbuf_len)
inbuf = python_data
arr = ct.c_ubyte * inbuf_len
outbuf_len = ct.c_uint # there is no ct.c_uint_p...
outbuf = libb.func(arr.from_buffer_copy(inbuf), inbuf_len, outbuf_len)
print hexlify(outbuf) #prints only the first 4 bytes of outbuf
The problems i have is:
I didn't find pointer to uint in ctypes types, so how can I pass the outbuf_len pointer to the C function?
when printing the outbuf, only the first 4 bytes that are pointed by the pointer are printed.
How do I free() the outbuf buffer from python?
I have the source of the C function so it is possible to change how arguments are passed the the C function.
Thanks.

If you'll be passing Python byte strings as the input buffer, here's a way to do it. I made a minimal example of the C call:
test.c
#include <stdint.h>
#include <stdlib.h>
#define API __declspec(dllexport)
// double every character in the input buffer as an example
API uint8_t *func(uint8_t *inbuf, uint32_t inbuf_len, uint32_t *outbuf_len) {
*outbuf_len = inbuf_len * 2;
uint8_t* outbuf = malloc(*outbuf_len);
for(uint32_t i = 0; i < inbuf_len; ++i) {
outbuf[i*2] = inbuf[i];
outbuf[i*2+1] = inbuf[i];
}
return outbuf;
}
API void freebuf(uint8_t* buf) {
free(buf);
}
test.py
import ctypes as ct
dll = ct.CDLL('./test')
# c_char_p accepts Python byte strings and is compatible with C uint8_t*
# but don't use it for the output buffer because ctypes converts the pointer
# back to a Python byte string and you would lose access to the pointer for
# later freeing it. Use POINTER(ct.c_char) to get the actual pointer back.
dll.func.argtypes = ct.c_char_p, ct.c_uint32, ct.POINTER(ct.c_uint32)
dll.func.restype = ct.POINTER(ct.c_char)
dll.freebuf.argtypes = ct.POINTER(ct.c_char),
dll.freebuf.restype = None
def func(inbuf):
outlen = ct.c_uint32() # create storage for the output length and pass by reference
outbuf = dll.func(inbuf, len(inbuf), ct.byref(outlen))
# Slicing the returned POINTER(c_char) returns a Python byte string.
# If you used POINTER(c_uint8) for the return value instead,
# you'd get a list of integer byte values.
data = outbuf[:outlen.value]
# Can free the pointer now if you want, or return it for freeing later
dll.freebuf(outbuf)
return data
print(func(b'ABC'))
Output:
b'AABBCC'

Python 3 ctypes call to a function that needs an indirect reference to a buffer through another structure

I have a C shared library with a function that takes one argument.
This argument is a pointer to a structure with two fields.
typedef struct
{
uint8_t *p_data; // Pointer to a fixed lenth buffer (100 bytes)
uint16_t len; // number of valid bytes in the buffer (range 1-100)
} data_t;
I need to setup a buffer of 100 bytes in my Python 3 script (I am using 3.7.2 / 3.7.3),
load the library and call this function.
int
fn_convert_buffer(data_t *data_p)
{
...
}
My Python 3 ctypes call attempt hits incompatible types.
import ctypes as ct
# load the library, etc...
# lib_cdll = ct.CDLL(mySharedLib)
def c_py_fn_convert_buffer(b_p):
global lib_cdll
val = lib_cdll.fn_convert_buffer(ct.byref(b_p))
return int(val)
data_a = bytearray(100)
# Initialize the buffer with data.
uint8_p = ct.c_uint8 * len(data_a)
class BufferStruct_t (ct.Structure):
_pack_ = 1
_fields_ = [
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
("len", ct.c_uint16)
]
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
# TypeError: incompatible types, c_ubyte_Array_100 instance
# instead of LP_c_ubyte_Array_100 instance
# Call C function in shared-library: int fn_convert_buffer(data_t *data_p);
z = c_py_fn_convert_buffer(data_buf)
I need help in understanding what I've missed in the BufferStruct_t definition above. The from_buffer is supposed to get a pointer to the buffer but it seems to get c_ubyte_ARRAY_100.
A byref() on that does not work either
data_buf = BufferStruct_t(ct.byref(uint8_p.from_buffer(data_a)), ct.c_uint16(8))
# TypeError: expected LP_c_ubyte_Array_100 instance, got CArgObject
To test the basics of my flow, I made a sample case that will send the buffer and length parameters individually.
def c_py_fn_convert_data(d_p,l):
global lib_cdll
val = lib_cdll.fn_convert_data(ct.byref(d_p),ct.c_uint32(l))
return int(val)
test_a = ct.c_uint8 * len(data_a)
# Call C function in shared-library:
# int fn_convert_data(uint8_t *data_p, uint32_t length);
z = c_py_fn_convert_data(test_a.from_buffer(data_a), 8)
This simplified case works.
How do I get about building a Python 3 object that carries a reference to a buffer that the shared-library function expects?
Update with two variations that worked.
Update 1 Tried a cast based on something I read later (I don't cast lightly :-))
Changed,
data_buf = BufferStruct_t(uint8_p.from_buffer(data_a), ct.c_uint16(8))
to a pointer that is cast to refer an Array of specific length,
data_buf = BufferStruct_t(cast(uint8_p.from_buffer(data_a),
ct.POINTER(ct.c_uint8 * len(data_a))),
ct.c_uint16(8))
Update 2 based on Mark's answer.
Changed _field_ from,
("p_data", ct.POINTER(ct.c_uint8 * len(data_a))),
to a simple-pointer form,
("p_data", ct.POINTER(ct.c_uint8)),
Both variations worked.
I am however curious to know which of these two ways is more safe/correct ctypes handling.
Is it better to cast to the Array form? or,
Is it better to use simple pointers and rely on the length sent independently?

Your structure definition declared a pointer to an array, not a simple pointer as in the C structure. Here's a working example with a simple implementation of the DLL where the function sums the data:
test.c
#include <stdint.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
typedef struct {
uint8_t *p_data;
uint16_t len;
} data_t;
API int fn_convert_buffer(data_t *data_p)
{
int i;
int sum = 0;
for(i = 0; i < data_p->len; ++i)
sum += data_p->p_data[i];
return sum;
}
test.py
import ctypes as ct
class BufferStruct_t(ct.Structure):
_pack_ = 1
_fields_ = [("p_data", ct.POINTER(ct.c_uint8)), # just a pointer
("len", ct.c_uint16)]
# Helper to initialize the data
def __init__(self,data):
self.p_data = (ct.c_uint8 * len(data))(*data)
self.len = len(data)
dll = ct.CDLL('test')
dll.fn_convert_buffer.argtypes = ct.POINTER(BufferStruct_t),
dll.fn_convert_buffer.restype = ct.c_int
data_buf = BufferStruct_t([1,2,3,4,5])
print(dll.fn_convert_buffer(data_buf))
Output:
15

Python ctypes, pass c_void_p as an out parameter to c function

I'm writing a Python module that wraps some C++ code. I used to use this function to pass a pointer from C++ to Python (the pointer would be freed by a different function
DLLEXPORT void* run(double input1, double input2)
I've added error return codes so my new function looks like
DLLEXPORT int run(double input1, double input2, void* output)
But now I can't seem to get the value of the pointer, in Python I set up the function with ctypes as follows
from ctypes import *
mydll = cdll.mydll
mydll.run.argtypes = [c_double, # input 1
c_double, # input 2
c_void_p] # output pointer
mydll.run.restype = c_int
Then I use the function by creating a new void pointer in Python and passing it to the dll function
p = c_void_p()
retval = mydll.run(1.2, 3.4, p)
print p
After running this code I am left with p equal to c_void_p(None).
Passing this pointer to other functions causes illegal access exceptions at address 0x0, so I don't think it is being updated.
I am expected some address to populate p after execution. I am missing something with ctypes? I can create a double array by allocating (c_double * 10)() and pass that to c function to be written to, why can't I pass a void pointer for the same purpose?

As eryksun's comment points out, for void* to be an output parameter, it should be void**:
# DLLEXPORT int run(double input1, double input2, void** output)
from ctypes import *
mydll = cdll.mydll
mydll.run.argtypes = [c_double, # input 1
c_double, # input 2
POINTER(c_void_p)] # output pointer
mydll.run.restype = c_int
p = c_void_p()
retval = mydll.run(1.2,3.4,byref(p))
print p.contents

python: handling c malloc variables with ctypes

I'm playing a bit with python ctypes, probably someone could tell me some elegant way to handle an exported buffer created in C with a malloc.
So, it's a very stupid c code that could explain what I'm looking for.
#include <stdio.h>
#include <stdlib.h>
char * stringa;
int numero = 0;
int * numero_2;
void crea_variabili_dyn() {
if ( numero_2 == NULL) {
numero_2 = malloc(1 * sizeof(int));
*numero_2 = 2;
}
}
void incrementa() {
numero += 1;
*numero_2 += 11;
}
void ciao() {
crea_variabili_dyn();
stringa = "cane.huu";
incrementa();
printf("%d\n", numero);
printf("%d\n", *numero_2);
printf("%s\n", stringa);
incrementa();
printf("%d\n", numero);
printf("%d\n", *numero_2);
printf("%s\n", stringa);
}
void main (void) {
//printf("%d\n", numero);
ciao();
//printf("%d\n", numero);
}
I compile it:
gcc -shared -o playing_ctypes.so playing_ctypes.c
and then I play it with python:
import ctypes
testlib = ctypes.CDLL('/home/gnommaro/Dev/C/playing_ctypes.so')
c_numero = ctypes.c_int.in_dll(testlib, "numero")
c_numero_2 = ctypes.c_int.in_dll(testlib, "numero_2")
c_numero
c_numero_2
testlib.ciao()
c_numero
c_numero_2
As used to know c_numero is a integer and when called from python terminal it returns c_long(54)
Meanwhile c_numero_2 is a buffer allocated dynamically and when it's called it returns c_void_p(147438576) or c_long(147438576)
It depend byt the declared ctypes export type.
When I call testlib.ciao() everything goes ok, but if I'd like to increment, decrement or simply change arbitrarily the value of one of these ctypes integers I can override it in this way:
c_numero.value = 89
As we Seen for integer it works very well. But for the malloched variable c_number_2, the attribute .value returns me the address of buffer (?) and How could do if I'd like to change the value, the integer, in it ?
Or, in other worlds, how could export a pointer with ctypes and playing with his content value in a elegant way.
Probably I'd use a memcpy or write a sort of python.ctypes handler but, first of write some ugly hardcoding, I have to ask your help.
Is any there ? :)

You can index ctypes pointers just like in C, but don't write past the end of the buffer, just like in C.
from ctypes import *
dll = CDLL('msvcrt')
dll.malloc.restype = c_void_p
n = dll.malloc(5 * sizeof(c_int))
n = cast(n,POINTER(c_int))
for i in range(5):
print('uninitialized value',n[i])
n[i] = i
print('new value',n[i])
Output:
uninitialized value 6815752
new value 0
uninitialized value 0
new value 1
uninitialized value 6128720
new value 2
uninitialized value 0
new value 3
uninitialized value 0
new value 4
Note you can lie about malloc's restype to skip the cast:
dll.malloc.restype = POINTER(c_int)
Your global variable can be accessed like so:
c_numero_2 = POINTER(c_int).in_dll(testlib, "numero_2")
c_numero_2[0] = 1
c_numero_2[1] = 2

Ok, it seems that in three line of code I could do the trick
c_long_p = ctypes.POINTER(ctypes.c_long)
c_numero_2_ptr = ctypes.cast(c_numero_2.value, c_long_p)
c_numero_2_ptr.content
that's all :)
cheers

It's definitely the most elegant way to do this:
c_numero_2_content = (ctypes.c_long).from_address(c_numero_2.value)