Develop Reference

Why do I get a calling convention mismatch when passing a pointer to a slice with a cdecl Rust function and Python's Ctypes?

An apparent calling convention mismatch exists where the position and contents of arguments are incorrect when loading a small function using Python's Ctypes module.
In the example I built up while trying to get something working, one positional argument gets another's value while the other gets garbage.
The Ctypes docs state that cdll.LoadLibrary expects the cdecl convention. Resulting standard boilerplate:
# Tell Rustc to output a dynamically linked library
crate-type = ["cdylib"]
// Specify clean symbol and cdecl calling convention
#[no_mangle]
pub extern "cdecl" fn boring_function(
n: *mut size_t,
in_data: *mut [c_ulong],
out_data: *mut [c_double],
garbage: *mut [c_double],
) -> c_int {
//...
Loading our library after build...
lib = ctypes.CDLL("nothing/lib/playtoys.so")
lib.boring_function.restype = ctypes.c_int
Load the result into Python and call it with some initialized data
data_len = 8
in_array_t = ctypes.c_ulong * data_len
out_array_t = ctypes.c_double * data_len
in_array = in_array_t(7, 7, 7, 7, 7, 8, 7, 7)
out_array = out_array_t(10000.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9)
val = ctypes.c_size_t(data_len)
in_array_p = ctypes.byref(in_array)
out_array_p = ctypes.byref(out_array)
n_p = ctypes.byref(val)
garbage = n_p
res = boring_function(n_p,
in_array_p,
# garbage cannot be observed in any callee arg
ctypes.cast(garbage, ctypes.POINTER(out_array_t)),
out_array_p)
Notice the garbage parameter. It is so-named because it winds up containing a garbage address. Note that its position is swapped with out_array_p in the Python call and the Rust declaration.
[src/hello.rs:29] n = 0x00007f56dbce5bc0
[src/hello.rs:30] in_data = 0x00007f56f81e3270
[src/hello.rs:31] out_data = 0x00007f56f81e3230
[src/hello.rs:32] garbage = 0x000000000000000a
in_data, out_data, and n print the correct values in this configuration. The positional swap between garbage and out_data makes this possible.
Other examples using more or less arguments reveal similar patterns of intermediate ordered variables containing odd values that resemble addresses earlier in the program or unrelated garbage.
Either I'm missing something in how I set up the calling convention or some special magic in argtypes must be missing. So far I had no luck with changing the declared calling conventions or explicit argtypes. Are there any other knobs I should try turning?

in_data: *mut [c_ulong],
A slice is not a FFI-safe data type. Namely, Rust's slices use fat pointers, which take up two pointer-sized values.
You need to pass the data pointer and length as two separate arguments.
See also:
Why can comparing two seemingly equal pointers with == return false?
Rust functions with slice arguments in The Rust FFI Omnibus
The complete example from the Omnibus:
extern crate libc;
use libc::{uint32_t, size_t};
use std::slice;
#[no_mangle]
pub extern fn sum_of_even(n: *const uint32_t, len: size_t) -> uint32_t {
let numbers = unsafe {
assert!(!n.is_null());
slice::from_raw_parts(n, len as usize)
};
let sum =
numbers.iter()
.filter(|&v| v % 2 == 0)
.fold(0, |acc, v| acc + v);
sum as uint32_t
}
#!/usr/bin/env python3
import sys, ctypes
from ctypes import POINTER, c_uint32, c_size_t
prefix = {'win32': ''}.get(sys.platform, 'lib')
extension = {'darwin': '.dylib', 'win32': '.dll'}.get(sys.platform, '.so')
lib = ctypes.cdll.LoadLibrary(prefix + "slice_arguments" + extension)
lib.sum_of_even.argtypes = (POINTER(c_uint32), c_size_t)
lib.sum_of_even.restype = ctypes.c_uint32
def sum_of_even(numbers):
buf_type = c_uint32 * len(numbers)
buf = buf_type(*numbers)
return lib.sum_of_even(buf, len(numbers))
print(sum_of_even([1,2,3,4,5,6]))
Disclaimer: I am the primary author of the Omnibus

Ctypes DLL call ArgumentError with c_char_Array

i'm trying to write a Python DLL Wrapper for a C Project
https://github.com/OpenEtherCATsociety/SOEM -> C Project
The Python DLL Wrapper can be found here:
https://github.com/GitHubStefan13/SOEM-for-Python
Original C Code that is important:
char IOmap[4096];
int ec_config_overlap(uint8 usetable, void *pIOmap);
Python Wrapper
IOMap = ctypes.POINTER(ctypes.c_char * 4096)
c_ec_config_overlap = ethercat.ec_config_overlap
c_ec_config_overlap.argtypes = [ctypes.c_unit8, IOMap]
c_ec_config_overlap.restype = ctypes.c_int
When im trying to define a function in Python
def ec_config_overlap(usetable, PIOMap):
return c_ec_config_overlap(usetable, PIOMap
and call it.
I receive the error
ctypes.ArgumentError: argument 2: : expected LPc_char_Array_4096 instance instead of _ctypes.PyPointerType.
I understand the Error but how do i go around to actually make the ctype a Array[4096] instead of the PyPointerType?

This syntax creates array instances:
>>> import ctypes
>>> (ctypes.c_char*4096)()
<__main__.c_char_Array_4096 object at 0x0000024D84E2D7C8>
Since it is a char array, you can also use:
>>> create_string_buffer(4096)
<__main__.c_char_Array_4096 object at 0x0000025AE48FE948>
The type of your function should be:
c_ec_config_overlap.argtypes = [ctypes.c_uint8, ctypes.c_void_p]
But for better type checking you can also use:
c_ec_config_overlap.argtypes = [ctypes.c_uint8, ctypes.c_char_p]

Pass ByteArray from Python to C function

I would like to pass a ByteArray variable from my Python program to my DLL written in C in order to accelerate some specific processing which is too slow in Python. I have gone through the Web, tried Ctypes with combinations of byref, cast, memoryviews, addressof, but nothing works. Is there any simple way to achieve this without copying my ByteArray into something else that will pass ?
Here is what I am trying to do:
/* My C DLL */
__declspec(dllexport) bool FastProc(char *P, int L)
{
/* Do some complex processing on the char buffer */
;
return true;
}
# My Python program
from ctypes import *
def main(argv):
MyData = ByteArray([1,2,3,4,5,6])
dll = CDLL('CHELPER.dll')
dll.FastProc.argtypes = (c_char_p, c_int)
dll.FastProc.restype = c_bool
Result = dll.FastProc(MyData, len(MyData))
print(Result)
But I get a type error when passing the first parameter (MyData) to the C function.
Is there any solution that doesn't require too much overhead that would waste the benefits of my C function ?
Olivier

I'll assume that ByteArray is supposed to be bytearray. We can use create_string_buffer to create a mutable character buffer which is a ctypes array of c_char. But create_string_buffer will not accept a bytearray, we need to pass it a bytes object to initialize it; fortunately, casting between bytes and bytearray is fast and efficient.
I don't have your DLL, so to test that the array behaves correctly I'll use the libc.strfry function to shuffle its chars.
from ctypes import CDLL, create_string_buffer
libc = CDLL("libc.so.6")
# Some test data, NUL-terminated so we can safely pass it to a str function.
mydata = bytearray([65, 66, 67, 68, 69, 70, 0])
print(mydata)
# Convert the Python bytearray to a C array of char
p = create_string_buffer(bytes(mydata), len(mydata))
#Shuffle the bytes before the NUL terminator byte, in-place.
libc.strfry(p)
# Convert the modified C array back to a Python bytearray
newdata = bytearray(p.raw)
print(newdata)
typical output
bytearray(b'ABCDEF\x00')
bytearray(b'BFDACE\x00')

Saving XML as ctypes c_byte in variable gives TypeError: an integer is required

In C header file I have :
long TEST_API test (
___OUT_ char DisplayText[41],
_IN____ const char XMLparams[2049]
);
In python code I have imported ctypes and I am trying to call "Test".
class A(Structure):
_fields_ = [("DisplayText", c_byte*41),
("XMLparams",c_byte*2049)
]
XMLparamsVal = (ctypes.c_byte*2049)(["<xml><MatchboxDataProviderValue>Openvez</MatchboxDataProviderValue><AlwaysPrintTwoTicketsFlag>FALSE</AlwaysPrintTwoTicketsFlag><DisplayWidthInCharacters>20</DisplayWidthInCharacters><!-- exclude Ikea-Card and Maestro from PAN truncation --><DontTruncateList>*119*1*</DontTruncateList></xml>"])
my_A = A("", XMLparamsVal)
lib.test(my_A.DisplayText, my_A.XMLparams)
I am gettting this error:
XMLparamsVal = (ctypes.c_byte*2049)(["<xml><MatchboxDataProviderValue>Openvez</MatchboxDataProviderValue><AlwaysPrintTwoTicketsFlag>FALSE</AlwaysPrintTwoTicketsFlag><DisplayWidthInCharacters>20</DisplayWidthInCharacters><!-- exclude Ikea-Card and Maestro from PAN truncation --><DontTruncateList>*119*1*</DontTruncateList></xml>"])
TypeError: an integer is required
How can I fix the issue. Thanks!

A c_byte array takes a variable number of ints as argument, you're trying to give it a list. Try this instead:
xml_bytes = bytearray(b'<xml>...')
XMLparamsVal = (ctypes.c_byte*2049)(*xml_bytes)
*xml_bytes is expanded to a series of positional int arguments.
For python3, you wouldn't need the bytearray, you can use a byte literal directly, as in python3 iterating a byte object yields ints:
XMLparamsVal = (ctypes.c_byte*2049)(*b'<xml>...')
Note that for the first argument of your A class, you'll also have to pass a c_byte_Array_41, not a string.

ctype question char**

I'm trying to figure out why this works after lots and lots of messing about with
obo.librar_version is a c function which requires char ** as the input and does a strcpy
to passed in char.
from ctypes import *
_OBO_C_DLL = 'obo.dll'
STRING = c_char_p
OBO_VERSION = _stdcall_libraries[_OBO_C_DLL].OBO_VERSION
OBO_VERSION.restype = c_int
OBO_VERSION.argtypes = [POINTER(STRING)]
def library_version():
s = create_string_buffer('\000' * 32)
t = cast(s, c_char_p)
res = obo.library_version(byref(t))
if res != 0:
raise Error("OBO error %r" % res)
return t.value, s.raw, s.value
library_version()
The above code returns
('OBO Version 1.0.1', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', '')
What I don't understand is why 's' does not have any value? Anyone have any ideas? Thx

When you cast s to c_char_p you store a new object in t, not a reference. So when you pass t to your function by reference, s doesn't get updated.
UPDATE:
You are indeed correct:
cast takes two parameters, a ctypes
object that is or can be converted to
a pointer of some kind, and a ctypes
pointer type. It returns an instance
of the second argument, which
references the same memory block as
the first argument.
In order to get a reference to your string buffer, you need to use the following for your cast:
t = cast(s, POINTER(c_char*33))
I have no idea why c_char_p doesn't create a reference where this does, but there you go.

Because library_version requires a char**, they don't want you to allocate the characters (as you're doing with create_string_buffer. Instead, they just want you to pass in a reference to a pointer so they can return the address of where to find the version string.
So all you need to do is allocate the pointer, and then pass in a reference to that pointer.
The following code should work, although I don't have obo.dll (or know of a suitable replacement) to test it.
from ctypes import *
_OBO_C_DLL = 'obo.dll'
STRING = c_char_p
_stdcall_libraries = dict()
_stdcall_libraries[_OBO_C_DLL] = WinDLL(_OBO_C_DLL)
OBO_VERSION = _stdcall_libraries[_OBO_C_DLL].OBO_VERSION
OBO_VERSION.restype = c_int
OBO_VERSION.argtypes = [POINTER(STRING)]
def library_version():
s_res = c_char_p()
res = OBO_VERSION(byref(s_res))
if res != 0:
raise Error("OBO error %r" % res)
return s_res.value
library_version()
[Edit]
I've gone a step further and written my own DLL that implements a possible implementation of OBO_VERSION that does not require an allocated character buffer, and is not subject to any memory leaks.
int OBO_VERSION(char **pp_version)
{
static char result[] = "Version 2.0";
*pp_version = result;
return 0; // success
}
As you can see, OBO_VERSION simply sets the value of *pp_version to a pointer to a null-terminated character array. This is likely how the real OBO_VERSION works. I've tested this against my originally suggested technique above, and it works as prescribed.