I would like to pass a ByteArray variable from my Python program to my DLL written in C in order to accelerate some specific processing which is too slow in Python. I have gone through the Web, tried Ctypes with combinations of byref, cast, memoryviews, addressof, but nothing works. Is there any simple way to achieve this without copying my ByteArray into something else that will pass ?
Here is what I am trying to do:
/* My C DLL */
__declspec(dllexport) bool FastProc(char *P, int L)
{
/* Do some complex processing on the char buffer */
;
return true;
}
# My Python program
from ctypes import *
def main(argv):
MyData = ByteArray([1,2,3,4,5,6])
dll = CDLL('CHELPER.dll')
dll.FastProc.argtypes = (c_char_p, c_int)
dll.FastProc.restype = c_bool
Result = dll.FastProc(MyData, len(MyData))
print(Result)
But I get a type error when passing the first parameter (MyData) to the C function.
Is there any solution that doesn't require too much overhead that would waste the benefits of my C function ?
Olivier
I'll assume that ByteArray is supposed to be bytearray. We can use create_string_buffer to create a mutable character buffer which is a ctypes array of c_char. But create_string_buffer will not accept a bytearray, we need to pass it a bytes object to initialize it; fortunately, casting between bytes and bytearray is fast and efficient.
I don't have your DLL, so to test that the array behaves correctly I'll use the libc.strfry function to shuffle its chars.
from ctypes import CDLL, create_string_buffer
libc = CDLL("libc.so.6")
# Some test data, NUL-terminated so we can safely pass it to a str function.
mydata = bytearray([65, 66, 67, 68, 69, 70, 0])
print(mydata)
# Convert the Python bytearray to a C array of char
p = create_string_buffer(bytes(mydata), len(mydata))
#Shuffle the bytes before the NUL terminator byte, in-place.
libc.strfry(p)
# Convert the modified C array back to a Python bytearray
newdata = bytearray(p.raw)
print(newdata)
typical output
bytearray(b'ABCDEF\x00')
bytearray(b'BFDACE\x00')
Related
I am new to both C and ctypes, but I cannot seem to find an answer on how to do this, particularly with a numpy array.
C Code
// Import/Export Macros
#define DllImport __declspec( dllimport )
#define DllExport __declspec( dllexport )
// Test function for receiving and transmitting arrays
extern "C"
DllExport void c_fun(char **string_array)
{
string_array[0] = "foo";
string_array[1] = "bar";
string_array[2] = "baz";
}
Python Code
import numpy as np
import ctypes
# Load the DLL library...
# Define function argtypes
lib.c_fun.argtypes = [np.ctypeslib.ndpointer(ctypes.c_char, ndim = 2, flags="C_CONTIGUOUS")]
# Initialize, call, and print
string_array = np.empty((3,10),dtype=ctypes.c_char)
lib.c_fun(string_array)
print(string_array)
I am sure there is some encoding/decoding that needs to happen as well, but I am not sure how/which. Thanks!
Addressing the C code part of the question only...
As noted in comments, C does not allow assignments in this way if the three variables shown are defined as char arrays:
string_array[0] = "foo";
string_array[1] = "bar";
string_array[2] = "baz";
Use the following:
strcpy(string_array[0], "foo");
strcpy(string_array[1], "bar");
strcpy(string_array[2], "baz");
And as long as the caller to this function is pre-allocating and freeing memory for the buffers, this part of the solution is now at least syntactically correct.
But if the strings do indeed need to be immutable to be compatible with Python, then in the caller function allocate memory to create char **string_array such that you can pass an array of 3 pointers as the argument. For example:
char **string_array = malloc(3*sizeof(*string_array));//creates array of 3 pointers.
Then call it as:
c_fun(string_array);
This allows use of the DLL call just as shown in in your original post.:
DllExport void c_fun(char **string_array)
{
//array of pointers being assigned to addresses of 3 string literals
string_array[0] = "foo";//these will now be immutable strings
string_array[1] = "bar";
string_array[2] = "baz";
}
Which data type should be used in this stringUpcase function in my DLL file
void __cdecl stringUpcase(char IN_str[], char OUT_str[], int32_t len);
I am trying ctype.c_char_p in Python 3.6 and This function should return the uppercase string "HELO" in OUT_str variable.
dl = ctypes.cdll.LoadLibrary("path/of/dll/file.dll")
IN_str = 'helo'
OUT_str = ''
IN_len = len(IN_str)
dl.stringUpcase.restype = ctypes.c_void_p
dl.stringUpcase.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_int32]
dl.stringUpcase(IN_str, OUT_str, IN_len);
Console error output is
line 21, in <module>
dl.stringUpcase(IN_str, OUT_str, IN_len);
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type
Thank you for any help you can provide.
You are trying to pass python string as an argument of type c_char_p. Python3 unlike python2 is pretty strict about typing in ctypes.
The working sample would look like this.
import ctypes
ins = ctypes.c_char_p(b'helo')
lens = len(ins.value)
outs = ctypes.create_string_buffer(lens+1)
lib = ctypes.cdll.LoadLibrary("./upper.so")
lib.stringUpcase.restype = None
lib.stringUpcase.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_int32]
lib.stringUpcase(ins, outs, lens)
print(outs.value)
Pay attention to the fact that c_char_p points to a zero-terminated string, so len argument is redundant here. Moreover to construct c_char_p you need to pass bytes object or and integer address rather than just a string, to be able to use string as an argument you need to use c_wchar_p type and wchar_t* in your library respectively.
One more thing to pay attention to is the fact that your C function does not allocate memory so you need for outs to be large enough to contain the result.
I have a C function that must be callable from C and Python.
I'm having trouble figuring out how to pass a python list of c-type structs,
each of which contains several nested structs, to the c function.
A single one of these structs looks like this in python:
class STATION_MM_NODE(ctypes.Structure):
_fields_ = [
("signal", MM_STRUCT),
("noise", MM_STRUCT),
("signalWindowLen", ctypes.c_double),
("metadata", SAC_PZ)
]
And like this in C:
typedef struct stationMMnode {
struct mantleMagStruct *signal;
struct mantleMagStruct *noise;
double signalWindowLen;
SAC_PZ metadata;
} stationMMnode_t;
The c function that takes an array of stationMMnode structs is callable as:
double magnitudeCompute_Mw_Mm_Event(stationMMnode_t **stationMMarray, int numStations);
For instance, I can call it purely from C as in:
int testfunc() {
stationMMnode_t *node1 = malloc(sizeof(struct stationMMnode));
node1->signalWindowLen = 500;
stationMMnode_t *node2 = malloc(sizeof(struct stationMMnode));
node2->signalWindowLen = 100;
struct stationMMnode *nodes[2];
nodes[0] = node1;
nodes[1] = node2;
magnitudeCompute_Mw_Mm_Event(nodes, 2); // Works!
}
In python, I can create a list of nodes that looks similar to the c array of structs:
stationMMnodes = []
...
node = get_stationMMnode() # Returns a STATION_MM_NODE
node.signal = mm_signal
node.noise = mm_noise
node.metadata = sacPoleZero
stationMMnodes.append(node)
...
wrap_lib.magnitudeCompute_Mw_Mm_Event(stationMMnodes, numStations) # Does NOT work
where I've defined the argtypes as:
wrap_lib.magnitudeCompute_Mw_Mm_Event.argtypes =
[ctypes.POINTER(STATION_MM_NODE), ctypes.c_int ]
The model I'm using above (passing a ctype pointer to a c-style struct to a c function that takes a pointer to struct) seems to work fine when I am passing in a pointer to a single struct, however, for a pointer to an array of structs, it seems to break down. In addition, I am uncertain of what the python memory layout is for a list of structs versus an array of pointers to struct (as the C function is expecting).
Any help would be greatly appreciated!
Update: I found the following link very helpful:
python ctypes array of structs
I solved my problem by:
1. Declaring an array of pointers to my struct:
nodeArrayType = ctypes.POINTER(STATION_MM_NODE) * 1024
nodeArray = nodeArrayType()
nstn = 0
2. Writing a C function to join the member structs into a larger struct (=a node) and return a pointer to that struct - which is stored in nodeArray[].
nodeArray[nstn] = wrap_libmth.libmth.makeNode(node.signal, node.noise, node.metadata)
nstn += 1
3. Fixing the argtype of the C function that receives the pointer to struct array:
wrap_libmth.libmth.magnitudeCompute_Mw_Mm_Event.argtypes = [ctypes.POINTER(ctypes.POINTER(STATION_MM_NODE)), ctypes.c_int]
So ... I have it working, but like most thing with Python, I feel like I'm holding the tiger by the tail as I don't fully understand exactly why it works and what (better) alternatives would be (e.g., the C hack makeNode() to return a pointer to a STATION_MM_NODE struct is less than satisfactory - it would be better to generate this struct fully in python).
I have a self-made C library that I want to access using python. The problem is that the code consists essentially of two parts, an initialization to read in data from a number of files and a few calculations that need to be done only once. The other part is called in a loop and uses the data generated before repeatedly. To this function I want to pass parameters from python.
My idea was to write two C wrapper functions, "init" and "loop" - "init" reads the data and returns a void pointer to a structure that "loop" can use together with additional parameters that I can pass on from python. Something like
void *init() {
struct *mystruct ret = (mystruct *)malloc(sizeof(mystruct));
/* Fill ret with data */
return ret;
}
float loop(void *data, float par1, float par2) {
/* do stuff with data, par1, par2, return result */
}
I tried calling "init" from python as a c_void_p, but since "loop" changes some of the contents of "data" and ctypes' void pointers are immutable, this did not work.
Other solutions to similar problems I saw seem to require knowledge of how much memory "init" would use, and I do not know that.
Is there a way to pass data from one C function to another through python without telling python exactly what or how much it is? Or is there another way to solve my problem?
I tried (and failed) to write a minimum crashing example, and after some debugging it turned out there was a bug in my C code. Thanks to everyone who replied!
Hoping that this might help other people, here is a sort-of-minimal working version (still without separate 'free' - sorry):
pybug.c:
#include <stdio.h>
#include <stdlib.h>
typedef struct inner_struct_s {
int length;
float *array;
} inner_struct_t;
typedef struct mystruct_S {
int id;
float start;
float end;
inner_struct_t *inner;
} mystruct_t;
void init(void **data) {
int i;
mystruct_t *mystruct = (mystruct_t *)malloc(sizeof(mystruct_t));
inner_struct_t *inner = (inner_struct_t *)malloc(sizeof(inner_struct_t));
inner->length = 10;
inner->array = calloc(inner->length, sizeof(float));
for (i=0; i<inner->length; i++)
inner->array[i] = 2*i;
mystruct->id = 0;
mystruct->start = 0;
mystruct->end = inner->length;
mystruct->inner = inner;
*data = mystruct;
}
float loop(void *data, float par1, float par2, int newsize) {
mystruct_t *str = data;
inner_struct_t *inner = str->inner;
int i;
inner->length = newsize;
inner->array = realloc(inner->array, newsize * sizeof(float));
for (i=0; i<inner->length; i++)
inner->array[i] = par1 + i * par2;
return inner->array[inner->length-1];
}
compile as
cc -c -fPIC pybug.c
cc -shared -o libbug.so pybug.o
Run in python:
from ctypes import *
sl = CDLL('libbug.so')
# What arguments do functions take / return?
sl.init.argtype = c_void_p
sl.loop.restype = c_float
sl.loop.argtypes = [c_void_p, c_float, c_float, c_int]
# Init takes a pointer to a pointer
px = c_void_p()
sl.init(byref(px))
# Call the loop a couple of times
for i in range(10):
print sl.loop(px, i, 5, 10*i+5)
You should have a corresponding function to free the data buffer when the caller is done. Otherwise I don't see the issue. Just pass the pointer to loop that you get from init.
init.restype = c_void_p
loop.argtypes = [c_void_p, c_float, c_float]
loop.restype = c_float
I'm not sure what you mean by "ctypes' void pointers are immutable", unless you're talking about c_char_p and c_wchar_p. The issue there is if you pass a Python string as an argument it uses Python's private pointer to the string buffer. If a function can change the string, you should first copy it to a c_char or c_wchar array.
Here's a simple example showing the problem of passing a Python string (2.x byte string) as an argument to a function that modifies it. In this case it changes index 0 to '\x00':
>>> import os
>>> from ctypes import *
>>> open('tmp.c', 'w').write("void f(char *s) {s[0] = 0;}")
>>> os.system('gcc -shared -fPIC -o tmp.so tmp.c')
0
>>> tmp = CDLL('./tmp.so')
>>> tmp.f.argtypes = [c_void_p]
>>> tmp.f.restype = None
>>> tmp.f('a')
>>> 'a'
'\x00'
>>> s = 'abc'
>>> tmp.f(s)
>>> s
'\x00bc'
This is specific to passing Python strings as arguments. It isn't a problem to pass pointers to data structures that are intended to be mutable, either ctypes data objects such as a Structure, or pointers returned by libraries.
Is your C code in a DLL? If so can might consider creating a global pointer in there. init() will do any initialization required and set the pointer equal to newly allocated memory and loop() will operate on that memory. Also don't forget to free it up with a close() function
I'm trying to figure out why this works after lots and lots of messing about with
obo.librar_version is a c function which requires char ** as the input and does a strcpy
to passed in char.
from ctypes import *
_OBO_C_DLL = 'obo.dll'
STRING = c_char_p
OBO_VERSION = _stdcall_libraries[_OBO_C_DLL].OBO_VERSION
OBO_VERSION.restype = c_int
OBO_VERSION.argtypes = [POINTER(STRING)]
def library_version():
s = create_string_buffer('\000' * 32)
t = cast(s, c_char_p)
res = obo.library_version(byref(t))
if res != 0:
raise Error("OBO error %r" % res)
return t.value, s.raw, s.value
library_version()
The above code returns
('OBO Version 1.0.1', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', '')
What I don't understand is why 's' does not have any value? Anyone have any ideas? Thx
When you cast s to c_char_p you store a new object in t, not a reference. So when you pass t to your function by reference, s doesn't get updated.
UPDATE:
You are indeed correct:
cast takes two parameters, a ctypes
object that is or can be converted to
a pointer of some kind, and a ctypes
pointer type. It returns an instance
of the second argument, which
references the same memory block as
the first argument.
In order to get a reference to your string buffer, you need to use the following for your cast:
t = cast(s, POINTER(c_char*33))
I have no idea why c_char_p doesn't create a reference where this does, but there you go.
Because library_version requires a char**, they don't want you to allocate the characters (as you're doing with create_string_buffer. Instead, they just want you to pass in a reference to a pointer so they can return the address of where to find the version string.
So all you need to do is allocate the pointer, and then pass in a reference to that pointer.
The following code should work, although I don't have obo.dll (or know of a suitable replacement) to test it.
from ctypes import *
_OBO_C_DLL = 'obo.dll'
STRING = c_char_p
_stdcall_libraries = dict()
_stdcall_libraries[_OBO_C_DLL] = WinDLL(_OBO_C_DLL)
OBO_VERSION = _stdcall_libraries[_OBO_C_DLL].OBO_VERSION
OBO_VERSION.restype = c_int
OBO_VERSION.argtypes = [POINTER(STRING)]
def library_version():
s_res = c_char_p()
res = OBO_VERSION(byref(s_res))
if res != 0:
raise Error("OBO error %r" % res)
return s_res.value
library_version()
[Edit]
I've gone a step further and written my own DLL that implements a possible implementation of OBO_VERSION that does not require an allocated character buffer, and is not subject to any memory leaks.
int OBO_VERSION(char **pp_version)
{
static char result[] = "Version 2.0";
*pp_version = result;
return 0; // success
}
As you can see, OBO_VERSION simply sets the value of *pp_version to a pointer to a null-terminated character array. This is likely how the real OBO_VERSION works. I've tested this against my originally suggested technique above, and it works as prescribed.