Develop Reference

Python ctypes misbehaves when a C function returns a dynamic array

I'm working on Python wrapper classes for Matlab's dynamic libraries to read Matlab MAT files in Python, and I'm encountering an odd behavior that I cannot explain from the ctypes interface.
The C function signature looks like this:
const mwSize *mxGetDimensions(const mxArray *);
Here, mwSize is a renamed size_t, and mxArray* is an opaque pointer. This function returns the "shape" of the Matlab array. The returned pointer points to the size_t array, which is stored internally within mxArray object and is not null terminated (its size is obtained via another function).
To call this function from Python, I set up the library as follows:
libmx = ctypes.cdll.LoadLibrary('libmx.dll')
libmx.mxGetDimensions.restype = ctypes.POINTER(ctypes.c_size_t)
libmx.mxGetDimensions.argtypes = [ctypes.c_void_p]
and after obtained mxArray* in VAR, I called:
dims = libmx.mxGetDimensions(VAR)
print(dims[0],dims[1])
VAR is known to be 2-D and has a shape of (1, 13) (validated with a C program) but my Python code returns (55834574849 0) in c_ulonglong... Results are consistently garbage across all the variables stored in the test MAT file.
What am I doing wrong? Other library calls using VAR seems be working properly, so VAR is pointing to the valid object. As stated above, mxGetDimensions() called in a C program works as expected.
Any inputs would be appreciated! Thanks

#Neitsa solved my immediate issue in his comment under the OP, and further investigation of libmx.dll resolved the remaining discrepancy between Python & C versions.
Because Matlab's libmx.dll goes back really long time as it was first written in the 32-bit era, the DLL contains multiple versions of its functions for backward compatibility. As it turned out, const mwSize *mxGetDimensions(const mxArray *); is the oldest version of the function, and the associated C header file (matrix.h) has the line #define mxGetDimensions mxGetDimensions_800 to override the function with its newest version. Obviously, Python's ctypes doesn't check the C header file; so, it was left for me to sift through the header file figuring out which version of the function to use.
In the end, the proper behavior with POINTER(c_size_t) was obtained when I changed my code to:
libmx = ctypes.cdll.LoadLibrary('libmx.dll')
libmx.mxGetDimensions_800.restype = ctypes.POINTER(ctypes.c_size_t)
libmx.mxGetDimensions_800.argtypes = [ctypes.c_void_p]
dims = libmx.mxGetDimensions_800(VAR)
So, there you have it: Study the associated header file thoroughly if you are wrapping a 3rd-party dynamic/shared library.

how to include shared object in python [duplicate]

I'm just getting started with ctypes and would like to use a C++ class that I have exported in a dll file from within python using ctypes.
So lets say my C++ code looks something like this:
class MyClass {
public:
int test();
...
I would know create a .dll file that contains this class and then load the .dll file in python using ctypes.
Now how would I create an Object of type MyClass and call its test function? Is that even possible with ctypes? Alternatively I would consider using SWIG or Boost.Python but ctypes seems like the easiest option for small projects.

Besides Boost.Python(which is probably a more friendly solution for larger projects that require one-to-one mapping of C++ classes to python classes), you could provide on the C++ side a C interface. It's one solution of many so it has its own trade offs, but I will present it for the benefit of those who aren't familiar with the technique. For full disclosure, with this approach one wouldn't be interfacing C++ to python, but C++ to C to Python. Below I included an example that meets your requirements to show you the general idea of the extern "c" facility of C++ compilers.
//YourFile.cpp (compiled into a .dll or .so file)
#include <new> //For std::nothrow
//Either include a header defining your class, or define it here.
extern "C" //Tells the compile to use C-linkage for the next scope.
{
//Note: The interface this linkage region needs to use C only.
void * CreateInstanceOfClass( void )
{
// Note: Inside the function body, I can use C++.
return new(std::nothrow) MyClass;
}
//Thanks Chris.
void DeleteInstanceOfClass (void *ptr)
{
delete(std::nothrow) ptr;
}
int CallMemberTest(void *ptr)
{
// Note: A downside here is the lack of type safety.
// You could always internally(in the C++ library) save a reference to all
// pointers created of type MyClass and verify it is an element in that
//structure.
//
// Per comments with Andre, we should avoid throwing exceptions.
try
{
MyClass * ref = reinterpret_cast<MyClass *>(ptr);
return ref->Test();
}
catch(...)
{
return -1; //assuming -1 is an error condition.
}
}
} //End C linkage scope.
You can compile this code with
gcc -shared -o test.so test.cpp
#creates test.so in your current working directory.
In your python code you could do something like this (interactive prompt from 2.7 shown):
>>> from ctypes import cdll
>>> stdc=cdll.LoadLibrary("libc.so.6") # or similar to load c library
>>> stdcpp=cdll.LoadLibrary("libstdc++.so.6") # or similar to load c++ library
>>> myLib=cdll.LoadLibrary("/path/to/test.so")
>>> spam = myLib.CreateInstanceOfClass()
>>> spam
[outputs the pointer address of the element]
>>> value=CallMemberTest(spam)
[does whatever Test does to the spam reference of the object]
I'm sure Boost.Python does something similar under the hood, but perhaps understanding the lower levels concepts is helpful. I would be more excited about this method if you were attempting to access functionality of a C++ library and a one-to-one mapping was not required.
For more information on C/C++ interaction check out this page from Sun: http://dsc.sun.com/solaris/articles/mixing.html#cpp_from_c

The short story is that there is no standard binary interface for C++ in the way that there is for C. Different compilers output different binaries for the same C++ dynamic libraries, due to name mangling and different ways to handle the stack between library function calls.
So, unfortunately, there really isn't a portable way to access C++ libraries in general. But, for one compiler at a time, it's no problem.
This blog post also has a short overview of why this currently won't work. Maybe after C++0x comes out, we'll have a standard ABI for C++? Until then, you're probably not going to have any way to access C++ classes through Python's ctypes.

The answer by AudaAero is very good but not complete (at least for me).
On my system (Debian Stretch x64 with GCC and G++ 6.3.0, Python 3.5.3) I have segfaults as soon has I call a member function that access a member value of the class.
I diagnosticated by printing pointer values to stdout that the void* pointer coded on 64 bits in wrappers is being represented on 32 bits in Python. Thus big problems occurs when it is passed back to a member function wrapper.
The solution I found is to change:
spam = myLib.CreateInstanceOfClass()
Into
Class_ctor_wrapper = myLib.CreateInstanceOfClass
Class_ctor_wrapper.restype = c_void_p
spam = c_void_p(Class_ctor_wrapper())
So two things were missing: setting the return type to c_void_p (the default is int) and then creating a c_void_p object (not just an integer).
I wish I could have written a comment but I still lack 27 rep points.

Extending AudaAero's and Gabriel Devillers answer I would complete the class object instance creation by:
stdc=c_void_p(cdll.LoadLibrary("libc.so.6"))
using ctypes c_void_p data type ensures the proper representation of the class object pointer within python.
Also make sure that the dll's memory management be handled by the dll (allocated memory in the dll should be deallocated also in the dll, and not in python)!

I ran into the same problem. From trial and error and some internet research (not necessarily from knowing the g++ compiler or C++ very well), I came across this particular solution that seems to be working quite well for me.
//model.hpp
class Model{
public:
static Model* CreateModel(char* model_name) asm("CreateModel"); // static method, creates an instance of the class
double GetValue(uint32_t index) asm("GetValue"); // object method
}
#model.py
from ctypes import ...
if __name__ == '__main__':
# load dll as model_dll
# Static Method Signature
fCreateModel = getattr(model_dll, 'CreateModel') # or model_dll.CreateModel
fCreateModel.argtypes = [c_char_p]
fCreateModel.restype = c_void_p
# Object Method Signature
fGetValue = getattr(model_dll, 'GetValue') # or model_dll.GetValue
fGetValue.argtypes = [c_void_p, c_uint32] # Notice two Params
fGetValue.restype = c_double
# Calling the Methods
obj_ptr = fCreateModel(c_char_p(b"new_model"))
val = fGetValue(obj_ptr, c_int32(0)) # pass in obj_ptr as first param of obj method
>>> nm -Dg libmodel.so
U cbrt#GLIBC_2.2.5
U close#GLIBC_2.2.5
00000000000033a0 T CreateModel # <----- Static Method
U __cxa_atexit#GLIBC_2.2.5
w __cxa_finalize#GLIBC_2.2.5
U fprintf#GLIBC_2.2.5
0000000000002b40 T GetValue # <----- Object Method
w __gmon_start__
...
...
... # Mangled Symbol Names Below
0000000000002430 T _ZN12SHMEMWrapper4HashEPKc
0000000000006120 B _ZN12SHMEMWrapper8info_mapE
00000000000033f0 T _ZN5Model12DestroyModelEPKc
0000000000002b20 T _ZN5Model14GetLinearIndexElll
First, I was able to avoid the extern "C" directive completely by instead using the asm keyword which, to my knowledge, asks the compiler to use a given name instead of the generated one when exporting the function to the shared object lib's symbol table. This allowed me to avoid the weird symbol names that the C++ compiler generates automatically. They look something like the _ZN1... pattern you see above. Then in a program using Python ctypes, I was able to access the class functions directly using the custom name I gave them. The program looks like fhandle = mydll.myfunc or fhandler = getattr(mydll, 'myfunc') instead of fhandle = getattr(mydll, '_ZN12...myfunc...'). Of course, you could just use the long name; it would make no difference, but I figure the shorter name is a little cleaner and doesn't require using nm to read the symbol table and extract the names in the first place.
Second, in the spirit of Python's style of object oriented programming, I decided to try passing in my class' object pointer as the first argument of the class object method, just like when we pass self in as the first method in Python object methods. To my surprise, it worked! See the Python section above. Apparently, if you set the first argument in the fhandle.argtypes argument to c_void_ptr and pass in the ptr you get from your class' static factory method, the program should execute cleanly. Class static methods seem to work as one would expect like in Python; just use the original function signature.
I'm using g++ 12.1.1, python 3.10.5 on Arch Linux. I hope this helps someone.

Passing structure with pointers to other structures in ctypes

I am trying to make a python wrapper for a C library using ctypes. The library has functions which require a pointer to a structure to be passed, which acts as a handle for future calls.
This structure has pointers to another internal structure that further has pointers to other structures.
typedef struct varnam {
char *scheme_file;
char *suggestions_file;
struct varnam_internal *internal;
} varnam;
The varnam_internal structure has pointers to an sqlite database and so forth
struct varnam_internal
{
sqlite3 *db;
sqlite3 *known_words;
struct varray_t *r;
struct token *v;
...
}
I tried ignoring the varnam_internal structure according to this SO answer. Something like
class Varnam(Structure):
__fields__ = [("scheme_file",c_char_p),
("suggestions_file",c_char_p),("internal",c_void_p)]
But this does not seem to work because I think the library needs to allocate varnam_internal for functioning properly.
Should I implement all the dependent structures in python? Is ctypes suitable for wrapping libraries like this? I have read about alternatives like Cython but I have no experience with Cython so is this doable in it?

There's no reason define the varnam_internal structure in ctypes because you should have no need to access it. The library you're calling will allocate it regardless of whether you define the structure or not. Whatever problem you're encountering it's not because you didn't define the structure in ctypes.
Make sure you're calling varnam_init correctly. It uses pointers to pointers as arguments, which means you can't just use your Varnam class directly. You'll want do something like this:
from ctypes import *
class Varnam(Structure):
__fields__ = [("scheme_file",c_char_p),
("suggestions_file",c_char_p),
("internal",c_void_p)]
varnam_ptr = POINTER(Varnam)
libvarnam = cdll.LoadLibrary("libvarnam.so") # on Linux
# libvarnam = cdll.libvarnam # on Windows
varnam_init = libvarnam.varnam_init
varnam_init.argtypes = [c_char_p, POINTER(varnam_ptr), POINTER(c_char_p)]
def my_varnam_init(scheme_file):
handle = varnam_ptr()
msg = c_char_p()
r = varnam_init(scheme_file. handle.byref(), msg.byref())
if r != 0:
raise Exception(msg)
return handle
The above code is completely untested, but shows you how you should be calling varnam_init.

Python call by reference or equivalent

I am trying to use SSL library in Python and I need to write a clone code of a C++ code.
In C SSL function is called like this:
EVP_MD_CTX mdctx;
EVP_MD_CTX_init(&mdctx)
In Python I reached SSL library:
import ctypes
import ctypes.util
import platform
from os import linesep
libraries = {}
libraries["c"] = ctypes.CDLL(ctypes.util.find_library("c"))
if platform.system() != "Windows":
libraries["ssl"] = ctypes.CDLL(ctypes.util.find_library("ssl"))
else:
libraries["ssl"] = ctypes.CDLL(ctypes.util.find_library("libeay32"))
EVP_MD_CTX_init = libraries['ssl'].EVP_MD_CTX_init
EVP_MD_CTX_init.restype = ctypes.c_void_p
EVP_MD_CTX_init.argtypes = [ctypes.c_void_p]
Which is taken from here
How may I pass EVP_MD_CTX_init function by reference as it is made in C?
Since I have no access to EVP_MD_CTX_init() method.I have to call this method with 'pass by reference'
I tried EVP_MD_CTX_init(ctypes.byref(mdctx)) but syntax error is occured.Is there any way to do this?

You should consult the ctypes documentation.
The way to go is to use byref() or pointer() (slower and more complex) functions exported by ctypes module, as seen in the example.
>>> print ctypes.byref.__doc__
byref(C instance[, offset=0]) -> byref-object
Return a pointer lookalike to a C instance, only usable
as function argument

This is explained in the ctypes docs, in Passing pointers (or: passing by reference):
Sometimes a C api function expects a pointer to a data type as parameter, probably to write into the corresponding location, or if the data is too large to be passed by value. This is also known as passing parameters by reference.
ctypes exports the byref() function which is used to pass parameters by reference. The same effect can be achieved with the pointer() function, although pointer() does a lot more work since it constructs a real pointer object, so it is faster to use byref() if you don’t need the pointer object in Python itself:
>>> i = c_int()
>>> f = c_float()
>>> s = create_string_buffer(b'\000' * 32)
>>> print(i.value, f.value, repr(s.value))
0 0.0 b''
>>> libc.sscanf(b"1 3.14 Hello", b"%d %f %s",
... byref(i), byref(f), s)
3
>>> print(i.value, f.value, repr(s.value))
1 3.1400001049 b'Hello'
However, now that you're provided the code, you're not trying to pass it to a C function that expects a pointer, you're trying to pass it to a Python function.
If you wanted to do that, as the docs explain, you need to use pointer rather than byref. And of course the Python function needs to expect a POINTER(c_int) rather than a c_int.
But in that, you're not even really doing that. You've written this code:
a=ctypes.c_int(35)
def foo(n):
n=ctypes.c_int(19)
You're not doing anything to the value that was passed in; you're just rebinding the local name n to a different, newly-constructed value.
All you need to do here is change the value:
def foo(n):
n.value = 19
Then there's no need for a byref or a pointer here.

python ctypes sending pointer to structure as parameter to native library

I am trying to write a wrapper to a native library in Linux. Problem is this:
definition in c:
int mymethod(mystruct* ptr)
in python:
_lib.mymethod.argtypes = (ctypes.POINTER(mystruct),)
_lib.mymethod.restype = ctypes.c_int
s = mystruct()
_lib.mymethod(ctypes.byref(s))
# raises: expected LP_mystruct instance instead of pointer to mystruct
_lib.mymethod(ctypes.pointer(s))
# raises expected LP_mystruct instance instead of LP_mystruct
errors. How to pass a structure as a pointer to a native method ?
Thanks.
Mete

The problem is that the higher level "POINTER" from ctypes is, in Python, a different object than "a generic pointer" (ctypes.CArgObject by ctypes.byref)which is returned or a single number representing a memory address (which is what is returned by ctype's adrresof) - you can either annotate your function to receive a `ctypes.c_voidp and call it with _lib.mymethod(ctypes.addressof(a)) instead -
Or if you want to work on the stronged-typed side to avoid errors that would crash Python (a type error raises a Python exception instead - a wrong parameter passed to a C unction would cause a segmentation fault on the Python interpreter itself), you have to create a variable to hold the new "type" which is a POINTER to your structure - and then create an instance of this type with the address of your structure:
mystruct_pointer = ctypes.POINTER(mystruct)
_lib.mymethod.argtypes = (mystruct_pointer,)
_lib.mymethod.restype = ctypes.c_int
s = mystruct()
_lib.mymethod(mystruct_pointer.from_address(ctypes.addressof(s)))

(I know that this is an old question, but I think the accepted answer is an unnecessary workaround, so I want to leave this here for posterity.)
Actually ctypes should explicitly support using byref() to pass a pointer like that:
ctypes exports the byref() function which is used to pass parameters by reference. The same effect can be achieved with the pointer() function, although pointer() does a lot more work since it constructs a real pointer object, so it is faster to use byref() if you don’t need the pointer object in Python itself.
The likely cause of this is that you have defined your struct in more than one place (e.g. in different modules) - if the argtypes assignment sees one definition and the function call sees the other, this confusing error arises. In other words, ctypes tries to match two mystruct types that are (probably) identical in contents, and have the exact same name, but they are not the same type. As long as the base struct type is a single type object, it doesn't matter if you construct a pointer to it using pointer(), byref() or POINTER()() - ctypes will detect that the underlying (pointed-to) type is the same.
To verify if this is the case, try assert(_lib.mymethod.argtypes[0]._type_ == type(s)) right before calling the external function.