Develop Reference

Return a ctypes C struct back to C

I'm trying to wrap my head around the ctypes API. I have a C structure (zmsg_t) I wish to create in Python and then return from Python to C.
So a Python method is called from C and needs to return an object which we then process in C again.
I have a binding to access and create the C methods and structures so in Python I can call the Zmsg() constructor. I now need to return this.
My python test method is simply:
def actor_test(*args, **kwargs):
print("test")
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return msg
the method is called from C as follows:
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
This correctly calls the method. However the returned object is of course a PyObject*. The debugger says it's
"<czmq._czmq_ctypes.Zmsg object at 0x7ffff5f18e50>" PyObject
[class] "<class 'czmq._czmq_ctypes.Zmsg'>"
[super class] "<class 'object'>"
[meta type] "<class 'type'>"
ob_refcnt 1 Py_ssize_t
How I can I get it back to the original C type (zmsg_t *)
Any help really appreciated.
The ctypes definitions for Zmsg can be seen here

I would try using return byref(msg) and do some C casting and watch out for garbage-collection. But tbh I am usually calling c from a python process.

I'm answering my own question as I was pointed by the python mailinglist to the fact that this question approaches it wrongly. It mixes both the ctypes API and the CPython API. Best is to use one approach. Since to embed Python you need the CPython API best is to use the CPython API to wrap to C methods. That way the when Python returns you'll endup with your own defined Python object in C and can access its internal structure.
This tutorial describes in detail how to wrap a C struct to a Python object.

how to include shared object in python [duplicate]

I'm just getting started with ctypes and would like to use a C++ class that I have exported in a dll file from within python using ctypes.
So lets say my C++ code looks something like this:
class MyClass {
public:
int test();
...
I would know create a .dll file that contains this class and then load the .dll file in python using ctypes.
Now how would I create an Object of type MyClass and call its test function? Is that even possible with ctypes? Alternatively I would consider using SWIG or Boost.Python but ctypes seems like the easiest option for small projects.

Besides Boost.Python(which is probably a more friendly solution for larger projects that require one-to-one mapping of C++ classes to python classes), you could provide on the C++ side a C interface. It's one solution of many so it has its own trade offs, but I will present it for the benefit of those who aren't familiar with the technique. For full disclosure, with this approach one wouldn't be interfacing C++ to python, but C++ to C to Python. Below I included an example that meets your requirements to show you the general idea of the extern "c" facility of C++ compilers.
//YourFile.cpp (compiled into a .dll or .so file)
#include <new> //For std::nothrow
//Either include a header defining your class, or define it here.
extern "C" //Tells the compile to use C-linkage for the next scope.
{
//Note: The interface this linkage region needs to use C only.
void * CreateInstanceOfClass( void )
{
// Note: Inside the function body, I can use C++.
return new(std::nothrow) MyClass;
}
//Thanks Chris.
void DeleteInstanceOfClass (void *ptr)
{
delete(std::nothrow) ptr;
}
int CallMemberTest(void *ptr)
{
// Note: A downside here is the lack of type safety.
// You could always internally(in the C++ library) save a reference to all
// pointers created of type MyClass and verify it is an element in that
//structure.
//
// Per comments with Andre, we should avoid throwing exceptions.
try
{
MyClass * ref = reinterpret_cast<MyClass *>(ptr);
return ref->Test();
}
catch(...)
{
return -1; //assuming -1 is an error condition.
}
}
} //End C linkage scope.
You can compile this code with
gcc -shared -o test.so test.cpp
#creates test.so in your current working directory.
In your python code you could do something like this (interactive prompt from 2.7 shown):
>>> from ctypes import cdll
>>> stdc=cdll.LoadLibrary("libc.so.6") # or similar to load c library
>>> stdcpp=cdll.LoadLibrary("libstdc++.so.6") # or similar to load c++ library
>>> myLib=cdll.LoadLibrary("/path/to/test.so")
>>> spam = myLib.CreateInstanceOfClass()
>>> spam
[outputs the pointer address of the element]
>>> value=CallMemberTest(spam)
[does whatever Test does to the spam reference of the object]
I'm sure Boost.Python does something similar under the hood, but perhaps understanding the lower levels concepts is helpful. I would be more excited about this method if you were attempting to access functionality of a C++ library and a one-to-one mapping was not required.
For more information on C/C++ interaction check out this page from Sun: http://dsc.sun.com/solaris/articles/mixing.html#cpp_from_c

The short story is that there is no standard binary interface for C++ in the way that there is for C. Different compilers output different binaries for the same C++ dynamic libraries, due to name mangling and different ways to handle the stack between library function calls.
So, unfortunately, there really isn't a portable way to access C++ libraries in general. But, for one compiler at a time, it's no problem.
This blog post also has a short overview of why this currently won't work. Maybe after C++0x comes out, we'll have a standard ABI for C++? Until then, you're probably not going to have any way to access C++ classes through Python's ctypes.

The answer by AudaAero is very good but not complete (at least for me).
On my system (Debian Stretch x64 with GCC and G++ 6.3.0, Python 3.5.3) I have segfaults as soon has I call a member function that access a member value of the class.
I diagnosticated by printing pointer values to stdout that the void* pointer coded on 64 bits in wrappers is being represented on 32 bits in Python. Thus big problems occurs when it is passed back to a member function wrapper.
The solution I found is to change:
spam = myLib.CreateInstanceOfClass()
Into
Class_ctor_wrapper = myLib.CreateInstanceOfClass
Class_ctor_wrapper.restype = c_void_p
spam = c_void_p(Class_ctor_wrapper())
So two things were missing: setting the return type to c_void_p (the default is int) and then creating a c_void_p object (not just an integer).
I wish I could have written a comment but I still lack 27 rep points.

Extending AudaAero's and Gabriel Devillers answer I would complete the class object instance creation by:
stdc=c_void_p(cdll.LoadLibrary("libc.so.6"))
using ctypes c_void_p data type ensures the proper representation of the class object pointer within python.
Also make sure that the dll's memory management be handled by the dll (allocated memory in the dll should be deallocated also in the dll, and not in python)!

I ran into the same problem. From trial and error and some internet research (not necessarily from knowing the g++ compiler or C++ very well), I came across this particular solution that seems to be working quite well for me.
//model.hpp
class Model{
public:
static Model* CreateModel(char* model_name) asm("CreateModel"); // static method, creates an instance of the class
double GetValue(uint32_t index) asm("GetValue"); // object method
}
#model.py
from ctypes import ...
if __name__ == '__main__':
# load dll as model_dll
# Static Method Signature
fCreateModel = getattr(model_dll, 'CreateModel') # or model_dll.CreateModel
fCreateModel.argtypes = [c_char_p]
fCreateModel.restype = c_void_p
# Object Method Signature
fGetValue = getattr(model_dll, 'GetValue') # or model_dll.GetValue
fGetValue.argtypes = [c_void_p, c_uint32] # Notice two Params
fGetValue.restype = c_double
# Calling the Methods
obj_ptr = fCreateModel(c_char_p(b"new_model"))
val = fGetValue(obj_ptr, c_int32(0)) # pass in obj_ptr as first param of obj method
>>> nm -Dg libmodel.so
U cbrt#GLIBC_2.2.5
U close#GLIBC_2.2.5
00000000000033a0 T CreateModel # <----- Static Method
U __cxa_atexit#GLIBC_2.2.5
w __cxa_finalize#GLIBC_2.2.5
U fprintf#GLIBC_2.2.5
0000000000002b40 T GetValue # <----- Object Method
w __gmon_start__
...
...
... # Mangled Symbol Names Below
0000000000002430 T _ZN12SHMEMWrapper4HashEPKc
0000000000006120 B _ZN12SHMEMWrapper8info_mapE
00000000000033f0 T _ZN5Model12DestroyModelEPKc
0000000000002b20 T _ZN5Model14GetLinearIndexElll
First, I was able to avoid the extern "C" directive completely by instead using the asm keyword which, to my knowledge, asks the compiler to use a given name instead of the generated one when exporting the function to the shared object lib's symbol table. This allowed me to avoid the weird symbol names that the C++ compiler generates automatically. They look something like the _ZN1... pattern you see above. Then in a program using Python ctypes, I was able to access the class functions directly using the custom name I gave them. The program looks like fhandle = mydll.myfunc or fhandler = getattr(mydll, 'myfunc') instead of fhandle = getattr(mydll, '_ZN12...myfunc...'). Of course, you could just use the long name; it would make no difference, but I figure the shorter name is a little cleaner and doesn't require using nm to read the symbol table and extract the names in the first place.
Second, in the spirit of Python's style of object oriented programming, I decided to try passing in my class' object pointer as the first argument of the class object method, just like when we pass self in as the first method in Python object methods. To my surprise, it worked! See the Python section above. Apparently, if you set the first argument in the fhandle.argtypes argument to c_void_ptr and pass in the ptr you get from your class' static factory method, the program should execute cleanly. Class static methods seem to work as one would expect like in Python; just use the original function signature.
I'm using g++ 12.1.1, python 3.10.5 on Arch Linux. I hope this helps someone.

Is it possible to call a C Function by reference from python??

I am a beginner in python and I need to test a C function which is being called by reference.
Here is my C File : myfile.c
#include<stdio.h>
int my_function(int *x)
{
int A;
printf("enter value of A");
scanf("%d",&A);
*x = 10 * A; // this sets the value to the address x is referencing to
return 0;
}
now,
my python script should call my_function() and pass the argument so that I can check and verify the result.
something like :
result = self.loaded_class.my_function(addressOf(some_variable))
self.assertEqual(some_variable,10)
Is this possible ?? how can i achieve this.
And I am writing scripts in python for auto tests , not using interactive python.

If you compile your file as a shared library or dll (I don't know how to do that), you can use ctypes like this (assuming it's a dll for this example):
import ctypes as ct
mylib = ct.cdll.myfile
c_int = ct.c_int(0)
mylib.my_function(ct.byref(c_int))
print c_int.value

You can write a Python interface for C function, a simple example is in Python doc. However, if all you want to do is test C function, you're probably better of with a C/C++ testing framework such as Google Test.

Too many arguments in my function - Python

I am trying to access my function in a dll and pass by value as integer in it. But I am getting a ValueError as:
Procedure probably called with too many arguments (4 bytes in excess)
My python .py script is as below:
func2.restype = c_int
func2.argtypes = [c_int]
func2(3)
...
My actual function in the dll is just a simple function like:
int DLLfun2(int argtest) { return argtest + 1; };
...
Looks like a simple problem, but I'm guessing I am missing out on something. Kindly help.
Cheers.

Looks weird because integer should be automatically cast, but try with a func2(c_int(3))
EDIT :
According to the ctypes doc, this exception could be raised because of a wrong calling convention (cdecl instead of stdcall). But the function is still called (just a sort of warning :s)
Be sure of you declaration in your dll.
If you load your library with windll, it must use stdcall calling convention, otherwise, use the cdll module.

How can I use a DLL from Python

I am trying to load a DLL in Python, I want to use its eConnect() function using ctypes
What I know from the source code of the DLL:
from the *.cpp source code: bool eConnect( const char *host, UINT port, int clientId=0);
from the dependency walker tool: function ordinal is 6 and undecorated name is
?eConnect#EClientSocket##UAE_NPBDIH#Z
I tried to access the eConnect from two ways myfunction and myfunction2, but I probably get it wrong. Here is my code:
from ctypes import *
def main():
IP = c_char_p('127.0.0.1')
port = c_uint(7496)
client_id = c_int(0)
myfunction = getattr(cdll.TwsSocketClient, "?eConnect#EClientSocket##UAE_NPBDIH#Z")
myfunction2= cdll.TwsSocketClient[6]
print myfunction
print myfunction2
print myfunction(IP, port, client_id,IP)
if __name__ == "__main__":
main()
I get the below error:
"WindowsError: exception: access violation reading 0x0000002D"
I would badly need some help here (I do not know c++).
Thanks!

To make things easier, more Pythonic, you might want to look into ctypesgen:
http://code.google.com/p/ctypesgen/
It will generate proper wrapper functions, data types and such for you. If you just want to know how to use ctypes, might as well start with the tutorial:
http://docs.python.org/library/ctypes.html
Anything more specific and I'll have to read the API for the DLL you're attempting to use.

The function being exported is a class member function of the class EClientSocket. You're attempting to call that function from Python without passing in an EClientSocket pointer as the this parameter; furthermore, ctypes doesn't know anything about the __thiscall calling convention, so even if you did pass in an EClientSocket instance, it would be on the stack instead of in the ECX register.
The only real solution to this would be to export a C wrapper from your DLL that forwards the call to eConnect. For example:
extern "C" DLLEXPORT
bool EClientSocket_eConnect(EClientSocket *This, const char *host, UINT port, int clientId)
{
return This->eConnect(host, port, clientId);
}
However, even in that case, you have to be extra-careful on the Python side to construct an appropriate EClientSocket instance. I'd strongly recommend reconsidering your approach here.

Consider looking into IronPython. It makes it easier to call on DLL files.

There is also Boost::Python

Thanks for your answers everyone. I took Adam's advise and reconsidered my approached. As I do not know know c++, it was a bad idea from the start.
There is an alternative API in R (not official) which is built on top of the official Java API. It is then quite easy to link R and Python using rPy2.