To avoid a memory leak in the sutuation described below, I would like to call Py_DecRef directly from Python. Is there a way to do this? Is there a better solution for this problem?
I am using ctypes to interface my Python code to a C library for which I do not have the code. The C library was not written for Python, so it doesn't know anything about Python objects.
The C library uses two callbacks: the first creates an object and returns a void* pointer to it, and the second gets the pointer as parameter and is supposed to destroy it. In the C header files, the types of these callback functions are defined as follows:
typedef void* (*CreateCallback)();
typedef void (*DestroyCallback)(void*);
These callbacks could be defined in Python as shown below (simplified code). The current code has a memory leak as explained in the comments.
import ctypes
CreateCallback = ctypes.CFUNCTYPE(ctypes.py_object)
DestroyCallback = ctypes.CFUNCTYPE(None, ctypes.py_object)
class Object:
pass # In the real application, this contains more code
#CreateCallback
def create():
return Object()
# Ctypes correctly increments the reference count of the
# object, to make sure it does not get garbage collected
# while the C code holds a reference to it.
#DestroyCallback
def destroy(object):
pass
# Above, the reference count of the object should be
# decremented, because the C code no longer holds a
# reference to it. However, Ctypes does not know this so
# cannot do it automatically. How can I do this from
# Python? Is it possible to call Py_DecRef or similar
# directly from Python?
One option would be to create a C function that call Py_DecRef, compile that C function into a dll (or so for Linux), and call that from the destroy function above. That solution has at least two disadvantages:
It seems overly complex to create a dll just for one function
The C code would have to be compiled against a specific version of Python, instead of using whatever version of Python is running my Python code. Note that I need this to work on Windows, where a dll cannot contain undefined globals.
I have a program that is written in C++ and can be extended with extensions that are written in python using python's C api, basically there is python interpreter in the program which can load python modules that contains some hooks to various events.
I can pass simple data types from C to python, such as string or integer. I can do the same the other way, I can call internal C++ api's I created from the python modules. What I can't do, however is to marshall more complex data types, such as classes. For example I have this thing:
class X
{
int a;
int b;
};
I could call some python hook that would look like this:
def some_hook(a, b):
print(str(a + b))
but I would rather do
def some_hook(x):
print(str(x.a + x.b))
I know how to call a python function from C using PyObject_CallObject and how to give it some simple python vars as parameters (like integer or string). But I have no idea how to construct python classes from within C/C++, fill them up with data, and then pass them as parameters to PyObject_CallObject.
My idea was to create some proxy classes like in SWIG (swig.org) that I would fill up with data from C++ class and then back. I created some .py file that contains the declaration of these classes, but still, I have no idea how would I instantiate them from within C/C++ and fill up? Neither how I would do it the other way - turn PyObject into C/C++ class?
In a nutshell: I want to be able to turn C++ class into a python class, call a python function with that class as a parameter, and then call some other function from python with this class that would be just a C++ API that would turn the python class into C++.
Some background reading on how to call python from C: http://docs.python.org/3/extending/embedding.html
Source code of my app C++ that loads and uses python extensions: https://github.com/huggle/huggle3-qt-lx/blob/master/huggle/pythonengine.cpp
I'm just getting started with ctypes and would like to use a C++ class that I have exported in a dll file from within python using ctypes.
So lets say my C++ code looks something like this:
class MyClass {
public:
int test();
...
I would know create a .dll file that contains this class and then load the .dll file in python using ctypes.
Now how would I create an Object of type MyClass and call its test function? Is that even possible with ctypes? Alternatively I would consider using SWIG or Boost.Python but ctypes seems like the easiest option for small projects.
Besides Boost.Python(which is probably a more friendly solution for larger projects that require one-to-one mapping of C++ classes to python classes), you could provide on the C++ side a C interface. It's one solution of many so it has its own trade offs, but I will present it for the benefit of those who aren't familiar with the technique. For full disclosure, with this approach one wouldn't be interfacing C++ to python, but C++ to C to Python. Below I included an example that meets your requirements to show you the general idea of the extern "c" facility of C++ compilers.
//YourFile.cpp (compiled into a .dll or .so file)
#include <new> //For std::nothrow
//Either include a header defining your class, or define it here.
extern "C" //Tells the compile to use C-linkage for the next scope.
{
//Note: The interface this linkage region needs to use C only.
void * CreateInstanceOfClass( void )
{
// Note: Inside the function body, I can use C++.
return new(std::nothrow) MyClass;
}
//Thanks Chris.
void DeleteInstanceOfClass (void *ptr)
{
delete(std::nothrow) ptr;
}
int CallMemberTest(void *ptr)
{
// Note: A downside here is the lack of type safety.
// You could always internally(in the C++ library) save a reference to all
// pointers created of type MyClass and verify it is an element in that
//structure.
//
// Per comments with Andre, we should avoid throwing exceptions.
try
{
MyClass * ref = reinterpret_cast<MyClass *>(ptr);
return ref->Test();
}
catch(...)
{
return -1; //assuming -1 is an error condition.
}
}
} //End C linkage scope.
You can compile this code with
gcc -shared -o test.so test.cpp
#creates test.so in your current working directory.
In your python code you could do something like this (interactive prompt from 2.7 shown):
>>> from ctypes import cdll
>>> stdc=cdll.LoadLibrary("libc.so.6") # or similar to load c library
>>> stdcpp=cdll.LoadLibrary("libstdc++.so.6") # or similar to load c++ library
>>> myLib=cdll.LoadLibrary("/path/to/test.so")
>>> spam = myLib.CreateInstanceOfClass()
>>> spam
[outputs the pointer address of the element]
>>> value=CallMemberTest(spam)
[does whatever Test does to the spam reference of the object]
I'm sure Boost.Python does something similar under the hood, but perhaps understanding the lower levels concepts is helpful. I would be more excited about this method if you were attempting to access functionality of a C++ library and a one-to-one mapping was not required.
For more information on C/C++ interaction check out this page from Sun: http://dsc.sun.com/solaris/articles/mixing.html#cpp_from_c
The short story is that there is no standard binary interface for C++ in the way that there is for C. Different compilers output different binaries for the same C++ dynamic libraries, due to name mangling and different ways to handle the stack between library function calls.
So, unfortunately, there really isn't a portable way to access C++ libraries in general. But, for one compiler at a time, it's no problem.
This blog post also has a short overview of why this currently won't work. Maybe after C++0x comes out, we'll have a standard ABI for C++? Until then, you're probably not going to have any way to access C++ classes through Python's ctypes.
The answer by AudaAero is very good but not complete (at least for me).
On my system (Debian Stretch x64 with GCC and G++ 6.3.0, Python 3.5.3) I have segfaults as soon has I call a member function that access a member value of the class.
I diagnosticated by printing pointer values to stdout that the void* pointer coded on 64 bits in wrappers is being represented on 32 bits in Python. Thus big problems occurs when it is passed back to a member function wrapper.
The solution I found is to change:
spam = myLib.CreateInstanceOfClass()
Into
Class_ctor_wrapper = myLib.CreateInstanceOfClass
Class_ctor_wrapper.restype = c_void_p
spam = c_void_p(Class_ctor_wrapper())
So two things were missing: setting the return type to c_void_p (the default is int) and then creating a c_void_p object (not just an integer).
I wish I could have written a comment but I still lack 27 rep points.
Extending AudaAero's and Gabriel Devillers answer I would complete the class object instance creation by:
stdc=c_void_p(cdll.LoadLibrary("libc.so.6"))
using ctypes c_void_p data type ensures the proper representation of the class object pointer within python.
Also make sure that the dll's memory management be handled by the dll (allocated memory in the dll should be deallocated also in the dll, and not in python)!
I ran into the same problem. From trial and error and some internet research (not necessarily from knowing the g++ compiler or C++ very well), I came across this particular solution that seems to be working quite well for me.
//model.hpp
class Model{
public:
static Model* CreateModel(char* model_name) asm("CreateModel"); // static method, creates an instance of the class
double GetValue(uint32_t index) asm("GetValue"); // object method
}
#model.py
from ctypes import ...
if __name__ == '__main__':
# load dll as model_dll
# Static Method Signature
fCreateModel = getattr(model_dll, 'CreateModel') # or model_dll.CreateModel
fCreateModel.argtypes = [c_char_p]
fCreateModel.restype = c_void_p
# Object Method Signature
fGetValue = getattr(model_dll, 'GetValue') # or model_dll.GetValue
fGetValue.argtypes = [c_void_p, c_uint32] # Notice two Params
fGetValue.restype = c_double
# Calling the Methods
obj_ptr = fCreateModel(c_char_p(b"new_model"))
val = fGetValue(obj_ptr, c_int32(0)) # pass in obj_ptr as first param of obj method
>>> nm -Dg libmodel.so
U cbrt#GLIBC_2.2.5
U close#GLIBC_2.2.5
00000000000033a0 T CreateModel # <----- Static Method
U __cxa_atexit#GLIBC_2.2.5
w __cxa_finalize#GLIBC_2.2.5
U fprintf#GLIBC_2.2.5
0000000000002b40 T GetValue # <----- Object Method
w __gmon_start__
...
...
... # Mangled Symbol Names Below
0000000000002430 T _ZN12SHMEMWrapper4HashEPKc
0000000000006120 B _ZN12SHMEMWrapper8info_mapE
00000000000033f0 T _ZN5Model12DestroyModelEPKc
0000000000002b20 T _ZN5Model14GetLinearIndexElll
First, I was able to avoid the extern "C" directive completely by instead using the asm keyword which, to my knowledge, asks the compiler to use a given name instead of the generated one when exporting the function to the shared object lib's symbol table. This allowed me to avoid the weird symbol names that the C++ compiler generates automatically. They look something like the _ZN1... pattern you see above. Then in a program using Python ctypes, I was able to access the class functions directly using the custom name I gave them. The program looks like fhandle = mydll.myfunc or fhandler = getattr(mydll, 'myfunc') instead of fhandle = getattr(mydll, '_ZN12...myfunc...'). Of course, you could just use the long name; it would make no difference, but I figure the shorter name is a little cleaner and doesn't require using nm to read the symbol table and extract the names in the first place.
Second, in the spirit of Python's style of object oriented programming, I decided to try passing in my class' object pointer as the first argument of the class object method, just like when we pass self in as the first method in Python object methods. To my surprise, it worked! See the Python section above. Apparently, if you set the first argument in the fhandle.argtypes argument to c_void_ptr and pass in the ptr you get from your class' static factory method, the program should execute cleanly. Class static methods seem to work as one would expect like in Python; just use the original function signature.
I'm using g++ 12.1.1, python 3.10.5 on Arch Linux. I hope this helps someone.
I wonder how I can create a PyObject in C++ and then return it to Python.
Sadly the documentation is not very explicit about it.
There is no PyObject_Create so I wonder whether allocating sizeof(PyObject) via PyObject_Malloc and initializing the struct is sufficient.
For now I only need an object with functions attached.
Do you really want a (1) PyObject, as in what Python calls object, or (2) an object of some subtype? That you "need an object with functions attached" seems to indicate you want either methods or attributes. That needs (2) in any case. I'm no expert on the C API, but generally you'd define your own PyTypeObject, then create an instance of that via PyObject_New (refcount and type field are initialized, other fields you might add are not).
I was looking at types.py to understand the built-in types and I came across this GetSetDescriptorType. From the Python documentation:
types.GetSetDescriptorType
The type of objects defined in extension modules with PyGetSetDef,
such as FrameType.f_locals or array.array.typecode. This type is used
as descriptor for object attributes; it has the same purpose as the
property type, but for classes defined in extension modules
I do understand the property type, but could not wrap my mind around this. Can some one who understands this throw some light ?
When you write a Python module using C, you define new types using a C API. This API has a lot of functions and structs to specify all the behavior of the new type.
One way to specify properties of a type using the C API is to define an array of PyGetSetDef structs:
static PyGetSetDef my_props[] = { /*... */ }
And then use the array in the initialization of the type (see this example for details).
Then, in Python, when you use MyType.my_property you have a value of types.GetSetDescriptorType, that is used to resolve the actual value of the property when you write my_obj.my_property.
As such, this type is an implementation detail, unlikely to be very useful.