I have an external class that is already wrapped (I mean, it is directly accessible by python with no further effort), and now I want it to be part of a larger cython module (in other words, embed it).
I could have explicitly python-import it. But the things is that the external class is already used in an extern function in the cython module (so that the class is eventually #included in source). Python import require the module to be compiled, then the two module may have two different copy of the same class...
How should I then use an external already wrapped class in cython?
(Possibly oversimplified) example:
Foo.cpp:
#include "Python.h"
#include "foo.hpp"
struct Foo_wrapper {
PyObject_HEAD
foo bar;
};
static int Foo_init(Foo_wrapper* self, PyObject* args, PyObject*) {
....
}
static PyTypeObject FooType {...};
Spam.pyx:
cdef extern from "some_functions.hpp":
cdef some_function1(some_type); // this returns a wrapped foo object
def spam(arg1, arg2, arg3):
// wrap arg1, arg2, arg3 to some_type
return some_function1(an_instance_of_some_type); // huh? but foo isn't available here!
And I want to use the foo class in spam.pyx.
This should be fine (almost) as it is. This cdef extern line isn't quite right:
cdef extern from "some_functions.hpp":
object some_function1(some_type); // this returns a wrapped foo object
Note the change to object - this tells Cython that the function returns a Python object. The C/C++ declaration would look like:
PyObject* some_function1(some_type);
// or
Foo_wrapper* some_function1(some_type);
Either will work.
The reason that the Cython code will be fine with it is that the PyObject_HEAD includes a pointer ob_type that points to the PyTypeObject FooType. This is set up when the object is created. The PyTypeObject contains all the details the Python interpreter needs to use the returned object and so everything should work fine.
The whole thing is basically the equivalent of the Python code:
# in "somemodule.py"
def a_function():
import something_else
return something_else.Class()
The Python interpreter can use the returned value, despite the Class not being known in the "global" namespace.
The one thing to be careful of is that you should ensure that the Foo module initialization function has been called at least once before creating a Foo_wrapper. The reason being is that this function usually does some things like calling PyType_Ready(&FooType) needed to make sure that FooType is properly set up. An easy way to do this would be to add the following lines to some_function1:
PyObject* m = PyImport_ImportModule("Foo");
if (m==NULL) return NULL; // an error
Py_CLEAR(m); // don't need to keep a reference to it
There are other ways of doing the same thing though.
Related
I am trying to use a C library "myClib" from python.
This library has a function "myCfunction" that return a pointer struct "myCStruct" and then had other functions that take a pointer of this struct in argument.
I don't have the definition of this struct I only get this in the .h file :
typedef struct myCStruct myCStruct
The library is provided with as a static library with a .a file. Some .so files are also provided they contains some dependencies used by the library. The .h files are also provided but the full definition of "myCStruct" seems not provided in any of the .h.
I don't have to read or process any information that would be contained in this object but I need to pass it in argument of an other C functions of the library.
Right now to execute the C code I am using cython to generate a .so file that I can import in my python code.
The definition of the function in my pyx file looks like this :
cdef extern from "myClib.h":
void myCfunction()
def py_myCfunction():
return myCfunction()
I didn't declare the return type of py_myCfunction as I didn't define any class matching the myCStruct since I don't have any idea about the members.
My problem is that when I call py_myCfunction in my python program I am getting an object of type NoneType.
So 2 questions :
Could I get a Nonetype object because I miss the declaration in the pyx ? Or does it necessary means that myCfunction return nothing (it's supposed to return something).
If the problem is there because I miss the declaration, which return type could my py_myCfunction has since I don't know the member of the C struct equivalent ?
You need to declare the C API so cython knows what it is and how to call it
You should wrap the C struct * in a Python class, and add methods
to the class that call the other C API functions that reference the struct
Here's example cython code based on your question:
# declare external stuff for cython
cdef extern from "myClib.h":
struct myCStruct:
pass
struct myCStruct *myCFunction();
int otherFunction(struct myCStruct *cstruct, int parameter);
# define python glue for python
cdef class MyPythonClass:
cdef myCStruct *cstruct
def __cinit__(self):
self.cstruct = myCFunction()
# plus methods that call other functions in myClib using the self.cstruct
def other(self, parameter):
return otherFunction(self.cstruct, parameter)
# if myCLib has "myCStruct_Free"
def __dealloc__(self):
if self.cstruct != NULL:
myCStruct_Free(self.cstruct)
self.cstruct = NULL
Say I have a .h file with the following code:
class MyClass
{
public:
int Attribute1;
int Attribute2;
MyClass(){};
virtual ~MyClass(){};
virtual void Method1(string var1);
virtual float Method2(float var2);
};
And a related .pyx file with:
cdef class PyClass:
cdef MyClass *classptr
[standard __cinit__ and __dealloc__ declarations ]
cdef int Attribute1;
def Method1(self, var1):
self.classptr.Method1(var1)
... and pxd file with:
cdef extern from "mycode.h":
cdef cppclass MyClass:
MyClass() except +
int Attribute1
void Method1(string)
And I use a setup.py to create a .so library to import in a python module.
Note that even though MyClass::Method2 and MyClass::Attribute2 exist in c++, I don't mention them in Cython, so they won't be visible in the .so library. Maybe I did so because they're indirectly used by MyClass::Method1(), or just because I don't plan to invoke them in the python code.
Is this a bad practice likely to cause a problem/weird behavior?
If so, why?
cdef class PyClass:
cdef MyClass *classptr
# ...
cdef int Attribute1;
Attribute1 doesn't do what you think. It's a separate value stored as part of PyClass and has nothing to do with the Attribute1 in classptr. You probably want to write a property instead.
However, to answer your question:
Yes, it's fine for you to only wrap the specific functions you're interested in. Cython doesn't need to know all the details of your C++ classes - it only needs to know enough details to generate valid C++ code using them. A few quick examples of things it's useful to omit:
Templates. For example std::string is really a template typedef, but it may not be necessarily for a Cython wrapper to know this, or for optional allocator template types, or for numeric template types which Cython doesn't really support.
Complicated inheritance hierarchies: doesn't matter if the functions you care about actually come from a base-class. Just wrap the derived class you're using.
Interfaces that return other classes - because then you'd need to wrap the second class (which might expose a third class...).
There really is no consequence beyond not being able to call the code you haven't wrapped from python. Cython's C++ support is (and will likely always be) somewhat incomplete and it's often necessary to give it a simplified C++ interface to get anything done.
I have a python extension module written in C++, which contains multiple functions. One of these generates an instance of a custom structure, which I then want to use with other functions of my module in Python as follows
import MyModule
var = MyModule.genFunc()
MyModule.readFunc(var)
To do this, I've tried using PyCapsule objects to pass a pointer to these objects between Python and C, but this produces errors when attempting to read them in the second C function ("PyCapsule_GetPointer called with invalid PyCapsule object"). Python, however, if asked to print the PyCapsule object (var) correctly identifies it as a "'capsule object "testcapsule"'. My C code appears as follows:
struct MyStruct {
int value;
};
static PyObject* genFunc(PyObject* self, PyObject *args) {
MyStruct var;
PyObject *capsuleTest;
var.value = 1;
capsuleTest = PyCapsule_New(&var, "testcapsule", NULL);
return capsuleTest;
}
static PyObject* readFunc(PyObject* self, PyObject *args) {
PyCapsule_GetPointer(args, "testcapsule");
return 0;
}
Thank you for your help.
Like stated in a comment to your question, you'll run into an issue when reading data from the local variable MyStruct var. For this you can use the third destructor to PyCapsule_New.
But that's not the reason for your problem just now. You're using PyCapsule_GetPointer(args, "testcapsule") on the args parameter. And since it's not a capsule, even though var is one, you might have defined the signature of the function as METH_VARARGS. Instead you need to unpack the tuple or use METH_O.
I have existing C++ code that defines some classes I need to use, but I need to be able to send those classes to Python code. Specifically, I need to create class instances in C++, create Python objects to serve as wrappers for these C++ objects, then pass these Python objects to Python code for processing. This is just one piece of a larger C++ program, so it needs to be done ultimately in C++ using the C/Python API.
To make my life easier, I have used Cython to define extension classes (cdef classes) that serve as the Python wrappers for my C++ objects. I am using the typical format where the cdef class contains a pointer to the C++ class, which is then initialized when the cdef class instance is created. Since I also want to be able to replace the pointer if I have an existing C++ object to wrap, I have added methods to my cdef classes to accept() the C++ object and take its pointer. My other cdef classes successfully use the accept() method in Cython, for example when one object owns another.
Here is a sample of my Cython code:
MyCPlus.pxd
cdef extern from "MyCPlus.h" namespace "mynamespace":
cdef cppclass MyCPlus_Class:
MyCPlus_Class() except +
PyModule.pyx
cimport MyCPlus
from libcpp cimport bool
cdef class Py_Class [object Py_Class, type PyType_Class]:
cdef MyCPlus.MyCPlus_Class* thisptr
cdef bool owned
cdef void accept(self, MyCPlus.MyCPlus_Class &indata):
if self.owned:
del self.thisptr
self.thisptr = &indata
self.owned = False
def __cinit__(self):
self.thisptr = new MyCPlus.MyCPlus_Class()
self.owned = True
def __dealloc__(self):
if self.owned:
del self.thisptr
The problem comes when I try to access the accept() method from C++. I tried using the public and api keywords on my cdef class and on the accept() method, but I cannot figure out how to expose this method in the C struct in Cython's auto-generated .h file. No matter what I try, the C struct looks like this:
PyModule.h (auto-generated by Cython)
struct Py_Class {
PyObject_HEAD
struct __pyx_vtabstruct_11PyModule_Py_Class *__pyx_vtab;
mynamespace::MyCPlus_Class *thisptr;
bool owned;
};
I also tried typing the self input as a Py_Class, and I even tried forward-declaring Py_Class with the public and api keywords. I also experimented with making accept() a static method. Nothing I've tried works to expose the accept() method so that I can use it from C++. I did try accessing it through __pyx_vtab, but I got a compiler error, "invalid use of incomplete type". I have searched quite a bit, but haven't seen a solution to this. Can anyone help me? Please and thank you!
As you pointed in your comment, it does seem that the __pyx_vtab member is for Cython use only, since it doesn't even define the struct type for it in the exported header(s).
Adding to your response, one approach could also be:
cdef api class Py_Class [object Py_Class, type Py_ClassType]:
...
cdef void accept(self, MyCPlus.MyCPlus_Class &indata):
... # do stuff here
...
cdef api void (*Py_Class_accept)(Py_Class self, MyCPlus.MyCPlus_Class &indata)
Py_Class_accept = &Py_Class.accept
Basically, we define a function pointer and set it to the extension's method we want to expose. This is not that much different to your response's cdef'd function; the main difference would be that we can define our methods as usual in the class definition without having to duplicate functionality or method/function calls to another function to expose it. One caveat is that we would've to define our function pointer's signature almost verbatim to the method's one in addition to the self's extension type (in this case) and etc; then again this also applies for regular functions.
Do note that I tried this up on a C-level Cython .pyx file, I haven't and do not intent to test it on a CPP implementation file. But hopefully this might work just as fine, I guess.
This is not really a solution, but I came up with a workaround for my problem. I am still hoping for a solution that allows me to tell Cython to expose the accept() method to C++.
My workaround is that I wrote a separate function for my Python class (not a method). I then gave the api keyword both to my Python class and to the new function:
cdef api class Py_Class [object Py_Class, type PyType_Class]:
(etc.)
cdef api Py_Class wrap_MyCPlusClass(MyCPlus.MyCPlus_Class &indata):
wrapper = Py_Class()
del wrapper.thisptr
wrapper.thisptr = &indata
wrapper.owned = False
return wrapper
This gets a little unwieldy with the number of different classes I need to wrap, but at least Cython puts the function in the API where it is easy to use:
struct Py_Class* wrap_MyCPlusClass(mynamespace::MyCPlusClass &);
You probably want to use cpdef instead of cdef when declaring accept. See the docs:
Callable from Python and C
* Are declared with the cpdef statement.
* Can be called from anywhere, because it uses a little Cython magic.
* Uses the faster C calling conventions when being called from other Cython code.
Try that!
I am trying to wrap some C++ code into Cython and I came up with some trouble trying to pass a method from a class as an argument to a function.
I do not know if it makes it more clear, but class A represents a statistical model (so myAMethod uses not only the arguments passed but many instance variables) and B has different methods for minimizing the function passed.
In C++ I have something of this style:
class A
{
public:
double myAMethod(double*)
};
class B
{
public:
double myBMethod(A&, double (A::*f) (double*)
}
So what I am trying to do is to use instances of both A and B in Cython code. I had no trouble wrapping the classes, but when I try to use myBMethod, I don't know how to pass a pointer of the kind A::*myAMethod
If I do this:
myBMethod(ptrToAObj[0], &ptrToAObj.myAMethod),
then Cython compiles this code to [...] &ptrToAObj->myAMethod [...], and I get the message one would expect from g++:
"ISO C++ forbids taking the address of a bound member function to form a pointer to member function."
But if I try to point straight to the class method, and do myBMethod(ptrToAObj[0], A.myAMethod), then Cython won't compile and say that
myAMethod is not a static member from A.
And that's pretty much all I was able to advance. I could work at C++ level and avoid any of these anoyances, but if I were able to use instances of A and B in Python (via Cython) interactively, it would help me in speedig my development pace.
Any help will be really appreciated, and I apologize if this question as been already answered and/or is available in a referece - I search SO, Cython reference and Smith's "Cython" book and I did not found this theme adressed.
Thanks in advance!
I have a partial (if horrendous) solution. I'm prepared to believe there's a better way, but I don't know it.
In cpp_bit.hpp:
class A {
public:
double myAMethod(double*) { return 0.0; }
};
typedef double (A::*A_f_ptr)(double *);
class B {
public:
double myBMethod(A& a, A_f_ptr f) {
double x = 0.1;
return (a.*f)(&x);
}
};
A_f_ptr getAMethod() {
return &A::myAMethod;
}
I've given the functions very basic implementations, just so I can check for really obvious crashes. I've also created a function pointer which returns a pointer to myAMethod. You'll need to do this for every method you want to wrap.
In py_bit.pyx
# distutils: language = c++
from cython.operator import dereference
cdef extern from "cpp_bit.hpp":
cdef cppclass A:
double myAMethod(double*)
cdef cppclass A_f_ptr:
pass
cdef cppclass B:
double myBMethod(A&, A_f_ptr)
cdef A_f_ptr getAMethod()
cdef class PyA:
cdef A* thisptr
def __cinit__(self):
self.thisptr = new A()
def __dealloc__(self):
del self.thisptr
cpdef myAMethod(self,double[:] o):
return self.thisptr.myAMethod(&o[0])
cdef class PyB:
cdef B* thisptr
def __cinit__(self):
self.thisptr = new B()
def __dealloc__(self):
del self.thisptr
cpdef myBMethod(self,PyA a):
return self.thisptr.myBMethod(dereference(a.thisptr),getAMethod())
I couldn't figure out how to typedef a member function pointer in Cython, so instead I created an empty cppclass with the same name. This works because cython just seems to use it for type-checking and nothing more, and since it includes cpp_bit.hpp (where it's defined) you can use it no problem.
All I've done is left the task of getting the member function pointer to c++ (in getAMethod, which I call). I don't think it's entirely satisfactory, but it looks workable, and is only a short extra c++ function for every member function you want to access. You could play with where you put it to encapsulate it more cleanly.
An alternative, untested approach: (Edit: further thought suggests this might be very tricky! Attempt this at your own risk!)
Personally, I'd be tempted to change the c++ interface so that myBMethod is defined as
double myBMethod(std::function<double (double*)>)
(since presumably you always call it with the A instance it's passed with). Then use lambda functions in c++(11!) to wrap the A instance and function together
b.myBMethod([&](double* d){ return a.myAMethod(d) };
It may then take a bit of hugely complicated Cython wrapping that I haven't yet considered, but it should be possible to convert a simple double (double*) function pointer to the c++ function object, and so use it more directly.
It's also possible that your actual design is more complicated in ways I haven't considered, and this approach isn't flexible enough anyway.