There's one thing I don't fully understand about the "nogil" applied to a cdef extern like this:
cdef extern from "pthread.h" nogil:
ctypedef struct pthread_mutex_t:
pass
cdef int pthread_mutex_init(pthread_mutex_t *, void *)
cdef int pthread_mutex_destroy(pthread_mutex_t *)
It isn't clear to me about the effect of that nogil, and I cannot find anything in the Cython's documentation about it. My interpetation is that the functions declared inside have the nogil implicitly. Am I right? And what about the struct?
Thanks for any info.
From Cython docs on binding with C code
The nogil function annotation declares that it is safe to call the
function without the GIL.
In your example, this means that the functions pthread_mutex_init and pthread_mutex_destroy are allowed in a with nogil block. In the absence of an explicit with nogil block, the GIL remains in place: it necessary but not sufficient to declare the function as done in the example.
Plain C variables can be used in the nogil blocks but you are responsible for the thread-safety.
Related
void worker(int *P,int l):
cdef i
with nogil, parallel():
for i in prange(l):
P[i]=1
I'm trying to use arrays in cython with multithreading.. but i get this error:
Coercion from Python not allowed without the GIL
the array P is initialized like this:
cdef int l=500000
cdef int *P=<int *> malloc(l* sizeof(int))
any help?
If you don't specify a type (e.g. cdef i) then i is a Python object. Therefore you are not allowed to use i within a nogil block. You probably want cdef int i or similar to specify that i is a C integer.
I am fairly new to cython, and I was attempting to wrap a templated vector class defined as
template < typename T, uint N >
struct Vector{}
and I am having a difficult time learning about how cython uses templates, especially those with an int as an argument. I read in the docs that ints are not yet supported as template parameters. How do I do this properly?
For the curious, the Cython wiki shows how to write a templated class in Cython:
cdef extern from "<vector>" namespace "std":
cdef cppclass vector[T]:
...
Furthermore, multiple template parameters are defined as a list.
To answer the OP's question, one would use cdef struct Vector[T, N].
I found an easy solution!
In a C++ header file, you can just declare a typedef, for example
typedef Vector<float,3>; Vector3f;
In your cython file you can just declare that and now you can use all the functions and operators within the class.
cdef extern from "Vector.h" namespace "ns":
cdef cppclass Vector3f:
Now, I had an additional issue, and that is with "specialized" functions, in my case a specialization for a Vector with 3 params.
template<typename T1, typename T2>
inline Vector<T1, 3 >Cross(const Vector <T1, 3 > & v1, const Vector<T2, 3> & v2)
To use this in cython, just declare it outside the class, in my case
cdef extern from "Vector.h" namespace "ns":
cdef cppclass Vector3f:
...
Vector3f Cross(Vector3f v1,Vector3f v2)
I have C++ dll which works with multiple threads.
So I wrapped this library with Cython and created special receiver callback-function, which must adds some results to asyncio.Queue.
cdef void __cdecl NewMessage(char* message) nogil:
I marked it as nogil, this callback calls from another thread.
In this callback I just use:
with gil:
print("hello") # instead adding to Queue. print("hello") is more simple operation to demostrate problem
And got deadlock here.
How to resolve it?
C++ callback declaration (header):
typedef void (*receiver_t)(char*);
void SetReceiver(receiver_t new_rec);
cpp:
static receiver_t receiver = nullptr;
void SetReceiver(receiver_t new_rec)
{
printf("setted%i\n", (int)new_rec);
a = 123;
if (new_rec != nullptr)
receiver = new_rec;
}
Cython code:
cdef extern from "TeamSpeak3.h":
ctypedef void (*receiver_t) (char*) nogil
cdef void __cdecl SetReceiver(receiver_t new_rec) nogil
cdef void __cdecl NewMessage(char* message) nogil:
with gil:
print("hello")
SetReceiver(NewMessage)
Full code:
.h http://pastebin.com/ZTCjc6NA
.cpp http://pastebin.com/MeygA8im
.pyx http://pastebin.com/k4X9c54P
.py http://pastebin.com/1YV7tMiF
This is a bit of a guess but you probably have a Cython/C/C++ loop running that's holding the GIL and never releasing it. The callback is then forced for wait forever for it.
In normal Python code the GIL is released every few instructions if another thread is waiting for it. In Cython that doesn't happen automatically. One way to ensure that it does happen every so often is to to your loop:
while True:
# ... do stuff
with nogil:
pass
This ensures the GIL is released once per loop.
Unfortunately it's not obvious to me where you have your main loop. I wonder if it's inside connect in your PyTeamSpeak3 class, and perhaps changing the definition of connect to:
def connect(self):
with nogil:
self.thisptr.Connect()
might help?
I am trying to wrap some C++ code into Cython and I came up with some trouble trying to pass a method from a class as an argument to a function.
I do not know if it makes it more clear, but class A represents a statistical model (so myAMethod uses not only the arguments passed but many instance variables) and B has different methods for minimizing the function passed.
In C++ I have something of this style:
class A
{
public:
double myAMethod(double*)
};
class B
{
public:
double myBMethod(A&, double (A::*f) (double*)
}
So what I am trying to do is to use instances of both A and B in Cython code. I had no trouble wrapping the classes, but when I try to use myBMethod, I don't know how to pass a pointer of the kind A::*myAMethod
If I do this:
myBMethod(ptrToAObj[0], &ptrToAObj.myAMethod),
then Cython compiles this code to [...] &ptrToAObj->myAMethod [...], and I get the message one would expect from g++:
"ISO C++ forbids taking the address of a bound member function to form a pointer to member function."
But if I try to point straight to the class method, and do myBMethod(ptrToAObj[0], A.myAMethod), then Cython won't compile and say that
myAMethod is not a static member from A.
And that's pretty much all I was able to advance. I could work at C++ level and avoid any of these anoyances, but if I were able to use instances of A and B in Python (via Cython) interactively, it would help me in speedig my development pace.
Any help will be really appreciated, and I apologize if this question as been already answered and/or is available in a referece - I search SO, Cython reference and Smith's "Cython" book and I did not found this theme adressed.
Thanks in advance!
I have a partial (if horrendous) solution. I'm prepared to believe there's a better way, but I don't know it.
In cpp_bit.hpp:
class A {
public:
double myAMethod(double*) { return 0.0; }
};
typedef double (A::*A_f_ptr)(double *);
class B {
public:
double myBMethod(A& a, A_f_ptr f) {
double x = 0.1;
return (a.*f)(&x);
}
};
A_f_ptr getAMethod() {
return &A::myAMethod;
}
I've given the functions very basic implementations, just so I can check for really obvious crashes. I've also created a function pointer which returns a pointer to myAMethod. You'll need to do this for every method you want to wrap.
In py_bit.pyx
# distutils: language = c++
from cython.operator import dereference
cdef extern from "cpp_bit.hpp":
cdef cppclass A:
double myAMethod(double*)
cdef cppclass A_f_ptr:
pass
cdef cppclass B:
double myBMethod(A&, A_f_ptr)
cdef A_f_ptr getAMethod()
cdef class PyA:
cdef A* thisptr
def __cinit__(self):
self.thisptr = new A()
def __dealloc__(self):
del self.thisptr
cpdef myAMethod(self,double[:] o):
return self.thisptr.myAMethod(&o[0])
cdef class PyB:
cdef B* thisptr
def __cinit__(self):
self.thisptr = new B()
def __dealloc__(self):
del self.thisptr
cpdef myBMethod(self,PyA a):
return self.thisptr.myBMethod(dereference(a.thisptr),getAMethod())
I couldn't figure out how to typedef a member function pointer in Cython, so instead I created an empty cppclass with the same name. This works because cython just seems to use it for type-checking and nothing more, and since it includes cpp_bit.hpp (where it's defined) you can use it no problem.
All I've done is left the task of getting the member function pointer to c++ (in getAMethod, which I call). I don't think it's entirely satisfactory, but it looks workable, and is only a short extra c++ function for every member function you want to access. You could play with where you put it to encapsulate it more cleanly.
An alternative, untested approach: (Edit: further thought suggests this might be very tricky! Attempt this at your own risk!)
Personally, I'd be tempted to change the c++ interface so that myBMethod is defined as
double myBMethod(std::function<double (double*)>)
(since presumably you always call it with the A instance it's passed with). Then use lambda functions in c++(11!) to wrap the A instance and function together
b.myBMethod([&](double* d){ return a.myAMethod(d) };
It may then take a bit of hugely complicated Cython wrapping that I haven't yet considered, but it should be possible to convert a simple double (double*) function pointer to the c++ function object, and so use it more directly.
It's also possible that your actual design is more complicated in ways I haven't considered, and this approach isn't flexible enough anyway.
I'm trying to wrap a C++ library in which the logic is implemented as templatized functors in .hpp files, and I'm struggling to find the right way to expose the C++ functors as Cython/Python functions. How are functors like the one below supposed to be wrapped in Cython?
I believe this should be possible, at least for template classes and functions, according to the Cython 0.20 docs.
Note: I think I've figured out how to wrap normal C++ functions—the problem occurs when I'm trying to wrap a templatized functor, i.e. a template struct that overloads the () operator (making it act like a function when a data type is fixed).
Disclaimer: I'm a total novice in C++ and very new to Cython so apologies if I'm making obvious mistakes here.
The functor I'm trying to wrap:
#include <vector>
#include "EMD_DEFS.hpp"
#include "flow_utils.hpp"
template<typename NUM_T, FLOW_TYPE_T FLOW_TYPE= NO_FLOW>
struct emd_hat_gd_metric {
NUM_T operator()(const std::vector<NUM_T>& P, const std::vector<NUM_T>& Q,
const std::vector< std::vector<NUM_T> >& C,
NUM_T extra_mass_penalty= -1,
std::vector< std::vector<NUM_T> >* F= NULL);
};
My wrapper.pyx file:
# distutils: language = c++
from libcpp.vector cimport vector
cdef extern from "lib/emd_hat.hpp":
# Apparently `cppclass` is necessary here even though
# `emd_hat_gd_metric` is not a class...?
cdef cppclass emd_hat_gd_metric[NUM_T]:
NUM_T operator()(vector[NUM_T]& P,
vector[NUM_T]& Q,
vector[vector[NUM_T]]& C) except +
cdef class EMD:
cdef emd_hat_gd_metric *__thisptr
def __cinit__(self):
self.__thisptr = new emd_hat_gd_metric()
def __dealloc__(self):
del self.__thisptr
def calculate(self, P, Q, C):
# What goes here? How do I call the functor as a function?
return self.__thisptr(P, Q, C)
The above just gives a Calling non-function type 'emd_hat_gd_metric[NUM_T]' error when I try to compile it with cython --cplus wrapper.pyx.
Here's the full library I'm trying to wrap.
End goal: to be able to call emd_hat_gd_metric as a Cython/Python function, with arguments being NumPy arrays.
I couldn't find a real solution, but here's a workaround (that requires modifying the C++ code): just instantiate the template function with the data type you need in the C++ header, then declare that function normally in your .pyx file.
It's a little unwieldy if you need many different data types, but I only needed double. It would also be nicer if it wasn't necessary to modify the external library… but it works.
In the C++ some_library.hpp file:
Instantiate the functor with the data type you need (say, double):
template<typename T>
struct some_template_functor {
T operator()(T x);
};
// Add this:
some_template_functor<double> some_template_functor_double;
In the Cython .pyx file:
Declare the function normally (no need for cppclass):
cdef extern from "path/to/some_library.hpp":
cdef double some_template_functor_double(double x)
Then you can call some_template_functor_double from within Cython.