Develop Reference

How can I transfer a pointer variable from one cython module to another using a Python script

Assume we have a cython class A with a pointer to float like in
# A.pyx
cdef class A:
cdef float * ptr
We also have a cython class B in another module which needs access to the data under ptr:
# B.pyx
cdef class B:
cdef float * f_ptr
cpdef submit(self, ptr_var):
self.f_ptr= get_from( ptr_var ) # ???
The corresponding Python code using A and B might be something like
from A import A
from B import B
a = A()
b = B()
ptr = a.get_ptr()
b.submit(ptr)
How can we define get_ptr() and what would we use for get_from in B?

The solution is to wrap the pointer variable into a Python object. Module libc.stdint offers a type named uintptr_t which is an integer large enough for storing any kind of pointer. With this the solution might look as follows.
from libc.stdint cimport uintptr_t
cdef class A:
cdef float * ptr
def get_ptr(self):
return <uintptr_t>self.ptr
The expression in angle brackets <uintptr_t> corresponds to a cast to uintptr_t. In class B we then have to cast the variable back to a pointer to float.
from libc.stdint cimport uintptr_t
cdef class B:
cdef float * f_ptr
cpdef submit(self, uintptr_t ptr_var):
self.f_ptr= <float *>ptr_var
This works for any kind of pointers not only for pointers to float. One has to make sure that both modules (A and B) deal with the same kind of pointer since that information is lost once the pointer is wrapped in a uintptr_t.

Return a struct from C to Python using Cython

I am trying to pass a struct back into my Python from a c file. Let's say I have a file pointc.c like this:
typedef struct Point {
int x;
int y;
} Point;
struct Point make_and_send_point(int x, int y);
struct Point make_and_send_point(int x, int y) {
struct Point p = {x, y};
return p;
}
Then I set-up a point.pyx file like this:
"# distutils: language = c"
# distutils: sources = pointc.c
cdef struct Point:
int x
int y
cdef extern from "pointc.c":
Point make_and_send_point(int x, int y)
def make_point(int x, int y):
return make_and_send_point(x, y) // This won't work, but compiles without the 'return' in-front of the function call
How do I get the returned struct into my Python? Is this kind of thing only possible by creating a struct in the Cython and sending by reference to a void c function?
As a reference, my setup.py is:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules = cythonize(
"point.pyx",
language="c"
)
)

Most typically you would write some kind of wrapper class that holds the c-level struct, for example:
# point.pyx
cdef extern from "pointc.c":
ctypedef struct Point:
int x
int y
Point make_and_send_point(int x, int y)
cdef class PyPoint:
cdef Point p
def __init__(self, x, y):
self.p = make_and_send_point(x, y)
#property
def x(self):
return self.p.x
#property
def y(self):
return self.p.y
In usage
>>> import point
>>> p = point.PyPoint(10, 10)
>>> p.x
10

Cython's default behaviour given a struct is to convert it to a Python dictionary, which may be good enough for you. (This only works for structs made up of simple types though).
There's a couple of reasons why this isn't working. First you should do cdef extern from from headers, not source files, otherwise you get errors about multiple definitions (I assume this is just a mistake in creating your minimal example). Second you need to put the definition of Point within your cdef extern block:
cdef extern from "pointc.h":
cdef struct Point:
int x
int y
If you don't do that then Cython creates a mangled internal name for your struct (__pyx_t_5point_Point) which doesn't match the C function signature and thus it fails.
With this corrected, you get the correct default behaviour of converting structs to dicts. (This should work both ways - you can convert dicts back to structs). In the event that this isn't what you want, follow #chrisb's answer

Using a struct from a different library in Cython

I am trying to call a C function in Cython and the header looks like this:
#include <stdio.h>
#include <stdint.h>
#include <inttypes.h>
#include <ctype.h>
#include <unistd.h>
#include <math.h>
#include <apriltag.h>
#include <tag36h11.h>
#include <common/getopt.h>
#include <common/image_u8.h>
#include <common/image_u8x4.h>
#include <common/pjpeg.h>
#include <common/zarray.h>
apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);
As you can see, I want to return an array of structs, which are type defined to be apriltag_detection_t. According to the documentation, in order to be able to use this in Cython, I must define some sort of pxd file which is essentially a copy of header.
However, apriltag_detection_t is a type that is already defined in apriltag.h. Furthermore, apriltag_detection_t has members that are already defined in apriltag.h. Do I have to recursively redefine all of these types (by hand) in the Cython file before being able to use this library? Where would I write them?
Thanks!
UPDATE 6
Finally at the step of wrapping a function!
from libc.stdint cimport uint8_t
cdef extern from "<apriltag.h>":
cdef struct apriltag_detection:
int id
double c[2]
double p[4][2]
ctypedef apriltag_detection apriltag_detection_t
cdef extern from "tag36h11_detector/tag36h11_detector.h":
apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);
def detect(width, height, frame):
return scan_frame(width, height, frame)
tag36h11_detector.pyx:15:21: Cannot convert 'apriltag_detection_t *' to Python object
apriltag_detection_t* is meant to be an array of structs
UPDATE 5
This seems to have worked.
from libc.stdint cimport uint8_t
cdef extern from "<apriltag.h>":
cdef struct apriltag_detection:
int id
double c[2]
double p[4][2]
ctypedef apriltag_detection apriltag_detection_t
cdef extern from "tag36h11_detector/tag36h11_detector.h":
apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);
UPDATE 4
Solved the previous issues by importing the necessary types.
from libc.stdint cimport uint8_t
cdef extern from "apriltag.h":
cdef struct apriltag_detection:
int id
double c[2]
double p[4][2]
ctypedef apriltag_detection apriltag_detection_t
cdef extern from "tag36h11_detector.h":
apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);
tag36h11_detector.c:533:10: fatal error: 'apriltag.h' file not found
I'm not sure where this is coming from because my header file, as provided in the original post, required <apriltag.h> and not "apriltag.h". This is what my setup.py looks like.
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules=cythonize(Extension(\
name='tag36h11_detector', \
sources=["tag36h11_detector.pyx", \
"tag36h11_detector/tag36h11_detector.c"], \
include_path=["/usr/local/include/apriltag"], \
libraries=["apriltag"])))
UPDATE 3
cdef extern from "apriltag.h":
cdef struct apriltag_detection:
int id
double c[2]
double p[4][2]
ctypedef apriltag_detection apriltag_detection_t
cdef extern from "tag36h11_detector.h":
apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);
tag36h11_detector.pyx:10:60: 'uint8_t' is not a type identifier
UPDATE 2
This is my current code and the following is the compilation error
// tag36h11_detector.pyx
cdef extern from "apriltag.h":
ctypedef apriltag_detection_t:
int id
double c[2]
double p[4][2]
cdef extern from "tag36h11_detector.h":
apriltag_detection_t* scan_frame(int width, int height, uint8_t* data);
// apriltag.h
...
typedef struct apriltag_detector apriltag_detector_t;
...
tag36h11_detector.pyx:2:33: Syntax error in ctypedef statement
UPDATE 1
So I'm trying to interface with the above header file with Python (which I wrote and implemented) with types defined in apriltag.h (from a library).
cdef extern from "apriltag.h":
struct apriltag_detection:
int id
double c[2]
double p[4][2]
cdef extern from "tag36h11_detector.h":
struct apriltag_detection* scan_frame(int width, int height, uint8_t* data);
When I try compiling the above, I get
tag36h11_detector.pyx:8:29: Syntax error in struct or union definition

This is basically covered in this part of cython documentation, which says that you only need to import parts, which you will be using in your cython code.
For example let's take a look at the following C interface:
#struct.h
struct Needed{
int a;
};
struct NotNeeded{
int b;
};
struct Combined{
struct Needed needed;
struct NotNeeded notneeded;
};
struct Combined create(void);
You would like to call the function create and use the value a from the Needed struct, that means you have to import struct Needed and parts of struct Combined but not NotNeeded in your cython code:
#struct_import.pyx
cdef extern from "struct.h":
struct Needed: # use "ctypedef struct Needed" if defined with typedef in h-file!
int a
struct Combined: #NotNeeded is missing!
Needed needed
Combined create()
def get_needed():
return create().needed.a #should be 42!
Now, using setup.py (its content can be seen further below, the same goes for the content of struct.cto) we get the expected result:
[] python setup.py build_ext --inplace
[] python -c "python -c "import struct_import as si; print si.get_needed()"
42
If you use cython to glue some C code together, it is possible, even less is necessary. In oour example, if we had a C-function which would extract the needed value from the Combined struct:
#struct.h
...
int extract(struct Combined combined);//returns combined.needed.a
We could use it as follows in the pyx-file:
#struct_import.pyx
cdef extern from "struct.h":
struct Combined:
pass #nothing imported!
Combined create()
int extract(Combined combined)
def get_needed():
return extract(create()) #should be 42!
And despite we didn't import the Needed struct at all, it works as good as the first version.
So if it becomes a chore to import all those structs, one could extend the c interface to make it unnecessary.
To make the example complete, here the missing setup.py and struct.c files:
#setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules=cythonize(Extension(
name='struct_import',
sources = ["struct_import.pyx", "struct.c"]
)))
and
//struct.c
#include "struct.h"
struct Combined create(){
struct Combined res;
res.needed.a=42;
res.notneeded.b=21;
return res;
}
int extract(struct Combined combined){
return combined.needed.a;
}

Wrapping std::array in Cython and Exposing it to memory views

It seems that there is currently a pull request in Cython's repo to wrap c++ std::array but until then, I could use some help. I am currently wrapping the std::array like so:
cdef extern from "<array>" namespace "std" nogil:
cdef cppclass array2 "std::array<double, 2>":
array2() except+
double& operator[](size_t)
This works, but I have to loop over a cython memory view say, double arr[:], and copy the values one by one. Is there an easier way to do this? Essentially I'd like to do the following:
cdef double arr[2]
arr[0] = 1.0
arr[1] = 2.0
cdef array2 array2_arr = arr
#and the reverse
cdef array2 reverse
reverse[0] = 1.0
reverse[1] = 2.0
cdef double reverse_arr[2] = reverse
Is this completely unreasonable? As a result, working with std::array is extremely tedious because I have a to have a for-loop to copy values from cython to c++ and from c++ to cython. Furthermore, since cython doesn't give us the ability to have non-type template parameters, I have to define a wrapper for every variation of std::array in my code. Any suggestions on how to efficiently work with std::array would be greatly appreciated.
edit:
I'm now able to go from a memory view to an array2 type using the following:
def __cinit__(self, double[:] mem):
cdef array2 *arr = <array2 *>(&mem[0])
But it seems that no matter what I do, I cannot get cython to convert an array2 type to a memoryview:
cdef array2 arr = self.thisptr.getArray()
# error: '__pyx_t_1' declared as a pointer to a reference of type 'double &'
cdef double[::1] mview = <double[:2]>(&arr[0])
#OR
# Stop must be provided to indicate shape
cdef double[::1] mview = <double[::2]>(&arr[0])
Please help me figure out how to cast a C++ pointer to a memory view. Every combination I have tried to date has resulted in some kind of casting error.
EDIT:
I found that I am to perform the following syntax with no error using a newer version of Cython (I was using Cythong 0.22) and upgraded to 0.23.5.
cdef double[::1] mview = <double[:4]>(&arr[0])
However, if I attempt to return mview from the function I am using it in, I get garbage memory. Returning the memoryview to the pointer of my array loses scope and therefore destructs my array automatically. As soon as I figure out how to properly return my array, I'll attempt to update the official answer.

After much fiddling, I found the answer to my question.
Definition of array and class that uses array:
cdef extern from "<array>" namespace "std" nogil:
cdef cppclass array4 "std::array<int, 4>":
array4() except+
int& operator[](size_t)
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass ArrayFun:
ArrayFun(array4&)
array4 getArray()
Python implementation
cdef class PyArrayFun:
cdef ArrayFun *thisptr # hold a C++ instance which we're wrapping
def __cinit__(self, int[:] mem):
#
# Conversion from memoryview to std::array<int,4>
#
cdef array4 *arr = <array4 *>(&mem[0])
self.thisptr = new ArrayFun(arr[0])
def getArray(self):
cdef array4 arr = self.thisptr.getArray()
#
# Conversion from std::array<int, 4> to memoryview
#
cdef int[::1] mview = <int[:4]>(&arr[0])
cdef int[::1] new_view = mview.copy()
for i in range(0,4):
print ("arr is ", arr[i])
print("new_view is ", new_view[i])
# A COPY MUST be returned because arr goes out of scope and is
# default destructed when this function exist. Therefore we have to
# copy again. This kinda of sucks because we have to copy the
# internal array out from C++, and then we have to copy the array
# out from Python, therefore 2 copies for one array.
return mview.copy()

How to expose a function returning a C++ object to Python without copying the object?

In another question I learnt how to expose a function returning a C++ object to Python by copying the object. Having to perform a copy does not seem optimal. How can I return the object without copying it? i.e. how can I directly access the peaks returned by self.thisptr.getPeaks(data) in PyPeakDetection.getPeaks (defined in peak_detection_.pyx)?
peak_detection.hpp
#ifndef PEAKDETECTION_H
#define PEAKDETECTION_H
#include <string>
#include <map>
#include <vector>
#include "peak.hpp"
class PeakDetection
{
public:
PeakDetection(std::map<std::string, std::string> config);
std::vector<Peak> getPeaks(std::vector<float> &data);
private:
float _threshold;
};
#endif
peak_detection.cpp
#include <iostream>
#include <string>
#include "peak.hpp"
#include "peak_detection.hpp"
using namespace std;
PeakDetection::PeakDetection(map<string, string> config)
{
_threshold = stof(config["_threshold"]);
}
vector<Peak> PeakDetection::getPeaks(vector<float> &data){
Peak peak1 = Peak(10,1);
Peak peak2 = Peak(20,2);
vector<Peak> test;
test.push_back(peak1);
test.push_back(peak2);
return test;
}
peak.hpp
#ifndef PEAK_H
#define PEAK_H
class Peak {
public:
float freq;
float mag;
Peak() : freq(), mag() {}
Peak(float f, float m) : freq(f), mag(m) {}
};
#endif
peak_detection_.pyx
# distutils: language = c++
# distutils: sources = peak_detection.cpp
from libcpp.vector cimport vector
from libcpp.map cimport map
from libcpp.string cimport string
cdef extern from "peak.hpp":
cdef cppclass Peak:
Peak()
Peak(Peak &)
float freq, mag
cdef class PyPeak:
cdef Peak *thisptr
def __cinit__(self):
self.thisptr = new Peak()
def __dealloc__(self):
del self.thisptr
cdef copy(self, Peak &other):
del self.thisptr
self.thisptr = new Peak(other)
def __repr__(self):
return "<Peak: freq={0}, mag={1}>".format(self.freq, self.mag)
property freq:
def __get__(self): return self.thisptr.freq
def __set__(self, freq): self.thisptr.freq = freq
property mag:
def __get__(self): return self.thisptr.mag
def __set__(self, mag): self.thisptr.mag = mag
cdef extern from "peak_detection.hpp":
cdef cppclass PeakDetection:
PeakDetection(map[string,string])
vector[Peak] getPeaks(vector[float])
cdef class PyPeakDetection:
cdef PeakDetection *thisptr
def __cinit__(self, map[string,string] config):
self.thisptr = new PeakDetection(config)
def __dealloc__(self):
del self.thisptr
def getPeaks(self, data):
cdef Peak peak
cdef PyPeak new_peak
cdef vector[Peak] peaks = self.thisptr.getPeaks(data)
retval = []
for peak in peaks:
new_peak = PyPeak()
new_peak.copy(peak) # how can I avoid that copy?
retval.append(new_peak)
return retval

If you have a modern C++ compiler and can use rvalue references, move constructors and std::move it's pretty straight-forward. I think the easiest way is to create a Cython wrapper for the vector, and then use a move constructor to take hold of the contents of the vector.
All code shown goes in peak_detection_.pyx.
First wrap std::move. For simplicity I've just wrapped the one case we want (vector<Peak>) rather than messing about with templates.
cdef extern from "<utility>":
vector[Peak]&& move(vector[Peak]&&) # just define for peak rather than anything else
Second, create a vector wrapper class. This defines the Python functions necessary to access it like a list. It also defines a function to call the move assignment operator
cdef class PyPeakVector:
cdef vector[Peak] vec
cdef move_from(self, vector[Peak]&& move_this):
self.vec = move(move_this)
def __getitem__(self,idx):
return PyPeak2(self,idx)
def __len__(self):
return self.vec.size()
Then define the class the wraps the Peak. This is slightly different to your other class in that it doesn't own the Peak it wraps (the vector does). Otherwise, most of the functions remain the same
cdef class PyPeak2:
cdef int idx
cdef PyPeakVector vector # keep this alive, since it owns the peak rather that PyPeak2
def __cinit__(self,PyPeakVector vec,idx):
self.vector = vec
self.idx = idx
cdef Peak* getthisptr(self):
# lookup the pointer each time - it isn't generally safe
# to store pointers incase the vector is resized
return &self.vector.vec[self.idx]
# rest of functions as is
# don't define a destructor since we don't own the Peak
Finally, implement getPeaks()
cdef class PyPeakDetection:
# ...
def getPeaks(self, data):
cdef Peak peak
cdef PyPeak new_peak
cdef vector[Peak] peaks = self.thisptr.getPeaks(data)
retval = PyPeakVector()
retval.move_from(move(peaks))
return retval
Alternative approaches:
If Peak was nontrivial you could go for an approach where you call move on Peak rather that on the vector, as you construct your PyPeaks. For the case you have here move and copy will be equivalent for `Peak.
If you can't use C++11 features you'll need to change the interface a little. Instead of having your C++ getPeaks function return a vector have it take an empty vector reference (owned by PyPeakVector) as an input argument and write into it. Much of the rest of the wrapping remains the same.

There are two projects that accomplish interfacing with C++ code into Python that have withstood the test of time Boost.Python and SWIG. Both work by adding additional markup to pertinent C/C++ code and generating dynamically loaded python extension libraries (.so files) and the related python modules.
However, depending on your use case there may still be some additional markup that looks like "copying." However, the copying should not be as extensive and it will all be exposed in the C++ code rather than being explicitly copied verbatim in Cython/Pyrex.