I'm trying to wrap a C++ api and I'm hitting a roadblock on some char* class members. It seems that boost-python will auto convert char const * and std::string types into python objects (based on this answer), but it balks at char* types. This is the error I get (in python):
TypeError: No to_python (by-value) converter found for C++ type: char*
It turns out that these particular char * members probably should have been declared as char const * since the strings are never altered.
I'm new to boost-python so maybe there is an obvious answer, but I'm not having much luck googling this one.
Is there an easy way to tell boost-python to auto convert these char* members?
(Unfortunately I can't change the declarations of char * to char const * since the API I am wrapping is not under my control.)
UPDATE:
Ok so I think that I need to add a custom converter to handle the char* members. I started writing one:
/** to-python convert for char* */
struct c_char_p_to_python_str
{
static PyObject* convert(char* s) {
return incref(object(const_cast<const char*>(s)).ptr());
}
};
// register the QString-to-python converter
to_python_converter<char*, c_char_p_to_python_str>();
Unfortunately this does not work. This is the error:
error: expected unqualified-id
to_python_converter<char*, c_char_p_to_python_str>();
^
Looking at the docs I can see that the template args have this signature:
template <class T, class Conversion, bool convertion_has_get_pytype_member=false>
Since char* isn't a class I'm guessing that's why this didn't work. Anyone have some insight?
UPDATE2:
Nope. Turns out to_python_converter needs to get called inside of the BOOST_PYTHON_MODULE call.
I got the to_python_converter working (with some modifications). I also wrote a function to convert form python and registered it with converter::registry::push_back. I can see my to_python code running, but the from_python code never seems to run.
Let's assume we're wrapping some third-party API, and set aside the awfulness of having those pointers exposed and mucking with them from the outside.
Here's a short proof of concept:
#include <boost/python.hpp>
namespace bp = boost::python;
class example
{
public:
example()
{
text = new char[1];
text[0] = '\0';
}
~example()
{
delete[] text;
}
public:
char* text;
};
char const* get_example_text(example* e)
{
return e->text;
}
void set_example_text(example* e, char const* new_text)
{
delete[] e->text;
size_t n(strlen(new_text));
e->text = new char[n+1];
strncpy(e->text, new_text, n);
e->text[n] = '\0';
}
BOOST_PYTHON_MODULE(so02)
{
bp::class_<example>("example")
.add_property("text", &get_example_text, &set_example_text)
;
}
Class example owns text, and is responsible for managing the memory.
We provide an external getter and setter function. The getter is simple, it just provides read access to the string. The setter frees the old string, allocates new memory of appropriate size, and copies the data.
Here's a simple test in python interpreter:
>>> import so02
>>> e = so02.example()
>>> e.text
''
>>> e.text = "foobar"
>>> e.text
'foobar'
Notes:
set_example_text() could perhaps take std::string or bp::object so that we have the lenght easily available, and potentially allow assignment from more than just strings.
If there are many member variables to wrap and the getter/setter pattern is similar, generate the code using templates, or even just few macros.
There may be a way to do this with the converters, I'll have a look into that tomorrow. However, as we're dealing with memory management here, i'd personally prefer to handle it this way, as it's much more obvious what's happening.
This expands on Dan's answer. I wrote some macro definitions which generate lambda expressions. The benefits of this approach are that it is not tied to a particular type or member name.
In the API I am wrapping, I have a few hundred classes to wrap. This allows me to make a single macro call for every char* class member.
Here is a modified version of Dan's example code:
#include <boost/python.hpp>
namespace bp = boost::python;
#define ADD_PROPERTY(TYPE, ATTR) add_property(#ATTR, SET_CHAR_P(TYPE, ATTR), \
GET_CHAR_P(TYPE, ATTR))
#define SET_CHAR_P(TYPE, ATTR) +[](const TYPE& e){ \
if (!e.ATTR) return ""; \
return (const char*)e.ATTR; \
}
#define GET_CHAR_P(TYPE, ATTR) +[](TYPE& e, char const* new_text){ \
delete[] e.ATTR; \
size_t n(strlen(new_text)); \
e.ATTR = new char[n+1]; \
strncpy(e.ATTR, new_text, n); \
e.ATTR[n] = '\0'; \
}
class example
{
public:
example()
{
text = new char[1];
text[0] = '\0';
}
~example()
{
delete[] text;
}
public:
char* text;
};
BOOST_PYTHON_MODULE(topics)
{
bp::class_<example>("example")
.ADD_PROPERTY(example, text);
}
Related
I have include a minimal working example below - it can be compiled using the typical pybind11 instructions (I use cmake).
I have an abstract base class, Abstract, which is pure virtual. I can easily wrap this in pybind11 using a "trampoline class" (this is well documented by pybind11).
Further, I have a concrete implementation of Abstract, ToBeWrapped, that is also wrapped using pybind11.
My issue is that I have some client code which accepts an arbitrary PyObject* (or, in the case of this example, pybind11's wrapper py::object) and expects to cast this to Abstract*.
However, as illustrated in my example, I am unable to cast the py::object to Abstract*.
I have no problem casting to ToBeWrapped* and then storing that as an Abstract*', however this would require my client code to know ahead of time what kind ofAbstract*` the python interpreter is sending, which defeats the purpose of the abstract base class.
TL;DR
Is it possible to modify this code such that the client accessMethod is able to arbitrarily handle an Abstract* passed from the python interpreter?
#include <pybind11/pybind11.h>
#include <iostream>
namespace py = pybind11;
// abstract base class - cannot be instantiated on its own
class Abstract
{
public:
virtual ~Abstract() = 0;
virtual std::string print() const = 0;
};
Abstract::~Abstract(){}
// concrete implementation of Abstract
class ToBeWrapped : public Abstract
{
public:
ToBeWrapped(const std::string& msg = "heh?")
: myMessage(msg){};
std::string print() const override
{
return myMessage;
}
private:
const std::string myMessage;
};
// We need a trampoline class in order to wrap this with pybind11
class AbstractPy : public Abstract
{
public:
using Abstract::Abstract;
std::string print() const override
{
PYBIND11_OVERLOAD_PURE(
std::string, // return type
Abstract, // parent class
print, // name of the function
// arguments (if any)
);
}
};
// I have client code that accepts a raw PyObject* - this client code base implements its
// own python interpreter, and calls this "accessMethod" expecting to convert the python
// object to its c++ type.
//
// Rather than mocking up the raw PyObject* method (which would be trivial) I elected to
// keep this minimal example 100% pybind11
void accessMethod(py::object obj)
{
// runtime error: py::cast_error
//Abstract* casted = obj.cast<Abstract*>();
// this works
Abstract* casted = obj.cast<ToBeWrapped*>();
}
PYBIND11_MODULE(PyMod, m)
{
m.doc() = R"pbdoc(
This is a python module
)pbdoc";
py::class_<Abstract, AbstractPy>(m, "Abstract")
.def("print", &Abstract::print)
;
py::class_<ToBeWrapped>(m, "WrappedClass")
.def(py::init<const std::string&>())
;
m.def("access", &accessMethod, "This method will attempt to access the wrapped type");
}
You need to declare the hierarchy relationship, so this:
py::class_<ToBeWrapped>(m, "WrappedClass")
should be:
py::class_<ToBeWrapped, Abstract>(m, "WrappedClass")
I have a legacy code in C++ (which would be a huge pain to edit) and I need to use it in Python 2 for speed reasons.
I have two classes. One is responsible for loading huge amount of data from memory, in a form of std::string and converting it to internal representation MiddleClass. Second one is converting it from internal representation MiddleClass back to std::string.
class Load {
Load(const std::string & data) { ... };
MiddleClass load() { ... };
};
class Save {
Save(std::string & data) { .... };
void save(const MiddleClass & middleclass) { ... };
};
My goal is, to use this setup in Python 2 like this:
import datahandler # my lib
import requests
request = request.get("url-to-data")
loader = datahandler.Load(request.content) # my C++ class Load
internal_representation = loader.load()
.
.
.
result_variable = str() # or None or something not important
saver = datahandler.Save(result_variable) # my C++ class Save
saver.save(internal_representation)
How can I achieve this?
I've run into trouble, right from the start.
Simple variant:
BOOST_PYTHON_MODULE(datahandler)
{
class_<MiddleClass>("MiddleClass");\
// some .defs - not important
class <Load>("Load", init<const std::string &>())
.def("load". &Load::load);
class <Save>("Save", init<std::string &>())
.def("save". &Save::save);
}
Will compile, no worries, but data which are loaded are somehow mangled, which leads me to thinking, that I am doing it terribly wrongly.
Also I found this bit offtopic SO question, which told me, that I can't have std::string &, because Python strings are immutable.
So conclusion: I have no idea what to do now :( Can anyone here help me? Thanks.
Take as reference this working example.
Define your C++ classes. For instance:
class MiddleClass {
public:
explicit MiddleClass(const std::string& data) : parent_data_(data) {}
void print() {
std::cout << parent_data_ << std::endl;
}
private:
std::string parent_data_;
};
class Loader {
public:
explicit Loader(const std::string& data) :
data_(data){
};
MiddleClass load() {
return MiddleClass(data_);
};
private:
std::string data_;
};
Create the boost bindings
boost::python::class_<MiddleClass>("MiddleClass",
boost::python::init<const std::string&>(boost::python::arg("data"), ""))
.def("print_data", &MiddleClass::print);
boost::python::class_<Loader>("Loader",
boost::python::init<const std::string&>(boost::python::arg("data"), ""))
.def("load", &Loader::load);
Install your library in the right python site-package.
Enjoy it in python:
from my_cool_package import MiddleClass, Loader
example_string = "whatever"
loader = Loader(data=example_string)
# Get the middle class
middle_class = loader.load()
# Print the data in the middle class
middle_class.print_data()
The expected output:
whatever
So, I have found a solution. Prove me wrong, but I think, that what am I trying to achieve is impossible.
Python has immutable strings, so passing a "reference" of string to function and expecting ability to change it from inside a function is simply not valid.
Take this code as an example:
variable = "Hello"
def changer(var):
var = "Bye"
changer(variable)
print(variable)
Prints "Hello". In Python, you can't make it work differently. (although to be exact, it is still being passed as a reference, but when you modify Python string, you just create a new one and a new reference).
So, how to get arround this?
Simple! Create a C++ wrapper, that will handle passing reference on std::string and return copy of resulting string. Not very effective, but you probably can't make it better.
Sample code of SaveWrapper class:
class SaveWrapper {
public:
// some constructor
std::string save(MiddleClass & value) {
std::string result;
Save saver(result);
saver.save(value);
return result;
}
};
Which can be easily "ported" to Python!
I have a C++ function which returns a raw float pointer, and another C++ function which accepts a raw float pointer as an argument. Something like:
float* ptr = something;
float* get_ptr(void) { return ptr; }
void use_ptr(float* ptr) { do_work(ptr); }
I want to be able to pass around pointers using Python. Something like this:
import my_native_functions as native
ptr = native.get_ptr()
native.use_ptr(ptr)
I am using pybind11 to create my native python module but I don't know how to create the bindings for the get_ptr() function. If I just do the following:
PYBIND11_MODULE(my_native_functions, m)
{
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
the get_ptr() function returns a Python Float object. I guess this makes sense because there are no pointer types in python. However, because this is now a simple Float, when I call the use_ptr() function and iterate over the pointer in C/C++, only the first element of the array is correct. The rest are garbage. To fix this, in C++, I have to cast my pointer to/from std::size_t. By doing this, everything works just fine.
However, I would like to ask: Is there a "right way" of achieving the above without the casting to/from std::size_t with pybind11?
In case you are curious why I do that:
I do understand that what I am doing is not type-safe. Also, I never touch the pointer/integer on the Python side. I just retrieve it from one native module and pass it to another. Also, I cannot cast the pointer to some kind of numpy view because the pointer is not always on the CPU. Sometimes I want to pass around CUDA pointers. Creating a py::array_t from a CUDA pointer is not possible unless I copy the data (and I do not want to do that).
Thank you.
Wrap the raw pointer in a custom "smart" pointer class (only pretending to be smart really) as described here. You can add some additional information to this class while you're at it, such as the size of the array element and the number of elements. This would make it a generalised array descriptor on the C++ side (but not on the Python side because you are not exposing the raw pointer to Python).
For a simpler option, just wrap your pointer in any old class in order to hide it from Python. No need to expose it to Python as a custom smart pointer. Here's an example that does just that:
#include <pybind11/pybind11.h>
#include <memory>
#include <iostream>
namespace py = pybind11;
template <class T> class ptr_wrapper
{
public:
ptr_wrapper() : ptr(nullptr) {}
ptr_wrapper(T* ptr) : ptr(ptr) {}
ptr_wrapper(const ptr_wrapper& other) : ptr(other.ptr) {}
T& operator* () const { return *ptr; }
T* operator->() const { return ptr; }
T* get() const { return ptr; }
void destroy() { delete ptr; }
T& operator[](std::size_t idx) const { return ptr[idx]; }
private:
T* ptr;
};
float array[3] = { 3.14, 2.18, -1 };
ptr_wrapper<float> get_ptr(void) { return array; }
void use_ptr(ptr_wrapper<float> ptr) {
for (int i = 0; i < 3; ++i)
std::cout << ptr[i] << " ";
std::cout << "\n";
}
PYBIND11_MODULE(Ptr,m)
{
py::class_<ptr_wrapper<float>>(m,"pfloat");
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
I am new to the business of writing custom Python modules and I am a bit confused how Capsules work. I use Python 2.7.6 from the system OSX installation and try to use Capsules (as recommended for Python > 2.7) for passing pointers around (before they used PyCObject for that). My code does not work at the moment and I would like to get some insights how things should be handled in principle here. The code should define a class LuscherClm and I want be able to do the following:
>>> c40=Luscher(4,0)
>>>
>>> c40(0.12)
>>> <print the result of the evaluation>
First question: at the moment I would have to do something like:
>>> c40=Luscher.init(4,0)
>>>
>>> c40.eval(0.12)
Segfault
My first question is therefore: how do I have to modify the method table to have more operator-style casts instead of the member functions init and eval.
However, my code has other problems and here is the relevant part (the underlying C++ class works smoothly, I use it in production a lot):
The destructor:
//destructor
static void clm_destruct(PyObject* capsule){
void* ptr=PyCapsule_GetPointer(capsule,"zetfunc");
Zetafunc* zetptr=static_cast<Zetafunc*>(ptr);
delete zetptr;
return;
}
The constructor: it returns the pointer to the capsule. I do not know whether this is correct. Because in this case when I call, clm=LuscherClm.init(l,m), the clm object is a PyCapsule and has no attribute eval so that I cannot call clm.eval(x) on that. How should this be handled?
//constructor
static PyObject* clm_init(PyObject* self, PyObject *args){
//return value
PyObject* result=NULL;
//parse variables
unsigned int lval=0;
int mval=0;
if(!PyArg_ParseTuple(args,"li",&lval,&mval)){
::std::cout << "Please specify l and m!" << ::std::endl;
return result;
}
//class instance:
Zetafunc* zetfunc=new Zetafunc(lval,mval);
instanceCapsule=PyCapsule_New(static_cast<void*> (zetfunc),"zetfunc",&clm_destruct);
return instanceCapsule;
}
So how is the capsule passed to the evaluate function? the code below is not correct since I have not updated it after moving from CObjects to Capsules. Shall the capsule be a global variable (I do not like that) or how can I pass it to the evaluation function? Or shall I call it on self, but what is self at the moment?
//evaluate the function
static PyObject* clm_evaluate(PyObject* self, PyObject* args){
//get the PyCObject from the capsule:
void* tmpzetfunc=PyCapsule_GetPointer(instanceCapsule,"zetfunc");
if (PyErr_Occurred()){
std::cerr << "Some Error occured!" << std::endl;
return NULL;
}
Zetafunc* zetfunc=static_cast< Zetafunc* >(tmpzetfunc);
//parse value:
double x;
if(!PyArg_ParseTuple(args,"d",&x)){
std::cerr << "Specify a number at which you want to evaluate the function" << std::endl;
return NULL;
}
double result=(*zetfunc)(x).re();
//return the result as a packed function:
return Py_BuildValue("d",result);
}
//methods
static PyMethodDef LuscherClmMethods[] = {
{"init", clm_init, METH_VARARGS, "Initialize clm class!"},
{"eval", clm_evaluate, METH_VARARGS, "Evaluate the Zeta-Function!"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
Python < 3 initialisation function:
PyMODINIT_FUNC
initLuscherClm(void)
{
PyObject *m = Py_InitModule("LuscherClm", LuscherClmMethods);
return;
}
Can you explain to me what is wrong and why? I would like to stay away from SWIG or boost if possible, since this module should be easily portable and I want to avoid having to install additional packages every time I want to use it somewhere else.
Further: what is the overhead produced by the C/API in calling the function? I need to call it an order of O(10^6) times and I would still like it to be fast.
Ok, I am using boost.python now but I get a segfault when I run object.eval(). That is my procedure now:
BOOST_PYTHON_MODULE(threevecd)
{
class_< threevec<double> >("threevecd",init<double,double,double>());
}
BOOST_PYTHON_MODULE(LuscherClm)
{
class_<Zetafunc>("LuscherClm",init<int,int, optional<double,threevec<double>,double,int> >())
.def("eval",&Zetafunc::operator(),return_value_policy<return_by_value>());
boost::python::to_python_converter<dcomplex,dcomplex_to_python_object>();
}
dcomplex is my own complex number implementation. So I had to write a converter:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
if(fabs(comp.im())<std::numeric_limits<double>::epsilon()){
boost::python::object result=boost::python::object(complex<double>(comp.re(),comp.im()));
return boost::python::incref(result.ptr());
}
else{
return Py_BuildValue("d",comp.re());
}
}
};
Complex128 is a numpy extension which is not understood by boost. So my questions are:
1) how can I return a complex number as a python datatype (is complex a standard python type?)
2) Why do I get a segfault. My result in my testcase is real so it should default to the else statement. I guess that the pointer runs out of scope and thats it. But even in the if-case (where I take care about ref-increments), it segfaults. Can someone help me with the type conversion issue?
Thanks
Thorsten
Ok, I got it. The following converter does the job:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
PyObject* result;
if(std::abs(comp.im())<=std::numeric_limits<double>::epsilon()){
result=PyFloat_FromDouble(comp.re());
}
else{
result=PyComplex_FromDoubles(comp.re(),comp.im());
}
Py_INCREF(result);
return result;
}
};
Using this converter and the post by Wouter, I suppose my question is answered. Thanks
Swig's manual is kinda confusing to me. I am wrapping my C library to python so that I can test my C code in python. Now I want to know how I can access the C pointer address in Python,
for example, this is the code I have
typedef struct _buffer_t {
char buf[1024];
char *next_ptr;
} buffer_t;
void parse_buffer(buffer_t * buf_p) {
buf_p -> next_ptr ++;
}
What I wanted to do is below, in C code
buffer_t my_buf;
my_buf.next_ptr = my_buf.buf;
parse_buffer(&my_buf);
expect_equal(&(my_buf.buf)+1, my_buf.next_ptr);
How do I do the same thing in Python?
After import the SWIG wrapped module, I have the buffer_t class in python.
The problem is that SWIG is trying to wrap the buffer as a String in Python, which is not just a pointer. The synthesised assignment for next_ptr will allocate memory and make a string copy, not just do a pointer assignment. You can work around this in several ways.
The simplest is to use %extend to add a "reset buffer" method in Python:
%extend {
void resetPtr() {
$self->next_ptr=$self->buf;
}
}
which can then be called in Python to make the assignment you want.
Alternatively if you want to force the buffer to be treated as just another pointer you could force SWIG to treat both members as void* instead of char* and char[]. I had hoped that would be as simple as a %apply per type, but it seems not to work correctly for memberin and memberout in my testing:
%apply void * { char *next };
%apply void * { char buf[ANY] };
Given that the memberin/memberout typemaps are critical to making this work I think %extend is by far the cleanest solution.