Develop Reference

python bytearray to C++ object

I'm wondering if I could get some help. For context, I'm using some C++ libraries to generate some large (think hundreds of Mb) objects that I want to send over a network from a server to a client.
On the server, I've got the following:
PyObject* CKKSwrapper::SerializePrivateKey() {
std::string s;
std::ostringstream os(s);
Serial::Serialize(m_keys.publicKey, os, SerType::BINARY);
auto msg = os.str();
return PyBytes_FromString(&msg[0]);
}
which gives me some Python object. I then send this directly to the client via python sockets. I'm reading it in like
def _safe_recv_abstract(socket: Socket, deserializer_func):
expected_length = _get_obj_size(socket)
running_length = 0
running_msg = bytearray()
while running_length < expected_length:
msg = socket.recv(expected_length)
if msg:
running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
running_length += len(msg)
socket.send(_add_header_to_payload(b"ACK"))
logger.debug("_safe_recv_unenc_obj: Received all data")
if optional_pycrypto_deserialize_func:
return deserializer_func(running_msg)
return running_msg
two things:
Accumulator (from cppwrapper.Accumulator() above) looks like
PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b){
return PyByteArray_Concat(&a, &b);
}
deserializer_func calls an underlying C++ function that looks like
void CKKSwrapper::DeserializeX(
const boost::python::list &pyvals) {
auto msg= pythonListToCppVectorBytes(pyvals);
LPPrivateKey<DCRTPoly> sk;
std::istringstream is(string(msg.begin(), msg.end()));
Serial::Deserialize(sk, is, SerType::BINARY);
this->m_keys.secretKey = sk;
}
I'm running into the following error:
Boost.Python.ArgumentError: Python argument types in
CKKSwrapper.Accumulator(bytearray, bytearray)
did not match C++ signature:
Accumulator(pycrypto::CKKSwrapper {lvalue}, _object*, _object*)
I completely understand what it is saying and that the types are wrong but I'm not sure WHY. From the docs
PyObject* PyByteArray_Concat(PyObject *a, PyObject *b)
Return value: New reference.
Concat **bytearrays** a and b and return a new bytearray with the result.
If I understand correctly, I AM passing in bytearrays but it says that it is expecting objects?
The reason I'm trying to do it this way is that when I use a bytearray or a list for the accumulation, i.e
while running_length < expected_length:
msg = socket.recv(expected_length)
if msg:
running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
running_length += len(msg)
the memory usage and runtime blow up

Partial answer for the simple mistake:
PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b)
This should be
PyObject* CKKSwrapper::Accumulator(PyObject* a, PyObject* b)
PyObject is never passed by value, always by pointer. In practice all useful Python objects look like:
struct Something{
PyObject ob_base;
Useful data;
};
This is essentially a C version of C++ inheritance. By passing by value you're losing everything but ob_base, exactly like passing a C++ derived class as its base by value.

Boost Python 2: Constructors using `std::string &`

I have a legacy code in C++ (which would be a huge pain to edit) and I need to use it in Python 2 for speed reasons.
I have two classes. One is responsible for loading huge amount of data from memory, in a form of std::string and converting it to internal representation MiddleClass. Second one is converting it from internal representation MiddleClass back to std::string.
class Load {
Load(const std::string & data) { ... };
MiddleClass load() { ... };
};
class Save {
Save(std::string & data) { .... };
void save(const MiddleClass & middleclass) { ... };
};
My goal is, to use this setup in Python 2 like this:
import datahandler # my lib
import requests
request = request.get("url-to-data")
loader = datahandler.Load(request.content) # my C++ class Load
internal_representation = loader.load()
.
.
.
result_variable = str() # or None or something not important
saver = datahandler.Save(result_variable) # my C++ class Save
saver.save(internal_representation)
How can I achieve this?
I've run into trouble, right from the start.
Simple variant:
BOOST_PYTHON_MODULE(datahandler)
{
class_<MiddleClass>("MiddleClass");\
// some .defs - not important
class <Load>("Load", init<const std::string &>())
.def("load". &Load::load);
class <Save>("Save", init<std::string &>())
.def("save". &Save::save);
}
Will compile, no worries, but data which are loaded are somehow mangled, which leads me to thinking, that I am doing it terribly wrongly.
Also I found this bit offtopic SO question, which told me, that I can't have std::string &, because Python strings are immutable.
So conclusion: I have no idea what to do now :( Can anyone here help me? Thanks.

Take as reference this working example.
Define your C++ classes. For instance:
class MiddleClass {
public:
explicit MiddleClass(const std::string& data) : parent_data_(data) {}
void print() {
std::cout << parent_data_ << std::endl;
}
private:
std::string parent_data_;
};
class Loader {
public:
explicit Loader(const std::string& data) :
data_(data){
};
MiddleClass load() {
return MiddleClass(data_);
};
private:
std::string data_;
};
Create the boost bindings
boost::python::class_<MiddleClass>("MiddleClass",
boost::python::init<const std::string&>(boost::python::arg("data"), ""))
.def("print_data", &MiddleClass::print);
boost::python::class_<Loader>("Loader",
boost::python::init<const std::string&>(boost::python::arg("data"), ""))
.def("load", &Loader::load);
Install your library in the right python site-package.
Enjoy it in python:
from my_cool_package import MiddleClass, Loader
example_string = "whatever"
loader = Loader(data=example_string)
# Get the middle class
middle_class = loader.load()
# Print the data in the middle class
middle_class.print_data()
The expected output:
whatever

So, I have found a solution. Prove me wrong, but I think, that what am I trying to achieve is impossible.
Python has immutable strings, so passing a "reference" of string to function and expecting ability to change it from inside a function is simply not valid.
Take this code as an example:
variable = "Hello"
def changer(var):
var = "Bye"
changer(variable)
print(variable)
Prints "Hello". In Python, you can't make it work differently. (although to be exact, it is still being passed as a reference, but when you modify Python string, you just create a new one and a new reference).
So, how to get arround this?
Simple! Create a C++ wrapper, that will handle passing reference on std::string and return copy of resulting string. Not very effective, but you probably can't make it better.
Sample code of SaveWrapper class:
class SaveWrapper {
public:
// some constructor
std::string save(MiddleClass & value) {
std::string result;
Save saver(result);
saver.save(value);
return result;
}
};
Which can be easily "ported" to Python!

Returning and passing around raw POD pointers (arrays) with Python, C++, and pybind11

I have a C++ function which returns a raw float pointer, and another C++ function which accepts a raw float pointer as an argument. Something like:
float* ptr = something;
float* get_ptr(void) { return ptr; }
void use_ptr(float* ptr) { do_work(ptr); }
I want to be able to pass around pointers using Python. Something like this:
import my_native_functions as native
ptr = native.get_ptr()
native.use_ptr(ptr)
I am using pybind11 to create my native python module but I don't know how to create the bindings for the get_ptr() function. If I just do the following:
PYBIND11_MODULE(my_native_functions, m)
{
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}
the get_ptr() function returns a Python Float object. I guess this makes sense because there are no pointer types in python. However, because this is now a simple Float, when I call the use_ptr() function and iterate over the pointer in C/C++, only the first element of the array is correct. The rest are garbage. To fix this, in C++, I have to cast my pointer to/from std::size_t. By doing this, everything works just fine.
However, I would like to ask: Is there a "right way" of achieving the above without the casting to/from std::size_t with pybind11?
In case you are curious why I do that:
I do understand that what I am doing is not type-safe. Also, I never touch the pointer/integer on the Python side. I just retrieve it from one native module and pass it to another. Also, I cannot cast the pointer to some kind of numpy view because the pointer is not always on the CPU. Sometimes I want to pass around CUDA pointers. Creating a py::array_t from a CUDA pointer is not possible unless I copy the data (and I do not want to do that).
Thank you.

Wrap the raw pointer in a custom "smart" pointer class (only pretending to be smart really) as described here. You can add some additional information to this class while you're at it, such as the size of the array element and the number of elements. This would make it a generalised array descriptor on the C++ side (but not on the Python side because you are not exposing the raw pointer to Python).
For a simpler option, just wrap your pointer in any old class in order to hide it from Python. No need to expose it to Python as a custom smart pointer. Here's an example that does just that:
#include <pybind11/pybind11.h>
#include <memory>
#include <iostream>
namespace py = pybind11;
template <class T> class ptr_wrapper
{
public:
ptr_wrapper() : ptr(nullptr) {}
ptr_wrapper(T* ptr) : ptr(ptr) {}
ptr_wrapper(const ptr_wrapper& other) : ptr(other.ptr) {}
T& operator* () const { return *ptr; }
T* operator->() const { return ptr; }
T* get() const { return ptr; }
void destroy() { delete ptr; }
T& operator[](std::size_t idx) const { return ptr[idx]; }
private:
T* ptr;
};
float array[3] = { 3.14, 2.18, -1 };
ptr_wrapper<float> get_ptr(void) { return array; }
void use_ptr(ptr_wrapper<float> ptr) {
for (int i = 0; i < 3; ++i)
std::cout << ptr[i] << " ";
std::cout << "\n";
}
PYBIND11_MODULE(Ptr,m)
{
py::class_<ptr_wrapper<float>>(m,"pfloat");
m.def("get_ptr", &get_ptr);
m.def("use_ptr", &use_ptr);
}

Passing a C++ pointer between C and Python

I have a python extension module written in C++, which contains multiple functions. One of these generates an instance of a custom structure, which I then want to use with other functions of my module in Python as follows
import MyModule
var = MyModule.genFunc()
MyModule.readFunc(var)
To do this, I've tried using PyCapsule objects to pass a pointer to these objects between Python and C, but this produces errors when attempting to read them in the second C function ("PyCapsule_GetPointer called with invalid PyCapsule object"). Python, however, if asked to print the PyCapsule object (var) correctly identifies it as a "'capsule object "testcapsule"'. My C code appears as follows:
struct MyStruct {
int value;
};
static PyObject* genFunc(PyObject* self, PyObject *args) {
MyStruct var;
PyObject *capsuleTest;
var.value = 1;
capsuleTest = PyCapsule_New(&var, "testcapsule", NULL);
return capsuleTest;
}
static PyObject* readFunc(PyObject* self, PyObject *args) {
PyCapsule_GetPointer(args, "testcapsule");
return 0;
}
Thank you for your help.

Like stated in a comment to your question, you'll run into an issue when reading data from the local variable MyStruct var. For this you can use the third destructor to PyCapsule_New.
But that's not the reason for your problem just now. You're using PyCapsule_GetPointer(args, "testcapsule") on the args parameter. And since it's not a capsule, even though var is one, you might have defined the signature of the function as METH_VARARGS. Instead you need to unpack the tuple or use METH_O.

Exposing virtual member functions from C++ to Python using boost::python

I try to expose two different classes to python, but I don't get it to compile. I tried to follow the boost::python example, which works quite well. But if I try to write the wrapper classes for my classes it doesn't work. I have provided two minimal examples below:
struct Base
{
virtual ~Base() {}
virtual std::unique_ptr<std::string> f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
std::unique_ptr<std::string> f()
{
return this->get_override("f")();
}
};
and
struct Base
{
virtual ~Base() {}
virtual void f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
void f()
{
return this->get_override("f")();
}
};
The first one does not compile because of the unique pointer(I think boost::python does not use unique pointers?) and the second example complains about the return statement inside the void function. Can someone help me how to solve this problems?

The examples are failing to compile because:
The first example attempts to convert an unspecified type (the return type of override::operator()) to an incompatible type. In particular, Boost.Python does not currently support std::unique_ptr, and hence will not convert to it.
The second example attempts to return the unspecified type mentioned above when the calling function declares that it returns void.
From a Python perspective, strings are immutable, and attempting to transferring ownership of a string from Python to C++ violates semantics. However, one could create a copy of a string within C++, and pass ownership of the copied string to C++. For example:
std::unique_ptr<std::string> BaseWrap::f()
{
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("f")();
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
The object returned from this->get_override("f")() has an unspecified type, but can be used to convert to C++ types. The invocation of the override will throw if Python throws, and the conversion to the C++ type will throw if the object returned from Python is not convertible to the C++ type.
Here is a complete example demonstrating two ways to adapt the returned Python object to a C++ object. As mentioned above, the override conversion can be used. Alternatively, one can use boost::python::extract<>, allowing one to check if the conversion will fail before performing the conversion:
#include <memory> // std::unique_ptr
#include <boost/algorithm/string.hpp> // boost::to_upper_copy
#include <boost/python.hpp>
struct base
{
virtual ~base() {}
virtual std::unique_ptr<std::string> perform() = 0;
};
struct base_wrap : base, boost::python::wrapper<base>
{
std::unique_ptr<std::string> perform()
{
namespace python = boost::python;
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("perform")();
// Alternatively, an extract could be used to defer extracting the
// result.
python::object method(this->get_override("perform"));
python::extract<std::string> extractor(method());
// Check that extractor contains a std::string without throwing.
assert(extractor.check());
// extractor() would throw if it did not contain a std::string.
assert(result == extractor());
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<base_wrap, boost::noncopyable>("Base", python::init<>())
.def("perform", python::pure_virtual(&base::perform))
;
python::def("make_upper", +[](base* object) {
auto result = object->perform(); // Force dispatch through base_wrap.
assert(result);
return boost::to_upper_copy(*result);
});
}
Interactive usage:
>>> import example
>>> class Derived(example.Base):
... def perform(self):
... return "abc"
...
>>> derived = Derived()
>>> assert("ABC" == example.make_upper(derived))