I have the following scenario: I have some code that is in C++ and I am writing a boost wrapper around it. There is a requirement that I need to serialize the objects to be sent over a socket (in Python)
I've tried doing it directly via
Pickle (and Dill)
Boost python pickle
but they both lead to errors when I'm trying to deserialize it on the other end. I was then given the following to work with
struct message_header {
T id{};
uint32_t size = 0;
};
template<typename T>
struct message {
// Header & Body vector
message_header<T> header{};
std::vector<uint8_t> body;
// Misc stuff
friend message<T> &operator<<(message<T> &msg, const std::string &s) {
const unsigned char *ucdata = reinterpret_cast<const unsigned char *>(s.c_str());
// Cache the location towards the end of the vector where the pulled data starts
size_t i = msg.body.size();
//determine the size of the incoming data
size_t strSize = s.size();
// Resize the vector by the size of the data being pushed
msg.body.resize(msg.body.size() + strSize);
// Physically copy the data into the newly allocated vector space
std::memcpy(msg.body.data() + i, ucdata, strSize);
// Recalculate the message size
msg.header.size = msg.size();
// Return the target message so it can be "chained"
return msg;
}
where the rough idea is to
serialize the data as bytes within C++
*Pass the serialized object to Python
Send the serialized object to the other machine
*Deserialize the object (within C++)
I've got the code to serialize the object in C++ (which uses the code above) but I was wondering
how I can get that C++ object into Python (most of my boost experience is with native types),
how I can pass the Python-ified object back to C++ to be deserialized
Related
I'm wondering if I could get some help. For context, I'm using some C++ libraries to generate some large (think hundreds of Mb) objects that I want to send over a network from a server to a client.
On the server, I've got the following:
PyObject* CKKSwrapper::SerializePrivateKey() {
std::string s;
std::ostringstream os(s);
Serial::Serialize(m_keys.publicKey, os, SerType::BINARY);
auto msg = os.str();
return PyBytes_FromString(&msg[0]);
}
which gives me some Python object. I then send this directly to the client via python sockets. I'm reading it in like
def _safe_recv_abstract(socket: Socket, deserializer_func):
expected_length = _get_obj_size(socket)
running_length = 0
running_msg = bytearray()
while running_length < expected_length:
msg = socket.recv(expected_length)
if msg:
running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
running_length += len(msg)
socket.send(_add_header_to_payload(b"ACK"))
logger.debug("_safe_recv_unenc_obj: Received all data")
if optional_pycrypto_deserialize_func:
return deserializer_func(running_msg)
return running_msg
two things:
Accumulator (from cppwrapper.Accumulator() above) looks like
PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b){
return PyByteArray_Concat(&a, &b);
}
deserializer_func calls an underlying C++ function that looks like
void CKKSwrapper::DeserializeX(
const boost::python::list &pyvals) {
auto msg= pythonListToCppVectorBytes(pyvals);
LPPrivateKey<DCRTPoly> sk;
std::istringstream is(string(msg.begin(), msg.end()));
Serial::Deserialize(sk, is, SerType::BINARY);
this->m_keys.secretKey = sk;
}
I'm running into the following error:
Boost.Python.ArgumentError: Python argument types in
CKKSwrapper.Accumulator(bytearray, bytearray)
did not match C++ signature:
Accumulator(pycrypto::CKKSwrapper {lvalue}, _object*, _object*)
I completely understand what it is saying and that the types are wrong but I'm not sure WHY. From the docs
PyObject* PyByteArray_Concat(PyObject *a, PyObject *b)
Return value: New reference.
Concat **bytearrays** a and b and return a new bytearray with the result.
If I understand correctly, I AM passing in bytearrays but it says that it is expecting objects?
The reason I'm trying to do it this way is that when I use a bytearray or a list for the accumulation, i.e
while running_length < expected_length:
msg = socket.recv(expected_length)
if msg:
running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
running_length += len(msg)
the memory usage and runtime blow up
Partial answer for the simple mistake:
PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b)
This should be
PyObject* CKKSwrapper::Accumulator(PyObject* a, PyObject* b)
PyObject is never passed by value, always by pointer. In practice all useful Python objects look like:
struct Something{
PyObject ob_base;
Useful data;
};
This is essentially a C version of C++ inheritance. By passing by value you're losing everything but ob_base, exactly like passing a C++ derived class as its base by value.
Currently when working with python + pybind11 I find it frustrating to work with the typed c++ classes/structs.
I would like to change my bindings so that they generate a simple python class, with an __init__ and a simple function like shown below. Is something like the feasible?
Reasoning:
I currently have a struct that I generate via c++, but it has a lot of heavy std::vector<float>s that I would like to pass to python, and keep as numpy arrays inside a similar interfacing python class. (bonus points if you can tell me how to move vectors to be numpy arrays quickly!)
I have already completely bound my c++ struct with pybind11, so I feel like I know what I'm doing... however I can't seem to figure out if this is possible!
So, as a learning exercise, can I make the following python class via pybind11?
>>> python
class MyStruct:
def __init__(self, A_in, descriptor_in):
self.A = A_in
self.descriptor = descriptor_in
def add_to_vec(f_in):
self.A.append(f_in)
<<< python
Edit:
I want to say I 'think' that this is doable with the python C api, but I'd like to avoid using that directly if I can. (but if you think that's the only way, please let me know :) )
Edit2: (response to #Erwan)
The only way I'm aware of to get class variables individually is this (shown below). You cannot use the pybind advertised buffer_protocol interface if you have more than one numpy array in the struct you would like to get. However this requires creating a python-interface only function .def (not ideal) that points to (what I think is a copy) of the original data (so it's probably slow, i haven't benchmarked it, but I'm not sure if this was is a hack or the correct way to get vectors into numpy arrays).
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <vector>
#include <string>
struct Pet {
Pet(const std::string &name) : name(name) {
bdata.push_back(22.);
bdata.push_back(23.1);
bdata.push_back(24.);
bdata.push_back(2222.);
}
void setName(const std::string &name_) { name = name_; }
const std::string &getName() const { return name; }
std::string name;
std::vector<float> bdata;
};
namespace py = pybind11;
PYBIND11_MODULE(example, m) {
py::class_<Pet>(m, "Pet")
.def(py::init<const std::string &>())
.def("setName", &Pet::setName)
.def("getName", &Pet::getName)
.def("bdata", [](Pet &m) -> py::array {
py::buffer_info buff_info(py::buffer_info(
m.bdata.data(), /* Pointer to buffer */
sizeof(float), /* Size of one scalar */
py::format_descriptor<float>::format(), /* Python struct-style format descriptor */
m.bdata.size() /* Number of dimensions */
));
return py::array(buff_info);
});
}
I don't understand your question in whole, but I'll take this part:
bonus points if you can tell me how to move vectors to be numpy arrays quickly!
If you use the return result of bdata.data() combined with numpy.frombuffer() and bdata.size() if need be, you can get a view on the vector data, which is guaranteed to be contiguous as of C++11. (The normal numpy.array() call will not honor copy=False in this case, but frombuffer acts like a cast.) Since there is no copy, that's probably as quick as it gets.
Below is an example in cppyy (which allows for easy testing, but the use of which is otherwise immaterial to the answer of how to mix std::vector and numpy.array per se). The gravy is in the last few lines: the update to 'arr' will show up in the original vector (and v.v.) b/c frombuffer is a view, not a copy:
import cppyy
import numpy as np
# load struct definition
cppyy.cppdef("""
struct Pet {
Pet(const std::string &name) : name(name) {
bdata.push_back(22.);
bdata.push_back(23.1);
bdata.push_back(24.);
bdata.push_back(2222.);
}
void setName(const std::string &name_) { name = name_; }
const std::string &getName() const { return name; }
std::string name;
std::vector<float> bdata;
};""")
# create a pet object
p = cppyy.gbl.Pet('fido')
print(p.bdata[0]) # for reference (prints 22, per above)
# create a numpy view on the std::vector's data
# add count=p.bdata.size() if need be
arr = np.frombuffer(p.bdata.data(), dtype=np.float32)
# prove that it worked as intended
arr[0] = 14
print(p.bdata[0]) # shows update to 14
p.bdata[2] = 17.5
print(arr[2]) # shows update to 17.5
which will print:
22.0
14.0
17.5
'arr' may become invalid if the std::vector resizes. If you know the maximum size, however, and it is not too large or will be fully used for sure, you can reserve that, so the vector's internal data will not be reallocated.
Depending on how/where you store the numpy array, I also recommend tying the life time of 'p' (and hence 'p.bdata') to 'arr' eg. by keeping them both as data members in an instance of the wrapper class you're after.
If you want to do the conversion in C++ instead, use PyArray_FromBuffer from NumPy's array API.
I am writing a C++ wrapper for Python using the Python C API. In my case I have to make bigger amounts of byte oriented data accessible for the Python script. For this purpose I use the PyByteArray_FromStringAndSize method to produce a Python bytearray (https://docs.python.org/2.7/c-api/bytearray.html).
When returning this bytearray directly I have not experienced any problems. When however adding the bytearray into a Python dict, the memory from the bytearray will not be released once the dict is destroyed.
This can be solved by calling Py_DECREF on the bytearray object after adding the bytearray object to the Python dict.
Below is a complete working example of my code containing a method dummyArrPlain returning the plain bytearray and a method dummyArrInDict returning a bytearray in a dict. The second method will produce a memory leak unless Py_DECREF(pyData); is called.
My question is: Why is Py_DECREF necessary at this point. Intuitively I would have expected that Py_DECREF should be called once the dict is destroyed.
Also I assign values like in the following to a dict:
PyDict_SetItem(dict, PyString_FromString("i"), PyInt_FromLong(i));
Will this also produce a memory leak when not calling Py_DECREF on the created string and long?
This is my dummy C++ wrapper:
#include <python2.7/Python.h>
static char module_docstring[] = "This is a module causing a memory leak";
static PyObject *dummyArrPlain(PyObject *self, PyObject *args);
static PyObject *dummyArrInDict(PyObject *self, PyObject *args);
static PyMethodDef module_methods[] = {
{"dummy_arr_plain", dummyArrPlain, METH_VARARGS, "returns a plain dummy bytearray"},
{"dummy_arr_in_dict", dummyArrInDict, METH_VARARGS, "returns a dummy bytearray in a dict"},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC initlibdummy(void)
{
PyObject *m = Py_InitModule("libdummy", module_methods);
if (m == NULL)
return;
}
static PyObject *dummyArrPlain(PyObject *self, PyObject *args)
{
int len = 10000000;
char* data = new char[len];
for(int i=0; i<len; i++) {
data[i] = 0;
}
PyObject * pyData = PyByteArray_FromStringAndSize(data, len);
delete [] data;
return pyData;
}
static PyObject *dummyArrInDict(PyObject *self, PyObject *args)
{
int len = 10000000;
char* data = new char[len];
for(int i=0; i<len; i++) {
data[i] = 0;
}
PyObject * pyData = PyByteArray_FromStringAndSize(data, len);
delete [] data;
PyObject *dict = PyDict_New();
PyDict_SetItem(dict, PyString_FromString("data"), pyData);
// memory leak without Py_DECREF(pyData);
return dict;
}
And a dummy python script using the wrapper:
import libdummy
import time
while True:
a = libdummy.dummy_arr_in_dict()
time.sleep(0.01)
It's a matter of [Python 2.0.Docs]: Ownership rules. I'm going to exemplify on Python 2.7.10 (pretty old, but I don't think that the behavior has (significantly) changed along the way).
PyByteArray_FromStringAndSize (bytearrayobject.c: 168) creates a new object (using PyObject_New, and allocates memory for the buffer as well).
By default, the refcount of that object (or better: of any newly created object) is 1 (set by _Py_NewReference), so that when the user calls del upon it, or at program exit, the refcount will be decreased, and when reaching 0, the object will be deallocated.
This is the behavior on the flow where the object is returned
But, in dummyArrInDict's case, PyDict_SetItem does (indirectly) a Py_INCREF of pyData (it does other stuff, but only this is relevant in the current situation), ending up with a refcount of 2 and therefore the memory leak
It's basically same thing that you're doing with data: you allocate memory for it, and when you no longer need it, you free it (this is because you're not returning it, you only use it temporarily).
Note: It's safer to use the X macros (e.g. [Python 2.Docs]: Py_XDECREF, especially since you're not testing for NULL the returned PyObjects).
For more details, also take a look at [Python 2.Docs]: C API Reference.
I have some C code that runs on an embedded system, generating a stream of data which my python code will read on the other side of a bluetooth/usb line. The protocol of the stream is still under heavy development, and changes frequently, but is defined in a single .h file. I would like to use SWIG to keep the python side of things up to date, specifically by providing access to the stream data layouts (structs)
Here is a sample .h file describing a number of structures and a few constants (as #defines), obviously a very small subset of the whole protocol, for brevity.
//datalayouts.h
#ifdef SWIG
#define __attribute__(x)
#endif
#define TOKEN_TYPE_SYNC_VALUE 1
#define TOKEN_TYPE_DELTA 2
typedef struct __attribute__((packed))
{
uint8_t token_type;
uint32_t timestamp;
uint32_t value;
} struct_token_type_sync_value;
typedef struct __attribute__((packed))
{
uint8_t token_type;
int16_t delta;
} struct_token_type_delta;
Coupled with this is the basic interface file
%module datalayouts
%{
#include "datalayouts.h"
%}
%include "datalayouts.h"
It all compiles and imports just fine. In python I can create a variable of type token_type_sync_value, but what I want to do is cast a portion of the data I'm reading from the stream (as a string), to impose the correct structure on it.
For example:
>>> from datalayouts token_type_sync_value
>>> data = stream.read() #returns 100+ bytes
>>> if ord(data[0]) == TOKEN_TYPE_SYNC_VALUE:
... #here I want to access data[0:9] as a token_type_sync_value
Is this possible, if so how?
You can do this with SWIG, the simplest solution is to use %extend to supply an extra constructor from within Python that takes a PyObect to use as a buffer:
%module test
%include <stdint.i>
%inline %{
#ifdef SWIG
#define __attribute__(x)
#endif
#define TOKEN_TYPE_SYNC_VALUE 1
#define TOKEN_TYPE_DELTA 2
typedef struct __attribute__((packed))
{
uint8_t token_type;
int16_t delta;
} struct_token_type_delta;
%}
%extend struct_token_type_delta {
struct_token_type_delta(PyObject *in) {
assert(PyObject_CheckBuffer(in));
Py_buffer view;
const int ret = PyObject_GetBuffer(in, &view, PyBUF_SIMPLE);
assert(0==ret);
assert(view.len >= sizeof(struct_token_type_delta));
struct_token_type_delta *result = new struct_token_type_delta(*static_cast<const struct_token_type_delta*>(view.buf));
PyBuffer_Release(&view); // Note you could/should retain view.obj for the life of this object to prevent use after free
return result;
}
}
You'd need to do this for each type you wanted to construct from a buffer, but the actual code for the constructor of each remains the same so could be wrapped as a macro (using %define) quite simply. You would also want to do something to prevent the use after free error, by retaining the reference to the underlying buffer for longer.
Personally if it were me doing this though I'd look for a different solution, because there are nicer ways of getting the same result and writing code that creates and maintains thin POD/bean like objects is tedious and dull in any language let alone 2 or more. Assuming protbuf is too heavyweight to use in your embedded system I'd look to solve this in reverse, using ctypes for Python and then having your Python code also generate the header for your C build tools as well. So something like:
import ctypes
class ProtocolStructure(type(ctypes.Structure)):
def __str__(self):
s='''
typedef struct __attribute__((packed)) {
\t%s
}'''
return s % '\n\t'.join(('%s %s;' % (ty.__name__[2:], name) for name,ty in self._fields_))
class struct_token_type_delta(ctypes.Structure, metaclass=ProtocolStructure):
_fields_ = (('token_type', ctypes.c_uint8),
('delta', ctypes.c_int16))
if __name__ == '__main__':
# when this file is run instead of imported print the header file to stdout
h='''
#ifndef PROTO_H
#define PROTO_H
%s
#endif
'''
print(h % ';\n'.join('%s %s;\n' % (ty, name) for name,ty in globals().items() if issubclass(type(ty), ProtocolStructure)))
Which then lets you write:
import proto
proto.struct_token_type_delta.from_buffer(bytearray(b'\xff\x11\x22'))
I am writing a Python app that makes use of PulseAudio API. The implementation is heavily using callbacks written in Python and invoked by PulseAudio's C code.
The most information is passed into the callback by a specific structure, for instance pa_sink_info, which is defined in C as follows:
typedef struct pa_sink_info {
const char *name;
uint32_t index;
const char *description;
pa_sample_spec sample_spec;
pa_channel_map channel_map;
uint32_t owner_module;
pa_cvolume volume;
int mute;
uint32_t monitor_source;
const char *monitor_source_name;
pa_usec_t latency;
const char *driver;
pa_sink_flags_t flags;
pa_proplist *proplist;
pa_usec_t configured_latency;
pa_volume_t base_volume;
pa_sink_state_t state;
uint32_t n_volume_steps;
uint32_t card;
uint32_t n_ports;
pa_sink_port_info** ports;
pa_sink_port_info* active_port;
uint8_t n_formats;
pa_format_info **formats;
} pa_sink_info;
From this structure it's very easy to get scalar values, eg.:
self.some_proc(
struct.contents.index,
struct.contents.name,
struct.contents.description)
But I have a difficulty dealing with ports and active_port, which in Python are described as:
('n_ports', uint32_t),
('ports', POINTER(POINTER(pa_sink_port_info))),
('active_port', POINTER(pa_sink_port_info)),
Here n_ports specifies number of elements in ports, which is a pointer to array of pointers to structures of type pa_sink_port_info. Actually, I don't even know how I can convert these to Python types at all.
What is the most efficient way of converting ports into Python dictionary containing pa_sink_port_info's?
Solving this problem required careful reading of Python's ctypes reference. Once the mechanism of ctypes type translation implementation was clear, it's not so difficult to get to the desired values.
The main idea about pointers is that you use their contents attribute to get to the data the pointer points to. Another useful thing to know is that pointers can be indexed like arrays (it's not validated by the interpreter, so it's your own responsibility to make sure it is indeed an array).
For this particular PulseAudio example, we can process the ports structure member (which is a pointer to array of pointers) as follows:
port_list = []
if struct.contents.ports:
i = 0
while True:
port_ptr = struct.contents.ports[i]
# NULL pointer terminates the array
if port_ptr:
port_struct = port_ptr.contents
port_list.append(port_struct.name)
i += 1
else:
break