I'm wondering if I could get some help. For context, I'm using some C++ libraries to generate some large (think hundreds of Mb) objects that I want to send over a network from a server to a client.
On the server, I've got the following:
PyObject* CKKSwrapper::SerializePrivateKey() {
std::string s;
std::ostringstream os(s);
Serial::Serialize(m_keys.publicKey, os, SerType::BINARY);
auto msg = os.str();
return PyBytes_FromString(&msg[0]);
}
which gives me some Python object. I then send this directly to the client via python sockets. I'm reading it in like
def _safe_recv_abstract(socket: Socket, deserializer_func):
expected_length = _get_obj_size(socket)
running_length = 0
running_msg = bytearray()
while running_length < expected_length:
msg = socket.recv(expected_length)
if msg:
running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
running_length += len(msg)
socket.send(_add_header_to_payload(b"ACK"))
logger.debug("_safe_recv_unenc_obj: Received all data")
if optional_pycrypto_deserialize_func:
return deserializer_func(running_msg)
return running_msg
two things:
Accumulator (from cppwrapper.Accumulator() above) looks like
PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b){
return PyByteArray_Concat(&a, &b);
}
deserializer_func calls an underlying C++ function that looks like
void CKKSwrapper::DeserializeX(
const boost::python::list &pyvals) {
auto msg= pythonListToCppVectorBytes(pyvals);
LPPrivateKey<DCRTPoly> sk;
std::istringstream is(string(msg.begin(), msg.end()));
Serial::Deserialize(sk, is, SerType::BINARY);
this->m_keys.secretKey = sk;
}
I'm running into the following error:
Boost.Python.ArgumentError: Python argument types in
CKKSwrapper.Accumulator(bytearray, bytearray)
did not match C++ signature:
Accumulator(pycrypto::CKKSwrapper {lvalue}, _object*, _object*)
I completely understand what it is saying and that the types are wrong but I'm not sure WHY. From the docs
PyObject* PyByteArray_Concat(PyObject *a, PyObject *b)
Return value: New reference.
Concat **bytearrays** a and b and return a new bytearray with the result.
If I understand correctly, I AM passing in bytearrays but it says that it is expecting objects?
The reason I'm trying to do it this way is that when I use a bytearray or a list for the accumulation, i.e
while running_length < expected_length:
msg = socket.recv(expected_length)
if msg:
running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
running_length += len(msg)
the memory usage and runtime blow up
Partial answer for the simple mistake:
PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b)
This should be
PyObject* CKKSwrapper::Accumulator(PyObject* a, PyObject* b)
PyObject is never passed by value, always by pointer. In practice all useful Python objects look like:
struct Something{
PyObject ob_base;
Useful data;
};
This is essentially a C version of C++ inheritance. By passing by value you're losing everything but ob_base, exactly like passing a C++ derived class as its base by value.
Related
I want to create a function in python, pass it's function pointer to c and execute it there.
So my python file:
import ctypes
import example
def tester_print():
print("Hello")
my_function_ptr = ctypes.CFUNCTYPE(None)(tester_print)
example.pass_func(my_function_ptr)
And here is what my function in c looks like:
typedef void (*MyFunctionType)(void);
PyObject* pass_func(PyObject *self, PyObject* args)
{
PyObject* callable_object;
if (!PyArg_ParseTuple(args, "O", &callable_object))
return NULL;
if (!PyCallable_Check(callable_object))
{
PyErr_SetString(PyExc_TypeError, "The object is not a callable function.");
return NULL;
}
PyObject* function_pointer = PyCapsule_New(callable_object, "my_function_capsule", NULL);
if (function_pointer == NULL) return NULL;
MyFunctionType my_function = (MyFunctionType) PyCapsule_GetPointer(function_pointer, "my_function_capsule");
if (my_function == NULL) return NULL;
my_function(); // Or (*my_function)() Both same result.
// PyCapsule_Free(function_pointer);
Py_RETURN_NONE;
}
Doing this causes a seg fault on my_function() call. How can I do this?
If you're just trying to pass a Python function to a C extension, pass it directly (don't use ctypes) and use PyObject_Call to call it:
example.pass_func(tester_print)
and
PyObject_CallNoArgs(callable_object);
If you need a real C function pointer for whatever reason, the usual approach is to write a C wrapper that takes the callable as an argument:
void callable_wrapper(PyObject *func) {
PyObject_CallNoArgs(func);
// plus whatever other code you need (e.g. reference counting, return value handling)
}
Most reasonable C APIs that take a callback function also provide a way to add an arbitrary argument to the callable ("user data"); for example, with pthreads:
result = pthread_create(&tid, &attr, callable_wrapper, callable_object);
Make sure to handle reference counting correctly: increment the reference on your callable object before passing it to the C API, and decrement the reference when it is no longer needed (e.g. if the callback is only called once, the callable_wrapper could DECREF before returning).
When using threads, you additionally need to ensure that you hold the GIL when calling any Python code; see https://docs.python.org/3/c-api/init.html#non-python-created-threads for more details and a code sample.
What your current code is doing is receiving a pointer to a ctypes CFUNCTYPE object as callable_object, placing that pointer in a capsule, taking it back out again, and calling it as if it was a C function pointer. This doesn't work, since it effectively attempts to call the CFUNCTYPE object as if it were a C function (the capsule stuff winds up being useless). When you're using the Python C API, there's almost never any need for ctypes in Python, because the C API can directly interact with Python objects.
Is it possible to store variables/arrays/objects in a C++ Object to store and modify such data through Python? For example, I want to store arrays/vectors of points/polygons/voxels/etc in a C++ Object in Python. And sometimes, I want to do something with them (change/add/remove).
Here is my simple code, but I've got an Exception:
C++:
class Foo{
public:
std::vector<openvdb::Vec3s> vertices_test;
void generateArray(std::vector<openvdb::Vec3s> ar){
// JUST DO SOMETHING GENERIC
for (int i = 0; i < 100; i++) {
ar.push_back(openvdb::Vec3s(0.3, 0.1, 0.2));
}
}
};
extern "C" {
__declspec(dllexport) Foo* Foo_new(){ return new Foo(); }
__declspec(dllexport) void Foo_generateArray(Foo* foo){ foo->generateArray(foo->vertices_test); }
}
Python part:
class Foo(object):
def __init__(self):
self.obj = lib.Foo_new()
def generateArray(self):
lib.Foo_generateArray(self.obj)
f = Foo()
f.generateArray()
But when I run Python I get an exception:
OSError: exception: access violation reading 0x0000000058ED7D28
As you can see I tried to store std::vectoropenvdb::Vec3s vertices_test object as a local variable in the C++ Object. But it looks like something goes wrong.
Also, is it possible to return float/int in the generateArray() function instead of void?
Thanks you.
From the ctypes documentation:
By default functions are assumed to return the C int type. Other return types can be specified by setting the restype attribute of the function object.
Your question implies that you're on 64-bit Windows, and on 64-bit Windows, pointers are 8 bytes and ints are only 4 bytes, so the upper 4 bytes of your pointer are being lost.
I have the following scenario: I have some code that is in C++ and I am writing a boost wrapper around it. There is a requirement that I need to serialize the objects to be sent over a socket (in Python)
I've tried doing it directly via
Pickle (and Dill)
Boost python pickle
but they both lead to errors when I'm trying to deserialize it on the other end. I was then given the following to work with
struct message_header {
T id{};
uint32_t size = 0;
};
template<typename T>
struct message {
// Header & Body vector
message_header<T> header{};
std::vector<uint8_t> body;
// Misc stuff
friend message<T> &operator<<(message<T> &msg, const std::string &s) {
const unsigned char *ucdata = reinterpret_cast<const unsigned char *>(s.c_str());
// Cache the location towards the end of the vector where the pulled data starts
size_t i = msg.body.size();
//determine the size of the incoming data
size_t strSize = s.size();
// Resize the vector by the size of the data being pushed
msg.body.resize(msg.body.size() + strSize);
// Physically copy the data into the newly allocated vector space
std::memcpy(msg.body.data() + i, ucdata, strSize);
// Recalculate the message size
msg.header.size = msg.size();
// Return the target message so it can be "chained"
return msg;
}
where the rough idea is to
serialize the data as bytes within C++
*Pass the serialized object to Python
Send the serialized object to the other machine
*Deserialize the object (within C++)
I've got the code to serialize the object in C++ (which uses the code above) but I was wondering
how I can get that C++ object into Python (most of my boost experience is with native types),
how I can pass the Python-ified object back to C++ to be deserialized
I have a python extension module written in C++, which contains multiple functions. One of these generates an instance of a custom structure, which I then want to use with other functions of my module in Python as follows
import MyModule
var = MyModule.genFunc()
MyModule.readFunc(var)
To do this, I've tried using PyCapsule objects to pass a pointer to these objects between Python and C, but this produces errors when attempting to read them in the second C function ("PyCapsule_GetPointer called with invalid PyCapsule object"). Python, however, if asked to print the PyCapsule object (var) correctly identifies it as a "'capsule object "testcapsule"'. My C code appears as follows:
struct MyStruct {
int value;
};
static PyObject* genFunc(PyObject* self, PyObject *args) {
MyStruct var;
PyObject *capsuleTest;
var.value = 1;
capsuleTest = PyCapsule_New(&var, "testcapsule", NULL);
return capsuleTest;
}
static PyObject* readFunc(PyObject* self, PyObject *args) {
PyCapsule_GetPointer(args, "testcapsule");
return 0;
}
Thank you for your help.
Like stated in a comment to your question, you'll run into an issue when reading data from the local variable MyStruct var. For this you can use the third destructor to PyCapsule_New.
But that's not the reason for your problem just now. You're using PyCapsule_GetPointer(args, "testcapsule") on the args parameter. And since it's not a capsule, even though var is one, you might have defined the signature of the function as METH_VARARGS. Instead you need to unpack the tuple or use METH_O.
How to parse userdefined types (or types from an existing non-standard library) using PyArg_ParseTuple?
Instead of using the plain O format, as Martijn suggested, I normally prefer using the O& format. It allows you to pass a function that will be called to convert any PyObject* to an arbitrary C (double) pointer. Here is some example usage, in which I'm converting a passed value to a pointer to my own object type:
/**
* This method should return 0 if it does not work or 1 in case it does
* PyArg_*() functions will handle the rest from there or let your program
* continue in case things are fine.
*/
int MyConverter(PyObject* o, MyOwnType** mine) {
//write the converter here.
}
Then, at the point you need to parse your object:
/**
* Simple example
*/
static PyObject* py_do_something(PyObject*, PyObject* args, PyObject* kwds) {
/* Parses input arguments in a single shot */
static char* kwlist[] = {"o", 0};
MyOwnType* obj = 0; ///< if things go OK, obj will be there after the next call
if (!PyArg_ParseTupleAndKeywords(args, kwds, "O&", kwlist, &MyConverter, &obj))
return 0; ///< we have failed, let Python check exceptions.
/* if you get to this point, your converter worked, just use */
/* your newly allocated object and free it, when done. */
}
The advantage of this approach is that you can encapsulate your MyConverter on a C-API and then re-use it in other functions for the the same job.
Custom python classes can be parsed using the O format:
O (object) [PyObject *]
Store a Python object (without any conversion) in a C object pointer. The C program thus receives the actual object that was passed. The object’s reference count is not increased. The pointer stored is not NULL.