I'm having difficulty with creating a PyTupleObject using the Python C api.
#include "Python.h"
int main() {
int err;
Py_ssize_t size = 2;
PyObject *the_tuple = PyTuple_New(size); // this line crashes the program
if (!the_tuple)
std::cerr << "the tuple is null" << std::endl;
err = PyTuple_SetItem(the_tuple, (Py_ssize_t) 0, PyLong_FromLong((long) 5.7));
if (err < 0) {
std::cerr << "first set item failed" << std::endl;
}
err = PyTuple_SetItem(the_tuple, (Py_ssize_t) 1, PyLong_FromLong((long) 5.7));
if (err < 0) {
std::cerr << "second set item failed" << std::endl;
}
return 0;
}
crashes with
Process finished with exit code -1073741819 (0xC0000005)
But so does everything else i've tried so far. Any ideas what I'm doing wrong? Not that I'm just trying to run the as a C++ program, as I'm just trying to do tests on the code before adding a swig typemap.
The commenter #asynts is correct in that you need to initialize the interpreter via Py_Initialize if you want to interact with Python objects (you are, in fact, embedding Python). There are a subset of functions from the API that can safely be called without initializing the interpreter, but creating Python objects do not fall within this subset.
Py_BuildValue may "work" (as in, not creating a segfault with those specific arguments), but it will cause issues elsewhere in the code if you try to do anything with it without having initialized the interpreter.
It seems that you're trying to extend Python rather than embed it, but you're embedding it to test the extension code. You may want to refer to the official documentation for extending Python with C/C++ to guide you through this process.
Related
I am just printing the value of car1.vehicle_id in python. I want it to print "1234" for the first 2 seconds and then when the value is changes in another thread to " 4543" the change should take effect in python. Is this possible or is there a simple example to help me with this?
c++
#include <pybind11/embed.h>
#include <string>
#include <thread>
#include <chrono>
// Define namespace for pybind11
namespace py = pybind11;
class Vehiclee
{
// Access specifier
public:
Vehiclee(){};
~Vehiclee() {}
// Data Members
int vehicle_id;
std::string vehicle_name;
std::string vehicle_color;
// Member Functions()
void printname()
{
std::cout << "Vehicle id is: " << vehicle_id;
std::cout << "Vehicle name is: " << vehicle_name;
std::cout << "Vehicle color is: " << vehicle_color;
}
};
PYBIND11_EMBEDDED_MODULE(embeded, m){
py::class_(m, "Vehiclee")
.def_readonly("vehicle_name", &Vehiclee::vehicle_name)
.def_readonly("vehicle_color", &Vehiclee::vehicle_color)
.def_readonly("vehicle_id", &Vehiclee::vehicle_id);
}
py::scoped_interpreter python{};
Vehiclee car1;
void threadFunc()
{
sleep(2);
std::cout<<"entering thread";
car1.vehicle_id = 4543;
std::cout<<"Modified val in thread";
}
int main() {
// Initialize the python interpreter
// Import all the functions from scripts by file name in the working directory
auto simpleFuncs = py::module::import("simpleFuncs");
// Test if C++ objects can be passed into python functions
car1.vehicle_id = 1234;
std::thread t1(threadFunc);
simpleFuncs.attr("simplePrint")(car1);
t1.join();
return 0;
}
python
> import time
> import importlib
> import embeded
>
> def simplePrint(argument):
> while(1):
> importlib.reload(embeded)
> print(argument.vehicle_id) time.sleep(1)
Current output
always 1234
Required output
1234 (for first 2 secs)
4543 (after 2 secs)
You need to understand the C++ rules for threading. In C++, threads can run far better in parallel than in Python. This is because in C++, threads are by default running entirely separate from each other, whereas Python uses a Global Interpreter Lock which causes a lot of thread synchronization.
So, in this case you do need the threads to synchronize, because the threads share a variable (car1). The challenge is that .def_readonly hides some boilerplate code which doesn't do synchronization - makes sense, because what object should it use to synchronize?
So what you need to do is make getter and setter methods in Vehicle, and add a std::mutex. In every getter and every setter, you lock and unlock this mutex. This is easy with a std::scoped_lock - this will automatically unlock the mutex when the method returns.
There are other options. For vehicle_id you could use a std::atomic_int, but you'd probably still need a getter method. I don't think pybind understands atomic variables.
Problem
I am trying to call my C++ code from Python 3.7 with Blender 2.82a (also happens in 2.83). The code should optimize a camera path. It can be used without Blender, however, I use Blender to setup a scene with camera path and query depth values in a scene.
I tried calling the optimization function in C++ and in the Python console. Both worked without any problems. The problem is, when I call it in Blender, Blender crashes.
This is the crash report:
# Blender 2.83.0, Commit date: 2020-06-03 14:38, Hash 211b6c29f771
# backtrace
./blender(BLI_system_backtrace+0x1d) [0x6989e9d]
./blender() [0xc1548f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa5fb3dc3c0]
/lib/x86_64-linux-gnu/libpthread.so.0(raise+0xcb) [0x7fa5fb3dc24b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa5fb3dc3c0]
./blender(_ZN5Eigen8IOFormatD1Ev+0xa3) [0x179bc43]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_Z2_zRK6CameraRKN5Eigen6MatrixIdLi9ELi1ELi0ELi9ELi1EEEiiRKSt8functionIFdRK3RayEE+0x2e2) [0x7fa5d1538e72]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_ZN11OpticalFlow13GradPathErrorERKSt6vectorI6CameraSaIS1_EEiiRKSt8functionIFdRK3RayEEd+0x5a7) [0x7fa5d1539c77]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_Z16_gradientDescentRKSt6vectorI6CameraSaIS0_EEiiRKSt8functionIFdRK3RayEEd+0x54b) [0x7fa5d153b5fb]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(_ZN11OpticalFlow12OptimizePathERKSt6vectorI6CameraSaIS1_EEiiRKSt8functionIFdRK3RayEEdNS_18OptimizationMethodE+0x22) [0x7fa5d153bcb2]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(+0x3d910) [0x7fa5d1533910]
/home/name/Programs/blender-2.83.0-linux64/2.83/python/lib/python3.7/optFlowCam.cpython-37m-x86_64-linux-gnu.so(+0x317ed) [0x7fa5d15277ed]
./blender(_PyMethodDef_RawFastCallKeywords+0x2f3) [0x570f373]
./blender(_PyCFunction_FastCallKeywords+0x25) [0x570f3f5]
./blender(_PyEval_EvalFrameDefault+0x7468) [0xc0fb48]
./blender(_PyEval_EvalCodeWithName+0xadc) [0x57c0d8c]
./blender(PyEval_EvalCodeEx+0x3e) [0x57c0ebe]
./blender(PyEval_EvalCode+0x1b) [0x57c0eeb]
./blender() [0x11f35ac]
./blender() [0x1600cde]
./blender() [0xec6a93]
./blender() [0xec6d07]
./blender(WM_operator_name_call_ptr+0x1a) [0xec720a]
./blender() [0x14f2082]
./blender() [0x15020d5]
./blender() [0xeca877]
./blender() [0xecaecc]
./blender(wm_event_do_handlers+0x310) [0xecb5e0]
./blender(WM_main+0x20) [0xec2230]
./blender(main+0x321) [0xb4bfd1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fa5facb50b3]
./blender() [0xc11c0c]
I am using Eigen for linear algebra calculations and pybind11 to compile it into a python module. The Eigen types are all fixed size as I don't need them to be dynamic (possible reason for the issue). I compile with gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0 on Ubuntu 20.04. I currently use c++11 standard but it's not a requirement.
Current findings
With faulthandler.enabled() it gives me
Fatal Python error: Segmentation fault
Current thread 0x00007fa5fab18040 (most recent call first):
File "/Text", line 16 in <module>
I already found that it crashes on the same line in the program, which is when the result of a matrix-vector multiplication is supposed to be returned and inserted into a std::vector. I printed the vector and the matrix prior, to make sure they do not contain garbage and that worked fine.
I also tried to store it in an intermediate variable and print it and then it crashes on printing. The multiplication itself seems not to cause the segfault.
I figured, I try to call the function, where it occurs, directly from Blender, but then it works and returns a result without segfault.
I suspect it to be some kind of memory alignment issue and tried everything suggested here and on the Eigen documentary. Namely, I use Eigen::aligned_allocator in every std::vector, only pass the Eigen objects as const & and have EIGEN_MAKE_ALIGNED_OPERATOR_NEW in the camera and ray class which have Eigen type members.
Using #define EIGEN_DONT_VECTORIZE and #define EIGEN_DISABLE_UNALIGNED_ARRAY_ASSERT only gave me partial success. It doesn't crash at the same line as before anymore. Weirdly enough, if I also add a cout before the return, the function finishes and returns.
Parts where crashes occur:
The project is not public and the C++ code is quite lengthy, so I only include parts of it. Let me know if you need more. The rest looks very similar, so if there is something conceptually wrong, it is probably wrong in here too. It is not a minimal example (and it contains some debug prints), as I don't know why it happens and the error is not always on the same part.
// in header
// this helped somehow
#define EIGEN_DONT_VECTORIZE
#define EIGEN_DISABLE_UNALIGNED_ARRAY_ASSERT
// ***********************************
#include <iostream>
#include <numeric>
#include <array>
#include <vector>
#include <Eigen/Dense>
#include <Eigen/StdVector>
#include "matrix_types.h"
#include "camera.h"
// *******************************************************************************
// Vector9d is a typedef
// in cpp
Vector9d _z(const Camera& cam, const Vector9d& derivX, int x_dir, int y_dir, const std::function<double(const Ray&)>& depthTest){
Eigen::IOFormat HeavyFmt(Eigen::FullPrecision, 0, ", ", ",\n", "[", "]", "[", "]");
Matrix9d M0 = OpticalFlow::M(cam, x_dir, y_dir, depthTest);
std::cout << "M_\n" << M0.format(HeavyFmt) << "\n" <<std::endl;
Vector9d z = M0 * derivX;
return z;
}
std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> OpticalFlow::GradPathError(const std::vector<Camera>& pathPositions, int x_dir, int y_dir, const std::function<double(const Ray&)>& depthTest, double h){
int n = pathPositions.size()-1;
Eigen::IOFormat HeavyFmt(Eigen::FullPrecision);
Eigen::IOFormat HeavyMtxFmt(Eigen::FullPrecision, 0, ", ", ",\n", "[", "]", "[", "]");
std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> gradPE;
gradPE.reserve(n+1);
// save values that will be used more often in calculations
std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> derivXs;
std::vector<Vector9d, Eigen::aligned_allocator<Vector9d>> zs;
std::vector<std::array<Matrix9d, 9>> gradMs;
derivXs.reserve(n+1);
zs.reserve(n+1);
gradMs.reserve(n+1);
for(int i = 0; i<n+1; ++i){
derivXs.push_back(_derivCamPath(pathPositions, i));
Camera cam = pathPositions[i];
Vector9d derivX = _derivCamPath(pathPositions, i);
zs.push_back(_z(cam, derivX, x_dir, y_dir, depthTest)); // <--- crashed here, if vectorization not turned off
gradMs.push_back(GradM(cam, x_dir, y_dir, depthTest, h));
}
for(int i = 0; i<n+1; ++i){
Vector9d derivZ = _derivZ(zs, i);
std::cout << "Zt_" << i << "\n" << derivZ.format(HeavyFmt) << "\n" << std::endl;
gradPE.push_back(1.0/(n+1) * _w(derivZ, derivXs[i], gradMs[i]));
}
// if this is included and vectorization turned off, it doesn't crash
// std::cout << "end" << std::endl;
return gradPE; // <-- crash here if vectorization is off
}
I hope someone can help me find the cause or what else I can try to track it down further. I am not very experienced with C++, so the code might have obvious issues.
I think I found the cause.
This line ./blender(_ZN5Eigen8IOFormatD1Ev+0xd3) [0x2041673] actually named the culprit in a not so readable format. I used gdb to debug the python call from Blender and in the backtrace, the same line is #0 0x0000000002041673 in Eigen::IOFormat::~IOFormat() ()
The problem was I used Eigen::IOFormat in my program to debug print the matrices and vectors for easier copying into another program where I just wanted to check if the values are correct. You can see it in the excerpt code I posted in the question. I only used it in those two functions and the segfault only occurred in those two functions. There might be something else wrong but for now, it seems to work.
I currently have a piece of hardware connected to C++ code using the MFC (Windows programming) framework. Basically the hardware is passing in image frames to my C++ code. In my C++ code, I am then calling a Python script using the CPython (Python embedding in C++) API to execute a model on that image. I've been noticing some weird behavior with the images though.
My C++ code is executing my Python script perfectly until some frame in the range of 80-90. After that point, my C++ code, for some reason, just stops executing the Python script. Despite that, the C++ code is still running normally - EXCEPT for the fact (which I just stated) that it's not executing the Python script.
Something to note: my Python script takes 5 seconds to execute the FIRST time, but then only 0.02 seconds to execute each frame after that first frame (I think due to the model getting set up).
At first, I thought it was a problem with the speed, so I replaced all my Python code with just a "time.sleep()" call with varying time, and, even if I sleep 5 seconds each C++ call to Python still always gets executed. As a result, I don't think it's a matter of the total time. For instance, if I do "time.sleep(1)" which sleeps for a second (which is longer than my Python script execution time AFTER the first frame), my Python script still always gets executed.
Does anyone have any idea why this might be happening? Could it be because of the uneven running times? Since it's taking 5 seconds to run the first frame and then significantly faster for each frame after that. Could it be that the Python is somehow unable to catch up after that time period?
This is my first time executing C++/Python on hardware, so I'm also new to this. Any helps would be greatly appreciated!
To give some idea of my code, here is a snippet:
if (pFuncFrame && PyCallable_Check(pFuncFrame)) {
PyObject* pArgs = PyTuple_New(1);
PyTuple_SetItem(pArgs, 0, PyUnicode_FromString("img.bmp"));
PyObject_CallObject(pFuncFrame, pArgs);
std::cout << "Called the frame function";
}
else {
std::cout << "Did not get the frame function";
}
I'm willing to bet that the first execution ends in a Python exception which isn't cleared until you execute some new Python statement in the second iteration, which therefore fails immediately. I recommend fixing the memory leaks and adding some error handling code to get some diagnostics (which will be useful either which way). For example (haven't tried, since you didn't provide a compilable example, but the following shouldn't be too far off):
if (pFuncFrame && PyCallable_Check(pFuncFrame)) {
PyObject* pArgs = PyTuple_New(1);
PyTuple_SetItem(pArgs, 0, PyUnicode_FromString("img.bmp"));
PyObject* res = PyObject_CallObject(pFuncFrame, pArgs);
if (!res) {
if (PyErr_Occurred()) PyErr_Print();
else std::cerr << "Python exception without error set\n";
} else {
Py_DECREF(res);
std::cout << "Called the frame function";
}
Py_DECREF(pArgs);
}
else {
std::cout << "Did not get the frame function";
}
I had one c++ program that inside for loop, calling a function.
The function is doing a heavy process, it is embedded with python and performing image processing.
My question is, why can it only run at the first instance of the variable?
Main function (I only show the part of code require in this title):
int main(){
for(int a = 0;a<5;a++){
for(int b=0;b<5;b++){
// I want every increment it go to PyRead() function, doing image processing, and compare
if(PyRead()==1){
// some application might be occur
}
else {
}
}
}
PyRead() function, the function in c++ to go into python environment performing image processing:
bool PyRead(){
string data2;
Py_Initialize();
PyRun_SimpleString("print 'hahahahahawwwwwwwwwwwww' ");
char filename[] = "testcapture";
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\".\")");
PyObject * moduleObj = PyImport_ImportModule(filename);
if (moduleObj)
{
PyRun_SimpleString("print 'hahahahaha' ");
char functionName[] = "test";
PyObject * functionObj = PyObject_GetAttrString(moduleObj, functionName);
if (functionObj)
{
if (PyCallable_Check(functionObj))
{
PyObject * argsObject = PyTuple_New(0);
if (argsObject)
{
PyObject * resultObject = PyEval_CallObject(functionObj, argsObject);
if (resultObject)
{
if ((resultObject != Py_None)&&(PyString_Check(resultObject)))
{
data2 = PyString_AsString(resultObject);
}
Py_DECREF(resultObject);
}
else if (PyErr_Occurred()) PyErr_Print();
Py_DECREF(argsObject);
}
}
Py_DECREF(functionObj);
}
else PyErr_Clear();
Py_DECREF(moduleObj);
}
Py_Finalize();
std::cout << "The Python test function returned: " << data2<< std::endl;
cout << "Data2 \n" << data2;
if(compareID(data2) == 1)
return true;
else
return false;
}
This is second time I ask this question in stack overflow. I hope this time this question will be more clear!
I can successful compile with no error.
When I run the program, I realize at a=0, b=0 it will go to PyRead() function and return value, after that it go to a=0, b=1, at that moment the whole program will end.
It supposes to go to PyRead() function again, but it does not do that and straight ending the program.
I must strongly mention that PyRead() function needed a long time to run (30seconds).
I had no idea what happens, seeking for somehelp. Please focus on the Bold part to understand my question.
Thanks.
See the comment in https://docs.python.org/2/c-api/init.html#c.Py_Finalize
Ideally, this frees all memory allocated by the Python interpreter.
Dynamically loaded extension modules loaded by Python are not unloaded.
Some extensions may not work properly if their initialization routine is called more than once
It seems your module, does not play well with this function.
A workaround can be - create the script on the fly and call it with python subprocess.
I have a C++ and a Python application on Windows (7+). I wish to send a ~50KB array of binary data (int[], float[], or double[]) from the C++ application to a NumPy array in the Python application in real time. I want <100ms latency, but can handle up to 500ms. I'm unsure of the correct way to do this.
I believe NumPy technically stores its arrays as just an array of binary data just like C++ (assuming a reasonable C++ compiler, like modern MSVC or GCC). Therefore technically it should be very easy, but I haven't been able to identify a good way to do this.
My current plan would be to use a memory mapped file, and then handle locking the memory-mapped file with more traditional IPC such as the Win32 message pump or a semaphore.
I'm however not sure whether NumPy can read straight from a memory mapped file. It can create a memory-map to a file on disk with numpy.memmap, but this doesn't seem to work for pure memory mapped file where I just have a name or a handle.
I don't know if this is the right approach. Maybe I can get it to work, but ideally I would also want to do it the right way and not be surprised by nasty consequences of me coding stuff I don't understand.
I would appreciate any help or pointers to material that might help me figure out the correct way to do this.
UPDATE:
My C++ code (proof-of-concept) would look like this:
// Host application.
// Creates 20 byte memory mapped file with name "Global\test_mmap_file" and
// containing 5 uint32s.
// Note: Requires SeCreateGlobalPrivilege to create global memory mapped
#include <windows.h>
#include <iostream>
#include <cassert>
file
int main()
{
HANDLE file_mapping_handle = NULL;
unsigned int* buffer = 0;
assert(sizeof(unsigned int) == 4); // Require compatability with np.uint32
const size_t buffer_sz = 5 * sizeof(unsigned int);
file_mapping_handle = CreateFileMapping(
INVALID_HANDLE_VALUE,
NULL,
PAGE_READWRITE,
0,
buffer_sz,
L"Global\\test_mmap_file");
if (!file_mapping_handle)
{
std::cout << "CreateFileMapping failed (Host).\n";
std::cout << "Error code: 0x" << std::hex << GetLastError() << std::endl;
std::cin.get();
return 1;
}
buffer = (unsigned int*)MapViewOfFile(
file_mapping_handle,
FILE_MAP_ALL_ACCESS,
0,
0,
buffer_sz);
if (!buffer)
{
CloseHandle(file_mapping_handle);
std::cout << " MapViewOfFile failed (Host).\n";
std::cout << "Error code: 0x" << std::hex << GetLastError() << std::endl;
std::cin.get();
return 1;
}
buffer[0] = 2;
buffer[1] = 3;
buffer[2] = 5;
buffer[3] = 7;
buffer[4] = 11;
std::cout << "Data sent, press enter to exit.\n";
std::cin.get();
UnmapViewOfFile(buffer);
CloseHandle(file_mapping_handle);
return 0;
}
I wanted some way to access this shared memory from Python and create a numpy array. I tried,
import numpy as np
L_mm = np.memmap('Global\\test_mmap_file')
but this fails as Global\test_mmap_file is not a filename. Following the hints given by abarnert I constructed the following client program which seems to work:
import numpy as np
import mmap
mm = mmap.mmap(0,20,'Global\\test_mmap_file')
L = np.frombuffer(mm,dtype = np.uint32)
print (L)
mm.close()
This requires admin privileges for both programs to run (or giving the user the right to SeCreateGlobalObjects). However I think this should easily be bypassed by not giving the shared memory a global name, and instead duplicating the handle and passing it to the Python program. It also doesn't control access to the shared memory properly, but that should be easy with a semaphore or some other such construct.