I'm a regular C user, but pretty new to Python.
I have a library written in C for performing calculations that I'm trying to make callable from a Python script. The library needs some user defined routines, for which I am trying to allow the use of Python scripts.
I am running into a problem which demonstrates to me that I do not understand something very fundamental. Here is the code for a simple program I cannot get to run. It should print to the screen the result of 7 (2+5).
The Python script test.py is called first. It loads ctypes and the library libfoo.so, and calls the C routine c_do_work:
from ctypes import cdll
lib = cdll.LoadLibrary('./libfoo.so')
print "Python 1: Going in..."
lib.c_do_work(2,5)
The C function c_do_work is defined in the library libfoo.so, which has the single module test.c. This routine should run the Python script my_func.py, which defines the function find_sum. A Python interpreter is initialized here:
//gcc test.c -I/usr/include/python2.7/ -L/usr/lib/python2.7/ -lpython2.7 -lm -fPIC -c
//gcc -shared -Wl,-soname,libfoo.so -o libfoo.so test.o
#include <Python.h>
#include <stdio.h>
#include <stdlib.h>
void c_do_work(int a,int b)
{
Py_Initialize();
PyObject* main_module = PyImport_AddModule("__main__");
PyObject* main_dict = PyModule_GetDict(main_module);
FILE* file_1 = fopen("my_func.py", "r");
PyRun_File(file_1, "my_func.py",Py_file_input,main_dict, main_dict);
PyObject* expression = PyDict_GetItemString(main_dict,"find_sum");
printf("C: calling Python function...\n");
PyObject_CallFunction(expression,"ii",a,b);
//Clean up
fclose(file_1);
Py_Finalize();
}
Lastly, the Python script my_func.py:
def find_sum(a, b):
print a+b
When I run "python test.py", I get a segmentation fault at the second line of the C function:
PyObject* main_module = PyImport_AddModule("__main__");
Why is this happening? If I slightly rewrite the C routine so it is main, and run that program directly, I get the desired result. The problem seems to be related to having a Python interpreter calling a routine that initializes a Python interpreter.
Related
I'm currently working on a project with a separate cli and gui. The gui part is written in C++ to keep the exectuable small. I compiled my python program with pyinstaller and need to call some functions from the C++ program.
Python part:
import configparser
def getconfig() -> list:
config = configparser.ConfigParser()
config.read("config.ini")
section1 = config["general"]
section2 = section["section2"]
return [section1, section2] #should be a list of dict?
Compiling it via pyinstaller --onefile --clean myprogram.py
What i would like to do in C++ is :
//please correct me if this is the wrong type,
//i know i probably need to do a bit of parsing between python and C++ types
std::vector<std::unordered_map<std::string, std::string>> > = myprogram.getconfig()
I just don't want to do the config parsing in C++ again, or would you recommend this as it's probably easier to call the compiled python binary?
This program just needs to run on linux, windows is not necessary.
If you are only planning to run on POSIX-compatible systems (ie. GNU/Linux), you could spawn a new process to run the Python script, e.g. using the C function exec. Example:
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
int main()
{
char* args[] = { "pythonscript.py", NULL };
int t = execv("pythonscript.py", args);
if(t < 0) printf("%s\n", strerror(errno));
return 0;
}
And then in pythonscript.py:
#!/usr/bin/python3
print("hello, world!")
This trick works with bash scripts as well.
The script must be executable though, so remember to run chmod +x pythonscript.py.
EDIT:
You will likely need to use pipes or other mechanisms for inter-process communication. (this is not my expertise!)
Following is the c++ code which I want to import.
#include <boost/python.hpp>
#include <string>
/*
* This is the C++ function we write and want to expose to Python.
*/
const std::string hello() {
return std::string("hello, zoo");
}
/*
* This is a macro Boost.Python provides to signify a Python extension module.
*/
BOOST_PYTHON_MODULE(zoo) {
// An established convention for using boost.python.
using namespace boost::python;
// Expose the function hello().
def("hello", hello);
}
The following code is python script.
import zoo # In zoo.cpp we expose hello() function, and it now exists
in the zoo module.
assert 'hello' in dir(zoo) # zoo.hello is a callable.
assert callable(zoo.hello) # Call the C++ hello() function from Python.
print zoo.hello()
When I try to run the script, I am not getting "hello, zoo" at the terminal as output. Where am I making the mistake?
Following is the error message I am getting:
import: not authorized zoo' # error/constitute.c/WriteImage/1028.
./visit_zoo.py: line 3: syntax error near unexpected token('
./visit_zoo.py: line 3: `assert 'hello' in dir(zoo)'
Aren't you forgetting to indicate the script should be run by Python, as I did?
You can either include the python executable in the header of your script file:
#!/usr/bin/env python2
and make the file executable or call the script with Python:
$ python <filename>
I'm trying to translate some simple cython to c++ :
cdef public int G = 1
and then use it in my c++ code :
#include <Python.h>
#include "test.h" // The file generated by the cython command
#include <iostream>
int main(int argc, char **argv) {
std::cout << "G : " << G << std::endl;
}
Output is :
G : 0
I've looked into the test.cpp cython-generated file, and on line 897, I have
G = 1;
So why is G evaluated to 0 in the main ?
Here are the commands used to compile :
cython.exe test.pyx -3 --cplus
g++ test.cpp test_main.cpp -IC:\Python36-32\include -LC:\Python36-32\libs -lpython36
What you generate when you use cython is a python extension module. You cannot link it directly into an executable, as it needs to be dynamically imported and linked to libpython. In that process, your extension's initialization function is run, which will cause your G to be set to 1.
So you should:
Build a python extension from your cython (using -shared and outputting a DLL).
Load python interpreter in your main. In your program, you don't even initialize python at the moment.
Import it in your main using PyImport_ImportModule("mymodule").
Untested, but your main should look like this:
#include <Python.h>
int main(int argc, char * argv[])
{
Py_Initialize();
PyObject * test_module = PyImport_ImportModule("test");
// do something with test_module
Py_Finalize();
return 0;
}
You can get G from python using PyObject_GetAttrString(), or since you declared it as cdef, you can access it directly using your OS's symbol resolution tools, such as GetProcAddress() for windows.
It may be possible to link dynamically at load time, yet still use importmodule to let python do its initialization magic, but I have no idea how to do that or if it's even possible.
I am trying to generate C++ based .SO file along with wrapper by using swig. This .SO was generated to make a call from python(Ubuntu Environment).
It works well if i tried with Simple C++ code, but when i try to build with OpenCV, facing some issues.
opencvtest.cpp:
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <iostream>
using namespace cv;
using namespace std;
int main()
{
Mat image;
image = imread("/home/swigtest/MyPic.jpg",1); // Read the file
namedWindow( "Display window", WINDOW_AUTOSIZE );// Create a window for display.
imshow( "Display window", image ); // Show our image inside it.
destroyWindow("Display Window");
waitKey(0); // Wait for a keystroke in the window
return 0;
}
opencvtest.i
%module opencvtest
%{
/* Put header files here or function declarations like below */
extern int main();
%}
extern int main();
So for Completed the Following steps:
opencvtest.cpp - Sample code that i wrote.
opencvtest.i - The corresponding intermediate file that i wrote
Executing the command : swig -c++ -python opencvtest.i
GCC command to create .o's(with wrapper) : g++ -fpic -c opencvtest.cpp opencvtest_wrap.cxx -I/usr/include/python2.7 -I/usr/local/include -I/usr/local/include/opencv -I/usr/local/include/opencv2
Command to create .so : g++ -shared opencvtest.o opencvtest_wrap.o -o
_opencvtest.so -L/usr/local/lib /usr/local/lib/libopencv_calib3d.so /usr/local/lib/libopencv_core.so /usr/local/lib/libopencv_features2d.so /usr/local/lib/libopencv_flann.so /usr/local/lib/libopencv_highgui.so /usr/local/lib/libopencv_imgcodecs.so /usr/local/lib/libopencv_imgproc.so /usr/local/lib/libopencv_ml.so /usr/local/lib/libopencv_objdetect.so /usr/local/lib/libopencv_photo.so /usr/local/lib/libopencv_shape.so /usr/local/lib/libopencv_stitching.so /usr/local/lib/libopencv_superres.so /usr/local/lib/libopencv_ts.a /usr/local/lib/libopencv_video.so /usr/local/lib/libopencv_videoio.so /usr/local/lib/libopencv_videostab.so
6.Generated .so file successfully
Then tried to test the python, followed the steps:
7.python
8.>>> import opencvtest
9.>>> opencvtest.main()
I got the error : Segmentation fault (core dumped)
Any one of the expert can help to clear the issue?
Just changed the method name main in opencvtest.cpp.
Remember that change the method name at step 9 also.
It's working fine for me!.
But I don't know the process running behind this method call.
I changed the method name from main to one.
In Step 9: >>> opencvtest.one()
So I have a Python program that's finding .txt file directories and then passing those directories as a list(I believe) to my C++ program. The problem I am having is that I am not sure how to pass the list to C++ properly. I have used :
subprocess.call(["path for C++ executable"] + file_list)
where file_list is the [] of txt file directories.
My arguments that my C++ code accepts are:
int main (int argc, string argv[])
Is this correct or should I be using a vector? When I do use this as my argument and try to print out the list I get the directory of my executable, the list, and then smiley faces, symbols, and then the program crashes.
Any suggestions? My main point that I am trying to find out is the proper syntax of utilizing subprocess.call. Any help would be appreciated! thanks!
Another option is to use cython, (not a direct answer). Here is a simple complete example:
Suppose you have the following files:
cython_file.cpp
python_file.py
setup.py
sum_my_vector.cpp
sum_my_vector.h
setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension(
name="cython_file",
sources=["cython_file.pyx", "sum_my_vector.cpp"],
extra_compile_args=["-std=c++11"],
language="c++",
)]
setup(
name = 'cython_file',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
)
cython_file.pyx
from libcpp.vector cimport vector
cdef extern from "sum_my_vector.h":
int sum_my_vector(vector[int] my_vector)
def sum_my_vector_cpp(my_list):
cdef vector[int] my_vector = my_list
return sum_my_vector(my_vector)
sum_my_vector.cpp
#include <iostream>
#include <vector>
#include "sum_my_vector.h"
using namespace::std;
int sum_my_vector(vector<int> my_vector)
{
int my_sum = 0;
for (auto iv = my_vector.begin(); iv != my_vector.end(); iv++)
my_sum += *iv;
return my_sum;
}
sum_my_vector.h
#ifndef SUM_MY_VECTOR
#define SUM_MY_VECTOR
using namespace::std;
int sum_my_vector(vector<int> my_vector);
#endif
python_file.py
from cython_file import sum_my_vector_cpp
print sum_my_vector_cpp([1,2,3,5])
Now run
python setup.py build_ext --inplace
and the you can run the python file
python python_file.py
11
"Passing a list through Python to C++"
An alternative approach would be to use Boost.Python, this may not answer your question directly, but still its worth pointing out another solution.
#include <boost/python.hpp>
#include <vector>
#include <string>
void get_dir_list( boost::python::list dir_list )
{
for (int i = 0; i < len(dir_list); ++i)
{
std::string x = boost::python::extract<std::string>(dir_list[i]);
// perform stuffs
std::cout << "This is " << x << std::endl ;
}
}
BOOST_PYTHON_MODULE(get_dir_list)
{
def("get_dir_list", get_dir_list);
}
Compiled Using :
g++ main.cpp -shared -fPIC -o get_dir_list.so -I/usr/include/python2.7 -lboost_python
Usage :
import get_dir_list
import os
get_dir_list.get_dir_list(os.listdir('.'))
Live Demo Here
I'll post this alternative solution since it would also work for other long lists of strings that needed to be passed.
In your Python script create a text file (I'll call it "masterFile") and write the file paths to the masterFile. You could give each file path a separate line. Then pass the masterFile's file path to your C++ program. This way you don't have to worry about the length of your command line arguments. Let your C++ program open and read the file for processing.
You can use something like os.remove() to get rid of the masterFile in your Python script once the C++ program has finished.
Also, you mentioned in the comments that you need to do different tasks dependent on different file paths: A suggestion would be to add a char at the beginning of each line in the masterFile to signal what needs to be done for the particular file. Example:
a Random/path/aFile.txt # a could mean do task 1
b Random2/path2/differentFile.c # b could mean do task 2
You pass a list to subprocess.call. subprocess.call converts this to what is needed for the system (which may vary, but certainly isn't a Python list). The system then arranges for this to be copied somewhere in the new process, and sets up the standard arguments to main, which are int, char**. In your C++ program, you must define main as int main( int argc, char** argv ); nothing else will work. (At least... a system could support int main( std::string const& ) or some such as an extension. But I've never heard of one that did.)