I wrote a simple Python extension function in C that just reads a Numpy array and it crashes.
static PyObject *test(PyObject *self, PyObject *args)
{
PyArrayObject *array = NULL;
if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) // Crash
return NULL;
return Py_BuildValue("d", 0);
}
Here is how it is called:
l = np.array([1,2,3,1,2,2,1,3])
print("%d" % extension.test(l))
What's wrong with my code?
I think the error is in the code you did not include in your example: have you remembered to call import_array() in your module init function:
... this subroutine must also contain calls to import_array() and/or import_ufunc() depending on which C-API is needed. Forgetting to place these commands will show itself as an ugly segmentation fault (crash) as soon as any C-API subroutine is actually called.
http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.how-to-extend.html#required-subroutine
I copied your example verbatim and added (using python 3)
PyMODINIT_FUNC
PyInit_numpytest(void)
{
import_array();
return PyModule_Create(&numpytest);
}
and the example ran without issues. Removing the call on the other hand produces a crash.
A difference between MacOS and Linux
The following works fine for me on Linux:
PyMODINIT_FUNC PyInit_numpytest(void) {
import_array();
return PyModule_Create(&numpytest);
}
However, when using MacOS, I get the following error when trying to import the library:
SystemError: лy () method: bad call flags
The following works fine on both MacOS and Linux:
PyMODINIT_FUNC PyInit_numpytest(void) {
PyObject* mod = PyModule_Create(&numpytest);
import_array();
return mod;
}
Notice that all I did was to change the order of PyModule_Create and import_array.
Related
I am using Python code within C++ code and trying to pass a list argument to a function written in Python. I tried the normal way of executing the Python code without passing any argument and it was all working fine but when I pass a list as an argument, I get a segmentation fault.
Here is my code:
#define PY_SSIZE_T_CLEAN
#include</usr/include/python3.6/Python.h>
#include <bits/stdc++.h>
using namespace std;
int callModuleFunc(int array[], size_t size) {
PyObject *mymodule = PyImport_ImportModule("test");
PyObject *myfunc = PyObject_GetAttrString(mymodule, "get_lists");
cout<<"Imported"<<endl;
PyObject *mylist = PyList_New(size);
cout<<"Imported3"<<endl;
for (size_t i = 0; i != size; ++i) {
PyList_SET_ITEM(mylist, i, PyLong_FromLong(array[i]));
}
PyObject *arglist = Py_BuildValue("(O)", mylist);
cout<<"Imported1"<<endl;
PyObject *result = PyObject_CallObject(myfunc, arglist); // getting segmentation fault here
cout<<"Imported5"<<endl;
int retval = (int)PyLong_AsLong(result);
Py_DECREF(result);
Py_DECREF(arglist);
Py_DECREF(mylist);
Py_DECREF(myfunc);
Py_DECREF(mymodule);
return retval;
}
int main(int argc, char const *argv[])
{
wchar_t * program = Py_DecodeLocale(argv[0], NULL);
if(!program){
cout<<"***Error***"<<endl;
exit(1);
}
Py_SetProgramName(program);
Py_Initialize();
PyObject *module = NULL, *result = NULL;
PyRun_SimpleString("print('Hello from python')\n"
"print('Hiii')");
int arr[5] = {1,3,4,5,6};
callModuleFunc(arr, 5);
if(Py_FinalizeEx() < 0){
cout<<"***Error***"<<endl;
exit(120);
}
PyMem_RawFree(program);
return 0;
}
When I call PyObject_CallObject(myfunc, arglist), I get a segmentation fault.
I am totally new to it so I'm just trying stuff from the internet.
I'm using Python version 3.6 with g++ compiler 7.5.
Here is my test.py:
def get_lists(l1):
print("Lists: ", l1)
return 10
Please let me know how I can resolve this.
Thanks
The Python function get_lists is not known during execution.
Note: The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.
see https://docs.python.org/3/library/test.html
If the name of the imported Python file is renamed to another name (e.g. list.py) the Python function can be found.
Additional Hints
For the environment of the OP the problem is then solved. For my environment (Python 3.9.5) I need additionally replace the PyRun_SimpleString with:
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\".\")");
Otherwise the module can't be imported.
Finally, it is advisable to check each function call for errors (e.g. whether it returns NULL) and to use PyErr_Print so that the actual cause of the error is printed instead of a crash due to a segmentation fault.
For example an error message could look like this:
AttributeError: module 'test' has no attribute 'get_lists'
Test
The output of the line Lists: [1, 3, 4, 5, 6] on the console shows that the Python function is called correctly when the above points are taken into account.
I am coding in c to solve a question. In the c code, I need to call a self-defined python to solve a equation. I am able to call the python function for twice time, however when i call the third time, I get an error:"Unhandled exception at 0x00007FEDF8A4E0C(multiarray.cp35-win_amd64.pyd) in Project.1.exe: 0xC0000005: Access violation writing location 0x000000000000000A." It is very strange because in the first and second time, I can call the function successfully and if I independently call the function with the parameters used in the third time, the function can run very well. However, if I try to do the work continuously, it causes the exception above. I have seen a similar question 'C's Python API: Calling python function 2X causes Access violation writing location' in STACKOVERFLOW, however, there is no answer for that problem.
PyObject* pRet = PyEval_CallObject(pv, args);
The c code above is the place where causing the exception when calling the python function the third time. The whole process calling the python function is described more clearly below.
Here comes the python function first
from sympy import *
init_printing(use_unicode=True)
def evaluate_dH(eps,bmax,H,B_H1,B_H2):
x=symbols("x")
expr=eps*(x-bmax)**2*x**(H+2)-(H+2)*x*B_H1+(H+1)*B_H2
result=solve(expr,x)#result is a list type
When solving out the equation, I used two condition that ">bmax" and "not a complex" to find the value in 'result' I need. the final is only one. Thus,
return result[0]
Second, here comes the calling process in c code.
Firstly, "double find_dH" is a c function I used in my main function.
double find_dH
(....)
{
double B_H2 = findBH(region, vecp_vertex, H + 2, index);
double B_H1 = findBH(region, vecp_vertex, H + 1, index);
double eps = 0.01;
double bmax = findbmax(region, vecp_vertex);
Py_Initialize();
PyObject *pModule = NULL;
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append('./')");
pModule = PyImport_ImportModule("evaluate_dH");
PyObject* pv = PyObject_GetAttrString(pModule, "evaluate_dH");
PyObject* args = PyTuple_New(5);
PyTuple_SetItem(args, 0, Py_BuildValue("d", double(eps)));
PyTuple_SetItem(args, 1, Py_BuildValue("d", double(bmax)));
PyTuple_SetItem(args, 2, Py_BuildValue("i", H));
PyTuple_SetItem(args, 3, Py_BuildValue("d", double(B_H1)));
PyTuple_SetItem(args, 4, Py_BuildValue("d", double(B_H2)));
PyObject* pRet = PyEval_CallObject(pv, args);
double result = PyFloat_AsDouble(pRet);
Py_Finalize();
return result;
}
Here is the place where causing the exception:"Unhandled exception at 0x00007FEDF8A4E0C(multiarray.cp35-win_amd64.pyd) in Project.1.exe: 0xC0000005: Access violation writing location 0x000000000000000A.". But, "PyEval_CallObject(pv, args)" works well before the third time.
PyObject* pRet = PyEval_CallObject(pv, args);
I guess the exception is caused by the reason that:
1. import module 'sympy' multitimes
2. no free for those 'PyObject's in the calling process after each calling.
thanks for your reading and answers!
Working in C++, I was getting a similar error when calling
PyObject* pValue = Py_BuildValue("(s)", param);
To solve the issue I copied the string into a char*
PyObject* getPythonValueString(string param)
{
char* paramVal = new char[param.length() + 1];
strcpy(paramVal, param.c_str());
PyObject* pValue = Py_BuildValue("(s)", paramVal);
PyErr_Print();
// Remember to clean up.
delete[] paramVal;
return pValue;
}
I am new to the business of writing custom Python modules and I am a bit confused how Capsules work. I use Python 2.7.6 from the system OSX installation and try to use Capsules (as recommended for Python > 2.7) for passing pointers around (before they used PyCObject for that). My code does not work at the moment and I would like to get some insights how things should be handled in principle here. The code should define a class LuscherClm and I want be able to do the following:
>>> c40=Luscher(4,0)
>>>
>>> c40(0.12)
>>> <print the result of the evaluation>
First question: at the moment I would have to do something like:
>>> c40=Luscher.init(4,0)
>>>
>>> c40.eval(0.12)
Segfault
My first question is therefore: how do I have to modify the method table to have more operator-style casts instead of the member functions init and eval.
However, my code has other problems and here is the relevant part (the underlying C++ class works smoothly, I use it in production a lot):
The destructor:
//destructor
static void clm_destruct(PyObject* capsule){
void* ptr=PyCapsule_GetPointer(capsule,"zetfunc");
Zetafunc* zetptr=static_cast<Zetafunc*>(ptr);
delete zetptr;
return;
}
The constructor: it returns the pointer to the capsule. I do not know whether this is correct. Because in this case when I call, clm=LuscherClm.init(l,m), the clm object is a PyCapsule and has no attribute eval so that I cannot call clm.eval(x) on that. How should this be handled?
//constructor
static PyObject* clm_init(PyObject* self, PyObject *args){
//return value
PyObject* result=NULL;
//parse variables
unsigned int lval=0;
int mval=0;
if(!PyArg_ParseTuple(args,"li",&lval,&mval)){
::std::cout << "Please specify l and m!" << ::std::endl;
return result;
}
//class instance:
Zetafunc* zetfunc=new Zetafunc(lval,mval);
instanceCapsule=PyCapsule_New(static_cast<void*> (zetfunc),"zetfunc",&clm_destruct);
return instanceCapsule;
}
So how is the capsule passed to the evaluate function? the code below is not correct since I have not updated it after moving from CObjects to Capsules. Shall the capsule be a global variable (I do not like that) or how can I pass it to the evaluation function? Or shall I call it on self, but what is self at the moment?
//evaluate the function
static PyObject* clm_evaluate(PyObject* self, PyObject* args){
//get the PyCObject from the capsule:
void* tmpzetfunc=PyCapsule_GetPointer(instanceCapsule,"zetfunc");
if (PyErr_Occurred()){
std::cerr << "Some Error occured!" << std::endl;
return NULL;
}
Zetafunc* zetfunc=static_cast< Zetafunc* >(tmpzetfunc);
//parse value:
double x;
if(!PyArg_ParseTuple(args,"d",&x)){
std::cerr << "Specify a number at which you want to evaluate the function" << std::endl;
return NULL;
}
double result=(*zetfunc)(x).re();
//return the result as a packed function:
return Py_BuildValue("d",result);
}
//methods
static PyMethodDef LuscherClmMethods[] = {
{"init", clm_init, METH_VARARGS, "Initialize clm class!"},
{"eval", clm_evaluate, METH_VARARGS, "Evaluate the Zeta-Function!"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
Python < 3 initialisation function:
PyMODINIT_FUNC
initLuscherClm(void)
{
PyObject *m = Py_InitModule("LuscherClm", LuscherClmMethods);
return;
}
Can you explain to me what is wrong and why? I would like to stay away from SWIG or boost if possible, since this module should be easily portable and I want to avoid having to install additional packages every time I want to use it somewhere else.
Further: what is the overhead produced by the C/API in calling the function? I need to call it an order of O(10^6) times and I would still like it to be fast.
Ok, I am using boost.python now but I get a segfault when I run object.eval(). That is my procedure now:
BOOST_PYTHON_MODULE(threevecd)
{
class_< threevec<double> >("threevecd",init<double,double,double>());
}
BOOST_PYTHON_MODULE(LuscherClm)
{
class_<Zetafunc>("LuscherClm",init<int,int, optional<double,threevec<double>,double,int> >())
.def("eval",&Zetafunc::operator(),return_value_policy<return_by_value>());
boost::python::to_python_converter<dcomplex,dcomplex_to_python_object>();
}
dcomplex is my own complex number implementation. So I had to write a converter:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
if(fabs(comp.im())<std::numeric_limits<double>::epsilon()){
boost::python::object result=boost::python::object(complex<double>(comp.re(),comp.im()));
return boost::python::incref(result.ptr());
}
else{
return Py_BuildValue("d",comp.re());
}
}
};
Complex128 is a numpy extension which is not understood by boost. So my questions are:
1) how can I return a complex number as a python datatype (is complex a standard python type?)
2) Why do I get a segfault. My result in my testcase is real so it should default to the else statement. I guess that the pointer runs out of scope and thats it. But even in the if-case (where I take care about ref-increments), it segfaults. Can someone help me with the type conversion issue?
Thanks
Thorsten
Ok, I got it. The following converter does the job:
struct dcomplex_to_python_object
{
static PyObject* convert(dcomplex const& comp)
{
PyObject* result;
if(std::abs(comp.im())<=std::numeric_limits<double>::epsilon()){
result=PyFloat_FromDouble(comp.re());
}
else{
result=PyComplex_FromDoubles(comp.re(),comp.im());
}
Py_INCREF(result);
return result;
}
};
Using this converter and the post by Wouter, I suppose my question is answered. Thanks
Hi I am trying to return a value (119599936) from the embedded python to C++
Py_Initialize();
PyObject* myModuleString = PyString_FromString((char*)"memory_leak_test");
PyObject* myModule = PyImport_Import(myModuleString);
PyObject* myFunction = PyObject_GetAttrString(myModule,(char*)"begin");
PyObject* args = PyTuple_Pack(1,PyString_FromString("CacheSetup"));
PyObject* myResult = PyObject_CallObject(myFunction, args);
double result = PyFloat_AsDouble(myResult);// <<<<<<<<< I guess thts where i'm goin wrong
std::cout << result << std::endl;
Py_Finalize();
I tried
unsigned long result = PyLong_AsUnsignedLong(myResult);
unsigned long long result = PyLong_AsUnsignedLongLongMask(myResult);
double result = PyLong_AsDouble(myResult2);
None of the above seemed to work for the when I try to return the number 119599936, the error i'm getting is
Exception SystemError: '..\\Objects\\longobject.c:739: bad argument to internal
function' in 'garbage collection' ignored
Fatal Python error: unexpected exception during garbage collection
Since your code performs no error checking whatsoever, and you don't provide the source to the memory_leak_test module, it's anyone's guess what went wrong. My guess would be that memory_leak_test failed to import, or that memory_leak_test.begin("CacheSetup") raised an exception or returned None instead of returning a number like your code expects.
To debug this problem, rewrite with the following guidelines in mind:
Check errors from Python/C functions that may return them. Whenever an error you can't handle appears, call PyErr_Print() and terminate current execution (unless you are expecting the error and want to handle it properly - in that case, call PyErr_Clear() instead.)
Invoke Py_DECREF on objects returned as new references once you no longer need them. Examples of these are the return values of PyString_FromString, PyTuple_Pack, and PyObject_CallOject.
Learn of and use convenience functions such as PyObject_CallFunction, PyObject_CallMethod, and PyImport_ImportModule which significantly reduce boilerplate code for creating temporary strings and tuples. The best way to find these is by perusing existing Python/C sources, such as those that come with Python.
Use PyObject_Print() to print intermediate results for debugging.
Here is a possible modification of your code according to the above:
Py_Initialize();
PyObject *module = PyImport_ImportModule("memory_leak_test");
if (!module) {
PyErr_Print();
return;
}
PyObject *result = PyObject_CallMethod(module, "begin", "s", "CacheSetup");
Py_DECREF(module);
if (!result) {
PyErr_Print();
return;
}
printf("got result of type %s\n", Py_TYPE(result)->tp_name);
PyObject_Print(result, stdout, 0);
putchar('\n');
Py_DECREF(result);
Py_Finalize();
I have been looking for this solution for some time and cannot find a solution. Some solutions have suggested using the "system" function in the . However, this suggestion is then always followed by 10 people saying never to use system as it lacks thread safety. Does anyone know how to do this?
Just some info: my python script gets and parses JSON from the internet, then creates a textfile, which the c program then uses. So, instead of having to run one, then the other, I want to run them both in succession from the C exe.
EDIT: Grammer
You can Use system function in c,c++
like this
system("python ashouri.py");
or
use this code
static PyObject *my_callback = NULL;
static PyObject *
my_set_callback(PyObject *dummy, PyObject *args)
{
PyObject *result = NULL;
PyObject *temp;
if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(my_callback); /* Dispose of previous callback */
my_callback = temp; /* Remember new callback */
/* Boilerplate to return "None" */
Py_INCREF(Py_None);
result = Py_None;
}
return result;
}
Be Successfull