I have a shared library (DLL) in C++ with some C-style API which I use from Python. There is a function which takes a C callback as an argument. As far as I understood, Python extension module (a separate DLL) is required to do that. This module shall pass a native "proxy" callback to API, then call Python object from that callback. I try do use it as follows (it fails):
from ctypes import CDLL
from test_extension import set_python_callback
def callback():
pass
if __name__ == '__main__':
library = CDLL('test_dll.shared-library')
set_python_callback(library, callback)
library.trigger_callback()
I'm using Windows 8 x64 environment with Python 2.7.6 (32 bit) and MinGW 32-bit compiler (mingw32). Python extension is built by distutils. API library can be changed if necessary, however, Python-specific code has to be kept in a separate library.
Here is my code, reduced. All error-checking was removed, however, when performed it showed no errors either from Python API or from Windows API.
API library:
/* typedef void (*callback_t)(); */
static callback_t s_callback = nullptr;
void DLL_PUBLIC set_callback(callback_t callback) {
s_callback = callback;
}
void DLL_PUBLIC trigger_callback() {
s_callback(); // <--------------------------(1)
}
Python extension module function set_python_callback(library, callback) takes a ctypes.CDLL object and Python callable. It then extracts native DLL handle from the former:
PyObject* handle_object = PyObject_GetAttrString(library, "_handle");
const HMODULE handle = static_cast<const HMODULE>(
PyLong_AsVoidPtr(handle_object));
const native_set_callback_t native_api_routine =
reinterpret_cast<const native_set_callback_t>(
GetProcAddress(handle, "set_callback"));
native_api_routine(common_callback);
C function common_callback() is implemented as follows:
extern "C" void DLL_PUBLIC common_callback() {
// <--------------------------------------- (2)
PyObject *arguments = Py_BuildValue("()");
PyObject *callback_result = PyEval_CallObject(s_callback, arguments);
Py_DECREF(arguments);
Py_XDECREF(callback_result);
}
The error message says:
Invoking callback from DLL... Traceback (most recent call last):
File "script.py", line 14, in <module>
library.trigger_callback()
WindowsError: exception: access violation reading 0x00000028
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute. The strange thing is, changing the way common_callback() call Python code (e. g. passing Py_None instead of empty tuple) changes the address in the error message.
Maybe this is somehow related to the calling convention, but I have no idea where it's wrong exactly or how to fix it.
Solved the issue by registering the thread for GIL as described in documentation.
Please note that original API DLL, Python extension DLL, and the script were executed in single-threaded mode and no additional thread had been created, either Python-managed, or not. However, there had been a synchronization issue for sure, because using Py_AddPendingCall() also worked. (Using is is discouraged; I found it analyzing signals module sources, it lead to the solution above.)
This statement of mine was incorrect:
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute.
I was mislead by the fact that some of Python API functions still worked (e. g. Py_BuildValue or Py_Initialize), but most didn't (e. g. PySys_WriteStdout, PyObject_Call, etc).
Related
I have the following python 3.8.2 code that attempts to access methods from my C# DLL
In the example it is using a method to calculate the area of a pipe.
import clr
clr.AddReference(r"calcsDLL")
from myDLL import calcs
calcs.Pipe_Area(0.2)
Here is my implementation of the C# DLL
namespace myDLL
{
public class calcs
{
public double pipe_area(double pipeDiameter)
{
double pipeArea = 0.25 * 3.1415 * Math.Pow(pipeDiameter, 2);
return pipeArea;
}
}
}
Most frustrating of all is that I had this exact code working already, I changed the DLL build inside Visual Studio to Any CPU to see if it would break the connection (it broke something) however when I changed it back to x64 build the python script was still broken. The python script still can connect and see the DLL methods however throws a type error when I try to execute them.
The error message I get from Python is:
TypeError: No method matches given arguments for Pipe_Area: ()
Solution: The classes must be static as I'm not creating a new instance within the python script.
I have been working on a project where I want to remove the boost dependencies and replace it with the Python C API.
I spent some time understanding the Python C API and I saw this
catch (error_already_set const &)
I read the boost docs but it explains where it is used. But I want to know why it is needed and how can I achieve the same functionality using the native Python C api.
Boost throws error_already_set when a Python error has occurred. So if you see code like this:
try {
bp::exec(bp::str("garbage code is garbage"));
} catch (const bp::error_already_set&) {
// your code here to examine Python traceback etc.
}
you'll replace it with:
your_ptr<PyObject> res = PyRun_String("garbage code is garbage");
if (!res) {
// your code here to examine Python traceback etc.
}
In other words, wherever you see catch(error_already_set), there you will likely want to do some error handling using whatever PyObject* or other value is involved to recognize when an error has occurred (and therefore you can examine the traceback, or convert it into a C++ exception).
Note that I'm constrained to use Python 2.6. I have a Python 2.6 application that uses a C++ multi-threaded API library built with boost-python. My use-case was simply to execute a Python function callback from a C++ boost thread but despite the many different attempts and researching all the available online resources I haven't found any way that works. All the proposed solutions revolve around a different combination of the functions: Py_Initialize*, PyEval_InitThreads, PyGILState_Ensure, PyGILState_Release but after trying all possible combinations nothing works in practice e.g.
Embedding Python in multi-threaded C++ applications with code here
PyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseum)
Therefore, this question: how can I start and run a Python thread from C++? I basically want to: create it, run it with a Python target function object and forget about it.
Is that possible?
Based on the text below from your question:
Run a Python thread from C++? I basically want to: create it, run it with a Python target function object and forget about it.
You may find useful to simply spawn a process using sytem:
system("python myscript.py")
And if you need to include arguments:
string args = "arg1 arg2 arg3 ... argn"
system("python myscript.py " + args)
You should call PyEval_InitThreads from your init routine (and leave the GIL held). Then, you can spawn a pthread (using boost), and in that thread, call PyGILState_Ensure, then call your python callback (PyObject_CallFunction?), release any returned value or deal with any error (PyErr_Print?), release the GIL with PyGILState_Release, and let the thread die.
void *my_thread(void *arg)
{
PyGILState_STATE gstate;
PyObject *result;
gstate = PyGILState_Ensure();
if ((result = PyObject_CallFunction(func, NULL))) {
Py_DECREF(result);
}
else {
PyErr_Print();
}
PyGILState_Release(gstate);
return NULL;
}
The answer to the OP How to call Python from a boost thread? also answers this OP or DP (Derived:)). It clearly demonstrates how to start a thread from C++ that callbacks to Python. I have tested it and works perfectly though needs adaptation for pre-C++11. It uses Boost Python, is indeed an all inclusive 5* answer and the example source code is here.
I'd like to know if it's possible prevent or allow a dll to be loaded by python ctypes, based on whether a condition is true from within the dll.
Some background:
My application uses various calculation algorithms, which I prototyped in Python, and then reimplemented in C++ for a speed boost. I still use Python for the application "glue" and GUI. I'm accessing the functions in the dll using a ctypes wrapper.
I now need to secure the software, so that it will only run if a security dongle is present. The open nature of Python makes this difficult, so I'd like to be able to stop a python script loading the dll unless a function which checks the dongle is present returns True.
Example Python wrapper:
from ctypes import cdll, c_int , c_float, c_bool
lib = cdll.LoadLibrary('my.dll')
cpp_sum = lib.sum
cpp_sum.argtypes = [c_int,c_int]
cpp_sum.restype = c_int
def wrapped_sum(value_1,value_2):
return cpp_sum(value_1,value_2)
And the code for the my.dll:
#include "stdafx.h"
#include <cmath>
#define DLLEXPORT extern "C" __declspec(dllexport)
DLLEXPORT int sum(int a, int b)
{return a + b;}
//pseudo dongle code:
bool is_dongle_present(){
if dongle present return true
else return false
Ideally the dll would fail to load if dongle_is_present returned false. Can anyone help?
Please tell me if this question is unclear!
Many thanks
Add DllMain function to your library.
An optional entry point into a dynamic-link library (DLL). When the
system starts or terminates a process or thread, it calls the
entry-point function for each loaded DLL using the first thread of the
process. The system also calls the entry-point function for a DLL when
it is loaded or unloaded using the LoadLibrary and FreeLibrary
functions.
You could prevent dll load by returning FALSE on DLL_PROCESS_ATTACH:
When the system calls the DllMain function with
the DLL_PROCESS_ATTACH value, the function returns TRUE if it succeeds
or FALSE if initialization fails. If the return value is FALSE when
DllMain is called because the process uses the LoadLibrary function,
LoadLibrary returns NULL. (The system immediately calls your
entry-point function with DLL_PROCESS_DETACH and unloads the DLL.) If
the return value is FALSE when DllMain is called during process
initialization, the process terminates with an error
See DllMain MSDN entry for the additional information.
I have a python extension function
static void EXTrender_effect(EffectGlobals_t *effect_handle,
std::string preset,
bp::object dest, int xdim, int ydim)
{ ... }
that I export as usual in my boost.python extension:
def("render_effect", EXTrender_effect);
When I call that from python(2.7), I get a C++ exception boost::python::error_already_set. Tracing that down in Visual Studio, I can see it's coming from a boost::python::objects::function::argument_error. So OK, I have an argument error; I'm probably calling it with the wrong args. What I'd like to do is print or throw something sensible in my python extension when this happens, so users of my extension will see the nice message that I know is lurking in PyErr_Fetch. (I can see the message getting built in the boost.python argument_error code.)
But I can't catch that error_already_set; boost.python does that internally around the funcall. And I can't check for python errors in my extension code, since my function never gets called (the argument error is detected by boost.python). What ends up happening is it just silently fails.
What can I hook up so I (or my users) can see the argument error messages? And ideally convert those into python exceptions?
This is all Win7, Python2.7.