Develop Reference

Cpp compilation from python fail but not in the shell

I have a cpp file that compiles fine with g++ by using the shell:
extern "C"{
#include <quadmath.h>
}
inline char* print( const __float128& val)
{
char* buffer = new char[128];
quadmath_snprintf(buffer,128,"%+.34Qe", val);
return buffer;
}
int main(){
__float128 a = 1.0;
print(a);
return 0;
}
However, when I try to compile it via a python scrit, it fails with the following error:
"undefined reference to quadmath_snprintf"
Here the code of the python script:
import commands
import string
import os
(status, output) = commands.getstatusoutput("(g++ test/*.c -O3 -lquadmath -m64)")
Any idea how to solve this? Thanks.

When you open a shell a whole of stuff is silently initialized for you, and most important for your issue, environment variables are set. What you most likely miss is the definition of LIBRARY_PATH, which is the variable used by the linker to look for libraries matching the ones you instruct it to link using the -lNAME flags.
What the linker needs is a list of directories where it will search for files matching libNAME.{a,so}. You can also pass these directories directly using the -L flag, but in general, you should probably try to use a program like CMake, Make or any other build tool.
This will give you access to commands like find_package and target_link_libraries (CMake), to find, respectively add libraries to your build targets, instead of having to maintain your python to compile your stuff.

Python callback invocation from C++ fails on native site

I have a shared library (DLL) in C++ with some C-style API which I use from Python. There is a function which takes a C callback as an argument. As far as I understood, Python extension module (a separate DLL) is required to do that. This module shall pass a native "proxy" callback to API, then call Python object from that callback. I try do use it as follows (it fails):
from ctypes import CDLL
from test_extension import set_python_callback
def callback():
pass
if __name__ == '__main__':
library = CDLL('test_dll.shared-library')
set_python_callback(library, callback)
library.trigger_callback()
I'm using Windows 8 x64 environment with Python 2.7.6 (32 bit) and MinGW 32-bit compiler (mingw32). Python extension is built by distutils. API library can be changed if necessary, however, Python-specific code has to be kept in a separate library.
Here is my code, reduced. All error-checking was removed, however, when performed it showed no errors either from Python API or from Windows API.
API library:
/* typedef void (*callback_t)(); */
static callback_t s_callback = nullptr;
void DLL_PUBLIC set_callback(callback_t callback) {
s_callback = callback;
}
void DLL_PUBLIC trigger_callback() {
s_callback(); // <--------------------------(1)
}
Python extension module function set_python_callback(library, callback) takes a ctypes.CDLL object and Python callable. It then extracts native DLL handle from the former:
PyObject* handle_object = PyObject_GetAttrString(library, "_handle");
const HMODULE handle = static_cast<const HMODULE>(
PyLong_AsVoidPtr(handle_object));
const native_set_callback_t native_api_routine =
reinterpret_cast<const native_set_callback_t>(
GetProcAddress(handle, "set_callback"));
native_api_routine(common_callback);
C function common_callback() is implemented as follows:
extern "C" void DLL_PUBLIC common_callback() {
// <--------------------------------------- (2)
PyObject *arguments = Py_BuildValue("()");
PyObject *callback_result = PyEval_CallObject(s_callback, arguments);
Py_DECREF(arguments);
Py_XDECREF(callback_result);
}
The error message says:
Invoking callback from DLL... Traceback (most recent call last):
File "script.py", line 14, in <module>
library.trigger_callback()
WindowsError: exception: access violation reading 0x00000028
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute. The strange thing is, changing the way common_callback() call Python code (e. g. passing Py_None instead of empty tuple) changes the address in the error message.
Maybe this is somehow related to the calling convention, but I have no idea where it's wrong exactly or how to fix it.

Solved the issue by registering the thread for GIL as described in documentation.
Please note that original API DLL, Python extension DLL, and the script were executed in single-threaded mode and no additional thread had been created, either Python-managed, or not. However, there had been a synchronization issue for sure, because using Py_AddPendingCall() also worked. (Using is is discouraged; I found it analyzing signals module sources, it lead to the solution above.)
This statement of mine was incorrect:
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute.
I was mislead by the fact that some of Python API functions still worked (e. g. Py_BuildValue or Py_Initialize), but most didn't (e. g. PySys_WriteStdout, PyObject_Call, etc).

ctypes - Beginner

I have the task of "wrapping" a c library into a python class. The docs are incredibly vague on this matter. It seems they expect only advanced python users would implement ctypes.
Some step by step help would be wonderful.
So I have my c library. What do I do? What files do I put where? How do I import the library? I read that there might be a way to "auto wrap" to Python?
(By the way I did the ctypes tutorial on python.net and it doesn't work. Meaning I'm thinking they are assuming I should be able to fill in the rest of the steps.)
In fact this is the error I get with their code:
File "importtest.py", line 1
>>> from ctypes import *
SyntaxError: invalid syntax
I could really use some step by step help on this!

Here's a quick and dirty ctypes tutorial.
First, write your C library. Here's a simple Hello world example:
testlib.c
#include <stdio.h>
void myprint(void);
void myprint()
{
printf("hello world\n");
}
Now compile it as a shared library (mac fix found here):
$ gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c
# or... for Mac OS X
$ gcc -shared -Wl,-install_name,testlib.so -o testlib.so -fPIC testlib.c
Then, write a wrapper using ctypes:
testlibwrapper.py
import ctypes
testlib = ctypes.CDLL('/full/path/to/testlib.so')
testlib.myprint()
Now execute it:
$ python testlibwrapper.py
And you should see the output
Hello world
$
If you already have a library in mind, you can skip the non-python part of the tutorial. Make sure ctypes can find the library by putting it in /usr/lib or another standard directory. If you do this, you don't need to specify the full path when writing the wrapper. If you choose not to do this, you must provide the full path of the library when calling ctypes.CDLL().
This isn't the place for a more comprehensive tutorial, but if you ask for help with specific problems on this site, I'm sure the community would help you out.
PS: I'm assuming you're on Linux because you've used ctypes.CDLL('libc.so.6'). If you're on another OS, things might change a little bit (or quite a lot).

The answer by Chinmay Kanchi is excellent but I wanted an example of a function which passes and returns a variables/arrays to a C++ code. I though I'd include it here in case it is useful to others.
Passing and returning an integer
The C++ code for a function which takes an integer and adds one to the returned value,
extern "C" int add_one(int i)
{
return i+1;
}
Saved as file test.cpp, note the required extern "C" (this can be removed for C code).
This is compiled using g++, with arguments similar to Chinmay Kanchi answer,
g++ -shared -o testlib.so -fPIC test.cpp
The Python code uses load_library from the numpy.ctypeslib assuming the path to the shared library in the same directory as the Python script,
import numpy.ctypeslib as ctl
import ctypes
libname = 'testlib.so'
libdir = './'
lib=ctl.load_library(libname, libdir)
py_add_one = lib.add_one
py_add_one.argtypes = [ctypes.c_int]
value = 5
results = py_add_one(value)
print(results)
This prints 6 as expected.
Passing and printing an array
You can also pass arrays as follows, for a C code to print the element of an array,
extern "C" void print_array(double* array, int N)
{
for (int i=0; i<N; i++)
cout << i << " " << array[i] << endl;
}
which is compiled as before and the imported in the same way. The extra Python code to use this function would then be,
import numpy as np
py_print_array = lib.print_array
py_print_array.argtypes = [ctl.ndpointer(np.float64,
flags='aligned, c_contiguous'),
ctypes.c_int]
A = np.array([1.4,2.6,3.0], dtype=np.float64)
py_print_array(A, 3)
where we specify the array, the first argument to print_array, as a pointer to a Numpy array of aligned, c_contiguous 64 bit floats and the second argument as an integer which tells the C code the number of elements in the Numpy array. This then printed by the C code as follows,
1.4
2.6
3.0

Firstly: The >>> code you see in python examples is a way to indicate that it is Python code. It's used to separate Python code from output. Like this:
>>> 4+5
9
Here we see that the line that starts with >>> is the Python code, and 9 is what it results in. This is exactly how it looks if you start a Python interpreter, which is why it's done like that.
You never enter the >>> part into a .py file.
That takes care of your syntax error.
Secondly, ctypes is just one of several ways of wrapping Python libraries. Other ways are SWIG, which will look at your Python library and generate a Python C extension module that exposes the C API. Another way is to use Cython.
They all have benefits and drawbacks.
SWIG will only expose your C API to Python. That means you don't get any objects or anything, you'll have to make a separate Python file doing that. It is however common to have a module called say "wowza" and a SWIG module called "_wowza" that is the wrapper around the C API. This is a nice and easy way of doing things.
Cython generates a C-Extension file. It has the benefit that all of the Python code you write is made into C, so the objects you write are also in C, which can be a performance improvement. But you'll have to learn how it interfaces with C so it's a little bit extra work to learn how to use it.
ctypes have the benefit that there is no C-code to compile, so it's very nice to use for wrapping standard libraries written by someone else, and already exists in binary versions for Windows and OS X.

list exported functions from dll with ctypes

Is there any way to know which functions are exported from the dll through python foreign function library ctypes?
And if possible to know details about the exported functions through ctypes.
If yes, could someone provide a snippet of code?

If you are on Linux, there is a handy utility nm to list the content of a shared library (there is always a handy utility on Linux, especially for C stuff).
Here is the question about it.
You use it with the -D flag: nm -D ./libMyLib.so

I don't think ctypes offers this functionality. On Windows with visual studio:
DUMPBIN -EXPORTS XXX.DLL
Or for mingw on windows:
objdump -p XXX.dll

In general, this is not possible, because, again in general, dynamically loaded libraries do not carry the meta-information you require. It may be possible to obtain that information in certain special cases through system-specific ways, but ctypes itself does not fetch that information. You can record such info via ctypes (see e.g. the restype and argtypes attributes of function pointers), but only after you have obtained it by different means.

The below approach worked for both Windows and Ubuntu. For Windows, Cygwin is required.
Suppose, there is a c file like below which name is test.c.
int func1(int a, int b){
return a + b;
}
int func2(int a, int b){
return a - b;
}
And the above c codes were compiled to test.dll file with the below commands:
gcc -shared -Wl,-soname,adder -o test.dll -fPIC test.c
And the below Python script finds which functions of the test.dll can be used by Python.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from subprocess import Popen, PIPE
out = Popen(
args="nm ./test.dll",
shell=True,
stdout=PIPE
).communicate()[0].decode("utf-8")
attrs = [
i.split(" ")[-1].replace("\r", "")
for i in out.split("\n") if " T " in i
]
from ctypes import CDLL
functions = [i for i in attrs if hasattr(CDLL("./test.dll"), i)]
print(functions)
The output I got in Windows is as below:
['func1', 'func2']
The output I got in Ubuntu is as below:
['_fini', 'func1', 'func2', '_init']
The items of the above list are objects of _FuncPtr class.

#Mark's answer uses Visual Studio tools.
On windows you can also use Dependency Walker to get the function names of dll exports.
Sometimes names are mangled and can't be used as a valid python function name.
You can use getattr to get a handle to mangled functions, e.g:
mylib = ctypes.cdll('mylib.dll')
my_func = getattr(mylib, '_my_func#0')
my_func()

If you've also got the source for said library, and you're looking for a fully automated all-python way, you could use pycparser
for the file: prog.c
typedef short int ret_t;
typedef short int param_t;
ret_t add(param_t a, param_t b) {
return (ret_t)(a + b);
}
ret_t passthrough(ret_t (* func)(param_t a, param_t b), param_t a, param_t b) {
// parameter intentionally altered.
// if this isn't done, compiler will deem entire function redundant
return func(a, b + 1);
}
compiling with gcc
gcc -I. -E ./prog.c > prog-preproc.c
gives us the pre-processed c file: prog-preproc.c
then in python:
import pycparser
parser = pycparser.c_parser.CParser()
with open('prog-preproc.c', 'r') as fh:
ast = parser.parse(fh.read())
class FunctionVisitor(pycparser.c_ast.NodeVisitor):
def visit_FuncDef(self, node):
print("found function: %s" % node.decl.name)
#node.show()
FunctionVisitor().visit(ast)
yields
found function: add
found function: passthrough
To dig further you can also get parameter and return types.
Uncomment node.show() for more information from within the Abstract Syntax Tree (AST)
edit: I've written a library to do a lot of the heavy lifting on this : https://github.com/fragmuffin/prlsc/tree/main/lib/py (examples can be found in that project)

YES! there is a very clever native method to do it.
let's say you are using Python ctypes. put something like this in your C code:
1) in your C code:
#define PYEXPORT extern "C" __declspec(dllexport)
now put PYEXPORT above the function you want to export:
PYEXPORT
int myfunc(params){
2) After compiling, go back into Python and open your .c file, and parse it similar to this:
source1_ = open(cfile_name + '.c')
source1 = source1_.read()
source1_.close()
fn = source1.split('PYEXPORT')[-1].split('(')[0].split(' ')[1]
shell input: fn
shell output: 'myfunc'
3) Now here's the clever part: define a new function in a string:
a1 = """
global get_c_fn
def get_c_fn(dll):
func = dll."""
a2 = """
return func"""
a3 = a1 + fn + a2
print(a3)
global get_c_fn
def get_c_fn(dll):
func = dll.myfunc
return func
NOW do exec(a3) and it will turn that string into a function that you can use.
4) do the usual:
mydll = ctypes.CDLL(cfile_name + '.dll')
c_fn = get_cuda_fn(mydll)
c_fn.argtypes = func_params (an array of C-converted inputs you need)
c_fn( *[params] )
and there you have a python wrapper for a C script without having to modify ten different things every time something changes.

Internally ctypes uses functions provided by dynamic link library (dlopen/dlsym on unix, LoadLibrary/GetProcAddress on windows) to load library and find address of function specified by function name; and then use cffi library to pass parameter dynamically.
Problem is that the dynamic link library that ctypes depends on doesn't include function to list symbol from the shared library, that's why you can't list symbol by ctypes.
To do that, you have to use specific tools to dump elf file (readelf on unix) and pe file for dll (dumpbin on windows).

How can I use a DLL file from Python?

What is the easiest way to use a DLL file from within Python?
Specifically, how can this be done without writing any additional wrapper C++ code to expose the functionality to Python?
Native Python functionality is strongly preferred over using a third-party library.

For ease of use, ctypes is the way to go.
The following example of ctypes is from actual code I've written (in Python 2.5). This has been, by far, the easiest way I've found for doing what you ask.
import ctypes
# Load DLL into memory.
hllDll = ctypes.WinDLL ("c:\\PComm\\ehlapi32.dll")
# Set up prototype and parameters for the desired function call.
# HLLAPI
hllApiProto = ctypes.WINFUNCTYPE (
ctypes.c_int, # Return type.
ctypes.c_void_p, # Parameters 1 ...
ctypes.c_void_p,
ctypes.c_void_p,
ctypes.c_void_p) # ... thru 4.
hllApiParams = (1, "p1", 0), (1, "p2", 0), (1, "p3",0), (1, "p4",0),
# Actually map the call ("HLLAPI(...)") to a Python name.
hllApi = hllApiProto (("HLLAPI", hllDll), hllApiParams)
# This is how you can actually call the DLL function.
# Set up the variables and call the Python name with them.
p1 = ctypes.c_int (1)
p2 = ctypes.c_char_p (sessionVar)
p3 = ctypes.c_int (1)
p4 = ctypes.c_int (0)
hllApi (ctypes.byref (p1), p2, ctypes.byref (p3), ctypes.byref (p4))
The ctypes stuff has all the C-type data types (int, char, short, void*, and so on) and can pass by value or reference. It can also return specific data types although my example doesn't do that (the HLL API returns values by modifying a variable passed by reference).
In terms of the specific example shown above, IBM's EHLLAPI is a fairly consistent interface.
All calls pass four void pointers (EHLLAPI sends the return code back through the fourth parameter, a pointer to an int so, while I specify int as the return type, I can safely ignore it) as per IBM's documentation here. In other words, the C variant of the function would be:
int hllApi (void *p1, void *p2, void *p3, void *p4)
This makes for a single, simple ctypes function able to do anything the EHLLAPI library provides, but it's likely that other libraries will need a separate ctypes function set up per library function.
The return value from WINFUNCTYPE is a function prototype but you still have to set up more parameter information (over and above the types). Each tuple in hllApiParams has a parameter "direction" (1 = input, 2 = output and so on), a parameter name and a default value - see the ctypes doco for details
Once you have the prototype and parameter information, you can create a Python "callable" hllApi with which to call the function. You simply create the needed variable (p1 through p4 in my case) and call the function with them.

This page has a very simple example of calling functions from a DLL file.
Paraphrasing the details here for completeness:
It's very easy to call a DLL function in Python. I have a self-made DLL file with two functions: add and sub which take two arguments.
add(a, b) returns addition of two numbers
sub(a, b) returns substraction of two numbers
The name of the DLL file will be "demo.dll"
Program:
from ctypes import*
# give location of dll
mydll = cdll.LoadLibrary("C:\\demo.dll")
result1= mydll.add(10,1)
result2= mydll.sub(10,1)
print("Addition value:"+result1)
print("Substraction:"+result2)
Output:
Addition value:11
Substraction:9

Building a DLL and linking it under Python using ctypes
I present a fully worked example on how building a shared library and using it under Python by means of ctypes. I consider the Windows case and deal with DLLs. Two steps are needed:
Build the DLL using Visual Studio's compiler either from the command line or from the IDE;
Link the DLL under Python using ctypes.
The shared library
The shared library I consider is the following and is contained in the testDLL.cpp file. The only function testDLL just receives an int and prints it.
#include <stdio.h>
​
extern "C" {
​
__declspec(dllexport)
​
void testDLL(const int i) {
printf("%d\n", i);
}
​
} // extern "C"
Building the DLL from the command line
To build a DLL with Visual Studio from the command line run
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\vsdevcmd"
to set the include path and then run
cl.exe /D_USRDLL /D_WINDLL testDLL.cpp /MT /link /DLL /OUT:testDLL.dll
to build the DLL.
Building the DLL from the IDE
Alternatively, the DLL can be build using Visual Studio as follows:
File -> New -> Project;
Installed -> Templates -> Visual C++ -> Windows -> Win32 -> Win32Project;
Next;
Application type -> DLL;
Additional options -> Empty project (select);
Additional options -> Precompiled header (unselect);
Project -> Properties -> Configuration Manager -> Active solution platform: x64;
Project -> Properties -> Configuration Manager -> Active solution configuration: Release.
Linking the DLL under Python
Under Python, do the following
import os
import sys
from ctypes import *
lib = cdll.LoadLibrary('testDLL.dll')
lib.testDLL(3)

ctypes can be used to access dlls, here's a tutorial:
http://docs.python.org/library/ctypes.html#module-ctypes

Maybe with Dispatch:
from win32com.client import Dispatch
zk = Dispatch("zkemkeeper.ZKEM")
Where zkemkeeper is a registered DLL file on the system...
After that, you can access functions just by calling them:
zk.Connect_Net(IP_address, port)

ctypes will be the easiest thing to use but (mis)using it makes Python subject to crashing. If you are trying to do something quickly, and you are careful, it's great.
I would encourage you to check out Boost Python. Yes, it requires that you write some C++ code and have a C++ compiler, but you don't actually need to learn C++ to use it, and you can get a free (as in beer) C++ compiler from Microsoft.

If the DLL is of type COM library, then you can use pythonnet.
pip install pythonnet
Then in your python code, try the following
import clr
clr.AddReference('path_to_your_dll')
# import the namespace and class
from Namespace import Class
# create an object of the class
obj = Class()
# access functions return type using object
value = obj.Function(<arguments>)
then instantiate an object as per the class in the DLL, and access the methods within it.