I have the task of "wrapping" a c library into a python class. The docs are incredibly vague on this matter. It seems they expect only advanced python users would implement ctypes.
Some step by step help would be wonderful.
So I have my c library. What do I do? What files do I put where? How do I import the library? I read that there might be a way to "auto wrap" to Python?
(By the way I did the ctypes tutorial on python.net and it doesn't work. Meaning I'm thinking they are assuming I should be able to fill in the rest of the steps.)
In fact this is the error I get with their code:
File "importtest.py", line 1
>>> from ctypes import *
SyntaxError: invalid syntax
I could really use some step by step help on this!
Here's a quick and dirty ctypes tutorial.
First, write your C library. Here's a simple Hello world example:
testlib.c
#include <stdio.h>
void myprint(void);
void myprint()
{
printf("hello world\n");
}
Now compile it as a shared library (mac fix found here):
$ gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c
# or... for Mac OS X
$ gcc -shared -Wl,-install_name,testlib.so -o testlib.so -fPIC testlib.c
Then, write a wrapper using ctypes:
testlibwrapper.py
import ctypes
testlib = ctypes.CDLL('/full/path/to/testlib.so')
testlib.myprint()
Now execute it:
$ python testlibwrapper.py
And you should see the output
Hello world
$
If you already have a library in mind, you can skip the non-python part of the tutorial. Make sure ctypes can find the library by putting it in /usr/lib or another standard directory. If you do this, you don't need to specify the full path when writing the wrapper. If you choose not to do this, you must provide the full path of the library when calling ctypes.CDLL().
This isn't the place for a more comprehensive tutorial, but if you ask for help with specific problems on this site, I'm sure the community would help you out.
PS: I'm assuming you're on Linux because you've used ctypes.CDLL('libc.so.6'). If you're on another OS, things might change a little bit (or quite a lot).
The answer by Chinmay Kanchi is excellent but I wanted an example of a function which passes and returns a variables/arrays to a C++ code. I though I'd include it here in case it is useful to others.
Passing and returning an integer
The C++ code for a function which takes an integer and adds one to the returned value,
extern "C" int add_one(int i)
{
return i+1;
}
Saved as file test.cpp, note the required extern "C" (this can be removed for C code).
This is compiled using g++, with arguments similar to Chinmay Kanchi answer,
g++ -shared -o testlib.so -fPIC test.cpp
The Python code uses load_library from the numpy.ctypeslib assuming the path to the shared library in the same directory as the Python script,
import numpy.ctypeslib as ctl
import ctypes
libname = 'testlib.so'
libdir = './'
lib=ctl.load_library(libname, libdir)
py_add_one = lib.add_one
py_add_one.argtypes = [ctypes.c_int]
value = 5
results = py_add_one(value)
print(results)
This prints 6 as expected.
Passing and printing an array
You can also pass arrays as follows, for a C code to print the element of an array,
extern "C" void print_array(double* array, int N)
{
for (int i=0; i<N; i++)
cout << i << " " << array[i] << endl;
}
which is compiled as before and the imported in the same way. The extra Python code to use this function would then be,
import numpy as np
py_print_array = lib.print_array
py_print_array.argtypes = [ctl.ndpointer(np.float64,
flags='aligned, c_contiguous'),
ctypes.c_int]
A = np.array([1.4,2.6,3.0], dtype=np.float64)
py_print_array(A, 3)
where we specify the array, the first argument to print_array, as a pointer to a Numpy array of aligned, c_contiguous 64 bit floats and the second argument as an integer which tells the C code the number of elements in the Numpy array. This then printed by the C code as follows,
1.4
2.6
3.0
Firstly: The >>> code you see in python examples is a way to indicate that it is Python code. It's used to separate Python code from output. Like this:
>>> 4+5
9
Here we see that the line that starts with >>> is the Python code, and 9 is what it results in. This is exactly how it looks if you start a Python interpreter, which is why it's done like that.
You never enter the >>> part into a .py file.
That takes care of your syntax error.
Secondly, ctypes is just one of several ways of wrapping Python libraries. Other ways are SWIG, which will look at your Python library and generate a Python C extension module that exposes the C API. Another way is to use Cython.
They all have benefits and drawbacks.
SWIG will only expose your C API to Python. That means you don't get any objects or anything, you'll have to make a separate Python file doing that. It is however common to have a module called say "wowza" and a SWIG module called "_wowza" that is the wrapper around the C API. This is a nice and easy way of doing things.
Cython generates a C-Extension file. It has the benefit that all of the Python code you write is made into C, so the objects you write are also in C, which can be a performance improvement. But you'll have to learn how it interfaces with C so it's a little bit extra work to learn how to use it.
ctypes have the benefit that there is no C-code to compile, so it's very nice to use for wrapping standard libraries written by someone else, and already exists in binary versions for Windows and OS X.
Related
I have a library which is a python wrapper for a C/C++ library, which needs to be imported to Python in Linux. Let's call the library abc.so. This library is dependent on another C/C++ library: xyz. Both these libraries used to have/have facilities that are dependent on Windows, Borland compiler or similar compilers. I am able to successfully build abc.so, after fixing some of the windows compiler related issues. However, I cannot import it to my python code. I receive the error:
ImportError: /usr/local/lib/abc.so: undefined symbol: _strnicmp
or a variant of this. I tried various import methods involving packages like ctpes, os, sys and flags like RTLD_LAZY, RTLD_GLOBAL, RTLD_NOW under the assumption that the method of import will fix this problem. However, none of them worked. This answer: undefined reference to stricmp (and the comment above) suggests that strnicmp should be replaced. It also points out that this is a link time error. However, I have not been able to identify part of these libraries expecting an implementation of strnicmp. What would be a good approach to find the source of this issue? Also, should I be trying some alternative path to fix this issue?
The stricmp() and strnicmp() functions are specific to Windows.
POSIX (Linux) uses the <strings.h> header and strcasecmp() and strncasecmp().
You can write a simple cover function or change the calls via a macro. For the cover function, you'd want to use appropriate conditional compilation (#ifdef SOMETHING / #endif — or maybe #ifndef SOMETHING / #endif). You might use an inline function if it only appears in one file. If it appears in many files, you may want a regular function, even though it's a smidgeon less efficient. If there's a convenient header included by all the files that you can modify to add (conditionally) the static inline function definition, that can work well — it's also about the only time you normally add a function body to a header file.
static inline int strnicmp(const char *s1, const char *s2, size_t len)
{
return strncasecmp(s1, s2, len);
}
or
#undef strnicmp
#define strnicmp(s1, s2, len) strncasecmp(s1, s2, len)
I am maintaining some DLL for Windows that is delivered in 32 and 64 bit versions. It has been build with Visual Studio 2010, but for some reasons I recently changed to MinGW (7.3, also tried 9.2). For the 64 bit version everything seems to be ok, but the 32 bit version doesn't work when loaded in Python or LabView; both can load the DLL but don't find the functions. A simplified example:
// mydll.h
extern "C" __declspec(dllexport) int __stdcall myFunc( int x );
// mydll.cpp
#include "mydll.h"
int _stdcall myFunc( int x )
{
return 2*x;
}
# test.py
import ctypes
lib = ctypes.windll.LoadLibrary('.\mydll.dll')
lib.myFunc.argtypes = [ctypes.c_int32]
print(lib.myFunc(1))
Commands, using MinGW 7.3 and Python 3.7.2 (32 bit):
> g++ -shared -o mydll.dll mydll.cpp
> python test.py
Traceback (most recent call last):
File "test.py", line 3, in <module>
print(lib.myFunc(1))
...
AttributeError: function 'myFunc' not found
The same code runs and prints "2" when compiled with MSVC-32bit or with MinGW-64bit.
Why doesn't it work with MinGW-32bit? I suspect it's an issue of the C name mangling: The exported symbol names of the DLL as reported by "dumpbin /exports mydll.dll" are _myFunc#4 (MSVC) and myFunc#4 (MinGW) - MinGW's output misses a leading underscore. As far as I understand, Python and LV expect the mangled names in the MSVC style (and hide the mangling in their user interface).
I have tried a couple of MinGW Options such as -Wl,-fleading_underscore, -Wl,--add-stdcall-alias, -Wl,--enable-stdcall-fixup to achieve compatible export names, but without success. One would expect that this is a common problem as the combination C/Python isn't really exotic, but I can't find any discussion of the problem on the net. What I am doing wrong?
I ran into the same problem. Theoretically dlltool might be able to help somehow, with its --add-stdcall-underscore option, but I couldn't get that to work.. it sucks that MinGW's ld doesn't have that option.
Anyway, I found another solution: Create your own mydll.def file with aliases, like
LIBRARY "mydll.dll"
EXPORTS
myFunc#4
_myFunc#4 = myFunc#4
myFunc2#42
_myFunc2#42 = myFunc2#42
(Using the gendef tool on your current state of the DLL could help to get an initial version of that .def, but it was a bit buggy for me, generating both myfunc#4 and myfunc#4#4 lines)
Then build the DLL with g++ -shared -o mydll.dll mydll.def mydll.cpp
You could also add -Wl,--add-stdcall-alias to also automatically create aliases without the #4 at the end (might be necessary to make GetProcAddress(myDllHandle, "myFunc"); work?), and it may be a good idea to use --static-libgcc so your DLL works without bundling libgcc_s_dw2-1.dll.
I have a cpp file that compiles fine with g++ by using the shell:
extern "C"{
#include <quadmath.h>
}
inline char* print( const __float128& val)
{
char* buffer = new char[128];
quadmath_snprintf(buffer,128,"%+.34Qe", val);
return buffer;
}
int main(){
__float128 a = 1.0;
print(a);
return 0;
}
However, when I try to compile it via a python scrit, it fails with the following error:
"undefined reference to quadmath_snprintf"
Here the code of the python script:
import commands
import string
import os
(status, output) = commands.getstatusoutput("(g++ test/*.c -O3 -lquadmath -m64)")
Any idea how to solve this? Thanks.
When you open a shell a whole of stuff is silently initialized for you, and most important for your issue, environment variables are set. What you most likely miss is the definition of LIBRARY_PATH, which is the variable used by the linker to look for libraries matching the ones you instruct it to link using the -lNAME flags.
What the linker needs is a list of directories where it will search for files matching libNAME.{a,so}. You can also pass these directories directly using the -L flag, but in general, you should probably try to use a program like CMake, Make or any other build tool.
This will give you access to commands like find_package and target_link_libraries (CMake), to find, respectively add libraries to your build targets, instead of having to maintain your python to compile your stuff.
Is there any way to know which functions are exported from the dll through python foreign function library ctypes?
And if possible to know details about the exported functions through ctypes.
If yes, could someone provide a snippet of code?
If you are on Linux, there is a handy utility nm to list the content of a shared library (there is always a handy utility on Linux, especially for C stuff).
Here is the question about it.
You use it with the -D flag: nm -D ./libMyLib.so
I don't think ctypes offers this functionality. On Windows with visual studio:
DUMPBIN -EXPORTS XXX.DLL
Or for mingw on windows:
objdump -p XXX.dll
In general, this is not possible, because, again in general, dynamically loaded libraries do not carry the meta-information you require. It may be possible to obtain that information in certain special cases through system-specific ways, but ctypes itself does not fetch that information. You can record such info via ctypes (see e.g. the restype and argtypes attributes of function pointers), but only after you have obtained it by different means.
The below approach worked for both Windows and Ubuntu. For Windows, Cygwin is required.
Suppose, there is a c file like below which name is test.c.
int func1(int a, int b){
return a + b;
}
int func2(int a, int b){
return a - b;
}
And the above c codes were compiled to test.dll file with the below commands:
gcc -shared -Wl,-soname,adder -o test.dll -fPIC test.c
And the below Python script finds which functions of the test.dll can be used by Python.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from subprocess import Popen, PIPE
out = Popen(
args="nm ./test.dll",
shell=True,
stdout=PIPE
).communicate()[0].decode("utf-8")
attrs = [
i.split(" ")[-1].replace("\r", "")
for i in out.split("\n") if " T " in i
]
from ctypes import CDLL
functions = [i for i in attrs if hasattr(CDLL("./test.dll"), i)]
print(functions)
The output I got in Windows is as below:
['func1', 'func2']
The output I got in Ubuntu is as below:
['_fini', 'func1', 'func2', '_init']
The items of the above list are objects of _FuncPtr class.
#Mark's answer uses Visual Studio tools.
On windows you can also use Dependency Walker to get the function names of dll exports.
Sometimes names are mangled and can't be used as a valid python function name.
You can use getattr to get a handle to mangled functions, e.g:
mylib = ctypes.cdll('mylib.dll')
my_func = getattr(mylib, '_my_func#0')
my_func()
If you've also got the source for said library, and you're looking for a fully automated all-python way, you could use pycparser
for the file: prog.c
typedef short int ret_t;
typedef short int param_t;
ret_t add(param_t a, param_t b) {
return (ret_t)(a + b);
}
ret_t passthrough(ret_t (* func)(param_t a, param_t b), param_t a, param_t b) {
// parameter intentionally altered.
// if this isn't done, compiler will deem entire function redundant
return func(a, b + 1);
}
compiling with gcc
gcc -I. -E ./prog.c > prog-preproc.c
gives us the pre-processed c file: prog-preproc.c
then in python:
import pycparser
parser = pycparser.c_parser.CParser()
with open('prog-preproc.c', 'r') as fh:
ast = parser.parse(fh.read())
class FunctionVisitor(pycparser.c_ast.NodeVisitor):
def visit_FuncDef(self, node):
print("found function: %s" % node.decl.name)
#node.show()
FunctionVisitor().visit(ast)
yields
found function: add
found function: passthrough
To dig further you can also get parameter and return types.
Uncomment node.show() for more information from within the Abstract Syntax Tree (AST)
edit: I've written a library to do a lot of the heavy lifting on this : https://github.com/fragmuffin/prlsc/tree/main/lib/py (examples can be found in that project)
YES! there is a very clever native method to do it.
let's say you are using Python ctypes. put something like this in your C code:
1) in your C code:
#define PYEXPORT extern "C" __declspec(dllexport)
now put PYEXPORT above the function you want to export:
PYEXPORT
int myfunc(params){
2) After compiling, go back into Python and open your .c file, and parse it similar to this:
source1_ = open(cfile_name + '.c')
source1 = source1_.read()
source1_.close()
fn = source1.split('PYEXPORT')[-1].split('(')[0].split(' ')[1]
shell input: fn
shell output: 'myfunc'
3) Now here's the clever part: define a new function in a string:
a1 = """
global get_c_fn
def get_c_fn(dll):
func = dll."""
a2 = """
return func"""
a3 = a1 + fn + a2
print(a3)
global get_c_fn
def get_c_fn(dll):
func = dll.myfunc
return func
NOW do exec(a3) and it will turn that string into a function that you can use.
4) do the usual:
mydll = ctypes.CDLL(cfile_name + '.dll')
c_fn = get_cuda_fn(mydll)
c_fn.argtypes = func_params (an array of C-converted inputs you need)
c_fn( *[params] )
and there you have a python wrapper for a C script without having to modify ten different things every time something changes.
Internally ctypes uses functions provided by dynamic link library (dlopen/dlsym on unix, LoadLibrary/GetProcAddress on windows) to load library and find address of function specified by function name; and then use cffi library to pass parameter dynamically.
Problem is that the dynamic link library that ctypes depends on doesn't include function to list symbol from the shared library, that's why you can't list symbol by ctypes.
To do that, you have to use specific tools to dump elf file (readelf on unix) and pe file for dll (dumpbin on windows).
What is the easiest way to use a DLL file from within Python?
Specifically, how can this be done without writing any additional wrapper C++ code to expose the functionality to Python?
Native Python functionality is strongly preferred over using a third-party library.
For ease of use, ctypes is the way to go.
The following example of ctypes is from actual code I've written (in Python 2.5). This has been, by far, the easiest way I've found for doing what you ask.
import ctypes
# Load DLL into memory.
hllDll = ctypes.WinDLL ("c:\\PComm\\ehlapi32.dll")
# Set up prototype and parameters for the desired function call.
# HLLAPI
hllApiProto = ctypes.WINFUNCTYPE (
ctypes.c_int, # Return type.
ctypes.c_void_p, # Parameters 1 ...
ctypes.c_void_p,
ctypes.c_void_p,
ctypes.c_void_p) # ... thru 4.
hllApiParams = (1, "p1", 0), (1, "p2", 0), (1, "p3",0), (1, "p4",0),
# Actually map the call ("HLLAPI(...)") to a Python name.
hllApi = hllApiProto (("HLLAPI", hllDll), hllApiParams)
# This is how you can actually call the DLL function.
# Set up the variables and call the Python name with them.
p1 = ctypes.c_int (1)
p2 = ctypes.c_char_p (sessionVar)
p3 = ctypes.c_int (1)
p4 = ctypes.c_int (0)
hllApi (ctypes.byref (p1), p2, ctypes.byref (p3), ctypes.byref (p4))
The ctypes stuff has all the C-type data types (int, char, short, void*, and so on) and can pass by value or reference. It can also return specific data types although my example doesn't do that (the HLL API returns values by modifying a variable passed by reference).
In terms of the specific example shown above, IBM's EHLLAPI is a fairly consistent interface.
All calls pass four void pointers (EHLLAPI sends the return code back through the fourth parameter, a pointer to an int so, while I specify int as the return type, I can safely ignore it) as per IBM's documentation here. In other words, the C variant of the function would be:
int hllApi (void *p1, void *p2, void *p3, void *p4)
This makes for a single, simple ctypes function able to do anything the EHLLAPI library provides, but it's likely that other libraries will need a separate ctypes function set up per library function.
The return value from WINFUNCTYPE is a function prototype but you still have to set up more parameter information (over and above the types). Each tuple in hllApiParams has a parameter "direction" (1 = input, 2 = output and so on), a parameter name and a default value - see the ctypes doco for details
Once you have the prototype and parameter information, you can create a Python "callable" hllApi with which to call the function. You simply create the needed variable (p1 through p4 in my case) and call the function with them.
This page has a very simple example of calling functions from a DLL file.
Paraphrasing the details here for completeness:
It's very easy to call a DLL function in Python. I have a self-made DLL file with two functions: add and sub which take two arguments.
add(a, b) returns addition of two numbers
sub(a, b) returns substraction of two numbers
The name of the DLL file will be "demo.dll"
Program:
from ctypes import*
# give location of dll
mydll = cdll.LoadLibrary("C:\\demo.dll")
result1= mydll.add(10,1)
result2= mydll.sub(10,1)
print("Addition value:"+result1)
print("Substraction:"+result2)
Output:
Addition value:11
Substraction:9
Building a DLL and linking it under Python using ctypes
I present a fully worked example on how building a shared library and using it under Python by means of ctypes. I consider the Windows case and deal with DLLs. Two steps are needed:
Build the DLL using Visual Studio's compiler either from the command line or from the IDE;
Link the DLL under Python using ctypes.
The shared library
The shared library I consider is the following and is contained in the testDLL.cpp file. The only function testDLL just receives an int and prints it.
#include <stdio.h>
extern "C" {
__declspec(dllexport)
void testDLL(const int i) {
printf("%d\n", i);
}
} // extern "C"
Building the DLL from the command line
To build a DLL with Visual Studio from the command line run
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\vsdevcmd"
to set the include path and then run
cl.exe /D_USRDLL /D_WINDLL testDLL.cpp /MT /link /DLL /OUT:testDLL.dll
to build the DLL.
Building the DLL from the IDE
Alternatively, the DLL can be build using Visual Studio as follows:
File -> New -> Project;
Installed -> Templates -> Visual C++ -> Windows -> Win32 -> Win32Project;
Next;
Application type -> DLL;
Additional options -> Empty project (select);
Additional options -> Precompiled header (unselect);
Project -> Properties -> Configuration Manager -> Active solution platform: x64;
Project -> Properties -> Configuration Manager -> Active solution configuration: Release.
Linking the DLL under Python
Under Python, do the following
import os
import sys
from ctypes import *
lib = cdll.LoadLibrary('testDLL.dll')
lib.testDLL(3)
ctypes can be used to access dlls, here's a tutorial:
http://docs.python.org/library/ctypes.html#module-ctypes
Maybe with Dispatch:
from win32com.client import Dispatch
zk = Dispatch("zkemkeeper.ZKEM")
Where zkemkeeper is a registered DLL file on the system...
After that, you can access functions just by calling them:
zk.Connect_Net(IP_address, port)
ctypes will be the easiest thing to use but (mis)using it makes Python subject to crashing. If you are trying to do something quickly, and you are careful, it's great.
I would encourage you to check out Boost Python. Yes, it requires that you write some C++ code and have a C++ compiler, but you don't actually need to learn C++ to use it, and you can get a free (as in beer) C++ compiler from Microsoft.
If the DLL is of type COM library, then you can use pythonnet.
pip install pythonnet
Then in your python code, try the following
import clr
clr.AddReference('path_to_your_dll')
# import the namespace and class
from Namespace import Class
# create an object of the class
obj = Class()
# access functions return type using object
value = obj.Function(<arguments>)
then instantiate an object as per the class in the DLL, and access the methods within it.