I want to use functions in dll's via ctype. I can call the function without errors and even the error code of the function is 0 meanig function successfuly finished. But when I try to acces the result variable ist is empty.
I have been implemented the lookup in free pascal severeal years ago and would transfer it to python right now. The interface allow to access via cdel convention and I tied to reimplement in python 3.7.4 with ctypes now
The last working Pascal Prototype have been:
PROCEDURE pGetCallInfo(DriveInfo: pointer; ACall: pointer; AInfo: pointer;
var AErrorCode: SmallInt); pascal; external 'raccd32a.dll';
My best version in python have been the following:
from ctypes import *
callBookDLL = CDLL('raccd32a')
AInfo = create_string_buffer(400)
err = callBookDLL.cGetCallInfo("self.txt_CallBookPath.text()","DG1ATN",AInfo)
The result ist:
err
0
AInfo.value
b''
AInfo should contain a max. 400 char long stringbuffer with an result containing Name, Adress and so on.
As I have a second library I have to acces same way I search for my fault but I was not able to find it. I think my problem is the work with pointer and the type conversion.
I checked teh ctypes howto allready but I can noht solve this trouble.
Thanks a lot so far ...
Check [Python 3.Docs]: ctypes - A foreign function library for Python. It contains (almost) every piece of info that you need.
There are a number of problems:
ctypes doesn't support pascal calling convention, only cdecl and stdcall (applies to 32bit only). That means (after reading the manual) that you shouldn't use the p* functions, but the c* (or s*)
You didn't specify argtypes (and restype) for your function. This results in UB. Some effects of this:
[SO]: Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated (#CristiFati's answer)
[SO]: python ctypes issue on different OSes (#CristiFati's answer)
It is a procedure (a function that returns void). Anyway this is a minor one
Here's some sample code (of course it's blind, as I didn't test it):
#!/usr/bin/env python3
import sys
import ctypes
dll = ctypes.CDLL("raccd32a.dll")
cGetCallInfo = dll.cGetCallInfo
cGetCallInfo.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.POINTER(ctypes.c_short)]
cGetCallInfo.restype = None
ADriveInfo = self.txt_CallBookPath.text().encode()
#ADriveInfo = b"C:\\callbook2019\\" # Notice the double bkslashes
ACall = b"DG1ATN"
AInfo = ctypes.create_string_buffer(400)
result = ctypes.c_short(0)
cGetCallInfo(ADriveInfo, ACall, AInfo, ctypes.byref(result))
#EDIT0:
From the beginning, I wanted yo say that the 1st argument passed to the function doesn't make much sense. Then, there are problems regarding the 2nd one as well. According to the manual ([AMT-I]: TECHNICAL INFORMATION about RACCD32a.DLL (emphasis is mine)):
ADriveInfo, ACall and AInfo are pointers to zero-terminated strings. These
strings has to exist at the moment of calling xGetCallInfo. The calling
program is responsible for creating them. AInfo must be long enough to
comfort xGetCallInfo (at least 400 characters).
Note: "Length of AInfo" refers to the length of the string AInfo points at.
ADriveInfo and ACall are treated in the same manner for short.
In ADriveInfo the procedure expects the path to the CD ROM drive. Use
"G:\"
if "G:" designates the CD ROM drive with the callbook CD ROM.
Keep in mind that this information is a *must* and the calling program
has to know it.
Note: If the active directory on drive G: is not the root, ADriveInfo = "G:"
will lead to an error 3. So always use "G:\".
The calling program has to ensure that the length of ADriveInfo does not
exceed 80 characters.
ACall contains the call you are looking for, all letters in lower case,
no additional spaces etc. The calling program has to ensure that ACall is
not longer than 15 characters. However, there is no call longer than 6
characters in the database.
I'm trying to translate the following line of C code into ctypes. Here's a snippet from the C program I'm trying to translate:
pIfRow = (MIB_IF_ROW2 *) malloc(sizeof(MIB_IF_ROW2));
SecureZeroMemory((PVOID)pIfRow, sizeof(MIB_IF_ROW2));
(Note that MIB_IF_ROW2 is a struct, defined in Netioapi.h)
Anyway, I can translate the first line fine in ctypes, assuming MIB_IF_ROW2 has already been defined as a ctypes struct:
from ctypes import *
# Translate first line of C Code
buff = create_string_buffer(sizeof(MIB_IF_ROW2))
p_if_row = cast(buff, POINTER(MIB_IF_ROW2))
# Second Line... ?
But when I get to the second line, I get stuck. I can't find anything in the docs or online with a ctypes equivalent for the function. What is the best way to go about this?
SecureZeroMemory will just fill the memory you pass it with zeroes. You should get the exact same result with ZeroMemory/memset or a plain loop in python. The thing that makes it "secure" is that it is not supposed to be optimized away by the compiler (when programming at a lower level like C/C++).
Using it on memory you just malloc'ed is not its intended purpose (not harmful though), it is supposed to be used like this:
char password[100];
AskUserForPassword(password);
DoSomething(password);
SecureZeroMemory(password, sizeof(password)); // Make sure password is no longer visible in memory in case the application is paged out or creates a memory dump in a crash
I have a working python 2.7 program that calls a DLL. I am trying to port the script to python 3.2. The DLL call seems to work (i.e. there is no error upon calling) but the returned data does not make sense.
Just in case it could be useful:
- The call takes three arguments: two int (input) and a pointer to a ushort array (output).
I have tried using both python and numpy arrays without success.
Can anyone enumerate the differences between Python 2.7 and 3.2 respecting ctypes?
Thanks in advance
EDIT
Here is some example code. The DLL is propietary so I do not have the code. But I do have the C header:
void example (int width, int height, unsigned short* pointer)
The python code is:
width, height = 40, 100
imagearray = np.zeros((width,height), dtype=np.dtype(np.ushort))
image = np.ascontiguousarray(imagearray)
ptrimage = image.ctypes.data_as(ct.POINTER(ct.c_ushort))
DLL.example(width, height, ptrimage)
This works in python 2.7 but not in 3.2.
EDIT 2
If the changes in ctypes are only those pointed out by Cedric, it does not make sense that python 3.2 will not work. So looking again at the code, I have found that there is a preparation function called before the function that I am mentioning. The signature is:
void prepare(char *table)
In python, I am calling by:
table = str(aNumber)
DLL.prepare(table)
Is it possible that the problem is due to the change in the Python string handling?
In Python 2.7, strings are byte-strings by default. In Python 3.x, they are unicode by default. Try explicitly making your string a byte string using .encode('ascii') before handing it to DLL.prepare.
Edit:
#another way of saying table=str(aNumber).encode('ascii')
table = bytes(str(aNumber), 'ascii')
DLL.prepare(table)
In our case, we had code looking like:
addr = clib.some_function()
data = some_struct.from_address(addr)
This worked in python 2 but not in 3. The reason turned out not to be any difference in ctypes, but rather a change in memory layout that unmasked a bug in the code above. In python 2, the address returned was always (by chance) small enough to fit inside a C int (32-bit), which is the default return type for all ctypes function calls. In python 3, the addresses were almost always too large, which caused the pointer address to become corrupted as it was coerced to int.
The solution is to set the function restype to a 64-bit integer type to ensure that it can accommodate the whole address, like:
clib.some_function.restype = c_longlong
or:
clib.some_function.restype = POINTER(some_struct)
According to the python documentation, the only changes between 2.7 and 3.2 is here
A new type, ctypes.c_ssize_t represents the C ssize_t datatype.
In 2.7, there was some other modifications introduced :
The ctypes module now always converts None to a C NULL pointer for
arguments declared as pointers. (Changed by Thomas Heller; issue
4606.) The underlying libffi library has been updated to version
3.0.9, containing various fixes for different platforms. (Updated by
Matthias Klose; issue 8142.)
I'm not sure it will explain the cause of your problem...
Alright, so a couple days ago I decided to try and write a primitive wrapper for the PARI library. Ever since then I've been playing with ctypes library in loading the dll and accessing the functions contained using code similar to the following:
from ctypes import *
libcyg=CDLL("<path/cygwin1.dll") #It needs cygwin to be loaded. Not sure why.
pari=CDLL("<path>/libpari-gmp-2.4.dll")
print pari.fibo #fibonacci function
#prints something like "<_FuncPtr object at 0x00BA5828>"
So the functions are there and they can potentially be accessed, but I always receive an access violation no matter what I try. For example:
pari.fibo(5) #access violation
pari.fibo(c_int(5)) #access violation
pari.fibo.argtypes = [c_long] #setting arguments manually
pari.fibo.restype = long #set the return type
pari.fibo(byref(c_int(5))) #access violation reading 0x04 consistently
and any variation on that, including setting argtypes to receive pointers.
The Pari .dll is written in C and the fibonacci function's syntax within the library is GEN fibo(long x).
Could it be the return type that's causing these errors, as it is not a standard int or long but a GEN type, which is unique to the PARI library? Any help would be appreciated. If anyone is able to successfully load the library and use ANY function from within python, please tell; I've been at this for hours now.
EDIT: Seems as though I was simply forgetting to initialize the library. After a quick pari.pari_init(4000000,500000) it stopped erroring. Now my problem lies in the in the fact that it returns a GEN object; which is fine, but whenever I try to reference the address to which it points, it's always 33554435, which I presume is still an address. I'm trying further commands and I'll update if I succeed in getting the correct value of something.
You have two problems here, one give fibo the correct return type and two convert the GEN return type to the value you are looking for.
Poking around the source code a bit, you'll find that GEN is defined as a pointer to a long. Also, at looks like the library provides some converting/printing GENs. I focused in on GENtostr since it would probably be safer for all the pari functions.
import cytpes
pari = ctypes.CDLL("./libpari.so.2.3.5") #I did this under linux
pari.fibo.restype = ctypes.POINTER(ctypes.c_long)
pari.GENtostr.restype = ctypes.POINTER(ctypes.c_char)
pari.pari_init(4000000,500000)
x = pari.fibo(100)
y = pari.GENtostr(x)
ctypes.string_at(y)
Results in:
'354224848179261915075'
I want to extend a large C project with some new functionality, but I really want to write it in Python. Basically, I want to call Python code from C code. However, Python->C wrappers like SWIG allow for the OPPOSITE, that is writing C modules and calling C from Python.
I'm considering an approach involving IPC or RPC (I don't mind having multiple processes); that is, having my pure-Python component run in a separate process (on the same machine) and having my C project communicate with it by writing/reading from a socket (or unix pipe). my python component can read/write to socket to communicate. Is that a reasonable approach? Is there something better? Like some special RPC mechanism?
Thanks for the answer so far - however, i'd like to focus on IPC-based approaches since I want to have my Python program in a separate process as my C program. I don't want to embed a Python interpreter. Thanks!
I recommend the approaches detailed here. It starts by explaining how to execute strings of Python code, then from there details how to set up a Python environment to interact with your C program, call Python functions from your C code, manipulate Python objects from your C code, etc.
EDIT: If you really want to go the route of IPC, then you'll want to use the struct module or better yet, protlib. Most communication between a Python and C process revolves around passing structs back and forth, either over a socket or through shared memory.
I recommend creating a Command struct with fields and codes to represent commands and their arguments. I can't give much more specific advice without knowing more about what you want to accomplish, but in general I recommend the protlib library, since it's what I use to communicate between C and Python programs (disclaimer: I am the author of protlib).
Have you considered just wrapping your python application in a shell script and invoking it from within your C application?
Not the most elegant solution, but it is very simple.
See the relevant chapter in the manual: http://docs.python.org/extending/
Essentially you'll have to embed the python interpreter into your program.
I haven't used an IPC approach for Python<->C communication but it should work pretty well. I would have the C program do a standard fork-exec and use redirected stdin and stdout in the child process for the communication. A nice text-based communication will make it very easy to develop and test the Python program.
If I had decided to go with IPC, I'd probably splurge with XML-RPC -- cross-platform, lets you easily put the Python server project on a different node later if you want, has many excellent implementations (see here for many, including C and Python ones, and here for the simple XML-RPC server that's part the Python standard library -- not as highly scalable as other approaches but probably fine and convenient for your use case).
It may not be a perfect IPC approach for all cases (or even a perfect RPC one, by all means!), but the convenience, flexibility, robustness, and broad range of implementations outweigh a lot of minor defects, in my opinion.
This seems quite nice http://thrift.apache.org/, there is even a book about it.
Details:
The Apache Thrift software framework, for scalable cross-language
services development, combines a software stack with a code generation
engine to build services that work efficiently and seamlessly between
C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa,
JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages.
I've used the "standard" approach of Embedding Python in Another Application. But it's complicated/tedious. Each new function in Python is painful to implement.
I saw an example of Calling PyPy from C. It uses CFFI to simplify the interface but it requires PyPy, not Python. Read and understand this example first, at least at a high level.
I modified the C/PyPy example to work with Python. Here's how to call Python from C using CFFI.
My example is more complicated because I implemented three functions in Python instead of one. I wanted to cover additional aspects of passing data back and forth.
The complicated part is now isolated to passing the address of api to Python. That only has to be implemented once. After that it's easy to add new functions in Python.
interface.h
// These are the three functions that I implemented in Python.
// Any additional function would be added here.
struct API {
double (*add_numbers)(double x, double y);
char* (*dump_buffer)(char *buffer, int buffer_size);
int (*release_object)(char *obj);
};
test_cffi.c
//
// Calling Python from C.
// Based on Calling PyPy from C:
// http://doc.pypy.org/en/latest/embedding.html#more-complete-example
//
#include <stdio.h>
#include <assert.h>
#include "Python.h"
#include "interface.h"
struct API api; /* global var */
int main(int argc, char *argv[])
{
int rc;
// Start Python interpreter and initialize "api" in interface.py using
// old style "Embedding Python in Another Application":
// https://docs.python.org/2/extending/embedding.html#embedding-python-in-another-application
PyObject *pName, *pModule, *py_results;
PyObject *fill_api;
#define PYVERIFY(exp) if ((exp) == 0) { fprintf(stderr, "%s[%d]: ", __FILE__, __LINE__); PyErr_Print(); exit(1); }
Py_SetProgramName(argv[0]); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString(
"import sys;"
"sys.path.insert(0, '.')" );
PYVERIFY( pName = PyString_FromString("interface") )
PYVERIFY( pModule = PyImport_Import(pName) )
Py_DECREF(pName);
PYVERIFY( fill_api = PyObject_GetAttrString(pModule, "fill_api") )
// "k" = [unsigned long],
// see https://docs.python.org/2/c-api/arg.html#c.Py_BuildValue
PYVERIFY( py_results = PyObject_CallFunction(fill_api, "k", &api) )
assert(py_results == Py_None);
// Call Python function from C using cffi.
printf("sum: %f\n", api.add_numbers(12.3, 45.6));
// More complex example.
char buffer[20];
char * result = api.dump_buffer(buffer, sizeof buffer);
assert(result != 0);
printf("buffer: %s\n", result);
// Let Python perform garbage collection on result now.
rc = api.release_object(result);
assert(rc == 0);
// Close Python interpreter.
Py_Finalize();
return 0;
}
interface.py
import cffi
import sys
import traceback
ffi = cffi.FFI()
ffi.cdef(file('interface.h').read())
# Hold references to objects to prevent garbage collection.
noGCDict = {}
# Add two numbers.
# This function was copied from the PyPy example.
#ffi.callback("double (double, double)")
def add_numbers(x, y):
return x + y
# Convert input buffer to repr(buffer).
#ffi.callback("char *(char*, int)")
def dump_buffer(buffer, buffer_len):
try:
# First attempt to access data in buffer.
# Using the ffi/lib objects:
# http://cffi.readthedocs.org/en/latest/using.html#using-the-ffi-lib-objects
# One char at time, Looks inefficient.
#data = ''.join([buffer[i] for i in xrange(buffer_len)])
# Second attempt.
# FFI Interface:
# http://cffi.readthedocs.org/en/latest/using.html#ffi-interface
# Works but doc says "str() gives inconsistent results".
#data = str( ffi.buffer(buffer, buffer_len) )
# Convert C buffer to Python str.
# Doc says [:] is recommended instead of str().
data = ffi.buffer(buffer, buffer_len)[:]
# The goal is to return repr(data)
# but it has to be converted to a C buffer.
result = ffi.new('char []', repr(data))
# Save reference to data so it's not freed until released by C program.
noGCDict[ffi.addressof(result)] = result
return result
except:
print >>sys.stderr, traceback.format_exc()
return ffi.NULL
# Release object so that Python can reclaim the memory.
#ffi.callback("int (char*)")
def release_object(ptr):
try:
del noGCDict[ptr]
return 0
except:
print >>sys.stderr, traceback.format_exc()
return 1
def fill_api(ptr):
global api
api = ffi.cast("struct API*", ptr)
api.add_numbers = add_numbers
api.dump_buffer = dump_buffer
api.release_object = release_object
Compile:
gcc -o test_cffi test_cffi.c -I/home/jmudd/pgsql-native/Python-2.7.10.install/include/python2.7 -L/home/jmudd/pgsql-native/Python-2.7.10.install/lib -lpython2.7
Execute:
$ test_cffi
sum: 57.900000
buffer: 'T\x9e\x04\x08\xa8\x93\xff\xbf]\x86\x04\x08\x00\x00\x00\x00\x00\x00\x00\x00'
$
Few tips for binding it with Python 3
file() not supported, use open()
ffi.cdef(open('interface.h').read())
PyObject* PyStr_FromString(const char *u)
Create a PyStr from a UTF-8 encoded null-terminated character buffer.
Python 2: PyString_FromString
Python 3: PyUnicode_FromString
Change to: PYVERIFY( pName = PyUnicode_FromString("interface") )
Program name
wchar_t *name = Py_DecodeLocale(argv[0], NULL);
Py_SetProgramName(name);
for compiling
gcc cc.c -o cc -I/usr/include/python3.6m -I/usr/include/x86_64-linux-gnu/python3.6m -lpython3.6m
I butchered dump def .. maybe it will give some ideas
def get_prediction(buffer, buffer_len):
try:
data = ffi.buffer(buffer, buffer_len)[:]
result = ffi.new('char []', data)
print('\n I am doing something here here........',data )
resultA = ffi.new('char []', b"Failed") ### New message
##noGCDict[ffi.addressof(resultA)] = resultA
return resultA
except:
print >>sys.stderr, traceback.format_exc()
return ffi.NULL
}
Hopefully it will help and save you some time
apparently Python need to be able to compile to win32 dll, it will solve the problem
In such a way that converting c# code to win32 dlls will make it usable by any development tool