https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-ntqueryinformationfile?redirectedfrom=MSDN
How can I call the above kernel method in python? I found an example on another stackoverflow post: Winapi: Get the process which has specific handle of a file
The answer on this other post is essentially what I want to do, but in python. The goal is to be able to get a list of processes which currently are accessing/locking a file. This NtQueryInformationFile method seems to be exactly what I want. I know this can be done with ctypes, but I am not familiar or comfortable enough with ctypes to do this myself. How can I do this?
If there's no available wrapper for the function, you'll need to call the function yourself using ctypes.
The dlls windows uses are exposed through ctypes.windll, with cytpes.windll.ntdll being the one that exposes the function you need.
To help python convert arguments, it's usually a good idea to specify the function's argument and return types, which can be done through the argtypes and restype attributes on the function object, like so:
function = cytpes.windll.ntdll.NtQueryInformationFile
function.argtypes = [ctypes.wintypes.HANDLE, ...]
function.restype = ctypes.c_long
ctypes exposes the common window types in the ctypes.wintypes module, though for most structures like the PIO_STATUS_BLOCK in your function you'll need to define the struct yourself and add it to the argument list to use it properly. In case it's optional a void pointer and passing it None will suffice.
Also, do mind that windows handles are not the file descriptors that python exposes, to convert to/from them you can use the ..._osfhandle functions from the msvcrt module
Related
To avoid a memory leak in the sutuation described below, I would like to call Py_DecRef directly from Python. Is there a way to do this? Is there a better solution for this problem?
I am using ctypes to interface my Python code to a C library for which I do not have the code. The C library was not written for Python, so it doesn't know anything about Python objects.
The C library uses two callbacks: the first creates an object and returns a void* pointer to it, and the second gets the pointer as parameter and is supposed to destroy it. In the C header files, the types of these callback functions are defined as follows:
typedef void* (*CreateCallback)();
typedef void (*DestroyCallback)(void*);
These callbacks could be defined in Python as shown below (simplified code). The current code has a memory leak as explained in the comments.
import ctypes
CreateCallback = ctypes.CFUNCTYPE(ctypes.py_object)
DestroyCallback = ctypes.CFUNCTYPE(None, ctypes.py_object)
class Object:
pass # In the real application, this contains more code
#CreateCallback
def create():
return Object()
# Ctypes correctly increments the reference count of the
# object, to make sure it does not get garbage collected
# while the C code holds a reference to it.
#DestroyCallback
def destroy(object):
pass
# Above, the reference count of the object should be
# decremented, because the C code no longer holds a
# reference to it. However, Ctypes does not know this so
# cannot do it automatically. How can I do this from
# Python? Is it possible to call Py_DecRef or similar
# directly from Python?
One option would be to create a C function that call Py_DecRef, compile that C function into a dll (or so for Linux), and call that from the destroy function above. That solution has at least two disadvantages:
It seems overly complex to create a dll just for one function
The C code would have to be compiled against a specific version of Python, instead of using whatever version of Python is running my Python code. Note that I need this to work on Windows, where a dll cannot contain undefined globals.
I was looking at types.py to understand the built-in types and I came across this GetSetDescriptorType. From the Python documentation:
types.GetSetDescriptorType
The type of objects defined in extension modules with PyGetSetDef,
such as FrameType.f_locals or array.array.typecode. This type is used
as descriptor for object attributes; it has the same purpose as the
property type, but for classes defined in extension modules
I do understand the property type, but could not wrap my mind around this. Can some one who understands this throw some light ?
When you write a Python module using C, you define new types using a C API. This API has a lot of functions and structs to specify all the behavior of the new type.
One way to specify properties of a type using the C API is to define an array of PyGetSetDef structs:
static PyGetSetDef my_props[] = { /*... */ }
And then use the array in the initialization of the type (see this example for details).
Then, in Python, when you use MyType.my_property you have a value of types.GetSetDescriptorType, that is used to resolve the actual value of the property when you write my_obj.my_property.
As such, this type is an implementation detail, unlikely to be very useful.
I'm using a 3rd party C library that defines an opaque type:
foo_t
And uses pointers to this type in its functions:
void foo_init(foo_t *foo);
Typical usage would be allocating a foo_t on the stack and passing a reference:
{
foo_t foo;
foo_init(&foo);
...
}
How do I call foo_init() with ctypes without knowing what constitutes a foo_t?
I think if I knew sizeof(foo_t) I could create a buffer of that size and cast, but is it possible to get the size with ctypes?
I could write a one-liner C program:
printf("sizeof(foo_t) = %zu\n", sizeof(foo_t));
and hard-code that value into my python, but that would get ugly in a hurry: I'd have to touch my python source with every upgrade to the library.
A slightly cleaner way would be to write a python c-ext to export the size value, but that too would require a recompile with every library upgrade.
Does anyone have a recipe for using ctypes with such opaque types?
I think this is the simplest solution...
Create a C file, say, foosizes.c:
size_t SIZEOF_FOO = sizeof(foo_t);
And compile it into a shared object, foosizes.so. Then in a python script:
from ctypes import *
foosizeslib = CDLL('foosizes.so')
sizeof_foo = c_ulong.in_dll(foosizeslib, 'SIZEOF_FOO')
I can then create a buffer of the appropriate size and pass it to functions, by reference, as a pointer to the opaque type. So far, so good.
It is not possible to get the size with ctypes as C does not support runtime reflection, as no metadata about types is stored in the compiled binary as is done with Java or C#/.Net.
As you said, one way to get the size is create a simple C program that includes the header that defines the type and then use the sizeof operator to print out the size. Taking that a step further you could utilize a C compiler written in Python to compile and execute your C code to get the size when your Python code is executed. You might even be able to get it without needing to actually execute the result by walking the data structures provided by the compiler.
That said, are you certain you need to create the memory yourself? Frequently C libraries provide a method to create an opaque type that their other functions operate on. Update: from the comments it is certain that the memory must be allocated by the caller.
I am trying to write a wrapper to a native library in Linux. Problem is this:
definition in c:
int mymethod(mystruct* ptr)
in python:
_lib.mymethod.argtypes = (ctypes.POINTER(mystruct),)
_lib.mymethod.restype = ctypes.c_int
s = mystruct()
_lib.mymethod(ctypes.byref(s))
# raises: expected LP_mystruct instance instead of pointer to mystruct
_lib.mymethod(ctypes.pointer(s))
# raises expected LP_mystruct instance instead of LP_mystruct
errors. How to pass a structure as a pointer to a native method ?
Thanks.
Mete
The problem is that the higher level "POINTER" from ctypes is, in Python, a different object than "a generic pointer" (ctypes.CArgObject by ctypes.byref)which is returned or a single number representing a memory address (which is what is returned by ctype's adrresof) - you can either annotate your function to receive a `ctypes.c_voidp and call it with _lib.mymethod(ctypes.addressof(a)) instead -
Or if you want to work on the stronged-typed side to avoid errors that would crash Python (a type error raises a Python exception instead - a wrong parameter passed to a C unction would cause a segmentation fault on the Python interpreter itself), you have to create a variable to hold the new "type" which is a POINTER to your structure - and then create an instance of this type with the address of your structure:
mystruct_pointer = ctypes.POINTER(mystruct)
_lib.mymethod.argtypes = (mystruct_pointer,)
_lib.mymethod.restype = ctypes.c_int
s = mystruct()
_lib.mymethod(mystruct_pointer.from_address(ctypes.addressof(s)))
(I know that this is an old question, but I think the accepted answer is an unnecessary workaround, so I want to leave this here for posterity.)
Actually ctypes should explicitly support using byref() to pass a pointer like that:
ctypes exports the byref() function which is used to pass parameters by reference. The same effect can be achieved with the pointer() function, although pointer() does a lot more work since it constructs a real pointer object, so it is faster to use byref() if you don’t need the pointer object in Python itself.
The likely cause of this is that you have defined your struct in more than one place (e.g. in different modules) - if the argtypes assignment sees one definition and the function call sees the other, this confusing error arises. In other words, ctypes tries to match two mystruct types that are (probably) identical in contents, and have the exact same name, but they are not the same type. As long as the base struct type is a single type object, it doesn't matter if you construct a pointer to it using pointer(), byref() or POINTER()() - ctypes will detect that the underlying (pointed-to) type is the same.
To verify if this is the case, try assert(_lib.mymethod.argtypes[0]._type_ == type(s)) right before calling the external function.
I have a C++ class that requires a function pointer in it's constructor (float(*myfunction)(vector<float>*))
I've already exposed some function pointers to Python.
The ideal way to use this class is something like this:
import mymodule
mymodule.some_class(mymodule.some_function)
So I tell Boost about this class like so:
class_<SomeClass>("some_class", init<float(*)(vector<float>*)>);
But I get:
error: no matching function for call to 'register_shared_ptr1(Sample (*)(std::vector<double, std::allocator<double> >*))'
when I try to compile it.
So, does anyone have any ideas on how I can fix the error without losing the flexibility gained from function pointers (ie no falling back to strings that indicate which function to call)?
Also, the main point of writing this code in C++ is for speed. So it would be nice if I was still able to keep that benefit (the function pointer gets assigned to a member variable during initialization and will get called over a million times later on).
OK, so this is a fairly difficult question to answer in general. The root cause of your problem is that there really is no python type which is exactly equivalent to a C function pointer. Python functions are sort-of close, but their interface doesn't match for a few reasons.
Firstly, I want to mention the technique for wrapping a constructor from here:
http://wiki.python.org/moin/boost.python/HowTo#namedconstructors.2BAC8factories.28asPythoninitializers.29. This lets you write an __init__ function for your object that doesn't directly correspond to an actual C++ constructor. Note also, that you might have to specify boost::python::no_init in the boost::python::class_ construction, and then def a real __init__ function later, if your object isn't default-constructible.
Back to the question:
Is there only a small set of functions that you'll usually want to pass in? In that case, you could just declare a special enum (or specialized class), make an overload of your constructor that accepts the enum, and use that to look up the real function pointer. You can't directly call the functions yourself from python using this approach, but it's not that bad, and the performance will be the same as using real function pointers.
If you want to provide a general approach that will work for any python callable, things get more complex. You'll have to add a constructor to your C++ object that accepts a general functor, e.g. using boost::function or std::tr1::function. You could replace the existing constructor if you wanted, because function pointers will convert to this type correctly.
So, assuming you've added a boost::function constructor to SomeClass, you should add these functions to your python wrapping code:
struct WrapPythonCallable
{
typedef float * result_type;
explicit WrapPythonCallable(const boost::python::object & wrapped)
: wrapped_(wrapped)
{ }
float * operator()(vector<float>* arg) const
{
//Do whatever you need to do to convert into a
//boost::python::object here
boost::python::object arg_as_python_object = /* ... */;
//Call out to python with the object - note that wrapped_
//is callable using an operator() overload, and returns
//a boost::python::object.
//Also, the call can throw boost::python::error_already_set -
//you might want to handle that here.
boost::python::object result_object = wrapped_(arg_as_python_object);
//Do whatever you need to do to extract a float * from result_object,
//maybe using boost::python::extract
float * result = /* ... */;
return result;
}
boost::python::object wrapped_;
};
//This function is the "constructor wrapper" that you'll add to SomeClass.
//Change the return type to match the holder type for SomeClass, like if it's
//held using a shared_ptr.
std::auto_ptr<SomeClass> CreateSomeClassFromPython(
const boost::python::object & callable)
{
return std::auto_ptr<SomeClass>(
new SomeClass(WrapPythonCallable(callable)));
}
//Later, when telling Boost.Python about SomeClass:
class_<SomeClass>("some_class", no_init)
.def("__init__", make_constructor(&CreateSomeClassFromPython));
I've left out details on how to convert pointers to and from python - that's obviously something that you'll have to work out, because there are object lifetime issues there.
If you need to call the function pointers that you'll pass in to this function from Python, then you'll need to def these functions using Boost.Python at some point. This second approach will work fine with these def'd functions, but calling them will be slow, because objects will be unnecessarily converted to and from Python every time they're called.
To fix this, you can modify CreateSomeClassFromPython to recognize known or common function objects, and replace them with their real function pointers. You can compare python objects' identity in C++ using object1.ptr() == object2.ptr(), equivalent to id(object1) == id(object2) in python.
Finally, you can of course combine the general approach with the enum approach. Be aware when doing this, that boost::python's overloading rules are different from C++'s, and this can bite you when dealing with functions like CreateSomeClassFromPython. Boost.Python tests functions in the order that they are def'd to see if the runtime arguments can be converted to the C++ argument types. So, CreateSomeClassFromPython will prevent single-argument constructors def'd later than it from being used, because its argument matches any python object. Be sure to put it after other single-argument __init__ functions.
If you find yourself doing this sort of thing a lot, then you might want to look at the general boost::function wrapping technique (mentioned on the same page with the named constructor technique): http://wiki.python.org/moin/boost.python/HowTo?action=AttachFile&do=view&target=py_boost_function.hpp.