Develop Reference

Python curses - textpad.Textbox() keyboard input not working with German umlauts

I'm trying to use the curses textpad.Textbox() function for text input. Everything is working fine so far, however, some keys don't get recognized, including the section sign (§) and all German umlauts (ä/ö/ü). I guess it's somehow related to the text encoding, but I have no idea how to fix this. My German keyboard layout works perfectly fine with input().
Here is some minimal example:
import curses
import curses.textpad as textpad
try:
stdtscr = curses.initscr()
curses.cbreak()
stdtscr.keypad(1)
curses.noecho()
textpad.Textbox(stdtscr).edit()
finally:
curses.nocbreak()
stdtscr.keypad(0)
curses.echo()
curses.endwin()

Just as in C, you should initialize the locale. It's spelled out in both the Python documentation:
Since version 5.4, the ncurses library decides how to interpret non-ASCII data using the nl_langinfo function. That means that you have to call locale.setlocale() in the application and encode Unicode strings using one of the system’s available encodings.
and the ncurses manual page:
The library uses the locale which the calling program has
initialized. That is normally done with setlocale:
setlocale(LC_ALL, "");
If the locale is not initialized, the library assumes that
characters are printable as in ISO-8859-1, to work with cer-
tain legacy programs. You should initialize the locale and
not rely on specific details of the library when the locale
has not been setup.
Addressing the followup comment, textpad.py does not expect UTF-8 input in any case. Essentially it "validates" its input, decides it isn't ASCII and ignores it when it's not.
Python's curses binding provides an interface to wgetch, which (with ncurses) gives the individual bytes for the UTF-8. (X/Open Curses specifies a different function wget_wch, for which Python has no binding).
textpad.py could be modified to work around the curses binding by assembling the bytes into a Unicode value, but you'd need the setlocale as the first step.

How do I tell which actual dll is being returned (x86 v x64)?

Let's focus on one dll: C:\Windows\System32\wbem\wmiutils.dll. Why? Because it's the file in which I personally discovered Windows delivers a different dll depending on process architecture.
TLDR; Is there a way to programmatically determine the actual path of the dll that was returned by the file system redirector?
I understand that if launched as a x86 process, I get C:\Windows\SysWOW64\wbem\wmiutils.dll. And, if launched as a x64 process, I get C:\Windows\System32\wbem\wmiutils.dll.
I need to determine which wmiutils.dll I'm actually looking at. The redirector makes system32\wbem\wmiutils.dll look and feel identical but it's not. If I use parent path, I get C:\Windows\System32\wbem even though I may/may not be looking at C:\Windows\SysWOW64\wbem.
Any sweet python magic to make this happen? I can't seem to see anything from other languages I can port. Based on my use case, I've come up with a couple hacks but they're just that. Hoping somebody has found a solution as easy as parent path that actually works in this case.

import ctypes, hashlib
k32 = ctypes.windll.kernel32
oldValue = ctypes.c_long(0)
k32.Wow64DisableWow64FsRedirection(ctypes.byref(oldValue)) # Should open 32-bit
with open(r"C:\Windows\System32\wbem\wmiutil.dll", "rb") as f:
checksum32 = hashlib.md5(f.read()).hexdigest()
k32.Wow64RevertWow64FsRedirection(oldValue) # Should use what Windows thinks you need
with open(r"C:\Windows\System32\wbem\wmiutil.dll", "rb") as f:
checksum64 = hashlib.md5(f.read()).hexdigest()
if (checksum32 != checksum64):
print("You're running 64bit wmiutil dll")
I don't have Windows Python to test this, but it should work according to https://msdn.microsoft.com/en-us/library/windows/desktop/aa365745%28v=vs.85%29.aspx.
I think an easier way would be to just do some test like creating a struct and seeing if it's 8 bytes or 4 bytes. Then you can assume that Windows is using the 64-bit version of DLLs if it's 8 bytes.

In python how can i know a given file is being used

I have a list of file and there is only one in used in a time. So i wanna know which file is being used by specific program. Since i can use 'unlocker' to find out a file that are in used like this question have mentioned. But i want a programming way so that my program can help me find out. Is there any way?
Specially, the simple 'open' function in whatever r/w mode CAN access the using file and python won't throw any exception. I can tell which file being used only by 'unlocker'.
I have find out that the python 'open' function in w mode have took the access permission from that specific program, and the program then don't work so well. In this moment i open the unlocker and i can see two process accessing the file. Is there any 'weak' method that can only detect whether the file is being used?

I'm not sure which of these two you want to find:
Are there are any existing HANDLEs for a given file, like the handle and Process Explorer tools shows?
Are there any existing locked HANDLEs for a given file, like the Unlocker tool shows.
But either way, the answer is similar.
Obviously it's doable, or those tools couldn't do it, right? Unfortunately, there is nothing in the Python stdlib that can help. So, how do you do it?
You will need to access Windows APIs functions—through pywin32, ctypes, or otherwise.
There are two ways to go about it. The first, mucking about with the NT kernel object APIs, is much harder, and only really needed if you need to work with very old versions of Windows. The second, NtQuerySystemInformation, is the one you probably want.
The details are pretty complicated (and not well documented), but they're explained in a CodeProject sample program and on the Sysinternals forums.
If you don't understand the code on those pages, or how to call it from Python, you really shouldn't be doing this. If you get the gist, but have questions, or get stuck, of course you can ask for help here (or at the sysinternals forums or elsewhere).
However, even if you have used ctypes for Windows before, there are a few caveats you may not know about:
Many of these functions either aren't documented, or the documentation doesn't tell you which DLL to find them in. They will all be in either ntdll or kernel32; in some cases, however, the documented NtFooBar name is just an alias for the real ZwFooBar function, so if you don't find NtFooBar in either DLL, look for ZwFooBar.
At least on older versions of Windows, ZwQuerySystemInformation does not act as you'd expect: You cannot call it with a 0 SystemInformationLength, check the ReturnLength, allocate a buffer of that size, and try again. The only thing you can do is start with a buffer with enough room for, say, 8 handles, try that, see if you get an error STATUS_INFO_LENGTH_MISMATCH, increase that number 8, and try again until it succeeds (or fails with a different error). The code looks something like this (assuming you've defined the SYSTEM_HANDLE structure):
STATUS_INFO_LENGTH_MISMATCH = 0xc0000004
i = 8
while True:
class SYSTEM_HANDLE_INFORMATION(Structure):
_fields_ = [('HandleCount', c_ulong),
('Handles', SYSTEM_HANDLE * i)]
buf = SYSTEM_HANDLE_INFORMATION()
return_length = sizeof(buf)
rc = ntdll.ZwQuerySystemInformation(SystemHandleInformation,
buf, sizeof(buf),
byref(return_length))
if rc == STATUS_INFO_LENGTH_MISMATCH:
i += 8
continue
elif rc == 0:
return buf.Handles[:buf.HandleCount]
else:
raise SomeKindOfError(rc)
Finally, the documentation doesn't really explain this anywhere, but the way to get from a HANDLE that you know is a file to a pathname is a bit convoluted. Just using NtQueryObject(ObjectNameInformation) returns you a kernel object space pathname, which you then have to map to either a DOS pathname, a possibly-UNC normal NT pathname, or a \?\ pathname. Of course the first doesn't work files on network drives without a mapped drive letter; neither of the first two work for files with very long pathnames.
Of course there's a simpler alternative: Just drive handle, Unlocker, or some other command-line tool via subprocess.
Or, somewhere in between, build the CodeProject project linked above and just open its DLL via ctypes and call its GetOpenedFiles method, and it will do the hard work for you.
Since the project builds a normal WinDLL-style DLL, you can call it in the normal ctypes way. If you've never used ctypes, the examples in the docs show you almost everything you need to know, but I'll give some pseudocode to get you started.
First, we need to create an OF_CALLBACK type for the callback function you're going to write, as described in Callback functions. Since the prototype for OF_CALLBACK is defined in a .h file that I can't get to here, I'm just guessing at it; you'll have to look at the real version and translate it yourself. But your code is going to look something like this:
from ctypes import windll, c_int, WINFUNCTYPE
from ctypes.wintypes import LPCWSTR, UINT_PTR, HANDLE
# assuming int (* OF_CALLBACK)(int, HANDLE, int, LPCWSTR, UINT_PTR)
OF_CALLBACK = WINFUNCTYPE(c_int, HANDLE, c_int, LPWCSTR, UINT_PTR)
def my_callback(handle, namelen, name, context):
# do whatever you want with each handle, and whatever other args you get
my_of_callback = OF_CALLBACK(my_callback)
OpenFileFinder = windll.OpenFileFinder
# Might be GetOpenedFilesW, as with most Microsoft DLLs, as explained in docs
OpenFileFinder.GetOpenedFiles.argtypes = (LPCWSTR, c_int, OF_CALLBACK, UINT_PTR)
OpenFileFinder.GetOpenedFiles.restype = None
OpenFileFinder.GetOpenedFiles(ru'C:\Path\To\File\To\Check.txt', 0, my_of_callback, None)
It's quite possible that what you get in the callback is actually a pointer to some kind of structure or list of structures you're going to have to define—likely the same SYSTEM_HANDLE you'd need for calling the Nt/Zw functions directly, so let me show that as an example—if you get back a SYSTEM_HANDLE *, not a HANDLE, in the callback, it's as simple as this:
class SYSTEM_HANDLE(Structure):
_fields_ = [('dwProcessId', DWORD),
('bObjectType', BYTE),
('bFlags', BYTE),
('wValue', WORD),
('pAddress', PVOID),
('GrantedAccess', DWORD)]
# assuming int (* OF_CALLBACK)(int, SYSTEM_HANDLE *, int, LPCWSTR, UINT_PTR)
OF_CALLBACK = WINFUNCTYPE(c_int, POINTER(SYSTEM_HANDLE), c_int, LPWCSTR, UINT_PTR)

You could use a try/except block.
check if a file is open in Python
try:
f = open("file.txt", "r")
except IOError:
print "This file is already in use"

Python: Windows System File

In python, how can I identify a file that is a "window system file". From the command line I can do this with the following command:
ATTRIB "c:\file_path_name.txt"
If the return has the "S" character, then it's a windows system file. I cannot figure out the equivilant in python. A few example of similar queries look like this:
Is a file writeable?
import os
filePath = r'c:\testfile.txt'
if os.access(filePath, os.W_OK):
print 'writable'
else:
print 'not writable'
another way...
import os
import stat
filePath = r'c:\testfile.txt'
attr = os.stat(filePath)[0]
if not attr & stat.S_IWRITE:
print 'not writable'
else:
print 'writable'
But I can't find a function or enum to identify a windows system file. Hopefully there's a built in way to do this. I'd prefer not to have to use win32com or another external module.
The reason I want to do this is because I am using os.walk to copy files from one drive to another. If there was a way to walk the directory tree while ignoring system files that may work too.
Thanks for reading.
Here's the solutions I came up with based on the answer:
Using win32api:
import win32api
import win32con
filePath = r'c:\test_file_path.txt'
if not win32api.GetFileAttributes(filePath) & win32con.FILE_ATTRIBUTE_SYSTEM:
print filePath, 'is not a windows system file'
else:
print filePath, 'is a windows system file'
and using ctypes:
import ctypes
import ctypes.wintypes as types
# From pywin32
FILE_ATTRIBUTE_SYSTEM = 0x4
kernel32dll = ctypes.windll.kernel32
class WIN32_FILE_ATTRIBUTE_DATA(ctypes.Structure):
_fields_ = [("dwFileAttributes", types.DWORD),
("ftCreationTime", types.FILETIME),
("ftLastAccessTime", types.FILETIME),
("ftLastWriteTime", types.FILETIME),
("nFileSizeHigh", types.DWORD),
("nFileSizeLow", types.DWORD)]
def isWindowsSystemFile(pFilepath):
GetFileExInfoStandard = 0
GetFileAttributesEx = kernel32dll.GetFileAttributesExA
GetFileAttributesEx.restype = ctypes.c_int
# I can't figure out the correct args here
#GetFileAttributesEx.argtypes = [ctypes.c_char, ctypes.c_int, WIN32_FILE_ATTRIBUTE_DATA]
wfad = WIN32_FILE_ATTRIBUTE_DATA()
GetFileAttributesEx(pFilepath, GetFileExInfoStandard, ctypes.byref(wfad))
return wfad.dwFileAttributes & FILE_ATTRIBUTE_SYSTEM
filePath = r'c:\test_file_path.txt'
if not isWindowsSystemFile(filePath):
print filePath, 'is not a windows system file'
else:
print filePath, 'is a windows system file'
I wonder if pasting the constant "FILE_ATTRIBUTE_SYSTEM" in my code is legit, or can I get its value using ctypes as well?

But I can't find a function or enum to identify a windows system file. Hopefully there's a built in way to do this.
There is no such thing. Python's file abstraction doesn't have any notion of "system file", so it doesn't give you any way to get it. Also, Python's stat is a very thin wrapper around the stat or _stat functions in Microsoft's C runtime library, which doesn't have any notion of "system file". The reason for this is that both Python files and Microsoft's C library are both designed to be "pretty much like POSIX".
Of course Windows also has a completely different abstraction for files. But this one isn't exposed by the open, stat, etc. functions; rather, there's a completely parallel set of functions like CreateFile, GetFileAttributes, etc. And you have to call those if you want that information.
I'd prefer not to have to use win32com or another external module.
Well, you don't need win32com, because this is just Windows API, not COM.
But win32api is the easiest way to do it. It provides a nice wrapper around GetFileAttributesEx, which is the function you want to call.
If you don't want to use an external module, you can always call Windows API functions via ctypes instead. Or use subprocess to run command-line tools (like ATTRIB—or, if you prefer, like DIR /S /A-S to let Windows do the recursive-walk-skipping-system-files bit for you…).
The ctypes docs show how to call Windows API functions, but it's a little tricky the first time.
First you need to go to the MSDN page to find out what DLL you need to load (kernel32), and whether your function has separate A and W variants (it does), and what values to pass for any constants (you have to follow a link to another page, and know how C enums works, to find out that GetFileExInfoStandard is 0), and then you need to figure out how to define any structs necessary. In this case, something like this:
from ctypes import *
kernel = windll.kernel32
GetFileExInfoStandard = 0
GetFileAttributesEx = kernel.GetFileAttributesEx
GetFileAttributesEx.restype = c_int
GetFileAttributesEx.argypes = # ...
If you really want to avoid using win32api, you can do the work to finish the ctypes wrapper yourself. Personally, I'd use win32api.
Meanwhile:
The reason I want to do this is because I am using os.walk to copy files from one drive to another. If there was a way to walk the directory tree while ignoring system files that may work too.
For that case, especially given your complaint that checking each file was too slow, you probably don't want to use os.walk either. Instead, use FindFirstFileEx, and do the recursion manually. You can distinguish files and directories without having to stat (or GetFileAttributesEx) each file (which os.walk does under the covers), you can filter out system files directly inside the find function instead of having to stat each file, etc.
Again, the options are the same: use win32api if you want it to be easy, use ctypes otherwise.
But in this case, I'd take a look at Ben Hoyt's betterwalk, because he's already done 99% of the ctypes-wrapping, and 95% of the rest of the code, that you want.

Handling lines with quotes using python's readline

I've written a simple shell-like program that uses readline in order to provide smart completion of arguments. I would like the mechanism to support arguments that have spaces and are quoted to signify as one argument (as with providing the shell with such).
I've seen that shlex.split() knows how to parse quoted arguments, but in case a user wants to complete mid-typing it fails (for example: 'complete "Hello ' would cause an exception to be thrown when passed to shlex, because of unbalanced quotes).
Is there code for doing this?
Thanks!

I don't know of any existing code for the task, but if I were to do this I'd catch the exception, try adding a fake trailing quote, and see how shlex.split does with the string thus modified.

GNU Readline allows for that scenario with the variable rl_completer_quote_characters. Unfortunatelly, Python does not export that option on the standard library's readline module (even on 3.7.1, the latest as of this writing).
I found a way of doing that with ctypes, though:
import ctypes
libreadline = ctypes.CDLL ("libreadline.so.6")
rl_completer_quote_characters = ctypes.c_char_p.in_dll (
libreadline,
"rl_completer_quote_characters"
)
rl_completer_quote_characters.value = '"'
Note this is clearly not portable (possibly even between Linux distros, as the libreadline version is hardcoded, but I didn't have plain libreadline.so on my computer), so you may have to adapt it for your environment.
Also, in my case, I set only double quotes as special for the completion feature, as that was my use case.
References
https://robots.thoughtbot.com/tab-completion-in-gnu-readline#adding-quoting-support
#eryksun's comment on how to set data to a global variable in a shared library using python

To make #caxcaxcoatl answer a little bit more portable, readline hardcoded version can be replaces with readline.__file__ and it will be:
import ctypes
import readline
libreadline = ctypes.CDLL (readline.__file__)
rl_completer_quote_characters = ctypes.c_char_p.in_dll (
libreadline,
"rl_completer_quote_characters"
)
rl_completer_quote_characters.value = '"'