Develop Reference

Different behaviours encountered when posting a mouse scroll message to different windows with win32gui

I was trying to send scroll commands to a window in the background, when I encountered strange behaviours that I don't really understand.
The Script
I wrote a script to illustrate what I encountered. To have it work, I used a Windows 10 system, and had one mail window and one command prompt window open in the background (not minimised). The script sends a scroll command to both windows with the mouse pointer located at the middle of the window.
In both windows, the cursor should hover over a region that allows for scrolling. The difference lies in the fact that for the mail window, only a portion of the window allows for scrolling (I used these windows because it is most likely that everyone using Windows has these applications).
import win32gui as wgui
import win32api as wapi
import win32con as wcon
# This script is meant to try sending scroll messages to a window.
def callback(hwnd, win_list):
""" Callback for win32gui.EnumWindows : appends the hwnd to win_list."""
win_list.append(hwnd)
def find_specific_window(tag):
""" This function finds a specific window and returns its handle and name.
:param tag: A key word that describes the window we look for.
.. note:
Always stops at the first match.
"""
win_list = []
wgui.EnumWindows(callback, win_list)
# Find the first window with the tag in its name:
hwnd = None
name = None
for h in win_list:
n = wgui.GetWindowText(h)
if tag in n:
name = n
hwnd = h
break
return hwnd, name
def scroll_bg(hwnd, pointer_x, pointer_y, child=True, delta=1):
""" Performs a scroll on a background window with handle hwnd (might need to use FindWindowEx).
:param pointor_x: relative horizontal position of the mouse (y is vertical).
:param child: Boolean telling whether to operate on the top child window or the main window.
"""
# Take the top child window handle
if child:
hwnd = wgui.FindWindowEx(hwnd, None, None, None)
# Transform the mouse coordinates into pixels for the given window.
# Obtain the window dimensions
rect = wgui.GetWindowRect(hwnd)
w = rect[2] # Width
h = rect[3] # Height
x_pix = int(round(w * pointer_x))
y_pix = int(round(h * pointer_y))
print("In scroll_bg: the coordinates for the window {} are : ({}, {})".format(hwnd, x_pix, y_pix))
# Transform them for the function
lparam = wapi.MAKELONG(x_pix, y_pix)
# Transform into wparam
wparam = wapi.MAKELONG(0, - delta * wcon.WHEEL_DELTA)
# Hover to the pointer position.
wgui.PostMessage(hwnd, wcon.WM_MOUSEMOVE, 0, lparam)
# Scroll
wgui.PostMessage(hwnd, wcon.WM_MOUSEWHEEL, wparam, lparam)
wgui.PostMessage(hwnd, wcon.WM_MOUSEMOVE, 0, lparam)
if __name__ == '__main__':
window1_tag = "Mail"
window2_tag = "Command"
hwnd1 = find_specific_window(window1_tag)
hwnd2 = find_specific_window(window2_tag)
print("first window handle: {}, second window handle: {}".format(hwnd1, hwnd2))
# In both windows we set the pointer at the middle.
scroll_bg(hwnd1[0], 1/2, 1/2)
scroll_bg(hwnd2[0], 1/2, 1/2, False)
In the script, I defined a scroll_bg function that can send the scroll instruction to a given window, identified by hwnd, or its first child (by setting child=True).
My Script Result
Running this script I can observe that the scroll has correctly been sent to the Command Prompt window, but nothing changed for the Mail window, despite targetting scrollable areas in both windows. I must note that I also tried sending the scroll message to the parent window, by running scroll_bg(hwnd1[0], 1/2, 1/2, False), but it didn't work either.
More Details
I must also note that I observed this issue in the same context as this question. I wanted to send scroll commands instead of dragging, and managed to successfully scroll in certain areas while not in others (for the same game).
I further observed that giving a scroll command would do the same animation as a click. The weird part about it is that changing the position of pointer_x didn't change the position of the animation, and it occurred at the left border of the window (horizontal pixel = 0). It looks like the pointer_x is set to 0, and this would explain the script results.
An other point worth mentioning is that maybe the scroll commands can only be interpreted by some other child window and that the wgui.PostMessage function sends the scroll commands to the wrong window thus they are not interpreted correctly. I'm a bit confused as to which window I should send the commands to. For instance, in the case of the game, posting messages to the parent window would not work and I had to send them to the first child. So, perhaps, the results of the script could be explained by the fact that I'm not sending to the correct child.
Versions I am using:
pywin32 v: 300
python v: 3.7.1
Thank you for your time, and any advice is welcome ! :)

How to click relative to a window/application in python and how to handle multiple scripts that click?

Is there any way to click relative to an open window? For example, clicking a set amount of pixels to the right/up/left/down of an open tab of google chrome? I know how to click using absolute coordinates, or click on something that matches an image file, but I haven't been able to find anything regarding relative clicking.
Another part to this - when automating some process on a computer that uses the mouse or keyboard to input commands, if you run two or more of the same script, is there a possibility that the commands interrupt each other? Like if you move the mouse and then click, but another script moves it again before the first one is allowed to click? Is there an easy solution for this? What my mind jumps to first is using a queuing process similar to handling multiple processes in an OS.

You can use the library pyautogui.
I put an example here:
import pyautogui as pya
start = pya.locateCenterOnScreen('start.png')#If the file is not a png file it will not work
print(start)
pya.moveTo(start)#Moves the mouse to the coordinates of the image
#even you can make click with
pya.click(button='left',clicks=2,x=start.x,y=start.y) # you can do two click on the image

For anyone in the future coming across this post, using Python to move their mouse relative to a specific windows coordinates:
EASIEST WAY
(Requires pyautogui & pywin32 (win32gui successor))
Step 1: Get the HWND or the 'ID' of the 'Active Window' you want to move your mouse relative to:
hwnd = win32gui.FindWindow(None, 'Untitled - Notepad')
Step 2: Now let's get the global X and Y coordinates of the top left and bottom right corner of this app to determine where it is on your screen at the moment:
x0, y0, x1, y1 = win32gui.GetWindowRect(hwnd)
You can also get the dimensions of the window here as well:
w = x1 - x0 # width
h = y1 - y0 # height
Step 3: We only need the X0 and Y0 (top left corner global coordinates on our screen), so return those if you put this in its own method
Step 4: All you have to do now is create a custom method to moveMouse(x, y) which takes your relative coordinates in as an argument, and translates them using the newly acquired X0, Y0 coordinates. You're basically just adding the offset required to be 'relative' to the active window and that offset is acquired by determining the top left corners x, y position.
def MoveMouse(x, y):
x0, y0 = GetMyWindowSize() # Method I created to return the GetWindowRect() values
pyautogui.moveTo(x + bsX, y + bsY)
return
Good luck! If you're not restricted to Python, AHK (Auto Hotkey) is also worth checking out for automation of mouse and keyboard inputs. There are native methods to recreate this functionality.

Is there a way to send a click event to a window in the background in python?

So I'm trying to build a bot to automate some actions in a mobile game that I'm running on my pc through Bluestacks.
My program takes a screenshot of the window, looks for certain button templates in the image and returns their coordinates.
I would now like to be able to send a click event to the window at those coordinates, but since I would also like to do other things while the bot runs in the background I'm looking for a way to send the mouse event directly to the window (even if it's minimized/in the background), without influencing the movement of the mouse while I'm doing other stuff or bringing the window to the foreground/unminimizing it. Is this possible?

Based on Dmitry's answer. The x and y must be the coordinates relative to the Bluestacks window not the screen.
def click(x, y):
hWnd = win32gui.FindWindow(None, "BlueStacks")
lParam = win32api.MAKELONG(x, y)
hWnd1= win32gui.FindWindowEx(hWnd, None, None, None)
win32gui.SendMessage(hWnd1, win32con.WM_LBUTTONDOWN, win32con.MK_LBUTTON, lParam)
win32gui.SendMessage(hWnd1, win32con.WM_LBUTTONUP, None, lParam)
click(100,100)

I solved this problem on windws 10 using win32api.
In Spy++ I looked at the mouse messages that occur when I click in Bluestacks.
I found that I should find the hwnd of bluestacks child window with the title "BlueStacks Android PluginAndroid". And send them mouse click events:
lParam = win32api.MAKELONG(x, y)
win32api.PostMessage(hWnd, win32con.WM_LBUTTONDOWN, win32con.MK_LBUTTON, lParam)
win32api.PostMessage(hWnd, win32con.WM_LBUTTONUP, None, lParam)
This works for me even if the window is minimized to the tray.

I learned that you'll need to use the correct libraries for this to work properly. I discovered that pynput works very well. Apart from what you're trying to do, it can fetch user input on the mouse and keyboard before your OS processes it.
Let me help you save a lot of time of going through all the different libraries: I suggest pynput. You can read about it here. Below you can find a piece of example code that I copied from this website (should it ever go offline):
from pynput.mouse import Button, Controller
mouse = Controller()
# Read pointer position
print('The current pointer position is {0}'.format(
mouse.position))
# Set pointer position
mouse.position = (10, 20)
print('Now we have moved it to {0}'.format(
mouse.position))
# Move pointer relative to current position
mouse.move(5, -5)
# Press and release
mouse.press(Button.left)
mouse.release(Button.left)
# Double click; this is different from pressing and releasing
# twice on Mac OSX
mouse.click(Button.left, 2)
# Scroll two steps down
mouse.scroll(0, 2)

If you really want to automate bluestacks, you could use adb (which could do the assigned work like clicking a button, even when the window is minimized) you could use
Ppadb (pure Python adb module) for automating it. And ppadb could be used in mac and Linux also, but win32 is restricted to Windows.

Python window detection [duplicate]

How can I get and set the window (any windows program) position and size with python?

Assuming you're on Windows, try using pywin32's win32gui module with its EnumWindows and GetWindowRect functions.
If you're using Mac OS X, you could try using appscript.
For Linux, you can try one of the many interfaces to X11.
Edit: Example for Windows (not tested):
import win32gui
def callback(hwnd, extra):
rect = win32gui.GetWindowRect(hwnd)
x = rect[0]
y = rect[1]
w = rect[2] - x
h = rect[3] - y
print("Window %s:" % win32gui.GetWindowText(hwnd))
print("\tLocation: (%d, %d)" % (x, y))
print("\t Size: (%d, %d)" % (w, h))
def main():
win32gui.EnumWindows(callback, None)
if __name__ == '__main__':
main()

You can get the window coordinates using the GetWindowRect function. For this, you need a handle to the window, which you can get using FindWindow, assuming you know something about the window (such as its title).
To call Win32 API functions from Python, use pywin32.

As Greg Hewgill mentioned, if you know the name of the window, you can simply use win32gui's FindWindow, and GetWindowRect. This is perhaps a little cleaner and efficient than previous methods.
from win32gui import FindWindow, GetWindowRect
# FindWindow takes the Window Class name (can be None if unknown), and the window's display text.
window_handle = FindWindow(None, "Diablo II")
window_rect = GetWindowRect(window_handle)
print(window_rect)
#(0, 0, 800, 600)
For future reference: PyWin32GUI has now moved to Github

this can return window rect from window title
Code
def GetWindowRectFromName(name:str)-> tuple:
hwnd = ctypes.windll.user32.FindWindowW(0, name)
rect = ctypes.wintypes.RECT()
ctypes.windll.user32.GetWindowRect(hwnd, ctypes.pointer(rect))
# print(hwnd)
# print(rect)
return (rect.left, rect.top, rect.right, rect.bottom)
if __name__ == "__main__":
print(GetWindowRectFromName('CALC'))
pass
Environment
Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:34:03) [MSC v.1916 64 bit (AMD64)] on win32
Windows 10 Pro 1909

For Linux you can use the tool I made here. The tool was meant for a slightly different use but you can use the API directly for your needs.
Install tool
sudo apt-get install xdotool xprop xwininfo
git clone https://github.com/Pithikos/winlaunch.git && cd winlaunch
In terminal
>>> from winlaunch import *
>>> wid, pid = launch('firefox')
>>> win_pos(wid)
[3210, 726]
wid and pid stand for window id and process id respectively.

This code will work on windows. It return the position and size of the active window.
from win32gui import GetWindowText, GetForegroundWindow
print(GetWindowRect(GetForegroundWindow()))

Something not mentioned in any of the other responses is that in newer Windows (Vista and up), "the Window Rect now includes the area occupied by the drop shadow.", which is what win32gui.GetWindowRect and ctypes.windll.user32.GetWindowRect are interfacing with.
If you want to get the positions and sizes without the dropshadow, you can:
Manually remove them. In my case there were 10 pixels on the left, bottom and right which had to be pruned.
Use the dwmapi to extract the DWMWA_EXTENDED_FRAME_BOUNDS as mentioned in the article
On using the dwmapi.DwmGetWindowAttribute (see here):
This function takes four arguments: The hwnd, the identifier for the attribute we are interested in, a pointer for the data structure in which to write the attribute, the size of this data structure. The identifier we get by checking this enum. In our case, the attribute DWMWA_EXTENDED_FRAME_BOUNDS is on position 9.
import ctypes
from ctypes.wintypes import HWND, DWORD, RECT
dwmapi = ctypes.WinDLL("dwmapi")
hwnd = 133116 # refer to the other answers on how to find the hwnd of your window
rect = RECT()
DMWA_EXTENDED_FRAME_BOUNDS = 9
dwmapi.DwmGetWindowAttribute(HWND(hwnd), DWORD(DMWA_EXTENDED_FRAME_BOUNDS),
ctypes.byref(rect), ctypes.sizeof(rect))
print(rect.left, rect.top, rect.right, rect.bottom)
Lastly: "Note that unlike the Window Rect, the DWM Extended Frame Bounds are not adjusted for DPI".

How to make python window run as "Always On Top"?

I am running a little program in python that launches a small window that needs to stay on top of all the other windows. I believe this is OS specific, how is it done in GNU-Linux with GNOME?
[Update - Solution for Windows]
Lovely, I think I got it working. I am using Python 2.5.4 with Pygame 1.9.1 in Eclipse on Vista 64-bit. Thus, this is for windows systems. The SetWindowPos function is documented Here. I will refer to this in my explanation.
Imports:
from ctypes import windll
Then I set up a variable that calls the "SetWindowPos" in user32:
SetWindowPos = windll.user32.SetWindowPos
Now, let's say I just made a window:
screen = pygame.display.set_mode((100,100), pygame.NOFRAME)
The next line is the key. This sets the window to be on top of other windows.
SetWindowPos(pygame.display.get_wm_info()['window'], -1, x, y, 0, 0, 0x0001)
Basically, You supply the hWnd(Window Handle) with the window ID returned from a call to display.get_wm_info(). Now the function can edit the window you just initialized.
The -1 is our hWndInsertAfter.
The MSDN site says:
A window can be made a topmost window either by setting the hWndInsertAfter parameter to HWND_TOPMOST and ensuring that the SWP_NOZORDER flag is not set, or by setting a window's position in the Z order so that it is above any existing topmost windows. When a non-topmost window is made topmost, its owned windows are also made topmost. Its owners, however, are not changed.
So, the -1 makes sure the window is above any other existing topmost windows, but this may not work in all cases. Maybe a -2 beats a -1? It currently works for me. :)
The x and y specify the new coordinates for the window being set. I wanted the window to stay at its current position when the SetWindowPos function was called on it. Alas, I couldn't find a way to easily pass the current window (x,y) position into the function. I was able to find a work around, but assume I shouldn't introduce a new topic into this question.
The 0, 0, are supposed to specify the new width and height of the window, in pixels. Well, that functionality is already in your pygame.display.set_mode() function, so I left them at 0. The 0x0001 ignores these parameters.
0x0001 corresponds to SWP_NOSIZE and is my only uFlag. A list of all the available uFlags are on the provided documentation page. Some of their Hex representations are as follows:
SWP_NOSIZE = 0x0001
SWP_NOMOVE = 0x0002
SWP_NOZORDER = 0x0004
SWP_NOREDRAW = 0x0008
SWP_NOACTIVATE = 0x0010
SWP_FRAMECHANGED = 0x0020
SWP_SHOWWINDOW = 0x0040
SWP_HIDEWINDOW = 0x0080
SWP_NOCOPYBITS = 0x0100
SWP_NOOWNERZORDER = 0x0200
SWP_NOSENDCHANGING = 0x0400
That should be it! Hope it works for you!
Credit to John Popplewell at john#johnnypops.demon.co.uk for his help.

The question is more like which windowing toolkit are you using ? PyGTK and similar educated googling gave me this:
gtk.Window.set_keep_above
As mentioned previously it is upto the window manager to respect this setting or not.
Edited to include SDL specific stuff
Pygame uses SDL to do display work and apprently does not play nice with Windowing toolkits. SDL Window can be put on top is discussed here.

I really don't know much Python at all, but Googling "pygtk always on top" gave me this:
http://www.mail-archive.com/pygtk#daa.com.au/msg01370.html
The solution posted there was:
transient.set_transient_for(main_window)
You might also want to search things like "x11 always on top". The underlying concept seems to be that you're giving the window manager a "hint" that it should keep the window above the others. The window manager, however, has free reign and can do whatever it wants.
I've also seen the concept of layers when using window managers like Fluxbox, so maybe there's a way to change the layer on which the window appears.

I was trying to figure out a similar issue and found this solution using the Pmw module
http://www.java2s.com/Code/Python/GUI-Pmw/Showglobalmodaldialog.htm