I would like to use python to control a windows application. I would like to click on certain button on a application using Python. Also, I would like to verify the reaction to click as expected --for example:pop up happens when it is expected. So far I have been reading PyAutoGUI to use it to automate click function( as well as to locate the button to click). To check for reaction, I have two option -
1) Do the Image Comparison between what shows up on application window and images of what is expected for asserting test pass/fail. But the problem is all text will be considered as image in comparison so I don't know how accurate will that be.
2) Use OCR to convert all readable text from screenshot of software(being tested) and compare text to text. The problem with this approach is I won't be able to check plots, bar.
I have never done testing of software application so I appreciate any insights.
Related
Is it possible to automate pressing a button when a color meets another color(check image for referance) in autohotkey or using python?
The image for what the script is for (Press E when orange goes over white)
Regarding AHK
To really activate an element using AHK you have to use the COM wrapper. Otherwise you have to resort to dirty tricks like finding your button by sending tabs, guessing its coordinates, using OCR etc.
Looking at your picture I assume it's from a game. That most likely excludes COM wrapper. If your button pops and goes that will exclude coodrinates and hotkeys. I didn't get the "color meets another color" part, but I suppose, optical recognition is your best bet.
Try your luck with this library, unless the game uses special rendering layers.
https://www.autohotkey.com/boards/viewtopic.php?f=6&t=18719
I have a script that is used to login in to a company-made application and click the right buttons like "continue", or "ok", etc. to perform a certain process. However, I have had to use screenshots of these buttons to click in order to do this using pyautogui. Is there any package or way to automate this process without using images. Maybe it can detect the text of the button and click it. I do not have identifiers for the buttons available and no access to the code/info behind the application. Let me know if you have any ideas. Thanks!
I have a few questions that may be helpful:
Does the layout of the buttons change? If it's always the same you can just program the correct locations and timing and not worry about reading the screen.
If you really have to read the screen, look into optical character recognition (ocr).
Is the application keyboard accessible? If so, using Tab and Enter to activate the buttons is simpler than controlling the mouse. Also, if it was made by superstars you can use find (ctrl-F) to search for the text on the buttons and go to them.
This answer is pretty vague, but I can only be as specific as the question asked.
Background
I have a program that calls various functions based on hotkeys and the clipboard. I am using pyperclip and aoikhotkey. This combination is working very well. However, I would like to improve it even more if possible. Currently, my workflow is as follows:
Highlight target text using my mouse
Press Ctrl+c to put it in the clipboard
Press my hotkey combination to call the function which uses the clipboard content.
I would like to eliminate step 2 and have the function called by the hotkey "scrape" the content on my screen (possibly using mouse or cursor event monitoring).
Question
Does anyone have any ideas about how I can do this? I suspect that I might be able to use Tkinter somehow to accomplish this, but I don't have any experience working with Tkinter, so if anyone has any suggestions or hints, I would be grateful.
Reference
Here's a post asking a similar question, but using the Autohotkey scripting language:
Get Selected Text Without Using the Clipboard
Update
The title of this question was originally "Getting selected text without using the clipboard". However, I changed it because the comments section to this question helped clarify my actual needs and goals.
Select Text without using the Clipboard is i think not possible, you will need the Clipboard to Copy the Text (Ctrl+c) - you can do that with your Keyboard Device by pressing the keys or you can do that by command Send a Hotkey stroke:
pyautogui.hotkey('Ctrl','c')
With python Packages pyautogui and pywinauto - you can send any text or hotkeys without to having to do a pressing on a Keyboard Device. - and if you want to use with your Mouse device you can use AutoPythonLauncher Software with this you can create Clickable Images on the Screen - watch this video Click Here
Beginner python learner here. I have a question that I have tried to Google but I just can't come up with the proper way to ask in just a few words (partly because I don't know the right terminology.)
How do I get python to detect other widgets? For example, if I wanted a script to check and see when I click my mouse if that click put focus on an entry widget on a (for example) website. I've been trying to get it to work in Tkinter and I can't figure out even where to begin.
I've seen this:
focus_displayof(self)
Return the widget which has currently the focus on the
display where this widget is located.
But the return value for that function seems to be some ambiguous long number I can't decipher, plus it only works in its own application.
Any direction would be much appreciated. :)
Do you mean inside your own GUI code, or some other application's/website's?
Sounds like you're looking for a GUI driver, or GUI test/automation driver. There are tons of these, some great, some awful, many abandoned. If you tell us more about what you want that will help narrow down the choices.
Is this for testing, or automation, or are you going to drive the mouse and button yourself and just want something to observe what is going on under the hood in the GUI?
>How do I get Python to detect other widgets?
On a machine, or in a browser? If in a machine, which platform: Linux/Windows (which)/Mac?
If in a browser, which browser (and major version)?
> But the return value for that function seems to be some ambiguous long number I can't decipher
Using longs as resource handles is par for the course, although good GUI drivers also work with string/regex matching on window and button names.
> plus it only works in its own application.
What do you mean, and what are you expecting it to return you? You should be able to look up that GUI object and access its title. Look for a GUI driver that works with window and button names.
Here is one list, read it through and see what sounds useful. I have used AutoIt under Win32, it's great, widely-used and actively-maintained; it can be called from Python (via subprocess).
Here are comparisons by the author of PyWinAuto on his and similar tools. Give a read to his criticisms of its structure from 2010. If none of these is what you want, at least you now have the vocabulary to tell us what would be...
I'm working on a project to increase the ability for wine to automatically test software packages. What I'm looking to do now is detect text in the screen capture of the current window. I can then parse all of the text and use autohotkey to give a mouse click on the coordinates of the text I want.
For example, in firefox, I might want to test different things, the first open being opening preferences. I would then need to parse the screenshot of firefox, detect all of the separate locations of text. I can then run these separate images of text into tesseract-ocr and detect which one, says "Edit". I then redo this again for "preferences".
I've tried to find a solution but so far can't find anything. I'd prefer a solution that uses python or has python binds as thats what I've been programing in so far.
A possible starting point is Project SIKULI. It is a tool to automate GUI testing. It is written in Java, nonetheless it includes a scripting environment based on Jython, hence modifying it to support python script may be not too difficult.