I wrote a scraper in python which uses selenium, chrome and an added extension. The scraper works locally with the display visible but not when I try to run it in headless mode.
On the scraped website there is a button which automatically opens a new window (I cannot access it via URL in order to open it in a new tab).
click_button() # the button opens a new window with a confirmation button
driver.switch_to.window(driver.window_handles[1]) # switching to the confirmation window
click_confirm() # when confirm is clicked the window closes automatically
driver.switch_to.window(driver.window_handles[0]) # switching back to original window
Chrome doesn't work in headless when it has extensions installed so I'm forced to use pyvirtualdisplay.
The problem is that the Display of pyvirtualdisplay opens only one browser window, therefore the confirmation window never appears. How can I allow pyvirtualdisplay to open new windows and alternate between them?
Turned out that pyvirtualdisplay did open a window, however its size was 1px by 1px. Apparently setting up the webdriver window size the following way doesn't work properly with pyvirtualdisplay:
options = Options()
options.add_argument('--window-size="1980,1080"')
driver = webdriver.Chrome(executable_path=chrome_driver_path, options=options)
Setting it up like bellow solved my problem and the window was of a normal size:
driver = webdriver.Chrome(executable_path=chrome_driver_path)
driver.set_window_size(1980,1080)
Related
I'm opening Firefox browser through selenium(3.141.0) on python 3.9, and it will always start minimized, even though in code I passed function .maximise_window() after opening the browser. When its started this way, it wont execute code to maximise and it wont execute code to switch tabs, and it wont even do it if im trying to switch tabs manually with my mouse.
If I immediately click on the initiated firefox browser on my taskbar when the program is started, it will function normally, but it will open my browsers home tab, and execute my code in new tab. Thats why you may see in code a part that closes that first tab. When i dont do it and when it starts "faulty", the home tab wont be opened.
Im also using the lenght of tabs as an indicator if the browser initiated faulty, where I tried to put it into loop and make it restart till it opens with 2 tabs, but it just wont unless I manually click it.
The only solution so far I can think of is kinda "hacky"...using pyautogui to scan my taskbar after initializing browser and clicking it fast, but I dont really like the idea.
The code goes through my company data warehouse site and manipulates it to download data.
Update
Other hacky solution I found is starting 2 browsers. First won't work, but second will. Meaning that browser works normally IF there is another Firefox browser open at the time.
Snippets of code:
from selenium import webdriver
import time
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
class DWH_browser:
def __init__(self):
self.browser = webdriver.Firefox()
self.browser.maximize_window()
self.browser.get("www.letskeepthecompanysiteasecret.org")
self.len_windows = len(self.browser.window_handles)
print(len(self.browser.window_handles))#this next part is used to close the extra tab when browser
#started normally and opens an extra tab
if len(self.browser.window_handles) == 2:
self.browser.switch_to.window(self.browser.window_handles[0])
self.browser.close()
self.browser.switch_to.window(self.browser.window_handles[0])
self.a = ActionChains(self.browser)
time.sleep(5)
DWH = DWH_browser()#and i initiate it in the code "normally"
#This was the other code I tried using to initate the browser and restart till its in 2 tabs, but not working
# issue_lenght = 1
# while issue_lenght == 1:
# DWH = DWH_browser()
# issue_lenght = DWH.len_windows
# if DWH.len_windows == 1:
# DWH.browser.quit()
# print("RESTARTING BROWSER")
Summarizing your issues:
Always start minimized.
Passed function maximise_window() after opening the browser.
Won't execute code to switch tabs.
To get rid of all these issues you need to ensure that:
Selenium is upgraded to current levels Version 4.4.0.
GeckoDriver is updated to current GeckoDriver v0.31.0 level.
Firefox Browser is updated to current firefox=103.0.2.
Additionally, you won't be needing self.browser.maximize_window() as firefox by default opens in a maximized mode.
Finally, trying to switch tabs manually with my mouse is a big no as the program execution may get interupted.
Working on web scraping since approximately 2 hours, I am using python selenium webdriver in order to extract a plain text table.
The webpage I am working on displays a dynamic table with the first 10 entries. Next to the table there is a print button that, once clicked, opens a new window with all the table entries in plain text (that's what I try to get).
The problem is that there is also the mac print popup window that also opens, waiting for pressing the cancel button before I can actually interact with the plain text table window.
When running the commands below, the last command print_button.click() keeps running in my terminal until I press the cancel button in the printer popup window.
I saw a couple of potential solutions to programmatically press the cancel print button (using ActionChains commands for instance), but I would first need to stop my print_button.click() running.
How could I make the print_button.click() command to stop once the printer window opens?
Or maybe prevent firefox to open any system print windows?
Pseudo-code:
import pandas as pd
from selenium import webdriver
driver = webdriver.Firefox()
driver.implicitly_wait(10)
driver.get('my_URL')
print_button = driver.find_element_by_xpath("/html/body/div/my_element/span")
# this is the command that keeps running until I click the cancel print button
print_button.click()
I am having trouble to login using selenium windows Internet Explorer. I am attaching the popup dialog
I need to use IE. I have tried
driver.get('https://username:password#xyz/Secure/Local/console/mc_index.html') and it does not work in IE. So I need to handle this popup dialog box.
with above code chrome works but I have to use IE (version ~ 11). So please avoid any other browser question.
Here is the code I am using
driver.get("https://xyz/Secure/Local/console/mc_index.html")
driver.find_element_by_id('moreInfoContainer').click()
driver.implicitly_wait(5)
parent_h = driver.current_window_handle
# click on the link that opens a new window
print(" before Popup")
driver.find_element_by_id("overridelink").click()
print("After popup")
It only prints up-to following 2 line:
Starting Opening the Automation
before Popup
After popup dialog it does not execute next code. So regardless what I do after the line "driver.find_element_by_id("overridelink").click()" selenium does nothing. It does not even print a simple "hello" after that line.
After the popup dialog it does not return the control to selenium.
It prints following line after i hit the "cancel" button.
After popup
The issue here is selenium never get the control back after popup dialog shows.
I am using Selenium with chromedriver in Python for testing 2 functionalities in parallel. Basically my webpage have 2 dropdown menus. Choosing an option from any dropdown menu will download a file. I have 2 scripts testing both dropdown menus separately. Now let say script-1 opens chrome and load the webpage. Now script-1 opens the dropdown-1 and it is in process of selecting an option from it as per the given input. During this if I navigate away from opened webpage to any other window eg. file explorer, the dropdown will get close without choosing the option and test case hangs. Also if I minimize the window before the dropdown is opened, the window will reopen itself when option is clicked for downloading the file. How can I stop the window from re-opening. Is there any Chrome setting for this?
Even if I move away from computer during execution, I want to run both the scripts in parallel side by side as they both open separate windows but after sometime one of them hangs the other window and I have to manually click on dropdown to bring it back to life. I am planning to run the scripts using multithreading with 2 threads referring to each of the scripts.
Note : I cannot use urlretrieve as there is no pre-defined URL for the files in webpage. PhantomJS do not support file download.
I am using Windows 10 machine.
There are several ways to do this
Run the parallel test using selenium grid and multiprocess, you could do this easily in docker, using docker/selenium, reference for using docker in https://github.com/SeleniumHQ/docker-selenium
Disable save dialog in chrome
options = webdriver.ChromeOptions()
options.add_experimental_option("prefs", {
"download.default_directory": r"yourpath",
"download.prompt_for_download": False
})
DesiredCapabilities caps = DesiredCapabilities.chrome();
driver = webdriver.Chrome(chrome_options=options)
Several times I have faced this issue: I run my Python-Selenium script using Firefox, but instead of browser window, updates downloading pop-up appears and I get WebDriverException: "The browser appears to have exited ". Just after download completed browser window opens, but script is already stopped.. So how can I avoid script exit and force webdriver to wait until Firefox updates completion...
Disable auto update by passing custom FirefoxProfile when you open the
browser.
from selenium import webdriver
fp = webdriver.FirefoxProfile()
fp.set_preference('app.update.auto', False)
fp.set_preference('app.update.enabled', False)
browser = webdriver.Firefox(firefox_profile=fp)
Another solution is to disable autoupdate manually Options>Advanced>Update by "Never check for updates" radio btn. In my case its better idea because I have full control of compability between FF and WebDriver version then. Once I want to move to newer/newest version I do it via Help>About Firefox or if Im interested in specific version I can always download it here