getting TimeoutException when using expected_conditions in heroku - python

I have a selenium robot that worked perfectly locally but on heroku TimeoutException raises whenever its on a expected_condition (element_to_be_clickable, visibility_of_element_located and presence_of_element_located). Anyone knows how to fix this problem in heroku.
here is an example where I used expected_conditions
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(#class, 'productlistning__btn')]")))
and the chrome arguments that I used in my code:
chrome_options = Options()
chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--no-sandbox")
driver = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), options=chrome_options)

I guess you didn't define the screen size for the driver while in headless mode the default screen size is 800,600.
So, to make your Selenium code working try setting the screen size to maximal or 1920,1080. As following:
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("window-size=1920,1080")
Or
options = Options()
options.add_argument("start-maximized")

Related

ChromeDriver not opening new page with chrome_options parameter

I'm trying to use the following code to open a new page using ChromeDriver
import selenium
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=r"path of chromedriver.exe",chrome_options=options)
I still get the "DevTools listening on...." print but no new page is being opened. If however I run:
driver = webdriver.Chrome(executable_path = r"path")
without the chrome_options parameter, the page opens. Not sure why this is?
chrome_options was deprecated long back.
DeprecationWarning: use options instead of chrome_options
you have to use an instance of options instead as well as pass the absolute path of the ChromeDriver along with the extension as follows:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=r"path of chromedriver.exe", options=options)
Use following code :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
#object of ChromeOptions
op = webdriver.ChromeOptions()
#add option
op.add_argument('--enable-extensions')
#pass option to webdriver object
driver = webdriver.Chrome(chrome_options=op)

Website not opening with chrome driver but opening with geckodriver

I am trying to scrape the website. First of all, it is not working with Beautifulsoup but when I am trying to open it with selenium chrome driver it's not opening. It's opening with firefox but it's very slow and gives an error on element click. Here is my code:
from selenium import webdriver
opt = webdriver.ChromeOptions()
opt.add_argument("--disable-xss-auditor")
opt.add_argument("--disable-web-security")
opt.add_argument("--allow-running-insecure-content")
opt.add_argument("--no-sandbox")
opt.add_argument("--disable-setuid-sandbox")
opt.add_argument("--disable-webgl")
opt.add_argument("--disable-popup-blocking")
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(f"http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=25&tableName=TABLE25&title=%B9%FA%B2%FA%D2%A9%C6%B7&bcId=152904713761213296322795806604&CbSlDlH0=qGrYrAktn7.tn7.tnznJalIvVetjcXpaapSdKuqmmoVqqWL")
Possibly Selenium driven ChromeDriver initiated google-chrome Browsing Context is geting detected as bot and the arguments you have added can't bypass the bot detection mechanism effectively.
Solution
You can evade the detection by adding a few arguments and experimental_option as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get("http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=25&tableName=TABLE25&title=%B9%FA%B2%FA%D2%A9%C6%B7&bcId=152904713761213296322795806604&CbSlDlH0=qGrYrAktn7.tn7.tnznJalIvVetjcXpaapSdKuqmmoVqqWL")

How to correctly specify webdriver path with add_argument()?

This is my code:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--no-sandbox")
options.add_argument("C:/webdrivers/chromedriver.exe")
driver = webdriver.Chrome(options=options)
driver.get("https://www.google.com")
But it did not use the webdriver that I am trying to specify and uses some different one. How to correctly specify path to webdriver in the code above?
So main point here is that I want to specify path to webdriver and also use it without sandbox. How can I do it?
This worked:
from selenium import webdriver
# start the browser
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
options.add_argument("--no-sandbox")
# options.add_argument("--disable-dev-shm-usage")
# options.add_argument("--disable-gpu")
# options.add_argument("--window-size=1920,1080")
driver = webdriver.Chrome(executable_path=r"C:/webdrivers/chromedriver.exe", options=options)
driver.get("https://www.google.com")

How to disable chrome notifications popup in python and selenium?

How to disable chrome notifications popup in python and selenium?
I tried:
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(chrome_options=chrome_options)
But then also it shows notification...
I tried the same codes answered
here but then also I
am not able to disable the notifications!
The selenium package has a ChromeOptions class, in which you can add many arguments. One of which is 'disable-notifications'. You can pass that class to the driver class when initializing it.
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('disable-notifications')
driver = webdriver.Chrome('chromedriver.exe', options=chrome_options)
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('disable-notifications')
driver = webdriver.Chrome(executable_path="C:\Users\PycharmProjects\chromedriver_win32\chromedriver.exe", options=chrome_options)
driver.maximize_window()
This works with the right placement of code:
Click here for Image-->ChromeOptions Pycharm Code Image
You can pass disable-notifications to your chrome options.
Here's an example I have using Javascript, should work the same with Python.
var o = new chrome.Options();
o.addArguments('user-data-dir=./chromeprofile');
o.addArguments('disable-infobars');
o.addArguments("disable-notifications");
o.setUserPreferences( { credentials_enable_service: false } );

Headless not working for windows

I am trying to run headless Chrome and it running normally without headless it just runs Chrome as normal. I have updated chrome recently so that's not the issue.
driver.maximize_window()
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('headless')
browser = webdriver.Chrome(chrome_options=options)
driver.get('https://www.youtube.com.au')
You need to make a couple of changes to your code block as follows :
You need to maximize the browser once the webdriver opens the browser instance.
As per best practices to maximize the Chrome Window use Options() Class.
To maximize the Chrome Window through Options() Class either use the start-maximized argument or use the window-size argument.
Here is the sample code block for your reference:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("--headless");
options.add_argument("window-size=1400,600");
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("http://google.com/")
print ("Headless Chrome Initialized")

Categories

Resources