Develop Reference

Chrome crashes when selenium launches a website

I am trying to open a website on chrome via selenium with the following code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
s=Service('C:/Users/Morteza/Documents/Dev/chromedriver.exe')
browser = webdriver.Chrome(service=s)
url='https://www.google.com'
browser.get(url)
link to problem: https://share.cleanshot.com/p1qu5y

This is not an issue or crash. After the specified actions are completed successfully, selenium closes the web browser. This program works fine.
Use the following code with a while True block
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)

This isn't any crash or an error as such. Selenium automatically closes the client i.e. the Chrome Browser instance after executing the last line of your code. However this practice may accumulate undeleted/zombie chromedriver instances within your system.
Ideally, you always need to invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully at the end of your tests.

the chromedrive path in your code, i guess your platform is windows.
but the picture you post revease that your machine is mac.
pay attention to the chromedriver file PATH and your PC platform, make sure they suit to each other.
or you could put chromedriver in system PATH variable, so you do not need to specify the chromedrive path in your code

Chrome Webdriver Refresh does not respond since Chrome 106

Since I have upgraded to Chrome 106 (and now to 107) the driver manages to open the URL at the first time but after some timeout (?) or other criteria, the driver.refresh() does not return (i.e. no resposne).
The way it looks to me, if the Refresh occurs within seconds since the initial open, then the refresh succeeds.
Yet, if longer interval has elapsed since the initial open, then the refresh() does not complete.
Code is in Python.
Before version 106 this seemed to work well
Does anyone experience similar behaviour? Any idea how to fix that?
#Code to create the driver:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-blink-features")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(r'... \chromedriver', options=chrome_options)`
#Then, the attempt to refresh:
driver.refresh()
Expecting the webpage to refresh

Can you do it on google colab? try this
import sys sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=chrome_options)
driver.get(url)

if your work does not depend much on it, just use earlier versions where it used to work.
if not, try the following:
driver.find_element_by_tag_name('body').send_keys(Keys.COMMAND + 'r')

undetected_chromedriver runs slowly, suggestions?

I'm making a price scraping program and have ran into the issue of antiscraping systems. I managed to get around these with the undetected_chromedriver but now I'm running into 2 issues
the first is that the UC is significantly slower than the standard chrome driver, through I need it for some sites, so I have some sites scraped with a normal driver and others with the UC
the second problem is that I have the standard Chrome driver install at the beginning of the program, but once I do that, the UC feels the need to install every time I open it?? this causes some sites to be scraped really slowly. can you help with why that is? and any other tips for running scraper faster would be appreciated.
I have this run at the beginning of the program as global variables:
chrome_path = Service(ChromeDriverManager().install())
options = webdriver.ChromeOptions()
options.headless = True
options.add_experimental_option('excludeSwitches', ['enable-logging'])
and this runs as a function every time I need a UC:
def start_uc():
options = webdriver.ChromeOptions()
# just some options passing in to skip annoying popups
options.add_argument('--no-first-run --no-service-autorun --password-store=basic')
driver = uc.Chrome(options=options)
driver.minimize_window()
return driver
My scraping functions just loop looking up the url and scrape the info, and restart the driver to clear the cookies if I run into a captcha .The scraping functions look like this (this is psuedo code to give you an idea):
driver = start_uc()
for url in url_list:
while true:
try:
driver.get(url)
#scrape info
break
except:
driver.close()
driver = start_uc()
I dont see why chrome_path would affect the UC? and are there any suggestions to make the scraping functions run more efficiently? Im not an expert on drivers and their intricacies so I could be doing something terribly wrong that I dont recognize.
thankyou in advance!

You can use https://github.com/seleniumbase/SeleniumBase to speed things up.
(It has a special undetected-chromedriver mode that works with headless mode.)
pip install -U seleniumbase
And then run the following with python:
from seleniumbase import Driver
from seleniumbase import page_actions
driver = Driver(headless=True, uc=True)
driver.get("https://nowsecure.nl")
page_actions.wait_for_text(driver, "OH YEAH, you passed!", "h1")
print(driver.find_element("css selector", "body").text)
screenshot_name = "now_secure_image.png"
driver.save_screenshot(screenshot_name)
print("\nScreenshot saved to: %s" % screenshot_name)
driver.quit()

Code stops executing after opening browser through webdriver using python

I have been trying to open multiple browser windows in internet explorer using webdriver in selenium. Once it reaches the get(url) line, it just halts there and eventually times out. I've added a print line, which does not execute. I've tried various methods and the one below is the Ie version of code I used to open multiple tabs in Chrome. Even if I remove the first 3 lines, it still only goes up to opening google.com. I've looked googled this issue and looked through other posts but nothing has helped. Would really appreciate any advice, thanks!
options = webdriver.IeOptions()
options.add_additional_option("detach", True)
driver = webdriver.Ie(options = options, executable_path=r'blahblah\IEDriverServer.exe')
driver.get("http://google.com")
print("syrfgf")
driver.execute_script("window.open('about:blank', 'tab2');")
driver.switch_to.window("tab2")
driver.get("http://yahoo.com")

You need to replace the url you have provided:
http://google.com
with a proper url as follows:
https://www.google.com/
Which should be represented as per the syntax diagram as follows:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome()
driver.get("https://github.com")
signin_link = driver.find_element(By.LINK_TEXT, "Sign in")
signin_link.click()
time.sleep(1)
user = driver.find_element(By.ID, "login_field")
user.send_keys("X")
passw = driver.find_element(By.ID, "password")
passw.send_keys("X")
passw.submit()
time.sleep(5)
driver.close()
I had this issue and writing this code seems to have made it work flawlessly. Adjust the sleep time as you want it. Putting my chromedriver.exe into my project folder also helped with some errors

Selenium working with Chrome, but not headless Chrome

I've developed a couple of Python scripts using Selenium and, at first, PhantomJS. While heading toward automated downloads, I switched for (headed) Firefox (which worked) and then Chrome with the headless option so I won't have the browser opening in front of me.
My first script, which accesses a page and a couple of HTML elements, works perfectly with headless Chrome.
The second one, however, works only with headed Chrome. If I add the "headless" option, it doesn't work anymore. When I try to print the HTML in headless mode to see why it cannot find the HTML element I'm looking for, all I have is :
<html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>
With the headed Chrome, I have a complete HTML printed.
This is how I start my headless Chrome :
options = webdriver.ChromeOptions()
options.add_argument("--ignore-certificate-errors")
options.add_argument("headless")
driver = webdriver.Chrome(chrome_options=options)
Again, note that this works in another of my script. The only difference here is that I need to log in to access the page, but even then, why would it work with the head? My script is made to log in automatically anyway by filling the form.
Python : 3.6.1, Chrome : 60.0.3112.78 (64 bits), Selenium : 3.4.3
Any idea?
Thanks.
** EDIT: Here is the beginning of the code**
url = 'https://10.11.227.21/tmui/'
driver.get(url + "login.jsp")
html_source = driver.page_source
print(html_source)
blocStatus = WebDriverWait(driver, TIMEOUT).until(EC.presence_of_element_located((By.ID, "username")))
inputElement = driver.find_element_by_id("username")
inputElement.send_keys('actualLogin')
inputElement = driver.find_element_by_id("passwd")
inputElement.send_keys('actualPassword')
inputElement.submit()

I had a same experience like you, and solved it by using xvfb and pyvirtualdisplay.
I use chromedrive=v2.3.1, chrome-browser=v60 and Selenium=3.4.3
In Headless chrome, some of script seems not to work as expected.
Please refer to vpassapera's comment in https://gist.github.com/addyosmani/5336747.
How about try it like below,
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
# Do Not use headless chrome option
# options.add_argument('headless')
url = 'https://10.11.227.21/tmui/'
driver.get(url + "login.jsp")
html_source = driver.page_source
print(html_source)
blocStatus = WebDriverWait(driver, TIMEOUT).until(EC.presence_of_element_located((By.ID, "username")))
inputElement = driver.find_element_by_id("username")
inputElement.send_keys('actualLogin')
inputElement = driver.find_element_by_id("passwd")
inputElement.send_keys('actualPassword')
inputElement.submit()
display.stop()
xvfb is required to use "pyvortualdisplay"
$ sudo apt-get install -y xvfb

Headless Chrome does not support insecure certificates and hence, websites with insecure certificates does not open living it blank. You need to add capabilities as follow:
from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
capabilities = DesiredCapabilities.CHROME.copy()
capabilities['acceptSslCerts'] = True
capabilities['acceptInsecureCerts'] = True
driver = webdriver.Chrome(chrome_options = chrome_options,executable_path='your path',desired_capabilities=capabilities)
driver.get("yourWebsite")
This will do the work.

I had the same issue and setting the window size in conftest.py solved it.
My code snippet:
#pytest.fixture
def chrome_options(chrome_options, pytestconfig):
if pytestconfig.getoption('headless'):
chrome_options.add_argument('--headless')
chrome_options.add_argument("window-size=1920,1080")
else:
chrome_options.add_argument("start-maximized");
return chrome_options

Headless chrome may be faster on same machine than headed, try adding some wait before locating password element.

For some if you remove the below, it would work.
driver.fullscreen_window()

refer https://github.com/SeleniumHQ/selenium/issues/4477
add below code
self.chrome_options = webdriver.ChromeOptions()
self.chrome_options.add_argument("--window-size=1920,1080")
self.chrome_options.add_argument("--disable-extensions")
self.chrome_options.add_argument("--proxy-server='direct://'")
self.chrome_options.add_argument("--proxy-bypass-list=*")
self.chrome_options.add_argument("--start-maximized")
self.chrome_options.add_argument('--headless')
self.chrome_options.add_argument('--disable-gpu')
self.chrome_options.add_argument('--disable-dev-shm-usage')
self.chrome_options.add_argument('--no-sandbox')
self.chrome_options.add_argument('--ignore-certificate-errors')
self.browser = webdriver.Chrome(options=self.chrome_options)

For my situation, headless did not work because I was behind a proxy. Apparently, Chrome is able to use the system proxy, but headless does not use the system proxy.
I simply needed to provide the proxy, then headless (as well as Chrome) worked.
options.add_argument('--proxy-server=http://myproxy:port')