Develop Reference

Selenium gives "Timed out receiving message from renderer" for all websites after some execution time

I have an application where I need a long running instance of Selenium web driver (I am using Chrome driver 83.0.4103.39 in headless mode). Basically the app continuously pull url-data from a queue and gives the extracted url to Selenium which should perform some analysis on the website. Many of these websites could be down, unreachable or broken, so I've put a page load timeout of 10 seconds to avoid Selenium wait forever for page load.
The problem I am having here is that after some execution time (let's say 10 minutes) Selenium starts to give Timed out receiving message from renderer error for every url. Initially it works properly, it correctly opens the good websites and goes on timeout on the bad ones (website fails to load), but after some time it starts to give timeout on everything, even websites that should open correctly (I've checked, they open correctly on Chrome browser).
I am having hard time to debug this problem, since every exception in the application is caught correctly. I have also noticed that this problem happens only in headless mode.
UPDATE *
During website analysis I also need to consider iframes (only top level), thus I've also added a logic to switch driver context to each iframe in the main page and extract the relative html.
This is a simplified version of the application:
import traceback
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
width = 1024
height = 768
chrome_options = Options()
chrome_options.page_load_strategy = 'normal'
chrome_options.add_argument('--enable-automation')
chrome_options.add_argument('disable-infobars')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--lang=en')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-insecure-localhost')
chrome_options.add_argument('--allow-running-insecure-content')
chrome_options.add_argument('--disable-notifications')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--disable-browser-side-navigation')
chrome_options.add_argument('--mute-audio')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--force-device-scale-factor=1')
chrome_options.add_argument(f'window-size={width}x{height}')
chrome_options.add_experimental_option(
'prefs', {
'intl.accept_languages': 'en,en_US',
'download.prompt_for_download': False,
'download.default_directory': '/dev/null',
'automatic_downloads': 2,
'download_restrictions': 3,
'notifications': 2,
'media_stream': 2,
'media_stream_mic': 2,
'media_stream_camera': 2,
'durable_storage': 2,
}
)
driver = webdriver.Chrome(options=options)
driver.set_page_load_timeout(10) # Timeout 10 seconds
# Polling queue
while True:
url = queue.pop()
# Try open url
try:
driver.get(url)
except BaseException as e:
print(e)
print(traceback.format_exc())
continue
# Take website screenshot
png = driver.get_screenshot_as_png()
# Extract html from iframes (if any)
htmls = [driver.page_source]
iframes = driver.find_elements_by_xpath("//iframe")
for index, iframe in enumerate(iframes):
try:
driver.switch_to.frame(index)
htmls.append(driver.page_source)
driver.switch_to.default_content()
except BaseException as e:
print(e)
print(traceback.format_exc())
continue
# Do some analysis
for html in htmls:
# ...
pass
# Wait a bit
sleep(0.1)
This is an example of stack trace:
Opening https://www.yourmechanic.com/user/appointment/3732777/?access_token=HLZYIg&ukey=6quWpg1724633&rcode=abttgi&utm_medium=sms&utm_source= rb
LOAD EXCEPTION Message: timeout: Timed out receiving message from renderer: 10.000
(Session info: headless chrome=83.0.4103.116)
Traceback (most recent call last):
File "/Users/macbmacbookpro4ookpro4/Documents/Projects/python/proj001/main.py", line 202, in inference
driver.get(url)
File "/opt/anaconda3/envs/cv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/opt/anaconda3/envs/cv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/opt/anaconda3/envs/cv/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 10.000
(Session info: headless chrome=83.0.4103.116)
Has anyone any clue why after a while of correct execution Selenium starts to give timeout exception for any url it tries to open?

This error message...
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 10.000
...implies that the ChromeDriver was unable to communicate with the Browsing Context i.e. Chrome Browser session.
Deep Dive
This error can arise due to several reasons. A couple of those reasons and the remedy are as follows:
disable-infobars and --enable-automation are almost analogous and disable-infobars is no more amaintained. --enable-automation will serve your purpose. So you need to drop:
chrome_options.add_argument('disable-infobars')
You can find a detailed discussion in Unable to hide “Chrome is being controlled by automated software” infobar within Chrome v76
--enable-automation is still a experimental_option so you need to:
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
You can find a detailed discussion in How can I use setExperimentalOption through Options using FirefoxDriver in Selenium IDE?
If you intend to use --enable-automation you need to use useAutomationExtension as well:
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
--disable-gpu is no longer necessary, so you need to drop:
chrome_options.add_argument('--disable-gpu')
You can find a detailed discussion in Chrome Options in Python Selenium : Disable GPU vs Headless
You can opt use a bigger Viewport through {width}x{height} e.g. 1920, 1080
chrome_options.add_argument("window-size=1920,1080")
You can find a detailed discussion in How to set window size in Selenium Chrome Python
To initiate a google-chrome-headless instead of chrome_options.add_argument('--headless') you need to use the headless attribute as follows:
chrome_options.headless = True
You can find a detailed discussion in How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
As you have enumerated all the iframe elements, it is worth to mention you can't switch_to all the <iframe> / <frame> as some of them may have the style attribute value set as display: none;.
You can find a detailed discussion in Expected condition failed: waiting for element to be clickable for element containing style=“display: none;”
Finally, to switch to a frame you need to induce WebDriverWait for the desired frame to be available and switch to it() as follows:
WebDriverWait(driver, 30).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#whovaIframeSpeaker")))
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
References
You can find a couple of relevant detailed discussions on Timed out receiving message from renderer in:
Timed out receiving message from renderer: 10.000
How to handle “Unable to receive message from renderer” in chrome driver?
Timed out receiving message from renderer

Getting error while opening a page using selenium webdriver [duplicate]

When I'm executing this code with Selenium using Python:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome(executable_path=r'/Users/qa/Documents/Python/chromedriver')
The error occurred:
Traceback (most recent call last):
File "/Users/qa/Documents/Python/try.py", line 4, in <module>
driver = webdriver.Chrome(executable_path=r'/Users/qa/Documents/Python/chromedriver')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
(Session info: chrome=71.0.3578.98)
(Driver info: chromedriver=2.44.609545 (c2f88692e98ce7233d2df7c724465ecacfe74df5),platform=Mac OS X 10.13.6 x86_64)
Can someone help me? Thanks.

I had a similar error, first getting the error message:
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited normally.
(unknown error: DevToolsActivePort file doesn't exist)
This was solved by adding options.add_argument("--remote-debugging-port=9230") to the ChromeOptions(). And the program runs once and I gained the same error message as above:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
(Session info: headless chrome=89.0.4389.114)
The problem here was that the chrome process does not close properly in the program so the process is still active on the debugging-port. To solve this problem close the active port sudo kill -9 $(sudo lsof -t -i:9230) and simply add the following lines to the end of the code:
driver.stop_client()
driver.close()
driver.quit()
Since I didn't find this answer anywhere, I hope it helps someone.

If you have options.add_argument("--remote-debugging-port=9222") change this to options.add_argument("--remote-debugging-port=9230")
or just simply Adding options.add_argument("--remote-debugging-port=9230") fixed in my case.

This error message...
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
...implies that the ChromeDriver was unable to initiate/spawn a new WebBrowser i.e. Chrome Browser session.
You need to consider a fact:
As you are using Mac OS X the Key executable_path must be supported with a Value as :
'/Users/qa/Documents/Python/chromedriver'
So line will be:
driver = webdriver.Chrome(executable_path='/Users/qa/Documents/Python/chromedriver')
Note: The path itself is a raw path so you don't need to add the switch r and drop it.
Additionally, ensure that /etc/hosts on your system contains the following entry :
127.0.0.1 localhost.localdomain localhost
#or
127.0.0.1 localhost loopback

I ran into this same issue on a Windows 10 machine. What I had to do to resolve the issue was to open the Task Manager and exit all Python.exe processes, along with all Chrome.exe processes. After doing this,

I am facing the same error and add the close code solve the problem, the code finnaly block look like this:
#staticmethod
def fetch_music_download_url(music_name: str):
chrome_driver_service = Service(ChromeDriverManager(chrome_type=ChromeType.GOOGLE).install())
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--remote-debugging-port=9230")
driver = webdriver.Chrome(service=chrome_driver_service, options=chrome_options)
try:
driver.maximize_window()
driver.get('http://example.cn/music/?page=audioPage&type=migu&name=' + music_name)
driver.implicitly_wait(5)
// do some logic
except Exception as e:
logger.error(e)
finally:
// add the close logic
driver.stop_client()
driver.close()
driver.quit()
chrome_driver_service.stop()
the key is you should close the chrome service after using it by add chrome_driver_service.stop().hope this code could help other people facing the same issue.

This worked for me on WINDOWS OS
from selenium import webdriver
import time
from bs4 import BeautifulSoup
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(options=options)
driver.get(url)
time.sleep(3)
page = driver.page_source
driver.quit()
soup = BeautifulSoup(page, 'html.parser')
Hope you'd find this useful. I also used it more comprehensively here.

Selenium driver.requests.Timeout does not stop the request after load timeout [duplicate]

In selenium tests, you open a webpage using
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("whateverpage.org.com")
How can I set the timeout of this command for selenium version 3.8.0 and python 2.7.12?

To set the time out for Page Loading you can induce the set_page_load_timeout(seconds).
set_page_load_timeout
Method Details
def set_page_load_timeout(self, time_to_wait):
"""
Set the amount of time to wait for a page load to complete
before throwing an error.
Args
time_to_wait: The amount of time to wait
Usage
driver.set_page_load_timeout(3)
Example
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome(executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.set_page_load_timeout(2)
try :
driver.get("https://www.booking.com/hotel/in/the-taj-mahal-palace-tower.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYATG4AQbIAQzYAQHoAQH4AQKSAgF5qAID;sid=338ad58d8e83c71e6aa78c67a2996616;dest_id=-2092174;dest_type=city;dist=0;group_adults=2;hip_dst=1;hpos=1;room1=A%2CA;sb_price_type=total;srfid=ccd41231d2f37b82d695970f081412152a59586aX1;srpvid=c71751e539ea01ce;type=total;ucfs=1&#hotelTmpl")
print("URL successfully Accessed")
driver.quit()
except TimeoutException as e:
print("Page load Timeout Occurred. Quitting !!!")
driver.quit()
Console Output
Page load Timeout Occurred. Quitting !!!
Documentation
You can find a detailed discussion on pageLoadTimeout here pageLoadTimeout in Selenium not working
Deep Dive
As per Python 3.x if we don't handle the exception the following log messages are observed :
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout
(Session info: chrome=62.0.3202.94)
(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 6.2.9200 x86_64)

How to set the timeout of 'driver.get' for python selenium 3.8.0?

In selenium tests, you open a webpage using
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("whateverpage.org.com")
How can I set the timeout of this command for selenium version 3.8.0 and python 2.7.12?

To set the time out for Page Loading you can induce the set_page_load_timeout(seconds).
set_page_load_timeout
Method Details
def set_page_load_timeout(self, time_to_wait):
"""
Set the amount of time to wait for a page load to complete
before throwing an error.
Args
time_to_wait: The amount of time to wait
Usage
driver.set_page_load_timeout(3)
Example
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome(executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.set_page_load_timeout(2)
try :
driver.get("https://www.booking.com/hotel/in/the-taj-mahal-palace-tower.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYATG4AQbIAQzYAQHoAQH4AQKSAgF5qAID;sid=338ad58d8e83c71e6aa78c67a2996616;dest_id=-2092174;dest_type=city;dist=0;group_adults=2;hip_dst=1;hpos=1;room1=A%2CA;sb_price_type=total;srfid=ccd41231d2f37b82d695970f081412152a59586aX1;srpvid=c71751e539ea01ce;type=total;ucfs=1&#hotelTmpl")
print("URL successfully Accessed")
driver.quit()
except TimeoutException as e:
print("Page load Timeout Occurred. Quitting !!!")
driver.quit()
Console Output
Page load Timeout Occurred. Quitting !!!
Documentation
You can find a detailed discussion on pageLoadTimeout here pageLoadTimeout in Selenium not working
Deep Dive
As per Python 3.x if we don't handle the exception the following log messages are observed :
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout
(Session info: chrome=62.0.3202.94)
(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 6.2.9200 x86_64)

selenium: chrome failed to start when using options

I am trying to use selenium with chrome web driver. When I run it without the options it works fine:
>>>driver = webdriver.Chrome(executable_path=path)
but when I add options (loaded profile) like this:
>>> options = webdriver.ChromeOptions()
>>> options.add_argument('--user-data-dir=%s' % chromeProfile)
>>> driver = webdriver.Chrome(executable_path=path,chrome_options=options)
it starts the browser window, which seems alive, but the command hangs until it finally declares:
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'unknown error: Chrome failed to start: exited normally\n (Driver info: chromedriver=2.9.248315,platform=Windows NT 6.1 SP1 x86_64)'
what could be causing this problem and how can I go about debugging it?
Thanks

OK - figured it out. The problem seems to be that one profile cant be opened while trying to connect a webdriver to it. I copied the default profile to a new location, and now it works fine. I'll leave this here in case this is useful to anyone.