Develop Reference

Python Selenium: Getting StaleElementReferenceException on click for SPA website

I'm trying to extract 1000 usernames and the team rosters for each username in a list from this SPA (single page application) sports website (requires free login): https://rumble.otmnft.com/contest/live?contest=Rumble
I've tried numerous strategies to try to avoid the common StaleElementReferenceException problem in Selenium including try / except block, WebDriverWait and time.sleep(5) among many others, but none have worked. Would be amazing if someone knows an alternative solution which could work as I've been trying to solve this by myself for quite a few days now.
Here is the relevant part of the latest attempted code which still throws StaleElementReferenceException below:
parent_dict = {}
parent_dict['data'] = []
USERNAME_XPATH = "//span[#class='hidden lg:flex font-semibold']"
ROSTER_XPATH = "//span[#class='text-white hidden lg:flex font-semibold']"
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, "//table")))
username_elements = driver.find_elements(By.XPATH, USERNAME_XPATH)
try:
for usr_element in username_elements:
usr_element.click()
except StaleElementReferenceException:
# Find the element again and continue with the loop
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, USERNAME_XPATH)))
username_elements = driver.find_elements(By.XPATH, USERNAME_XPATH)
time.sleep(5)
for usr_element in username_elements:
usr_element.click()
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, ROSTER_XPATH)))
roster_elements = driver.find_elements(By.XPATH, ROSTER_XPATH)
roster = [cell.text for cell in roster_elements]
TITLE_USER_XPATH = "//div[#class='flex flex-col space-y-2']/span"
title_user = driver.find_element(By.XPATH, TITLE_USER_XPATH)
child_dict = {"username": title_user.text, "roster": roster }
print("CHILD_DICT", child_dict)
parent_dict['data'].append(child_dict)
with open('data_v2.json', 'w') as outfile:
json.dump(parent_dict, outfile)
print("PARENT_DICT", parent_dict)
return parent_dict
In the browser I can see that the script is able to click on the very first username but it fails when trying click on the second username, but reports the error for the line usr_element.click() both inside the try and except blocks
Full error trace is:
Traceback (most recent call last):
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/utils.py", line 23, in sub_func
usr_element.click()
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 88, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <span class="hidden lg:flex font-semibold"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
Stacktrace:
RemoteError#chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:180:5
StaleElementReferenceError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:461:5
element.resolveElement#chrome://remote/content/marionette/element.sys.mjs:674:11
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:255:31
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:263:29
receiveMessage#chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:74:34
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/rumble_client.py", line 53, in <module>
sub_func(driver)
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/utils.py", line 30, in sub_func
usr_element.click()
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 88, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <span class="hidden lg:flex font-semibold"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
Stacktrace:
RemoteError#chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:180:5
StaleElementReferenceError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:461:5
element.resolveElement#chrome://remote/content/marionette/element.sys.mjs:674:11
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:255:31
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:263:29
receiveMessage#chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:74:34

Seems that after you do usr_element.click() DOM updated and so you cannot iterate over username_elements since all elements in this list are stale.
Try this solution
username_elements_length = len(driver.find_elements(By.XPATH, USERNAME_XPATH))
for usr_element_index in range(username_elements_length):
username_element = driver.find_elements(By.XPATH, USERNAME_XPATH)[usr_element_index]
username_element.click()
WebDriverWait(driver, 5).until(EC.staleness_of(username_element))

How to edit chromes search and homepage with selenium/python?

I am trying to edit chrome browsers search and homepage using selenium/python. After navigating to chrome://settings/searchEngines and targeting the 'add' button with the ID 'addSearchEngine', I get an error when I run a .click function. How do I target this element correctly, or is there another way to update chromes search/startpage with python?
I'm guessing this element is trapped inside an iframe but I'm unable to find one on the page using the dev tools, xpath noted the following about the absolute xpath: "It might be a child of iframe from different src & it is not supported currently."
from selenium import webdriver
driver = webdriver.Chrome()
driver.set_page_load_timeout(10)
driver.get("chrome://settings/searchEngines")
driver.find_element_by_id("addSearchEngine").click()
Traceback (most recent call last):
File "C:/Users/Jonathan/PycharmProjects/test_project/test_project/Main.py", line 20, in <module>
driver.find_element_by_id("addSearchEngine").click()
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="addSearchEngine"]"}
(Session info: chrome=75.0.3770.142)

chrome://settings/searchEngines has Shadow DOM elements . You will need to use the driver.execute_script() to get handle of the shadowRoot element and ultimately get to the 'addSearchEngine' element.
Example python : shadowRoot python
Example Java : For chrome://downloads/ shadowRoot java

Refer to this post for detailed explanation.
In your case you can do the below.
url = "chrome://settings/searchEngines"
driver.get(url)
addButton = driver.execute_script("return document.querySelector('settings-ui')"
".shadowRoot.querySelector('#main')"
".shadowRoot.querySelector('settings-basic-page.showing-subpage')"
".shadowRoot.querySelector('settings-search-page')"
".shadowRoot.querySelector('settings-search-engines-page')"
".shadowRoot.querySelector('#addSearchEngine')")
addButton.click()
Screenshot:

Error when scrolling in selenium with python

I am trying to scroll to the bottom of the page, as I usually do. But on this website it seems not to work. It's one of those where you need to scroll to the bottom in order to load new links.
This is the command I am using:
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
The error I am getting is:
Traceback (most recent call last):
File "/somePathTo/someProgram.py", line 12, in <module>
browser.execute_script("window.scrollTo(0,document.body.scrollHeight);")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
'args': converted_args})['value']
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Cannot read property 'offsetTop' of null
(Session info: chrome=73.0.3683.86)
(Driver info: chromedriver=72.0.3626.69 (3c16f8a135abc0d4da2dff33804db79b849a7c38),platform=Mac OS X 10.14.3 x86_64)
Any other scroll does not work as well, something is wrong with the JS script...
I don't know what element is null and therefore can't have the 'offsetTop' property... maybe I should identify some element that can have that property, but I don't know much more about JS than this script
EDIT:
I did solve my specific problem with this code
browser.execute_script('arguments[0].scrollIntoView(true);', target)
where target is an element to the bottom. However, I would like to know what the issue is with the JS code.

how to delete youtube's watch later video in selenium?

I want to delete youtube's watch later videos. but my codes don't work.
<button class="yt-uix-button yt-uix-button-size-default yt-uix-button-default yt-uix-button-empty yt-uix-button-has-icon no-icon-markup pl-video-edit-remove yt-uix-tooltip" type="button" onclick=";return false;" title="Remove"></button>
my code is this.
_sDriver.get('https://www.youtube.com/playlist?list=WL')
_sDriver.find_element_by_class_name('pl-video-edit-remove').click()
exception is this.
>>> _sDriver.find_element_by_class_name('pl-video-edit-remove')
<selenium.webdriver.remote.webelement.WebElement (session="283d423c-5ff7-478f-89
b9-002499413126", element="{e00dbb1e-e0ca-4f79-8652-23c955b464e7}")>
>>> _sDriver.find_element_by_class_name('pl-video-edit-remove').click()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 72, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 461, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 236, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py"
, line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not c
urrently visible and so may not be interacted with
Stacktrace:
at fxdriver.preconditions.visible (file:///c:/users/admini~1/appdata/local/t
emp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command-processor.js
:10092)
at DelayedCommand.prototype.checkPreconditions_ (file:///c:/users/admini~1/a
ppdata/local/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/comman
d-processor.js:12644)
at DelayedCommand.prototype.executeInternal_/h (file:///c:/users/admini~1/ap
pdata/local/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command
-processor.js:12661)
at DelayedCommand.prototype.executeInternal_ (file:///c:/users/admini~1/appd
ata/local/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command-p
rocessor.js:12666)
at DelayedCommand.prototype.execute/< (file:///c:/users/admini~1/appdata/loc
al/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command-processo
r.js:12608)
I don't know what can I do for this.
I need your help.
thank you.

The element that you are trying to click is not visible normally. It is only visible when you hover the mouse on it. So either you fire a mouseover event before clicking and try if it works.
The other solution is that you can try executing the JavaScript.
driver.execute_script("document.getElementsByClassName('pl-video-edit-remove')[0].click()")
The script is working at my end.
you may want to add some wait before executing the script so that the page loads properly.

Selenium error At the end of the page

I have written this code using selenium python so parse a webpage(dynamically loading) so
that it goes till the end and then stop when "load more" button is not clickable anymore.
wait=WebDriverWait(driver,50)
wait.until(EC.element_to_be_clickable((By.ID, 'bottomPager'))) while
EC.element_to_be_clickable((By.ID,'bottomPager')):
driver.find_element_by_xpath('.//div[#id="bottomPager"]').click()
if not driver.find_element_by_id('bottomPager').is_enabled():
break
wait.until(EC.element_to_be_clickable((By.ID,'bottomPager')))
but I am getting the error when it reaches the end of the page
Traceback (most recent call last): File "python_org_search2.py", line 24, in <module>
driver.find_element_by_xpath('.//div[#id="bottomPager"]').click() File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 60, in click
self._execute(Command.CLICK_ELEMENT) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 370, in _execute
return self._parent.execute(command, params) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 166, in execute
self.error_handler.check_response(response) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: u'unknown error: Element is not clickable at point (643, 628)\n (Session info: chrome=34.0.1847.137)\n (Driver info: chromedriver=2.9.248304,platform=Linux 3.13.0-24-generic x86_64)'
What are the changes required in code so as to rectify it ?

In my experience, this exception only occurs when using the Chrome driver with Selenium. Can you use Firefox instead? (I believe Firefox is the primary intended browser for webdriver invocations via Selenium.)
driver = webdriver.Firefox()
That should clear up this exception.
(You can get portable versions of Firefox at this site.)