Develop Reference

Python Selenium: Getting StaleElementReferenceException on click for SPA website

I'm trying to extract 1000 usernames and the team rosters for each username in a list from this SPA (single page application) sports website (requires free login): https://rumble.otmnft.com/contest/live?contest=Rumble
I've tried numerous strategies to try to avoid the common StaleElementReferenceException problem in Selenium including try / except block, WebDriverWait and time.sleep(5) among many others, but none have worked. Would be amazing if someone knows an alternative solution which could work as I've been trying to solve this by myself for quite a few days now.
Here is the relevant part of the latest attempted code which still throws StaleElementReferenceException below:
parent_dict = {}
parent_dict['data'] = []
USERNAME_XPATH = "//span[#class='hidden lg:flex font-semibold']"
ROSTER_XPATH = "//span[#class='text-white hidden lg:flex font-semibold']"
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, "//table")))
username_elements = driver.find_elements(By.XPATH, USERNAME_XPATH)
try:
for usr_element in username_elements:
usr_element.click()
except StaleElementReferenceException:
# Find the element again and continue with the loop
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, USERNAME_XPATH)))
username_elements = driver.find_elements(By.XPATH, USERNAME_XPATH)
time.sleep(5)
for usr_element in username_elements:
usr_element.click()
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, ROSTER_XPATH)))
roster_elements = driver.find_elements(By.XPATH, ROSTER_XPATH)
roster = [cell.text for cell in roster_elements]
TITLE_USER_XPATH = "//div[#class='flex flex-col space-y-2']/span"
title_user = driver.find_element(By.XPATH, TITLE_USER_XPATH)
child_dict = {"username": title_user.text, "roster": roster }
print("CHILD_DICT", child_dict)
parent_dict['data'].append(child_dict)
with open('data_v2.json', 'w') as outfile:
json.dump(parent_dict, outfile)
print("PARENT_DICT", parent_dict)
return parent_dict
In the browser I can see that the script is able to click on the very first username but it fails when trying click on the second username, but reports the error for the line usr_element.click() both inside the try and except blocks
Full error trace is:
Traceback (most recent call last):
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/utils.py", line 23, in sub_func
usr_element.click()
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 88, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <span class="hidden lg:flex font-semibold"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
Stacktrace:
RemoteError#chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:180:5
StaleElementReferenceError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:461:5
element.resolveElement#chrome://remote/content/marionette/element.sys.mjs:674:11
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:255:31
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:263:29
receiveMessage#chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:74:34
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/rumble_client.py", line 53, in <module>
sub_func(driver)
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/utils.py", line 30, in sub_func
usr_element.click()
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 88, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <span class="hidden lg:flex font-semibold"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
Stacktrace:
RemoteError#chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:180:5
StaleElementReferenceError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:461:5
element.resolveElement#chrome://remote/content/marionette/element.sys.mjs:674:11
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:255:31
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:263:29
receiveMessage#chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:74:34

Seems that after you do usr_element.click() DOM updated and so you cannot iterate over username_elements since all elements in this list are stale.
Try this solution
username_elements_length = len(driver.find_elements(By.XPATH, USERNAME_XPATH))
for usr_element_index in range(username_elements_length):
username_element = driver.find_elements(By.XPATH, USERNAME_XPATH)[usr_element_index]
username_element.click()
WebDriverWait(driver, 5).until(EC.staleness_of(username_element))

Stuck getting selenium code to click my link

I am trying to click a button but I am having an error when finding the element. In general I am trying to refresh a page until the link to join the waiting list goes live. When I run the script the webpage pulls up but does not refresh or click the join waitlist button when it is live.
I've tried finding element by class, link text, xpath, each time I get an error saying the element wasn't found.
code
while not joinList:
try:
addToCartBtn = addButton = driver.find_element_by_class_name("disabled")
print("Button isnt ready yet.")
time.sleep(2)
driver.refresh()
except:
addToCartBtn = addButton = driver.find_element_by_class_name("signup-btn")
print("Join")
addToCartBtn.click()
joinList = True
Where the element for the disabled join waitlist button is:
button _ngcontent-yur-c149="" disabled="" class="signup-btn ng-tns-c149-6 ng-star-inserted" style="background-color: rgb(0, 59, 153);"> Waitlist is currently closed </button
and the element for the enabled waitlist is:
button _ngcontent-yur-c149="" class="signup-btn ng-tns-c149-4 ng-star-inserted" tabindex="0" style="background-color: rgb(0, 59, 153);"> Join waitlist </button
The error I get is:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\Pycharm\MCattempt2\Main test folder\Test 1.py", line 20, in <module>
addToCartBtn = addButton = driver.find_element_by_class_name("signup-btn")
File "C:\Python\Pycharm\MCattempt2\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 564, in find_element_by_class_name
return self.find_element(by=By.CLASS_NAME, value=name)
File "C:\Python\Pycharm\MCattempt2\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Python\Pycharm\MCattempt2\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Python\Pycharm\MCattempt2\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".signup-btn"}
(Session info: chrome=88.0.4324.190)
Any advice on what I am doing wrong?

Get a Text from a Iframe with selenium and python

I need to get text from emails and save for send in webwhatsapp, but the text is in a iframe and returns a error when I try to get him
Thats the html code:
<div id="mailContent" class="mail-content" style="height: 724px;">
<div id="phishingLinkScanContent"></div>
<iframe id="bodyMailViewer"
class="body-mail-viewer" style="height: 719px;"
cd_frame_id_="18a0312a828e12092646c9a6b1ca58c4">
#document
<html><head></head>
<body>MetaTrader email test -> cilabassa#terra.com.br<br><div id="ozoneAttachments"></div></body>
</html>
</iframe>
</div>
Thats my code:
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
texto = driver.find_element_by_id('bodyMailViewer').text
Thats the error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="bodyMailViewer"]"}
(Session info: chrome=75.0.3770.142)

First switch to the iframe and then locate the body tag under iframe and finally get the text from body tag. Try the below code.
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
textInMail = driver.find_element_by_tag_name('body').text

Selenium can't find elements

I am trying to login to a site through Selenium. All the sites on which I try to log in, I could do this less. After putting this part of the code shows me a Python 2.7 error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 350, in send_keys
'value': keys_to_typing(value)})
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 499, in _execute
return self._parent.execute(command, params)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: element not visible
(Session info: chrome=61.0.3163.100)
(Driver info: chromedriver=2.32.498537 (cb2f855cbc7b82e20387eaf9a43f6b99b6105061),platform=Mac OS X 10.12.6 x86_64)
Code:
# coding=utf-8
from selenium import webdriver
driver = webdriver.Chrome('/Users/Martin/Desktop/chromedriver')
driver.get("https://car.com")
user= driver.find_element_by_xpath('//*[#id="onevideo_login_username"]')
user.send_keys("email")
Inspect:
<input type="text" id="onevideo_login_username" name="adaptv_email" data-i18n-placeholder="login.username" ng-model="models.login.email" class="ng-pristine ng-valid" placeholder="Username">
As I said before, I can login in all web sites I was trying, but this one is a headache!

Is the url correct?
If you have id,then its better to use,
Try:
user=driver.find_element_by_id('onevideo_login_username');

Please check the URL, I guess your URL is not correct. First of all I did not find any element like this in URL, but as per html I suggest you locator to locate element
<input type="text" id="onevideo_login_username" name="adaptv_email" data-i18n-placeholder="login.username" ng-model="models.login.email" class="ng-pristine ng-valid" placeholder="Username">
Use ID
user=driver.find_element_by_id('onevideo_login_username');
Use Name
user=driver.find_element_by_name('adaptv_email');
Use xpath
user= driver.find_element_by_xpath('//*[#name="adaptv_email"]')

how to delete youtube's watch later video in selenium?

I want to delete youtube's watch later videos. but my codes don't work.
<button class="yt-uix-button yt-uix-button-size-default yt-uix-button-default yt-uix-button-empty yt-uix-button-has-icon no-icon-markup pl-video-edit-remove yt-uix-tooltip" type="button" onclick=";return false;" title="Remove"></button>
my code is this.
_sDriver.get('https://www.youtube.com/playlist?list=WL')
_sDriver.find_element_by_class_name('pl-video-edit-remove').click()
exception is this.
>>> _sDriver.find_element_by_class_name('pl-video-edit-remove')
<selenium.webdriver.remote.webelement.WebElement (session="283d423c-5ff7-478f-89
b9-002499413126", element="{e00dbb1e-e0ca-4f79-8652-23c955b464e7}")>
>>> _sDriver.find_element_by_class_name('pl-video-edit-remove').click()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 72, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 461, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 236, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py"
, line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not c
urrently visible and so may not be interacted with
Stacktrace:
at fxdriver.preconditions.visible (file:///c:/users/admini~1/appdata/local/t
emp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command-processor.js
:10092)
at DelayedCommand.prototype.checkPreconditions_ (file:///c:/users/admini~1/a
ppdata/local/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/comman
d-processor.js:12644)
at DelayedCommand.prototype.executeInternal_/h (file:///c:/users/admini~1/ap
pdata/local/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command
-processor.js:12661)
at DelayedCommand.prototype.executeInternal_ (file:///c:/users/admini~1/appd
ata/local/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command-p
rocessor.js:12666)
at DelayedCommand.prototype.execute/< (file:///c:/users/admini~1/appdata/loc
al/temp/tmppqz9zq/extensions/fxdriver#googlecode.com/components/command-processo
r.js:12608)
I don't know what can I do for this.
I need your help.
thank you.

The element that you are trying to click is not visible normally. It is only visible when you hover the mouse on it. So either you fire a mouseover event before clicking and try if it works.
The other solution is that you can try executing the JavaScript.
driver.execute_script("document.getElementsByClassName('pl-video-edit-remove')[0].click()")
The script is working at my end.
you may want to add some wait before executing the script so that the page loads properly.