I want to have Selenium wait until a webpage is loaded. I use Safari 14.1.2 on macOS 10.15.7
So I tried as below:
// login process and it successes and the following page shows up
login_submit.click()
// the page after the authorization, make selenium wait until ```hoge``` is clickable
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.ID, 'hoge')))
This code does not work as expected. It fails with an error selenium.common.exceptions.NoSuchFrameException:. The error comes immediately, which seems there is no 10 s wait.
The stack trace is as below: (Paths are modified.)
Traceback (most recent call last):
File "/login.py", line 99, in <module>
login_schedule().quit()
File "/login.py", line 67, in login_schedule
wait.until(EC.presence_of_element_located((By.ID, 'HOGEHOGE')))
File "/selenium/webdriver/support/wait.py", line 71, in until
value = method(self._driver)
File "/selenium/webdriver/support/expected_conditions.py", line 64, in __call__
return _find_element(driver, self.locator)
File "/selenium/webdriver/support/expected_conditions.py", line 415, in _find_element
raise e
File "/selenium/webdriver/support/expected_conditions.py", line 411, in _find_element
return driver.find_element(*by)
File "/selenium/webdriver/remote/webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchFrameException: Message:
When I put time.sleep(10) instead as below,
import time
// login process and it successes and the following page shows up
login_submit.click()
time.sleep(10)
// the page after the authorization, make selenium wait until ```hoge``` is clickable
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.ID, 'hoge')))
it works as expected.
Note that the last part can even be replaced
browser.find_element_by_xpath("//select[#id='hoge']")
What is wrong with my Python code?
try to use this function to wait until the element or something appears:
def wait_for(xpath):
while True:
try:
driver.find_element_by_xpath(xpath)
return True
except NoSuchElementException:
continue
do not forget
from selenium.common.exceptions import NoSuchElementException
you can use anything it is not important to be XPATH as a method.
Related
I'm trying to extract 1000 usernames and the team rosters for each username in a list from this SPA (single page application) sports website (requires free login): https://rumble.otmnft.com/contest/live?contest=Rumble
I've tried numerous strategies to try to avoid the common StaleElementReferenceException problem in Selenium including try / except block, WebDriverWait and time.sleep(5) among many others, but none have worked. Would be amazing if someone knows an alternative solution which could work as I've been trying to solve this by myself for quite a few days now.
Here is the relevant part of the latest attempted code which still throws StaleElementReferenceException below:
parent_dict = {}
parent_dict['data'] = []
USERNAME_XPATH = "//span[#class='hidden lg:flex font-semibold']"
ROSTER_XPATH = "//span[#class='text-white hidden lg:flex font-semibold']"
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, "//table")))
username_elements = driver.find_elements(By.XPATH, USERNAME_XPATH)
try:
for usr_element in username_elements:
usr_element.click()
except StaleElementReferenceException:
# Find the element again and continue with the loop
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, USERNAME_XPATH)))
username_elements = driver.find_elements(By.XPATH, USERNAME_XPATH)
time.sleep(5)
for usr_element in username_elements:
usr_element.click()
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH, ROSTER_XPATH)))
roster_elements = driver.find_elements(By.XPATH, ROSTER_XPATH)
roster = [cell.text for cell in roster_elements]
TITLE_USER_XPATH = "//div[#class='flex flex-col space-y-2']/span"
title_user = driver.find_element(By.XPATH, TITLE_USER_XPATH)
child_dict = {"username": title_user.text, "roster": roster }
print("CHILD_DICT", child_dict)
parent_dict['data'].append(child_dict)
with open('data_v2.json', 'w') as outfile:
json.dump(parent_dict, outfile)
print("PARENT_DICT", parent_dict)
return parent_dict
In the browser I can see that the script is able to click on the very first username but it fails when trying click on the second username, but reports the error for the line usr_element.click() both inside the try and except blocks
Full error trace is:
Traceback (most recent call last):
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/utils.py", line 23, in sub_func
usr_element.click()
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 88, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <span class="hidden lg:flex font-semibold"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
Stacktrace:
RemoteError#chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:180:5
StaleElementReferenceError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:461:5
element.resolveElement#chrome://remote/content/marionette/element.sys.mjs:674:11
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:255:31
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:263:29
receiveMessage#chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:74:34
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/rumble_client.py", line 53, in <module>
sub_func(driver)
File "/Users/danpro12/Dropbox/01 FREE TIME STUFF/01 WEB DEVELOPMENT/12 DAPPER RELATED/otm_python_scraper/otm_specific/utils.py", line 30, in sub_func
usr_element.click()
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 88, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/Users/danpro12/.virtualenvs/otm/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <span class="hidden lg:flex font-semibold"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
Stacktrace:
RemoteError#chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:180:5
StaleElementReferenceError#chrome://remote/content/shared/webdriver/Errors.sys.mjs:461:5
element.resolveElement#chrome://remote/content/marionette/element.sys.mjs:674:11
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:255:31
evaluate.fromJSON#chrome://remote/content/marionette/evaluate.sys.mjs:263:29
receiveMessage#chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:74:34
Seems that after you do usr_element.click() DOM updated and so you cannot iterate over username_elements since all elements in this list are stale.
Try this solution
username_elements_length = len(driver.find_elements(By.XPATH, USERNAME_XPATH))
for usr_element_index in range(username_elements_length):
username_element = driver.find_elements(By.XPATH, USERNAME_XPATH)[usr_element_index]
username_element.click()
WebDriverWait(driver, 5).until(EC.staleness_of(username_element))
from selenium import webdriver
import time
DRIVER = 'chromedriver'
driver = webdriver.Chrome(DRIVER)
driver.get("https://www.tradingview.com/#signin")
driver.find_element_by_name("username").send_keys("Username")
driver.find_element_by_name("password").send_keys("Password")
driver.find_element_by_xpath("//button[#type='submit']").click()
driver.maximize_window()
driver.find_element_by_link_text('Chart').click()
driver.find_element_by_class_name('input-3lfOzLDc').send_keys('SBIN')
time.sleep(20)
driver.quit()
Everything is working fine here but when I try to find element which needs to be change( the one sending SBIN , I get this error
Traceback (most recent call last):
File "tview.py", line 11, in <module>
driver.find_element_by_xpath("//input[#class='input-3lfOzLDc']").send_keys('SBIN')
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
'value': value})['value']
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//input[#class='input-3lfOzLDc']"}
Can someone please help out
waiting a little bit longer works fine. I imported keys too because you had to clear that textbox
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
DRIVER = 'chromedriver'
driver = webdriver.Chrome(DRIVER)
driver.get("https://www.tradingview.com/#signin")
driver.find_element_by_name("username").send_keys("sibyl666")
driver.find_element_by_name("password").send_keys("417305628m")
driver.find_element_by_xpath("//button[#type='submit']").click()
driver.maximize_window()
driver.find_element_by_link_text('Chart').click()
time.sleep(5)
driver.find_element_by_class_name('input-3lfOzLDc').send_keys(Keys.CONTROL + "a")
driver.find_element_by_class_name('input-3lfOzLDc').send_keys(Keys.DELETE);
driver.find_element_by_class_name('input-3lfOzLDc').send_keys('SBIN')
time.sleep(20)
driver.quit()
this may be too late, but here it is what works for me. Basically, I use .clear() to get rid of the previous value and add some waiting times (depends on how fast the website loads on your machine).
ticker = driver.find_element_by_class_name('input-3lfOzLDc')
time.sleep(2)
ticker.clear()
time.sleep(2)
ticker.send_keys('SBIN')
time.sleep(2)
ticker.send_keys(Keys.RETURN)
So I'm losing my mind over here. I want to click on a specific button but there is an error which I don't understand:
Traceback (most recent call last):
File "C:\Users\leosc\PycharmProjects\ogameBot\guideline.py", line 22, in guideline
if metalMineLVL() > (10):
File "C:\Users\leosc\PycharmProjects\ogameBot\guideline.py", line 10, in <lambda>
metalMineLVL = lambda: c.metalMine.checkLVL()
File "C:\Users\leosc\PycharmProjects\ogameBot\classes.py", line 121, in checkLVL
a = self.sparte()
File "C:\Users\leosc\PycharmProjects\ogameBot\classes.py", line 54, in <lambda>
(By.XPATH, ('//*[contains(text(),\'{}\')]').format(sparte))))
File "C:\Users\leosc\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\support\wait.py", line 71, in until
value = method(self._driver)
File "C:\Users\leosc\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\support\expected_conditions.py", line 64, in __call__
return _find_element(driver, self.locator)
File "C:\Users\leosc\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\support\expected_conditions.py", line 415, in _find_element
raise e
File "C:\Users\leosc\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\support\expected_conditions.py", line 411, in _find_element
return driver.find_element(*by)
File "C:\Users\leosc\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\leosc\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\leosc\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchWindowException: Message: no such window: target window already closed
from unknown error: web view not found
(Session info: chrome=72.0.3626.121)
(Driver info: chromedriver=2.46.628402 (536cd7adbad73a3783fdc2cab92ab2ba7ec361e1),platform=Windows NT 10.0.17134 x86_64)
Here the code:
self.sparte = lambda: wait.until(
EC.presence_of_element_located(
(By.XPATH, ('//*[contains(text(),\'{}\')]').format(sparte))))
self.LVL = lambda: int(
wait.until(EC.presence_of_element_located((By.XPATH, ('//*
[#ref=\'' + str(ref) + '\']/span/span')))).text)
def checkLVL(self):
time.sleep(1)
self.sparte().click()
time.sleep(1)
return self.LVL()
metalMineLVL = lambda: c.metalMine.checkLVL()
roboFabLVL = lambda: c.roboFab.checkLVL()
try:
if metalMineLVL() > (10):
if roboFabLVL() <(5):
c.roboFab.build()
except:
pass
When I activate the debugger I see that every thing is processed until it gets to
self.sparte = lambda: wait.until(
EC.presence_of_element_located(
(By.XPATH, ('//*[contains(text(),\'{}\')]').format(sparte))))
and then it goes to the exception part. BUT I use this line of code in the rest of the program all the time and usually it works perfectly. What do I do wrong?
Thanks in advance!!
I don't know why you get that error. My best guess is that it's because of namespace conflicts but I'm not sure. This is the code I use for clicking on things:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
def clickElement(driver, XP):
WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, XP)))
driver.find_element_by_xpath(XP).click()
Example:
clickElement(driver, '/html/body/div[3]/div/a[1]')
Also check out PEP 498 for string formating.
I now found out that the problem was that the program opens 4 additional tabs and then closes the initial one. Normally, I then did:
newTabs = c.driver.window_handles
for tab in newTabs:
c.driver.switch_to.window(tab)
to get Selenium working on the new tabs. Yet, writing the new code, I forgor to do so, so Selenium thought it's still operating on the initial tab, which was already closed.
Thanks a lot anyway!!
I've wrote a code and works perfectly on my computer, then i decided to move it on a server cloud, and then I create an account on pythonanywhere.com
Then, after a lot of try I understand that chromedriver is not working over there, but just firefox, no big deal .
Then I did some test and works fine, but when I tried to use:
browser.find_element_by_link_text('Log in').click()
It doesn't work
well, my code is:`
from pyvirtualdisplay import Display
from selenium import webdriver
from time import sleep
with Display():
browser = webdriver.Firefox()
try:
browser.get('https://www.instagram.com')
print("get instagram.com ")
browser.implicitly_wait(60)
browser.find_element_by_link_text('Log in').click()
print("click log in")
sleep(10)
finally:
browser.quit()
the error is :
get instagram.com
Traceback (most recent call last):
File "/home/fobu36/.virtualenvs/prova_selenium.py", line 16, in <module>
browser.find_element_by_link_text('Log in').click()
File "/home/fobu36/.local/lib/python3.6/site-
packages/selenium/webdriver/remote/webdriver.py", l
ine 317, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "/home/fobu36/.local/lib/python3.6/site-
packages/selenium/webdriver/remote/webdriver.py", l
ine 752, in find_element
'value': value})['value']
File "/home/fobu36/.local/lib/python3.6/site-
packages/selenium/webdriver/remote/webdriver.py", l
ine 236, in execute
self.error_handler.check_response(response)
File "/home/fobu36/.local/lib/python3.6/site-
packages/selenium/webdriver/remote/errorhandler.py"
, line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: {"method":"l
ink text","selector":"Log in"}
Stacktrace:
at FirefoxDriver.findElementInternal_
(file:///tmp/tmpbydkaxfk/extensions/fxdriver#googlecode.
com/components/driver-component.js:10770)
at fxdriver.Timer.setTimeout/<.notify
(file:///tmp/tmpbydkaxfk/extensions/fxdriver#googlecode.
com/components/driver-component.js:625)
seems like that it can't find the log in button, but i'm sure that in local it's working
I have written this code using selenium python so parse a webpage(dynamically loading) so
that it goes till the end and then stop when "load more" button is not clickable anymore.
wait=WebDriverWait(driver,50)
wait.until(EC.element_to_be_clickable((By.ID, 'bottomPager'))) while
EC.element_to_be_clickable((By.ID,'bottomPager')):
driver.find_element_by_xpath('.//div[#id="bottomPager"]').click()
if not driver.find_element_by_id('bottomPager').is_enabled():
break
wait.until(EC.element_to_be_clickable((By.ID,'bottomPager')))
but I am getting the error when it reaches the end of the page
Traceback (most recent call last): File "python_org_search2.py", line 24, in <module>
driver.find_element_by_xpath('.//div[#id="bottomPager"]').click() File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 60, in click
self._execute(Command.CLICK_ELEMENT) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 370, in _execute
return self._parent.execute(command, params) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 166, in execute
self.error_handler.check_response(response) File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: u'unknown error: Element is not clickable at point (643, 628)\n (Session info: chrome=34.0.1847.137)\n (Driver info: chromedriver=2.9.248304,platform=Linux 3.13.0-24-generic x86_64)'
What are the changes required in code so as to rectify it ?
In my experience, this exception only occurs when using the Chrome driver with Selenium. Can you use Firefox instead? (I believe Firefox is the primary intended browser for webdriver invocations via Selenium.)
driver = webdriver.Firefox()
That should clear up this exception.
(You can get portable versions of Firefox at this site.)