driver = webdriver.Chrome(service=s)
url="https://fourminutebooks.com/book-summaries/"
driver.get(url)
page_tabs = driver.find_elements(By.CSS_SELECTOR, "a[class='post_title w4pl_post_title']")
#html = driver.find_elements(By.CSS_SELECTOR,"header[class='entry-header page-header']")
length_page_tabs = len(page_tabs)
in_length = len(page_tabs)
for i in range(length_page_tabs):
ran = random.randint(0,in_length)
page_tabs[ran].click()
driver.execute_script("window.history.go(-1)")
time.sleep(10)
#need to get page source of html and then open it to a new file, extract what I want and add it to the email
I am trying to click one of the links, get the html code, email it to myself, and then go back a page and repeat. However after clicking the first random link, the code stops working and instead I get this error
You have to be very careful, when you put some elements collection to the variable, and going to iterate and perform some actions.
page_tabs = driver.find_elements...
All the elements in this case are cached, and each web browser action of navigate to another page, refrech the page, etc. will make all of these cached elements stale. This means they bacame like out-of-date and not possible to interact them any more.
So, to avoid stale element reference errors, you have to prevent any page reloads, or just refresh the elements every time after the page state has been changed.
StaleElementReferenceException
StaleElementReferenceException is a type of WebDriverException which is thrown when a reference to an element have gone stale, i.e. the element no longer appears on the HTML DOM of the page.
Some of the possible causes of StaleElementReferenceException include:
You are no longer on the same page, or the page may have refreshed since the element was last located.
The element may have been removed and re-added to the DOM Tree, since it was located. Such as an element being relocated. This can happen typically with a javascript framework when values are updated and the node is rebuilt.
Element may have been inside an iframe or another context which was refreshed.
This usecase
In your usecase, you have created a list of webelement i.e. page_tabs using the locator strategy:
page_tabs = driver.find_elements(By.CSS_SELECTOR, "a[class='post_title w4pl_post_title']")
Next within the loop whenever you invoke click on page_tabs[ran] you are redirected to a new page, where the elements within the list page_tabs becomes stale and new elements are loaded.
Moving forward when you invoke driver.execute_script("window.history.go(-1)") you are moving back to the main page where the elements of page_tabs were present and they reload again. At this point of time, the list page_tabs still continues to hold the webelements of the previous search, which have now become stale. Hence during the second iteration you face StaleElementReferenceException
Solution
In your usecase to avoid StaleElementReferenceException as the desired elements are <A> tag so instead of saving the elements you can store the href attributes in a list and invoke get(href) as follows:
driver.get("https://fourminutebooks.com/book-summaries/")
hrefs = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[class='post_title w4pl_post_title']")))]
for href in hrefs:
driver.get(href)
print("Placeholder to perform the desired operations on the respective page")
driver.quit()
References
You can find a couple of relevant detailed discussions in:
StaleElementException when iterating with Python
Message: stale element reference: element is not attached to the page document in Python
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document with Selenium and Python
Use driver.execute_script and javascript. Javascript is never stale because it evaluates right away. In other words, if you select an element with Python and later interact with it, there's a decent chance it won't be there anymore. The only way you can be sure it's still there is to evaluate it as you interact with it and the only way to do that is to stay in the browser context.
Related
I'm trying to find all the my subject in my dashboard of my college website.
I'm using selenium to do it.
The site is a little slow so first I wait
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#class='multiline']")))
then I find all the elements with
course = driver.find_elements_by_xpath("//span[#class='multiline']")
after that in a for loop I try to traverse it the 0th place of the "course" works fine and I'm able to click it and go to webpage but when the loop runs for the secon d time that is for the 1st place in "course" it gives me error selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
So I tried adding a lit bit wait time to using 2 method it still gives me error
driver.implicitly_wait(20)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#class='multiline']")))
the loop
for i in course[1::]:
#driver.implicitly_wait(20)
#WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#class='multiline']")))
print(i)
i.click()
driver.implicitly_wait(2)
driver.back()
a snippet of the website
Thanks in advance
Answering my own question after extensive research
A common technique used for simulating a tabbed UI in a web app is to prepare DIVs for each tab, but only attach
one at a time, storing the rest in variables. In this case my code have a reference
to an element that is no longer attached to the DOM (that is, that has an ancestor which is "document.documentElement").
If WebDriver throws a stale element exception in this case, even though the element still exists, the reference
is lost. You should discard the current reference you hold and replace it, possibly by locating the element again
once it is attached to the DOM
for i in range(len(course)):
# here you need to find all the elements again because once we
leave the page the reference will be lost and we need to find it again
course = driver.find_elements_by_xpath("//span[#class='multiline']")
print(course[i].text)
course[i].click()
driver.implicitly_wait(2)
driver.back()
I desire to iterate thru a set of URLs using Selenium. From time to time I get 'element is not attached to the page document'. Thus after reading a couple of other questions indicated that it's because I am changing the page that is looking at. But I am not satisfied with that argument since:
for url in urlList:
driver.get(url)
WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, '//div/div')))
#^ WebDriverWait shall had taken care of it
myString = driver.find_element_by_xpath('//div/div').get_attribute("innerHTML")
# ^ Error occurs here
# Then I call this function to go thru other elements given other conditions not shown
if myString:
getMoreElements(driver)
But if I add a delay like this:
for url in urlList:
driver.get(url)
time.sleep(5) # <<< IT WORKS, BUT WHY?
element = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, '//div/div')))
myString = driver.find_element_by_xpath('//div/div').get_attribute("innerHTML") # Error occured here
I feel I am hiding the problem by adding the delay right there. I have implicity_wait set to 30s and set_page_load_timeout to 90s, that would had been sufficient. So, why am I still facing to add what looks like useless time.sleep?
Did you try the xpath: //div/div manually in dev tool to see how many div will be found on the page? I thinks there should be many. So your below explicity wait code can very easy to satisfied, maybe no more than 1 second, selenium can find such one div after browser.get() and your wait end.
WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, '//div/div')))
Consider following possiblity:
Due to your above explicity wait issue, the page loading not complete, more and more //div/div are rendering to page, at this time point, you ask selenium to find such one div and to interact with it.
Think about the possiblity of the first found div by selenium won't be deleted or moved to another DOM node.
What do you think the rate of above possiblity will be high or low? I think it's very hight, because div is very common tag in nowdays web page and you use such a relaxed xpath which lead to so many matched div will be found, and each one of them is possible to cause the 'Element Stale' issue
To resolve your issue, please use more strict locator to wait some special element, rather than such hasty xpath which result in finding very common and many exist element.
What you observe as element is not attached to the page document is pretty much possible.
Analysis:
In your code, while iterating over the urlList, we are opening an url then waiting for the WebElement with XPATH as //div/div with ExpectedConditions clause set to presence_of_element_located which does not necessarily mean that the element is visible or clickable.
Hence, next when you try to driver.find_element_by_xpath('//div/div').get_attribute("innerHTML") the reference of previous search/find_element is not found.
Solution:
The solution to your question would be to change the ExpectedConditions clause from presence_of_element_located to element_to_be_clickable which checks that element is visible and enabled such that you can even click it.
Code Block:
Your optimized code block may look like:
for url in urlList:
driver.get(url)
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, '//div/div')))
myString = driver.find_element_by_xpath('//div/div').get_attribute("innerHTML")
Your other solution:
Your other solution works because you are trying to covering up Selenium's work through time.sleep(5) which is not a part of best practices.
I have the following python code :
iframe = driver.find_element_by_name("iframe_name")
driver.switch_to_frame(iframe)
elem = driver.find_element_by_xpath("/html/body/iframe")
It is able to find the first iframe element and then switch to it however once it is in it and I try to access the second iframe element (by xpath since it does not have a name or id) I keep getting a "no such element error".
Can someone please help. I am trying to access the interior iframe so that I can get the src attribute within it.
Possible solutions.
Try using wait for an element by XPath ("/html/body/iframe"), as often times the driver will fail to wait till switching to the frame is completed.
Make sure that your XPath ("/html/body/iframe") is working. Also try identifying element using the tag name if there only one IFrame in the IFrame.
Hope that helps.
I have a python function that should click through all options of a product:
submit_button = driver.find_element_by_id('quantityactionbox')
elementList = submit_button.find_elements_by_tag_name("option")
for x in elementList:
x.click()
After I clicked 2 elements I get this error:
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
Can you maybe tell me why this error appaer and what can I do to go successfully through all elements?
The easy way to overcome many of these types of errors is to just add some sort of delay:
import time
time.sleep(1)
DOM manipulation after an event is fired usually takes a bit of time so you're not really losing that much of performance.
You have the explanation and the solution on The Element is not Attached to the DOM:
A common technique used for simulating a tabbed UI in a web app is to
prepare DIVs for each tab, but only attach one at a time, storing the
rest in variables. In this case, it's entirely possible that your code
might have a reference to an element that is no longer attached to the
DOM (that is, that has an ancestor which is
"document.documentElement").
If WebDriver throws a stale element
exception in this case, even though the element still exists, the
reference is lost. You should discard the current reference you hold
and replace it, possibly by locating the element again once it is
attached to the DOM.
In my case it was because the page had changed and the element no longer existed but my script was trying to call it. It was not readily obvious because the page did have essentially the same element but it had been reloaded and was therefore, not the exact same element and not available in the current page document. I had to redefine the element after the page was reloaded.
I trying automating testing with Selenium (python bindings), specifically want to log in on tokmonet.zendesk.com.
I create a script which takes email field, password field and sign in button by id.
But when I ran script it fails with
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"id","selector":"user_email"}
Inspecting page with Firebug I see these elements. But when trying to get them with Firefinder it couldn't.
So, I perform
html_source = driver.page_source
print(html_source)
and get the only
<html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>
When I check page source it contains only js scripts and no mark up.
Please advice how I could handle this elements?
I see that elements that you are trying to log in are in an iframe in tokmonet.zendesk.com and so you are not able to get the elements. To handle such situation try to switch to the iframe first and then get the elements. Here's how to do it in Java -
driver.switchTo().frame(driver.findElement(By.tagName("iframe")));
(new WebDriverWait(driver, 20))
.until(ExpectedConditions.presenceOfElementLocated(By.id("user_email"))).sendKeys("username");
//Similarly you can get other elements too
You similarly implement it in other languages. Hope this helps.
You need to switch to the IFRAME, then send_keys() to the element which you can find by ID. Don't forget to switch back to the default content if you need to access elements outside the IFRAME.
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
driver.find_element_by_id("user_email").send_keys("username")
driver.find_element_by_id("user_password").send_keys("password")
// do whatever else
driver.switch_to.default_content()