I'm using selenium for the first time to get some information about a fantasy soccer game I play with my friends (we have a competition). I'm facing issues iterating through a list of webelements. Apparently they become stale.
Here's some code and details:
I was able to get to the competition's page by myself. This page has cards for every team in the competition and they look like this
<span class="cartola-card-thin__nome__time">TEAM1</span>
When clicked, those cards lead to that team's page. This page contains a dropdown menu that looks like this
<span class="cartola-dropdown-bg__botao cartola-dropdown-bg-botao-rodada-id cartola-dropdown-bg__botao--aberto" ng-class="'cartola-dropdown-bg-botao-' + name"></span>
and this menu contains a div for each round of the competition. It looks like this
<div ng-if="!hasDescription" class="cartola-dropdown-bg__selecao" ng-bind="item.label">rodada 25</div>
When clicked, each div loads that specific team's formation, and its points during that round. The points are shown on the page like this:
<div class="cartola-time-adv__pontuacao pont-positiva" ng-class="{'pont-positiva': ctrl.timeService.dadosTime.pontos > 0,
'pont-negativa': ctrl.timeService.dadosTime.pontos < 0}" ng-bind="ctrl.timeService.dadosTime.pontos != null ? ctrl.timeService.dadosTime.pontos : ''">78.17</div>
My goal: I want to gather each team's points during each one of the rounds in a dict['round'] = points.
What I've tried already: I've tried to keep the teams in a list by doing
teams = browser.find_elements_by_class_name("cartola-card-thin__nome__time")
Then, for each team in teams I'd click on it.
When on that page I'd find each round like this
rounds = browser.find_elements_by_class_name("cartola-dropdown-bg__selecao")
Then, for each round in rounds I'd click on it and get that round's points.
The problem: those loops where I iterate through teams and rounds are not working because apparently those webelements become stale after the whole process inside the loop (clicking, etc)
How can I approach this problem?
Angular drop down elements are rebuild in runtime. After drop down is collapsed - previously found drop down item is no longer an element of DOM. It is added to DOM one more time, when drop down is expanded again - but it is not the same element for WebDriver (even if it can be found with the same locator).
So, you're following this logic:
Expand drop down
Get drop down elements ->
teams = browser.find_elements_by_class_name("cartola-card-thin__nome__time")
Do something for each team -> here, I suppose, that drop down is collapsed. So found WebElements are no longer in DOM -> stale element exception
What you have to do?
teamsCount = len(teams);
teamIndexes = range(teamsCount)
for(i in temIndexes)
team = driver.find_element(locator_that_usesTeamIndex)
Related
I'm very new to programming so apologies in advance if I'm not communicating my issue clearly.
Essentially, using Selenium I have created a list of elements on a webpage by finding all the elements with the same class name I'm looking for.
In this case, I'm finding songs, which have the html class 'item-song' on this website.
On the website, there are lots of clickable options for each listed song . I just want to click the title of the song, which opens a popup modal window in which I edit the note attached to the song, then click save, which closes the popup.
I have successfully been able to do that by using what I guess would be called the title’s XPATH 'relative' to the song class.
songs = driver.find_elements(By.CLASS_NAME, "item-song")
songs[0].find_element(By.XPATH, "div[5]/a").click()
# other code that ends by closing popup
This works, hooray! It also works for any other list index that I put in that line of code.
However, it does not work sequentially, or in a for loop.
i.e.
songs[0].find_element(By.XPATH, "div[5]/a").click()
# other code
time.sleep(5) # to ensure the popup has finished closing
songs[1].find_element(By.XPATH, "div[5]/a").click()
Does not work.
for song in songs:
song.find_element(By.XPATH, "div[5]/a").click()
# other code
time.sleep(5)
continue
Also does not work.
I get a traceback error:
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
After going back to the original page, the song does now say note(1) so I suppose the site has changed slightly. But as far as I can tell, the 'songs' list object and the xpath for the title of the next song should be exactly the same. To verify this, I even tried:
for song in songs:
print(song)
print(songs)
print()
song.find_element(By.XPATH, "div[5]/a").click()
# other code
Sure enough, on the first iteration, print(song) matched the first index of print(songs) and on the second iteration, print(song) matches the second index of print(songs). And print(songs) is identical both times. (Only prints twice as the error happens halfway through the second iteration)
Any help is greatly appreciated, I'm stumped!
---------------------------------
Edit: Of course, it would be easier if my songs list could be all the song titles instead of the class ‘item-song’, that was what I was trying first. However I couldn’t find anything common between the titles in the HTML that would let me use find_elements to just get the song title element, as each song has a different title, and there are also other items like videos listed in between each song.
Through the comments, the solution is to use an iterative loop and an xpath.
songs = driver.find_elements(By.CLASS_NAME, "item-song")
for i in range(songs.count):
driver.find_element(By.XPATH, "(//*[#class='item-song'][" + i + "])/div[5]/a").click()
Breaking this down:
this: By.XPATH, "//*[#class='item-song']" is the same as this: By.CLASS_NAME, "item-song". The former is the xpath equivalent of the latter. I did this so we can build a single identification string to the link instead of trying to find elements within elements.
The [" + i + "] is the iteration for the the loop. If you were to print this you'd see (//*[#class='item-song'][1])") then (//*[#class='item-song'][2])"). That [x] is the ordinal identifier - it means the xth instance of the element in the DOM. The brackets around it ensure the entire thing is matched for the next part - you can sometimes get unexpected matches without it.
The last part /div[5]/a is just the original solution. Doing div[5] isn't great. Your link must ALWAYS be inside the 5th div else it will fail - but as i can't see your application I can't comment on another way.
The original approach throws a StaleElementReferenceException because of the way Selenium stores identified elements.
Once you've identified an element by doing driver.find_elements(By.CLASS_NAME, "item-song") Selenium essentially captures a reference to it - it doesn't store the identifier you used. Stick a break point and after you identify an element and you'll see something like this:
That image is from visual studio as I have it hand but you can see it's a GUID on the ID.
Once you change the page that reference is lost.
Repeat the same steps, identify the same object and the ID is unique every time. This is same break point, same element on a second test run:
Page has changed == Selenium can no longer find it == Stale element.
The solution in this answer works because we're not storing an element.
Every action in the loop freshly identifies the element.
..Then add some clever pun about fresh vs stale... ;-)
There is a table that I want to get the XPATH of, however the amount of rows and columns is inconsistent across results so I can't just right click and get copy the full XPATH.
My current code:
result_priority_number = driver.find_element(By.XPATH, "/html/body/div/div[2]/div[6]/div/div[2]/table/tbody/tr[18]/td[2]")
The table header names though are always consistent. How do I get the value of an element where the table header specifically says something (i.e. "Priority Number")
I can't just right click and get copy the full XPATH.
Never use this method. Xpath has a very useful feature for search! It isn't just for nested pathing!
//td[contains(text(),'header value')]
or if it has many tables and you want only one of its:
//table[#id='id_of_table']//td[contains(text(),'header value')]
or the table hasn't id or class:
//table[2]//td[contains(text(),'header value')]
where 2 is index of table in page
and other many feature for searching in html nodes
in your case, for get Filing language:
//td[contains(text(),'Filing language')]/following-sibling::td
In my project, I am downloading all the reports by clicking each link written as a "Date". Below is the image of the table.
I have to extract a report of each date mentioned in the table column "Payment Date". Each date is a link for a report. So, I am clicking all the dates one-by-one to get the report downloaded.
for dt in driver.find_elements_by_xpath('//*[#id="tr-undefined"]/td[1]/span'):
dt.click()
time.sleep(random.randint(5, 10))
So, the process here going is when I click one date it will download a report of that date. Then, I will click next date to get a report of that date. So, I made a for loop to loop through all the links and get a report of all the dates.
But it is giving me Stale element exception. After clicking 1st date it is not able to click the next date. I am getting error and code stops.
How can I solve this?
You're getting a stale element exception because the DOM is updating elements in your selection on each click.
An example: on-click, a tag "clicked" is appended to an element's class. Since the list you've selected contains elements which have changed (1st element has a new class), it throws an error.
A quick and dirty solution is to re-perform your query after each iteration. This is especially helpful if the list of values grows or shrinks with clicks.
# Create an anonymous function to re-use
# This function can contain any selector
get_elements = lambda: driver.find_elements_by_xpath('//*[#id="tr-undefined"]/td[1]/span')
i = 0
while True:
elements = get_elements()
# Exit if you're finished iterating
if not elements or i>len(elements):
break
# This should always work
element[i].click()
# sleep
time.sleep(random.randint(5, 10))
# Update your counter
i+=1
The simplest way to solve it is to get a specific link each time before clicking on it.
links = driver.find_elements_by_xpath('//*[#id="tr-undefined"]/td[1]/span')
for i in range(len(links)):
element = driver.find_elements_by_xpath('(//*[#id="tr-undefined"]/td[1]/span)[i+1]')
element.click()
time.sleep(random.randint(5, 10))
I am trying to create a web scraper and I ran into problem. I am trying to iterate over elements on the left side of the widget and if name starts with 'a', I want to click on minus sign and move it to the right side. I managed to find all the elements, however, once the element move to the right is side is executed, right after that loop I get the following error.
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=80.0.3987.163)
JS widget.
You need to refactor your code. Your code pattern is likely something like this (of course with different id-s but since you did not include your code or the page source this is the best I can offer):
container = driver.find_elements_by_xpath('//*[#class="window_of_elements"]')
elements = container.find_elements_by_xpath('//*[#class="my_selected_class"]')
for e in elements:
minus_part = e.find_element_by_xpath('//span[#class="remove"]')
minus_part.click()
When you click the minus_part, the container of your elements is probably getting re-rendered/reloaded and all your previously found elements turn stale.
To bypass this you should try a different approach:
container = driver.find_elements_by_xpath('//*[#class="window_of_elements"]')
to_be_removed_count = len(container.find_elements_by_xpath('//*[#class="my_selected_class"]'))
for _ in range(to_be_removed_count):
target_element = container.find_element_by_xpath('//*[#class="window_of_elements"]//*[#class="my_selected_class"]')
minus_part = target_element.find_element_by_xpath('//span[#class="remove"]')
minus_part.click()
So basically you should:
find out how many elements you should find to be clicked
in a for loop find and click them one by one
I would like to get the text value of a span class "currency-coins value" to be used in a comparison.
Basically I want to check the market value of a specific player. I get the player listed 20 times in a container. So the "currency-coins value" is shown 20 times on the page.
Now I need to get the "200" as shown in the screenshot of the HTML code above as value I can work with. And this for all 20 results on the page. The value might be different for all 20 results.
After I got all 20 values, I want to check which one is the lowest.
I will then afterwards use the lowest value as price to list my element on the market.
Is there a way to do this? Since I am learning python for a bit more than one week now, I cant figure it out myself.
The idea is to first iterate over the player containers - usually, these are table rows, and, for each container, locate that price element within. For instance:
for row in driver.find_elements_by_css_selector("table tbody > tr"):
coin_value = float(row.find_element_by_css_selector(".currency-coins.value").text)
print(coin_value)
Note that table tbody > tr is used as an example, your locator for table rows or player containers is likely different.