In my project, I am downloading all the reports by clicking each link written as a "Date". Below is the image of the table.
I have to extract a report of each date mentioned in the table column "Payment Date". Each date is a link for a report. So, I am clicking all the dates one-by-one to get the report downloaded.
for dt in driver.find_elements_by_xpath('//*[#id="tr-undefined"]/td[1]/span'):
dt.click()
time.sleep(random.randint(5, 10))
So, the process here going is when I click one date it will download a report of that date. Then, I will click next date to get a report of that date. So, I made a for loop to loop through all the links and get a report of all the dates.
But it is giving me Stale element exception. After clicking 1st date it is not able to click the next date. I am getting error and code stops.
How can I solve this?
You're getting a stale element exception because the DOM is updating elements in your selection on each click.
An example: on-click, a tag "clicked" is appended to an element's class. Since the list you've selected contains elements which have changed (1st element has a new class), it throws an error.
A quick and dirty solution is to re-perform your query after each iteration. This is especially helpful if the list of values grows or shrinks with clicks.
# Create an anonymous function to re-use
# This function can contain any selector
get_elements = lambda: driver.find_elements_by_xpath('//*[#id="tr-undefined"]/td[1]/span')
i = 0
while True:
elements = get_elements()
# Exit if you're finished iterating
if not elements or i>len(elements):
break
# This should always work
element[i].click()
# sleep
time.sleep(random.randint(5, 10))
# Update your counter
i+=1
The simplest way to solve it is to get a specific link each time before clicking on it.
links = driver.find_elements_by_xpath('//*[#id="tr-undefined"]/td[1]/span')
for i in range(len(links)):
element = driver.find_elements_by_xpath('(//*[#id="tr-undefined"]/td[1]/span)[i+1]')
element.click()
time.sleep(random.randint(5, 10))
Related
I've been creating a webscraper with selenium that will select several items, fill a cart, and then cycle through a list of zip codes, cities, and states to calculate the total cost + shipping + tax for a variety of locations.
The scraper has been set up successfully and works, but the problem that I am running into is that the text of the element I am searching for gives the base price, and then updates to include tax and shipping. I've tried using implicit wait or time.sleep() to slow the process down to give it time to update, but it continues to only get the original price. Without putting in the entire script this is what I am running into:
for item in c_s_z_list:
cty = driver.find_element(By.CSS_SELECTOR, 'input[id="city"]')
cty.clear()
time.sleep(1)
cty.send_keys(item[0])
dropdown1 = Select(driver.find_element(By.CSS_SELECTOR,'select[id="state"]'))
dropdown1.select_by_visible_text(item[1])
zcode = driver.find_element(By.CSS_SELECTOR,'input[id="zip"]')
zcode.clear()
time.sleep(1)
zcode.send_keys(item[2])
time.sleep(20)
fpt = driver.find_element(By.CSS_SELECTOR,'span[class="order-wizard-cart-summary-grid-right"]')
fp = fpt.text
print(fp)
Before entering in the city, state and zip info the html looks like this:
and after the info is added it updates to this:
Even after giving enough sleep for the page to load, the element.text always returns the initial value, not the updated one.
I've played around with waits, and sleeps and even refreshing to see if any of that makes a difference, but so far no dice
This is the main landing page
https://www.performbetter.com/First-Place-Gravity-Kettlebell
And this is the cart specific page
https://checkout.performbetter.com/sca-dev-vinson/checkout.ssp?is=checkout&fragment=opc&_ga=2.53878443.2036683866.1667331254-1479525696.1666912626#opc
There are two matches for your locator value, so it returns the SubTotal, change locator value to:
driver.find_element(By.CSS_SELECTOR,'.order-wizard-cart-summary-total .order-wizard-cart-summary-grid-right')
I'm very new to programming so apologies in advance if I'm not communicating my issue clearly.
Essentially, using Selenium I have created a list of elements on a webpage by finding all the elements with the same class name I'm looking for.
In this case, I'm finding songs, which have the html class 'item-song' on this website.
On the website, there are lots of clickable options for each listed song . I just want to click the title of the song, which opens a popup modal window in which I edit the note attached to the song, then click save, which closes the popup.
I have successfully been able to do that by using what I guess would be called the title’s XPATH 'relative' to the song class.
songs = driver.find_elements(By.CLASS_NAME, "item-song")
songs[0].find_element(By.XPATH, "div[5]/a").click()
# other code that ends by closing popup
This works, hooray! It also works for any other list index that I put in that line of code.
However, it does not work sequentially, or in a for loop.
i.e.
songs[0].find_element(By.XPATH, "div[5]/a").click()
# other code
time.sleep(5) # to ensure the popup has finished closing
songs[1].find_element(By.XPATH, "div[5]/a").click()
Does not work.
for song in songs:
song.find_element(By.XPATH, "div[5]/a").click()
# other code
time.sleep(5)
continue
Also does not work.
I get a traceback error:
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
After going back to the original page, the song does now say note(1) so I suppose the site has changed slightly. But as far as I can tell, the 'songs' list object and the xpath for the title of the next song should be exactly the same. To verify this, I even tried:
for song in songs:
print(song)
print(songs)
print()
song.find_element(By.XPATH, "div[5]/a").click()
# other code
Sure enough, on the first iteration, print(song) matched the first index of print(songs) and on the second iteration, print(song) matches the second index of print(songs). And print(songs) is identical both times. (Only prints twice as the error happens halfway through the second iteration)
Any help is greatly appreciated, I'm stumped!
---------------------------------
Edit: Of course, it would be easier if my songs list could be all the song titles instead of the class ‘item-song’, that was what I was trying first. However I couldn’t find anything common between the titles in the HTML that would let me use find_elements to just get the song title element, as each song has a different title, and there are also other items like videos listed in between each song.
Through the comments, the solution is to use an iterative loop and an xpath.
songs = driver.find_elements(By.CLASS_NAME, "item-song")
for i in range(songs.count):
driver.find_element(By.XPATH, "(//*[#class='item-song'][" + i + "])/div[5]/a").click()
Breaking this down:
this: By.XPATH, "//*[#class='item-song']" is the same as this: By.CLASS_NAME, "item-song". The former is the xpath equivalent of the latter. I did this so we can build a single identification string to the link instead of trying to find elements within elements.
The [" + i + "] is the iteration for the the loop. If you were to print this you'd see (//*[#class='item-song'][1])") then (//*[#class='item-song'][2])"). That [x] is the ordinal identifier - it means the xth instance of the element in the DOM. The brackets around it ensure the entire thing is matched for the next part - you can sometimes get unexpected matches without it.
The last part /div[5]/a is just the original solution. Doing div[5] isn't great. Your link must ALWAYS be inside the 5th div else it will fail - but as i can't see your application I can't comment on another way.
The original approach throws a StaleElementReferenceException because of the way Selenium stores identified elements.
Once you've identified an element by doing driver.find_elements(By.CLASS_NAME, "item-song") Selenium essentially captures a reference to it - it doesn't store the identifier you used. Stick a break point and after you identify an element and you'll see something like this:
That image is from visual studio as I have it hand but you can see it's a GUID on the ID.
Once you change the page that reference is lost.
Repeat the same steps, identify the same object and the ID is unique every time. This is same break point, same element on a second test run:
Page has changed == Selenium can no longer find it == Stale element.
The solution in this answer works because we're not storing an element.
Every action in the loop freshly identifies the element.
..Then add some clever pun about fresh vs stale... ;-)
Today I'm having troubles due to a "a href" button that does not have any ID to be identified, then let's explain a little bit more about the problem... I have an structure like this one(let's assume XXX is an anonymous path):
wait = WebDriverWait(driver, 5)
el=wait.until(EC.presence_of_element_located((By.ID, 'XXX1')))
entries = el.find_elements_by_tag_name('tr')
for i in range(len(entries)):
if(entries[i].find_element_by_xpath(XXX2).text==compare):
el = wait.until(EC.element_to_be_clickable((By.ID,XXX3)))
el.click()
el=wait.until(EC.presence_of_element_located((By.ID, XXX4)))
entries2 = el.find_elements_by_tag_name('tr')
for j in range(len(entries2)):
#Some statements...
xpath = ".../a"
your_element=WebDriverWait(driver,10)\
.until(EC.element_to_be_clickable((By.XPATH,xpath)))##Here the problem
your_element.click()
Then, I'm getting information from an hibrid page (dynamic and static) using as driver a ChromeDriver one, once I get a big table, inside every row there is a button that shows more info, then I need to click it for open it too, the main problem is when this operation iterates, that error is shown by the output. This driver is a ChromeDriver one. In summary, first I search something and click on search button, then I get a table where every row(at the end in the last column) has a button that once is open, shows more information, consequently I need to open it and close it due to the next row, it works with the first row, but with the second one, it crashes. I would really appreciate any advice of how to handle this problem.
Thanks in advance!
The problem is that you change with the click the dom within your loop. For mr this never worked.
One solution is, to try to re-query within the loop to make sure your at the correct position. Your third line:
entries = el.find_elements_by_tag_name('tr')
should be executed every time and with a counter make sure you are at the correct position of your <tr> entries.
I am trying to create a web scraper and I ran into problem. I am trying to iterate over elements on the left side of the widget and if name starts with 'a', I want to click on minus sign and move it to the right side. I managed to find all the elements, however, once the element move to the right is side is executed, right after that loop I get the following error.
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=80.0.3987.163)
JS widget.
You need to refactor your code. Your code pattern is likely something like this (of course with different id-s but since you did not include your code or the page source this is the best I can offer):
container = driver.find_elements_by_xpath('//*[#class="window_of_elements"]')
elements = container.find_elements_by_xpath('//*[#class="my_selected_class"]')
for e in elements:
minus_part = e.find_element_by_xpath('//span[#class="remove"]')
minus_part.click()
When you click the minus_part, the container of your elements is probably getting re-rendered/reloaded and all your previously found elements turn stale.
To bypass this you should try a different approach:
container = driver.find_elements_by_xpath('//*[#class="window_of_elements"]')
to_be_removed_count = len(container.find_elements_by_xpath('//*[#class="my_selected_class"]'))
for _ in range(to_be_removed_count):
target_element = container.find_element_by_xpath('//*[#class="window_of_elements"]//*[#class="my_selected_class"]')
minus_part = target_element.find_element_by_xpath('//span[#class="remove"]')
minus_part.click()
So basically you should:
find out how many elements you should find to be clicked
in a for loop find and click them one by one
I am trying to read the value of a text from a page using Selenium Python. The element is always visible and then goes invisible and then becomes visible and the text value changes rapidly until it reaches the final value. I think it is using some form of javascript to get the value before displaying the final value.
The page is https://www.junglescout.com/estimator/
I am trying to get the result of the estimate from the element js-magic-result.
I'm able to use Selenium to fill the three forms and click the 'calculates sales' button
I am using the chromedriver in python selenium to read the value.
all the tests gets me one of the intermediate values before it finishes loading.
I tried using the folling inplace of the browser2.implicitly_wait(5):
driver.implicitly_wait(5)
wait = WebDriverWait(browser2,3)
wait.until(EC.invisibility_of_element_located((By.CLASS_NAME,'js-magic-result')))
wait.until(EC.visibility_of_element_located((By.CLASS_NAME,'js-magic-result')))
Here is the full code I am using
browser2 = webdriver.Chrome(options=options,executable_path=driverPath)
url = 'https://www.junglescout.com/estimator/'
browser2.get(url)
container = browser2.find_element_by_class_name('js-estimation-section')
rankField = container.find_element_by_name('theRankInput')
rankField.send_keys('345')
# Click for storesList drop down
storeGroup = container.find_element_by_class_name('js-est-stores-list-input')
storeGroup.find_element_by_class_name('x-icon-caret-down').click()
# Get Stores List items
wait = WebDriverWait(browser2,3)
stores = wait.until(EC.visibility_of_element_located((By.CLASS_NAME,'js-est-stores-list')))
stores.find_elements_by_tag_name('span')[0].click()
# Wait for storesList is invisible
wait.until(EC.invisibility_of_element_located((By.CLASS_NAME,'js-est-stores-list')))
# Click for Categories list drop down
catGroup = container.find_element_by_class_name('js-est-categories-list-input')
catGroup.find_element_by_tag_name('input').click()
# Get Categories List items
cats = wait.until(EC.visibility_of_element_located((By.CLASS_NAME,'js-est-categories-list')))
# Get Categories List items
for cat in cats.find_elements_by_class_name('us-available'):
if (cat.find_element_by_tag_name('span').text == 'Electronics'):
cat.click()
break
# Wait for storesList is invisible
wait.until(EC.invisibility_of_element_located((By.CLASS_NAME,'js-est-categories-list')))
# wait5 = WebDriverWait(browser2,3)
submit = wait.until(EC.visibility_of_element_located((By.CLASS_NAME,'js-est-btn')))
submit.click()
browser2.implicitly_wait(5)
print(container.find_element_by_class_name('js-magic-result').text)
What I expect is to get the final value returned but what I get is one of the intermediate values from the element.
Instead Of
print(container.find_element_by_class_name('js-magic-result').text)
Please try this.
print(browser2.find_element_by_xpath("//table[#class='js-estimation-section']//tbody//tr/td[2]/p").text)
Make sure before print some delay should be there so please replace only print code.
Let me know if this works.