I'm using selenium with python to interact with a webpage.
There is a table in the webpage. I'm trying to access to its rows with this code:
rows = driver.find_elements_by_class_name("data-row")
It works as expected. It returns all elements of the table.
The question is, is the order of the returned elements guaranteed to be the same as they appear on the page?
For example, Will the first row that I see in the table in browser ALWAYS be the 0th index in the array?
You shouldn't be depending on the fact whether Selenium returns the elements in the same order as they appear on the webpage or DOM Tree.
Each WebElement within the HTML DOM can be identified uniquely using either of the Locator Strategies.
Though you were able to pull out all the desired elements using find_elements_by_class_name() as follows:
rows = driver.find_elements_by_class_name("data-row")
Ideally, you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CLASS_NAME:
element = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "data-row")))
Using CSS_SELECTOR:
element = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".data-row")))
Using XPATH:
element = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[#class='data-row']")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a detailed discussion in WebDriverWait not working as expected
Related
I am working with this website= https://www.offerte.smartpaws.de/
In the final page once you have put all your variables in, we get a page with this info:
and the HTML for the price looks like this:
I am looking to store the prices for each product, but it seems like I have to select the parent element based on the text "Optimal Care", "Classic Case", etc.. and go down to the child element to extract the price.
I am not sure how to do this, my current code for this specific scenario:
price_optimal = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//form[#class="summary-table-title" and #text="Optimal Care"]/*/*/'))).get_attribute("text()")
How would I take a text based on a parent element text?
Thanks
If you want to only retrieve price then you can use the below XPath:
//div[#class='summary-table-price price-monthly']//div[2]
with find_elements or with explicit waits.
With find_elements
Code:
for price in driver.find_elements(By.XPATH, "//div[#class='summary-table-price price-monthly']//div[2]"):
print(price.get_attribute('innerText'))
With ExplicitWaits:
for price in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[#class='summary-table-price price-monthly']//div[2]"))):
print(price.get_attribute('innerText'))
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
If you want to extract based on the plan name such as Classic Care
You should use the below XPath:
//div[text()='Classic Care']//following-sibling::div[#class='summary-table-price price-monthly']/descendant::div[2]
and use it like this:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Classic Care']//following-sibling::div[#class='summary-table-price price-monthly']/descendant::div[2]"))).get_attribute('innerText'))
Also, for other plans, all you will have to do is to replace Classic care with Optimal Care or Basic care in the XPath.
In python we have:
select = Select(driver.find_element_by_id('fruits01'))
But what if I want to have a list of all select tags and not just the first one?
I tried:
select = Select(driver.find_elements_by_id('fruits01'))
But it didn't work for me.
As per the Selenium Python API Docs of Select():
class selenium.webdriver.support.select.Select(webelement)
A check is made that the given element is, indeed, a SELECT tag. If it is not, then an UnexpectedTagNameException is thrown.
So Select() takes a webelement as an argument and you won't be able to pass webelements.
Moreover, each webelement is unique within the DOM Tree. So it's highly unlikely multiple elements would be identified using the same value of id attribute i.e. fruits01.
This usecase
However, considering the usecase as a valid usecase the simplest approach would be similar to #Arundeep Chohan's answer but ideally inducing WebDriverWait for the visibility_of_all_elements_located() as follows:
elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.ID, "fruits01")))
for element in elements:
select = Select(element)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
elems=driver.find_elements_by_id('fruits01')
for elem in elems:
select=Select(elem)
Why not just use a list to loop them ? I don't think you can do it the way you want to.
Hello i'm trying to click those links but when I try to click with
driver.find_element_by_xpath('//*[#id="productDetails_detailBullets_sections1"]/tbody/tr[6]/td/span/span[2]/a').click()
its work's but problem is every items has different path and its changing and it does not work for some items
URL: https://www.amazon.com/MICHELANGELO-Piece-Rainbow-Kitchen-Knife/dp/B074T6C4YS/ref=zg_bs_289857_1?_encoding=UTF8&psc=1&refRID=K5GAX1GF2SDZMN3NS403>
The Amazon webpage has 3 entries for Best Sellers Rank. An effective approach would be to collect the href of all the three(3) Best Sellers, store them in a list and open in a seperate tab to scrape. To construct the list you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get('https://www.amazon.com/dp/B074T6C4YS')
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table#productDetails_detailBullets_sections1 td>span>span a")))])
Using CSS_SELECTOR in a single line:
driver.get('https://www.amazon.com/dp/B074T6C4YS')
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[#id='productDetails_detailBullets_sections1']//td/span/span//a")))])
Console Output:
['https://www.amazon.com/gp/bestsellers/kitchen/ref=pd_zg_ts_kitchen', 'https://www.amazon.com/gp/bestsellers/kitchen/289857/ref=pd_zg_hrsr_kitchen', 'https://www.amazon.com/gp/bestsellers/kitchen/289862/ref=pd_zg_hrsr_kitchen']
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
This is easy one, even you did not specify which link do you want, only from the table all different links that will transfer you to table.
You need to use customized xpath like e.g.
//*[#id="productDetails_detailBullets_sections1"]/tbody/tr[6]/td/span/span['+i+']/a'
Where i will be your iterator in for loop. To get valu of i use something like
driver.find_elements_by_xpath('//*[#id="productDetails_detailBullets_sections1"]/tbody/tr[6]/td/span/span').size();
I want to get the text of an element in selenium. First I did this:
team1_names = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".home span"))
)
for kir in team1_names:
print(kir.text)
It didn’t work out. So I tried this:
team1_name = driver.find_elements_by_css_selector('.home span')
print(team1_name.getText())
so team1_name.text doesn’t work either.
So what's wrong with it?
You need to take care of a couple of things here:
presence_of_element_located() is the expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible.
Additionally, presence_of_element_located() returns a WebElement, not a list. Hence, iterating through for won't work.
Solution
As a solution you need to induce WebDriverWait for the visibility_of_element_located(), and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and text attribute:
print(WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".home span"))).text)
Using XPATH and get_attribute():
print(WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.XPATH, "//*[#class='home']//span"))).get_attribute("innerHTML"))
Note: You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
Outro
Link to useful documentation:
The get_attribute() method Gets the given attribute or property of the element.
The text attribute returns The text of the element.
Difference between text and innerHTML using Selenium
I have an input HTML element like this in Django
<input id="id" type="number" maxlength="50">
When I want to find and clear it
elm_input = self.wait.until(EC.presence_of_element_located((By.ID, elm_id)))
elm_input.clear()
elm_input.send_keys(value)
It's got error InvalidElementStateException
InvalidElementStateException invalid element state: Element must be user-editable in order to clear it"
We cannot send key clear because selenium know CLEAR or DELETE Keys is an Charactics Keys not an Number Keys, It's don't send Keys to element input. So how can I we fix it, I had try ActionChains but It's not working with as well
This error message...
InvalidElementStateException invalid element state: Element must be user-editable in order to clear it"
...implies that the WebDriver instance was unable to clear the existing contents of the element.
A bit more of the outerHTML of the element would have helped us to analyze the issue in a better way. However you need to take care of a couple of things as follows:
While sending a character sequence instead of using presence_of_element_located() you have to induce WebDriverWait for the element_to_be_clickable().
Ensure the Locator Strategy uniquely identifies the WebElement and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
elm_input = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#id[type='number'][maxlength='50']")))
elm_input.clear()
elm_input.send_keys("1234567890")
Using XPATH:
elm_input = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#id='id' and #type='number'][#maxlength='50']")))
elm_input.clear()
elm_input.send_keys("1234567890")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a relevant discussion in:
Invalid element state: Element must be user-editable in order to clear it error trying to click and insert a date on a dropdown-toggle using Selenium