How would you find an element in selenium (with python) of the following html:
Login
As per the HTML you have shared to find the element by href you can use either of the following:
css_selector:
driver.find_element_by_css_selector("a[href=/user/login]")
xpath:
driver.find_element_by_xpath("//a[#href='/user/login']")
xpath (Multiple Attributes):
driver.find_element_by_xpath("//a[#href='/user/login' and text()='Login']")
Related
I'm attempting to extract "479" from this sample HTML:
<div data-testid="testid">
"479"
" Miles Away"
</div>
I'm using the following Selenium code in Python:
xpath = 'html/body/div/text()[1]'
WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, xpath)))
distance = driver.find_element(By.XPATH, xpath)
print(distance)
Which returns the following error:
'The result of the xpath expression "html/body/div/text()[1]" is: [object Text]. It should be an element.'
I've attempted to remove 'text()[1]' from the end of my xpath, theoretically printing off all data contained the in the HTML div, but it will instead print a blank line when I do so.
Note: I'm an amateur and self-taught (via mostly Google, YouTube, and this site), so some of my wordage may not be correct. I apologize in advanced.
Given the html:
<div data-testid="testid">
"479"
" Miles Away"
</div>
Both the texts 479 and Miles Away are with in 2 different text nodes.
Selenium doesn't supports text() as it returns a text node, where as Selenium expects back a WebElement. Hence you see the error:
The result of the xpath expression "html/body/div/text()[1]" is: [object Text]. It should be an element.
Solution
To extract the text 479 you can use either of the following locator strategies:
Using xpath through execute_script() and textContent:
print(driver.execute_script('return arguments[0].firstChild.textContent;', WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//div[#data-testid='testid']")))).strip())
Using xpath through splitlines() and get_attribute():
print(WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//div[#data-testid='testid']"))).get_attribute("innerHTML").splitlines()[1])
The problem is that you can't treat text like that, the text() function returns everything as a string including a line break. I think there is no split function that can help you with that, I advise you to get the text in a python variable and do a split('\n') to the text.
xpath = 'html/body/div/text()'
WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.XPATH, xpath)))
distance = driver.find_element(By.XPATH, xpath)
print(distance.split('\n')[0])
You should take the entire element (without text()) using only
html/body/div
then from returned element get text, which will be: "479" " Miles Away" .
Then using split method from python you can take that number(split by \n, space, or ").
Selenium doesn't support the following xpath
xpath = 'html/body/div/text()[1]'
To identify the element uniquely, Your xpath should be like
xpath = '//div[#data-testid="testid"]'
WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, xpath)))
distance = driver.find_element(By.XPATH, xpath).text
print(distance)
To get the text of the element you have to use element.text
I am trying to extract some information from the amazon website using selenium. But I am not able to scrape that information using xpath in selenium.
In the image below I want to extract the info highlighted.
This is the code I am using
try:
path = "//div[#id='desktop_buybox']//div[#class='a-box-inner']//span[#class='a-size-small')]"
seller_element = WebDriverWait(driver, 5).until(
EC.visibility_of_element_located((By.XPATH, path)))
except Exception as e:
print(e)
When I run this code, it shows that there is an error with seller_element = WebDriverWait(driver, 5).until( EC.visibility_of_element_located((By.XPATH, path))) but does not say what exception it is.
I tried looking online and found that this happens when selenium is not able to find the element in the webpage.
But I think the path I have specified is right. Please help me.
Thanks in advance
[EDIT-1]
This is the exception I am getting
Message:
//div[class='a-section a-spacing-none a-spacing-top-base']//span[class='a-size-small a-color-secondary']
XPath could be something like this. You can shorten this.
CSS selector could be and so forth.
.a-section.a-spacing-none.a-spacing-top-base
.a-size-small.a-color-secondary
I think the reason is xpath expression is not correct.
Take the following element as an example, it means the span has two class:
<span class="a-size-small a-color-secondary">
So, span[#class='a-size-small') will not work.
Instead of this, you can ues xpath as
//span[contains(#class, 'a-size-small') and contains(#class, 'a-color-secondary')]
or cssSelector as
span.a-size-small.a-color-secondary
Amazon is updating its content on the basis of the country you are living in, as I have clicked on the link provided by you, there I did not find the element you are looking for simply because the item is not sold here in India.
So in short if you are sitting in India and try to find your element, it is not there, but as you change the location to "United States". it is appearing there.
Solution - Change the location
To print the Ships from and sold by Amazon.com of an element you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and get_attribute():
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.a-section.a-spacing-none.a-spacing-top-base > span.a-size-small.a-color-secondary"))).get_attribute("innerHTML"))
Using XPATH and text attribute:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#class='a-section a-spacing-none a-spacing-top-base']/span[#class='a-size-small a-color-secondary']"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
Outro
Link to useful documentation:
get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium
I'm using selenium with python,now I want to locate an element by part of its id name,what can I do?
For example,now I've already located a item by id name coption5 :
sixth_item = driver.find_element_by_id("coption5")
Is there anyway I can locate this element only by using coption?
To find the element which you have located with:
sixth_item = driver.find_element_by_id("coption5")
To locate this element only by using coption you can use can use either of the following Locator Strategies:
Using XPATH and starts-with():
sixth_item = driver.find_element_by_xpath("//*[starts-with(#id, 'coption')]")
Using XPATH and contains():
sixth_item = driver.find_element_by_xpath("//*[contains(#id, 'coption')]")
Using CSS_SELECTOR and ^ (wildcard of starts-with):
sixth_item = driver.find_element_by_css_selector("[id^='coption']")
Using CSS_SELECTOR and * (wildcard of contains):
sixth_item = driver.find_element_by_css_selector("[id*='coption']")
Reference
You can find a detailed discussion on dynamic CssSelectors in:
How to get selectors with dynamic part inside using Selenium with Python?
Java Selenium webdriver expression finding dynamic element by ccs that starts with and ends with
How to click a dynamic link with in a drupal 8 website using xpath/css selector while automating through Selenium and Python
Finding elements by CSS selector with ChromeDriver (Selenium) in Python
I'm using selenium with python,now I want to locate an element by part of its id name,what can I do?
For example,now I've already located a item by id name coption5 :
sixth_item = driver.find_element_by_id("coption5")
Is there anyway I can locate this element only by using coption?
To find the element which you have located with:
sixth_item = driver.find_element_by_id("coption5")
To locate this element only by using coption you can use can use either of the following Locator Strategies:
Using XPATH and starts-with():
sixth_item = driver.find_element_by_xpath("//*[starts-with(#id, 'coption')]")
Using XPATH and contains():
sixth_item = driver.find_element_by_xpath("//*[contains(#id, 'coption')]")
Using CSS_SELECTOR and ^ (wildcard of starts-with):
sixth_item = driver.find_element_by_css_selector("[id^='coption']")
Using CSS_SELECTOR and * (wildcard of contains):
sixth_item = driver.find_element_by_css_selector("[id*='coption']")
Reference
You can find a detailed discussion on dynamic CssSelectors in:
How to get selectors with dynamic part inside using Selenium with Python?
Java Selenium webdriver expression finding dynamic element by ccs that starts with and ends with
How to click a dynamic link with in a drupal 8 website using xpath/css selector while automating through Selenium and Python
Finding elements by CSS selector with ChromeDriver (Selenium) in Python
I look up the information that lxml does not support xpath2.0 so that it can't use ends-with, so selenium can't use ends-with how to use it or replace ends-with. thank you very much indeed！！!
HTML sample
<span id="xxxxx_close">wwwww</span>
The 'xxxxx' part of #id is random
You can apply an ends-with CSS selector:
By.cssSelector("[id$=_close]")
There's no need of including span tag in css selector search as well.
The ends-with XPath Constraint Function is part of XPath v2.0 but as per the current implementation Selenium supports XPath v1.0.
As per the HTML you have shared to identify the element you can use either of the Locator Strategies:
XPath using contains():
xpath using contains for id attribute:
driver.findElement(By.xpath("//span[contains(#id,'_close')]")).click();
xpath using contains for id and innerHTML attribute:
driver.findElement(By.xpath("//span[contains(#id,'_close') and contains(.,'wwwww')]")).click();
Alternatively, you can also use CssSelector as follows:
css_selector using ends-with (i.e. $ wildcard) clause for id attribute:
driver.find_element_by_css_selector("span[id$='_close']").click();
css_selector using contains (i.e. * wildcard) clause for id attribute:
driver.find_element_by_css_selector("span[id*='_close']").click();