Develop Reference

Extract list of text from g element within svg tag using Selenium and Python

I would like to extract the following text fields that are located within a g tag which is located in a svg tag (the url: https://www.msci.com/our-solutions/esg-investing/esg-ratings-climate-search-tool). I put in a company name and search for it, expand the last drop down menu and want to extract information from the <svg>.
HTML Part I:
HTML Part II:
I tried what has been suggested here: Extracting text from svg using python and selenium
but I did not mange to get it.
My code:
test = driver.find_element(By.XPATH, "//*[local-name()='svg' and #class='highcharts-root']//*[local-name()='g' and #class='highcharts-axis-labels highcharts-xaxis-labels']//*[name()='text']").text
print(test)

To extract the texts e.g. Sep-18, Nov-22 you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following locator strategies:
Using CSS_SELECTOR and text attribute:
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "svg.highcharts-root g.highcharts-axis-labels.highcharts-xaxis-labels text")))])
Using XPATH and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[local-name()='svg' and #class='highcharts-root']//*[local-name()='g' and #class='highcharts-axis-labels highcharts-xaxis-labels']//*[local-name()='text']")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant detailed discussions in:
Reading title tag in svg?
Creating XPATH for svg tag

No such element error but the element can be located in the DOM through xpath expression

I have a python script logging into our company website to then get each row of data. It logs me in, locates the iframe where the table with row values is located, from there I try to locate a row in the iframe table and attached is a picture:
Highlighting the correct xpath expression but I need all 31 values.
All my code works properly except this:
elem5 = browser.find_element_by_xpath("//td/label[contains(#id, 'driver')][1]")

find_element_by_* returns a single webelement, where as you are looking for all the 31 elements. So you need find_elements* and you can use either of the following Locator Strategies:
Using css_selector:
elements = driver.find_elements(By.CSS_SELECTOR, "td > label[id^='driver']")
Using xpath:
elements = driver.find_elements(By.XPATH, "//td/label[starts-with(#id, 'driver')]")
To find all the desired elements ideally you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "td > label[id^='driver']")))
Using XPATH:
elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//td/label[starts-with(#id, 'driver')]")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Selenium find attribute value (Python)

I would like to get all attribute values names 'href' from a website, there are like 10 of them. I have successfully got one using the following method:
url = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[1]/div/div[1]/span/a"))).get_attribute("href")
The problem with this is it's only giving back one not all of them. I have tried to go by ID but it doesn't return anything:
url = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "a"))).get_attribute("href")
Also, the other href values are located in different xpaths:
/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[2]/div/div[1]/span/a
/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[3]/div/div[1]/span/a
/html/body/div[2]/div[2]/div/div/div[2]/section[1]/div/div[2]/div[3]/ul/li[4]/div/div[1]/span/a
Here's my element:
<a ph-tevent="job_click" ref="linkEle" href.bind="getUrl(linkEle, 'job', eachJob, '', eachJob.jobUrl)" data-ph-at-id="job-link" data-ph-id="ph-page-element-page20-CRUCUZ" class="au-target" au-target-id="181" ph-click-ctx="job" ph-tref="12313123213" ph-tag="ph-search-results-v2" href="https://hyperlink.com" data-ph-at-job-title-text="title" data-ph-at-job-location-text="Unknown" data-ph-at-job-location-area-text="asd" data-ph-at-job-category-text="Manufacturing" data-access-list-item="2" data-ph-at-job-id-text="A123124" data-ph-at-job-type-text="Regular" data-ph-at-job-industry-text="Manufacturing" data-ph-at-job-post-date-text="2021-12-09T00:00:00.000Z" data-ph-at-job-seqno-text="ASD212ASFS" aria-label="Senior Manager">
<div class="job-title" data-ph-id="ph-page-element-page20-0Mi3Ce">
<!--anchor-->
<!--anchor-->
<span data-ph-id="ph-page-element-page20-PLxqta">Senior Manager </span>
</div><!--anchor--> </a>
Any help is appreciated!

For finding more than one web element, you should either use find_elements or if you are using Explicit waits then you can use presence_of_all_elements_located or visibility_of_all_elements_located.
Based on the HTML that you've shared, if
this css
a[ph-tevent='job_click'][ref='linkEle']
or this xpath
//a[#ph-tevent='job_click' and #ref='linkEle']
represent all the nodes, to check below are the steps:
Please check in the dev tools (Google chrome) if we have all desired nodes entry in HTML DOM or not.
Steps to check:
Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired elements are getting highlighted or not.
If they are then the below code should work:
for ele in driver.find_elements(By.XPATH, "//a[#ph-tevent='job_click' and #ref='linkEle']"):
print(ele.get_attribute('href'))

To extract the value of the href attributes using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[ph-tevent='job_click'][ref='linkEle'][data-ph-at-id='job-link'][ph-click-ctx='job'][href]")))])
Using XPATH:
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[#ph-tevent='job_click' and #ref='linkEle'][#data-ph-at-id='job-link' and #ph-click-ctx='job'][#href]")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Selenium Python: Selecting all items in a class and putting them into a list?

Is there any way to get all the elements with a certain class and put into a list? for example:
<li class="Fruit">Apple</li>
<li class="Fruit">Orange</li>
I would like to put Apple and Orange in a list.

To extract and print the texts e.g. Apple, Orange, etc from all of the <li class="Fruit"> using Selenium and python you can use either of the following Locator Strategies:
Using class_name and get_attribute("textContent"):
print([my_elem.get_attribute("textContent") for my_elem in driver.find_elements_by_class_name("Fruit")])
Using css_selector and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in driver.find_elements_by_css_selector(".Fruit")])
Using xpath and text attribute:
print([my_elem.text for my_elem in driver.find_elements_by_xpath("//*[#class='Fruit']")])
Ideally you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CLASS_NAME and get_attribute("textContent"):
print([my_elem.get_attribute("textContent") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "Fruit")))])
Using CSS_SELECTOR and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".Fruit")))])
Using XPATH and text attribute:
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[#class='Fruit']")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Outro
Link to useful documentation:
get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

driver.find_elements_by_class_name("Fruit")
Find elements returns all elements that much the given criteria ,here the class name , it returns list
you can print the text as:
https://selenium-python.readthedocs.io/locating-elements.html
elems = driver.find_elements_by_class_name("Fruit")
for elem in elems:
print(elem.text)
or
elemsText = [i.text for i in driver.find_elements_by_class_name("Fruit")]
print(elemsText]

Python Selenium - get href value

I am trying to copy the href value from a website, and the html code looks like this:
<p class="sc-eYdvao kvdWiq">
<a href="https://www.iproperty.com.my/property/setia-eco-park/sale-
1653165/">Shah Alam Setia Eco Park, Setia Eco Park
</a>
</p>
I've tried driver.find_elements_by_css_selector(".sc-eYdvao.kvdWiq").get_attribute("href") but it returned 'list' object has no attribute 'get_attribute'. Using driver.find_element_by_css_selector(".sc-eYdvao.kvdWiq").get_attribute("href") returned None. But i cant use xpath because the website has like 20+ href which i need to copy all. Using xpath would only copy one.
If it helps, all the 20+ href are categorised under the same class which is sc-eYdvao kvdWiq.
Ultimately i would want to copy all the 20+ href and export them out to a csv file.
Appreciate any help possible.

You want driver.find_elements if more than one element. This will return a list. For the css selector you want to ensure you are selecting for those classes that have a child href
elems = driver.find_elements_by_css_selector(".sc-eYdvao.kvdWiq [href]")
links = [elem.get_attribute('href') for elem in elems]
You might also need a wait condition for presence of all elements located by css selector.
elems = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".sc-eYdvao.kvdWiq [href]")))

As per the given HTML:
<p class="sc-eYdvao kvdWiq">
Shah Alam Setia Eco Park, Setia Eco Park
</p>
As the href attribute is within the <a> tag ideally you need to move deeper till the <a> node. So to extract the value of the href attribute you can use either of the following Locator Strategies:
Using css_selector:
print(driver.find_element_by_css_selector("p.sc-eYdvao.kvdWiq > a").get_attribute('href'))
Using xpath:
print(driver.find_element_by_xpath("//p[#class='sc-eYdvao kvdWiq']/a").get_attribute('href'))
If you want to extract all the values of the href attribute you need to use find_elements* instead:
Using css_selector:
print([my_elem.get_attribute("href") for my_elem in driver.find_elements_by_css_selector("p.sc-eYdvao.kvdWiq > a")])
Using xpath:
print([my_elem.get_attribute("href") for my_elem in driver.find_elements_by_xpath("//p[#class='sc-eYdvao kvdWiq']/a")])
Dynamic elements
However, if you observe the values of class attributes i.e. sc-eYdvao and kvdWiq ideally those are dynamic values. So to extract the href attribute you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "p.sc-eYdvao.kvdWiq > a"))).get_attribute('href'))
Using XPATH:
print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//p[#class='sc-eYdvao kvdWiq']/a"))).get_attribute('href'))
If you want to extract all the values of the href attribute you can use visibility_of_all_elements_located() instead:
Using CSS_SELECTOR:
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "p.sc-eYdvao.kvdWiq > a")))])
Using XPATH:
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//p[#class='sc-eYdvao kvdWiq']/a")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

The XPATH
//p[#class='sc-eYdvao kvdWiq']/a
return the elements you are looking for.
Writing the data to CSV file is not related to the scraping challenge. Just try to look at examples and you will be able to do it.

To crawl any hyperlink or Href, proxycrwal API is ideal as it uses pre-built functions for fetching desired information. Just pip install the API and follow the code to get the required output. The second approach to fetch Href links using python selenium is to run the following code.
Source Code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
import time
list = ['https://www.heliosholland.com/Ampullendoos-voor-63-ampullen','https://www.heliosholland.com/lege-testdozen’]
driver = webdriver.Chrome()
wait = WebDriverWait(driver,29)
for i in list:
driver.get(i)
image = wait.until(EC.visibility_of_element_located((By.XPATH,'/html/body/div[1]/div[3]/div[2]/div/div[2]/div/div/form/div[1]/div[1]/div/div/div/div[1]/div/img'))).get_attribute('src')
print(image)
To scrape the link, use .get_attribute(‘src’).

Get the whole element you want with driver.find_elements(By.XPATH, 'path').
To extract the href link use get_attribute('href').
Which gives,
driver.find_elements(By.XPATH, 'path').get_attribute('href')

try something like:
elems = driver.find_elements_by_xpath("//p[contains(#class, 'sc-eYdvao') and contains(#class='kvdWiq')]/a")
for elem in elems:
print elem.get_attribute['href']