Develop Reference

Extract id of the div using selenium in Python

Can someone help me to extract the id text (the underline one in the below-attached image) from this link: https://opennem.org.au/facilities/au/?selected=DEIBDL&status=operating,committed,commissioning,retired
To get the right on the item in any of the items in the table and inspect.
I knew how to extract the table text but I want the id text which I underlined

The selected element can be located by this XPath
//div[#class='card is-selected']
Or this CSS Selector
div.is-selected
So, you can use this Selenium code to extract the id attribute value:
id = driver.find_element(By.XPATH,"//div[#class='card is-selected']").get_attribute("id")
But this will give you the id appearing on the dev tools, this is not always the text you see as a user on the page.
To get the text user sees you will need this:
name = driver.find_element(By.CSS_SELECTOR,"div.is-selected .station-name").text
This can be done woth XPath as well

Python Selenium pulling child property from parent?

I'm trying to scrape a webform for text in specific fields however i can't do it with xpath because some forms are missing fields which won't be included in the page when it loads (i.e. if /html/blah/blah/p[3] is the initials field for one form it might be first name on another form but have the same xpath. The structure for the fields is like this:
<p><strong>Initials:</strong> WT</p>
so using python selenium i'm doing
driver.find_element_by_xpath("//*[contains(text(), 'Initials:')]") which does successfully pull the "Initials:" text between the strong tags but i specifically need the child text after it, in this case WT. It has the attribute "nextSibling.data" which contains the WT value but from my googling i don't think its possible to pull that attribute with python selenium. Does anyone know a way to pull the WT text following the xpath query?

The 'WT' text is in a weird spot. I don't think it is actually a sibling per-se. The only way I know to grab that text would be to use p_element.get_attribute('outerHTML'), which in this instance should grab the string '<p><strong>Initials:</strong> WT</p>'. I doubt this is the cleanest solution, but here's a way to parse that text out:
strong_close_tag = '</strong>'
p_close_tag = '</p>'
p_element = driver.find_element_by_xpath("//*[contains(text(), 'Initials:')]/parent")
print(p_element.get_attribute('outerHTML')[text.index(strong_close_tag)+len(strong_close_tag):text.index(p_close_tag)])
OR -- use p_element.get_attribute('innerHTML'), which should return just <strong>Initials:</strong> WT. Then, similarly, grab the text after the </strong> closing tab, maybe like this:
p_element = driver.find_element_by_xpath("//*[contains(text(), 'Initials:')]/parent")
print p_element.get_attribute('innerHTML').split("</strong>",1)[1]

How would you click all texts on a page with Xpath - Python

So, this won't be a long description, but I am trying to have xpath click on all of the elements (more specifically text elements) that are on a page. I really don't know where to start, and all of the other questions on clicking everything on a page is based on a class, not a text using xpath.
Here is some of my code:
browser.find_element_by_xpath("//*[text()='sample']").click()
I really don't know how I would go about to make it click all of the "sample" texts throughout the whole page.
Thanks in advance!

Well, let's say that you have lots of Divs or spans that contains text. Let's figure out Divs :
<div class="some class name" visibility ="visible" some other attribute> Text here </div>
Now when you go to developer mode(F12) in elements section and if you do this //div[contains(#class,'some class name')] and if there are more than 1 entry then you can store all of them in a list just like below :
driver.find_elements(By.XPATH, '//div[contains(#class,'some class name')]')
this will give you a list of divs web element.
div_list = driver.find_elements(By.XPATH, '//div[contains(#class,'some class name')]')
Now you have a python list and you can manipulate this list as per your requirement.
for div_text in div_list:
print(div_text.text)
Same way you can try for span or different web elements.

You just need to use that xpath to define an array of elements instead, like this:
my_elements = browser.find_elements_by_xpath("//*[text()='sample']")
for element in my_elements:
element.click();
That loop may not work as is (you could maybe add a wait for element) but that's the idea.

how to resolve changes in Element's id in html?

After deployement id of elements are changed. So that can not select those element by id in selenium using python.
Suppose I want to find the below element in HTML.
<input class="o_form_input c_field-65 o_form_field o_form_required" id="o_field_input_22" type="text">
I can not use the element's class because there will be elements with the same class.
I would like to find element without using Xpath because if new fields are added in the development side then Xpath will be changed.

In that case you should not be locating your element by id (since it not steady).
For example you can use Xpath (other locators may also work):
login_form = driver.find_element_by_xpath("//form[#id='loginForm']")
or
username = driver.find_element_by_xpath("//form[#id='loginForm']/input[1]")
Now the only thing that you can't copy from this, is the xpath it self. That is specific to your website/html/DOM.
An easy way of getting a correct xpath is by inspecting elements using f12 and then right click on the element, go to copy and select [copy xpath]. You can than paste it into your code.
Let me know it this was helpfull!

Xpath clicking not working at all

Quick info: I'm using Mac OS, Python 3.
I have like 800 links that need to be clicked on a page (and many more pages to go so need automation).
They were hidden because you only see those links when you hover over.
I fixed that by injecting CSS rule (just saying in case its the reason it's not working).
When I try to find elements by xpath it does not want to click the links afterwards and it also doesn't find all of them always just 4 (even when more are displayed in view).
HTML:
Display
When i click ok copy xpath in inspect it gives me:
//*[#id="tiles"]/li[3]/div[2]/ul/li[2]/a
But it doesn't work when I use it like this:
driver.find_elements_by_xpath('//*[#id="tiles"]/li[3]/div[2]/ul/li[2]/a')
So two questions:
How do I get them all?
How do I get it to click on each of them?
The pattern in the XPath is the same, with the /li[3] being the only number that changes, for this I created a for loop to create them all based on the count on page which I did successfully.
So if it can be done with the XPaths generated by myself that are corresponding to when I copy XPath in inspector then I only need question 2 answered.
PS.: this is HTML of parent of that first HTML:
<li onclick="openPopup(event, 'collect', {item_id: 165214})" class="collect" data-item-id="165214">Display</li>

This XPath,
//a[.="Display"]
will select all a links with anchor text equal to "Display".

As per your question, the HTML you have shared and your code attempts there is no necessity to get the <li> tags. Instead we will get the <a> tags in a list. So to answer your first question How do I get them all you can use the following line of code :
all_Display = driver.find_elements_by_xpath("//*[#id='tiles']//li/div[2]/ul/li[#class='collect']/a[#title='Display']")
Next to click on each of them you have to create a loop to iterate through all the <a> tag as follows :
all_Display = driver.find_elements_by_xpath("//*[#id='tiles']//li/div[2]/ul/li[#class='collect']/a[#title='Display']")
for each_Display in all_Display :
each_Display.click()

Using an XPath with elements by position is not ideal. Instead use a CSS selector to match the attributes for the targeted elements.
Something like:
all_Display = driver.find_elements_by_css_selector("#tiles li[onclick][data-item-id] a[title]")
You can then click them in a loop if none of them is loading a new page:
for element in all_Display:
element.click()