Develop Reference

Locating an element using Python and Selenium via innerHTML

I'm new to Selenium and I'm trying to write my first real script using the package for Python.
I'm using:
Windows 10
Python 3.10.5
Selenium 4.3.0
So far I've been able to do everything I need with different selectors, like ID, name, XPATH etc.
However I've stumbled upon an issue where I need to find a specific element by using the innerHTML of it.
The issue I'm facing is that I need to find an element with the innerHTML-value of "Changed" as seen in the HTML below.
The first challenge I'm facing is that the element doesn't have a unique ID, name or otherwise to identify it and there's many objects/elements of "dlx-treeview-node".
The second challenge is that XPATH won't work because the element changes position depending on where you are on the website (the number of "dlx-treeview-node"-elements change), so if I use XPATH I'll get the wrong element depending on where I am.
I can successfully get the name by using the below XPATH, "get_attribute" and printing to console, which is why I know it's innerHTML and not innerText, but as mentioned this will change depending on where I am on the website.
I would really appreciate any help I can get to solve this challenge and to learn more about the use of Selenium with Python.
Code trials:
select_filter_name = wait.until(EC.element_to_be_clickable((By.XPATH, "/html/body/div/app-root/dlx-select-filter-attribute-dialog/dlx-dialog-window/div/div[2]/div/div/div[5]/div/div/dlx-view-column-selector-component/div[1]/dlx-treeview/div/dlx-treeview-nodes/div/dlx-treeview-nodes/div/dlx-treeview-node[16]/div/div/div/div[2]/div/dlx-text-truncater/div")))
filter_name = select_filter_name.get_attribute("innerHTML")
print(filter_name)
HTML:
<dlx-treeview-node _nghost-nrk-c188="" class="ng-star-inserted">
<div _ngcontent-nrk-c188="" dlx-droppable="" dlx-draggable="" dlx-file-drop="" class="d-flex flex-column position-relative dlx-hover on-hover-show-expandable-menu bg-control-active bg-control-hover">
<div _ngcontent-nrk-c188="" class="d-flex flex-row ml-2">
<div _ngcontent-nrk-c188="" class="d-flex flex-row text-nowrap expand-horizontal" style="padding-left: 15px;">
<!---->
<div _ngcontent-nrk-c188="" class="d-flex align-self-center ng-star-inserted" style="min-width: 16px; margin-left: 3px;">
<!---->
</div>
<!---->
<div _ngcontent-nrk-c188="" class="d-flex flex-1 flex-no-overflow-x" style="padding: 3.5px 0px;">
<div class="d-flex flex-row justify-content-start flex-no-overflow-x align-items-center expand-horizontal ng-star-inserted">
<!---->
<dlx-text-truncater class="overflow-hidden d-flex flex-no-overflow-x ng-star-inserted">
<div class="text-truncate expand-horizontal ng-star-inserted">Changed</div>
<!---->
<!---->
</dlx-text-truncater>
<!---->
</div>
<!---->
<!---->
<!---->
</div>
</div>
<!---->
<!---->
</div>
</div>
<!---->
<dlx-attachment-content _ngcontent-nrk-c188="">
<div style="position: fixed; z-index: 10001; left: -10000px; top: -10000px; pointer-events: auto;">
<!---->
<!---->
</div>
</dlx-attachment-content>
</dlx-treeview-node>
Edit-1:
NOTE: I'm not sure I'm using the correct terms for HTML, so please correct me if I'm wrong.
I've learned that I have a follow up question:
How do I search for the text as described, but only searching in the "dlx-treeview-node" (there's about 100 of these)? So basically searching in the "children" of these.
The question is because I've learned that there are more elements with the specific text I'm searching for in other places.
Edit-2/solution:
I ended up finding my own solution before I received answers - I'm writing it here in case it can help anyone else.
The reply that is marked as "answer" is because this came the closest to what I needed.
The final code ended up like this (first searching the nodes - then searching the children for the specific innerHTML):
select_filter_name = wait.until(EC.element_to_be_clickable((By.XPATH, "//dlx-treeview-node[.//div[text()='Changed']]")))

Presuming the innerText of the <div> element as a unique text within the HTML DOM to locate the element with the innerHTML as Changed you can use either of the following xpath based locator strategies:
Using xpath and text():
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Changed']")))
Using xpath and contains():
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[contains(., 'Changed')]")))

just run this code on your page and you will get an array of all elements which are a div with the value of Changed
# Define XPath Function (used in the next step)
driver.execute_script("function getXPathOfElement(elt) {var path = "";for (; elt && elt.nodeType == 1; elt = elt.parentNode) { idx = getElementIdx(elt); xname = elt.tagName; if (idx > 1) xname += "[" + idx + "]"; path = "/" + xname + path;} return path;}")
# Get all XPaths for all nodes which are a div with the text of "changed"
xpaths = driver.execute_script("return Array.from(document.querySelectorAll(\"div\")).find(el => el.textContent.includes('Changed')).map((node)=>{ return getXPathOfElement(node)});');
write up
the first execute adds a javascript function to the dom called getXPathOfElement this function accepts a html node element and will provide the xpath string for said node.
the second execute gets all elements which are a div with the text of Changed this will then loop through each element and then provide you with an array of strings, where each string is an xpath by calling the above getXPathOfElement function on each node.
the js is quite simple and harmless.
Tips
check if xpaths length is more than or equal to 1
index xpaths such as xpaths[0] or do loops to make your changes
you will now have an xpath which can be used like a normal selector.
good luck
Edit 1
execute_script() synchronously executes JavaScript in the current window/frame.
or find more here

Selenium + Python: Print the text attribute of an element

I would like to navigate through a website, find an element and print it.
Python version: 3.10; Selenium Webdriver: Firefox; IDE: PyCharm 2021.3.2 (CE);
OS: Fedora 35 VM
I am able to navigate to the appropriate page where the text is generated in a drop down menu.
When I locate the element by CSS Selector and attempt to print it, the output does print the text "None".
I would like it to print the Plan Name which in this case is "Dual Complete Plan 1".
The element is not always present so I also need to catch any exceptions.
The relevant HTML code of the element I am trying to print:
<span class="OSFillParent" data-expression="" style="font-size: 12px; margin-top: 5px;">Dual Complete Plan 1</span>
More of the HTML code of the element I am trying to print (element I am trying to capture is below the fourth div):
<td data-header="Plan Name">
<div id="b8-b40-l1_0-132_0-$b2" class="OSBlockWidget" data-block="Content.AccordionItem">
<div id="b8-b40-l1_0-132_0-b2-SectionItem" class="section-expandable open is--open small-accordion" data-container="" data-expanded="true" aria-expanded="true" aria-disabled="false" role="tab">
<div id="b8-b40-l1_0-132_0-b2-TitleWrapper" class="section-expandable-title" data-container="" style="cursor: pointer;" role+"button" aria-hidden="false" aria-expanmded="true" tabindex="0" aria-controls="b8-b40-l1_0-132_0-b2-Content" EVENT FLEX
<div id="b8-b40-l1_0-132_0-b2-Title" class="dividers full-width">
<span class="OSFillParent" data-expression="" style="font-size: 12px; margin-top: 5px;">Dual Complete Plan 1</span>
</div>
<div class="section-expandable-icon" data-container="" aria-hidden="true"
::after
</div>
</div>
<div id="b8-b40-l1_0-132_0-b2-ContentWrapper" class="section-expandable-content no-padding is--expanded" data-container="" tabindex="0" aria-hidden="false" aria-labelledby="b8-b40-l1_0-132_0-b2-TitleWrapper">
<div id="b8-b40-l1_0-132_0-b2-Content" role="tabpanel">
<a data-link="" href="https://www.communityplan.com" target="_blank" title="Click for more information"> EVENT
<span class="OSFillParent" data-expression="" style="font-size: 12px;">www.CommunityPlan.com</span>
</a>
<span class="OSFillParent" data-expression="" style="font-size: 12px:">Phone Number: 8005224700</span>
</div>
</div>
</div>
</div>
</td>
My relevant Selenium code:
# Find the Plan Name & if present set it to the variable "Advantage"
try:
Advantage = (WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "#b8-b40-l1_0-132_0-b2-Title > span:nth-child(1)"))).get_attribute("value"))
except:
pass
print('\033[91;46m', Advantage, '\033[0m')
I expect the output to be "Dual Complete Plan 1", which is what I see on the screen and in the HTML. Instead I get the following:
None
Apparently the "Advantage" variable is being set to "None".
Why?
I can see the text "Dual Complete Plan 1" that I want to print in the HTML code above.
What am I doing wrong?
I feel like I need a primer on "get attribute"?

To get the text Dual Complete Plan 1 you need to use
element.text
or
element.get_attribute("innerHTML")
or
element.get_attribute("textContent")
Instead of presence_of_element_located() use visibility_of_element_located()
and following css selector to identify
div[id*='Title'] > span.OSFillParent
Or
div.dividers.full-width > span.OSFillParent
Code:
try:
Advantage = WebDriverWait(driver, 5).until(
EC.visibility_of_element_located((By.CSS_SELECTOR, "div[id*='Title'] > span.OSFillParent"))).text
except:
pass
print(Advantage )

Unable to Locate Element for Selenium Webdriver Path in Python

I've been trying to reference the two inputs in the following HTML code using Selenium's get_element_by_...() with the purpose of automatically entering a username and password into the fields. I'm running into trouble with accessing the proper HTML elements.
<body>
<div id="app">
<div>
<section id="root" class="page-login">
<section class="main">
<section class="auth-page register-fill">
<div class="page-center">
<a class="switch-lang" href="/cn/login">cn</a>
<a class="btn-back" href="en/login">
<span class="isvg loaded">
<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100" viewBox="0 0 24 24">
<path class="a" d="M0,0H24V24H0Z"></path>
<path class="b" d="M20,11H7.83l5.59-5.59L12,4,4,12l8,8,1.41-1.41L7.83,13H20Z"></path>
</svg>
</span>
</a>
<div class="auth-page-wrapper">
<h1 class="auth-page-title">Welcome Back</h1>
<h2 class="auth-page-subtitle">Login to get started</h2>
<div class="auth-page-fields">
<div class="custom-input">
<input type="text" placeholder="Email">
<label>Email</label>
</div>
<div class="custom-input custom-input-password">
<a class="forgot-link" href="en/forgot-password">Forgot Password?</a>
<input type="password" placeholder="Minimum 6 characters">
<label>Password</label>
</div>
</div>
Note: The HTML may have some mistakes, I just quickly formatted it, but the mistakes should not affect the question.
Tried:
I've tried using all the different functions of the 'form find_element_by_...' and referencing a variety of things, but all tend to return "no such element: unable to locate element ..."
I've done absolute xpaths, relative xpaths, css selectors, ids, and combinations of these.
Some examples:
username = browser.find_element_by_xpath("//input[1]")
username = browser.find_element_by_xpath("//html/body/div/div/section/.../input")
The error begins at this point in the path:
username = browser.find_element_by_xpath("//html/body/div/div/section")
I'm likely making a stupid mistake, so sorry if this question isn't great, but other StackOverflow answers all have ids and such that are referable on the input field. (eg. <input id=username ...>)

To identify the Email field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[placeholder='Email']")))
Using XPATH:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#placeholder='Email']")))
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

There are more than one way to do this. But, this should work browser.find_element_by_css_selector('[placeholder="Email"]')

Can find element within an iframe using Firefox but not Chrome

So I'm trying to write a test for a webpage which has some elements within an iframe. I've been able to successfully run the test using webdriver.Firefox() without any problems but if I switch it over to webdriver.Chrome() I get a timeout exception on the following lines of code:
self.driver.switch_to.frame(0)
self.activity_status = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#overview > div.details.w-66 > div > div.duration-and-status > span.status.stat_incomplete#')))
It'd be great to get a solution to this as I'm all out of ideas.
Thanks for your help.
edit, partial html for the page:
<iframe id="iframe_course_details" allowfullscreen="" src="../Course/Details.aspx?HidePageNav=true&IsInIframe=true"></iframe>
Close
Edit (Inactive)
Edit
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_favourite" class="favourite button tooltipstered" style="display: none;">Favourite</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_basket_dull" class="add-to-basket button delete tooltipstered" style="display: none;">Enrolled (Remove From Enrolments)</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_basket" class="add-to-basket button tooltipstered">Add to Enrolments</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_print" class="print button tooltipstered">Print</span>
</div>
<section id="overview" style="opacity: 1;">
<div id="fullname" class="fullname w-100" style="display: none;">
</div>
<div class="image w-33" style="cursor: pointer;">
<div style="background-image:url(/App_Themes/MainTheme-responsive/Images/Course/webcast.jpg);"></div></div>
<div class="details w-66">
<div class="inner">
<h2>testing activity</h2>
<div class="star-rating-num-ratings">
<div class="star-rating">
<span></span><span></span><span></span><span></span><span></span>
</div>
<span class="num-of-ratings">0 Ratings</span>
</div>
<div class="duration-and-status">
<span class="duration">
<label>
Duration:
</label>
<span>0</span>
</span>
<span class="status stat_incomplete">Started</span>
</div>
Edit 2:
So we've managed to find a solution to this and its even more confusing than the original problem
WebDriverWait(self.driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, 'iframe_course_details')))
time.sleep(0)
self.activity_status = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//*[#id="overview"]/div[3]/div/div[2]/span[2]')))
I'd be really curious to hear some theories on why this works, it times out without the 'time.sleep(0).

If you reference the iframe directly rather then an integer that will work between Firefox/Chrome.
self.driver.switch_to.frame(driver.find_element_by_name("iframe"))
You can find the iframe element any way you wish e.g by css/xpath etc

As the the desired element is within an <iframe> so to invoke click() on the element you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Code Block:
# as per your comment assuming -> there is only one frame on the page
WebDriverWait(self.driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME,"iframe")))
self.element = self.activity_status = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#overview > div.details.w-66 > div > div.duration-and-status > span.status.stat_incomplete#')))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a relevant detailed discussion in:
Ways to deal with #document under iframe

Unable to check Checkbox

I am trying to check the checkbox but i am getting following error
selenium.common.exceptions.NoSuchElementException: Message: no such
element: Unable to locate element: {"method":"xpath","selector":"//mat-
checkbox[#id='mat-checkbox-1']/label/div"}
I tried selecting using id, css-selector and xpath but keep on getting error.
reg_date_checkbox = self.browser.find_element_by_xpath("//matcheckbox[#id='mat-checkbox-1']/label/div")
reg_date_checkbox.location_once_scrolled_into_view
reg_date_checkbox.click()
The HTML of checkbox is
<div _ngcontent-fep-c23="" class="last-date my-12 ng-star-inserted" fxlayout="column" style="flex-direction: column; box-sizing: border-box; display: flex;">
<mat-checkbox _ngcontent-fep-c23="" class="mat-checkbox mat-accent ng-untouched ng-pristine ng-valid" formcontrolname="deadline_enabled" id="mat-checkbox-1">
<label class="mat-checkbox-layout" for="mat-checkbox-1-input">
<div class="mat-checkbox-inner-container">
<input class="mat-checkbox-input cdk-visually-hidden" type="checkbox" id="mat-checkbox-1-input" tabindex="0" aria-checked="false">
<div class="mat-checkbox-ripple mat-ripple" matripple="">
<div class="mat-ripple-element mat-checkbox-persistent-ripple"></div>
</div>
<div class="mat-checkbox-frame"></div>
<div class="mat-checkbox-background">
<svg xml:space="preserve" class="mat-checkbox-checkmark" focusable="false" version="1.1" viewBox="0 0 24 24">
<path class="mat-checkbox-checkmark-path" d="M4.1,12.7 9,17.6 20.3,6.3" fill="none" stroke="white"></path>
</svg>
<div class="mat-checkbox-mixedmark"></div>
</div>
</div><span class="mat-checkbox-label"><span style="display:none"> </span>The event has a registration deadline</span>
</label>
</mat-checkbox>
<!---->

You are probably trying to click on the element which is NOT of type 'checkbox'. I see there is an input with type checkbox. Please try below mentioned code
reg_date_checkbox = self.browser.find_element_by_xpath("//input[#id='mat-checkbox-1-input']")
reg_date_checkbox.click()
Additionally if you wish you perform uncheck you can look for input element's aria-checked property to be true before performing a click.

You are using this xpath //matcheckbox[#id='mat-checkbox-1']/label/div , it should be //mat-checkbox[#id='mat-checkbox-1']/label/div , there's a - in between, you are missing that.
Still you can try with this css selector :
div.mat-checkbox-inner-container input[class^='mat-checkbox-input'][id^='mat-checkbox']
There may be chances that the checkbox is in iframe, if it is the case then first switch to frame and then try to click on check box.

You could try this:
element = self.browser.find_element_by_css("input#mat-checkbox-1-input")
element.click()