Develop Reference

Cannot input date to field with shadow-root (user-agent) with Selenium Python

I am trying to write a stable code to input a date into calendar input field.
Native Selenium inputs work only locally, but when I run this code in CICD I see that numbers are sometime inputted into wrong places.
For example, the timestamp is 2022-07-22 00:56:42.
Expected input by user is 220720220056.
By experimenting I found out that year works if there are two zeros in front of the year. Therefore, what I actually send is 22070020220056
input_field = driver.find_element(By.XPATH, "//div[#role='dialog']//input[#type='datetime-local']")
input_field.click()
input_field.clear()
input_field.send_keys(22070020220056)
<input _ngcontent-pau-c188="" clrinput="" type="datetime-local" name="expiry" aria-describedby="clr-form-control-9-helper" id="clr-form-control-9" class="clr-input ng-pristine ng-valid ng-touched">
#shadow-root (user-agent) == $0
<div pseudo="-webkit-datetime-edit" id="date-time-edit" datetimeformat="dd/MM/y, HH:mm">
<div pseudo="-webkit-datetime-edit-fields-wrapper">
<span role="spinbutton" aria-placeholder="dd" aria-valuemin="1" aria-valuemax="31" aria-label="Day" pseudo="-webkit-datetime-edit-day-field" aria-valuenow="25" aria-valuetext="25">25</span>
<div pseudo="-webkit-datetime-edit-text">/</div>
<span role="spinbutton" aria-placeholder="mm" aria-valuemin="1" aria-valuemax="12" aria-label="Month" pseudo="-webkit-datetime-edit-month-field" aria-valuenow="7" aria-valuetext="07">07</span>
<div pseudo="-webkit-datetime-edit-text">/</div>
<span role="spinbutton" aria-placeholder="yyyy" aria-valuemin="1" aria-valuemax="275760" aria-label="Year" pseudo="-webkit-datetime-edit-year-field" aria-valuenow="2022" aria-valuetext="2022">2022</span>
<div pseudo="-webkit-datetime-edit-text">, </div>
<span role="spinbutton" aria-placeholder="--" aria-valuemin="0" aria-valuemax="23" aria-label="Hours" pseudo="-webkit-datetime-edit-hour-field" aria-valuenow="0" aria-valuetext="00">00</span>
<div pseudo="-webkit-datetime-edit-text">:</div>
<span role="spinbutton" aria-placeholder="--" aria-valuemin="0" aria-valuemax="59" aria-label="Minutes" pseudo="-webkit-datetime-edit-minute-field" aria-valuenow="1" aria-valuetext="01">01</span>
</div>
</div>
</input>
Field looks like this:
I tried few other options to make the test more stable.
Option 1.
Tried to set the value directly.
driver.execute_script(f"arguments[0].setAttribute('value', '{my_timestamp}')", input_field)
Option 2
Tried to access shadow root:
date_field = driver.execute_script("return document.querySelector('input[type=datetime-local]').shadowRoot.getElementById('#date-time-edit')")
date_field.send_keys(my_timestamp)
Option 3
Tried to get parent element with execute_script instead of Selenium and send value:
date_field = driver.execute_script("return document.querySelector('input[type=datetime-local]')")
date_field.send_keys(my_timestamp)
Option 4
Tried to send text with a small delay before each letter:
for letter in timestamp:
time.sleep(0.1)
print(f"Sending letter {letter}")
date_field.send_keys(letter)
None of these methods worked on CICD, Only initial one and Option 3 always work locally.
I think the only stable solution would be the use of Javascipt code, but still not sure.
Any suggestion would be greatly appreciated.

Ah yes, the shadow root. For me, what works is something like the following:
shadowRootElement = driver.execute_script('''return document.querySelector("selector for shadowRoot's parent").shadowRoot''')
shadowRootChild = driver.execute_script('''return arguments[0].querySelector("selector for shadowRoot's parent").shadowRoot''', shadowRootElement)
shadowRootChild.find_element(By.CSS_SELECTOR, 'input[type=datetime-local]').send_keys("This is a test");

The <input> looks perfectly out of #shadow-root (user-agent) ambit.
<input _ngcontent-pau-c188="" clrinput="" type="datetime-local" name="expiry" aria-describedby="clr-form-control-9-helper" id="clr-form-control-9" class="clr-input ng-pristine ng-valid ng-touched">
#shadow-root (user-agent) == $0
<div pseudo="-webkit-datetime-edit" id="date-time-edit" datetimeformat="dd/MM/y, HH:mm">
<div pseudo="-webkit-datetime-edit-fields-wrapper">
.
.
.
</div>
</div>
</input>
However, the desired element is a Angular element, so ideally to send a character sequence to the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategy:
Using XPATH:
input_field = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#class='clr-input ng-pristine ng-valid ng-touched' and #name='expiry'][#type='datetime-local']")))
driver.execute_script(f"arguments[0].setAttribute('value', '{my_timestamp}')", input_field)
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Is there a way to select an item on a webpage with Selenium with no unique ID?

My objective is to open a webpage, and click the app button for a specific app, like Anaplan. In the past, I've used get element by CSS selector with the combination of class, and ID, as shown in this past post.
first_item = driver.find_element_by_id("anaplan")
I've come across a webpage where the buttons seem to have literally no ID whatsoever, or unique values:
HTML output of the Anaplan App button:
<a
aria-label="launch app Anaplan"
class="chiclet a--no-decoration"
data-se="app-card"
href="https://gartner.okta.com/home/anaplan/0oaforg08lyATdLuw4x6/2487"
draggable="true"
><article class="chiclet--article">
<button
class="chiclet--action"
tabindex="0"
aria-label="Settings for Anaplan"
data-se="app-card-settings-button"
>
<svg
class="chiclet--action-kebab"
width="20"
height="4"
viewBox="0 0 20 4"
fill="#B7BCC0"
xmlns="http://www.w3.org/2000/svg"
>
<circle cx="2" cy="2" r="2"></circle>
<circle cx="10" cy="2" r="2"></circle>
<circle cx="18" cy="2" r="2"></circle>
</svg>
</button>
<section class="chiclet--main" data-se="app-card-main">
<img
class="chiclet--main-logo"
src="https://ok11static.oktacdn.com/fs/bcg/4/gfs1ev15ab63zqgZ91d8"
alt="Anaplan logo"
/>
</section>
<footer class="chiclet--footer" data-se="app-card-footer">
<o-tooltip content="Anaplan" position="bottom" class="hydrated"
><div slot="content"></div>
<div aria-describedby="o-tooltip-0">
<h1 class="chiclet--app-title" data-se="app-card-title">Anaplan</h1>
</div>
</o-tooltip>
</footer>
</article>
</a>
I grabbed the Xpath of the Anaplan button, which shows the following:
/html[#class='hydrated wf-proximanova-n4-inactive wf-
inactive']/body[#class='default']/div[#id='root']
/div[#class='enduser-app ']/section[#class='content-frame']
/main[#class='main-container has-top-bar']/div[#class='dashboard--main']/section[#id='main-
content']/section[#class='chiclet-area']
/section[#class='chiclet-grid--container']
/section/section[#class='chiclet-grid section-appear-done section-enter-done']
/a[#class='chiclet a--no-decoration'][1]/article[#class='chiclet--article']
The only differences between apps is the number in the bracket:
/a[#class='chiclet a--no-decoration'][1], where 1 seems to be Anaplan, 3 is G Drive, and so on. Is there a way to select elements such as this where there appears to be no unique identifier at all?

To locate the first button you can use one of the following xpaths //a[#aria-label='launch app Anaplan'] or //a[contains(#href,'anaplan')] and there are many other unique combinations. The same can be done with css selectors
Similarly to the above there are several combinations for all the other navigation buttons you provided here.
In case the element located inside <iframe> you have to switch to that <iframe> first and get out of it after that.
Locate the <iframe> with
iframe = driver.find_element_by_xpath("//iframe[#name='iframeName']") or whatever locator that it matches
Then switch_to the <iframe>:
driver.switch_to.frame(iframe)
If after that you need to continue anywhere out of the <iframe> switch out of it with
driver.switch_to.default_content()

It is possible both with xpath and css.
Example of xpath:
Anaplan:
//a[contains(#aria-label, 'Anaplan')]/article/button
Or:
//button[contains(#aria-label, 'Settings for Anaplan')]
Spam Quarantine:
//a[contains(#aria-label, 'Spam Quarantine')]
G-suite
//a[contains(#aria-label, 'G Suite Drive')]
The main idea is that you can find an element by writing a partial name of an attribute.
Update:
If an element is located inside an iframe, you should wait for it to load and switch to it. Selenium has very convenient method for it: frame_to_be_available_and_switch_to_it
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
driver = webdriver.Chrome()
driver.get(url)
wait = WebDriverWait(driver, 15)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[data-testid=shell-content]")))
After switching to iframe you work with elements inside it.

How to get the text inside span data-bind part of html using selenium Python?

This is the html snippet I am trying to web scrape using Python Selenium.
I am trying to get the text Add to bag which is inside a span data-bind.
<div class="is-add-item-saving" data-bind="visible: isBusy" style="display: none;"></div>
<span class="aria-live" aria-role="status" aria-live="polite" data-bind="{ text: ariaLiveText }"></span>
<button data-bind="click: addToBag, css : buttonCss, attr: { 'aria-label': resources.pdp_cta_add_to_bag, disabled: isBusy }, markAndMeasure: 'pdp:add_to_bag_interactive'" data-test-id="add-button" aria-label="Add to bag">
<span class="product-tick" data-bind="visible: showProductTick" style="display: none;"></span>
<span data-bind="text: buttonText">Add to bag</span>
</button>
This is what I have tried so far.
instock_element = driver.find_elements_by_xpath("//span[contains(#data-bind,'text: buttonText')]")
instock_element = driver.find_elements_by_xpath("//*[contains(text(), 'Add to bag')]")
When I iterate over these instock_elements,
for value in instock_element:
print("text : ",value.text)
print(" id : ",value.id)
if len(value.text) == 0:
text = value.id
else:
print(value.text)
text = value.text
ins_list.append(text)
These are giving me random values like 6489355d-9dd3-4d77-a0d7-b134ce48fae7 but not the text Add to bag.

To print the text Add to bag you can use either of the following Locator Strategies:
Using css_selector and get_attribute("innerHTML"):
print(driver.find_element_by_css_selector("button[data-test-id='add-button'][aria-label='Add to bag'] span").get_attribute("innerHTML"))
Using xpath and text attribute:
print(driver.find_element_by_xpath("//button[#data-test-id='add-button' and #aria-label='Add to bag']//span").text)
Ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and text attribute:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button[data-test-id='add-button'][aria-label='Add to bag'] span"))).text)
Using XPATH and get_attribute():
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//button[#data-test-id='add-button' and #aria-label='Add to bag']//span"))).get_attribute("innerHTML"))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
References
Link to useful documentation:
get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

Try this (in particular the xpath):
from lxml import html
sample = """<div class="is-add-item-saving" data-bind="visible: isBusy" style="display: none;"></div>
<span class="aria-live" aria-role="status" aria-live="polite" data-bind="{ text: ariaLiveText }"></span>
<button data-bind="click: addToBag, css : buttonCss, attr: { 'aria-label': resources.pdp_cta_add_to_bag, disabled: isBusy }, markAndMeasure: 'pdp:add_to_bag_interactive'" data-test-id="add-button" aria-label="Add to bag">
<span class="product-tick" data-bind="visible: showProductTick" style="display: none;"></span>
<span data-bind="text: buttonText">Add to bag</span>
</button>"""
print(html.fromstring(sample).xpath("//*[#data-bind='text: buttonText']/text()"))
Output:
['Add to bag']

https://www.selenium.dev/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webelement.html#module-selenium.webdriver.remote.webelement
id
Internal ID used by selenium.
This is mainly for internal use. Simple use cases such as checking if
2 webelements refer to the same element, can be done using ==:
if element1 == element2:
print("These 2 are equal")
use value.get_attribute("id") instead to get id
to get text use:
value.text
if it fails use:
value.get_attribute("textContent")
as value.text retrieves only text that is displayed in UI

If you are having hard time finding the right element,
Easy way is to instead of finding all the elements of associated with xpath,
You must use the full xpath of the individual tag and then get the text of it using .text
Example:
text = driver.find_element_by_xpath("full xpath of the element").text

You may also use BeautifulSoup for this :
from bs4 import BeautifulSoup
html = """<div class="is-add-item-saving" data-bind="visible: isBusy" style="display: none;"></div>
<span class="aria-live" aria-role="status" aria-live="polite" data-bind="{ text: ariaLiveText }"></span>
<button data-bind="click: addToBag, css : buttonCss, attr: { 'aria-label': resources.pdp_cta_add_to_bag, disabled: isBusy }, markAndMeasure: 'pdp:add_to_bag_interactive'" data-test-id="add-button" aria-label="Add to bag">
<span class="product-tick" data-bind="visible: showProductTick" style="display: none;"></span>
<span data-bind="text: buttonText">Add to bag</span>
</button>"""
soup = BeautifulSoup(html)
tag = soup.find('span',{'data-bind':'text: buttonText'})
print(tag.text)
Output
Add to bag

Get value when ID is changing

I need to retrieve value "[wkSu-'$bS[U#;" from HTML bellow. The issue is that id="ext-comp-1328" and id="ext-gen1578" are always changing. My code so far to get to a specific page:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
my_login="SuperMegaUser"
my_pin="960790"+input()
driver=webdriver.Chrome()
driver.get(r'https://cyberark.supermegacorp.com/PasswordVault/logon.aspx?ReturnUrl=%2fPasswordVault%2fdefault.aspx')
assert "Password Vault Sign In" in driver.title
driver.find_element_by_xpath(r'//*[#id="pvBody_PageTemplate_innerHolder_ctrlLogon_txtUsername"]').send_keys(my_login)
driver.find_element_by_xpath(r'//*[#id="pvBody_PageTemplate_innerHolder_ctrlLogon_txtPassword"]').send_keys(my_pin)
driver.find_element_by_xpath(r'//*[#id="pvBody_PageTemplate_innerHolder_ctrlLogon_btnLogon"]').click()
time.sleep(10)
#driver.find_element_by_xpath(r'//*[#id="ext-gen159"]/div/table/tbody/tr/td[19]').click()
driver.find_element_by_xpath(r'//*[#id="ext-gen159"]/div/table/tbody/tr/td[17]/div').click()
time.sleep(5)
driver.find_element_by_xpath(r'//*[#id="reason"]').send_keys("0")
driver.find_element_by_xpath(r'//*[#id="reason"]').send_keys(Keys.RETURN)
time.sleep(5)
HTML:
<div class="password-windows-fieldset-body password-windows-fieldset-body-noheader" id="ext-gen1578"><label id="ext-comp-1327" class=" account-password-display password-labels">[wkSu-'$bS[U#;</label><table id="ext-comp-1328" cellspacing="0" class="x-btn password-window-copy-btn password-labels x-btn-noicon" style="width: auto;"><tbody class="x-btn-small x-btn-icon-small-left"><tr><td class="x-btn-tl"><i> </i></td><td class="x-btn-tc"></td><td class="x-btn-tr"><i> </i></td></tr><tr><td class="x-btn-ml"><i> </i></td><td class="x-btn-mc"><em class="" unselectable="on"><button type="button" id="ext-gen1593" class=" x-btn-text">Copy</button></em></td><td class="x-btn-mr"><i> </i></td></tr><tr><td class="x-btn-bl"><i> </i></td><td class="x-btn-bc"></td><td class="x-btn-br"><i> </i></td></tr></tbody></table></div>

To fetch text use text instead of click() only text no parenthesis
For ext-
driver.find_element_by_xpath(r'//*[contains(#id,"ext-gen")]/div/table/tbody/tr/td[17]/div').click()
OR
driver.find_element_by_xpath(r'//*[matches(#id,"^.*?ext-gen.*$")]/div/table/tbody/tr/td[17]/div').click()
OR
driver.find_element_by_xpath(r'//*[matches(#id,"ext-gen")]/div/table/tbody/tr/td[17]/div').click()
For ext-comp-
driver.find_element_by_xpath(r'//*[contains(#id,"ext-comp-")]/div/table/tbody/tr/td[17]/div').click()
OR
driver.find_element_by_xpath(r'//*[matches(#id,"^.*?ext-comp-.*$")]/div/table/tbody/tr/td[17]/div').click()
OR
driver.find_element_by_xpath(r'//*[matches(#id,"ext-comp-")]/div/table/tbody/tr/td[17]/div').click()
I found one more that is starts-with
driver.find_element_by_xpath(r'//*[starts-with(#id,"ext-gen")]/div/table/tbody/tr/td[17]/div').click()
driver.find_element_by_xpath(r'//*[starts-with(#id,"ext-comp-")]/div/table/tbody/tr/td[17]/div').click()

Given the HTML you provided, the line below should get you what you want.
driver.find_element_by_css_selector("label.account-password-display").text

how to find element selenium

How do I find
element = driver.find_element_by_id("id","class","class")
Im trying to click an ad
doing direct with xpath will not work:
/html/body/div/div[1]/div[1]/div/a/img
Traceback (most recent call last):
File "a.py", line 14, in <module>
element = driver.find_element_by_id("/html/body/div/div[1]/div[1]/div/a/img")
HTML shown as follows:
</head>
<body scroll="no">
<div id="widget" class="widget">
<div class="plug">
<div class="thumbBorder">
<div class="thumb">
<div class="ton" style="display: block;">
<div class="title_bg"> </div>
<a class="title q" target="_blank" href="//prwidgets.com/t/ghxa/g0us/7433c239e19107a4301ad9959d2d37440/aHR0cDovL3Ry‌​aXBsZXh2aWQuY29tLw==">Kiss N Tell</a>
</div>
<a class="q" target="_blank" href="//prwidgets.com/t/ghxa/g0us/7433c239e19107a4301ad9959d2d37440/aHR0cDovL3Ry‌​aXBsZXh2aWQuY29tLw=="> <img title="Title" src="//prstatics.com/prplugs/0/747604/160x120.jpg"

find_element_by_id in Selenium python binding accepts one parameter which is the value of the id attribute. Such as
login_form = driver.find_element_by_id('loginForm')
Please refer to the doc here
Addition to that you can use
driver.find_element(By.ID, 'your ID')

In this case you can try xpath- and axis i.e. following-sibling
element = driver.find_element_by_xpath("//a[class='q']/following-sibling::img[1]")
element.click()
N.B I have assumed there is no a with class name q in the whole html doument.

This may not work for you but when there is not an easy ID or NAME to grab, I go into the browser (I will refer to Firefox) right click on the element, select 'Inspect Element', then right click on the highlighted area in the inspection window and select 'Copy Unique Selector'. Then you can paste this into your code and use:
selector = 'pasted string here'
element = driver.find_element_by_css_selector(selector)
element.click()
EDIT: using the selector provided by #James below:
selector = 'div.plug:nth-child(1) > div:nth-child(1) > div:nth-child(1) > a:nth-child(2) > img:nth-child(1)'
element = driver.find_element_by_css_selector(selector)
element.click()
This usually works quite well for me.
EDIT: Add a real example. Try this and see if it works.
# open google search page and click the "About" link
from selenium import webdriver
driver = webdriver.Firefox()
driver.maximize_window()
driver.get('www.google.com/ncr')
# got the selector below using Firefox 'Inspect Element -> Copy Unique Selector'
about_selector = 'a._Gs:nth-child(3)'
about = driver.find_element_by_css_selector(about_selector)
about.click()