Develop Reference

Cannot input date to field with shadow-root (user-agent) with Selenium Python

I am trying to write a stable code to input a date into calendar input field.
Native Selenium inputs work only locally, but when I run this code in CICD I see that numbers are sometime inputted into wrong places.
For example, the timestamp is 2022-07-22 00:56:42.
Expected input by user is 220720220056.
By experimenting I found out that year works if there are two zeros in front of the year. Therefore, what I actually send is 22070020220056
input_field = driver.find_element(By.XPATH, "//div[#role='dialog']//input[#type='datetime-local']")
input_field.click()
input_field.clear()
input_field.send_keys(22070020220056)
<input _ngcontent-pau-c188="" clrinput="" type="datetime-local" name="expiry" aria-describedby="clr-form-control-9-helper" id="clr-form-control-9" class="clr-input ng-pristine ng-valid ng-touched">
#shadow-root (user-agent) == $0
<div pseudo="-webkit-datetime-edit" id="date-time-edit" datetimeformat="dd/MM/y, HH:mm">
<div pseudo="-webkit-datetime-edit-fields-wrapper">
<span role="spinbutton" aria-placeholder="dd" aria-valuemin="1" aria-valuemax="31" aria-label="Day" pseudo="-webkit-datetime-edit-day-field" aria-valuenow="25" aria-valuetext="25">25</span>
<div pseudo="-webkit-datetime-edit-text">/</div>
<span role="spinbutton" aria-placeholder="mm" aria-valuemin="1" aria-valuemax="12" aria-label="Month" pseudo="-webkit-datetime-edit-month-field" aria-valuenow="7" aria-valuetext="07">07</span>
<div pseudo="-webkit-datetime-edit-text">/</div>
<span role="spinbutton" aria-placeholder="yyyy" aria-valuemin="1" aria-valuemax="275760" aria-label="Year" pseudo="-webkit-datetime-edit-year-field" aria-valuenow="2022" aria-valuetext="2022">2022</span>
<div pseudo="-webkit-datetime-edit-text">, </div>
<span role="spinbutton" aria-placeholder="--" aria-valuemin="0" aria-valuemax="23" aria-label="Hours" pseudo="-webkit-datetime-edit-hour-field" aria-valuenow="0" aria-valuetext="00">00</span>
<div pseudo="-webkit-datetime-edit-text">:</div>
<span role="spinbutton" aria-placeholder="--" aria-valuemin="0" aria-valuemax="59" aria-label="Minutes" pseudo="-webkit-datetime-edit-minute-field" aria-valuenow="1" aria-valuetext="01">01</span>
</div>
</div>
</input>
Field looks like this:
I tried few other options to make the test more stable.
Option 1.
Tried to set the value directly.
driver.execute_script(f"arguments[0].setAttribute('value', '{my_timestamp}')", input_field)
Option 2
Tried to access shadow root:
date_field = driver.execute_script("return document.querySelector('input[type=datetime-local]').shadowRoot.getElementById('#date-time-edit')")
date_field.send_keys(my_timestamp)
Option 3
Tried to get parent element with execute_script instead of Selenium and send value:
date_field = driver.execute_script("return document.querySelector('input[type=datetime-local]')")
date_field.send_keys(my_timestamp)
Option 4
Tried to send text with a small delay before each letter:
for letter in timestamp:
time.sleep(0.1)
print(f"Sending letter {letter}")
date_field.send_keys(letter)
None of these methods worked on CICD, Only initial one and Option 3 always work locally.
I think the only stable solution would be the use of Javascipt code, but still not sure.
Any suggestion would be greatly appreciated.

Ah yes, the shadow root. For me, what works is something like the following:
shadowRootElement = driver.execute_script('''return document.querySelector("selector for shadowRoot's parent").shadowRoot''')
shadowRootChild = driver.execute_script('''return arguments[0].querySelector("selector for shadowRoot's parent").shadowRoot''', shadowRootElement)
shadowRootChild.find_element(By.CSS_SELECTOR, 'input[type=datetime-local]').send_keys("This is a test");

The <input> looks perfectly out of #shadow-root (user-agent) ambit.
<input _ngcontent-pau-c188="" clrinput="" type="datetime-local" name="expiry" aria-describedby="clr-form-control-9-helper" id="clr-form-control-9" class="clr-input ng-pristine ng-valid ng-touched">
#shadow-root (user-agent) == $0
<div pseudo="-webkit-datetime-edit" id="date-time-edit" datetimeformat="dd/MM/y, HH:mm">
<div pseudo="-webkit-datetime-edit-fields-wrapper">
.
.
.
</div>
</div>
</input>
However, the desired element is a Angular element, so ideally to send a character sequence to the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategy:
Using XPATH:
input_field = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#class='clr-input ng-pristine ng-valid ng-touched' and #name='expiry'][#type='datetime-local']")))
driver.execute_script(f"arguments[0].setAttribute('value', '{my_timestamp}')", input_field)
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Selenium cannot find element by name or id (Python)

I am trying to automate logging into a website using selenium, but I am getting "no such element message" error. Here is my code, with the link to the website included:
from selenium import webdriver
import time
import datetime
driver = webdriver.Chrome("C:\\Users\\Family\\Downloads\\chromedriver_win32\\chromedriver.exe")
driver.get("https://login.microsoftonline.com/c4d72b4d-8155-4a90-9155-7705148c41ca/saml2?SAMLRequest=jdE9a8MwEAbgvdD%2fYLRbkh3ZVoQdCO0SSJek7dClnJVzYrClVCeX%2fvw6DaEdu90HLzzc1espntwOPyakmGweG0YwDuHav3eqqFSHOZRQKZAZdJDpDjToKisXiCx5xUC9dw3LuWTJhmjCjaMILs4jmWepVKnMnrPCyNIscq601pVUbyxZE2GIc%2fbBO5pGDHsMn73Fl922YacYz2SEiAdqOQ4IwfXu6F2E0HtuQRzyQQxnAbNeDP7YO3Fxby8Vn3cs%2bRoHRw2bgjMeqCfjYEQy0Zr9%2bmlrZq45Bx%2b99QNb3d8lSf2DD%2f8Jwo3OVjdokdlSVrZKUS10quTSphrLIi00drrVebksWh7RzYch3ob%2beIp0Bovc%2bvGXXosrYgbV4u9nVt8%3d&RelayState=%2fd2l%2fhome&sso_reload=true")
login_button = driver.find_element_by_id("i0116")
login_button.send_keys("sajjad.jessa#student.tdsb.on.ca")
And here is the element I am trying to access with my code:
<input type="email" name="loginfmt" id="i0116" maxlength="113" lang="en" class="form-control ltr_override input ext-input text-box ext-text-box" aria-required="true" data-bind="
externalCss: {
'input': true,
'text-box': true,
'has-error': usernameTextbox.error },
ariaLabel: tenantBranding.UserIdLabel || str['CT_PWD_STR_Username_AriaLabel'],
ariaDescribedBy: 'loginHeader' + (pageDescription && !svr.fHideLoginDesc ? ' loginDescription' : ''),
textInput: usernameTextbox.value,
hasFocusEx: usernameTextbox.focused,
placeholder: $placeholderText" aria-label="Enter your TDSB email address here, then click Next" aria-describedby="loginHeader" placeholder="Enter your TDSB email address here, then click Next">
From other answers I understand that you have to use driver.find_element_by_css_selector() and driver.switch_to.frame(), but if you look at the full hypertext of the website, the first frame to go into is a "div" tag without any attributes. It is however the only "div" tag alongside two "script" tags. I need the correct code to go into the frame, or another method to automate logging in.

It seems synchronization Issue.
Induce WebDriverWait() and wait for element_to_be_clickable()
login_button =WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.ID,"i0116")))
login_button.send_keys("sajjad.jessa#student.tdsb.on.ca")
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
login_button =WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.NAME,"loginfmt")))

The most of the large websites now uses dynamic layout. You should look for the parents with static css selector, and then do something like that:
.parent_css_selector > div:nth-child(1) > div:nth-child(3) > img > input
Token '>' is supposed to look only for direct childs, token ':nth-child' asserts to the child number in parent.
Example:
<div class="parent_css_selector">
<div id="random_id_12312313">
<div id="unneccesary_123123"></div>
<div id="random_id_341234">
<form id="random_id_345545">
<span></span>
<span></span>
<span></span>
<span></span>
<span>
<input id="neccesary_13843942"></input>
</span>
</form>
</div>
</div>
</div>
You can use this selector to access the input:
.parent_css_selector > div > div:nth-child(2) > form > span:nth-child(5) > input

clicking through all rows in a table from angular using selenium python web driver

I'm trying to iterate through a certain column of rows on a table/grid of an HTML page with I assume is a dynamic angular element.
I have tried to iterate through the rows by creating a list of common xpaths between each row. This only help me achieve 32 rows and not the full amount which is 332. I also tried waiting to see if the webpage would load and then have the full amount of web-elements. Then I tried to run a loop on searching for similar xpaths by scrolling down to the last element in the list. None of these ways helped me to iterate through the rows. Also I will not be able to share the website since the website is private.
python
webelement = []
driver.implicitly_wait(20)
ranSleep()
for webelement in driver.find_elements_by_xpath('//a[#class="ng-pristine ng-untouched ng-valid ng-binding ng-scope ng-not-empty"]'):
driver.implicitly_wait(20)
html for the rows
<a ng-model="row.entity.siteCode"
ng-click="grid.appScope.openSite(row.entity)"
style="cursor:pointer"
class="ng-pristine ng-untouched ng-valid ng-binding ng-scope ng-not-empty">
Albuquerque
<span title="Open defect(s) on site"
ng-show="row.entity.openDeficiencies"
style="background-color:yellow; color:#000;"
class="ng-hide">
!
</span>
</a>
I expect to be able to click all the links in each row once this is solved
Here is the snippet of the html code
<div id="table1" class="container-fluid">
<div ui-i18n="en"
class="grid advanceSearch ui-grid ng-isolate-scope grid1554731599680"
id="grid1" ui-grid="gridOptions"
ui-grid-expandable="" ui-grid-rowedit=""
ui-grid-resize-columns="" ui-grid-selection=""
ui-grid-edit="" ui-grid-move-columns="">
<!-- TODO (c0bra): add "scoped" attr here, eventually? -->
<style ui-grid-style="" class="ng-binding">
.grid1554731599680 {
/* Styles for the grid */
}
here is how the page looks with the table format
Here is the rows that I want to click through all of them

You might still be able to increment through each link by appending to the class name, as they seem to be a little unique in nature and using the last number as a char from the alphabet. Perhaps something like below could work :) Expanding on the classname's last character, in-case there's an increase, should solve the problem of there being more than 26.
Steps taken: increment classnames >append successes to list >move to link within list >click link >List item
import string
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
alpha = string.ascii_uppercase
successfulIncs = []
for char in alpha:
className = 'ng-pristine.ng-scope.ui-grid-coluiGrid-000' + char
try:
driver.find_elements_by_class_name(className)
successfullIncs.append(className)
except NoSuchElementException:
print("Element not found")
### First move to our element
for line in successfulIncs:
link = WebDriverWait(driver, 3).until(EC.visibility_of_element_located
(By.CLASS_NAME, line))
ActionChains(driver).move_to_element(link).perform()
#Click
ActionChains(driver).move_to_element(link).click(link).perform()

Can't trigger a click on a certain link using selenium

I've written a script in python with selenium to click on a certain link in a webpage to download an excel file. However, when I execute my script, it throws timeout exception. How can I make it work? Any help will be greatly appreciated.
Link to the site: webpage
Script I've tried with:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get('replace_with_above_link')
item = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".hasmore #dlink")))
item.click()
driver.quit()
Html elements which contain the dropdown options:
<li class="hasmore drophover"><span>Share & more</span><div><ul><li><button class="tooltip" tip="Use a customizable report creator that can<br>output HTML, CSV, or a shareable link." id="share_on_ajax_result_table">Modify & Share Table</button></li><li><button class="tooltip" tip="Get a bit of widget code to emed this table on your site">Embed this Table</button></li><li><button class="tooltip" tip="Convert the table below to comma-separated values<br>suitable for use with excel">Get as Excel Workbook (experimental)</button><a id="dlink" style="display: none;"></a></li><li><button class="tooltip" tip="Export table as <br>suitable for use with excel">Get table as CSV (for Excel)</button></li><li><button class="tooltip" tip="">Strip Mobile Formatting</button></li><li><a id="a_ajax_result_table" name="ajax_result_table" href="#ajax_result_table::none">Copy Link to Table to Clipboard</a></li><li><button class="tooltip" tip="">About Sharing Tools</button></li><li><button class="tooltip" tip="">Video: SR Sharing Tools & How-to</button></li><li><button class="tooltip" tip="">Video: Stats Table Tips & Tricks</button></li></ul></div></li>
Location of that file in that webpage (the desired link is marked with pencil):

Target link is hidden and so wait for its visibility will always fail. You should try to handle button node instead:
item = wait.until(EC.visibility_of_element_located((By.XPATH, "//li[span='Share & more']")))
item.click()
wait.until(lambda driver: "drophover" in item.get_attribute("class"))
item.find_element_by_xpath("//button[.='Get as Excel Workbook (experimental)']").click()

As you are trying to click on the link with text as Get as Excel Workbook (experimental) and as per your comment you are already able to click on the Share&more link in the first place and found it working next your intended <a> tagged element contains the attribute style set to display: none;. So to invoke click() to download you can use the following code block :
Get_as_Excel_Workbook_link = driver.find_element_by_xpath("//li[#class='hasmore drophover']//ul//li//a[#id='dlink']")
driver.execute_script("arguments[0].removeAttribute('style')", Get_as_Excel_Workbook_link)
Get_as_Excel_Workbook_link.click()
Update A
As per your comment :
I am not sure if the xpath which you have used is a valid one or not :
"//li[a[#id='dlink']]/a"
You tried using :
Get_link = driver.find_element_by_xpath("//li[a[#id='dlink']]/a")
print(Get_link.get_attribute("outerHTML"))
But why? Is there any necessity?
As per my research and analysis you can be assured that you are at the right place. See the formatted version of the HTML you have shared and the resolution of the xpath I have provided.
<li class="hasmore drophover"><span>Share & more</span>
<div>
<ul>
<li><button class="tooltip" tip="Use a customizable report creator that can<br>output HTML, CSV, or a shareable link." id="share_on_ajax_result_table">Modify & Share Table</button></li>
<li><button class="tooltip" tip="Get a bit of widget code to emed this table on your site">Embed this Table</button></li>
<li><button class="tooltip" tip="Convert the table below to comma-separated values<br>suitable for use with excel">Get as Excel Workbook (experimental)</button>
<a id="dlink" style="display: none;"></a>
</li>
<li><button class="tooltip" tip="Export table as <br>suitable for use with excel">Get table as CSV (for Excel)</button></li>
<li><button class="tooltip" tip="">Strip Mobile Formatting</button></li>
<li><a id="a_ajax_result_table" name="ajax_result_table" href="#ajax_result_table::none">Copy Link to Table to Clipboard</a></li>
<li><button class="tooltip" tip="">About Sharing Tools</button></li>
<li><button class="tooltip" tip="">Video: SR Sharing Tools & How-to</button></li>
<li><button class="tooltip" tip="">Video: Stats Table Tips & Tricks</button></li>
</ul>
</div>
</li>
So the result you have seen is pretty correct. Now, for you understanding I have inserted some text as MyLink within the intended tag :
<a id="dlink" style="display: none;"></a>
Converted as :
<a id="dlink" style="display: none;">MyLink</a>
See the result :
Check out my solution once again I can ensure that works.
Update B
unable to locate element is good message to debug perhaps apart from "display: none;" you have pulled a rug over the actual issue by mentioning clicked on the share&more link in the first place and found it working. Troubles come up when i try to initiate a click on the link.
If you observe the HTML the element is within class="tooltip" so you need to induce a waiter as follows :
//perform click on the link Share&more
Get_as_Excel_Workbook_link = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//li[#class='hasmore drophover']//ul//li//a[#id='dlink']")))
driver.execute_script("arguments[0].removeAttribute('style')", Get_as_Excel_Workbook_link)
Get_as_Excel_Workbook_link.click()

how to find element selenium

How do I find
element = driver.find_element_by_id("id","class","class")
Im trying to click an ad
doing direct with xpath will not work:
/html/body/div/div[1]/div[1]/div/a/img
Traceback (most recent call last):
File "a.py", line 14, in <module>
element = driver.find_element_by_id("/html/body/div/div[1]/div[1]/div/a/img")
HTML shown as follows:
</head>
<body scroll="no">
<div id="widget" class="widget">
<div class="plug">
<div class="thumbBorder">
<div class="thumb">
<div class="ton" style="display: block;">
<div class="title_bg"> </div>
<a class="title q" target="_blank" href="//prwidgets.com/t/ghxa/g0us/7433c239e19107a4301ad9959d2d37440/aHR0cDovL3Ry‌​aXBsZXh2aWQuY29tLw==">Kiss N Tell</a>
</div>
<a class="q" target="_blank" href="//prwidgets.com/t/ghxa/g0us/7433c239e19107a4301ad9959d2d37440/aHR0cDovL3Ry‌​aXBsZXh2aWQuY29tLw=="> <img title="Title" src="//prstatics.com/prplugs/0/747604/160x120.jpg"

find_element_by_id in Selenium python binding accepts one parameter which is the value of the id attribute. Such as
login_form = driver.find_element_by_id('loginForm')
Please refer to the doc here
Addition to that you can use
driver.find_element(By.ID, 'your ID')

In this case you can try xpath- and axis i.e. following-sibling
element = driver.find_element_by_xpath("//a[class='q']/following-sibling::img[1]")
element.click()
N.B I have assumed there is no a with class name q in the whole html doument.

This may not work for you but when there is not an easy ID or NAME to grab, I go into the browser (I will refer to Firefox) right click on the element, select 'Inspect Element', then right click on the highlighted area in the inspection window and select 'Copy Unique Selector'. Then you can paste this into your code and use:
selector = 'pasted string here'
element = driver.find_element_by_css_selector(selector)
element.click()
EDIT: using the selector provided by #James below:
selector = 'div.plug:nth-child(1) > div:nth-child(1) > div:nth-child(1) > a:nth-child(2) > img:nth-child(1)'
element = driver.find_element_by_css_selector(selector)
element.click()
This usually works quite well for me.
EDIT: Add a real example. Try this and see if it works.
# open google search page and click the "About" link
from selenium import webdriver
driver = webdriver.Firefox()
driver.maximize_window()
driver.get('www.google.com/ncr')
# got the selector below using Firefox 'Inspect Element -> Copy Unique Selector'
about_selector = 'a._Gs:nth-child(3)'
about = driver.find_element_by_css_selector(about_selector)
about.click()