Develop Reference

Selenium cant find element on Javascript page

I am still quite new to python and selenium, However have managed to get quite far with what I am doing. But I appear to now be stuck. The page in question is an internal business page. I have tried using ID, name and XPATH with very little success.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
import time
PATH = r"C:\Users\p819364\Downloads\chromedriver.exe"
options = webdriver.ChromeOptions()
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--ignore-certificate-errors')
driver = webdriver.Chrome(PATH, options=options)
driver.get("https://10.47.31.102/3/Login")
driver.implicitly_wait(15)
print (driver.title)
username = driver.find_element(By.NAME, value='j_username')
username.send_keys("username")
password = driver.find_element(By.NAME, value='j_password')
password.send_keys("password")
password.send_keys(Keys.RETURN)
driver.implicitly_wait(20)
Settings = driver.find_element(By.XPATH, value="//*[#id='evo_widget_TBFisheyeItem_4']")
Settings.click()
driver.implicitly_wait(20)
Filter = driver.find_element(By.XPATH, value="//*[#id='tableForm:authenticationPolicyTable:tableActions']")
driver.implicitly_wait(20)
Filter.click()
The problem I am having is with the filter, I think it may be because this is a page within a page. I am sorry I cannot share the page as its internal. But I need to be able to click the filter and click and options
I keep getting the following error
Message: no such element; Unable to locate element: {"method":"css selector","selector":["//*[#id='tableForm:authenticationPolicyTable:tableActions']"
The XPATH is as follows:
//*[#id="tableForm:authenticationPolicyTable:tableActions"]
This is the full XPATH (which I have tried):
/html/body/form[3]/table/thead/tr[1]/th/table/tbody/tr/td[2]/select
I appreciate any help given and I also am not sure if its caused this because the field I want to select is not in view until I scroll. However I cannot seem to scroll as the element is within another as pictured
Thanks
Edit:
The iFrame was stopping it i had to switch to it first
iframe = driver.find_element_by_xpath("//*[#id='consoleCanvas']")
driver.switch_to.frame(iframe)

I think if the page is within page then you need to check if that element is within iframe. If iframe is there then you should first switch to frame and then click on filter. If you can post html code then it will be easy to understand issue.

How to click on the shadowbox in selenium-python?

I am trying to create an automation program. I want to click on the "Accept Cookies" shadowbox on the given website.
Here's how I have tried to achieve this:
driver = webdriver.Chrome('chromedriver')
driver.get(r'https://www.studydrive.net/')
script = '''return document.querySelector('#usercentrics-root').shadowRoot.querySelector('button[aria-label="Accept All"]')'''
accept_all_btn = driver.execute_script(script)
accept_all_btn.click()
Here's the error that I get after following this approach:
AttributeError: 'NoneType' object has no attribute 'click'
I don't know, what I am doing wrong here. Any help is appreciated. Thank you in advance.

wait=WebDriverWait(driver, 60)
driver.get("https://www.studydrive.net/")
elem = wait.until(EC.presence_of_element_located((By.ID,"usercentrics-root")))
script = '''return document.querySelector('#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')'''
accept_all_btn = driver.execute_script(script)
accept_all_btn.click()
Simply wait for the element and then proceed to click the accept all button. No aria-label was specified so I used another attribute.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

The element Accept all button is within a #shadow-root (open)
Solution
Tto click() on the desired element you need to use shadowRoot.querySelector()
You can use the following Locator Strategy:
driver = webdriver.Chrome(service=s, options=options)
driver.get("https://www.studydrive.net/")
time.sleep(5)
accept_all = driver.execute_script('''return document.querySelector("#usercentrics-root").shadowRoot.querySelector("button[data-testid='uc-accept-all-button']")''')
accept_all.click()
PS: The cookie popup surfaces on the screen after significant amount of time, so you may have to induce some waits
References
You can find a couple of relevant detailed discussions in:
How to locate the First name field within shadow-root (open) within the website https://www.virustotal.com using Selenium and Python
How to get past a cookie agreement page using Python and Selenium?
Unable to locate the Sign In element within #shadow-root (open) using Selenium and Python

Element Not Clickable - even though it is there

Hoping you can help. I'm relatively new to Python and Selenium. I'm trying to pull together a simple script that will automate news searching on various websites. The primary focus was football and to go and get me the latest Manchester United news from a couple of places and save the list of link titles and URLs for me. I could then look through the links myself and choose anything I wanted to review.
In trying the the independent newspaper (https://www.independent.co.uk/) I seem to have come up against a problem with element not interactable when using the following approaches:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('chromedriver')
driver.get('https://www.independent.co.uk')
time.sleep(3)
#accept the cookies/privacy bit
OK = driver.find_element_by_id('qcCmpButtons')
OK.click()
#wait a few seconds, just in case
time.sleep(5)
search_toggle = driver.find_element_by_class_name('icon-search.dropdown-toggle')
search_toggle.click()
This throws the selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable error
I've also tried with XPATH
search_toggle = driver.find_element_by_xpath('//*[#id="quick-search-toggle"]')
and I also tried ID.
I did a lot of reading on here and then also tried using WebDriverWait and execute_script methods:
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="quick-search-toggle"]')))
driver.execute_script("arguments[0].click();", element)
This didn't seem to error but the search box never appeared, i.e. the appropriate click didn't happen.
Any help you could give would be fantastic.
Thanks,
Pete

Your locator is //*[#id="quick-search-toggle"], there are 2 on the page. The first is invisible and the second is visible. By default selenium refers to the first element, sadly the element you mean is the second one, so you need another unique locator. Try this:
search_toggle = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//div[#class="row secondary"]//a[#id="quick-search-toggle"]')))
search_toggle.click()

First you need to open search box, then send search keys:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import os
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
browser = webdriver.Chrome(executable_path=os.path.abspath(os.getcwd()) + "/chromedriver", options=chrome_options)
link = 'https://www.independent.co.uk'
browser.get(link)
# accept privacy
button = browser.find_element_by_xpath('//*[#id="qcCmpButtons"]/button').click()
# open search box
li = browser.find_element_by_xpath('//*[#id="masthead"]/div[3]/nav[2]/ul/li[1]')
search_tab = li.find_element_by_tag_name('a').click()
# send keys to search box
search = browser.find_element_by_xpath('//*[#id="gsc-i-id1"]')
search.send_keys("python")
search.send_keys(Keys.RETURN)

Can you try with below steps
search_toggle = driver.find_element_by_xpath('//*[#class="row secondary"]/nav[2]/ul/li[1]/a')
search_toggle.click()

Selenium not finding an element

I am trying to retrieve an element that I would like to click on. Here's the opening of the website with Selenium in Python:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--dns-prefetch-disable')
driver = webdriver.Chrome("./chromedriver", options=chrome_options)
website = "https://www.agronet.gov.co/estadistica/Paginas/home.aspx?cod=4"
driver.get(website) # loads the page
Then, I look for the element I'm interested in:
driver.find_element_by_xpath('//*[#id="cmbDepartamentos"]')
which raises a NoSuchElementException error. When looking at the html source (driver.page_source), indeed "cmbDepartamentos" does not exist! and the text of the dropdown menu I am trying to locate which is "Departamentos:" does not exist either. How can I deal with this?

This should work:
iframe=driver.find_element_by_xpath('//div[#class="iframe"]//iframe')
driver.switch_to.frame(iframe)
driver.find_element_by_xpath('//*[#id="cmbDepartamentos"]').click()
Notes:
The reason for NoSuchElementException error is that the element is
inside an iframe. Unless you switch your driver to that iframe,
the identification will not work.
CTRL + F in the Dev Tools panel, then search for the xpath you
defined in your script is always a good way to rule out issues with
your xpath definition, as cause for NoSuchElementException error (and in your case, the xpath is correct)
You might want to consider adding a WebdriverWait for a complete load of the search area/iframe before attempting to find the "Departamentos" field

Python Selenium find element XPath doesn't work

I try to find refresing elements (time minute) on the webpage. My code worked only for simple text earlier. Now I use Ctrl+Shift+I and point out my element and "Copy Xpath".
Also, I have Chrome extension "XPath helper" and tried to do that with it one. There is more longer XPath, than in my code below. And it doesn't work too.
Error: NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id....
And, I also tried to use find by class, by tag, by CSS selector.. It only worked by tag and no perfect, on different page.
And I don't even say about print it, sometimes find_element(By.XPATH,'//*[...).text work, sometimes not.
I don't understand, why it work on one page and not on second.. I want to work with find elements by XPath in flash later.
UPDATE Now I retrying code and it work! But still doesn't work on the next webpage.. why it is so changeable? XPath change, when page reload or what? What is the simplest way to get text(refresing) info from flash, opened in chrome browser?
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(r"C:\Users\vishniakov\Desktop\python bj\driver\chromedriver.exe",chrome_options=options)
driver.get("https://www.betfair.com/sport/football/event?eventId=28935432")
print(driver.title)
elem =driver.find_element(By.XPATH,'//*[#id="yui_3_5_0_1_1538670363144_2571"]').text
print(elem)

This will work with assumption you want data of that page not of any specific element:
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get("https://www.betfair.com/sport/football/event?eventId=28935730")
print(driver.title)
elem =driver.find_element(By.CSS_SELECTOR,'.scroller.context-event').text
print(elem)

Assuming you do want spcipic data, you can use the contains() Xpath method... You can read about this here
For your case:
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(r"C:\Users\vishniakov\Desktop\python bj\driver\chromedriver.exe",chrome_options=options)
driver.get("https://www.betfair.com/sport/football/event?eventId=28935432")
print(driver.title)
elements =driver.find_elements(By.XPATH,'//*[contains(#id, "yui_3_5_0_1_")]')
print([i.text for i in elements])
You can play around with the contains() if my example didn't work... You must find what part of the id changes and exclude that part of the ID.
Hope this helps you!