Develop Reference

Selenium Python is unable to locate an element

I'm new to webdrivers and am experimenting with them.
I'm trying to click a button when opening a webpage and it keeps giving the error of unable to locate element.
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("html page")
button = driver.find_element(By.ID, "onetrust-accept-btn-handler")
button.click()
i have tried id and xpath but i dont know what else to use.
the path for the button is:
/html/body/div[3]/div[3]/div/div/div[2]/div/div/button
<button id="onetrust-accept-btn-handler" tabindex="0">Run</button>

You can use implicit wait for wait until the page is fully loaded
driver.implicitly_wait(10)

2nd 'Element not interactable exception' when iterating on the loop, SELENIUM PYTHON

Selenium does see displayed element visible on the page on the second iteration.
I click on a link, and a box within a website appears. I need to close that box.
This action will be performed 1000+ times. On the first iteration, Selenium opens the link and closes the box. On the second iteration, Selenium opens the link, and cannot close the box. At this point, it gives error message:
Exception has occurred: ElementNotInteractableException Message: element not interactable (Session info: chrome=105.0.5195.102)
My code + HTML of relevant element below.
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r"D:\SeleniumDriver\chromedriver.exe")
driver.get('https://sprawozdaniaopp.niw.gov.pl/')
find_button = driver.find_element("id", "btnsearch")
find_button.click()
interesting_links = driver.find_elements(By.CLASS_NAME, "dialog")
for i in range(len(interesting_links)):
interesting_links[i].click()
time.sleep(10) # I tried 60 seconds, no change
#
# HERE I WOULD DO MY THINGS
#
close_box = driver.find_element(By.CLASS_NAME, "ui-dialog-titlebar-close")
print(close_box.is_displayed())
close_box.click() # Here is where the program crushes on the 2nd iteration
if i == 4: # Stop the program after 5 iterations
break
HTML code of the relevant element:
<span class="ui-icon ui-icon-closethick">close</span>
I tried to locate the element that closes the box by CSS SELECTOR AND XPATH.
The CSS SELECTOR of the X/close button is the same every time, but
only the first time Selenium will see the X button displayed.
THE XPATH is strange. On the first opening of the link, X/close button will have path:
/html/body/div[6]/div[1]/a
However, if you open the next link, path will look this:
/html/body/div[8]/div[1]/a
Let me know what you think of that :-)

This is one way to achieve your goal:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time as t
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,720")
webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(browser, 20)
url = 'https://sprawozdaniaopp.niw.gov.pl/'
browser.get(url)
wait.until(EC.element_to_be_clickable((By.ID, "btnsearch"))).click()
links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "a[class='dialog']")))
counter = 0
for link in links[:5]:
link.click()
print('clicked link', link.text)
### do your stuff ###
t.sleep(1)
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'span[class="ui-icon ui-icon-closethick"]')))[counter].click()
print('closed the popup')
counter = counter+1
This will print out in terminal:
clicked link STOWARZYSZENIE POMOCY DZIECIOM Z PORAŻENIEM MÓZGOWYM "JASNY CEL"
closed the popup
clicked link FUNDACJA NA RZECZ POMOCY DZIECIOM Z GRODZIEŃSZCZYZNY
closed the popup
clicked link FUNDACJA "ADAMA"
closed the popup
clicked link KUJAWSKO-POMORSKI ZWIĄZEK LEKKIEJ ATLETYKI
closed the popup
clicked link "RYBNICKI KLUB PIŁKARSKI - SZKÓŁKA PIŁKARSKA ROW W RYBNIKU"
closed the popup
Every time you click on a link, a new popup is created. When you close it, that popup will not disappear, but it will stay hidden. So when you click on a new link and then you want to close the new popup, you need to select the new (nth) close button. This should also apply to popup elements, so make sure you account for it. I stopped after the 5th link, of course you will need to remove the slicing to handle all links present in page.
Selenium setup above is chromedriver on linux - you just have to observe the imports, and the code after defining the browser(driver).
Selenium documentation can be found at https://www.selenium.dev/documentation/

Edge webdriver cannot find element visible to Firefox driver on SharePoint site

I am trying to access a dialog box on SharePoint using the Edge driver, however, my code doesn't find it. Using the Firefox driver, it works without problems:
[The reason I have to use Edge is because I am in a corporate environment, and Firefox is not supported by our single sign-on, proxies etc.]
from selenium import webdriver
import time
driver = webdriver.Edge()
#driver = webdriver.Firefox()
base_url = r"..../project/carb2_0/filedirectory/_layouts/15/groups.aspx"
driver.get(base_url)
time.sleep(1)
group = "AT01"
button= driver.find_element_by_xpath(r"//a[contains(text(),'File Directory - "+group+"')]")
button.click()
time.sleep(0.2)
button = driver.find_element_by_link_text("New")
button.click()
time.sleep(0.2)
driver.switch_to.frame(1)
time.sleep(0.2)
#t = driver.page_source
elem = driver.find_element_by_xpath(
r"//div[#id='ctl00_PlaceHolderMain_peoplePicker_TopSpan']/input[2]"
)
The last line throws an exception when using Edge,
NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#id='ctl00_PlaceHolderMain_peoplePicker_TopSpan']/input[2]"}
(Session info: MicrosoftEdge=85.0.564.41)
Firefox is perfectly fine. Digging deeper, reading the page source with driver.page_source yields two completely different results. For Edge it is
<html><head></head><body></body></html>
For Firefox I get a full page, including an iframe, which can be found by find_element_by_xpath.
Any help would be appreciated.

The solution to the above issue was 2-fold:
switch to the correct frame
find the exact xpath denoting the <input> field in the page
For the first part I used the following code:
for i, frame in enumerate(frames):
if re.search('ms-dlgFrame', frame.get_attribute('outerHTML')):
frame_id = i
break
driver.switch_to.frame(frame_id)
It turns out the the correct frame_id was 2, not 1. It is not clear to me why the firefox (gecko) driver would work anyways (and the selenium IDE would select 1).
The second part boiled down to searching through the source of the HTML page and find the exact input element. Interestingly enough, the selenium IDE had only located a <div> around that element, and the firefox webdriver was forgiving enough to type into the correct field. The edge driver would not do that and claim that element was not writable. Once I had found the correct <input> field, everything worked fine.

Selenium not finding an element

I am trying to retrieve an element that I would like to click on. Here's the opening of the website with Selenium in Python:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--dns-prefetch-disable')
driver = webdriver.Chrome("./chromedriver", options=chrome_options)
website = "https://www.agronet.gov.co/estadistica/Paginas/home.aspx?cod=4"
driver.get(website) # loads the page
Then, I look for the element I'm interested in:
driver.find_element_by_xpath('//*[#id="cmbDepartamentos"]')
which raises a NoSuchElementException error. When looking at the html source (driver.page_source), indeed "cmbDepartamentos" does not exist! and the text of the dropdown menu I am trying to locate which is "Departamentos:" does not exist either. How can I deal with this?

This should work:
iframe=driver.find_element_by_xpath('//div[#class="iframe"]//iframe')
driver.switch_to.frame(iframe)
driver.find_element_by_xpath('//*[#id="cmbDepartamentos"]').click()
Notes:
The reason for NoSuchElementException error is that the element is
inside an iframe. Unless you switch your driver to that iframe,
the identification will not work.
CTRL + F in the Dev Tools panel, then search for the xpath you
defined in your script is always a good way to rule out issues with
your xpath definition, as cause for NoSuchElementException error (and in your case, the xpath is correct)
You might want to consider adding a WebdriverWait for a complete load of the search area/iframe before attempting to find the "Departamentos" field

no such element: Unable to locate element using chromedriver and Selenium in production environment

I have a problem with selenium chromedriver which I cannot figure out what's causing it. Some weeks ago everything was working OK, and suddenly this error started to show up.
The problem is coming from the following function.
def login_(browser):
try:
browser.get("some_url")
# user credentials
user = browser.find_element_by_xpath('//*[#id="username"]')
user.send_keys(config('user'))
password = browser.find_element_by_xpath('//*[#id="password"]')
password.send_keys(config('pass'))
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
login.send_keys("\n")
time.sleep(1)
sidebar = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/a')
sidebar.send_keys("\n")
app_submit = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/ul/li[1]/a')
app_submit.send_keys("\n")
except TimeoutException or NoSuchElementException:
raise LoginException
This function works with no problem in the development environment (macOS 10.11), but throws the following error in the production environment:
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="sidebar"]/ul/li[1]/a"}
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee97XXX),platform=Linux 4.4.0-116-generic x86_64)
I already updated both Chrome and chromedriver (v67 & 2.40, respectively) in each environment. I also gave it more time.sleep(15). But the problem persists. My latest guess is that maybe the initialization of the webdriver is not working properly:
def initiate_webdriver():
option = webdriver.ChromeOptions()
option.binary_location = config('GOOGLE_CHROME_BIN')
option.add_argument('--disable-gpu')
option.add_argument('window-size=1600,900')
option.add_argument('--no-sandbox')
if not config('DEBUG', cast=bool):
display = Display(visible=0, size=(1600, 900))
display.start()
option.add_argument("--headless")
else:
option.add_argument("--incognito")
return webdriver.Chrome(executable_path=config('CHROMEDRIVER_PATH'), chrome_options=option)
Because, if the Display is not working, then there may not be the mentioned sidebar but some other button.
So my questions are: does anybody have had a similar issue? Is there a way to know what is the page showing at the time the driver is looking for such an element?

It's report that the element not found error after you supplying the login , so I think the login failed and the page redirected to somewhere. You can use screenshot option to take a screenshot of the page and then see which page the driver load.
driver.save_screenshot("path to save screen.jpeg")
Also you can save the raw html code and inspect the same page.
Webdriver Screenshot
Using Selenium in Python to save a webpage on Firefox

A couple of things as per the login_(browser) method:
As you have identified the Login button through:
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
I would suggest rather invoking send_keys("\n") take help of the onclick() event through login.click() to mock the clicking of Login button as follows:
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
login.click()
Next when you identify the sidebar induce WebDriverWait for the element to be clickable as follows:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="sidebar"]/ul/li[1]/a'))).click()
As you mentioned your code code block works perfect in macOS 10.11 environment but throws the following error in the production environment (Linux) it is highly possible that different browsers renders the HTML DOM differently in different OS architecture. So instead of absolute xpath you must use relative xpath as follows:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#attribute='value']"))).click()
A couple of things as per the initiate_webdriver() method:
As per Getting Started with Headless Chrome the argument --disable-gpu is applicable only for Windows but not a valid configuration for Linux OS. So need o remove:
option.add_argument('--disable-gpu')
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Whenever I encounter strange issues in Selenium like this, I prefer retrying to find the particular element which is causing intermittent troubles. One way is to wrap it around a try-except block:
try:
sidebar = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/a')
except NoSuchElementException:
time.sleep(10)
print("Unable to find element in first time, trying it again")
sidebar = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/a')
You could also put the try code in a loop with a suitable count variable to make the automation code work. (Check this). In my experience with JAVA, this idea has resolved multiple issues.

You need to wait until the element is visible or else you will get this error. Try something like this:
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support.expected_conditions import visibility_of_element_located
from selenium.webdriver.common.by import By
TIMEOUT = 5
...
xpath = '//*[#id="sidebar"]/ul/li[1]/a'
WebDriverWait(self.selenium, TIMEOUT).until(visibility_of_element_located((By.XPATH, xpath)))
browser.find_element_by_xpath(xpath)
...