Develop Reference

Trying to locate an element in a webpage but getting NoSuchElementException

I am trying to get the webdriver to click a button on the site random.org The button is a generator that generates a random integer between 1 and 100. It looks like this:
After inspecting the webpage, I found the corresponding element on the webpage looks something like this:
It is inside an iframe and someone suggested that I should first switch over to that iframe to locate the element, so I incorporated that in my code but I am constantly getting NoSuchElementException error. I have attached my code and the error measage below for your reference. I can't understand why it cannot locate the button element despite referencing the ID, which is supposed to unique in the entire document.
The code:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Edge()
driver.get("https://www.random.org/")
driver.implicitly_wait(15)
driver.switch_to.frame(driver.find_element(By.TAG_NAME, "iframe"))
button = driver.find_element(By.CSS_SELECTOR, "input[id='hnbzsqjufzxezy-button']")
button.click()
The error message:

Make sure that there are no more Iframes on the page. If there are a few an not only one do this:
iframes = driver.find_elements(By.CSS, 'iframe')
// try switching to each iframe:
driver.switch_to.frame(iframes[0])
driver.switch_to.frame(iframes[1])
You can't find the button because its name contain random letters. Every time you will refresh the page you can see that the name value will change. So, do this:
button = driver.findElement(By.CSS, 'input[type="button"][value="Generate"]')
button.click()

There are several issues with your code:
First you need to close the cookies banner
The locator of the button is wrong. It's id is dynamic.
You need to use WebDriverWait to wait for elements clickability.
The following code works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = 'https://www.random.org/'
driver.get(url)
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[onclick*='all']"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME, "iframe")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[id*='button']"))).click()

Click button using Selenium Python

I am new to python selenium. I want to click the Iniciar session(Login button) on this web site. I have tried using the XPath, CSS selector, CSS Path, and class name.
I use the below code to click the button:
from selenium import webdriver
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome("C:/Users/pralo/Downloads/DE/chromedriver", chrome_options=options)
driver.get('https://www.panamacompra.gob.pa/Inicio/#/')
sleep(10)
driver.find_element(By.XPATH,'//button[text()=" Iniciar sesión"]').click()
I get the below error for the above code execution:
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (389, 1470)
Can anyone help me understand the reason for this error?

You can utilize the driver.execute_script() method:
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
driver = webdriver.Chrome()
driver.get('https://www.panamacompra.gob.pa/Inicio/#/')
sleep(10)
element = driver.find_element(By.CLASS_NAME, 'btn-blue-dark')
driver.execute_script("arguments[0].click();", element)
The driver.execute_script() method allows you to run JavaScript code in your Python program!

Button element not interactable Selenium

I'm trying to write a script to automate some tasks with Selenium and Python, and every time I try to click on a button
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
driver = webdriver.Chrome(options=options)
xpath = "/html/body/app-root/app-prime/div/mat-sidenav-container//app-detail-component/main//div/span/button[#aria-label='Prenota']"
# Wait for the element to be visible, always true
WebDriverWait(driver, 5).until(expected_conditions.presence_of_element_located((By.XPATH, xpath)))
# Try to click on element, get an error
driver.find_element(By.XPATH, xpath).click()
I get the following error
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
I know for sure that the element gets located correctly and has to match, but it should act only on the first one.
I tried:
Trying to click child tags, such as others divs and span
Waiting for it to be clickable
Waiting with an implicit wait
None of those activities were successful
Edit: Apparently the issue olly exist on my machine

It could be not clickable for a number of reasons. You might want to check if there is some element on the page layered on top so that that element is not interactable at that time, e.g some popup/iframe etc. There could be some other element that will receive click at that time.
You could try an actions click - something like this
myElement = driver.find_element_by_xpath("myXpath")
webdriver.ActionChains(driver).move_to_element(myElement).click(myElement).perform()

One of this should work.
IJavaScriptExecutor executor = (IJavaScriptExecutor)WebDriver.Driver;
executor.ExecuteScript("arguments[0].click();", webElement);
Actions actions = new Actions(WebDriver.Driver);
actions.MoveToElement(webElement).Click().Perform();
Note - these are c sharp code. try to do the same in java.

Unable to select element from Dropdown that hides - Python Selenium

I'm trying to get to the results page of a search but I have to first click on the dropdown option to complete the search. When I do this manually, the dropdown hides if I do not click on it right as it appears, when I code it, I get a the following error:
ElementNotInteractableException: Message: Element <div id="_esgratingsprofile_autocomplete-results-container" class="autocomplete-results-container msci-ac-search-data-dropdown"> could not be scrolled into view
This is my code so far, you can visit the url and see how it is yourself as well:
from selenium.webdriver import Firefox
from selenium.webdriver.support.ui import Select
from selenium.webdriver.firefox.options import Options
opts = Options()
opts.set_headless()
assert opts.headless
browser = Firefox(options=opts)
browser.get('https://www.msci.com/esg-ratings')
search_form = browser.find_element_by_id('_esgratingsprofile_keywords')
search_form.send_keys('MSFT')
browser.find_element_by_xpath("//div[#id='_esgratingsprofile_autocomplete-results-container']/ul[#id='ui-id-1']/li[#class='msci-ac-search-section-title ui-menu-item']").click()
I looked through many other answers but they didnt seem to deal with the case where the dropdown was not a directly clickable element or where it hides if you dont click on it right away. Any help is appreciated.

Try the below code, This code is working for me. Let me know if it shows any error.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.maximize_window()
wait = WebDriverWait(driver, 5)
action = ActionChains(driver)
driver.get("https://www.msci.com/esg-ratings")
Drop_Down = driver.find_element_by_xpath('//*[#id="_esgratingsprofile_keywords"]')
Drop_Down.send_keys("MSFT")
# Select the First Result from the search.
Result = wait.until(
EC.presence_of_element_located((By.XPATH, "//div[contains(#class,'autocomplete-results-container')]/ul/li[1]")))
action.move_to_element(Result).click().perform()

Selenium crashes when I'm trying to parse the next page (and seven after it) on a website. Any way to tackle this?

I want to parse an IMDb film rating located here on around 8 pages. In order to do that I'm using Selenium, and I'm having trouble with clicks, proceeding algorithm to next page. In the end I need 1000 titles when I'll continue using BeautifulSoup. Code below isn't working, I need to use button 'NEXT' with this HTML:
<a class="flat-button lister-page-next next-page" href="/list/ls000004717/?page=2">
Next
</a>
This is the code:
from selenium import webdriver as wb
browser = wb.Chrome()
browser.get('https://www.imdb.com/list/ls000004717/')
field = browser.find_element_by_name("flat-button lister-page-next next-page").click()
Error is the following:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".flat-button lister-page-next next-page"}
(Session info: chrome=78.0.3904.108)
I suppose I lack knowledge of syntax needed, or maybe I mixed it up a little. I tried searching on SO, though every example is pretty unique and I don't possess the knowledge to extrapolate these cases fully. Any way Selenium can handle that?

You could try using an XPath to query on the Next text inside the button. You should also probably invoke WebDriverWait since you are navigating across multiple pages, then scroll into view since this is at the bottom of the page:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from time import sleep
browser = wb.Chrome()
browser.get('https://www.imdb.com/list/ls000004717/')
# keep clicking next until we reach the end
for i in range(0,9):
# wait up to 10s before locating next button
try:
next_button = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[contains(#class, 'page') and contains(text(), 'Next')]")))
# scroll down to button using Javascript
browser.execute_script("arguments[0].scrollIntoView(true);", next_button)
# click the button
# next_button.click() this throws exception -- replace with JS click
browser.execute_script("arguments[0].click();", next_button)
# I never recommend using sleep like this, but WebDriverWait is not waiting on next button to fully load, so it goes stale.
sleep(5)
# case: next button no longer exists, we have reached the end
except TimeoutException:
break
I also wrapped everything in a try / except TimeoutException block to handle the case where we have reached the end of pages, and Next button no longer exists, thus breaking out of the loop. This worked on multiple pages for me.
I also had to add an explicit sleep(5) because even after invoking WebDriverWait on element_to_be_clickable, next_button was still throwing StaleElementReferenceException. It seems like WebDriverWait was finishing before page was fully loaded, causing the status of next_button to change after it had been located. Normally adding sleep(5) is bad practice, but there did not seem to be another workaround here. If anyone else has a suggestion on this, feel free to comment / edit the answer.

There are a couple of ways that could work:
1. Use a selector for the next button and loop until the end:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
browser = webdriver.Chrome()
browser.get('https://www.imdb.com/list/ls000004717/')
selector = 'a[class*="next-page"]'
num_pages = 10
for page in range(pages):
# Wait for the element to load
WebDriverWait(browser, 10).until(ec.presence_of_element_located((By.CSS_SELECTOR, selector)))
# ... Do rating parsing here
browser.find_element_by_css_selector(selector).click()
Instead of clicking on the element, the other option could be to navigate to the next page using broswer.get('...'):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
# Set up browser as before and navigate to the page
browser = webdriver.Chrome()
browser.get('https://www.imdb.com/list/ls000004717/')
selector = 'a[class*="next-page"]'
base_url = 'https://www.imdb.com/list/ls000004717/'
page_extension = '?page='
# Already at page = 1, so only needs to loop 9 times
for page in range(2, pages + 1):
# Wait for the page to load
WebDriverWait(browser, 10).until(ec.presence_of_element_located((By.CSS_SELECTOR, selector)))
# ... Do rating parsing here
next_page = base_url + page_extension + str(page)
browser.get(next_page)
As a note: field = browser.find_element_by_name("...").click() will not assign field to a webelement, as the click() method has no return value.

You could try a partial css selector.
browser.find_element_by_css_selector("a[class*='next-page']").click()

To click on the element with text as NEXT till the 901 - 1,000 of 1,000 page you have to:
scrollIntoView() the element once the visibility_of_element_located() is achieved.
Induce WebDriverWait for the element_to_be_clickable()
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.imdb.com/list/ls000004717/')
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.pagination-range"))))
while True:
try:
WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.CSS_SELECTOR, "div.row.text-center.lister-working.hidden")))
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.pagination-range"))))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.flat-button.lister-page-next.next-page"))).click()
print("Clicked on NEXT button")
except TimeoutException as e:
print("No more NEXT button")
break
driver.quit()
Console Output:
Clicked on NEXT button
Clicked on NEXT button
Clicked on NEXT button
Clicked on NEXT button
Clicked on NEXT button
Clicked on NEXT button
Clicked on NEXT button
Clicked on NEXT button
Clicked on NEXT button
No more NEXT button