Selenium does see displayed element visible on the page on the second iteration.
I click on a link, and a box within a website appears. I need to close that box.
This action will be performed 1000+ times. On the first iteration, Selenium opens the link and closes the box. On the second iteration, Selenium opens the link, and cannot close the box. At this point, it gives error message:
Exception has occurred: ElementNotInteractableException Message: element not interactable (Session info: chrome=105.0.5195.102)
My code + HTML of relevant element below.
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r"D:\SeleniumDriver\chromedriver.exe")
driver.get('https://sprawozdaniaopp.niw.gov.pl/')
find_button = driver.find_element("id", "btnsearch")
find_button.click()
interesting_links = driver.find_elements(By.CLASS_NAME, "dialog")
for i in range(len(interesting_links)):
interesting_links[i].click()
time.sleep(10) # I tried 60 seconds, no change
#
# HERE I WOULD DO MY THINGS
#
close_box = driver.find_element(By.CLASS_NAME, "ui-dialog-titlebar-close")
print(close_box.is_displayed())
close_box.click() # Here is where the program crushes on the 2nd iteration
if i == 4: # Stop the program after 5 iterations
break
HTML code of the relevant element:
<span class="ui-icon ui-icon-closethick">close</span>
I tried to locate the element that closes the box by CSS SELECTOR AND XPATH.
The CSS SELECTOR of the X/close button is the same every time, but
only the first time Selenium will see the X button displayed.
THE XPATH is strange. On the first opening of the link, X/close button will have path:
/html/body/div[6]/div[1]/a
However, if you open the next link, path will look this:
/html/body/div[8]/div[1]/a
Let me know what you think of that :-)
This is one way to achieve your goal:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time as t
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,720")
webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(browser, 20)
url = 'https://sprawozdaniaopp.niw.gov.pl/'
browser.get(url)
wait.until(EC.element_to_be_clickable((By.ID, "btnsearch"))).click()
links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "a[class='dialog']")))
counter = 0
for link in links[:5]:
link.click()
print('clicked link', link.text)
### do your stuff ###
t.sleep(1)
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'span[class="ui-icon ui-icon-closethick"]')))[counter].click()
print('closed the popup')
counter = counter+1
This will print out in terminal:
clicked link STOWARZYSZENIE POMOCY DZIECIOM Z PORAŻENIEM MÓZGOWYM "JASNY CEL"
closed the popup
clicked link FUNDACJA NA RZECZ POMOCY DZIECIOM Z GRODZIEŃSZCZYZNY
closed the popup
clicked link FUNDACJA "ADAMA"
closed the popup
clicked link KUJAWSKO-POMORSKI ZWIĄZEK LEKKIEJ ATLETYKI
closed the popup
clicked link "RYBNICKI KLUB PIŁKARSKI - SZKÓŁKA PIŁKARSKA ROW W RYBNIKU"
closed the popup
Every time you click on a link, a new popup is created. When you close it, that popup will not disappear, but it will stay hidden. So when you click on a new link and then you want to close the new popup, you need to select the new (nth) close button. This should also apply to popup elements, so make sure you account for it. I stopped after the 5th link, of course you will need to remove the slicing to handle all links present in page.
Selenium setup above is chromedriver on linux - you just have to observe the imports, and the code after defining the browser(driver).
Selenium documentation can be found at https://www.selenium.dev/documentation/
Related
I am currently scraping website https://glasschain.org (to be precise let's focus on example - https://glasschain.org/btc/wallet/100478015). There is a tab named 'Addresses', when I want to go to the X-th page (in this example equals 5 000) from available over 12 000 (to parallel some operations and not do it iteratively). To do it manually, I need to click on ... button and write a number and click ENTER. I use Python and selenium library to do it automatically and I have managed with clicking the button but I can't send a value to a field which appears after my click (in source code of website after clicking to button there is no new input field where I can send that value so I have tried to send it to the only existing element - earlier clicked button). Can anyone help me with that ? :)
My code:
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(path, options=chrome_options)
driver.get('https://glasschain.org/btc/wallet/100478015')
time.sleep(5)
adr_button = driver.find_elements_by_id('tab-Addresses')[1]
driver.execute_script("arguments[0].click();", adr_button)
time.sleep(5)
dots_button = driver.find_elements_by_class_name('ellipse')[0]
driver.execute_script("arguments[0].click();", dots_button)
dots_button.send_keys(5000)
My error:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable (Session info: chrome=110.0.5481.77)
Version of selenium I am working with: 3.141.0
I don't have your specific version of selenium, but here's a solution that works with a more recent version, selenium 4.7.2:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from time import sleep
s = Service('bin/chromedriver.exe')
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(service=s, options=chrome_options)
driver.get('https://glasschain.org/btc/wallet/100478015')
delay = 3 # seconds
try:
adr_button = WebDriverWait(driver, delay).until(ec.presence_of_element_located((By.ID, 'tab-Addresses')))
except TimeoutException:
print("Loading timed out, exiting...")
exit()
# click the GDPR cookies accept button, if that hasn't happened yet
gdpr_button = driver.find_element(by='id', value='gdpr')
if gdpr_button:
driver.execute_script("gdpr('accept');")
driver.execute_script("arguments[0].click();", adr_button)
try:
dots_button = WebDriverWait(driver, delay).until(ec.presence_of_element_located((By.CLASS_NAME, 'ellipse')))
except TimeoutException:
print("Loading timed out, exiting...")
exit()
# click the dots button, to make the input appear
driver.execute_script("arguments[0].click();", dots_button)
# select the input, modify the value and then make it lose focus
dots_input = dots_button.find_element('xpath', "//span[#class='ellipse clickable']/input")
print(dots_input)
driver.execute_script("arguments[0].setAttribute('value','5000');", dots_input)
driver.execute_script("arguments[0].blur();", dots_input)
# just so you can see before the driver closes the browser again
sleep(10)
Note that, instead of waiting a number of seconds, the script simply waits until a specific element is loaded and then continues. There's still a delay defined, but this is now the maximum amount of time the script will wait for the element to load, instead of just waiting a fixed amount of time.
Once the button is clicked, a nested input element appears, which is selected and then JavaScript is executed to change its value and cause it to lose focus (blur), so that the new value gets applied.
(note that I have the Chrome driver in a subfolder next to the script, called bin)
i basically want to click every load more button thats on the page before running the rest of my code because otherwise i wont be able to access each profile.
There are 2 Problems:
first how do i even access it? i tried similar methods to the fancyCompLabel part of my code but it wont work.
second im not sure how i should loop through all buttons since i would assume the second button only starts loading until the first one is clicked.
heres the relevant html part and a picture of the button
<span type="button" class="md-text-button button-orange-white" onclick="loadFollowing();">mehr anzeigen</span>
Heres the code to access each profile but as you can see it only runs until the first load button.
from bs4 import BeautifulSoup
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
time.sleep(3)
# Set some Selenium Options
options = webdriver.ChromeOptions()
# options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
# Webdriver
wd = webdriver.Chrome(executable_path='/usr/bin/chromedriver', options=options)
# URL
url = 'https://www.techpilot.de/zulieferer-suchen?laserschneiden%202d%20(laserstrahlschneiden)'
# Load URL
wd.get(url)
# Get HTML
soup = BeautifulSoup(wd.page_source, 'html.parser')
wait = WebDriverWait(wd, 15)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#bodyJSP #CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "#efficientSearchIframe")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".hideFunctionalScrollbar #CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll"))).click()
#wd.switch_to.default_content() # you do not need to switch to default content because iframe is closed already
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".fancyCompLabel")))
results = wd.find_elements_by_css_selector(".fancyCompLabel")
for profil in results:
print(profil.text) #heres the rest of my code but its not relevant
wd.close()
As I see the second pop-up element located by .hideFunctionalScrollbar #CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll is initially appearing out of the visible screen.
So after switching to the iframe you need to scroll to that element before trying to click on it.
Also, presence_of_all_elements_located doesn't actually wait for all the elements presence. It even doesn't know how many such elements will be. It returns once it finds at least 1 element matching the passed locator.
So I'd advice to add a short sleep after that line to allow all those elements to be actually loaded.
from selenium.webdriver.common.action_chains import ActionChains
wait = WebDriverWait(wd, 15)
actions = ActionChains(wd)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#bodyJSP #CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "#efficientSearchIframe")))
second_pop_up = wd.find_element_by_css(".hideFunctionalScrollbar #CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll")
actions.move_to_element(second_pop_up).build().perform()
time.sleep(0.5)
second_pop_up.click()
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".fancyCompLabel")))
time.sleep(0.5)
for profil in results:
print(profil.text) #heres the rest of my code but its not relevant
wd.close()
I am trying to click on the "CSV" button at this website https://covid19-vaccine-report.ecdc.europa.eu/#6_Reported_data using selenium.
However, with the below code I am receiving the error selenium.common.exceptions.ElementClickInterceptedException: Message: Element <button class="dt-button buttons-csv buttons-html5" type="button"> is not clickable at point (464,600) because another element <div class="cckBannerInner"> obscures it.
The code:
import os
import glob
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
from selenium.webdriver.firefox.options import Options
# get directory of the current script
CURR_DIR = os.path.dirname(os.path.abspath(__file__))
URL = 'https://covid19-vaccine-report.ecdc.europa.eu/#6_Reported_data'
ACCEPT_COOKIES_ID = 'cckAcceptCookies'
VACCINES_XPATH = '//*[#id="DataTables_Table_0_wrapper"]/div[5]/button[2]'
# before downloading new files, remove old ones
for f in glob.glob(os.path.join(CURR_DIR, "dataset_2021-W*")):
os.remove(f)
# this will prevent opening browser window
# (https://stackoverflow.com/a/60627463/1979665)
options = Options()
#options.add_argument("--headless")
# (https://stackoverflow.com/a/18440478/1979665)
# To prevent download dialog
profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2) # custom location
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference('browser.download.dir', CURR_DIR)
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'text/csv')
# (https://stackoverflow.com/a/32416545/1979665)
profile.set_preference("network.cookie.cookieBehavior", 2)
# (https://stackoverflow.com/a/58727916/1979665)
browser = webdriver.Firefox(profile,
executable_path=GeckoDriverManager().install(),
options=options)
browser.get(URL)
wait = WebDriverWait(browser, 2000000)
# accept cookies
accept_cookies = browser.find_element_by_id(ACCEPT_COOKIES_ID)
browser.execute_script("arguments[0].click();", accept_cookies)
# download vaccines as CSV
wait.until(EC.element_to_be_clickable((By.XPATH, VACCINES_XPATH))).click()
browser.quit()
The XPATH of the button was taken with the copy XPath feature of the Firefox web developer console.
I have read many other suggested answers and tried their solutions but none worked.
I use JavaScript to accept the cookies (browser.execute_script("arguments[0].click();", accept_cookies)), in order for the banner to go away, because when I tried with other methods I received the error could not be scrolled into view.
Then I am using WebDriverWait to wait for the desired button to be clickable (with element_to_be_clickable).
When the code runs, I see what's happening in the browser window, and I do see that the cookie are accepted and the banner goes away. Then the page is scrolled down to the CSV button, but I receive the aforementioned error, like if the button is still hidden by the cookie banner.
The exact error (I repeat it down here), indeed refers to the cookie banner this way:
selenium.common.exceptions.ElementClickInterceptedException: Message: Element <button class="dt-button buttons-csv buttons-html5" type="button"> is not clickable at point (464,600) because another element <p class="cckInform"> obscures it
I also tried increasing the wait time up to 5000000 but it did not make any difference.
WORKING CODE
Thanks to #Prophet I was able to make it work. The trick was to scroll down the page until the button becomes visible and only then click on it.
I also removed the "accept coockie" part this way, which is great given that now I have to deal with only one element this way.
The updated working code:
import os
import glob
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
from selenium.webdriver.firefox.options import Options
# get directory of the current script
CURR_DIR = os.path.dirname(os.path.abspath(__file__))
URL = 'https://covid19-vaccine-report.ecdc.europa.eu/#6_Reported_data'
VACCINES_XPATH = '//*[#id="DataTables_Table_0_wrapper"]/div[5]/button[2]'
# before downloading new files, remove old ones
for f in glob.glob(os.path.join(CURR_DIR, "dataset_2021-W*")):
os.remove(f)
# this will prevent opening browser window
# (https://stackoverflow.com/a/60627463/1979665)
options = Options()
#options.add_argument("--headless")
# (https://stackoverflow.com/a/18440478/1979665)
# To prevent download dialog
profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2) # custom location
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference('browser.download.dir', CURR_DIR)
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'text/csv')
# (https://stackoverflow.com/a/32416545/1979665)
profile.set_preference("network.cookie.cookieBehavior", 2)
# (https://stackoverflow.com/a/58727916/1979665)
driver = webdriver.Firefox(profile,
executable_path=GeckoDriverManager().install(),
options=options)
driver.get(URL)
wait = WebDriverWait(driver, 2000000)
vaccines = wait.until(EC.element_to_be_clickable((By.XPATH, VACCINES_XPATH)))
driver.execute_script("arguments[0].scrollIntoView();", vaccines)
vaccines.click()
driver.quit()
Try the following: scroll into view some other element that is below the desired button so that the desired button will not be covered by the cookies banner and then try clicking the button.
national_references = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#national-references")))
actions = ActionChains(driver)
actions.move_to_element(national_references).perform()
wait.until(EC.element_to_be_clickable((By.XPATH, VACCINES_XPATH))).click()
UPD:
Since scrolling to the national_references was not possible since it is out of bounds of viewport width (1366) and height (630) according to the exception thrown when trying so scroll it into the view with actions.move_to_element(national_references).perform() we can scroll page down until national_references becomes visible.
Now we can successfully click the vaccines CSV button.
So I am trying to scrape a website, and I want to click an element, go to the page that opens from the click, find another element and click that one. The first click seems to work, no errors, but the next page doesn't open, thus I get an error. Here is a screenshot of what I want to click on the fist page: https://prnt.sc/10l8xa4. Clicking that should redirect to the second page. The problem seems to be that the driver clicks the element but nothing happens:
import sys, csv, os
from selenium import webdriver # Selenium 3.141.0
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
from datetime import datetime
from time import sleep
class Scraper(object):
''' A lot of the messy code is just playing with the tags from the page'''
def __init__(self, link):
self.link = link
self.driver = self.configure_driver() # The simulated browser
# Configuring the browser simulator, named driver, that will get all the information
def configure_driver(self):
# Add additional Options to the webdriver
chrome_options = Options()
# add the argument and make the browser Headless. It will work smoother& faster but it will miss the first category
# chrome_options.add_argument("--headless")
driver = webdriver.Chrome(options = chrome_options)
return driver
def click_element(self, selector): # Clicks the provided element from the page, even if not visible
element = WebDriverWait(self.driver, 20).until(
EC.presence_of_element_located((By.CSS_SELECTOR, selector)))
ActionChains(self.driver).move_to_element(element).click(element).perform()
if __name__ == '__main__':
product_link = 'https://www.action.com/nl-nl/click-and-collect-producten/' # An example of a product
app = Scraper(product_link)
with app.driver:
app.driver.get(product_link)
app.click_element('a.content-card.has-text.card-theme--light.card-size--s.card-align--bottom-left') # This gets clicked and should open new page, but it doesn't
sleep(10)
app.click_element('a.product-card__link') # This throws a Timeout, because the element can't be found, which is obvoius because the second page(which has this element) didn't open
sleep(20)
Try use like that:
with app.driver:
app.driver.get(product_link)
sleep(2)
app.click_element('li.has-submenu')
sleep(2)
app.click_element(
'div.grid-item.grid-item--content') # This gets clicked and should open new page, but it doesn't
sleep(2)
you should add step of openning pop-up and then click on your aimed button
Code with required argument but, by xpath:
def click_element(self, selector, by=By.CSS_SELECTOR): # Clicks the provided element from the page, even if not visible
element = WebDriverWait(self.driver, 20).until(
EC.presence_of_element_located((by, selector)))
ActionChains(self.driver).move_to_element(element).click(element).perform()
if __name__ == '__main__':
product_link = 'https://www.action.com/nl-nl/click-and-collect-producten/' # An example of a product
app = Scraper(product_link)
with app.driver:
app.driver.get(product_link)
sleep(2)
app.click_element("//section[#class='grid']/div[#class='grid-item grid-item--content'][1]", By.XPATH)
sleep(2)
I'm working on a selenium based python automation project. The code automates some regular tasks by connecting to the company's internal webpage url. However, the code occasionally throws the same exception between two button click actions. Could you please help me to figure out what's the point that I miss? Thanks in advance.
You can find my code snippet and error screenshot here:
selenium.common.exceptions.ElementClickInterceptedException: Message:
element click intercepted: Element ... is not clickable at point (180,
447). Other element would receive the click: ...
pycharm_error_output_screenshot
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
timeout = 200
options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)
driver.implicitly_wait(3)
wait = WebDriverWait(driver, timeout)
driver.get("https://internal_web_appplication_page_for_company_x")
apply_button_v4 = wait.until(EC.visibility_of_element_located((By.XPATH, "//body//button[2]")))
apply_button_v4.click()
both_button = wait.until(EC.element_to_be_clickable((By.XPATH, "// label[3] // span[1]")))
wait.until_not(EC.element_to_be_clickable((By.CLASS_NAME, "map-mask")))
wait.until_not(EC.element_to_be_clickable((By.XPATH, "// *[ contains( # class, 'map-mask')]")))
both_button.click()
The intercepted element is full screen "loading" notification which is visible for a short period of time following any click action.
Moreover, my code works as expected if i put time.sleep(5) before clicking "both_button".
Try below solution for clicking on both_button. There are multiple reasons of this exception like javascript or ajax call , element is not in the view port.
actions = ActionChains(driver)
actions.move_to_element(both_button)
actions.click(both_button)
actions.perform()