Develop Reference

Selenium cannot click button because there is a div overlay

I am trying to scrap this webpage https://www.tecnocasa.es/venta/piso/barcelona/barcelona/510567.html, the code i use is the following, and i think is correct:
options = Options()
options.headless = False
driver = webdriver.Firefox(options=options, executable_path=r'geckodriver.exe')
driver.get(url)
# ("Headless Firefox Initialized")
#dins del id= cookie-banner
frame = driver.find_element(by=By.CSS_SELECTOR,value = '[id = "cookie-banner"]')
print(frame)
buttons = frame.find_elements(by=By.TAG_NAME, value='button')
print(buttons)
buttons[0].click()
time.sleep(5)
The buttons are found but when i try to click one i have the error <button class="btn-default"> is not clickable at point (1138,829) because another element <div id="hide-overlay" class="hide-overlay fade-leave-active fade-leave-to"> obscures it
How can i disable this overlay and click the button to accept the conditions of the page?

Your code works without any problem on my computer, try this one
driver.find_element(By.CSS_SELECTOR, '#close').click()
If also this will not work, you can try this code which hides the banner. However, you should run it each time that a new page is loaded
overlay = driver.find_element(By.CSS_SELECTOR, 'div.cookies-overlay')
driver.execute_script("arguments[0].style.display = 'none';", overlay)
banner = driver.find_element(By.CSS_SELECTOR, '#cookie-banner')
driver.execute_script("arguments[0].style.display = 'none';", banner)

Q: Can't keep clicking button by chorme web driver in python

Question:
How can I show all reviews by clicking "Show more reviews" button?
What did I do:
To scrape all reviews, I decided to Keep clicking until that button disappears.
But some new reviews didn't appear after clicking for 8 times (using while below).
I already checked xpath of the button but it didn't changed.
If I click the button manually without the driver, I can see new reviews.
import time, random
from selenium import webdriver
from latest_user_agents import get_random_user_agent
### ser user_agent
user_agent = get_random_user_agent()
options = webdriver.ChromeOptions()
options.add_argument("--user-agent=" + user_agent)
driverpath = "C:/Users/~~/chromedriver.exe"
driver = webdriver.Chrome(chrome_options=options,executable_path=driverpath)
### target url
url = "https://www.sony.co.uk/electronics/truly-wireless/wf-l900/reviews-ratings"
### open URL
driver.get(url)
time.sleep(5)
# accept cookies
driver.find_element_by_xpath('//*[#id="onetrust-accept-btn-handler"]').click()
# show all reviews
try:
while True:
driver.execute_script('window.scroll(0,1000000);')
driver.find_element_by_xpath('//*[#id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button').click()
time.sleep(random.randrange(2, 5, 1))
except:
print("---------------- finish showing all reviews ----------------")

Try going to the end of the webpage and then do it as the button might not been loaded fully.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
You might also try to press the button only when it is available.
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "myElement")))

You can try javascript click:
element = driver.find_element_by_xpath('//*[#id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button')
driver.execute_script("arguments[0].click();", element)
In this method, the scrolling shouldn't be necessary. See more info about the subject here.

How can I automatically click on the text "Show more matches"? My code doesn't work

There is a text button that says "Show more matches", it is used to scroll down the page, because no more results come out. I would like to press it automatically, but there is something wrong with my code. Can you help me please?
IMPORTANT: It is not yet on this page, but soon there will be a second button "Show more meetings", because I scroll down and find "Show more meetings", then in a few months if I go down even further and find a second button "Show more matches". In the page there is no second button yet, but I would like to make it press that too (it's the same same button, so I don't think it's complicated).
So I would like to press 2, but also 3 the same, at the same time
P.S: The purpose of the request and the code is only for personal study reasons, so for personal didactic reasons, no profit. This question and this code is not for commercial or profit-making purposes.
driver.get("url")
driver.implicitly_wait(12)
wait = WebDriverWait(driver, 12)
try:
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "a[class^='event__more event__more--static']"))).click() #because another element <div class="skOT skOT--ot"> obscures it
except Exception as ex:
print('EX:', ex)
UPDATE:Before driver.get and the link, there is this
torexe_linux = os.popen('/home/xxxxx/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US')
profile = FirefoxProfile('/home/xxxx/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False) #certi la tengono True
profile.update_preferences()
firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = '/usr/bin/firefox'
driver = webdriver.Firefox(
firefox_profile=profile, options=firefox_options,
executable_path='/usr/bin/geckodriver')

You need to close accept cookies panel.
Show more button appears on the bottom of the page, you need to scroll it to the view in order to click it.
After clicking it you need to wait for the page loading in order to get that element again, scroll to it again and the click it again.
As following:
from selenium.webdriver.common.action_chains import ActionChains
driver.get("link")
driver.implicitly_wait(12)
wait = WebDriverWait(driver, 12)
actions = ActionChains(driver)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
while(driver.find_elements_by_css_selector('a.event__more.event__more--static')):
show_more = driver.find_element_by_css_selector('a.event__more.event__more--static')
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
actions.move_to_element(show_more).perform()
time.sleep(0.5)
show_more = driver.find_element_by_css_selector('a.event__more.event__more--static')
show_more.click()
time.sleep(3)

Selenium + Python Click on Search Selects but Does Not Click

I have all of my options selected on this form and have the following code to click the Search button:
WebDriverWait(wd, 10).until(EC.element_to_be_clickable((By.ID, "alMatchFrequencies"))).click()
I have also tried this:
search = WebDriverWait(wd, 10).until(EC.element_to_be_clickable((By.ID, "alMatchFrequencies")))
search.send_keys(Keys.ENTER)
I have also tried subbing in RETURN instead of ENTER.
Here is a screenshot after that line is run:
The search bar is highlighted, but no results are being displayed. I can replicate this screen on my own and am able to press the search button so I don't think it is missing fields.

I believe you are actually clicking the search button but are not seeing the results because they are below the view you show. The scrollable area in your screenshot was a clue.
Clicking the search button is as easy as search.click() after you've executed the selector. But then you need to wait and scroll the page.
To Scroll a page you need to introduceActionChains in the Selenium library and to wait for the load you'll need to use Python's time.sleep(sec) function.
Full code to produce the attached picture:
#Imports
import time
from selenium.webdriver.common.action_chains import ActionChains
#...Code to Load and Select Filters...#
search = WebDriverWait(wd, 3000).until(EC.element_to_be_clickable((By.ID, "alMatchFrequencies")))
search.click()
# Wait for element to load
time.sleep(3)
# Locate element to scroll to
results = WebDriverWait(wd,3000).until(EC.presence_of_element_located((By.XPATH, "/html/body/div[1]/section/div/div[2]/div/div[2]/div[5]/div/div[1]/div[5]")))
actions = ActionChains(wd)
# Scroll
actions.move_to_element(results).perform()
wd.get_screenshot_as_file("test3.png")

Selenium is misclicking

I'm trying to make Selenium click a button, Selenium has to scroll in order to click that button. But what happens is that when Selenium scrolls down, the button gets behind the sticky bar and Selenium clicks on the sticky bar instead of the button.
I'm using Firefox webdriver.
I tried the following:
DesiredCapabilities.FIREFOX["elementScrollBehavior"] = 1
self.driver = webdriver.Firefox()
When I have elementScrollBehavior set to 1, then it doesn't scroll at all.
How can I make Selenium scroll down properly so that the button doesn't get behind the sticky bar?
-edit-
DesiredCapabilities.FIREFOX["elementScrollBehavior"] = 1
does work now, but it still isn't able to click on the button. Even tho the sticky bar doesn't get in the way now.

You could implement something like this:
driver = webdriver.Firefox()
driver.get("http://somewebpage.example")
time.sleep(5)
while True:
try:
# Edit this to how you're currently scrolling
driver.execute_script("window.scrollBy(0, 150);")
button = driver.find_element(By.XPATH, YOUR_BUTTON_XPATH)
button.click()
break
except WebDriverException:
driver.execute_script("window.scrollBy(0, -100);")
time.sleep(2)
If your script scrolls to far and throws an exception, it will catch this error and scroll back up the page 100 pixels.
This should guarantee your button gets clicked, it will constantly move 50px down the page until it finds it but you can implement your normal way of scrolling and then catch the WebDriverException and scroll back up a couple of pixels.

You could try to use a javascript scroll to element then scroll a little more with an offset
eg in c#.
IJavaScriptExecutor jse = (IJavaScriptExecutor)driver;
jse.ExecuteScript("arguments[0].scrollIntoView(true)", element)
jse.ExecuteScript("window.scrollBy(0,50)");