Here is the link to the site I am currently viewing: https://messari.io/tool/fb8d86ca-d3cf-4568-8d48-1a052c95364e. Scroll to the bottom of the page and click "View More".
I am trying to figure out how to click the x button but what I have tried hasn't worked. I get a "no such element: Unable to locate element error.
I have tried all three of these:
driver.find_element(By.CLASS_NAME, "button").click()
driver.find_element_by_xpath("//button[contains(#data-testid='CloseIcon')]").click()
driver.find_element_by_tag_name("svg").click()
Check the Xpath of the load more button //*[#id="root"]/div[2]/div/div[2]/div[2]/div[3]/button -
And then check the full Xpath of the X button that will close the pop-up window
full Xpath of the X button - '/html/body/div[2]/div[3]/div/h2/button'
working code -
import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1920x1080")
options.add_argument("--disable-extensions")
chrome_driver = webdriver.Chrome(
service=Service(ChromeDriverManager().install()),
options=options
)
def messari_scraper():
URL = "https://messari.io/tool/fb8d86ca-d3cf-4568-8d48-1a052c95364e"
with chrome_driver as driver:
driver.implicitly_wait(15) # wait max 15 sec for any element to find
driver.get(URL)
time.sleep(3)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") # scroll to the end of the page
# click the button
driver.find_element(By.XPATH, '//*[#id="root"]/div[2]/div/div[2]/div[2]/div[3]/button').click()
time.sleep(3)
# get the full Xpath of the close `x` button of the pop-up
driver.find_element(By.XPATH, '/html/body/div[2]/div[3]/div/h2/button').click()
# pop up window closed
time.sleep(5)
# do your tasks here....
messari_scraper()
Related
I tried with XPath but selenium can't click this image/button.
from undetected_chromedriver.v2 import Chrome
def test():
driver = Chrome()
driver.get('https://bandit.camp')
WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,"/html/body/div[1]/div/main/div/div/div/div/div[5]/div/div[2]/div/div[3]/div"))).click()
if __name__ == "__main__":
test()
Try the below one, I checked it, and it is working fine, while clicking on the link it is opening a separate window for login.
free_case = driver.find_element(By.XPATH, ".//p[contains(text(),'Open your free')]")
driver.execute_script("arguments[0].scrollIntoView(true)", free_case)
time.sleep(1)
driver.execute_script("arguments[0].click();", free_case)
First you need to wait for presence of that element.
Then you need to scroll the page down to make that element visible since initially this element is out of the visible screen so you can't click it.
Now you can click it and it works.
The code below is working:
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
options.add_argument('--disable-notifications')
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://bandit.camp/"
driver.get(url)
element = wait.until(EC.presence_of_element_located((By.XPATH, "//div[contains(.,'daily case')][contains(#class,'v-responsive__content')]")))
element.location_once_scrolled_into_view
time.sleep(0.3)
element.click()
I just need to click the load more button once to reveal a bunch more information so that I can scrape more HTML than what is loaded.
The following "should" go to github.com/topics and find the one and only button element and click it one time.
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Edge()
driver.get("https://github.com/topics")
time.sleep(5)
btn = driver.find_element(By.TAG_NAME, "button")
btn.click()
time.sleep(3)
driver.quit()
I'm told Message: element not interactable so I'm obviously doing something wrong but I'm not sure what.
use
btn = driver.findElementsByXPath("//button[contains(text(),'Load more')]");
You are not finding the right element. This is the reason why it is not "interactable"
There are several issues with your code:
The "Load more" button is initially out of the view, so you have to scroll the page in order to click it.
Your locator is bad.
You need to wait for elements to appear on the page before accessing them. WebDriverWait expected_conditions explicit waits should be used for that, not hardcoded sleeps.
The following code works, it scrolls the page and clicks "Load more" 1 time.
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 20)
url = "https://github.com/topics"
driver.get(url)
load_more = wait.until(EC.presence_of_element_located((By.XPATH, "//button[contains(.,'Load more')]")))
load_more.location_once_scrolled_into_view
time.sleep(1)
load_more.click()
UPD
You can simply modify the above code to make it clicking Load more button while it presented.
I implemented this with infinite while loop making a break if Load more button not found. This code works.
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)
url = "https://github.com/topics"
driver.get(url)
while True:
try:
load_more = wait.until(EC.presence_of_element_located((By.XPATH, "//button[contains(.,'Load more')]")))
load_more.location_once_scrolled_into_view
time.sleep(1)
load_more.click()
except:
break
I keep getting the ElementClickInterceptedException on this script I'm writing, I'm supposed to click a link that will open a new window, scrape from the new window and close it and move to the next link to scrape, but it just won't work, it gives the error after max 3 link clicks. I saw a similar question here and I tried using wait.until(EC.element_to_be_clickable()) and also maximized my screen but still did not work for me. Here is the site I am scraping from trying to scrape all the games for each day and here is a chunk of the code I'm using
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium.common.exceptions import TimeoutException, NoSuchElementException, ElementNotInteractableException, StaleElementReferenceException
from time import sleep
l = "https://www.flashscore.com/"
options = FirefoxOptions()
#options.add_argument("--headless")
driver = webdriver.Firefox(executable_path="geckodriver.exe",
firefox_options=options)
driver.install_addon('C:\\Windows\\adblock_plus-3.10.1-an+fx.xpi')
driver.maximize_window()
driver.get(l)
driver.implicitly_wait(5)
cnt = 0
sleep(5)
wait = WebDriverWait(driver, 20)
a = driver.window_handles[0]
b = driver.window_handles[1]
driver.switch_to.window(a)
# Close Adblock tab
if 'Adblock' in driver.title:
driver.close()
driver.switch_to.window(a)
else:
driver.switch_to.window(b)
driver.close()
driver.switch_to.window(a)
var1 = driver.find_elements_by_xpath("//div[#class='leagues--live ']/div/div")
knt = 0
for i in range(len(var1)):
if (var1[i].get_attribute("id")):
knt += 1
#sleep(2)
#driver.switch_to.window(driver.window_handles)
var1[i].click()
sleep(2)
#var2 = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(#classs, 'event__match event__match--last event__match--twoLine')]")))
print(len(driver.window_handles))
driver.switch_to.window(driver.window_handles[1])
try:
sleep(4)
driver.close()
driver.switch_to.window(a)
#sleep(3)
except(Exception):
print("Exception caught")
#WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.CLASS_NAME, "event__match event__match--last event__match--twoLine")))
sleep(10)
driver.close()
Any ideas to help please.
It looks like the element you are trying to click on is covered by a banner ad or something else like a cookie message.
To fix this you can scroll down to the last element using the following code:
driver.execute_script('\
let items = document.querySelectorAll(\'div[title="Click for match detail!"]\'); \
items[items.length - 1].scrollIntoView();'
)
Add it before clicking on the desired element in the loop.
I tried to make a working example for you but it works on chromedriver not gecodriver:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation", "enable-logging"])
service = Service(executable_path='your\path\to\chromedriver.exe')
driver = webdriver.Chrome(service=service, options=options)
wait = WebDriverWait(driver, 5)
url = 'https://www.flashscore.com/'
driver.get(url)
# accept cookies
wait.until(EC.presence_of_element_located((By.ID, 'onetrust-accept-btn-handler'))).click()
matches = driver.find_elements(By.CSS_SELECTOR, 'div[title="Click for match detail!"]')
for match in matches:
driver.execute_script('\
let items = document.querySelectorAll(\'div[title="Click for match detail!"]\'); \
items[items.length - 1].scrollIntoView();'
)
match.click()
driver.switch_to.window(driver.window_handles[1])
print('get data from open page')
driver.close()
driver.switch_to.window(driver.window_handles[0])
driver.quit()
It works in both normal and headless mode
I'm trying to scrape this site:https://www.wagr.com/mens-ranking, at the bottom right of the table there is a button to click to the next page, but selenium keeps throwing exceptions when I try to click it. The code below is what I'm using to click the button.
next = driver.find_element(By.CSS_SELECTOR,'.next > a:nth-child(1)')
next.click()
Here's a screenshot of the traceback:
I can't understand why this isn't working, I'd be grateful for any tips.
You need to
Handle the cookie pop up
Scroll down at the bottom of the page, so that the button could be visible
Here is a working code -
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
chrome_driver = webdriver.Chrome(
service=Service(ChromeDriverManager().install()),
options=options
)
with chrome_driver as driver:
driver.implicitly_wait(15)
driver.get('https://www.wagr.com/mens-ranking')
time.sleep(3)
# click cookie popup
cookie_btn = driver.find_element(By.XPATH, "/html/body/div[2]/div[3]/div/div/div[2]/div[1]/button")
cookie_btn.click()
time.sleep(0.3)
# scrolling bottom
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)
next_btn = driver.find_element(By.CSS_SELECTOR, '.next > a:nth-child(1)') # li.next
# next_btn = driver.find_element(By.XPATH, "//li[#class='next']")
print("found and click next", next_btn.tag_name)
next_btn.click()
time.sleep(2)
driver.quit()
I am new to selenium and web development. I am working on a project to take screenshots from the websites from web.archive.org.
Here is a link to the sample webpage. I am trying to click on the close button (on the top right of this page) before capturing the screenshots. I am not sure what kind of element is the close button and I was unsuccessful in my attempts.
Here is the element I am attempting to click from selenium:
Here is the corresponding HTML
<a id="wm-tb-close" href="#close" style="top:-2px;" title="Close the toolbar"><span class="iconochive-remove-circle" style="color:#888888;font-size:240%;"></span></a>
Here is my code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_argument('--disable-dev-shm-usage')
options.binary_location = "<path-to-local-dir>/google-chrome/opt/google/chrome/google-chrome"
driver = webdriver.Chrome(options=options)
driver.get('https://web.archive.org/web/20220315011343/https://stackoverflow.com/')
# My attempts at closing the wayback toolbar:
# driver.find_element(By.LINK_TEXT, 'close').click() # Attempt 1
# driver.find_element(By.ID, 'wm-tb-close').click() # Attempt 2
# driver.find_element_by_xpath("a[#title='Close the toolbar']").click() # Attempt 3
# Capture full webpage screenshot (with scrolling)
original_size = driver.get_window_size()
# required_width = driver.execute_script('return document.body.parentNode.scrollWidth')
required_height = driver.execute_script('return document.body.parentNode.scrollHeight')
driver.set_window_size(1920, max(required_height, 1080))
driver.find_element(By.TAG_NAME, 'body').screenshot('webpage_screenshot.png') # avoids scrollbar
driver.set_window_size(original_size['width'], original_size['height']) # reset to defaults
driver.quit()
When I make an attempt to click on the close button using the three techniques (shown in the code), I receive the following error:
selenium.common.exceptions.NoSuchElementException: Message: no such
element: Unable to locate element
The element you are trying to click is inside the SHADOW-ROOT, to access such elements you need to use some special techniques.
Also you need to use Expected Conditions explicit waits to let the elements loaded before accessing them.
This should work:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_argument('--disable-dev-shm-usage')
options.binary_location = "<path-to-local-dir>/google-chrome/opt/google/chrome/google-chrome"
driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 20)
driver.get('https://web.archive.org/web/20220315011343/https://stackoverflow.com/')
#locate the shadow root element
root_element = wait.until(EC.presence_of_element_located((By.ID, "wm-ipp-base")))
shadow_root = driver.execute_script('return arguments[0].shadowRoot', root_element)
#access the close button inside the shadow root
shadow_root.find_element_by_css_selector("span.iconochive-remove-circle").click()