I am new to selenium and web development. I am working on a project to take screenshots from the websites from web.archive.org.
Here is a link to the sample webpage. I am trying to click on the close button (on the top right of this page) before capturing the screenshots. I am not sure what kind of element is the close button and I was unsuccessful in my attempts.
Here is the element I am attempting to click from selenium:
Here is the corresponding HTML
<a id="wm-tb-close" href="#close" style="top:-2px;" title="Close the toolbar"><span class="iconochive-remove-circle" style="color:#888888;font-size:240%;"></span></a>
Here is my code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_argument('--disable-dev-shm-usage')
options.binary_location = "<path-to-local-dir>/google-chrome/opt/google/chrome/google-chrome"
driver = webdriver.Chrome(options=options)
driver.get('https://web.archive.org/web/20220315011343/https://stackoverflow.com/')
# My attempts at closing the wayback toolbar:
# driver.find_element(By.LINK_TEXT, 'close').click() # Attempt 1
# driver.find_element(By.ID, 'wm-tb-close').click() # Attempt 2
# driver.find_element_by_xpath("a[#title='Close the toolbar']").click() # Attempt 3
# Capture full webpage screenshot (with scrolling)
original_size = driver.get_window_size()
# required_width = driver.execute_script('return document.body.parentNode.scrollWidth')
required_height = driver.execute_script('return document.body.parentNode.scrollHeight')
driver.set_window_size(1920, max(required_height, 1080))
driver.find_element(By.TAG_NAME, 'body').screenshot('webpage_screenshot.png') # avoids scrollbar
driver.set_window_size(original_size['width'], original_size['height']) # reset to defaults
driver.quit()
When I make an attempt to click on the close button using the three techniques (shown in the code), I receive the following error:
selenium.common.exceptions.NoSuchElementException: Message: no such
element: Unable to locate element
The element you are trying to click is inside the SHADOW-ROOT, to access such elements you need to use some special techniques.
Also you need to use Expected Conditions explicit waits to let the elements loaded before accessing them.
This should work:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_argument('--disable-dev-shm-usage')
options.binary_location = "<path-to-local-dir>/google-chrome/opt/google/chrome/google-chrome"
driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 20)
driver.get('https://web.archive.org/web/20220315011343/https://stackoverflow.com/')
#locate the shadow root element
root_element = wait.until(EC.presence_of_element_located((By.ID, "wm-ipp-base")))
shadow_root = driver.execute_script('return arguments[0].shadowRoot', root_element)
#access the close button inside the shadow root
shadow_root.find_element_by_css_selector("span.iconochive-remove-circle").click()
Related
I tried to get data from the menu in this website. I've written this script which seems work well in debug mode. In this script, i close the chrome every time I've get information for certain city.
In debug mode, the detail information is shown when 'element' is clicked by the script. However, if i run the script, it seems that it doesn't do anything after city information is sent. The button 'Visualize Results' is enabled in the website when city data is entered, but the detail information that supposed to be shown after clicking this button by the script is not shown as if it is not clicked. Is there something that i miss?
thank you in advance.
for city in Cities:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('prefs', prefs)
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get(link)
driver.find_element(By.ID, "inputText").send_keys(city)
driver.find_element(By.ID, 'btninputText').click()
element = wait(driver, 5).until(EC.presence_of_element_located((By.XPATH,div[#id="btviewPVGridGraph"]'))).click()
driver.close()
You need to wait for clickability of all 3 elements you accessing here. Presence is not enough.
Need to add a short delay between clicking the search button and clicking visualization button
The following code works:
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 30)
url = "https://re.jrc.ec.europa.eu/pvg_tools/en/"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.ID, "inputText"))).send_keys("Paris")
wait.until(EC.element_to_be_clickable((By.ID, "btninputText"))).click()
time.sleep(2)
wait.until(EC.element_to_be_clickable((By.ID, "btviewPVGridGraph"))).click()
Th result is:
Try below xpath
wait(driver, 5).until(EC.presence_of_element_located(By.XPATH,'//div[#id="btviewPVGridGraph"]')).click()
I want to change or at least make any small effect in the korean custom website but it seems they are not accessible by my code! Maybe they are in the internal iframe or not, I don't know. I want to change dropdown, write something in textbox and click search button, but I cannot.
May anyone help me?
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
webdriver_service = Service('C:\Webdriver\chromedriver.exe')
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
url = 'http://www.kita.org/kStat/byCom_AllCount.do'
browser.get(url)
time.sleep(5)
select = Select(WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, '/html/body/table/tbody/tr/td/table/tbody/tr/td/form/table/tbody/tr/td/table[3]/tbody/tr/td/table/tbody/tr/td/table/tbody/tr[2]/td[2]/select')))) # select dropdown
select.select_by_index(1)
browser.find_element(By.XPATH,'/html/body/table/tbody/tr/td/table/tbody/tr/td/form/table/tbody/tr/td/table[2]/tbody/tr/td[2]/table/tbody/tr/td[2]/a/img').click() #click seach Button
time.sleep(5)
There is an iframe there.
You need to switch to it first in order to access elements in it.
Also, you should improve your locators.
And insert some text into the item input.
The following code works
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = "http://www.kita.org/kStat/byCom_AllCount.do"
driver.get(url)
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID,"iframe_stat")))
select = Select(WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[name='cond_choosefield']"))))
select.select_by_index(1)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[name='cond_prdt_cd']"))).send_keys("kuku")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='searchForm']"))).click()
Here is the link to the site I am currently viewing: https://messari.io/tool/fb8d86ca-d3cf-4568-8d48-1a052c95364e. Scroll to the bottom of the page and click "View More".
I am trying to figure out how to click the x button but what I have tried hasn't worked. I get a "no such element: Unable to locate element error.
I have tried all three of these:
driver.find_element(By.CLASS_NAME, "button").click()
driver.find_element_by_xpath("//button[contains(#data-testid='CloseIcon')]").click()
driver.find_element_by_tag_name("svg").click()
Check the Xpath of the load more button //*[#id="root"]/div[2]/div/div[2]/div[2]/div[3]/button -
And then check the full Xpath of the X button that will close the pop-up window
full Xpath of the X button - '/html/body/div[2]/div[3]/div/h2/button'
working code -
import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1920x1080")
options.add_argument("--disable-extensions")
chrome_driver = webdriver.Chrome(
service=Service(ChromeDriverManager().install()),
options=options
)
def messari_scraper():
URL = "https://messari.io/tool/fb8d86ca-d3cf-4568-8d48-1a052c95364e"
with chrome_driver as driver:
driver.implicitly_wait(15) # wait max 15 sec for any element to find
driver.get(URL)
time.sleep(3)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") # scroll to the end of the page
# click the button
driver.find_element(By.XPATH, '//*[#id="root"]/div[2]/div/div[2]/div[2]/div[3]/button').click()
time.sleep(3)
# get the full Xpath of the close `x` button of the pop-up
driver.find_element(By.XPATH, '/html/body/div[2]/div[3]/div/h2/button').click()
# pop up window closed
time.sleep(5)
# do your tasks here....
messari_scraper()
When I run this program, I can successfully open up the sign-in page for w3schools.com, but; I cannot send keys into a textbox and the driver.wait times out. My original thought was that I didn't have the right name for the box, although testing it with XPath gave the same result. I later discovered that this can be an issue with frames on the HTML page instead of the names of the textboxes.
I tried to use driver.switch_to.frame(driver.find_element(By.NAME, 'email')) to go to the frame of the textbox, however; I got an error saying that ...Message: no such frame: element is not a frame Trying to locate the frame by searching for the XPath of the form says that it is not a frame.
Website I am trying to create an account for:
https://profile.w3schools.com/signup?redirect_url=https%3A%2F%2Fspaces.w3schools.com%2Fcreatespace%2Fstepone
from selenium.webdriver import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
# options.add_argument('--headless')
# options.add_argument('--disable-gpu')
options.add_argument('--disable-extensions')
s = Service('C:/Users/username/PycharmProjects/project/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.set_window_size(2048, 1080)
driver.set_window_position(1200, 200, windowHandle='current')
def create_account():
driver.find_element(By.TAG_NAME, 'body').send_keys(Keys.LEFT_CONTROL + 't')
driver.execute_script("window.open('https://profile.w3schools.com/signup?redirect_url=https%3A%2F%2Fspaces.w3schools.com%2Fcreatespace%2Fstepone');")
WebDriverWait(driver, 20).until(ec.element_to_be_clickable(
(By.NAME, 'email'))).click().send_keys('example#gmail.com')
if __name__ == '__main__':
create_account()
driver.delete_all_cookies()
driver.quit()
I am having some trouble trying to automate some web inputs, but first i need to click some buttons and i cannot do it. I've tried a lot of stuff but i cannot complete it :'(
webpage: https://vacunacovid.catsalut.gencat.cat/
I cannot go past the image the code i have:
rom selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
# browser config
options = webdriver.ChromeOptions()
options.add_argument('--start-maximized')
options.add_argument('--disable-extensions')
driver_path= "/C:/chromedriver_linux64/chromedriver" # este es el driver del navegador https://chromedriver.chromium.org/
driver = webdriver.Chrome(driver_path,chrome_options=options)
#starting screen in optimal position
driver.set_window_position(2000,0)
driver.maximize_window()
time.sleep(1)
#getting website
driver.get("https://vacunacovid.catsalut.gencat.cat/")
#go to the element if its clickable
WebDriverWait(driver, 5)\
.until(EC.element_to_be_clickable((By.XPATH, BLABLABLA)))\
.click()
it doesn't find anything throwugh this #shadow (open) , how can i do it?
https://i.stack.imgur.com/it2nQ.png
It is under Shadow-dom #shadow-root (open) So you have not mentioned exactly which button you want to click, so I'm clicking on the first button Demana o modifica cita
#adding some wait for application to load properly
sleep(5)
You just take the JS path of the desire element as below and return the element for that
press F12->Element Tab -> right click(on the element)->copy JS path
javascript = 'return document.querySelector("body > vaccinapp-app").shadowRoot.querySelector("#pages > vaccinapp-shell").shadowRoot.querySelector("#main-shell-content > appointment-shell").shadowRoot.querySelector("#appointment-shell-content > appointment-onboarding").shadowRoot.querySelector("#dismiss-btn").shadowRoot.querySelector("#button")'
By using execute_script will access the element under shadow-root (open)
element = driver.execute_script(javascript)
element.click()
code
options = webdriver.ChromeOptions()
options.add_argument('--start-maximized')
options.add_argument('--disable-extensions')
driver_path= "/C:/chromedriver_linux64/chromedriver"
driver = webdriver.Chrome(driver_path,chrome_options=options)
#starting screen in optimal position
driver.set_window_position(2000,0)
driver.maximize_window()
time.sleep(1)
#getting website
driver.get("https://vacunacovid.catsalut.gencat.cat/")
sleep(5)
javascript = 'return document.querySelector("body > vaccinapp-app").shadowRoot.querySelector("#pages > vaccinapp-shell").shadowRoot.querySelector("#main-shell-content > appointment-shell").shadowRoot.querySelector("#appointment-shell-content > appointment-onboarding").shadowRoot.querySelector("#dismiss-btn").shadowRoot.querySelector("#button")'
element = driver.execute_script(javascript)
element.click()
For reference check here