Hello there I try to scrape this website - https://dom.ria.com/uk/realtors/ and I get a popup message below about cookies when I press accept it dismiss and I can access phone numbers but When I try to press this button using selenium I get erro ElementNotInterectable.
Here is my code to handle it:
cookies = driver.find_element(By.XPATH, "//label[#class='button large']")
driver.implicitly_wait(20)
cookies.click()
I tried to use driver.implicitly_wait() but it still doesn't work.
How can I fix this?
Your xpath matches two elements on the page. In this case, selenium simply grabs the first element, which does not happen to be the one that you want. Try something like this:
cookies = driver.find_elements(By.XPATH, "//label[#class='button large']")
# wait if you have to
cookies[1].click()
A reliable way of accepting cookies on that page would be:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
### other imports, setting up selenium, etc ##
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(browser, 20)
url = 'https://dom.ria.com/uk/realtors/'
browser.get(url)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, '//div[#class="nowrap gdpr_settings_not_checked"]/label'))).click()
print('accepted cookies')
except Exception as e:
print('no cookie button!')
Selenium docs can be found at https://www.selenium.dev/documentation/
Related
I am trying to get the webdriver to click a button on the site random.org The button is a generator that generates a random integer between 1 and 100. It looks like this:
After inspecting the webpage, I found the corresponding element on the webpage looks something like this:
It is inside an iframe and someone suggested that I should first switch over to that iframe to locate the element, so I incorporated that in my code but I am constantly getting NoSuchElementException error. I have attached my code and the error measage below for your reference. I can't understand why it cannot locate the button element despite referencing the ID, which is supposed to unique in the entire document.
The code:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Edge()
driver.get("https://www.random.org/")
driver.implicitly_wait(15)
driver.switch_to.frame(driver.find_element(By.TAG_NAME, "iframe"))
button = driver.find_element(By.CSS_SELECTOR, "input[id='hnbzsqjufzxezy-button']")
button.click()
The error message:
Make sure that there are no more Iframes on the page. If there are a few an not only one do this:
iframes = driver.find_elements(By.CSS, 'iframe')
// try switching to each iframe:
driver.switch_to.frame(iframes[0])
driver.switch_to.frame(iframes[1])
You can't find the button because its name contain random letters. Every time you will refresh the page you can see that the name value will change. So, do this:
button = driver.findElement(By.CSS, 'input[type="button"][value="Generate"]')
button.click()
There are several issues with your code:
First you need to close the cookies banner
The locator of the button is wrong. It's id is dynamic.
You need to use WebDriverWait to wait for elements clickability.
The following code works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = 'https://www.random.org/'
driver.get(url)
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[onclick*='all']"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME, "iframe")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[id*='button']"))).click()
I need to accept cookies on a specific website but I keep getting the NoSuchElementException. This is the code for entering the website:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
chrome_options = Options()
driver = webdriver.Chrome(executable_path='./chromedriver', options=chrome_options)
page_url = 'https://www.boerse.de/historische-kurse/Erdgaspreis/XD0002745517'
driver.get(page_url)
time.sleep(10)
I tried accepting the cookie button using the following:
driver.find_element_by_class_name('message-component message-button no-children focusable button global-font sp_choice_type_11 last-focusable-el').click()
driver.find_element_by_xpath('//*[#id="notice"]').click()
driver.find_element_by_xpath('/html/body/div/div[2]/div[4]/div/button').click()
I got the xpaths from copying the xpath and the full xpath from the element while using google chrome.
I am a beginner when it comes to selenium, just wanted to use it for a short workaround. Would appreciate some help.
The button Zustimmen is in iframe so first you'd have to switch to the respective iframe and then you can interact with that button.
Code:
driver.maximize_window()
page_url = 'https://www.boerse.de/historische-kurse/Erdgaspreis/XD0002745517'
driver.get(page_url)
wait = WebDriverWait(driver, 30)
try:
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[starts-with(#id,'sp_message_iframe')]")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Zustimmen']"))).click()
print('Clicked successfully')
except:
print('Could not click')
pass
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I've got a similar problem with a page. I've tried to address the iframe and the "accept all" button with selenium (as suggested by #cruisebandey above).
However, the pop-up on this page seems to work differently:
https://www.kreiszeitung-wochenblatt.de/hanstedt/c-panorama/mega-faslams-umzug-in-hanstedt_a270327
This is what I've tried:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(executable_path="C:\\Users\\***\\chromedriver.exe")
driver.maximize_window()
try:
driver.get("https://www.kreiszeitung-wochenblatt.de/hanstedt/c-panorama/mega-faslams-umzug-in-hanstedt_a270327")
except:
print('Site not found')
wait = WebDriverWait(driver,10)
try:
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,'/html/body/iframe')))
except:
print('paywall-layover not found')
try:
cookie = wait.until(EC.element_to_be_clickable((By.XPATH,'//*[#id="consentDialog"]/div[3]/div/div[2]/div/div[2]/div/div[1]/div[2]/div')))
cookie.click()
except:
print('Button to accept all not found')
im quite noob in python and right now building up a web scraper in Selenium that would take all URL's for products in the clicked 'tab' on web page. But my code take the URL's from the first 'tab'. Code below. Thank you guys. Im starting to be kind of frustrated lol.
Screenshot
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from lxml import html
PATH = 'C:\Program Files (x86)\chromedriver.exe'
driver = webdriver.Chrome(PATH)
url = 'https://www.alza.sk/vypredaj-akcia-zlava/e0.htm'
driver.get(url)
driver.find_element_by_xpath('//*[#id="tabs"]/ul/li[2]').click()
links = []
try:
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, 'blockFilter')))
link = driver.find_elements_by_xpath("//a[#class='name browsinglink impression-binded']")
for i in link:
links.append(i.get_attribute('href'))
finally:
driver.quit()
print(links)
To select current tab:
current_tab = driver.current_window_handle
To switch between tabs:
driver.switch_to_window(driver.window_handles[1])
driver.switch_to.window(driver.window_handles[-1])
Assuming you have the new tab url as TAB_URL, you should try:
from selenium.webdriver.common.action_chains import ActionChains
action = ActionChains(driver)
action.key_down(Keys.CONTROL).click(TAB_URL).key_up(Keys.CONTROL).perform()
Also, apparently the li doesn't have a click event, are you sure this element you are getting '//*[#id="tabs"]/ul/li[2]' has the aria-selected property set to true or any of these classes: ui-tabs-active ui-state-active?
If not, you should call click on the a tag inside this li.
Then you should increase the timeout parameter of your WebDriverWait to guarantee that the div is loaded.
I've been trying to make a Python script to login into a certain website, navigate through the menu, fill out a form and save the file it generates to a folder.
I've been using Selenium trying to make the website fully load so i can find the elements for the login, but i'm being unsucessful, maybe because the website does a lot of JavaScript content before it fully loads, but i can't make it fully load and show me the data i want.
I tried Robobrowser, Selenium, Requests and BeautifulSoup to get it done.
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
url = "https://directa.natal.rn.gov.br/"
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
driver.get(url)
html = driver.execute_script("return document.documentElement.outerHTML")
sel_soup = BeautifulSoup(html, 'html.parser')
senha = driver.find_element_by_xpath('//*[#id="senha"]')
senha.send_keys("123")
I expected to have filled the password (senha) field with "123" but i can't even find the element.
It seems like what's needed here is a little bit of a scroll, wait and switch, incase the login fields just aren't ready for input :) The below should work, whereby we actually scroll to the element, having switch to the iframe, before we interact with the rest of the login form. You're able to adjust the delay from 5 seconds to anything of your preference.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
""" Variables """
url = "https://directa.natal.rn.gov.br/"
delay = 5 # seconds
""" Initiate driver """
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
""" Go to url """
driver.get(url)
""" Iframe switch """
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
""" Attempt to get all our elements """
try:
""" Username """
usuario = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'usuario')))
""" Password """
senha = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'senha')))
print("All elements located!")
except TimeoutException:
print("Loading took too much time!")
exit(0)
"""Scroll to our element """
driver.execute_script("arguments[0].scrollIntoView();", usuario)
""" Input data into our fields """
usuario.send_keys("username")
senha.send_keys("password")
""" Locate our login element """
login = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'acessar')))
""" Click Login """
login.click()
To send the character sequence 123 to the password (senha) field, as the the desired elements are within a <frame> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://directa.natal.rn.gov.br/")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.input[name='usuario']"))).send_keys("Tads")
driver.find_element_by_css_selector("input.input[name='senha']").send_keys("123")
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe
I am trying to scrape a website. Where in I have to press a link. for this purpose, I am using selenium library with chrome drive.
from selenium import webdriver
url = 'https://sjobs.brassring.com/TGnewUI/Search/Home/Home?partnerid=25222&siteid=5011&noback=1&fromSM=true#Applications'
browser = webdriver.Chrome()
browser.get(url)
time.sleep(3)
link = browser.find_element_by_link_text("Don't have an account yet?")
link.click()
But it is not working. Any ideas why it is not working? Is there a workaround?
You can get it done in several ways. Here is one of such. I've used driver.execute_script() command to force the clicking. You should not go for hardcoded delay as they are very inconsistent.
Modified script:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://sjobs.brassring.com/TGnewUI/Search/Home/Home?partnerid=25222&siteid=5011&noback=1&fromSM=true#Applications'
driver = webdriver.Chrome()
driver.get(url)
item = wait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[ng-click='newAccntScreen()']")))
driver.execute_script("arguments[0].click();",item)