I am experimenting with Selenium and I have a code that load some URL and click on the button, so far my code looks like this:
import time
from selenium import webdriver
driver = webdriver.Chrome()
url = "https://example.org/"
driver.get(url)
button = driver.find_element_by_class_name("cool-button")
button.click()
driver.quit()
This code works as expected, but what am trying to achieve is this:
when this script finished it should start again, and again, let's say like infinite loop, so click on the button, close, open URL again, close, and so on:
I am new to Python and I try something like this:
import time
from selenium import webdriver
while True:
driver = webdriver.Chrome()
url = "https://example.org/"
driver.get(url)
button = driver.find_element_by_class_name("cool-button")
button.click()
driver.quit()
Can anybody try to help me with this?
We do the same thing here where we test when we want to attempt a "retry" of a test. One thing to note in your code above; you give no wait time for the "cool-button" to appear, become clickable or it's presence known; that will often lead to a failing result. I added support for that; as well as encompassed the entire code in an keyboard interrupted while loop.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
try:
while True:
driver = webdriver.Chrome()
url = "https://example.org/"
driver.get(url)
try:
button = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CLASS_NAME, "cool-button")))
button.click()
except:
print('could not find the "cool" button')
pass
driver.quit()
print("Press CTRL+C to escape!")
time.sleep(3)
except KeyboardInterrupt:
quit()
I'd also suggest this youtube as a very good resource on the topic of selenium:
https://www.youtube.com/watch?v=b5jt2bhSeXs
Related
It closes when click runs.
and ı wanna just click button.
ı wanna do click that xpath. but its doesnt work. it close without doesnt click. so it open edge. waiting waiting and close. it didint run my click command. and there a error. its "webdriver object has no attribute find_element_by_xpath"
from selenium import webdriver
import time
driver = webdriver.Edge()
driver.get("https://www.instagram.com/")
time.sleep(7)
log_in = driver.find_element_by_xpath("//*[#id='mount_0_0_+h']/div/div/div/div[1]/div/div/div/div[1]/section/main/article/div[2]/div[2]/div/p/a/span")
log_in.click()
driver.close()
I think this is what you want:
# Needed libs
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
#Open the browser
driver = webdriver.Edge()
driver.get("https://www.instagram.com/")
# Wait and click for allow cookies button
allow_cookies_button = date = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, f"//button[text()='Only allow essential cookies']")))
allow_cookies_button.click()
# Wait and click for Sign in button
signup_button = date = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, f"//span[text()='Sign up']")))
signup_button.click()
# This time.sleep is to allow you to see what happened, you can delete it
time.sleep(5)
# Close the browser
driver.close()
Why your code did not work? Because as you console told you "find_element_by_xpath" is not a valid method/attribute for driver.
Advises for automation/Scraping:
Wait for the element you will use (it does not matter if you want to write, click or whatever, make sure the element is on the screen)
Try to use xpaths (Or any selector) that are understandable, you will reuse them in the future. What do you understand better, //span[text()='Sign up'] or //*[#id='mount_0_0_+h']/div/div/div/div[1]/div/div/div/div[1]/section/main/article/div[2]/div[2]/div/p/a/span? Good selectors will save a lot of time in the future
Hello there I try to scrape this website - https://dom.ria.com/uk/realtors/ and I get a popup message below about cookies when I press accept it dismiss and I can access phone numbers but When I try to press this button using selenium I get erro ElementNotInterectable.
Here is my code to handle it:
cookies = driver.find_element(By.XPATH, "//label[#class='button large']")
driver.implicitly_wait(20)
cookies.click()
I tried to use driver.implicitly_wait() but it still doesn't work.
How can I fix this?
Your xpath matches two elements on the page. In this case, selenium simply grabs the first element, which does not happen to be the one that you want. Try something like this:
cookies = driver.find_elements(By.XPATH, "//label[#class='button large']")
# wait if you have to
cookies[1].click()
A reliable way of accepting cookies on that page would be:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
### other imports, setting up selenium, etc ##
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(browser, 20)
url = 'https://dom.ria.com/uk/realtors/'
browser.get(url)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, '//div[#class="nowrap gdpr_settings_not_checked"]/label'))).click()
print('accepted cookies')
except Exception as e:
print('no cookie button!')
Selenium docs can be found at https://www.selenium.dev/documentation/
I have the following code as part of a data download function in selenium / chrome driver
driver.find_element_by_class_name("mt-n1").click()
driver.implicitly_wait(5)
When I 'step through' the code it works fine, but in normal execution it fails to action this link on the web page, therefore does not open the next webpage as required by the function
I have inspected the web page and identified the xpath, so I amended the code as follows
driver.find_element_by_xpath("//div[#class = 'mt-n1 flex-auto']").click()
driver.implicitly_wait(5)
This produces the same result, i.e. only works if I step through the code one line at a time
Hopefully just something obvious I'm not seeing?
The full code is given below for ref.
# Download file from github function
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
workDir = r"/C:\Users\Oem\Desktop"
### Set the download directory
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_experimental_option("prefs", {"download.default_directory": workDir})
### Open a web-browser for automated control
driver = webdriver.Chrome(options=chromeOptions)
driver.get("https://github.com")
driver.fullscreen_window()
time.sleep(5)
assert "GitHub" in driver.title
elem = driver.find_element_by_class_name("header-search-input")
elem.clear()
elem.send_keys("acca-edx/python-practical-automation")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
### Go to the first result.
# Find the results on the page using "mt-n1" and clicking on it.
# Wait for the page to complete.
driver.find_element_by_class_name("mt-n1").click()
driver.implicitly_wait(5)
driver.find_element_by_xpath("//a[#title='detection.xlsx']").click()
driver.implicitly_wait(5)
# Find the raw-url to download the file we are after.
driver.find_element_by_id('raw-url').click()
time.sleep(10)
driver.close()
if your code works while going step by step through, it might be that web page needs time to be rendered. try this, it will wait 50 seconds or until the element presence has been located.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome(options=chromeOptions)
try:
element = WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.CLASS_NAME, 'mt-n1')))
except TimeoutException as e:
# do something if element is not found
I can't figure out how to get around this in Python. The email pop-up is preventing Selenium from clicking on one of the footer links because the pop-up blocks the view of it. I ideally would like to click the "X" and not enter an email.
I've tried using what was in the Selenium documentation about prompts but none of it worked or perhaps I implemented it incorrectly. I tried some of what I already found in stack overflow, which you can see in the commented out code, but kept getting all kinds of errors.
import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup
from urllib.request import urlopen as uReq
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.support import ui
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
url = "https://www.standardmedia.co.ke/"
driver = webdriver.Chrome()
driver.get(url)
html = driver.page_source.encode('utf-8')
page_num = 0
##options = Firefox_options();
##options.addPreference("dom.disable_beforeunload", true)
##driver = webdriver.Firefox(options);
#click on the headings in the footer
for i in range (0,1):
footer = driver.find_elements_by_css_selector('#top-footer li')[i]
## if(driver.switch_to.alert != null):
## alert = driver.switch_to.alert
## alert.dismiss()
try:
WebDriverWait(driver, 10).until(EC.alert_is_present())
alert = driver.switch_to_alert()
alert.dismiss()
print("Alert dismissed.")
except TimeoutException:
print("No alert.")
footer.click()
print("alert dismissed")
page_num += 1
subheadings = driver.find_elements_by_css_selector('.title a')
len(subheadings)
The most recent error for a Firefox web driver was "No connection could be made because the target machine actively refused it."
WebDriver Alert class is designed to work with JavaScript Alerts to wit:
Window.alert()
Window.confirm()
Window.prompt()
At the page you're trying to test this modal popup is a normal <div>
therefore you will not be able to use Alert class, you will have to normally locate close button using find_element() function and click it.
Open the page:
driver.get("https://www.standardmedia.co.ke/")
Wait for the popup to appear
popup = WebDriverWait(driver, 10).until(expected_conditions.presence_of_element_located((By.CLASS_NAME, "mc-closeModal")))
Click the close button
popup.click()
Aside from the selenium connection issue, to address the main question, it looks like you are not calling the switch_to_alert function. Once you get selenium connecting, try the following:
alert = driver.switch_to_alert()
alert.dismiss()
Also note, your wait times are in seconds which seem pretty high/long in your code.
I've written some code in python in combination with selenium to parse the different questions from quora.com. My scraper is doing it's job at this moment. The thing is I've used here hardcoded delay for the scraper to work, even when Explicit Wait has already been defined. As the page is an infinite scrolling one, i tried to make the scrolling process to a limited number. Now, I have got two questions:
Why wait.until(EC.staleness_of(page)) is not working within my scraper. It is commented out now.
If i use something else instead of page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link"))) the scraper throws an error: can't focus element.
Btw, I do not wish to go for page = driver.find_element_by_tag_name('body') this option.
Here is what I've written so far:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.quora.com/topic/C-programming-language")
wait = WebDriverWait(driver, 10)
page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link")))
for scroll in range(10):
page.send_keys(Keys.PAGE_DOWN)
time.sleep(2)
# wait.until(EC.staleness_of(page))
for item in wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "rendered_qtext"))):
print(item.text)
driver.quit()
You can try below code to get as much XHR as possible and then parse the page:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome()
driver.get("https://www.quora.com/topic/C-programming-language")
wait = WebDriverWait(driver, 10)
page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link")))
links_counter = len(wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "question_link"))))
while True:
page.send_keys(Keys.END)
try:
wait.until(lambda driver: len(driver.find_elements_by_class_name("question_link")) > links_counter)
links_counter = len(driver.find_elements_by_class_name("question_link"))
except TimeoutException:
break
for item in wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "rendered_qtext"))):
print(item.text)
driver.quit()
Here we scroll page down and wait up to 10 seconds for more links to be loaded or break the while loop if the number of links remains the same
As for your questions:
wait.until(EC.staleness_of(page)) is not working because when you scroll page down you don't get the new DOM - you just make XHR which adds more links into existed DOM, so the first link (page) will not be stale in this case
(I'm not quite confident about this, but...) I guess you can send keys only to nodes that can be focused (user can set focus manually), e.g. links, input fields, textareas, buttons..., but not content division (div), paragraphs (p), etc