I am trying to scrape a table with multiple pages. The next page is obtained by clicking on the 'Next Page button' (See code snippet).
<a class="botons" id="btn2" href="javascript:void(0)">
Next Page
<i class="fas fa-long-arrow-alt-right"></i>
</a>
Selenium finds the "button" and has no trouble "clicking" via the following code:
btn_next = self.browser.find_element_by_partial_link_text("Next Page")
btn_next.click()
However, the page just refreshes and the table doesn't update to its next page.
Any clues to what's going wrong here?
Edit: table can be found at https://www.proxy-list.download/HTTPS
Edit2:
chrome_options = Options()
chrome_options.add_argument("--enable-javascript")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--headless")
There is one id assigned to that button btn2 and it's unique too.
You should give preference to id over link Text.
That said, Next Page link isn't present in view point.For that first you have to move the focus of driver like this :
wait = WebDriverWait(self.browser,10)
next_page = wait.until(EC.visibility_of_element_located((By.ID, "btn2")))
ActionChains(self.browser).move_to_element(next_page).perform()
next_page.click()
Imports :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
The desired element is an JavaScript enabled element so to locate and click() on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.botons[id^='btn2'] i.fas.fa-long-arrow-alt-right"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='botons' and normalize-space()='Next Page']/i[#class='fas fa-long-arrow-alt-right']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Related
I'm having trouble performing a click operation on a website. I'm getting a error message NoSuchElementException, but I'm not sure why because I got the class name from the site.
What am I missing?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
s = Service('C:/Program Files (x86)/chromedriver.exe')
chromeOptions = Options()
chromeOptions.headless = False
driver = webdriver.Chrome(service=s, options=chromeOptions)
list_data = []
def initialize_browser():
driver.get("https://virtualracingschool.appspot.com/#/Home")
print("starting_Driver")
click_button = driver.find_element(By.CLASS_NAME, "white-text")
driver.implicitly_wait(15)
click_button.click()
initialize_browser()
Site & Code:
I tried referencing some documents from the selenium site and it mentions for a format
<p class="content">Site content goes here.</p>`
write the code:
content = driver.find_element(By.CLASS_NAME, 'content')`
I felt like I did this properly but my site has
<a class="white-text" style="" ...>
<span>Login</span>
</a>
format. Is the <a> and "style" element hindering my code?
Use the below XPath expression:
//span[text()='Login']
Your code should look like this:
click_button = driver.find_element(By.XPATH, "//span[text()='Login']")
Below is the inspect element by this XPath for your reference:
As per your code trials:
click_button = driver.find_element(By.CLASS_NAME, "white-text")
By.CLASS_NAME, "white-text" identifies four elements within the HTML DOM:
Hence you see the see the error.
Solution
To click on the clickable element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get('https://virtualracingschool.appspot.com/#/Home')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.white-text > span"))).click()
Using XPATH:
driver.get('https://virtualracingschool.appspot.com/#/Home')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='white-text']//span"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser snapshot:
I'm having trouble performing a click operation on a website. I'm getting a error message NoSuchElementException, but I'm not sure why because I got the class name from the site.
What am I missing?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
s = Service('C:/Program Files (x86)/chromedriver.exe')
chromeOptions = Options()
chromeOptions.headless = False
driver = webdriver.Chrome(service=s, options=chromeOptions)
list_data = []
def initialize_browser():
driver.get("https://virtualracingschool.appspot.com/#/Home")
print("starting_Driver")
click_button = driver.find_element(By.CLASS_NAME, "white-text")
driver.implicitly_wait(15)
click_button.click()
initialize_browser()
Site & Code:
I tried referencing some documents from the selenium site and it mentions for a format
<p class="content">Site content goes here.</p>`
write the code:
content = driver.find_element(By.CLASS_NAME, 'content')`
I felt like I did this properly but my site has
<a class="white-text" style="" ...>
<span>Login</span>
</a>
format. Is the <a> and "style" element hindering my code?
Use the below XPath expression:
//span[text()='Login']
Your code should look like this:
click_button = driver.find_element(By.XPATH, "//span[text()='Login']")
Below is the inspect element by this XPath for your reference:
As per your code trials:
click_button = driver.find_element(By.CLASS_NAME, "white-text")
By.CLASS_NAME, "white-text" identifies four elements within the HTML DOM:
Hence you see the see the error.
Solution
To click on the clickable element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get('https://virtualracingschool.appspot.com/#/Home')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.white-text > span"))).click()
Using XPATH:
driver.get('https://virtualracingschool.appspot.com/#/Home')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='white-text']//span"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser snapshot:
I am using Selenium with Python to webscrape a page which includes JavaScript.
The racecourse result tabs towards the top of the page eg "Ludlow","Dundalk" are manually clickable but do not have any obvious hyperlinks attached to them.
...
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path='C:/A38/chromedriver_win32/chromedriver.exe')
driver.implicitly_wait(30)
driver.maximize_window()
# Navigate to the application home page
driver.get("https://www.sportinglife.com/racing/results/2020-11-23")
...
This works so far. I have used BeautifulSoup to find the label names of the NewGenericTabs eg "Ludlow","Dundalk" etc. However, the following code, to attempt to automate clicking a tab, times out each time.
WebDriverWait(driver, 60).until(EC.element_to_be_clickable((By.LINK_TEXT, "Ludlow"))).click()
Would welcome any help.
The WebElement aren't <a> tags but <span> tags so By.LINK_TEXT wouldn't work.
To click on the desired elements you can use either of the following xpath based Locator Strategies:
Ludlow:
driver.get("https://www.sportinglife.com/racing/results/2020-11-23")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#mode='primary']"))).click()
driver.find_element_by_xpath("//span[#data-test-id='generic-tab' and text()='Ludlow']").click()
Dundalk:
driver.get("https://www.sportinglife.com/racing/results/2020-11-23")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#mode='primary']"))).click()
driver.find_element_by_xpath("//span[#data-test-id='generic-tab' and text()='Dundalk']").click()
Ideally, to click on the elements you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following xpath based Locator Strategies:
Ludlow:
driver.get("https://www.sportinglife.com/racing/results/2020-11-23")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#mode='primary']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#data-test-id='generic-tab' and text()='Ludlow']"))).click()
Dundalk:
driver.get("https://www.sportinglife.com/racing/results/2020-11-23")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#mode='primary']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#data-test-id='generic-tab' and text()='Dundalk']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
So the webpage has a button that after clicking will add an element to the webpage, in which I can't find using selenium
Some imaginary code as follows to explain the problem I experience:
from selenium import webdriver
d = webdriver.Chrome()
#Go to my target website
d.get("https://some_website_url") #ref1
#Okay now loading of the website is done. `d` will not be updated and this is the problem!!
#Click my target button and an element with id="SecretButton" is loaded.
d.find_element_by_css_selector("#secretlyupdatethewebpage").click()
#Find #SecretButton but to no avail.
#It can be found in the html panel of Chrome Developer Tools
#but cannot be found in the webdriver `d`, as `d` won't be
#updated after #ref1
d.find_element_by_css_selector("#SecretButton").click()
How can I find that #SecretButton?
To find and invoke click() on the secret button you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using ID:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "SecretButton"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#SecretButton"))).click()
Using XPATH:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[#id='SecretButton']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am new to the world of python and I am trying to select a couple of options on the following website and then click the search button to update results. However, I cannot get the button to respond.
I tried using search button.click() and .submit() and I have tried to implicitly wait. I have also used the code below to wait until the button is clickable. When executing the code, it highlights the button but doesn't seem to release the click; almost like a half click.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Safari()
driver.get('https://leasing.com/personal/car-leasing/')
element = driver.find_element_by_id('selUpfront')
select = Select(element)
select.select_by_value("3")
element = driver.find_element_by_id('selMileage')
select = Select(element)
select.select_by_value("8000")
searchbutton = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, "search-button")))
searchbutton.click()
I would expect the search results to be updated with the conditions above.
Seems you were close. To click() on the element you have to induce WebDriverWait for the element to be clickable and you can use either of the following solutions:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#search-button>i.fa.fa-search#search-button-icon"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='search-button']/i[#class='fa fa-search' and #id='search-button-icon']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
There are 2 elements with the search-button you need to use the xpath to locate specific
searchbutton = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='search-button']")))
searchbutton.click()