How to scroll a web page down until specific button to click? - python

I am working with Python and Selenium but I got stuck scraping a page trying to scroll down until selenium finds the classic "Show more" button (that seems to be hidden).
What I have done 'til now is the following:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
element = browser.find_element_by_xpath('//*[#id="ember371"]/button') # you can use ANY way to locate element
coordinates = element.location_once_scrolled_into_view # returns dict of X, Y coordinates
browser.execute_script('window.scrollTo({}, {});'.format(coordinates['x'], coordinates['y']))
However, whatever I try to do Python throws me several kind of errors, for example the below.
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="ember371"]/button"}
(Session info: chrome=80.0.3987.122)
Could it be due to the fact that the button is hidden?
Below is the HTML code for the button I am looking for and that I would like to click on.
<button class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid" aria-controls="skill-categories-expanded" aria-expanded="false" data-control-name="skill_details" data-ember-action="" data-ember-action-2182="2182">
<span aria-hidden="true">
Show more
</span>
<span class="visually-hidden">
BLA BLA BLA
</span>
<li-icon aria-hidden="true" type="chevron-down-icon" class="pv-skills-section__chevron-icon" size="small"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" data-supported-dps="16x16" fill="currentColor" width="16" height="16" focusable="false">
<path d="M8 9l5.93-4L15 6.54l-6.15 4.2a1.5 1.5 0 01-1.69 0L1 6.54 2.07 5z"></path>
</svg></li-icon>
</button>
#################################################################################
It worked with the following:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# scroll down smoothly
scheight = .1
while scheight < 1.0:
browser.execute_script("window.scrollTo(0, document.body.scrollHeight*%s);" % scheight)
scheight += .1
time.sleep(1)
try:
browser.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.XPATH, "//button[#aria-controls='skill-categories-expanded' and #data-control-name='skill_details']/span[normalize-space()='Show more']"))))
browser.execute_script("arguments[0].click();", WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#aria-controls='skill-categories-expanded' and #data-control-name='skill_details']/span[normalize-space()='Show more']"))))
except:
print('NO')

The element Show more within linkedin website is EmberJS enabled element. So to scrollIntoView the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button[aria-controls='skill-categories-expanded'][data-control-name='skill_details']>span"))))
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-controls='skill-categories-expanded'][data-control-name='skill_details']>span"))))
Using XPATH:
driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//button[#aria-controls='skill-categories-expanded' and #data-control-name='skill_details']/span[normalize-space()='Show more']"))))
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#aria-controls='skill-categories-expanded' and #data-control-name='skill_details']/span[normalize-space()='Show more']"))))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can try using a while loop here, check for the existence of the button as the condition, and put the browser.execute_script in the body. In this way, you would start by checking if the button exists, if not then scroll for 200 px or so (just make sure you don't scroll past the button) until the button appears.

You can first try to find if the "show more" button is enabled by using the isEnabled() method. This method returns “true” value if the specified web element is enabled on the web page otherwise returns “false” value if the web element is disabled on the web page.
If isEnabled() method returns False, you can then try scrolling down the page till the element is visible and use the isEnabled() method again.
Also, try replacing id with XPath. It may work.

Related

How to click on the element with svg tag containing the attribute focusable="false" using Selenium and Python

I'm quite new in Selenium in Python. This time I have a problem with clicking on an element (This is a closing button).
<div class="CloseButton_Background__27knc">
<svg class="MuiSvgIcon-root CloseButton_Icon__7wksG" focusable="false" viewBox="0 0 24 24"
aria-hidden="true" style="font-size: 36px;">
<path d="M19 6.41L17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59
19 19 17.59 13.41 12 19 6.41z"></path></svg></div>
What I do first is: I'm looking for an element on the site:
closeButton = self.driver.find_elements_by_xpath('//div[#class="CloseButton_Background__27knc"]/*[name()="svg"][#class="MuiSvgIcon-root CloseButton_Icon__7wksG"]')
The element is found on the page. And then I try to call click () method on it and I get this error:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
I'm waiting for the button to be visible.
Please help! Thanks!
You can wait until the element is loaded and then perform the click operation.
wait = WebDriverWait(browser, 10)
close_btn= wait.until(EC.visibility_of_element_located((By.XPATH,'//div[#class="CloseButton_Background__27knc"]/*[name()="svg"][#class="MuiSvgIcon-root CloseButton_Icon__7wksG"]')
close_btn.click()
Moreover, you can also use action chain to click on your element.
driver.implicitly_wait(10)
ActionChains(driver).move_to_element(close_btn).click(close_btn).perform()
ElementNotInteractableException is caused when an element is found, but you can not interact with it.
possible reasons : element is not visible , hidden or behind of another element
closeButton = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.XPATH, '//div[#class="CloseButton_Background__27knc"]/*[name()="svg"][#class="MuiSvgIcon-root CloseButton_Icon__7wksG"]')))
actionChains.move_to_element(closeButton).click().perform()
Note: Dont forget to add below imports to your solution:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as Wait
from selenium.webdriver.common.action_chains import ActionChains
Javascript :
closeButton = self.driver.find_elements_by_xpath('//div[#class="CloseButton_Background__27knc"]/*[name()="svg"][#class="MuiSvgIcon-root CloseButton_Icon__7wksG"]')
driver.execute_script("arguments[0].click();", closeButton)
focusable="false"
As per the documentation in Managing focus in SVG:
The focusable attribute defined by SVG Tiny 1.2 is only implemented in Internet Explorer and Microsoft Edge. Unlike tabindex this attribute has a boolean value, where focusable="true" equals tabindex="0" and focusable="false" makes the element inert. Besides removing SVG links from the tabbing order by way of <a xlink:href="…" focusable="false">, this property also comes in handy to prevent the SVG's root element from unnecessarily receiving focus: <svg focusable="false">.
This usecase
To click on the desired element, it would be better to avoid the <svg> tag and you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class^='CloseButton_Background']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[starts-with(#class, 'CloseButton_Background')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on ElementNotInteractableException in:
Element not Interactable when using the selenium function find element by name but works when using find element by xpath method
ElementNotInteractableException: Message: element not interactable error sending text in search field using Selenium Python
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable when clicking on an element using Selenium Python

Selenium not performing click action

I am trying to scrape a table with multiple pages. The next page is obtained by clicking on the 'Next Page button' (See code snippet).
<a class="botons" id="btn2" href="javascript:void(0)">
Next Page
<i class="fas fa-long-arrow-alt-right"></i>
</a>
Selenium finds the "button" and has no trouble "clicking" via the following code:
btn_next = self.browser.find_element_by_partial_link_text("Next Page")
btn_next.click()
However, the page just refreshes and the table doesn't update to its next page.
Any clues to what's going wrong here?
Edit: table can be found at https://www.proxy-list.download/HTTPS
Edit2:
chrome_options = Options()
chrome_options.add_argument("--enable-javascript")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--headless")
There is one id assigned to that button btn2 and it's unique too.
You should give preference to id over link Text.
That said, Next Page link isn't present in view point.For that first you have to move the focus of driver like this :
wait = WebDriverWait(self.browser,10)
next_page = wait.until(EC.visibility_of_element_located((By.ID, "btn2")))
ActionChains(self.browser).move_to_element(next_page).perform()
next_page.click()
Imports :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
The desired element is an JavaScript enabled element so to locate and click() on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.botons[id^='btn2'] i.fas.fa-long-arrow-alt-right"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='botons' and normalize-space()='Next Page']/i[#class='fas fa-long-arrow-alt-right']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to click javascript:void(0) link with selenium?

I am using selenium and chrome to automate the clicks of a page with python.
I am getting stuck not being able to click on the following href link:
<a href="javascript:void(0)" class="addSuppData-trigger pts" data-target="edit_3-1" style="padding-right:6px;float:right;">
<i class="material-icons black-text tiny-small">edit</i></a>
I have tried using xpath, css, and linktext to no avail.
sample code:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="row-group-1"]/td[1]/div/div[2]/a/i' ))).click()
The goal is to click the pen, then select the item from the drop down.
screen shot of button
The second highlight line is the pen.
html tree
The element seems to be a dynamic element and to click on it you need to induce WebDriverWait for the desired element to be clickable and you can use either of the following solutions:
Using CSS_SELECTOR:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.addSuppData-trigger.pts[data-target^='edit_']>i.material-icons.black-text.tiny-small"))).click()
Using XPATH:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='addSuppData-trigger pts' and starts-with(#data-target, 'edit_')]/i[#class='material-icons black-text tiny-small' and contains(., 'edit')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to click the button as per the html using Selenium and Python

I am learning Selenium and trying to click the GO button:
https://speedtest.telstra.com/
<button class="button background-primary-hover text-primary" aria-label="start your speedtest">
<span style="font-size: unset; overflow-wrap: unset;">GO</span></button>
What are all possible Selenium methods to get that button clicked,
elem = driver.find_element_by_....???
I also would like to see what I found, so should print(elem.text) then be used?
As per the website https://speedtest.telstra.com/ the desired element is within an <iframe> so you need to induce WebDriverWait to switch to the <iframe> and then look out for the element and you can use the following solution:
Using XPATH:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#class='speed-test' and #src='//telstra-nbn.speedtestcustom.com']")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='button background-primary-hover text-primary']/span[contains(.,'GO')]"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"//iframe.speed-test[src*='speedtestcustom']")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.button.background-primary-hover.text-primary[aria-label='start your speedtest']>span"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You have to use xpath, there is xpath helper tool for chrome. You can install it.
button = driver.find_element_by_xpath("your xpath")
button.click()
Try this:
browser.find_element_by_class_name("button background-primary-hover text-primary").click()
Since it will select the element and click it.

Button"+" is not being clicked in selenium

Source code:
<input type="button" value="+" id="hour_add" class="ui-button ui-widget ui-state-default ui-corner-all" role="button" aria-disabled="false">
My code:
driver.find_element_by_xpath("//input[contains(#id, 'hour_add')]").click();
This button is not clicked.
There could be multiple reasons for the problem and there is certainly not enough information to answer, but here are the possible reasons:
there are multiple elements matching the XPath expression and a wrong element is clicked
you might need to move to the element and click:
from selenium.webdriver.common.action_chains import ActionChains
ActionChains(driver).move_to_element(element).click(element).perform()
you might need to scroll into view of the element:
driver.execute_script("arguments[0].scrollIntoView();", element)
you might need to click via javascript:
driver.execute_script("arguments[0].click();", element)
you might need to wait for element to be clickable:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.ID, "hour_add"))
)
element.click()
sometimes even maximizing the browser window can help:
driver.maximize_window()
When you have Id available for the element you want to click on, then just use find_element_by_id.
driver.find_element_by_id('hour_add').click()

Categories

Resources