I'm using Selenium with Chrome driver to scrap pages that contain SVG .
I need a way to make Selenium wait until the svg is completely loaded otherwise I will get some incomplete charts when I scrap.
For the moment the script wait for 10sec before it start scrapping but that's is a lot for scraping 20000 pages .
def page_loaded(driver):
path = "//*[local-name() = 'svg']"
time.sleep(10)
return driver.find_element_by_xpath(path)
wait = WebDriverWait(self.driver, 10)
wait.until(page_loaded)
is there any efficient way to check if the SVG is loaded before starting to scrap?
An example from Selenium documentation:
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, 'someid')))
So in your case it should be :
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(self.driver, 10)
element = wait.until(EC.presence_of_element_located((By.XPATH, path)))
Here 10 in WebDriverWait(driver, 10) is the maximum seconds of wait. ie it waits until 10 or condition whichever is first.
Some common conditions that are frequently of use when automating web browsers:
title_is title_contains
presence_of_element_located
visibility_of_element_located visibility_of
presence_of_all_elements_located
text_to_be_present_in_element
text_to_be_present_in_element_value
etc.
More available here.
Also here's the documentation for expected conditions support.
Another way you can tackle this is write your on method like:
def find_svg(driver):
element = driver.find_element_by_xpath(path)
if element:
return element
else:
return False
And then call Webdriver wait like:
element = WebDriverWait(driver, max_secs).until(find_svg)
Related
I try to webscrape linkedin sales navigator but the page loads other elements only if i scroll down and wait a bit. I tried to execute this a following way but it just added me 3 more elements so I get 6 out of 25.
profile_url = "https://www.linkedin.com/sales/search/people?query=(recentSearchParam%3A(doLogHistory%3Atrue)%2Cfilters%3AList((type%3AREGION%2Cvalues%3AList((id%3A102393603%2Ctext%3AAsia%2CselectionType%3AINCLUDED)))))&sessionId=fvwzsZ8CTAKdGri5T5mYZw%3D%3D"
driver.get(profile_url)
time.sleep(20)
action = webdriver.ActionChains(driver)
to_scroll = driver.find_element(By.XPATH, "//button[#id='ember105']")
to_scroll_up = driver.find_element(By.XPATH, "//button[#id='ember141']")
action.move_to_element(to_scroll)
action.perform()
time.sleep(3)
action.move_to_element(to_scroll_up)
action.perform()
time.sleep(3)
action.move_to_element(to_scroll)
action.perform()
time.sleep(3)
How to solve it?
You can do this by using Explicit Wait. As it will make sure the element is visible on UI. You can identify an element that gets loaded after the page is completely loaded and replace XPATH_of_element with its XPATH. Also, add this code before extracting the data so it works properly.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 15).until(EC.visibility_of_element_located((By.XPATH, 'XPATH_of_element')))
You can visit this blog by Shreya Bose for more information.
So I'm trying to submit a form but something is either preventing me from accessing the box or I'm using a wrong element but I think I'm using the correct one.
Here is my code:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path = 'mypath/chromedriver.exe')
driver.maximize_window()
#driver.implicitly_wait(50)
driver.get("https://ai.fmcsa.dot.gov/SMS")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Close']"))).click()
wait = WebDriverWait(driver, 20)
driver.find_element_by_xpath('//*[#id="home-body"]/div[1]/div/div[1]/form/label').click()
driver.find_element_by_xpath('//*[#id="home-body"]/div[1]/div/div[1]/form/label').send_keys('1818437')
driver.find_element_by_xpath('/html/body/div[3]/div[2]/article/section[2]/div[1]/div/div[1]/form/input[2]').click();
What I'm getting on the output is
ElementClickInterceptedException: Message: element click intercepted:
Element ... is
not clickable at point (553, 728). Other element would receive the
click:
(Session info: chrome=93.0.4577.63)
What might be the issue?
Things to noted down in this scenario :-
When you define an explicit waits wait = WebDriverWait(driver, 20), you can always use wait reference in the scope. you do not need to create again and again in same class.
Try to avoid absolute xpath /html/body/div[3]/div[2]/article/section[2]/div[1]/div/div[1]/form/input[2], try with relative xpath/xpath axes.
When we try to send keys to some element, in general it should be a input tag, not label
You may have to scroll, may be not in this case but when you scroll manually to interact with elements in UI, same has to automated with Selenium as well.
Also I observed to this webapp that search and input tags are duplicated, so I have used xpath indexing [2] to handle.
Sample code :-
driver = webdriver.Chrome(executable_path = 'mypath/chromedriver.exe')
driver.maximize_window()
#driver.implicitly_wait(50)
driver.get("https://ai.fmcsa.dot.gov/SMS")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Close']"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "(//input[#name='MCSearch'])[2]"))).send_keys('1818437')
wait.until(EC.element_to_be_clickable((By.XPATH, "(//input[#name='search'])[2]"))).click()
You can use below xpaths too.
driver.find_element_by_xpath("//div[#class='sms-search-box']//input[1]").send_keys('1818437')
driver.find_element_by_xpath("//div[#class='sms-search-box']//input[2]").click()
Xpath you are using is not right. Your xpath for the input field should be like this.
driver.find_element_by_xpath("//input[#name='MCSearch' and #placeholder='Type Name or U.S. DOT#']").send_keys("1818437")
driver.find_element_by_xpath("//input[#placeholder='Type Name or U.S. DOT#']//following::input[#value='Search']").click();
I'm very new to Selenium with Python and I'm simply trying to click 2 consecutive buttons on this website: https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28
I want to click the settings icon in the top right corner and then click the celsius option to convert temperatures on the site.
I can achieve this whilst debugging the code and doing the steps one at a time, but when I run the code normally, it is unable to find the second element with the error:
NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="wuSettings-quick"]/div/a[2]"}
(Session info: chrome=85.0.4183.83)
I have tried identifying the elements by id and xpath as follows:
Button 1: driver.find_element_by_id('wuSettings').click()
Button 2: driver.find_element_by_xpath('//*[#id="wuSettings-quick"]/div/a[2]').click()
Hoping it's an easy fix. All help appreciated. Thanks.
Code below:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28")
driver.implicitly_wait(15)
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, 'wuSettings'))
)
element.click()
# element = WebDriverWait(driver, 10).until(
# EC.presence_of_element_located((By.XPATH, '//*[#id="wuSettings-quick"]/div/a[2]'))
# )
# element.click()
except:
driver.quit()
Your code works fine if I replace this line:
driver.implicitly_wait(15)
with
time.sleep(5)
When I need to await for an element to be clickable, I usually have the best luck with something like this
can_click=False
while not can_click:
try:
element=driver.find_element_by_id('wuSettings')
element.click()
# note that if the issue is that the element is not visible in
# the viewport, you may need to do
# driver.execute_script("arguments[0].click();", element)
# instead
can_click=True
except:
pass
basically it should throw an exception until you can click on the element you want and once it does the loop terminates.
First of all you should not use both Implicit and explicit wait together. A cocktail of it could cause an unpredictable wait times. As you are using explicit wait for both buttons, it should be more than sufficient.
Also use of time.sleep might work in this case but its never been a good way to wait in test script as it makes test brittle and unpredictable.
As you can see even after page load few ajax components (Record for day passed in url is loading) are still loading and even though setting element is present its not giving option to change temp, unit record is fully loaded. So explicit / implicit wait on setting icon is not working.
Try below code and it should work.
try:
driver.get("https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28")
WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "//div[#class='legend-def temperature']"))) # To insure record for the day is loaded
ele = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, "wuSettings")))
driver.execute_script("arguments[0].scrollIntoView();", ele)
ele.click() #Scroll to Setting icon
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Switch to Metric']"))).click()
except:
driver.quit()
I tried different things, but it looks like you have to agree cookies use before you can click on another button.
I tried Matt's solution but I'm pretty sure that the commercial loading on top of the page screws up the first click on the settings button.
This is working fine for me :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28")
cookies = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//button[#id="truste-consent-button"][#type="button"]'))
)
cookies.click()
time.sleep(5)
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//button[#id="wuSettings"][#class="wu-settings"]'))
)
element.click()
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="wuSettings-quick"]/div/a[2]'))
)
element.click()
I've created a python script together with selenium to parse a specific content from a webpage. I can get this result AARONS INC located under QUOTE in many different ways but the way I wish to scrape that is by using pseudo selector which unfortunately selenium doesn't support. The commented out line within the script below represents that selenium doesn't support pseudo selector.
However, when I use pseudo selector within driver.execute_script() then I can parse it flawlessly. To make this work I had to use hardcoded delay for the element to be avilable. Now, I wish to do the same wrapping this driver.execute_script() within Explicit Wait condition.
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
driver.get("https://www.nyse.com/quote/XNYS:AAN")
time.sleep(15)
# item = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span:contains('AARONS')")))
item = driver.execute_script('''return $('span:contains("AARONS")')[0];''')
print(item.text)
How can I wrap driver.execute_script() within Explicit Wait condition?
This is one of the ways you can achieve that. Give it a shot.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
with webdriver.Chrome() as driver:
wait = WebDriverWait(driver, 10)
driver.get('https://www.nyse.com/quote/XNYS:AAN')
item = wait.until(
lambda driver: driver.execute_script('''return $('span:contains("AARONS")')[0];''')
)
print(item.text)
You could do the while thing in the browser script which is probably safer:
item = driver.execute_async_script("""
var span, interval = setInterval(() => {
if(span = $('span:contains("AARONS")')[0]){
clearInterval(interval)
arguments[0](span)
}
}, 1000)
""")
Here is the simple approach.
url = 'https://www.nyse.com/quote/XNYS:AAN'
driver.get(url)
# wait for the elment to be presented
ele = WebDriverWait(driver, 30).until(lambda driver: driver.execute_script('''return $('span:contains("AARONS")')[0];'''))
# print the text of the element
print (ele.text)
A similar question was asked on another thread
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Firefox()
element = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.ID, "element_id"))
This answer doesn't work because the element already existed in the previous page. I want to be able to wait until the page is done loading and then grab the element.
As the element existed in the previous page and now you want to grab the element after the page is done loading you can induce WebDriverWait initially for the staleness_of(element) then once again for the element_to_be_clickable(locator) and you can use the following solution:
WebDriverWait(browser, 10).until(EC.staleness_of(driver.find_element_by_id("element_id"))
element = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.ID, "element_id"))
You can use javascriptexecutor if you want to wait for page to load and then execute further steps.This will return "complete" only if the page is loaded completely.
driver.execute_script("return document.readyState")=="complete"