I am facing inconsistencies in Selenium execution.
Last line in the code snippet I pasted below doesn't execute consistently. Sometimes it works, sometimes it throws an error saying that element is not found. Doesn't Selenium "block" for the element to appear before attempting to execute the click? I generated it using Selenium IDE. What I am missing here?
self.driver.find_element(By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)").click()
self.driver.switch_to.default_content()
self.driver.find_element(By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText").click()
Selenium may not find elements if they happen to be loaded dynamically by JS and if you search for them before they are loaded.
You can try either an implicit wait or an explicit wait.
In case of implicit waiting, the docs say:
An implicit wait tells WebDriver to poll the DOM for a certain amount of time when trying to find any element (or elements) not immediately available.
You could do with something like:
from selenium import webdriver
driver = webdriver.Chrome()
driver.implicitly_wait(10) #wait and poll for 10 seconds
Whereas the explicit waiting means to explicitly specify the element which is to be waited for it to be available. As per the docs:
An explicit wait is a code you define to wait for a certain condition to occur before proceeding further in the code.
You can do this with something like:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
element1 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)")))
element1.click()
self.driver.switch_to.default_content()
element2 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText")))
element2.click()
As you are using the line of code:
self.driver.switch_to.default_content()
Presumably you are switching Selenium's focus from a frame or iframe to the Top Level Content. Hence you need to induce WebDriverWait for the desired element to be clickable and you can use the following Locator Strategy:
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
Note:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant detailed discussions in:
How to send text to the Password field within https://mail.protonmail.com registration page?
How to switch between iframes using Selenium and Python?
Wait for the element to be loaded
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)"))).click()
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
The number is how long the driver should spend looking for the element before moving on.
Related
I'm automating a bot to book a certain time (preferred time and second preferred time).
I need help to use if else statement to allow the browser to search for the preferred time and if not present to search for the second preferred time.
Below is my code:
preferred_time=driver.find_element(By.XPATH,'//div[#class="booking-start-time-label"][contains(., "12:33pm")]')
preferred_time.click()
second_preferd_time= driver.find_element(By.XPATH,'//div[#class="booking-start-time-label"][contains(., "1:33pm")]')
second_preferd_time.click()
An ideal approach will be to try to click on the initial desired element inducing WebDriverWait for element_to_be_clickable() and incase it fails catch the TimeoutException and attempt to click on the second desired element as follows:
try:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//div[#class="booking-start-time-label"][contains(., "12:33pm")]'))).click()
except TimeoutException:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//div[#class="booking-start-time-label"][contains(., "1:33pm")]'))).click()
Note: You have to add the following imports :
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
the link I'm trying to scrape is https://hcad.org/property-search/real-property/real-property-search-by-account-number/
I'm using this method currently
driver.get('https://hcad.org/property-search/real-property/real-property-search-by-account-number/')
driver.implicitly_wait(2)
driver.find_element_by_xpath('/html/body/div[1]/div[3]/div/h1').click()
actions.send_keys(Keys.TAB * 2 )
actions.send_keys(account_num)
actions.send_keys(Keys.TAB)
actions.send_keys(Keys.RETURN)
actions.perform()
to type the account number in search box and search it which gives me a new form.
I tried to find the input area using full xpath but it never works.
Is there is anyway I can write account num using element xpath or I can excess the form elements other then actions method.
I tried different methods.
like searching input area using (id,full xpath) using driver implicity wait function but none of them work please help me here so that I can grab elements using your method on the form I get after typing account number I'm lost.
You are using wrong locator.
Also, I don't understand why are you using Actions instead of Selenium.send_keys()?
I would do this as following:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
wait = WebDriverWait(driver, 20)
driver.get('https://hcad.org/property-search/real-property/real-property-search-by-account-number/')
wait.until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_xpath("//iframe")))
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'input[name="searchval"]'))).send_keys(account_num)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'input[type="submit"]'))).click()
The search field is in iframe, you would need to switch the driver focus :
code :
driver.maximize_window()
wait = WebDriverWait(driver, 10)
driver.get('https://hcad.org/property-search/real-property/real-property-search-by-account-number/')
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[src='https://public.hcad.org/records/Real.asp']")))
driver.find_element_by_id('acct').send_keys("12345")
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input[value='Search']"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
There are HTML elements on the website (https://www.immowelt.de/expose/2ult44w) that Selenium does not recognize at all. But I would like to be able to address them. I still recognize the element "body" without any problems, however "div [# class = 'cdk-overlay-container']" is not. Errors are not thrown.
from selenium import webdriver
import time
driver = webdriver.Chrome('C:\\go2\\installation\\chromedriver.exe')
driver.get("https://www.immowelt.de/expose/2ult44w");
driver.execute_script("return document.readyState") == "complete"
time.sleep(10)
#just so that a message is clicked away:
datenschutz = driver.find_elements_by_xpath("//button[#id='uc-btn-accept-banner']")
if len(datenschutz) > 0: datenschutz[0].click()
#that is not recognized:
example = driver.find_elements_by_xpath("//div[#class='cdk-overlay-container']")
print("Counts:"+str(len(example))) #result: Counts:0
The reason you are unable to found it because the element is present inside an iframe.
You need to switch to iframe first.
To Switch to iframe
Induce WebDriverWait() and wait for frame_to_be_available_and_switch_to_it()
And
Induce WebDriverWait() and wait for presence_of_all_elements_located()
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome('C:\\go2\\installation\\chromedriver.exe')
driver.get("https://www.immowelt.de/expose/2ult44w")
btn=WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//button[#id='uc-btn-accept-banner']")))
driver.execute_script("arguments[0].click();", btn)
WebDriverWait(driver,20).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"externalViewerStage")))
example=WebDriverWait(driver,20).until(EC.presence_of_all_elements_located((By.XPATH,"//div[#class='cdk-overlay-container']")))
print("Counts:"+str(len(example)))
driver.switch_to.default_content()
i'm having a problem with selenium. I load this page: Investing 3M, and i want to click got it in a pop-out window.
I already tryed with this code bellow, but it doesn't work:
driver = webdriver.Chrome()
driver.get("https://www.investing.com/equities/3m-co-financial-summary")
driver.implicitly_wait(10)
x =driver.find_elements_by_xpath("/html/body/div[8]/div[1]/div/div[2]/div[2]/a[1]")
print(len(x))
print(x)
Can anyone provide any solution to this?
The element is present inside an iframe.You need to switch to frame first.
To handle dynamic element you need to
Induce WebDriverWait() and wait for frame_to_be_available_and_switch_to_it() and following xpath
Induce WebDriverWait() and wait for element_to_be_clickable() and following xpath
code:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.investing.com/equities/3m-co-financial-summary")
WebDriverWait(driver,20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#title='TrustArc Cookie Consent Manager']")))
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//a[text()='Got it']"))).click()
Browser snapshot after clicking
I'm trying to click on Jetzt bewerben available in this webpage using selenium. The script that I've written so far can click on that grid If I stick with hardcoded delay. I would like to do the clicking without using any hardcoded delay.
I've tried with:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = "https://jobs.deloitte.de/job/D%C3%BCsseldorf-Werkstudent-%28mwd%29-Administration-Business-Process-Solutions/522320501/"
driver = webdriver.Chrome()
driver.get(url)
wait = WebDriverWait(driver,10)
time.sleep(5)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"a#bewerben_top > h1"))).click()
How can I click on that grid shaking off hardcoded delay?
There is not a problem with a fact that button is not clickable or not visible.
The problem is that there is Javascript doing some "magic" to the button.
All you need to do is to wait until the document is complete.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = "https://jobs.deloitte.de/job/D%C3%BCsseldorf-Werkstudent-%28mwd%29-Administration-Business-Process-Solutions/522320501/"
driver = webdriver.Chrome()
driver.get(url)
wait = WebDriverWait(driver,10)
wait.until(lambda d: d.execute_script("return document.readyState")=='complete')
driver.find_element_by_css_selector("a#bewerben_top > h1").click()
Your locator strategy was perfect. However, there are a couple of things:
It seems the Browsing Context of the website https://jobs.deloitte.de/job/D%C3%BCsseldorf-Werkstudent-%28mwd%29-Administration-Business-Process-Solutions/522320501/ returns back the control even before document.readyState turns complete.
Ideally, you should acknowledge the Acceptance of the website's cookie policy.
As per best practices, while invoking click() you need to induce WebDriverWait for the element_to_be_clickable().
You can use the following Locator Strategies:
driver.get("https://jobs.deloitte.de/job/D%C3%BCsseldorf-Werkstudent-%28mwd%29-Administration-Business-Process-Solutions/522320501/")
WebDriverWait(driver, 10).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#cookie-acknowledge"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#bewerben_top > h1"))).click()
Note: You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
PS: In such one off scenarios you may require to wait for the jQuery to complete as follows:
WebDriverWait(driver, 20).until(lambda driver: driver.execute_script('return jQuery.active') == 0)