I use python/selenium for web-scraping and automation through Chrome. Recently I've come across a problem where some pages or websites would load endlessly as a means to prevent bots from scraping them (It often starts to happen after first time). I had no idea how to counter this until I came across the function in question:
wait = WebDriverWait(self.driver, 3)
element = wait.until(
EC.presence_of_element_located((By.XPATH, '//button[.="Sign up"]'))
)
The way I understood this function is that it sat a timeout for the page (3 seconds in this example) and at the same time checking whether an element has been loaded into the page or not. If the element has been loaded or if the wait times out, the program stops waiting for the page to load and continues on with the next line of code, which is, in my case, a print("timeout") function.
Thing is, as I use the function as demonstrated above the program still waits for the page to load, as "timeout" only appears in the console after I manually stop the page loading from the browser.
I am sure the element is present in the page (and quite visible) as it continues to load endlessly, because once I press the X button and stop the page loading, "timeout" is printed and the element is interacted with successfully.
I also tried EC.visibility_of_element_locatedto the same effect.
What is going wrong and how do I achieve the functionality I need?
Since you didn't share a link to the page you are working on I can only guess.
So my guess is: you are using a wrong locator.
Try using this
wait = WebDriverWait(self.driver, 3)
element = wait.until(EC.presence_of_element_located((By.XPATH, '//button[contains(.,"Sign up")]')))
Or this
wait = WebDriverWait(self.driver, 3)
element = wait.until(EC.presence_of_element_located((By.XPATH, '//button[contains(.,"sign up")]')))
Related
I am trying to automate a process to download data from a website. The code works if I run it step by step but if I run it all at once it fails. Giving the error
ElementNotInteractableException: Message: element not interactable
I have got around this using time.sleep(x amount of time) but it still seems to fail intermittently. I am having trouble implementing implicit waits. Any help would be appreciated. Code below.
import selenium
browser = webdriver.Chrome(executable_path=r'path\to\chromedriver.exe')
browser.get("https://map.sarig.sa.gov.au/")
browser.maximize_window()
browser.switch_to.frame(browser.find_element_by_id('MapViewer'))
browser.find_element_by_xpath('//*[#id="TourWidget"]/div[1]/span').click()
browser.find_element_by_xpath('//*[#id="menuAllMapLayers"]/div[2]/p').click()
browser.find_element_by_xpath('//*[#id="238"]/li[1]/div/div/span[1]').click()
time.sleep(3)
browser.find_element_by_xpath('//*[#id="238"]/li[1]/div/div/span[1]').click()
browser.find_element_by_xpath('//*[#id="238"]/li[3]/div/div/label/span').click()
browser.find_element_by_xpath('//*[#id="239"]/li[1]/div/div/span[1]').click()
browser.find_element_by_xpath('//*[#id="239"]/li[3]/div/div/label/span').click()
browser.find_element_by_xpath('//*[#id="menuActiveLayers"]').click()
browser.find_element_by_xpath('//*[#id="groupOptions238"]/span').click()
time.sleep(3)
browser.find_element_by_xpath('//*[#id="238"]/li[2]/div/div[3]/div[2]/span').click()
browser.find_element_by_xpath('//*[#id="groupOptions239"]/span').click()
time.sleep(3)
browser.find_element_by_xpath('//*[#id="239"]/li[2]/div/div[3]/div[2]/span').click()
Use ActionChains and get access to pause(3) instead of using sleep(3) but it could also help to use Waits and checking if your elements actually are "visible" rather than "present" (see expected_conditions)
It's a lot of dropdowns so maybe there are not visible all the time, but you can run these checks after doing a move_to_element() so it would actually be present.
I'm a noob and trying to automate some online form filling in a certain site. My problem is that some buttons need some time before clicking them, otherwise they don't work (but no error!, execution continues).
My only solution so far is to add a time.sleep(6) before these buttons but this is not ideal.
I am trying to find a better solution.
So far, I have this function:
def Send_Click_dk(bywhat,what):
WebDriverWait(browser, 10).until(EC.presence_of_element_located((bywhat,what)))
WebDriverWait(browser, 10).until(EC.visibility_of(browser.find_element(bywhat, what)))
WebDriverWait(browser, 10).until(EC.element_to_be_clickable(browser.find_element(bywhat, what)))
browser.find_element(bywhat, what).click()
Send_Click_dk(By.NAME, "mainpanel_parentSection_1b0a0b")
First of all, is this a good approach? Am I misunderstanding something?
Secondly, if this is the right approach, what else could I check before clicking the button? So far, all of these checks pass instantly and the .click() is executed but doesn't produce the expected result. Only by adding time.sleep(6) the clicking works as intended.
This is a snapshot of that particular part of the page, I'm having trouble copying the raw text.
The first marked button reveals the second marked button. But if without the time.sleep(x) it just stays closed without revealing the second one.
Many thanks in advance for any help! Cheers!
You definitely can reduce all your code to a single line of
def Send_Click_dk(bywhat,what):
WebDriverWait(browser, 10).until(EC.element_to_be_clickable(browser.find_element(bywhat, what))).click()
Send_Click_dk(By.NAME, "mainpanel_parentSection_1b0a0b")
visibility_of expected condition includes presence_of_element_located since element can't be visible without being present.
element_to_be_clickable internally includes visibility_of.
Also WebDriverWait(browser, 10).until(EC.element_to_be_clickable(browser.find_element(bywhat, what))) returns web element object, so you can click it directly.
Also, make sure the mainpanel_parentSection_1b0a0b name attribute is a fixed value. 1b0a0b suffix seems to be dynamically generated
I run a multithread python selenium script which gives me unexpected behavior which I can not understand. So after opening a new browser I try to clear the cache with below code ;
driver.get('chrome://settings/clearBrowserData')
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//settings-ui')))
driver.find_element_by_xpath('//settings-ui').send_keys(Keys.ENTER)
90 out of 100 times this works but sometimes it throws me this error :
element not interactable
To my understanding somehow the page is not loaded fully and selenium is too "fast".
However I do wait until the element is visible which would eliminate this plus to my understanding the driver.get also waits until the "full get" is returned and finished.
Can someone explain me why this behavior occurs ?
Sub question would be is my understanding right driver.get(page) waits for full page loading done ?
The ElementNotInteractableException exception can occur for a few reasons:
The element that you want to interact with is disabled.
There's an overlay element covering the one you want to interact with, such as a loading spinner.
If it's 1, use EC.element_to_be_clickable instead of EC.visibility_of_element_located.
If it's 2, you'll need to either wait for the overlay element to go away on it's own. Or, perform an action to make the overlay element go away.
I have a website where I need to automate some actions. Every time a customer does a purchase, a div with an input and a submit button appears. On that div I need to enter a value and click submit. The div then closes until the next purchase appears in the same div. I need to do the same actions, and so on. It's indefinite.
I already found some solutions that point to the direction:
Selenium - wait until element is present, visible and interactable
WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".reply-button"))).click()
But I can't provide a specific time to wait. I need a solution that keeps going in a loop indefinitely and progress all purchases throughout the day
Every solution that I find solves the problem that the website takes time to load. But I have a completely different problem underlying. I need to wait for a purchases to happen. So I can't set a specific time to wait. It could be indefinite.
What Selenium function could help me -in a best practice way- with my problem?
You can use try and except
while true:
try:
WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".reply-button"))).click()
except:
continue
else:
break
I'm making sure I wait for the page to load with WebDriverWait but it's still running the timeout exception and I can't figure out why. I also checked to make sure the XPath was present in the chrome developer inspector and confirmed that it is. Here is the snippet if anyone can help me.
Thanks!
url2 = 'https://www.rotorooter.com/adelantoca/'
driver.get(url2)
delay = 3
try:
name = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.XPATH, '/html/body/app-root/div/app-not-found/div/app-local-page/app-local-map/div/div/div/div[2]/div[4]/div[1]/strong')))
print("Page is ready!",name)
except TimeoutException:
print("Loading took too much time!")
Returns:
Loading took too much time!
There are several issues here.
The main problem is that the element you are checking is far out of the initially visible screen view. So when the page is opened this element is not loaded until scroll down is performed. This is why your WebDriverWait fails on timeout.
You should never use automatically generated locators like /html/body/app-root/div/app-not-found/div/app-local-page/app-local-map/div/div/div/div[2]/div[4]/div[1]/strong. They are extremely unreliable and breakable.
It's recommended to use larger timeouts for WebDriverWaits. Making them too should may lead to false errors caused by low internet / slow web pages. They do not cause extra runtime in the normal cases since Selenium code will continue immediately after detecting the checked condition.