https://www.testim.io/blog/how-to-wait-for-a-page-to-load-in-selenium/
I want to understand how "Explicit Wait" is implemented under selenium. Can you show some example python code to demonstrate how selenium's "Explicit Wait" is implemented without using selenium?
Is the logic just wait for some time, then test for if an element is available, if not wait more time, check again, ..., until the element is available?
To understand explicit waits better, I found I needed to understand what's happening in the following:
The expected condition function.
The WebdriverWait.until method
The simplest expected condition is presence_of_element_located. It is just a wrapper around driver.find_element()
def presence_of_element_located(locator):
def _predicate(driver):
return driver.find_element(*locator)
return _predicate
Other expected conditions will check the element for certain conditions but I'll keep this example simple.
The result of presence_of_element_located is passed into WebDriverWait.until, typically examples will look like this:
wait = WebDriverWait(driver, timeout=30)
element = wait.until(ec.presence_of_element_located((By.ID, "my_id")))
When I break down what's happening in the above it starts to become a little more clear.
a_callable_method = ec.presence_of_element_located((By.ID, "my_id"))
wait = WebDriverWait(driver, timeout=30)
element = wait.until(a_callable_method)
WebDriverWait.until is simply a while loop calling a_callable_method you passed in. The method/function we are passing in always takes driver as an argument.
def until(self, method, message: str = ""):
"""Calls the method provided with the driver as an argument until the \
return value does not evaluate to ``False``.
:param method: callable(WebDriver)
:param message: optional message for :exc:`TimeoutException`
:returns: the result of the last call to `method`
:raises: :exc:`selenium.common.exceptions.TimeoutException` if timeout occurs
"""
screen = None
stacktrace = None
end_time = time.monotonic() + self._timeout
while True:
try:
value = method(self._driver)
if value:
return value
except self._ignored_exceptions as exc:
screen = getattr(exc, 'screen', None)
stacktrace = getattr(exc, 'stacktrace', None)
time.sleep(self._poll)
if time.monotonic() > end_time:
break
raise TimeoutException(message, screen, stacktrace)
In other words, explicit waiting is just a retry loop until a certain condition is met, or the timer runs out. In our example it's trying to find the element with id="my_id". If/when the element is found, it will be returned, otherwise a TimeoutException will be raised.
The real power of explicit wait starts to shine when looking at some of the other expected conditions:
def visibility_of(element):
def _predicate(_):
return _element_if_visible(element)
return _predicate
def _element_if_visible(element, visibility=True):
return element if element.is_displayed() == visibility else False
Using visibility_of will look for the element but then also check to see if the element has the condition consistent with being visible.
Related
I have the following code:
for button in buttons:
ActionChains(driver).move_to_element(button).perform()
time.sleep(2)
button.click()
time.sleep(2)
try:
wait_button.until(EC.presence_of_element_located((By.XPATH,'//div/h2')))
time.sleep(2)
name = driver.find_element_by_xpath('//div/h2').text
except:
wait_button.until(EC.presence_of_element_located((By.XPATH,'//span[#id="chat-header-title"]')))
time.sleep(2)
name = driver.find_element_by_xpath('//span[#id="chat-header-title"]').text
def pull_ul() -> list:
chat_frame = driver.find_element_by_xpath("//iframe[starts-with(#id, 'experience-container-')]")
driver.switch_to.frame(chat_frame)
wait_button.until(EC.presence_of_element_located((By.XPATH,'//ul[#aria-label="Chat content"]')))
the_ul =driver.find_element(By.XPATH,'//ul[#aria-label="Chat content"]')
new_lis =the_ul.find_elements(By.TAG_NAME,'li')
return new_lis
def pull_ul_again() -> list:
the_ul =driver.find_element(By.XPATH,'//ul[#aria-label="Chat content"]')
new_lis_2 =the_ul.find_elements(By.TAG_NAME,'li')
return new_lis_2
lis = pull_ul()
print(f"Archiving Chat with {name} ...\n")
print("this is len lis: ",len(lis), "for " + name)
And here is what the terminal shows:
As you can see, the code actually does run past the line that threw up the error, how is this possible? Also, why would this be happening , I successfully ran the code multiple times and suddenly it starts throwing the follwing error?
The loop that you're doing,
for button in buttons:
ActionChains(driver).move_to_element(button).perform()
...
is causing your StaleElementReferenceException because inside that you have driver.switch_to.frame(chat_frame), but you never switch back to default content or to the parent frame, which means that your buttons won't exist in that level, and so Selenium throws StaleElementReferenceException as a result. That could also happen if you navigate away from the page (which might also be happening depending on the button clicks).
If you ever switch frames or leave pages, you have to re-find elements in order to interact with them.
I am trying to programmatically solve the problem presented at this address: https://www.arealme.com/brain-memory-game/en/, in short it will display several flashing numbers in a row, and asking you to input the numbers in reverse order.
Now here are the details, when you get to the website, there will be a start button which has a blue bar that fills it from the left, I have found that if you let your bot click that button before the blue bar fills up the button, then the function of the button won't be triggered and the rest of the script won't run.
After the start button is clicked, the page will refresh, and you will see this flashing text:
The numbers are coming soon. Please pay attention...
And after that, you will see several flashing numbers, they flash by changing alpha, they change from fully visible to fully invisible, when a number disappears another number will appear, they are different numbers even if they have the same value.
Then there will be this message:
Please click on the number you just saw IN REVERSE ORDER.
And a number pad with ten buttons.
Do as it said and then a button will appear that tells whether you are right or not and you need to click it to proceed to the next question.
There are ten questions in total.
Now with the technical details.
I have managed to find the class names and xpaths and ids of the elements involved.
The id of the start button is 'start', and I have found time.sleep(3) is sufficient for it to fill up.
The hierarchy of the classes is as follows:
<div class="questionWrapper">
<div class="question">
<div class="gnumber_title">
<div class="gend-tip">Please click on the number you just saw IN REVERSE ORDER.</div>
</div>
<div class="gnumber_btns" id="gbtn0" style=""><button data-n="9">9</button><button data-n="8">8</button><button data-n="7">7</button><button data-n="6">6</button><button data-n="5">5</button><button data-n="4">4</button><button data-n="3">3</button><button data-n="2">2</button><button data-n="1">1</button><button data-n="0">0</button></div>
</div>
<div class="answer" value="10">1</div>
</div>
The questions are wrapped in "questionWrapper" class, in each there is one and only one instance of class "gnumber_title", the object contains the messages and numbers.
At all times the class holds exactly one element, what the class holds changes with time, but they all share the same xpath relative to said class: './div'
At the start of questions, when the prompt is the first message, the object inside the said location is of class "gstart-tip blink_me".
Then it will disappear and in its place there will be an object of class 'gflash-num', whose content can be accessed using .text attribute.
It blinks by changing style, the style will change from "opacity: 1.000000;" to "opacity: 0.000000;" (I don't know the exact values, but it is a float with six decimal places from 1 to 0), then it will become "display: none;", then it will be deleted and another instance of "glash-num" will appear.
After several instances of "gflash-num", the second message appears, and its class is "gend-tip", located at the same xpath as the numbers and first message.
And the number pad will be visible, whose class is "gnumber_btns".
After the same number of displayed numbers of the buttons have been clicked, the button whose class is "answer" appears, clicking it will proceed to the next question.
Here is my attempt to solve the problem:
import time
from selenium import webdriver
Firefox = webdriver.Firefox()
Firefox.get('https://www.arealme.com/brain-memory-game/en/')
time.sleep(3)
Firefox.find_element_by_id('start').click()
questions = Firefox.find_elements_by_class_name("questionWrapper")
for q in questions:
numbers = []
title = q.find_element_by_class_name('gnumber_title')
while True:
if title.find_element_by_xpath('./div').get_attribute('class') != 'gstart-tip blink_me':
break
while True:
if title.find_element_by_xpath('./div').get_attribute('class') == "gend-tip":
break
numbers.append(title.find_element_by_class_name('gflash-num').text)
while True:
if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('opacity'):
break
while True:
if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('display'):
break
buttons = q.find_element_by_class_name("gnumber_btns")
for n in reversed(numbers):
buttons.find_element_by_xpath(f'.//*[text() = "{n}"]').click()
time.sleep(0.5)
q.find_element_by_class_name("answer").click()
And the errors:
---------------------------------------------------------------------------
StaleElementReferenceException Traceback (most recent call last)
<ipython-input-1-9d34e88c4046> in <module>
20 numbers.append(title.find_element_by_class_name('gflash-num').text)
21 while True:
---> 22 if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('opacity'):
23 break
24 while True:
c:\program files\python39\lib\site-packages\selenium\webdriver\remote\webelement.py in get_attribute(self, name)
137 attributeValue = ''
138 if self._w3c:
--> 139 attributeValue = self.parent.execute_script(
140 "return (%s).apply(null, arguments);" % getAttribute_js,
141 self, name)
c:\program files\python39\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute_script(self, script, *args)
632 command = Command.EXECUTE_SCRIPT
633
--> 634 return self.execute(command, {
635 'script': script,
636 'args': converted_args})['value']
c:\program files\python39\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
c:\program files\python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
StaleElementReferenceException: Message: The element reference of <div class="gflash-num"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
The exception can be raised at any iteration of the dynamic element, I have to wait until the element located at that xpath is no longer "gstart-tip blink_me" to start the next stage of code execution, then add the value of "gflash-num" and wait until the element is gone to add the next value, and then finally when it becomes "gend-tip" click the buttons in reverse order.
I have tried to avoid the exception by not assigning variables and getting the attribute on the fly, but the exception still got raised.
But the time when it raises exceptions is exactly when it is supposed to wake up from its sleep and add the numbers.
So how can I wait until an element becomes stale/invisible/deleted/non-existent/whatsoever, all the Google searching tells exactly how to do the opposite: wait until the element is NO LONGER STALE, but I want to wait until the element is GONE, so how to do this?
I think I have found something very important to solving the problem, using ffmpeg to extract frames from screen recording of the process, I am able to determine the exact during of the flashings.
The first prompt shows exactly 3 seconds, and each number flashes exactly 1 second.
to wait till any elelemt becomes invisible we have this Expected conditions :
invisibility_of_element
also in code I could see :
class invisibility_of_element(invisibility_of_element_located):
""" An Expectation for checking that an element is either invisible or not
present on the DOM.
element is either a locator (text) or an WebElement
"""
def __init(self, element):
self.target = element
if you have a running webdriverwait object, then you can try this :
WebDriverWait(driver, 10).until(EC.invisibility_of_element((By.XPATH, "xpath here")))
this will wait till the invisibility of element, defined by xpath.
I have solved my problem, without using invisibility_of_element, maybe I circumvented the problem, but the code doesn't throw exceptions now.
Since I know the exact "lifespans" of the elements, I could use time.sleep() rather than relying on EC.invisibility_of_element() whose behavior might be unpredictable.
The code:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Firefox = webdriver.Firefox()
Firefox.get('https://www.arealme.com/brain-memory-game/en/')
time.sleep(3)
Firefox.find_element_by_id('start').click()
questions = Firefox.find_elements_by_class_name("questionWrapper")
wait = WebDriverWait(Firefox, 3)
for i, q in enumerate(questions):
wait.until(EC.visibility_of_element_located((By.ID, f'q{i + 1}')))
numbers = []
ids = set()
title = q.find_element_by_class_name('gnumber_title')
if title.find_element_by_xpath('./div').get_attribute('class') == 'gstart-tip blink_me':
time.sleep(3)
while True:
if title.find_element_by_xpath('./div').get_attribute('class') == "gend-tip":
break
number = title.find_element_by_class_name('gflash-num')
if number.id not in ids:
numbers.append(number.text)
ids.add(number.id)
time.sleep(1)
wait.until(EC.visibility_of_element_located((By.ID, f'gbtn{i}')))
buttons = q.find_element_by_class_name("gnumber_btns")
for n in reversed(numbers):
buttons.find_element_by_xpath(f'.//*[#data-n = "{n}"]').click()
time.sleep(0.1)
time.sleep(0.5)
q.find_element_by_class_name("answer").click()
You can use invisibility_of_element_located() or until_not() methods.
WebDriverWait(self.driver, timeout).until(ec.invisibility_of_element_located(locator))
WebDriverWait(self.driver, timeout).until_not(ec.visibility_of_element_located(locator))
I need to wait for the presence of an element with specific text in it.
I want to fetch the info from that element in the moment the element is present and has text in it. I appears sometime after submitting a form and normaly fills with info a little later.
My current solution looks like this:
wait = WebDriverWait(self.driver, MAXIMUM_LOAD_TIME)
try:
wait.until(ec.presence_of_element_located((By.ID,"IdOfElement")))
wait.until(ec.text_to_be_present_in_element((By.ID, "IdOfElement"), "theText"))
data = self._extract_data()
except TimeoutException:
raise WebsiteTimeoutError(MAXIMUM_LOAD_TIME)
This runs perfectly in >99% of the cases but now it happened that i got an error on ec.text_to_be_present_in_element.
The error was:
File "my/Path", line 436, in _scrape_quote
wait.until(ec.text_to_be_present_in_element((By.ID, "IdOfElement"), "theText"))
File "C:\Program Files (x86)\Python38-32\lib\site-packages\selenium\webdriver\support\wait.py", line 71, in until
value = method(self._driver)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\selenium\webdriver\support\expected_conditions.py", line 210, in __call__
return self.text in element_text
TypeError: argument of type 'NoneType' is not iterable
Apparently the element dissapeared again. Is my assumption right? What is the best way to fix this?
If you take a look at the text_to_be_present_in_element implementation it assume the value is never None.
def __call__(self, driver):
try:
element_text = _find_element(driver, self.locator).text
return self.text in element_text
except StaleElementReferenceException:
return False
If the value of element_text is None (which can be sometime), it will throw you an exception
TypeError: argument of type 'NoneType' is not iterable
Now your code is a little cumbersome as you need to ensure the element is present first, then retrieve the same element to find the proper value. There is no class in the EC module that offer both at the same time. So how about implementing your class.
You can implement a class that will ensure it is present and at the same time fix the issue to handle the case of None in the text return.
class FindPresenceOfTextInElement(object):
def __init__(self, locator, text):
self.locator = locator
self.text = text
def __call__(self, driver):
try:
text = driver.find_element(*self.locator).text
return self.text in text
except (NoSuchElementException, StaleElementReferenceException, TypeError):
return False
Then your code becomes:
wait = WebDriverWait(self.driver, MAXIMUM_LOAD_TIME)
try:
wait.until(FindPresenceOfTextInElement((By.ID,"IdOfElement")))
data = self._extract_data()
except TimeoutException:
raise WebsiteTimeoutError(MAXIMUM_LOAD_TIME)
You will need the following imports:
from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException
UPDATE
AS the above is the best approach, I believe, you could have work around it by adding the exception handler in the wait obj as follow:
wait = WebDriverWait(driver, 30, ignored_exceptions=(TypeError, ))
If you add the NoSuchElementException as well, you basically added to ensure element is present too.
My selenium code checks for a completed subroutine to be done by waiting on the site's title to change which worked perfectly. Code looks like this:
waitUntilDone = WebDriverWait(session, 15).until(EC.title_contains(somestring))
However, this can fail sometimes since the site's landing page changes after manual website visits. The server remembers where you left off. This forces me to check for an alternate condition (website title = "somestring2).
Here is what I came up with so far (also works as far as I can tell):
try:
waitUntilDone = WebDriverWait(session, 15).until(EC.title_contains(somestring)) # the old condition
except:
try:
waitUntilDone = WebDriverWait(session, 15).until(EC.title_contains(somestring2)) # the new other condition which is also valid
except:
print "oh crap" # we should never reach this point
Either one of these conditions is always true. I don't know which one thou.
Is there any way to include an "OR" inside these waits or make the try/except block look nicer?
Looks like selenium will let you do this by creating your own class. Check out the documentation here: http://selenium-python.readthedocs.io/waits.html
Here's a quick example for your case. Note the key is to have a method named __call__ in your class that defines the check you want. Selenium will call that function every 500 milliseconds until it returns True or some not null value.
class title_is_either(object):
def __init__(self, locator, string1, string2):
self.locator = locator
self.string1 = string1
self.string2 = string2
def __call__(self, driver):
element = driver.find_element(*self.locator) # Finding the referenced element
title = element.text
if self.string1 in title or self.string2 in title
return element
else:
return False
# Wait until an element with id='ID-of-title' contains text from one of your two strings
somestring = "Title 1"
somestring2 = "Title 2"
wait = WebDriverWait(driver, 10)
element = wait.until(title_is_either((By.ID, 'ID-of-title'), somestring, somestring2))
I am using the following code to check if a new message has arrived on Facebook Messages :
def waitForNextMessage():
messageList=driver.find_elements_by_css_selector('.null')
message=''
while True:
driver.implicitly_wait(10)
element = driver.find_elements_by_css_selector('.null')
if not(element == messageList):
message=element[-1].find_elements_by_css_selector("*")[0].text
print(message)
break
Basically, at the start, I make a list of all the elements of class .null (the message boxes) and in an infinite loop create the list again and check if it is different. If it is, it means that a new message has arrived.
Is there a better way to do this, hopefully without an Infinite Loop?
You can use explicit wait to wait for the messages null element to disappear
messageList = driver.find_elements_by_css_selector('.null')
try:
WebDriverWait(driver, 60).until(expected_conditions.staleness_of(messageList[1]))
message = element[1].find_elements_by_css_selector("*")[0].text
print(message)
except TimeoutException:
# no new message
This will wait up to 60 seconds for the second item in messageList to disappear from the DOM, i.e. you have new message. If there is new message, the code will continue to the message printing, else there will be TimeoutException.
Edit
To run this for certain amount of time you can set timer
now = time.time()
howLong = now + 10
while time.time() < howLong:
waitForNextMessage()
pass
To run this continuously in addition to other tests you can run waitForNextMessage() in thread.
Edit 2
expected_conditions.staleness_of() is an expectation for an element not to be attached to the DOM. You can find more expected conditions here