Develop Reference

How Explicit Wait is implemented under selenium?

https://www.testim.io/blog/how-to-wait-for-a-page-to-load-in-selenium/
I want to understand how "Explicit Wait" is implemented under selenium. Can you show some example python code to demonstrate how selenium's "Explicit Wait" is implemented without using selenium?
Is the logic just wait for some time, then test for if an element is available, if not wait more time, check again, ..., until the element is available?

To understand explicit waits better, I found I needed to understand what's happening in the following:
The expected condition function.
The WebdriverWait.until method
The simplest expected condition is presence_of_element_located. It is just a wrapper around driver.find_element()
def presence_of_element_located(locator):
def _predicate(driver):
return driver.find_element(*locator)
return _predicate
Other expected conditions will check the element for certain conditions but I'll keep this example simple.
The result of presence_of_element_located is passed into WebDriverWait.until, typically examples will look like this:
wait = WebDriverWait(driver, timeout=30)
element = wait.until(ec.presence_of_element_located((By.ID, "my_id")))
When I break down what's happening in the above it starts to become a little more clear.
a_callable_method = ec.presence_of_element_located((By.ID, "my_id"))
wait = WebDriverWait(driver, timeout=30)
element = wait.until(a_callable_method)
WebDriverWait.until is simply a while loop calling a_callable_method you passed in. The method/function we are passing in always takes driver as an argument.
def until(self, method, message: str = ""):
"""Calls the method provided with the driver as an argument until the \
return value does not evaluate to ``False``.
:param method: callable(WebDriver)
:param message: optional message for :exc:`TimeoutException`
:returns: the result of the last call to `method`
:raises: :exc:`selenium.common.exceptions.TimeoutException` if timeout occurs
"""
screen = None
stacktrace = None
end_time = time.monotonic() + self._timeout
while True:
try:
value = method(self._driver)
if value:
return value
except self._ignored_exceptions as exc:
screen = getattr(exc, 'screen', None)
stacktrace = getattr(exc, 'stacktrace', None)
time.sleep(self._poll)
if time.monotonic() > end_time:
break
raise TimeoutException(message, screen, stacktrace)
In other words, explicit waiting is just a retry loop until a certain condition is met, or the timer runs out. In our example it's trying to find the element with id="my_id". If/when the element is found, it will be returned, otherwise a TimeoutException will be raised.
The real power of explicit wait starts to shine when looking at some of the other expected conditions:
def visibility_of(element):
def _predicate(_):
return _element_if_visible(element)
return _predicate
def _element_if_visible(element, visibility=True):
return element if element.is_displayed() == visibility else False
Using visibility_of will look for the element but then also check to see if the element has the condition consistent with being visible.

Selenium Hover over element gives me an Error AFTER is goes past the code?

I have the following code:
for button in buttons:
ActionChains(driver).move_to_element(button).perform()
time.sleep(2)
button.click()
time.sleep(2)
try:
wait_button.until(EC.presence_of_element_located((By.XPATH,'//div/h2')))
time.sleep(2)
name = driver.find_element_by_xpath('//div/h2').text
except:
wait_button.until(EC.presence_of_element_located((By.XPATH,'//span[#id="chat-header-title"]')))
time.sleep(2)
name = driver.find_element_by_xpath('//span[#id="chat-header-title"]').text
def pull_ul() -> list:
chat_frame = driver.find_element_by_xpath("//iframe[starts-with(#id, 'experience-container-')]")
driver.switch_to.frame(chat_frame)
wait_button.until(EC.presence_of_element_located((By.XPATH,'//ul[#aria-label="Chat content"]')))
the_ul =driver.find_element(By.XPATH,'//ul[#aria-label="Chat content"]')
new_lis =the_ul.find_elements(By.TAG_NAME,'li')
return new_lis
def pull_ul_again() -> list:
the_ul =driver.find_element(By.XPATH,'//ul[#aria-label="Chat content"]')
new_lis_2 =the_ul.find_elements(By.TAG_NAME,'li')
return new_lis_2
lis = pull_ul()
print(f"Archiving Chat with {name} ...\n")
print("this is len lis: ",len(lis), "for " + name)
And here is what the terminal shows:
As you can see, the code actually does run past the line that threw up the error, how is this possible? Also, why would this be happening , I successfully ran the code multiple times and suddenly it starts throwing the follwing error?

The loop that you're doing,
for button in buttons:
ActionChains(driver).move_to_element(button).perform()
...
is causing your StaleElementReferenceException because inside that you have driver.switch_to.frame(chat_frame), but you never switch back to default content or to the parent frame, which means that your buttons won't exist in that level, and so Selenium throws StaleElementReferenceException as a result. That could also happen if you navigate away from the page (which might also be happening depending on the button clicks).
If you ever switch frames or leave pages, you have to re-find elements in order to interact with them.

Element <span> is not clickable at point (1039,84) because another element <div class="_2dDPU CkGkG"> obscures it

Im trying to click the like button with selenium this is the exception error I'm getting:
[-] Passed One Message: Element <span class="glyphsSpriteHeart__filled__16__white u-__7"> is not clickable at point (1039,84) because another element <div class="_2dDPU CkGkG"> obscures it
I have tried solving it with a loop that clicks all of the aria label like elements with Xpath, see my code:
while count <= likes:
try:
likeopt = bot.find_elements_by_xpath("//*[#aria-label='Like'] ")
for likeopti in likeopt:
likeopti.click()
#bot.find_element_by_class_name("fr66n").click()
print("[+] Liked !", count)
count += 1
time.sleep(random.randint(8, 15))
bot.find_element_by_class_name("_65Bje").click()
time.sleep(random.randint(8, 15))
except Exception as e:
try:
bot.find_element_by_class_name("_65Bje").click()
time.sleep(random.randint(3, 7))
print("[-] Passed One", e)
except Exception as x:
bot.find_elements_by_xpath("//*[#aria-label='Close'] ")
continue
print(x,"[+]click on X, finished all posts")
How do you think can it can be solved?
Thanks

Watch your script running, and pay attention to the point where it fails. You will likely see that there's some sort of drop down menu, pop-up 'dialog', or even an advertisement, that has displayed over your target link. You'll need to close/dismiss this to be able to get at the target link.
Alternatively, you can hack it by instructing the driver to send a click command via Javascript. bot.execute_script("arguments[0].click();", likeopt) I typically save this approach as a last resort since it does not represent how a user would interact with the page.

Selenium + Python how to listen for change of element

The idea is to create a bot to read message from a chat, and all message are in a ul>li(not neccesary to write message), e.g
<ul class="message-list">
<li class="message">
Hello There
</li>
<li class="message">
Hello
</li>
</ul>
I found expected_conditions from Selenium, to handle if element exist/found, but it's always true, because there is a message but I don't want to continue the code if a message was already in chat, that's not the main idea. And after that, I found EventFiringWebDriver, AbstractEventListener and nothing.
from selenium import webdriver as wb
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.events import EventFiringWebDriver, AbstractEventListener
from selenium.webdriver.common.keys import Keys
class MyListener(AbstractEventListener):
def before_navigate_to(self, url, driver):
print("Before navigate to %s" % url)
def after_navigate_to(self, url, driver):
print("After navigate to %s" % url)
driver = wb.Chrome('C:\python\selenium\chromedriver.exe')
wait = WebDriverWait(driver, 300)
# ef_driver = EventFiringWebDriver(driver, MyListener())
driver.implicitly_wait(15)
driver.get('https://socket.io/demos/chat/')
driver.switch_to.frame(driver.find_element_by_class_name('iframe-class'))
InputName = driver.find_element_by_xpath('/html/body/ul/li[2]/div/input')
InputName.send_keys('MyNameIsJeff')
InputName.send_keys(Keys.ENTER)

I think you can write a loop and inside the loop, get the number of "li"s inside the desired "ul". if the number of "li"s increased, you have a new message.
# get list of li inside of ul
ulList = driver.find_elements_by_xpath("//ul[#class='message-list']")
lenOfLis = len(ulList)
# wait for new message...
while (true):
ulList = driver.find_elements_by_xpath("//ul[#class='message-list']")
if (len(ulList) > lenOfLis): # you have new message
for (i in range(len(ulList)-lenOfLis)):
yourLastMessage = ulList[-1-(i-1)]
# you can do whatever you want with your last messages
LenOfLis = len(ulList) # update length of ul
# wait 15 seconds
sleep(5)

You can do some kind of listener in a loop that checks the text of your message elements to determine whether or not it is a new message, you just have to determine a set time frame that you want to wait between 'checks' -- 10 seconds, 30 seconds, 1 minute, etc.
I don't think you need an event listener for this. You can just grab the latest message, then keep checking it to see if it's different than the previous value.
from time import sleep
# get list of currently displayed messages
messages = driver.find_elements_by_xpath("//li[#class='message']")
# get text from most recent message
# depending on element order on the page, might need to do messages.last
last_message_text = messages[0].text
# do something here to trigger a new message coming in?
# wait for last_message_text to be something different
while (true):
# get list of currently displayed messages
messages = driver.find_elements_by_xpath("//li[#class='message']")
# new message has arrived if we hit this statement
if (messages[0].text != last_message_text) break;
# wait 15 seconds
sleep(15)
This example will grab the list of messages currently displayed. Then, it grabs the last received message. The code enters a loop and re-fetches the list of messages, checks the last received message, and compares its text to the last message value we saved earlier.
If the last received message is different from saved value, the loop will break, meaning a new message has arrived.
There's a few unclear things here -- the most recent message may be either the first element, or last element, in messages list. The other issue -- are you doing something on your end to trigger a new message appearing? Such as InputName.send_keys('MyNameIsJeff')? If this is the case, the order of the code may need to change a bit.
Lastly -- in your comments, you mentioned sometimes the ul is empty. I'm not really sure what this means, but if some messages are appearing and there is no HTML on the page for them, then this solution won't really work.

Python Selenium - How to handle alert that doesn't show up until after WebDriverWait finishes?

Currently I'm trying to automate an order process and at some point, if an order has been recently placed, an alert will pop-up stating, "Customer has placed an order for these same items within 2 days. Please check for duplicates or check Override Duplicate Order to continue."
I've tried the following to intercept that alert and allow the script to continue running:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.alert import Alert
from selenium.webdriver.support import expected_conditions as ExCon
from selenium.common.exceptions import TimeoutException, NoSuchElementException, NoAlertPresentException,\
UnexpectedAlertPresentException
from oleaprocs import Order
"""
Will put the following constants in their own file later
"""
GEN_NEXT_ID = "cmdNext"
SHIP_DET_METH_ID = "cboAvailItemsShip"
SHIP_DET_NEXT_ID = "cmdNext"
ORDER_REP_ID = "cboServiceRep"
EMAIL_ID = "ctl00_ctl00_main_content_main_content_ctl00_txtEmail"
SEARCH_RESULT_ID = "ctl00_ctl00_main_content_main_content_ctl00_dgResults_ctl03_lnkCustName"
ADD_ORDER_ID = "cmdAddNewOrder"
LOGIN_ID = 'ctl00_ctl00_main_content_main_content_ctl00_UserName'
PASS_ID = 'ctl00_ctl00_main_content_main_content_ctl00_password'
SIGNIN_ID = 'ctl00_ctl00_main_content_main_content_ctl00_SigninBtn'
PLACED_VIA = 'Phone'
BILL_OPTION = 'Visa x1111'
username = 'user'
password = 'pass'
customer = 'user'
items = {'Item1': '100', 'Item2': '125'} # One or many items - also noting quantity
# This setup will likely be steamlined to determine whose server to use, etc
driver = webdriver.Chrome()
driver.maximize_window()
driver.get("website-placeholder-for-security-reasons")
# Login will be separated out into a function, to allow for various logins
# log-in
driver.find_element_by_id(LOGIN_ID).send_keys(username)
driver.find_element_by_id(PASS_ID).send_keys(password)
driver.find_element_by_id(SIGNIN_ID).click()
driver.find_element_by_xpath('//a[text()="Customers"]').click()
# Type in search for user-email - separated to function later
search_email = driver.find_element_by_id(EMAIL_ID)
search_email.send_keys(customer)
search_email.send_keys(Keys.RETURN)
# Click the (hopefully) singular search result
driver.find_element_by_id(SEARCH_RESULT_ID).click()
#search_result = driver.find_element_by_id(SEARCH_RESULT_ID).click()
#search_result.click()
### Start creating new order ###
## Order Items ##
# Accept Customer Orders alert
"""
alert_obj = driver.switch_to.alert
alert_obj.accept()
"""
Alert(driver).accept()
# Click on "Add New Order"
driver.find_element_by_id(ADD_ORDER_ID).click()
order = Order(driver)
# Set Placed Via
placed_via = Select(driver.find_element_by_name('cboPlacedVia'))
placed_via.select_by_visible_text(PLACED_VIA) # could also set IDs in a dict
# Set Billing Option
bill_option = Select(driver.find_element_by_name('ctrlBillOption$cboBillOption'))
bill_option.select_by_visible_text(BILL_OPTION)
# Add items to order
order.add_items(items)
# Will likely do some assertions here at some point, for custom / printed items
driver.find_element_by_id('cmdNext').click()
## Shipping Details ##
# Select Shipping Method
# If similar order recently placed - confirm alert and try/except duplicate order override
# driver.implicitly_wait(2)
try:
print("WebDriverWait - 2 seconds")
WebDriverWait(driver, 2).until(ExCon.alert_is_present())
Alert(driver).accept()
print("Switching to Alert")
#alert_dupe = driver.switch_to.alert
# print(str(alert_dupe.text))
# alert_dupe.accept()
except TimeoutException:
print("Timeout")
except NoAlertPresentException:
print("No Alert Present")
except UnexpectedAlertPresentException:
print("Unexpected alert, yo!")
try:
driver.find_element_by_id('chkDuplicateOrderOverride').click()
except NoSuchElementException:
print("No duplicate order detected")
# Continue
driver.find_element_by_id('cmdNext').click()
## Order Group Details ##
# Check for Authorize button (I could bypass this when I implement "order_type" later
try:
driver.find_element_by_id('cmdAuthorize').click()
except NoSuchElementException:
print("No Authorize Element Found - This payment method does not require authorization")
driver.find_element_by_id('cmdPlaceOrder').click()
And the other file referenced:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
"""
Order will be used to keep track of the state of an order throughout
the order-process. This can include things like the order total, items
included, and can even get as complicated as 1:1 mirroring all of the
details of an order. This will depend on our requirements.
"""
class Order(object):
def __init__(self, driver):
self.driver = driver
# self.product_name_field = self.driver.find_element_by_id('txtProductName')
# self.quantity_field = self.driver.find_element_by_id('txtQuantity')
# self.add_item_button = self.driver.find_element_by_id('cmdAddSaveItem')
# Add items to an order
def add_items(self, items):
"""
Will need to make the quantity check in here.
"""
# product_name_field = self.driver.find_element_by_id('txtProductName')
# quantity_field = self.driver.find_element_by_id('txtQuantity')
# add_item_button = self.driver.find_element_by_id('cmdAddSaveItem')
"""
Cannot simply one-time set the above because of the references
going stale x_x - need to see if I can ignore stale-references.
"""
for item, quantity in items.items():
product_name_field = self.driver.find_element_by_id('txtProductName')
product_name_field.send_keys(item)
product_name_field.send_keys(Keys.TAB)
quantity_field = self.driver.find_element_by_id('txtQuantity')
quantity_field.send_keys(quantity)
quantity_field.send_keys(Keys.TAB)
add_item_button = self.driver.find_element_by_id('cmdAddSaveItem')
add_item_button.click()
I made a comment about the previous alert potentially not releasing resources... I have the following code a good bit earlier to handle an alert that always pops up, so I don't have to try that one:
alert_obj = driver.switch_to.alert
alert_obj.accept()
I use the driver.switch_to.alert variation here because I've tried both variations in the earlier code and neither worked.
Basically what happens when the code gets to WebDriverWait(drive, 2)... is that the browser sits there and waits 2 seconds, THEN the alert pops up. I've tried various arbitrary lengths of time and it will always wait that amount of time, then show the alert. I feel like I'm being trolled here :|
Here is the error info I receive after this happens:
Traceback (most recent call last):
File "C:\Users\user\Documents\Projects\Selenium\proj.py", line 136, in <module>
driver.find_element_by_id('cmdAuthorize').click()
File "C:\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 353, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 957, in find_element
'value': value})['value']
File "C:\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 314, in execute
self.error_handler.check_response(response)
File "C:\Python\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 241, in check_response
raise exception_class(message, screen, stacktrace, alert_text)
selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: None
Message: unexpected alert open: {Alert text : Customer has placed an order for these same items within 2 days. Please check for duplicates or check Override Duplicate Order to continue.}
(Session info: chrome=66.0.3359.181)
(Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Windows NT 10.0.16299 x86_64)
Any idea why this might be occurring? If I recall correctly, I don't believe we really have any alerts that also have checkboxes or input, so I wouldn't be 100% opposed to just 100% blocking alerts from being acknowledged by Selenium (or auto-accepting them), but I would like to avoid that if I can.
The code runs completely fine and places an order all the way through properly if no order has been placed recently (or I use a different account to test the order creation under over and over).
The manual steps from the point of adding items (just before the problematic page) would be:
1) Enter item name, then enter the item quantity
2) Do this for each item in the "items" dict
3) Hit the next button
4) Next page with shipping options "loads"
5) Issue occurs if an order was recently placed -- the WebDriverWait seems to be halting the loading of the page? It seems like this is the case since the alert will not show up until after the WebDriverWait time runs out.
I'm not really doing anything special after telling Selenium to hit the next button to reach the next page. Nothing different from the previous "hit the next button" actions, and the page shouldn't really be behaving any different than the other pages - even though the Alert is apparently just not having it x_x

I didn't have a "click next button" before the alert check -- the alert doesn't pop up until after I try and submit the shipping details first. Reason why it seemed to continue on to further steps after the alert-check failure.
Outline the process beforehand, kids.