My selenium code checks for a completed subroutine to be done by waiting on the site's title to change which worked perfectly. Code looks like this:
waitUntilDone = WebDriverWait(session, 15).until(EC.title_contains(somestring))
However, this can fail sometimes since the site's landing page changes after manual website visits. The server remembers where you left off. This forces me to check for an alternate condition (website title = "somestring2).
Here is what I came up with so far (also works as far as I can tell):
try:
waitUntilDone = WebDriverWait(session, 15).until(EC.title_contains(somestring)) # the old condition
except:
try:
waitUntilDone = WebDriverWait(session, 15).until(EC.title_contains(somestring2)) # the new other condition which is also valid
except:
print "oh crap" # we should never reach this point
Either one of these conditions is always true. I don't know which one thou.
Is there any way to include an "OR" inside these waits or make the try/except block look nicer?
Looks like selenium will let you do this by creating your own class. Check out the documentation here: http://selenium-python.readthedocs.io/waits.html
Here's a quick example for your case. Note the key is to have a method named __call__ in your class that defines the check you want. Selenium will call that function every 500 milliseconds until it returns True or some not null value.
class title_is_either(object):
def __init__(self, locator, string1, string2):
self.locator = locator
self.string1 = string1
self.string2 = string2
def __call__(self, driver):
element = driver.find_element(*self.locator) # Finding the referenced element
title = element.text
if self.string1 in title or self.string2 in title
return element
else:
return False
# Wait until an element with id='ID-of-title' contains text from one of your two strings
somestring = "Title 1"
somestring2 = "Title 2"
wait = WebDriverWait(driver, 10)
element = wait.until(title_is_either((By.ID, 'ID-of-title'), somestring, somestring2))
Related
https://www.testim.io/blog/how-to-wait-for-a-page-to-load-in-selenium/
I want to understand how "Explicit Wait" is implemented under selenium. Can you show some example python code to demonstrate how selenium's "Explicit Wait" is implemented without using selenium?
Is the logic just wait for some time, then test for if an element is available, if not wait more time, check again, ..., until the element is available?
To understand explicit waits better, I found I needed to understand what's happening in the following:
The expected condition function.
The WebdriverWait.until method
The simplest expected condition is presence_of_element_located. It is just a wrapper around driver.find_element()
def presence_of_element_located(locator):
def _predicate(driver):
return driver.find_element(*locator)
return _predicate
Other expected conditions will check the element for certain conditions but I'll keep this example simple.
The result of presence_of_element_located is passed into WebDriverWait.until, typically examples will look like this:
wait = WebDriverWait(driver, timeout=30)
element = wait.until(ec.presence_of_element_located((By.ID, "my_id")))
When I break down what's happening in the above it starts to become a little more clear.
a_callable_method = ec.presence_of_element_located((By.ID, "my_id"))
wait = WebDriverWait(driver, timeout=30)
element = wait.until(a_callable_method)
WebDriverWait.until is simply a while loop calling a_callable_method you passed in. The method/function we are passing in always takes driver as an argument.
def until(self, method, message: str = ""):
"""Calls the method provided with the driver as an argument until the \
return value does not evaluate to ``False``.
:param method: callable(WebDriver)
:param message: optional message for :exc:`TimeoutException`
:returns: the result of the last call to `method`
:raises: :exc:`selenium.common.exceptions.TimeoutException` if timeout occurs
"""
screen = None
stacktrace = None
end_time = time.monotonic() + self._timeout
while True:
try:
value = method(self._driver)
if value:
return value
except self._ignored_exceptions as exc:
screen = getattr(exc, 'screen', None)
stacktrace = getattr(exc, 'stacktrace', None)
time.sleep(self._poll)
if time.monotonic() > end_time:
break
raise TimeoutException(message, screen, stacktrace)
In other words, explicit waiting is just a retry loop until a certain condition is met, or the timer runs out. In our example it's trying to find the element with id="my_id". If/when the element is found, it will be returned, otherwise a TimeoutException will be raised.
The real power of explicit wait starts to shine when looking at some of the other expected conditions:
def visibility_of(element):
def _predicate(_):
return _element_if_visible(element)
return _predicate
def _element_if_visible(element, visibility=True):
return element if element.is_displayed() == visibility else False
Using visibility_of will look for the element but then also check to see if the element has the condition consistent with being visible.
So I have this project: It is a website with multiple WebElements on it. I find those WebElements by their class name (they all have the same one obviously) and then iterate through them to click on each of them. After clicking on them I have to click on another button "next". Some of them then open a website in a new tab (others don't). I then immediately close the newly opened tab and try to iterate through the next element when I get the StaleElementReferenceException.
Don't get me wrong here, I know what a StaleElementReferenceException is, I just don't know why it occurs. The DOM of the initial website doesn't seem to change and more importantly: The WebElement I'm trying to reach in the next iteration is still known so I can print it out, but not click on it.
I have tried working around this issue by creating a new class CustomElement to permanently "save" the found WebElements to be able to reach them after the DOM has changed but that also doesn't seem to be working.
Whatever here's some code for you guys:
def myMethod():
driver.get("https://initialwebsite.com")
time.sleep(1)
scrollToBottom() #custom Method to scroll to the bottom of the website to make sure I find all webelemnts
ways = driver.find_elements_by_class_name("sc-dYzWWc")
waysCounter = 1
for way in ways:
# print("clicking: " + str(way)) ##this will get executed even if there was a new tab opened in the previous iteration....
driver.execute_script("arguments[0].click();", way)
# print("clicked: " + str(way)) ##...but this won't get executed
try:
text = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(entryWaysCounter) + "]/div[1]/div/div/div[1]").text
except:
waysCounter += 1
text = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(entryWaysCounter) + "]/div[1]/div/div/div[1]").text
methode = None
#big bunch of if and else statements to give methode a specific number based on what text reads
print(methode)
weiterButton = driver.find_element_by_xpath(
"/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(
entryWaysCounter) + "]/div[2]/div/div/div/div/div/div[2]/button[2]")
try:
driver.execute_script("arguments[0].click();", weiterButton)
except:
pass
if (methode == 19):
time.sleep(0.2)
try:
driver.switch_to.window(driver.window_handles[1])
driver.close()
time.sleep(0.5)
driver.switch_to.window(driver.window_handles[0])
time.sleep(0.5)
except:
pass
waysCounter += 1
time.sleep(0.5)
And for those who are curious here's the workaround class I set up:
class CustomElement:
def __init__(self, text, id, location):
self.text = text
self.id = id
self.location = location
def __str__(self):
return str(str(self.text) + " \t" + str(self.id) + " \t" + str(self.location))
def storeWebElements(seleniumElements):
result = []
for elem in seleniumElements:
result.append(CustomElement(elem.text, elem.id, elem.location))
return result
I tried then working with the id and "re-finding" the WebElement ("way") by id but apparently the id that gets saved is a completely different id.
So what can I say I really tried my best, searched nearly every forum but didn't come up with a good soluation, I really hope you got one for me :)
Thanks!
Are you crawling links? If so then you want to save the destination, not the element.
Otherwise you could force the link to open in a new window (perhaps like https://stackoverflow.com/a/19152396/1387701), switch to that wind, parse the page, close the page and still have the original window open.
I have this automation tool I've built with Selenium 2 and Browsermob proxy that works quite well for most of what I need. However, I've run into a snag on capturing network traffic.
I basically want to capture the har that a click provides before the page redirects. For example, I have an analytics call happening on the click that I want to capture, then another analytics call on the page load that I don't want to capture.
All of my attempts currently capture the har too late, so I see both the click analytics call and the page load one. Is there any way to get this working? I've included my current relevant code sections below
METHODS INSIDE HELPER CLASS
class _check_for_page_load(object):
def __init__(self, browser, parent):
self.browser = browser
self.maxWait = 5
self.parent = parent
def __enter__(self):
self.old_page = self.browser.find_element_by_tag_name('html')
def wait_for(self,condition_function):
start_time = time.time()
while time.time() < start_time + self.maxWait:
if condition_function():
return True
else:
time.sleep(0.01)
raise Exception(
'Timeout waiting for {}'.format(condition_function.__name__)
)
def page_has_loaded(self):
new_page = self.browser.find_element_by_tag_name('html')
###self.parent.log("testing ---- " + str(new_page.id) + " " + str(self.old_page.id))
return new_page.id != self.old_page.id
def __exit__(self, *_):
try:
self.wait_for(self.page_has_loaded)
except:
pass
def startNetworkCalls(self):
if self._p != None:
self._p.new_har("Step"+str(self._currStep))
def getNetworkCalls(self, waitForTrafficToStop = True):
if self._p != None:
if waitForTrafficToStop:
self._p.wait_for_traffic_to_stop(5000, 30*1000);
return self._p.har
else:
return "{}"
def click(self, selector):
''' clicks on an element '''
self.log("Clicking element '" + selector + "'")
el = self.findEl(selector)
traffic = ""
with self._check_for_page_load(self._d, self):
try:
self._curr_window = self._d.window_handles[0]
el.click()
except:
actions = ActionChains(self._d);
actions.move_to_element(el).click().perform()
traffic = self.getNetworkCalls(False)
try:
popup = self._d.switch_to.alert
if popup != None:
popup.dismiss()
except:
pass
try:
window_after = self._d.window_handles[1]
if window_after != self._curr_window:
self._d.close()
self._d.switch_to_window(self._curr_window)
except:
pass
return traffic
INSIDE FILE THAT RUNS MULTIPLE SELENIUM ACTIONS
##inside a for loop, we get an action that looks like "click('#selector')"
util.startNetworkCalls()
if action.startswith("click"):
temp_traffic = eval(action)
if temp_traffic == "":
temp_traffic = util.getNetworkCalls()
traffic = json.dumps(temp_traffic, sort_keys=True) ##gives json har info that is saved later
You can see from these couple snippets that I initiate the "click" function which returns network traffic. Inside the click function, you can see it references the class "_check_for_page_load". However, the first time it reaches this line:
###self.parent.log("testing ---- " + str(new_page.id) + " " + str(self.old_page.id))
The log (when enabled) shows that the element ids don't match on the first time it logs, indicating the page load has already started to happen. I'm pretty stuck right now as I've tried everything I can think of to try to accomplish this functionality.
I found a solution to my own question - though it isn't perfect. I told my network calls to capture headers:
def startNetworkCalls(self):
if self._p != None:
self._p.new_har("Step"+str(self._currStep),{"captureHeaders": "true"})
Then, when I retrieve the har data, I can look for the "Referer" header and compare that with the page that was initially loaded (before the redirect from the click). From there, I can split the har into two separate lists of network calls to further process later.
This works for my needs, but it isn't perfect. Some things, like image requests, sometimes get the same referrer that the previous page's url matched, so the splitting puts those into the first bucket rather than the appropriate second bucket. However, since I'm more interested in requests that aren't on the same domain, this isn't really an issue.
I am trying to create my own Selenium class with custom functions so that test scripting will become a bit more intuitive and more robust in some scenarios, for my taste at least. One of my current tasks is to wrap all Selenium expected conditions (described here) so that eventually I will have a single function that looks something like that:
def waitForElement(self, elementName, expectedCondition, searchBy)
Where:
elementName - the name of the element I am looking for. That could be id, name, xpath, css, etc...
expectedCondition - this is where the Selenium expected condition is set. So that can be: element_to_be_clickable, visibility_of_element_located, etc...
The above function internally implements the standard Selenium WebDriverWait as follows:
try:
if expectedCondition == "element_to_be_clickable":
element = WebDriverWait(self.driver, defaultWait).until(EC.element_to_be_clickable((searchBy, elementName)))
elif expectedCondition == "visibility_of_element_located":
element = WebDriverWait(self.driver, defaultWait).until(EC.visibility_of_element_located((searchBy, elementName)))
All is good but I have a bit of trouble with passing the searchBy as a parameter. To remind, searchBy can be one of the following:
By.ID
By.NAME
By.CLASS_NAME
...
When I call this wrapper function from the main code, I do it with the below line:
self.waitForElement("elementName", "element_to_be_clickable", "By.NAME", "test")
So all parameters are passed as strings which is fine for everything except of the searchBy part.
So my question is: How can I pass the By.X part as a parameter to my function?
Hopefully I was able to describe my situation well. If I wasn't I will be happy to clarify.
Eventually I was able to solve this problem after asking this question. So in order to obtain the desired functionality, the above-mentioned method will look like this:
def waitForElement(self, elementName, expectedCondition, searchBy):
try:
if expectedCondition == "element_to_be_clickable":
element = WebDriverWait(self.driver, self.defaultWait).until(EC.element_to_be_clickable((getattr(By, searchBy), elementName)))
elif expectedCondition == "visibility_of_element_located":
element = WebDriverWait(self.driver, self.defaultWait).until(EC.visibility_of_element_located((getattr(By, searchBy), elementName)))
. . .
So it can be called like this:
self.waitForElement("elementName", "element_to_be_clickable", "NAME")
You can start like this:
Create main findElement method, that accepts By instance:
WebElement findElement(By by) {
try {
return driver.findElement(by);
} catch (NoSuchElementException e) {
logException("ERROR: Could not find - '" + by + "' on page " + driver.getCurrentUrl());
throw e;
}
Then create wait method, that uses the findElement method:
WebElement findElementAndWaitElementToPresent(By by, int timeoutInSeconds) {
try {
WebDriverWait wait = new WebDriverWait(driver, timeoutInSeconds);
wait.until(ExpectedConditions.presenceOfElementLocated(by));
return findElement(by);
} catch (TimeoutException e) {
logException("ERROR: Element is not present: " + by + " on page " + driver.getCurrentUrl());
throw e;
}
}
And pass By instance to the findElementAndWaitElementToPresent method:
findElementAndWaitElementToPresent(By.xpath("//your_xpath"), 10);
or
findElementAndWaitElementToPresent(By.name("name"), 10);
something like this is done in the framework i am working on/with
Issue with your code is that you are converting "By" datatype to string.
Simple way would be to pass it without quotes like below:
self.waitForElement("elementName", "element_to_be_clickable", By.NAME, "test")
Only additional thing you need to do is to add import as below for By in python module from where you are calling above method:
from selenium.webdriver.common.by import By
I have a comprehensive list of Australian postcodes, and I need to use the search function of a specific site to get corresponding remoteness codes. I created a Python script to do that, and it does it efficiently.
Except that, at a seemingly random time during the iteration, it throws a 'Modal dialog present' exception. The problem is, I see no dialog! The webpage looks as usual, and I can interact normally with it with my mouse. What could be the problem and is there a solution?
browser = webdriver.Firefox() # Get local session of firefox
browser.set_page_load_timeout(30)
browser.get("http://www.doctorconnect.gov.au/internet/otd/Publishing.nsf/Content/locator") # Load page
assert "Locator" in browser.title
search = browser.find_element_by_name("Search") # Find the query box
ret = browser.find_element_by_id("searchButton")
doha_addr = []
doha_ra = []
for i in search_string_list:
search.send_keys(i)
ret.send_keys(Keys.RETURN)
addr = browser.find_element_by_xpath("//table/tbody/tr[2]/td[2]")
doha_addr.append(addr.text)
ra = browser.find_element_by_xpath("//table/tbody/tr[4]/td[2]")
doha_ra.append(ra.text)
try:
browser.find_element_by_xpath("//html/body/div/div[3]/div/div/div/div/div/div[7]/div/div[13]/div[1]").click()
except:
pass
search.clear()
I seem to have caught a glimpse of a popup dialog that shows up and hides itself while my script was running. Anyway, this becomes irrelevant with a while switch and a try/except clause... :7D
doha_ra = {}
for i in search_string_list:
switch = True
while switch == True:
search.send_keys(i)
ret.send_keys(Keys.RETURN)
try:
addr = browser.find_element_by_xpath("//table/tbody/tr[2]/td[2]")
ra = browser.find_element_by_xpath("//table/tbody/tr[4]/td[2]")
doha_ra[addr.text] = ra.text
switch = False
except:
pass
search.clear()