So I have this project: It is a website with multiple WebElements on it. I find those WebElements by their class name (they all have the same one obviously) and then iterate through them to click on each of them. After clicking on them I have to click on another button "next". Some of them then open a website in a new tab (others don't). I then immediately close the newly opened tab and try to iterate through the next element when I get the StaleElementReferenceException.
Don't get me wrong here, I know what a StaleElementReferenceException is, I just don't know why it occurs. The DOM of the initial website doesn't seem to change and more importantly: The WebElement I'm trying to reach in the next iteration is still known so I can print it out, but not click on it.
I have tried working around this issue by creating a new class CustomElement to permanently "save" the found WebElements to be able to reach them after the DOM has changed but that also doesn't seem to be working.
Whatever here's some code for you guys:
def myMethod():
driver.get("https://initialwebsite.com")
time.sleep(1)
scrollToBottom() #custom Method to scroll to the bottom of the website to make sure I find all webelemnts
ways = driver.find_elements_by_class_name("sc-dYzWWc")
waysCounter = 1
for way in ways:
# print("clicking: " + str(way)) ##this will get executed even if there was a new tab opened in the previous iteration....
driver.execute_script("arguments[0].click();", way)
# print("clicked: " + str(way)) ##...but this won't get executed
try:
text = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(entryWaysCounter) + "]/div[1]/div/div/div[1]").text
except:
waysCounter += 1
text = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(entryWaysCounter) + "]/div[1]/div/div/div[1]").text
methode = None
#big bunch of if and else statements to give methode a specific number based on what text reads
print(methode)
weiterButton = driver.find_element_by_xpath(
"/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(
entryWaysCounter) + "]/div[2]/div/div/div/div/div/div[2]/button[2]")
try:
driver.execute_script("arguments[0].click();", weiterButton)
except:
pass
if (methode == 19):
time.sleep(0.2)
try:
driver.switch_to.window(driver.window_handles[1])
driver.close()
time.sleep(0.5)
driver.switch_to.window(driver.window_handles[0])
time.sleep(0.5)
except:
pass
waysCounter += 1
time.sleep(0.5)
And for those who are curious here's the workaround class I set up:
class CustomElement:
def __init__(self, text, id, location):
self.text = text
self.id = id
self.location = location
def __str__(self):
return str(str(self.text) + " \t" + str(self.id) + " \t" + str(self.location))
def storeWebElements(seleniumElements):
result = []
for elem in seleniumElements:
result.append(CustomElement(elem.text, elem.id, elem.location))
return result
I tried then working with the id and "re-finding" the WebElement ("way") by id but apparently the id that gets saved is a completely different id.
So what can I say I really tried my best, searched nearly every forum but didn't come up with a good soluation, I really hope you got one for me :)
Thanks!
Are you crawling links? If so then you want to save the destination, not the element.
Otherwise you could force the link to open in a new window (perhaps like https://stackoverflow.com/a/19152396/1387701), switch to that wind, parse the page, close the page and still have the original window open.
Related
I have the following code:
for button in buttons:
ActionChains(driver).move_to_element(button).perform()
time.sleep(2)
button.click()
time.sleep(2)
try:
wait_button.until(EC.presence_of_element_located((By.XPATH,'//div/h2')))
time.sleep(2)
name = driver.find_element_by_xpath('//div/h2').text
except:
wait_button.until(EC.presence_of_element_located((By.XPATH,'//span[#id="chat-header-title"]')))
time.sleep(2)
name = driver.find_element_by_xpath('//span[#id="chat-header-title"]').text
def pull_ul() -> list:
chat_frame = driver.find_element_by_xpath("//iframe[starts-with(#id, 'experience-container-')]")
driver.switch_to.frame(chat_frame)
wait_button.until(EC.presence_of_element_located((By.XPATH,'//ul[#aria-label="Chat content"]')))
the_ul =driver.find_element(By.XPATH,'//ul[#aria-label="Chat content"]')
new_lis =the_ul.find_elements(By.TAG_NAME,'li')
return new_lis
def pull_ul_again() -> list:
the_ul =driver.find_element(By.XPATH,'//ul[#aria-label="Chat content"]')
new_lis_2 =the_ul.find_elements(By.TAG_NAME,'li')
return new_lis_2
lis = pull_ul()
print(f"Archiving Chat with {name} ...\n")
print("this is len lis: ",len(lis), "for " + name)
And here is what the terminal shows:
As you can see, the code actually does run past the line that threw up the error, how is this possible? Also, why would this be happening , I successfully ran the code multiple times and suddenly it starts throwing the follwing error?
The loop that you're doing,
for button in buttons:
ActionChains(driver).move_to_element(button).perform()
...
is causing your StaleElementReferenceException because inside that you have driver.switch_to.frame(chat_frame), but you never switch back to default content or to the parent frame, which means that your buttons won't exist in that level, and so Selenium throws StaleElementReferenceException as a result. That could also happen if you navigate away from the page (which might also be happening depending on the button clicks).
If you ever switch frames or leave pages, you have to re-find elements in order to interact with them.
I am writing a test, where i create an ip pool for the application , and then in the next step, delete it.
The function is as follows:
def remove_passed_ip(self, ip):
""" Find and ip and delete it
Args:
ip (str): the name of ip to click and delete
Returns:
webelement
"""
index = -1
try:
ipDelBtnList = self.wait_for_presence_of_all_elements(self.onboarding_select_address_pools_delete_btn_list)
ipList = self.wait_for_presence_of_all_elements(self.onboarding_select_address_pools_list)
time.sleep(5)
except:
self.log.info("dasdda")
else:
for ips in ipList:
index +=1
temp = ips.text.split('/')
if str(ip)==str(temp[0]):
ipHandle = ipDelBtnList[index]
time.sleep(5)
ipHandle.click()
time.sleep(15)
The delete action works fine and the created ip is deleted, but after this when the test ends, it gives the error as
Message: Element is no longer attached to the DOM
Please provide with any pointers to resolve this issue. If any other clarification is required regarding the question, please let me know.
This is because after the delete the DOM changes and all 'references' that Selenium has with the browser are lost.
To solve this, you need to get the element/s again from the page after the previous one is deleted (or any other action that determines the DOM to be changed).
I am hoping someone may be able to point out the error I am making; it is probably very straight forward!
What I am trying to do is run some code previous what I have shown below, then when I get to this point I need to get it to hold for the 600 seconds and then reload the download page:
try:
# Clicks OK in Download Requested window
driver.implicitly_wait(10)
ClickOkay = driver.find_element_by_css_selector("._42ft._42fu.layerCancel.uiOverlayButton.selected._42g-._42gy")
ClickOkay.click()
except:
print("error 2")
# Wait Time
# time.sleep(600) # Allow Facebook to compile the archive
# Reload Settings page
GoToURL('https://www.facebook.com/' + 'settings', driver)
# Goes back through to Download page
link = driver.find_element_by_link_text('Download a copy')
link.click()
At this point if the archive has finished being created then the button changes from Start Archive to Download Archive. However depending on the size of the profile the time taken to compile the archive varies. so what i was attempting (with the code below and a couple of attempts with the if and while arguments) was to get it to check if the button exists and if not go back and wait 300 seconds before trying again. Once the button appears it will then continue on to download using additional code.
try:
print("Checking if button exists")
DownloadArchive = driver.find_elements_by_css_selector("._42ft._42fu.selected._42gz._42gy")
print(DownloadArchive.count())
if(DownloadArchive.count() > 0):
print("button exists")
else:
print("button does not exist")
# Button to initiate password entry popup window
#driver.implicitly_wait(10)
#while (DownloadArchive = driver.find_element_by_css_selector("._42ft._42fu.selected._42gz._42gy")):
# if (DownloadArchive = True):
# DownloadArchive.click()
# print("wait")
# else:time.sleep(300)
Thanks in advance, James
You're mixing the assignment operator (=) with the equal operator (==).
So it should be:
while (DownloadArchive == driver.find_element_by_css_selector("._42ft._42fu.selected._42gz._42gy")):
if (DownloadArchive == True):
Or just:
while DownloadArchive == driver.find_element_by_css_selector("._42ft._42fu.selected._42gz._42gy"):
Hope it helps!
I have this automation tool I've built with Selenium 2 and Browsermob proxy that works quite well for most of what I need. However, I've run into a snag on capturing network traffic.
I basically want to capture the har that a click provides before the page redirects. For example, I have an analytics call happening on the click that I want to capture, then another analytics call on the page load that I don't want to capture.
All of my attempts currently capture the har too late, so I see both the click analytics call and the page load one. Is there any way to get this working? I've included my current relevant code sections below
METHODS INSIDE HELPER CLASS
class _check_for_page_load(object):
def __init__(self, browser, parent):
self.browser = browser
self.maxWait = 5
self.parent = parent
def __enter__(self):
self.old_page = self.browser.find_element_by_tag_name('html')
def wait_for(self,condition_function):
start_time = time.time()
while time.time() < start_time + self.maxWait:
if condition_function():
return True
else:
time.sleep(0.01)
raise Exception(
'Timeout waiting for {}'.format(condition_function.__name__)
)
def page_has_loaded(self):
new_page = self.browser.find_element_by_tag_name('html')
###self.parent.log("testing ---- " + str(new_page.id) + " " + str(self.old_page.id))
return new_page.id != self.old_page.id
def __exit__(self, *_):
try:
self.wait_for(self.page_has_loaded)
except:
pass
def startNetworkCalls(self):
if self._p != None:
self._p.new_har("Step"+str(self._currStep))
def getNetworkCalls(self, waitForTrafficToStop = True):
if self._p != None:
if waitForTrafficToStop:
self._p.wait_for_traffic_to_stop(5000, 30*1000);
return self._p.har
else:
return "{}"
def click(self, selector):
''' clicks on an element '''
self.log("Clicking element '" + selector + "'")
el = self.findEl(selector)
traffic = ""
with self._check_for_page_load(self._d, self):
try:
self._curr_window = self._d.window_handles[0]
el.click()
except:
actions = ActionChains(self._d);
actions.move_to_element(el).click().perform()
traffic = self.getNetworkCalls(False)
try:
popup = self._d.switch_to.alert
if popup != None:
popup.dismiss()
except:
pass
try:
window_after = self._d.window_handles[1]
if window_after != self._curr_window:
self._d.close()
self._d.switch_to_window(self._curr_window)
except:
pass
return traffic
INSIDE FILE THAT RUNS MULTIPLE SELENIUM ACTIONS
##inside a for loop, we get an action that looks like "click('#selector')"
util.startNetworkCalls()
if action.startswith("click"):
temp_traffic = eval(action)
if temp_traffic == "":
temp_traffic = util.getNetworkCalls()
traffic = json.dumps(temp_traffic, sort_keys=True) ##gives json har info that is saved later
You can see from these couple snippets that I initiate the "click" function which returns network traffic. Inside the click function, you can see it references the class "_check_for_page_load". However, the first time it reaches this line:
###self.parent.log("testing ---- " + str(new_page.id) + " " + str(self.old_page.id))
The log (when enabled) shows that the element ids don't match on the first time it logs, indicating the page load has already started to happen. I'm pretty stuck right now as I've tried everything I can think of to try to accomplish this functionality.
I found a solution to my own question - though it isn't perfect. I told my network calls to capture headers:
def startNetworkCalls(self):
if self._p != None:
self._p.new_har("Step"+str(self._currStep),{"captureHeaders": "true"})
Then, when I retrieve the har data, I can look for the "Referer" header and compare that with the page that was initially loaded (before the redirect from the click). From there, I can split the har into two separate lists of network calls to further process later.
This works for my needs, but it isn't perfect. Some things, like image requests, sometimes get the same referrer that the previous page's url matched, so the splitting puts those into the first bucket rather than the appropriate second bucket. However, since I'm more interested in requests that aren't on the same domain, this isn't really an issue.
I have a comprehensive list of Australian postcodes, and I need to use the search function of a specific site to get corresponding remoteness codes. I created a Python script to do that, and it does it efficiently.
Except that, at a seemingly random time during the iteration, it throws a 'Modal dialog present' exception. The problem is, I see no dialog! The webpage looks as usual, and I can interact normally with it with my mouse. What could be the problem and is there a solution?
browser = webdriver.Firefox() # Get local session of firefox
browser.set_page_load_timeout(30)
browser.get("http://www.doctorconnect.gov.au/internet/otd/Publishing.nsf/Content/locator") # Load page
assert "Locator" in browser.title
search = browser.find_element_by_name("Search") # Find the query box
ret = browser.find_element_by_id("searchButton")
doha_addr = []
doha_ra = []
for i in search_string_list:
search.send_keys(i)
ret.send_keys(Keys.RETURN)
addr = browser.find_element_by_xpath("//table/tbody/tr[2]/td[2]")
doha_addr.append(addr.text)
ra = browser.find_element_by_xpath("//table/tbody/tr[4]/td[2]")
doha_ra.append(ra.text)
try:
browser.find_element_by_xpath("//html/body/div/div[3]/div/div/div/div/div/div[7]/div/div[13]/div[1]").click()
except:
pass
search.clear()
I seem to have caught a glimpse of a popup dialog that shows up and hides itself while my script was running. Anyway, this becomes irrelevant with a while switch and a try/except clause... :7D
doha_ra = {}
for i in search_string_list:
switch = True
while switch == True:
search.send_keys(i)
ret.send_keys(Keys.RETURN)
try:
addr = browser.find_element_by_xpath("//table/tbody/tr[2]/td[2]")
ra = browser.find_element_by_xpath("//table/tbody/tr[4]/td[2]")
doha_ra[addr.text] = ra.text
switch = False
except:
pass
search.clear()