I have a comprehensive list of Australian postcodes, and I need to use the search function of a specific site to get corresponding remoteness codes. I created a Python script to do that, and it does it efficiently.
Except that, at a seemingly random time during the iteration, it throws a 'Modal dialog present' exception. The problem is, I see no dialog! The webpage looks as usual, and I can interact normally with it with my mouse. What could be the problem and is there a solution?
browser = webdriver.Firefox() # Get local session of firefox
browser.set_page_load_timeout(30)
browser.get("http://www.doctorconnect.gov.au/internet/otd/Publishing.nsf/Content/locator") # Load page
assert "Locator" in browser.title
search = browser.find_element_by_name("Search") # Find the query box
ret = browser.find_element_by_id("searchButton")
doha_addr = []
doha_ra = []
for i in search_string_list:
search.send_keys(i)
ret.send_keys(Keys.RETURN)
addr = browser.find_element_by_xpath("//table/tbody/tr[2]/td[2]")
doha_addr.append(addr.text)
ra = browser.find_element_by_xpath("//table/tbody/tr[4]/td[2]")
doha_ra.append(ra.text)
try:
browser.find_element_by_xpath("//html/body/div/div[3]/div/div/div/div/div/div[7]/div/div[13]/div[1]").click()
except:
pass
search.clear()
I seem to have caught a glimpse of a popup dialog that shows up and hides itself while my script was running. Anyway, this becomes irrelevant with a while switch and a try/except clause... :7D
doha_ra = {}
for i in search_string_list:
switch = True
while switch == True:
search.send_keys(i)
ret.send_keys(Keys.RETURN)
try:
addr = browser.find_element_by_xpath("//table/tbody/tr[2]/td[2]")
ra = browser.find_element_by_xpath("//table/tbody/tr[4]/td[2]")
doha_ra[addr.text] = ra.text
switch = False
except:
pass
search.clear()
Related
So I have this project: It is a website with multiple WebElements on it. I find those WebElements by their class name (they all have the same one obviously) and then iterate through them to click on each of them. After clicking on them I have to click on another button "next". Some of them then open a website in a new tab (others don't). I then immediately close the newly opened tab and try to iterate through the next element when I get the StaleElementReferenceException.
Don't get me wrong here, I know what a StaleElementReferenceException is, I just don't know why it occurs. The DOM of the initial website doesn't seem to change and more importantly: The WebElement I'm trying to reach in the next iteration is still known so I can print it out, but not click on it.
I have tried working around this issue by creating a new class CustomElement to permanently "save" the found WebElements to be able to reach them after the DOM has changed but that also doesn't seem to be working.
Whatever here's some code for you guys:
def myMethod():
driver.get("https://initialwebsite.com")
time.sleep(1)
scrollToBottom() #custom Method to scroll to the bottom of the website to make sure I find all webelemnts
ways = driver.find_elements_by_class_name("sc-dYzWWc")
waysCounter = 1
for way in ways:
# print("clicking: " + str(way)) ##this will get executed even if there was a new tab opened in the previous iteration....
driver.execute_script("arguments[0].click();", way)
# print("clicked: " + str(way)) ##...but this won't get executed
try:
text = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(entryWaysCounter) + "]/div[1]/div/div/div[1]").text
except:
waysCounter += 1
text = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(entryWaysCounter) + "]/div[1]/div/div/div[1]").text
methode = None
#big bunch of if and else statements to give methode a specific number based on what text reads
print(methode)
weiterButton = driver.find_element_by_xpath(
"/html/body/div[1]/div/div[2]/div/div[1]/div/div[7]/div[2]/div[" + str(
entryWaysCounter) + "]/div[2]/div/div/div/div/div/div[2]/button[2]")
try:
driver.execute_script("arguments[0].click();", weiterButton)
except:
pass
if (methode == 19):
time.sleep(0.2)
try:
driver.switch_to.window(driver.window_handles[1])
driver.close()
time.sleep(0.5)
driver.switch_to.window(driver.window_handles[0])
time.sleep(0.5)
except:
pass
waysCounter += 1
time.sleep(0.5)
And for those who are curious here's the workaround class I set up:
class CustomElement:
def __init__(self, text, id, location):
self.text = text
self.id = id
self.location = location
def __str__(self):
return str(str(self.text) + " \t" + str(self.id) + " \t" + str(self.location))
def storeWebElements(seleniumElements):
result = []
for elem in seleniumElements:
result.append(CustomElement(elem.text, elem.id, elem.location))
return result
I tried then working with the id and "re-finding" the WebElement ("way") by id but apparently the id that gets saved is a completely different id.
So what can I say I really tried my best, searched nearly every forum but didn't come up with a good soluation, I really hope you got one for me :)
Thanks!
Are you crawling links? If so then you want to save the destination, not the element.
Otherwise you could force the link to open in a new window (perhaps like https://stackoverflow.com/a/19152396/1387701), switch to that wind, parse the page, close the page and still have the original window open.
I have the following program in python 2.7 to launch a web page and do an activity. I have parsed the HTML DOM of a web page and was able to make this work when launched via PyCharm IDE or via an Exe created with PyInstaller or via a bat file that wraps an Exe.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By
import time
import sys
def main():
try:
mgmt_console_server_name = sys.argv[1]
user_session_to_terminate = sys.argv[2]
# Phase 1: Launch a browser and open the management console page
# Using IE to access web
driver = webdriver.Ie('IEDriverServer32.exe')
return_code = 0
# Open the management console page.
driver.get("http://" + mgmt_console_server_name + ":8183/mgmt/security/login.html")
# Phase 2: Enter the LDAP credentials and hit enter
# Select the username and the password text boxes
username = driver.find_element_by_class_name('gwt-TextBox')
password = driver.find_element_by_class_name('gwt-PasswordTextBox')
# Send username information
username.send_keys('sa-tcnxbot-gds')
# Send password information
password.send_keys('zzzzzzzzz')
# Login to Teamcenter management console with the above set credentials.
signinbutton = driver.find_element_by_class_name('GFHWBTICII')
signinbutton.click()
# Phase 3: Click on the 'TcServer' component to reach the UI where we can find and kill the user sessions.
# We are now in the main page. We need to wait till we find an XML element by the ID
# 'Console'. This signifies that the page has loaded. And we wait at the maximum of 20 seconds.
WebDriverWait(driver, 20).until(ec.presence_of_element_located((By.ID, "Console")))
# We find the XML element whose InnerHTML is 'Tc Server'.
tcserverelement = driver.find_element_by_xpath("//div[text()='Tc Server']")
# We then find its immediate parent, which is the clickable element and finally click it.
tcserverclickableelement = tcserverelement.find_element_by_xpath('..')
tcserverclickableelement.click()
# Phase 4: We enter the user id whose session is to be terminated in the search field and hit the search.
WebDriverWait(driver, 20).until(ec.presence_of_element_located((By.XPATH, "//div[text()='Assigned User:']")))
# We are at the HTML element that has the text 'Assigned User:'
assigneduserelement = driver.find_element_by_xpath("//div[text()='Assigned User:']")
# print(assigneduserelement)
# An immediate parent of this HTML element is the TD element.
assigneduserelementparent = assigneduserelement.find_element_by_xpath('..')
# print(assigneduserelementparent)
# An immediate sibling of this TD HTML element is another TD element which contains INPUT HTML element.
assigneduserelementparent.find_element_by_xpath("//TD/following-sibling::TD")
# print(assigneduserelementparent_sibling)
# assignedusertextbox = assigneduserelementparent_sibling.find_elements_by_xpath(".//*")
# print(assignedusertextbox[0])
# Send username information
driver.find_elements_by_class_name('gwt-TextBox')[1].send_keys(user_session_to_terminate)
search_button = driver.find_element_by_xpath("//button[text()='Search']")
# We then find its immediate parent, which is the Search button and finally click it.
search_button.click()
time.sleep(4)
# There would be three cases here.
# Case 1: The input user has no sessions on his name.
# Case 2: The input user has one session on his name.
# Case 3: The input user has more than one sessions on his name.
# Case 1: This should be one of the table header elements containing the list of servers assigned to the user.
assigned_server_rows = driver.find_elements_by_xpath("//div[text()='Assigned']")
if len(assigned_server_rows) == 0:
return_code = 4
else:
# Case 2 and Case 3:
for each_assigned_server in assigned_server_rows:
print each_assigned_server.tag_name
assigned_server_parent_td = each_assigned_server.find_element_by_xpath('..') # td
print assigned_server_parent_td.tag_name
assigned_server_parent_td.click()
# assigned_server_parent_tr = assigned_server_parent_td.find_element_by_xpath('..') #tr
# print assigned_server_parent_tr.tag_name
# assigned_server_parent_tr.click()
shutdown_button_element = driver.find_element_by_xpath("//button[text()='Shutdown Server']")
shutdown_button_element.click()
time.sleep(2)
main_page = driver.window_handles[0]
alert_window = driver.switch_to.alert
alert_window.accept()
driver.switch_to.window(main_page)
except:
return_code = 1
finally:
driver.quit()
if return_code == 0:
print('=> Successfully terminated sessions for the user ' + user_session_to_terminate)
elif return_code == 4:
print('=> There are no sessions assigned to the user ' + user_session_to_terminate)
else:
print('=> An error occurred while tying to terminate sessions for the user ' + user_session_to_terminate)
return return_code
if __name__ == "__main__":
main()
But as part of Azure Chatbot, when we tried to plug it in NodeJS code, we are seeing this problem. Is this really some problem with my code?. How should I correctly correct it.
I read on scoping of python but in this case everything is inside the main function and got to know that python try-catches do not add new block scope. So my expectation is that everything should have worked. Please let me know what I am missing.
PyAutomationScript Error:Error: Command failed: C:\tcnxbot_data\automated_script\auto.exe localhost pdittaka
[9904] Failed to execute script auto
Traceback (most recent call last):
File "auto.py", line 131, in <module>
File "auto.py", line 118, in main
UnboundLocalError: local variable 'driver' referenced before assignment
Thanks,
Pavan.
This might be an issue with the line inside the finally function. What you are essentially saying is irrespective of whether there is an error or not (whether it goes into try or except) execute driver.quit().
But if your code runs into and error inside the try before or during the driver is defined, it goes into the except, then the finally where you are attempting to close the driver. However, this is not possible because the driver hasn't been defined yet. What you can do is add this at the beginning of the main function:
driver = None
and change the code inside the finally: to be this:
if driver is not None:
driver.quit()
I am making a script that will fill the forms to create a Gmail account with Python. When a username is already used a message becomes visible. In dutch: "Dez gebruikersnaam worst al gebruikt. Probeer been andere gebruikersnaam." To be able to deal with this I need to detect when this message is visible, but I have no idea how to do that. If anyone can help, that would be amazing
Try this code:
from selenium import webdriver
import time
path = '/home/avionerman/Documents/stack'
driver = webdriver.Firefox(path)
driver.get('https://www.google.com/intl/nl/gmail/about/#')
try:
print('locating create account button')
create_account_button = driver.find_element_by_class_name('h-c-button')
except:
print("error, couldn't find create account button")
try:
create_account_button.click()
print('navigating to creation page')
except:
print('error navigating to creation page')
time.sleep(8)
handles = driver.window_handles
size = len(handles)
driver.switch_to.window(handles[1])
print(driver.title)
first_name_input = driver.find_element_by_id('firstName')
first_name_input.click()
first_name_input.send_keys("WhateverYouWant")
last_name_input = driver.find_element_by_id('lastName')
last_name_input.click()
last_name_input.send_keys("WhateverYouWant2")
username_input = driver.find_element_by_id('username')
username_input.click()
username_input.send_keys('papadopoulos')
pswd_input = driver.find_element_by_name('Passwd')
pswd_input.click()
pswd_input.send_keys('whateveryouwant')
pswd_conf_input = driver.find_element_by_name('ConfirmPasswd')
pswd_conf_input.click()
pswd_conf_input.send_keys('whateveryouwant')
next_button = driver.find_element_by_id("accountDetailsNext")
next_button.click()
# In the variable x I save the text that is included inside the specific xpath
x = driver.find_element_by_xpath("//*[#id='view_container']/form/div[2]/div/div[1]/div[2]/div[1]/div/div[2]/div[2]/div").text
print(x)
# I assert in order to check if the variable x has the proper text value (the expected one)
assert x == 'Dez gebruikersnaam worst al gebruikt. Probeer been andere gebruikersnaam.'
time.sleep(10)
In this way, if the assert passes mean that you see the warning message and in any other case the username for the mail is unique. So, the part of the code that I wrote to you, you can insert it inside if statements and you can handle it as you wish. If you need any further info feel free to ping me.
I am hoping someone may be able to point out the error I am making; it is probably very straight forward!
What I am trying to do is run some code previous what I have shown below, then when I get to this point I need to get it to hold for the 600 seconds and then reload the download page:
try:
# Clicks OK in Download Requested window
driver.implicitly_wait(10)
ClickOkay = driver.find_element_by_css_selector("._42ft._42fu.layerCancel.uiOverlayButton.selected._42g-._42gy")
ClickOkay.click()
except:
print("error 2")
# Wait Time
# time.sleep(600) # Allow Facebook to compile the archive
# Reload Settings page
GoToURL('https://www.facebook.com/' + 'settings', driver)
# Goes back through to Download page
link = driver.find_element_by_link_text('Download a copy')
link.click()
At this point if the archive has finished being created then the button changes from Start Archive to Download Archive. However depending on the size of the profile the time taken to compile the archive varies. so what i was attempting (with the code below and a couple of attempts with the if and while arguments) was to get it to check if the button exists and if not go back and wait 300 seconds before trying again. Once the button appears it will then continue on to download using additional code.
try:
print("Checking if button exists")
DownloadArchive = driver.find_elements_by_css_selector("._42ft._42fu.selected._42gz._42gy")
print(DownloadArchive.count())
if(DownloadArchive.count() > 0):
print("button exists")
else:
print("button does not exist")
# Button to initiate password entry popup window
#driver.implicitly_wait(10)
#while (DownloadArchive = driver.find_element_by_css_selector("._42ft._42fu.selected._42gz._42gy")):
# if (DownloadArchive = True):
# DownloadArchive.click()
# print("wait")
# else:time.sleep(300)
Thanks in advance, James
You're mixing the assignment operator (=) with the equal operator (==).
So it should be:
while (DownloadArchive == driver.find_element_by_css_selector("._42ft._42fu.selected._42gz._42gy")):
if (DownloadArchive == True):
Or just:
while DownloadArchive == driver.find_element_by_css_selector("._42ft._42fu.selected._42gz._42gy"):
Hope it helps!
I have this automation tool I've built with Selenium 2 and Browsermob proxy that works quite well for most of what I need. However, I've run into a snag on capturing network traffic.
I basically want to capture the har that a click provides before the page redirects. For example, I have an analytics call happening on the click that I want to capture, then another analytics call on the page load that I don't want to capture.
All of my attempts currently capture the har too late, so I see both the click analytics call and the page load one. Is there any way to get this working? I've included my current relevant code sections below
METHODS INSIDE HELPER CLASS
class _check_for_page_load(object):
def __init__(self, browser, parent):
self.browser = browser
self.maxWait = 5
self.parent = parent
def __enter__(self):
self.old_page = self.browser.find_element_by_tag_name('html')
def wait_for(self,condition_function):
start_time = time.time()
while time.time() < start_time + self.maxWait:
if condition_function():
return True
else:
time.sleep(0.01)
raise Exception(
'Timeout waiting for {}'.format(condition_function.__name__)
)
def page_has_loaded(self):
new_page = self.browser.find_element_by_tag_name('html')
###self.parent.log("testing ---- " + str(new_page.id) + " " + str(self.old_page.id))
return new_page.id != self.old_page.id
def __exit__(self, *_):
try:
self.wait_for(self.page_has_loaded)
except:
pass
def startNetworkCalls(self):
if self._p != None:
self._p.new_har("Step"+str(self._currStep))
def getNetworkCalls(self, waitForTrafficToStop = True):
if self._p != None:
if waitForTrafficToStop:
self._p.wait_for_traffic_to_stop(5000, 30*1000);
return self._p.har
else:
return "{}"
def click(self, selector):
''' clicks on an element '''
self.log("Clicking element '" + selector + "'")
el = self.findEl(selector)
traffic = ""
with self._check_for_page_load(self._d, self):
try:
self._curr_window = self._d.window_handles[0]
el.click()
except:
actions = ActionChains(self._d);
actions.move_to_element(el).click().perform()
traffic = self.getNetworkCalls(False)
try:
popup = self._d.switch_to.alert
if popup != None:
popup.dismiss()
except:
pass
try:
window_after = self._d.window_handles[1]
if window_after != self._curr_window:
self._d.close()
self._d.switch_to_window(self._curr_window)
except:
pass
return traffic
INSIDE FILE THAT RUNS MULTIPLE SELENIUM ACTIONS
##inside a for loop, we get an action that looks like "click('#selector')"
util.startNetworkCalls()
if action.startswith("click"):
temp_traffic = eval(action)
if temp_traffic == "":
temp_traffic = util.getNetworkCalls()
traffic = json.dumps(temp_traffic, sort_keys=True) ##gives json har info that is saved later
You can see from these couple snippets that I initiate the "click" function which returns network traffic. Inside the click function, you can see it references the class "_check_for_page_load". However, the first time it reaches this line:
###self.parent.log("testing ---- " + str(new_page.id) + " " + str(self.old_page.id))
The log (when enabled) shows that the element ids don't match on the first time it logs, indicating the page load has already started to happen. I'm pretty stuck right now as I've tried everything I can think of to try to accomplish this functionality.
I found a solution to my own question - though it isn't perfect. I told my network calls to capture headers:
def startNetworkCalls(self):
if self._p != None:
self._p.new_har("Step"+str(self._currStep),{"captureHeaders": "true"})
Then, when I retrieve the har data, I can look for the "Referer" header and compare that with the page that was initially loaded (before the redirect from the click). From there, I can split the har into two separate lists of network calls to further process later.
This works for my needs, but it isn't perfect. Some things, like image requests, sometimes get the same referrer that the previous page's url matched, so the splitting puts those into the first bucket rather than the appropriate second bucket. However, since I'm more interested in requests that aren't on the same domain, this isn't really an issue.