I have the following issue with Selenium Webdriver. There are two dropdown menus on a page i am testing "brand" and "items". The options of "items" depend on which brand you choose. I am trying to iterate through all possible choices and print brand-item pairs. I use two possible ways to pick an option from each dropdown menu
Using Select():
def retryingSelectOption(name,n):
result=False
attempts=0
while attempts<5:
try:
element=Select(driver.find_element_by_name(name))
element.select_by_index(n)
print element.all_selected_options[0].text
result=True
break
except StaleElementReferenceException:
pass
attempts+=1
return result
And using .click():
def retryingClickOption(name,n):
result=False
attempts=0
while attempts<5:
try:
driver.find_element_by_name(name).find_elements_by_tag_name("option")[n].click()
result=True
break
except StaleElementReferenceException:
pass
attempts+=1
return result
My problem is that at ,what seem to me as random moments (sometimes it works sometimes it does not), even though the above functions return True and printing out the selected option shows me the correct answer, the browser still displays the previous option. So basically Selenium tells me i have picked the right option but the browser displays the previous one.No idea what is wrong.
After checking it a bit more it seems that if i manually click somewhere in the browser the option actually refreshes to the correct value. It seems like the browser hangs.
Related
When visiting some websites with selenium there are times where the page doesn't load correctly. So I want to write a code that checks if the website loaded correctly by searching for a particular button. If it can't find the button it needs to refresh until it finds the button and if the code does find the button it needs to execute the rest of the code.
So for example: Button not there: >>> refresh >>> (checks again if button is not there) Button not there >>> refresh >>> (checks again if button is not there) Button is there >>> rest of code
The loop that I currently have looks like this but after refreshing the loop doesn't restart and runs the else: function.
So the question is how do I make a loop that restarts the loop after it refreshes.
while not (driver.find_elements(By.CLASS_NAME, "button")):
driver.refresh()
else:
Rest of code
Help would be much appreciated, thanks in advance
You can have an infinite while loop, and an if condition with find_elements, Please note that find_elements does not return any exception, it returns a list of web elements or either 0.
Code:
while True:
try:
if len(driver.find_elements(By.XPATH, "xpath of the button")) >0:
print("Button is present, since find elements list is non empty")
# execute the code, may be click on the button or whatever it is.
#make sure to exit from infinite loop as well
break
else:
driver.refresh()
#may be put some delay here to let button available to click again.
except:
print("There was some problem trying to find the button element, tearing it down")
break
I am trying to get followers with python selenium. But sometimes python clicks by itself.
I want to make an error-free program. I try to I've tried "try catch" constructs but it didn't work. Here is my code:
def getFollowers(self):
try:
self.browser.get(f"https://www.instagram.com/{self.username}")
time.sleep(2)
followers=self.browser.find_element_by_xpath("//*[#id='react-root']/section/main/div/header/section/ul/li[2]/a").click()
time.sleep(2)
dialog=self.browser.find_element_by_xpath("/html/body/div[5]/div/div/div[2]")
followerCount=len(dialog.find_elements_by_tag_name("li"))
print(f"first count:{followerCount}")
action=webdriver.ActionChains(self.browser)
//*******************************************Probly my problem is here****************************************
while True:
dialog.click()
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
//**********************************************************************************************************
followers=dialog.find_elements_by_tag_name("li")
followersList=[]
for user in followers:
link=user.find_element_by_css_selector("a").get_attribute("href")
# print(link)
followersList.append(link)
with open("followers.txt","w",encoding="UTF-8") as file:
for item in followersList:
file.write(item+"\n")
time.sleep(5)
except:
pass
I also have def getfollowing and it works flawlessly. If you want I can show it too. But they are almost same.
EDIT: #RohanShah solved my problem. At the bottom of the page you can see the solution.
Edit: I am new here thats why sometimes my questions could be meanless.But please dont decrease my points. Stackoverflow not gonna accept my questions anymore. Please increase my points.
I've had this exact same problem while scrolling the popups. What happens is your dialog.click(), while attempting to focus your key down on the popup, occasionally clicks a user and loads their profile. Your script then crashes as the popup is no longer on the screen.
After a lot of research into solving this problem, I noticed it only happens with usernames that are long. Regardless, I implemented a simple hack to get around this problem.
First we get the url of what the standard scroll looks like. When opening and scrolling the popup, this is the url we are on.
https://www.instagram.com/some_username/followers/
2.Now I have created a function to hold the code for opening the popup. This will be very useful so trap the necessary code into a function. (I don't have the classnames or xpath's on me so please customize the function for yourself)
def openPopup():
self.browser.get(f"https://www.instagram.com/{self.username}")
global popup # we will need to access this variable outside of the function
popup = driver.find_element_by_class_name('popupClass') #you don't have to use class_name
popup.click()
Now we have to tell our while loop to not scan when Selenium accidentally clicks on a user. We will use our URL from step 1. Please make sure the following if-statement is inserted at the TOP of your loop so if there is a break, it will handle it first before trying to access the popup.
while True:
check_url = self.browser.current_url #returns string with current_url
if check_url != 'https://www.instagram.com/some_username/followers/':
#if this code is executed, this means there has been an accidental click
openPopup() #this will bring back to the page and reopen popup
#the rest of your code
popup.click() # variable from our function
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
check_url = self.browser.current_url #we must recheck the current_url every time the loop runs to see if there has been a misclick
Now, whenever your loop detects the URL is no longer one of the popup, it will automatically call openPopup() which will get you back to the page and back in the popup, and your loop will continue as if nothing happened.
I made a crawler for this page (http://www.bobaedream.co.kr/cyber/CyberCar.php?gubun=I) to collect the stock list of specific manufacturers. The process is to start from selecting the drop-down menu in the first row of upper part of search menu.
Each right drop-down menu is child menu of its left drop-down menu. What I would like to do is to select each first item in each drop-down menu and click the "search" button for the first run. After crawling of its stock list, then I set the second item of the last drop-down menu and click the "search" button.
But the problem is occurred here. I saved each items of each drop-down menu as tuple. When I try to call the second item of the last drop-down menu for the second round of crawling, "StaleElementReferenceException" or "NoSuchElementException" is occurred with the message of "Element is no longer attached to the DOM". Thus, I would like to make the element wait until the entire round of each drop-down iteration is completed.
Below is my code, but still have the error message. My error usually occurs at the second while loop. At this moment, I guess some type of "wait.until(EC.~)" code in the second "try" function can work this out, but I have no specific idea for this. Please help or give me any advice.
def option2_menu_loaded(inDriver):
path = '//select[#id="level2_no"]'
return inDriver.find_element_by_xpath(path)
self.wait.until(option2_menu_loaded)
while True:
try:
select_option2_values = [
('%s' % o.get_attribute('text'), '%s' % o.get_attribute('value'))
for o
in self.getNewSelect("#level2_no").options
if o.get_attribute('text') != '세부등급']
except (StaleElementReferenceException, NoSuchElementException):
print("Exception Found")
continue
break
for option2 in select_option2_values:
self.csv.setCarTitle(ma, mo, de, option1[0], option2[0])
print(option2[0], option2[1])
self.driver.implicitly_wait(0.5)
while True:
try:
self.getNewSelect("#level2_no").select_by_value(option2[1])
except (StaleElementReferenceException, NoSuchElementException):
self.getNewSelect("#level2_no").options
print("Exception Found")
continue
break
If you google the StaleElementException you will see solutions that try to find again the element within a loop. So that is one idea, in your exception above try 3 times with 1 sec delay before each try to find_Element again, see if this helps.
Another idea is to refresh the page (certainly not ideal but it might work) between every crawl. You can do this in Python using:
driver.refresh()
Finally, you can also avoid looping (this might be causing the StaleElementException) through all the different elements when crawling as Selenium has a solution for that. You can save eveything in tuple/array without looping through each record by using find_ElementS instead of find_ElemenT. Try this see if it improves your overall performance:
a=[];
a = driver.find_elements_by_xpath(path)
Best of luck!
So, using WebDriver python binding, I came across a problem that I need to wait until an element is refreshed on the page. Say, I have two radio buttons, and by clicking either of them, a label text is changed.
Currently, if I click on one and get the text and then click on the other and get the text again, I will get the same text although it has changed. Thus, I think I need to wait for the element to be refreshed.
On Java documentation, there is a refreshed expected condition which appears to be useful in this case. But I'm unable to find the python version of it. What is its equivalent? How can I workaround this?
text_to_be_present_in_element() under expected_conditions.py is similar to what you're looking for. If it's not the value that's changing but instead another field like textContent, you could roll a custom solution which is similar to text_to_be_present_in_element().
def attribute_text_is_in_element(text, locator, attribute):
try:
element_text = driver.find_element(locator).get_attribute(attribute)
if text == element_text:
return True
else:
return False
except StaleElementReferenceException:
return False
In my test I have to click a button to add an item to the cart. At that point a model dialog comes up and allows some options and links, one of which I am trying to click; View Cart and Checkout.
Every attempt I make to find my element with find_element_by fails with a NoSuchElementException. Doesn't matter how I try to find it; ID, Class, Link Text, etc.
I have tried setting up WebDriverWait in a try/catch statement to see if pausing for the model to be added to the page would help, and it doesn't. I have also checked to see if there is another window or frame to switch to, but there is only the one window.
Are there any other ways to get what I need done, or troubleshooting steps that I can do to figure this out?
Below is what Firefox shows the HTML to be.
<div class="summary">
<dl></dl>
<a class="bb-button" href="/shopping/cart">View Cart and Checkout</a>
<div class="continue"></div>
The python I have written is:
product_details.addDocumentToCart('Printed Edition')
try:
WebDriverWait(self.driver, 10).until(EC.presence_of_element_located(By.CLASS_NAME, 'bb-button'))
print 'bb-button found'
except:
print 'bb-button not found'
windows = self.driver.window_handles
print 'Windows after: ' + str(len(windows))
element = self.driver.find_element_by_class_name('bb-button')
element.click()
The messages I printed out return:
added_to_cart-container not found
and
Windows after: 1
The error I get when running this is:
NoSuchElementException: Message: Unable to locate element: {"method":"class name","selector":"bb-button"}
Thanks Mikko, with your advice about the pbd.set_trace() I was able to discover that my problem was really with the button that I thought was being clicked but wasn't. Further troubleshooting showed that my addDocumentToCart() method was clicking the wrong element, and so my model didn't come up, but no error was generated which lead me to believe that things were working when the really were not.