When visiting some websites with selenium there are times where the page doesn't load correctly. So I want to write a code that checks if the website loaded correctly by searching for a particular button. If it can't find the button it needs to refresh until it finds the button and if the code does find the button it needs to execute the rest of the code.
So for example: Button not there: >>> refresh >>> (checks again if button is not there) Button not there >>> refresh >>> (checks again if button is not there) Button is there >>> rest of code
The loop that I currently have looks like this but after refreshing the loop doesn't restart and runs the else: function.
So the question is how do I make a loop that restarts the loop after it refreshes.
while not (driver.find_elements(By.CLASS_NAME, "button")):
driver.refresh()
else:
Rest of code
Help would be much appreciated, thanks in advance
You can have an infinite while loop, and an if condition with find_elements, Please note that find_elements does not return any exception, it returns a list of web elements or either 0.
Code:
while True:
try:
if len(driver.find_elements(By.XPATH, "xpath of the button")) >0:
print("Button is present, since find elements list is non empty")
# execute the code, may be click on the button or whatever it is.
#make sure to exit from infinite loop as well
break
else:
driver.refresh()
#may be put some delay here to let button available to click again.
except:
print("There was some problem trying to find the button element, tearing it down")
break
Related
in my scraping script in python I'm in a situation where, while collapsing multiple buttons on a page, randomically a couple of pop up appear in the page and automatically the script fails.
These two pop ups are already managed and the beginning of the script but the website in a non systematic way dedices to show these two.
This is the part of the script interested where the script sleeps for 3 secs between one click to the other:
collapes = driver.find_elements_by_css_selector('.suf-CompetitionMarketGroup suf-CompetitionMarketGroup-collapsed ')
for collapes in collapes:
collapes.click()
sleep(3)
These are the two lines of the scipt where I click on the pop ups at the beginning
wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/div[3]/div/div[2]/div[2]'))).click()
driver.find_element(By.XPATH, '/html/body/div[1]/div/div[4]/div[1]/div/div[3]/div[4]/div[3]/div').click()
DO you think there's a way to continue running the process being ready to click on these two without going on error?
You can try to close the popups every time your code in for loop fails:
def try_closing_popups():
try:
wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/div[3]/div/div[2]/div[2]'))).click()
driver.find_element(By.XPATH, '/html/body/div[1]/div/div[4]/div[1]/div/div[3]/div[4]/div[3]/div').click()
except:
pass
collapes = driver.find_elements_by_css_selector('.suf-CompetitionMarketGroup suf-CompetitionMarketGroup-collapsed ')
for collapes in collapes:
try:
collapes.click()
except:
try_closing_popups()
sleep(3)
I have a piece of pywinauto code which attempts to delete an Item from an ItemList. Occasionally the type_keys() isn't working, as if the window is going out of focus. Interestingly enough, if I click on the Command Prompt window where the python code is running, this suddenly triggers the code to properly set focus on the application and continue running.
Here's an overview of the steps:
Opens an Open Diaglog window in certain application
(here its Tradestation's Development Environment)
Selects an item from the ItemList
Type DELETE key
(confirmation window pops up)
Select 'Yes' in the confirmation window
Most of the time this sequence works, except occasionally (maybe 1 out of every 5 or so tries) it fails where the DELETE key press isn't happening. The item is selected, but the window appears out of focus (or so it seems).
I've been chasing this issue for days/weeks, trying various different things, i.e. added delays, adjusted timeouts, adjusted pywinauto.Timings.fast/slow, tried setting focus in various places, etc... I also put the code into an infinite 'while' loop hoping that it would work on subsequent tries.
Interestingly enough, when it gets stuck, if I click on the Windows Command Prompt window where the python application is running, then suddenly the Open Diaglog box goes back into focus and continues to run successfully.
I'm looking for suggestions as to the root cause OR what else I could try.
Here's where the code has ended up:
strategy = self.window.child_window(title=name, control_type='ListItem')
Timings.slow()
while True:
# Issues where delete/confirm window not showing
# Try solving with SetFocus in case the open diag is going out of focus
_logger.info("Set focus")
handle = win32gui.FindWindow(None, OPEN_DLG_TITLE)
win32gui.SetForegroundWindow(handle)
# Highlight the strategy
_logger.info("Select Strategy")
strategy.select()
if strategy.is_selected() == 0:
time.sleep(2)
continue
_logger.info("Strategy selected")
# Delete it...
_logger.info("Type DELETE")
#strategy.type_keys('{DELETE}')
strategy.type_keys('{DELETE}', set_foreground=False)
# Delay in case return comes back too quickly
time.sleep(2)
# Click 'Yes' in Confirmation Window
_logger.info("Check confirmation window")
confirm_window = self.window.child_window(
title="Confirm file removal",
control_type="Window"
)
if confirm_window.exists() is False:
try:
_logger.info('... does not exist. Briefly wait 5 seconds')
confirm_window.wait('exists', timeout=5)
except pywinauto.timings.TimeoutError:
#breakpoint()
_logger.info('---- Retrying -----')
continue
assert confirm_window.exists() is True
break
Timings.fast()
I am trying to get followers with python selenium. But sometimes python clicks by itself.
I want to make an error-free program. I try to I've tried "try catch" constructs but it didn't work. Here is my code:
def getFollowers(self):
try:
self.browser.get(f"https://www.instagram.com/{self.username}")
time.sleep(2)
followers=self.browser.find_element_by_xpath("//*[#id='react-root']/section/main/div/header/section/ul/li[2]/a").click()
time.sleep(2)
dialog=self.browser.find_element_by_xpath("/html/body/div[5]/div/div/div[2]")
followerCount=len(dialog.find_elements_by_tag_name("li"))
print(f"first count:{followerCount}")
action=webdriver.ActionChains(self.browser)
//*******************************************Probly my problem is here****************************************
while True:
dialog.click()
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
//**********************************************************************************************************
followers=dialog.find_elements_by_tag_name("li")
followersList=[]
for user in followers:
link=user.find_element_by_css_selector("a").get_attribute("href")
# print(link)
followersList.append(link)
with open("followers.txt","w",encoding="UTF-8") as file:
for item in followersList:
file.write(item+"\n")
time.sleep(5)
except:
pass
I also have def getfollowing and it works flawlessly. If you want I can show it too. But they are almost same.
EDIT: #RohanShah solved my problem. At the bottom of the page you can see the solution.
Edit: I am new here thats why sometimes my questions could be meanless.But please dont decrease my points. Stackoverflow not gonna accept my questions anymore. Please increase my points.
I've had this exact same problem while scrolling the popups. What happens is your dialog.click(), while attempting to focus your key down on the popup, occasionally clicks a user and loads their profile. Your script then crashes as the popup is no longer on the screen.
After a lot of research into solving this problem, I noticed it only happens with usernames that are long. Regardless, I implemented a simple hack to get around this problem.
First we get the url of what the standard scroll looks like. When opening and scrolling the popup, this is the url we are on.
https://www.instagram.com/some_username/followers/
2.Now I have created a function to hold the code for opening the popup. This will be very useful so trap the necessary code into a function. (I don't have the classnames or xpath's on me so please customize the function for yourself)
def openPopup():
self.browser.get(f"https://www.instagram.com/{self.username}")
global popup # we will need to access this variable outside of the function
popup = driver.find_element_by_class_name('popupClass') #you don't have to use class_name
popup.click()
Now we have to tell our while loop to not scan when Selenium accidentally clicks on a user. We will use our URL from step 1. Please make sure the following if-statement is inserted at the TOP of your loop so if there is a break, it will handle it first before trying to access the popup.
while True:
check_url = self.browser.current_url #returns string with current_url
if check_url != 'https://www.instagram.com/some_username/followers/':
#if this code is executed, this means there has been an accidental click
openPopup() #this will bring back to the page and reopen popup
#the rest of your code
popup.click() # variable from our function
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
check_url = self.browser.current_url #we must recheck the current_url every time the loop runs to see if there has been a misclick
Now, whenever your loop detects the URL is no longer one of the popup, it will automatically call openPopup() which will get you back to the page and back in the popup, and your loop will continue as if nothing happened.
I made a crawler for this page (http://www.bobaedream.co.kr/cyber/CyberCar.php?gubun=I) to collect the stock list of specific manufacturers. The process is to start from selecting the drop-down menu in the first row of upper part of search menu.
Each right drop-down menu is child menu of its left drop-down menu. What I would like to do is to select each first item in each drop-down menu and click the "search" button for the first run. After crawling of its stock list, then I set the second item of the last drop-down menu and click the "search" button.
But the problem is occurred here. I saved each items of each drop-down menu as tuple. When I try to call the second item of the last drop-down menu for the second round of crawling, "StaleElementReferenceException" or "NoSuchElementException" is occurred with the message of "Element is no longer attached to the DOM". Thus, I would like to make the element wait until the entire round of each drop-down iteration is completed.
Below is my code, but still have the error message. My error usually occurs at the second while loop. At this moment, I guess some type of "wait.until(EC.~)" code in the second "try" function can work this out, but I have no specific idea for this. Please help or give me any advice.
def option2_menu_loaded(inDriver):
path = '//select[#id="level2_no"]'
return inDriver.find_element_by_xpath(path)
self.wait.until(option2_menu_loaded)
while True:
try:
select_option2_values = [
('%s' % o.get_attribute('text'), '%s' % o.get_attribute('value'))
for o
in self.getNewSelect("#level2_no").options
if o.get_attribute('text') != '세부등급']
except (StaleElementReferenceException, NoSuchElementException):
print("Exception Found")
continue
break
for option2 in select_option2_values:
self.csv.setCarTitle(ma, mo, de, option1[0], option2[0])
print(option2[0], option2[1])
self.driver.implicitly_wait(0.5)
while True:
try:
self.getNewSelect("#level2_no").select_by_value(option2[1])
except (StaleElementReferenceException, NoSuchElementException):
self.getNewSelect("#level2_no").options
print("Exception Found")
continue
break
If you google the StaleElementException you will see solutions that try to find again the element within a loop. So that is one idea, in your exception above try 3 times with 1 sec delay before each try to find_Element again, see if this helps.
Another idea is to refresh the page (certainly not ideal but it might work) between every crawl. You can do this in Python using:
driver.refresh()
Finally, you can also avoid looping (this might be causing the StaleElementException) through all the different elements when crawling as Selenium has a solution for that. You can save eveything in tuple/array without looping through each record by using find_ElementS instead of find_ElemenT. Try this see if it improves your overall performance:
a=[];
a = driver.find_elements_by_xpath(path)
Best of luck!
I have the following issue with Selenium Webdriver. There are two dropdown menus on a page i am testing "brand" and "items". The options of "items" depend on which brand you choose. I am trying to iterate through all possible choices and print brand-item pairs. I use two possible ways to pick an option from each dropdown menu
Using Select():
def retryingSelectOption(name,n):
result=False
attempts=0
while attempts<5:
try:
element=Select(driver.find_element_by_name(name))
element.select_by_index(n)
print element.all_selected_options[0].text
result=True
break
except StaleElementReferenceException:
pass
attempts+=1
return result
And using .click():
def retryingClickOption(name,n):
result=False
attempts=0
while attempts<5:
try:
driver.find_element_by_name(name).find_elements_by_tag_name("option")[n].click()
result=True
break
except StaleElementReferenceException:
pass
attempts+=1
return result
My problem is that at ,what seem to me as random moments (sometimes it works sometimes it does not), even though the above functions return True and printing out the selected option shows me the correct answer, the browser still displays the previous option. So basically Selenium tells me i have picked the right option but the browser displays the previous one.No idea what is wrong.
After checking it a bit more it seems that if i manually click somewhere in the browser the option actually refreshes to the correct value. It seems like the browser hangs.