in my scraping script in python I'm in a situation where, while collapsing multiple buttons on a page, randomically a couple of pop up appear in the page and automatically the script fails.
These two pop ups are already managed and the beginning of the script but the website in a non systematic way dedices to show these two.
This is the part of the script interested where the script sleeps for 3 secs between one click to the other:
collapes = driver.find_elements_by_css_selector('.suf-CompetitionMarketGroup suf-CompetitionMarketGroup-collapsed ')
for collapes in collapes:
collapes.click()
sleep(3)
These are the two lines of the scipt where I click on the pop ups at the beginning
wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/div[3]/div/div[2]/div[2]'))).click()
driver.find_element(By.XPATH, '/html/body/div[1]/div/div[4]/div[1]/div/div[3]/div[4]/div[3]/div').click()
DO you think there's a way to continue running the process being ready to click on these two without going on error?
You can try to close the popups every time your code in for loop fails:
def try_closing_popups():
try:
wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/div[3]/div/div[2]/div[2]'))).click()
driver.find_element(By.XPATH, '/html/body/div[1]/div/div[4]/div[1]/div/div[3]/div[4]/div[3]/div').click()
except:
pass
collapes = driver.find_elements_by_css_selector('.suf-CompetitionMarketGroup suf-CompetitionMarketGroup-collapsed ')
for collapes in collapes:
try:
collapes.click()
except:
try_closing_popups()
sleep(3)
Related
When visiting some websites with selenium there are times where the page doesn't load correctly. So I want to write a code that checks if the website loaded correctly by searching for a particular button. If it can't find the button it needs to refresh until it finds the button and if the code does find the button it needs to execute the rest of the code.
So for example: Button not there: >>> refresh >>> (checks again if button is not there) Button not there >>> refresh >>> (checks again if button is not there) Button is there >>> rest of code
The loop that I currently have looks like this but after refreshing the loop doesn't restart and runs the else: function.
So the question is how do I make a loop that restarts the loop after it refreshes.
while not (driver.find_elements(By.CLASS_NAME, "button")):
driver.refresh()
else:
Rest of code
Help would be much appreciated, thanks in advance
You can have an infinite while loop, and an if condition with find_elements, Please note that find_elements does not return any exception, it returns a list of web elements or either 0.
Code:
while True:
try:
if len(driver.find_elements(By.XPATH, "xpath of the button")) >0:
print("Button is present, since find elements list is non empty")
# execute the code, may be click on the button or whatever it is.
#make sure to exit from infinite loop as well
break
else:
driver.refresh()
#may be put some delay here to let button available to click again.
except:
print("There was some problem trying to find the button element, tearing it down")
break
I am trying to get followers with python selenium. But sometimes python clicks by itself.
I want to make an error-free program. I try to I've tried "try catch" constructs but it didn't work. Here is my code:
def getFollowers(self):
try:
self.browser.get(f"https://www.instagram.com/{self.username}")
time.sleep(2)
followers=self.browser.find_element_by_xpath("//*[#id='react-root']/section/main/div/header/section/ul/li[2]/a").click()
time.sleep(2)
dialog=self.browser.find_element_by_xpath("/html/body/div[5]/div/div/div[2]")
followerCount=len(dialog.find_elements_by_tag_name("li"))
print(f"first count:{followerCount}")
action=webdriver.ActionChains(self.browser)
//*******************************************Probly my problem is here****************************************
while True:
dialog.click()
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
//**********************************************************************************************************
followers=dialog.find_elements_by_tag_name("li")
followersList=[]
for user in followers:
link=user.find_element_by_css_selector("a").get_attribute("href")
# print(link)
followersList.append(link)
with open("followers.txt","w",encoding="UTF-8") as file:
for item in followersList:
file.write(item+"\n")
time.sleep(5)
except:
pass
I also have def getfollowing and it works flawlessly. If you want I can show it too. But they are almost same.
EDIT: #RohanShah solved my problem. At the bottom of the page you can see the solution.
Edit: I am new here thats why sometimes my questions could be meanless.But please dont decrease my points. Stackoverflow not gonna accept my questions anymore. Please increase my points.
I've had this exact same problem while scrolling the popups. What happens is your dialog.click(), while attempting to focus your key down on the popup, occasionally clicks a user and loads their profile. Your script then crashes as the popup is no longer on the screen.
After a lot of research into solving this problem, I noticed it only happens with usernames that are long. Regardless, I implemented a simple hack to get around this problem.
First we get the url of what the standard scroll looks like. When opening and scrolling the popup, this is the url we are on.
https://www.instagram.com/some_username/followers/
2.Now I have created a function to hold the code for opening the popup. This will be very useful so trap the necessary code into a function. (I don't have the classnames or xpath's on me so please customize the function for yourself)
def openPopup():
self.browser.get(f"https://www.instagram.com/{self.username}")
global popup # we will need to access this variable outside of the function
popup = driver.find_element_by_class_name('popupClass') #you don't have to use class_name
popup.click()
Now we have to tell our while loop to not scan when Selenium accidentally clicks on a user. We will use our URL from step 1. Please make sure the following if-statement is inserted at the TOP of your loop so if there is a break, it will handle it first before trying to access the popup.
while True:
check_url = self.browser.current_url #returns string with current_url
if check_url != 'https://www.instagram.com/some_username/followers/':
#if this code is executed, this means there has been an accidental click
openPopup() #this will bring back to the page and reopen popup
#the rest of your code
popup.click() # variable from our function
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
check_url = self.browser.current_url #we must recheck the current_url every time the loop runs to see if there has been a misclick
Now, whenever your loop detects the URL is no longer one of the popup, it will automatically call openPopup() which will get you back to the page and back in the popup, and your loop will continue as if nothing happened.
I'm trying to .click() a few elements in a pop up-list on a webpage, but keep getting the StaleElementReferenceException when i try to move_to_elements.
The code is based around a number of clickable elements in a feed. When clicked, these elements result in a pop-up box with more clickable elements that I want to access.
I access the pop up-box with the following code, where popupbox_links are a list with coordinates and links for the pop up boxes:
for coordinate in popupbox_links:
actions = ActionChains(driver)
actions.move_to_element(coordinate["Popupbox location"]).perform()
time.sleep(3)
popupboxpath = coordinate["Popupbox link"]
popupboxpath.click()
time.sleep(3)
This works fine. But when the pop up box is opened, I want to perform the following:
seemore = driver.find_element_by_link_text("See More")
time.sleep(2)
actions.move_to_element(seemore).perform()
time.sleep(2)
seemore.click()
time.sleep(3)
findbuttons = driver.find_elements_by_link_text("Button")
time.sleep(2)
print(findbutton)
for button in findbuttons:
time.sleep(2)
actions.move_to_element(button).perform()
time.sleep(2)
button.click()
time.sleep(randint(1, 5))
The trouble starts at actions.move_to_element on both "See more" and "Button". Even though the print(findbutton) actually returns a list with contents, containing the elements that I want to click, Selenium seems to be unable to move_to_element on these. Instead, it throws StaleElementReferenceException.
To make it more confusing, the script seems to work at times. Although usually it just crashes.
Any clues on how to solve this? Big thanks in advance.
I'm running the latest Selenium on Python 3.6 with the Chrome WebDriver.
StaleElementReferenceException Says that element is stage because something has changed in page after you created the webElement object. In you case that could be happening due to button.click().
The simplest solution is to create new element every time, rather than iterate element from the loop.
following changes might work.
findbuttons = driver.find_elements_by_link_text("Button")
time.sleep(2)
print(findbuttons)
for i in range(len(findbuttons)):
time.sleep(2)
elem = driver.find_elements_by_link_text("Button")[i]
actions.move_to_element(elem).perform()
time.sleep(2)
elem.click()
time.sleep(randint(1, 5))
I made a crawler for this page (http://www.bobaedream.co.kr/cyber/CyberCar.php?gubun=I) to collect the stock list of specific manufacturers. The process is to start from selecting the drop-down menu in the first row of upper part of search menu.
Each right drop-down menu is child menu of its left drop-down menu. What I would like to do is to select each first item in each drop-down menu and click the "search" button for the first run. After crawling of its stock list, then I set the second item of the last drop-down menu and click the "search" button.
But the problem is occurred here. I saved each items of each drop-down menu as tuple. When I try to call the second item of the last drop-down menu for the second round of crawling, "StaleElementReferenceException" or "NoSuchElementException" is occurred with the message of "Element is no longer attached to the DOM". Thus, I would like to make the element wait until the entire round of each drop-down iteration is completed.
Below is my code, but still have the error message. My error usually occurs at the second while loop. At this moment, I guess some type of "wait.until(EC.~)" code in the second "try" function can work this out, but I have no specific idea for this. Please help or give me any advice.
def option2_menu_loaded(inDriver):
path = '//select[#id="level2_no"]'
return inDriver.find_element_by_xpath(path)
self.wait.until(option2_menu_loaded)
while True:
try:
select_option2_values = [
('%s' % o.get_attribute('text'), '%s' % o.get_attribute('value'))
for o
in self.getNewSelect("#level2_no").options
if o.get_attribute('text') != '세부등급']
except (StaleElementReferenceException, NoSuchElementException):
print("Exception Found")
continue
break
for option2 in select_option2_values:
self.csv.setCarTitle(ma, mo, de, option1[0], option2[0])
print(option2[0], option2[1])
self.driver.implicitly_wait(0.5)
while True:
try:
self.getNewSelect("#level2_no").select_by_value(option2[1])
except (StaleElementReferenceException, NoSuchElementException):
self.getNewSelect("#level2_no").options
print("Exception Found")
continue
break
If you google the StaleElementException you will see solutions that try to find again the element within a loop. So that is one idea, in your exception above try 3 times with 1 sec delay before each try to find_Element again, see if this helps.
Another idea is to refresh the page (certainly not ideal but it might work) between every crawl. You can do this in Python using:
driver.refresh()
Finally, you can also avoid looping (this might be causing the StaleElementException) through all the different elements when crawling as Selenium has a solution for that. You can save eveything in tuple/array without looping through each record by using find_ElementS instead of find_ElemenT. Try this see if it improves your overall performance:
a=[];
a = driver.find_elements_by_xpath(path)
Best of luck!
I have a dynamic page that loads products when the user scrolls down a page. I want to get the total number of products rendered on the display page. Currently I am using the following code to get to the bottom until all the products are being displayed.
elems = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
print len(elems)
a = len(elems)
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(4)
elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
b = len(elem1)
while b > a:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(4)
elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
a = b
b = len(elem1)
print b
This is working nicely, but I want to know whether there is any better option of doing this?
You can perform this action easily using this line of code
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
And if you want to scroll down for ever you should try this.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Firefox()
driver.get("https://twitter.com/BarackObama")
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(3)
I am not sure about time.sleep(x value) cause loading data my take longer .. or less ..
for more information please check the official Doc page
have fun :)
I think you could condense your code down to this:
prior = 0
while True:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
current = len(WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x"))))
if current == prior:
return current
prior = current
I did away with all the identical lines by moving them all into the loop, which necessitated making the loop a while True: and moving the condition checking into the loop (because unfortunately, Python lacks any do-while).
I also threw out the sleep and print statements - I'm not sure what their purpose was, but on my own page, I have found that the same number of elements load whether I sleep between scrolls or not. Further, in my own case, I don't need to know the count at any point, I just need to know when it has exhausted the list (but I added in a return variable so you can get the final count if you happen to need it. If you really want to print ever intermediate count, you can print current right after it's assigned in the loop.
If you have no idea how many elements might be added to the page, but you just want to get all of them, it might be good to loop thusly:
scroll down as described above
wait a few seconds
save the size of the page source (xxx.page_source)
if the size of the page source is larger than the last page source size saved, loop back and scroll down some more
I suppose that screenshot size might work fine too, depending upon the page you're loading, but this is working in my current program.