I have a dynamic page that loads products when the user scrolls down a page. I want to get the total number of products rendered on the display page. Currently I am using the following code to get to the bottom until all the products are being displayed.
elems = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
print len(elems)
a = len(elems)
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(4)
elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
b = len(elem1)
while b > a:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(4)
elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
a = b
b = len(elem1)
print b
This is working nicely, but I want to know whether there is any better option of doing this?
You can perform this action easily using this line of code
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
And if you want to scroll down for ever you should try this.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Firefox()
driver.get("https://twitter.com/BarackObama")
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(3)
I am not sure about time.sleep(x value) cause loading data my take longer .. or less ..
for more information please check the official Doc page
have fun :)
I think you could condense your code down to this:
prior = 0
while True:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
current = len(WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x"))))
if current == prior:
return current
prior = current
I did away with all the identical lines by moving them all into the loop, which necessitated making the loop a while True: and moving the condition checking into the loop (because unfortunately, Python lacks any do-while).
I also threw out the sleep and print statements - I'm not sure what their purpose was, but on my own page, I have found that the same number of elements load whether I sleep between scrolls or not. Further, in my own case, I don't need to know the count at any point, I just need to know when it has exhausted the list (but I added in a return variable so you can get the final count if you happen to need it. If you really want to print ever intermediate count, you can print current right after it's assigned in the loop.
If you have no idea how many elements might be added to the page, but you just want to get all of them, it might be good to loop thusly:
scroll down as described above
wait a few seconds
save the size of the page source (xxx.page_source)
if the size of the page source is larger than the last page source size saved, loop back and scroll down some more
I suppose that screenshot size might work fine too, depending upon the page you're loading, but this is working in my current program.
Related
I am trying to get followers with python selenium. But sometimes python clicks by itself.
I want to make an error-free program. I try to I've tried "try catch" constructs but it didn't work. Here is my code:
def getFollowers(self):
try:
self.browser.get(f"https://www.instagram.com/{self.username}")
time.sleep(2)
followers=self.browser.find_element_by_xpath("//*[#id='react-root']/section/main/div/header/section/ul/li[2]/a").click()
time.sleep(2)
dialog=self.browser.find_element_by_xpath("/html/body/div[5]/div/div/div[2]")
followerCount=len(dialog.find_elements_by_tag_name("li"))
print(f"first count:{followerCount}")
action=webdriver.ActionChains(self.browser)
//*******************************************Probly my problem is here****************************************
while True:
dialog.click()
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
//**********************************************************************************************************
followers=dialog.find_elements_by_tag_name("li")
followersList=[]
for user in followers:
link=user.find_element_by_css_selector("a").get_attribute("href")
# print(link)
followersList.append(link)
with open("followers.txt","w",encoding="UTF-8") as file:
for item in followersList:
file.write(item+"\n")
time.sleep(5)
except:
pass
I also have def getfollowing and it works flawlessly. If you want I can show it too. But they are almost same.
EDIT: #RohanShah solved my problem. At the bottom of the page you can see the solution.
Edit: I am new here thats why sometimes my questions could be meanless.But please dont decrease my points. Stackoverflow not gonna accept my questions anymore. Please increase my points.
I've had this exact same problem while scrolling the popups. What happens is your dialog.click(), while attempting to focus your key down on the popup, occasionally clicks a user and loads their profile. Your script then crashes as the popup is no longer on the screen.
After a lot of research into solving this problem, I noticed it only happens with usernames that are long. Regardless, I implemented a simple hack to get around this problem.
First we get the url of what the standard scroll looks like. When opening and scrolling the popup, this is the url we are on.
https://www.instagram.com/some_username/followers/
2.Now I have created a function to hold the code for opening the popup. This will be very useful so trap the necessary code into a function. (I don't have the classnames or xpath's on me so please customize the function for yourself)
def openPopup():
self.browser.get(f"https://www.instagram.com/{self.username}")
global popup # we will need to access this variable outside of the function
popup = driver.find_element_by_class_name('popupClass') #you don't have to use class_name
popup.click()
Now we have to tell our while loop to not scan when Selenium accidentally clicks on a user. We will use our URL from step 1. Please make sure the following if-statement is inserted at the TOP of your loop so if there is a break, it will handle it first before trying to access the popup.
while True:
check_url = self.browser.current_url #returns string with current_url
if check_url != 'https://www.instagram.com/some_username/followers/':
#if this code is executed, this means there has been an accidental click
openPopup() #this will bring back to the page and reopen popup
#the rest of your code
popup.click() # variable from our function
action.key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
time.sleep(3)
newCount=len(dialog.find_elements_by_tag_name("li"))
if followerCount!=newCount or newCount==24:
print(f"New count:{newCount}")
time.sleep(3)
followerCount=newCount
else:
break
check_url = self.browser.current_url #we must recheck the current_url every time the loop runs to see if there has been a misclick
Now, whenever your loop detects the URL is no longer one of the popup, it will automatically call openPopup() which will get you back to the page and back in the popup, and your loop will continue as if nothing happened.
I want to scrape data from an HTML table for different combinations of drop-down values via looping over those combinations. After a combination is chosen, the changes need to be submitted. This is, however, causing an error since it refreshes the page.
This is what I've done so far:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
browser.get('https://daten.ktbl.de/feldarbeit/entry.html')
# Selecting the constant values of some of the drop downs:
fertilizer = Select(browser.find_element_by_name("hgId"))
fertilizer.select_by_value("2")
fertilizer = Select(browser.find_element_by_name("gId"))
fertilizer.select_by_value("193")
fertilizer = Select(browser.find_element_by_name("avId"))
fertilizer.select_by_value("383")
fertilizer = Select(browser.find_element_by_name("hofID"))
fertilizer.select_by_value("2")
# Looping over different combinations of plot size and amount of fertilizer:
size = Select(browser.find_element_by_name("flaecheID"))
for size_values in size.options:
size.select_by_value(size_values.get_attribute("value"))
time.sleep(1)
amount= Select(browser.find_element_by_name("mengeID"))
for amount_values in amount.options:
amount.select_by_value(amount_values.get_attribute("value"))
time.sleep(1)
#Refreshing the page after the two variable values are chosen:
button = browser.find_element_by_xpath("//*[#type='submit']")
button.click()
time.sleep(5)
This leads to the error: selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <option> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed.
Obviously the issue is that I did indeed refresh the document.
I tried it with .submit():
button = browser.find_element_by_xpath("//*[#type='submit']")
button.submit()
This leads to the error: selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: ./ancestor-or-self::form
Time seems not to be the issue here (adding time.sleep or b.implicit.wait didn't change anything).
How can I update the page without breaking from the loop?
EDIT since it was marked as duplicate: I believe that this question does not answer my problem posed here, at least not on my level of python skills, even though the error message is similar. In the other question the problem was solved via added a wait(driver, x) and an expected condition if I'm not mistaken. Here on the other hand the issue was indeed that I did not call the relevant drop-down inside AND outside the loop, as correctly stated by Zaraki Kenpachi.
You need to initiate size select outside the loop to collect sizes and inside loop to grab current page state.
size = Select(browser.find_element_by_name("flaecheID"))
for size in [item.get_attribute("value") for item in size.options]:
size_select = Select(browser.find_element_by_name("flaecheID"))
size_select.select_by_value(str(size))
button = browser.find_element_by_xpath("//input[#type='submit']")
button.click()
I'm trying to .click() a few elements in a pop up-list on a webpage, but keep getting the StaleElementReferenceException when i try to move_to_elements.
The code is based around a number of clickable elements in a feed. When clicked, these elements result in a pop-up box with more clickable elements that I want to access.
I access the pop up-box with the following code, where popupbox_links are a list with coordinates and links for the pop up boxes:
for coordinate in popupbox_links:
actions = ActionChains(driver)
actions.move_to_element(coordinate["Popupbox location"]).perform()
time.sleep(3)
popupboxpath = coordinate["Popupbox link"]
popupboxpath.click()
time.sleep(3)
This works fine. But when the pop up box is opened, I want to perform the following:
seemore = driver.find_element_by_link_text("See More")
time.sleep(2)
actions.move_to_element(seemore).perform()
time.sleep(2)
seemore.click()
time.sleep(3)
findbuttons = driver.find_elements_by_link_text("Button")
time.sleep(2)
print(findbutton)
for button in findbuttons:
time.sleep(2)
actions.move_to_element(button).perform()
time.sleep(2)
button.click()
time.sleep(randint(1, 5))
The trouble starts at actions.move_to_element on both "See more" and "Button". Even though the print(findbutton) actually returns a list with contents, containing the elements that I want to click, Selenium seems to be unable to move_to_element on these. Instead, it throws StaleElementReferenceException.
To make it more confusing, the script seems to work at times. Although usually it just crashes.
Any clues on how to solve this? Big thanks in advance.
I'm running the latest Selenium on Python 3.6 with the Chrome WebDriver.
StaleElementReferenceException Says that element is stage because something has changed in page after you created the webElement object. In you case that could be happening due to button.click().
The simplest solution is to create new element every time, rather than iterate element from the loop.
following changes might work.
findbuttons = driver.find_elements_by_link_text("Button")
time.sleep(2)
print(findbuttons)
for i in range(len(findbuttons)):
time.sleep(2)
elem = driver.find_elements_by_link_text("Button")[i]
actions.move_to_element(elem).perform()
time.sleep(2)
elem.click()
time.sleep(randint(1, 5))
I developed a selenium script, which makes automatic comments in facebook groups.
It works relatively good, but it does not execute the click() method if the targeted element is not visible on the browser.
As a working around I'm using the execute_script("window.scrollTo(x,y";), method, but it's not ideal script. The piece of code that must be improved is the following:
text_box = driver.find_element_by_class_name("UFIAddCommentInput")
try:
driver.execute_script("window.scrollTo(100, 0);")
text_box.click()
except:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
text_box.click()
element = driver.switch_to.active_element
element.send_keys(frase)
element.send_keys(Keys.RETURN)
It first tries for the element on the top of the page, and, if does not get to execute click(), it tries at the bottom.
However, there is a more effective way to scroll at the element found by the find_element_by_class_namemethod?
You can try
text_box.location_once_scrolled_into_view
text_box.click()
to scroll page down right to required element and click it
Having issue with scrolling method. Here is my code, by looks like it works fine, but completely ignoring self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") line.
Is there any other method by chance?
Keep in mind, I cannot use any particular element like 'self.driver.find_element_by_id' at the end, because they keep changing all the time.
def ViewEventHistory(self):
self.login()
self.controlMenu('2')
time.sleep(2)
view_event = self.driver.find_element_by_id('com.eightdevelopment.eas.android:id/btn_event_history')
view_event.click()
time.sleep(10)
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)
As I understand, the problem here is that your are trying to scroll the window, but actually you need to scroll the pop-up that is opened by view_event.click().
You can use a snippet I wrote for one of my projects. It accepts an optional Selenium element and scrolls it. If it's not provided, the function scrolls the main window.
def _scroll(self, scroll_obj=None):
if scroll_obj:
self.driver.execute_script(
'arguments[0].scrollTop = arguments[0].scrollHeight;', scroll_obj)
else:
self.driver.execute_script(
'document.body.scrollTop = document.body.scrollHeight;')