hey guys I'm scrapping abnb with Selenium-Python. The issue is that I want to scrap the reviews from each abnb but in order to scrap them I need to scroll down the pop up element that appears.
As you can see on the image I click the button then the pop-up element appears and then I want to scroll but I can't. Of course as you can see all the reviews don't appear and I have to scroll if I want them to appear, they also don't appear on the DevTools from google chrome unless I scroll.
I tried:
reviews_lista = []
boton_revs = driver.find_element(By.XPATH, "//button[#data-testid='pdp-show-all-reviews-button']")
boton_revs.click()
I clicked on the button that you see on the image.
I selected the whole POP UP
bloque_pop = driver.find_element(By.XPATH, "//div[#class='_14vzertx']")
I tried to scroll with this code but it didn't work
scroll = 0
while scroll < 5: # scroll 5 times
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollTop + arguments[0].offsetHeight;', bloque_pop)
time.sleep(1)
scroll += 1
I also tried
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
and this too:
bloque_pop = driver.find_element(By.XPATH, "//div[#class='_14vzertx']")
for i in range(5):
#driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", bloque_pop)
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)", bloque_pop)
time.sleep(1.5)
None of them work.
The easiest way to scroll inside a popup is to scrape elements contained in the popup and then scroll to the last element by using the javascript command scrollIntoView.
popup_reviews = driver.find_elements(By.CSS_SELECTOR, 'div[role=dialog] div[data-review-id]')
driver.execute_script('arguments[0].scrollIntoView({block: "center", behavior: "smooth"});', popup_reviews[-1])
Related
Essentially title - I've seen plenty of methods here for scrolling on webpages but not on the side panel of a webpage. Here's the code - the scrolling part seems to have no effect:
driver = webdriver.Chrome(chromedriver_path) # chromedriver path
driver.get('https://www.google.com/maps/search/gas/#40,-75.2,12z/data=!4m2!2m1!6e2')
time.sleep(3)
driver.execute_script("window.scrollTo(0, 100)")
time.sleep(3)
Here's the page I'm trying to access. Any tips?
You can scroll to a specific element using
els = driver.find_elements(By.CSS_SELECTOR, '.TFQHme')
driver.execute_script("arguments[0].scrollIntoView();", els[-1])
In this example the last one loaded.
The list will grow as you scroll down, to keep scrolling, just retrieve the list again.
You can implement an infinite scroll using the following:
def infinite_scroll(driver):
number_of_elements_found = 0
while True:
els = driver.find_elements(By.CSS_SELECTOR, '.TFQHme')
if number_of_elements_found == len(els):
# Reached the end of loadable elements
break
try:
driver.execute_script("arguments[0].scrollIntoView();", els[-1])
number_of_elements_found = len(els)
except StaleElementReferenceException:
# Possible to get a StaleElementReferenceException. Ignore it and retry.
pass
so I am trying to automate a program that would log in to a google account I created, go to this canvas website, and draw a circle,
(Circle just as a placeholder because im trying to make it draw some cool stuff like a car, just to test if it will work first.) But the main issue is when you first go into the website, there is a pop up that displays and that pop up has 2 options, "learn more" and "Get started". I tried to make selenium click on the "Get started" using driver.find_element_by_id('Get-started').click but it does not seem to work, I also tried to use CSS selector but that does not seem to work either. So, I'm stuck on this. Any advice or help to click the get started button? (Feel free to also give me advice on how to draw in the canvas as well!)
Here's the HTML:
<paper-button id="get-started" dialog-confirm="" class="primary" aria-label="Get started" role="button" tabindex="0" animated="" elevation="0" aria-disabled="false">
Get started
</paper-button>
here's the code:
from selenium import webdriver
import time
from PrivateInfo import*
driver = webdriver.Chrome(r"D:\chromeDriver\chromedriver.exe")
link = "https://www.google.com/"
driver.get(link)
def linkText(element):
button = driver.find_element_by_link_text(element)
button.click()
def byClass(className):
button2 = driver.find_element_by_class_name(className)
button2.click()
def type(text, elements):
input = driver.find_elements_by_name(elements)
for element in input:
driver.implicitly_wait(2)
element.send_keys(text)
linkText("Sign in")
type(email, "identifier")
byClass("VfPpkd-vQzf8d")
type(pw, "password")
driver.find_element_by_id("passwordNext").click()
time.sleep(1)
driver.get("https://canvas.apps.chrome/")
driver.implicitly_wait(3)
driver.find_element_by_css_selector('paper-button[id="get-started"]').click()
edit: I also tried this
getStart = driver.find_elements_by_css_selector('paper-button[id="get-started"]')
for start in getStart:
start.click()
it doesn't give me any errors but it does not do anything.
Ah yeah, forgot to mention but im new to using selenium.
The content of the popup is nested inside shadowRoots.
More on ShadowRoot here:
The ShadowRoot interface of the Shadow DOM API is the root node of a
DOM subtree that is rendered separately from a document's main DOM
tree.
To be able to control the element you will need to switch between DOMs. Here's how:
drawing_app_el = driver.execute_script("return arguments[0].shadowRoot", driver.find_element(By.CSS_SELECTOR, 'drawing-app'))
This code will retrieve the drawing_app first and then return the content of the shadowRoot.
To have access to the button getStarted, here's how:
# Get the shadowRoot of drawing-app
drawing_app_el = driver.execute_script("return arguments[0].shadowRoot", driver.find_element(By.CSS_SELECTOR, 'drawing-app'))
# Get the shadowRoot of welcome-dialog within drawing_app
welcome_dialog_el = driver.execute_script("return arguments[0].shadowRoot", drawing_app_el.find_element(By.CSS_SELECTOR, 'welcome-dialog'))
# Get the paper-dialog from welcome_dialog
paper_dialog_el = welcome_dialog_el.find_element(By.CSS_SELECTOR, 'paper-dialog')
# Finally, retrieve the getStarted button
get_started_button = paper_dialog_el.find_element(By.CSS_SELECTOR, '#get-started')
get_started_button.click()
main url :https://www.kaplanpathways.com/degree-finder/#/search-result?status=7&institution_short_name=Arizona-State-University-Downtown-Phoenix-Campus&subject_area_name=&university=38°ree_level=20
try:
for i in range(1, 20):
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
driver.find_element_by_xpath("/html/body/div[5]/main/div/div/app-root/app-search-result/div/div[2]/div[2]/div[2]/button[1]").click()
except:
pass
course_list = driver.find_elements_by_xpath("//*[#class='wrap-result']")
print("Total courses: ", len(course_list))
Problem:
When you are moving to the bottom of the page, you actually move out of the clickable area for the "Show more" button. selenium does not click the button if that is not clickable (i.e. out of the screen, or behind some div like accept cookies div in your case).
Solution:
Try clicking by injecting javascript.
driver.execute_script("""document.querySelector(".dgf-show-more-button").click()""")
Note: Also don't forgot to click the "I accept" button for cookies.
I'm writing a Python program to access a website and click buttons that have the same class name.
Conditions:
There are 6 buttons provided per one time.
When a button is clicked, it disappears.
After 6 buttons are clicked, only the section contains the 6 buttons refreshes and the new 6 buttons appear. (then loop like this for a certain amount of times)
The problem is that I can get it to click only the 6 buttons that appear for the first time. After a section contains the buttons refreshes, find_elements_by_class_name cannot find the new 6 buttons.
Question: Is there a way to get find_elements_by_class_name or any alternatives to find the new 6 buttons every time the section refreshes?
Thank you so much in advance for your help. Highly appreciate!
Here is a part of my code.
webdriver_path = '/xxxxxxx/xxxxx/xxxx'
options = Options()
options.headless = True
options.add_argument('--window-size=1920x1080')
driver = webdriver.Chrome(options=options, executable_path=webdriver_path)
driver.get('https://xxxxx.xxx/')
click_n = 1
While True:
all_clicks = driver.find_elements_by_class_name('button_me')
all_clicks[click_n].click()
print('Click SUCESSFUL')
click_n +=1
# if already 6 clicks wait for refresh
if click_n = 6:
driver.implicitly_wait(10)
click_n -= 6
continue
else:
# Wait till click success notice appear
NoticeNCon = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="Hint"]/div/b[contains(text(),"success")]')))
I need to click on a button using selenium in python. This is what I have:
read_more_buttons = responses[0].find_elements_by_class_name("read-more")
if len(read_more_buttons) > 0:
read_more_buttons[0].click()
It works fine most of the time but sometimes there's an overlay on the bottom of the page, which can not be dismissed. Therefore I'd get this error:
[element] is not clickable at point (665.7333145141602,883.4666748046875) because another element <div class="raq-module js-raq-module"> obscures it
I tried to scroll down the page with this code right before calling click():
driver.execute_script("window.scrollTo(0, " + str(read_more_buttons[0].location["y"] + 120) + ")")
However, I'm still receiving the same error. It seems like by calling .click() the element is scrolled to the very bottom of the page, which is right under the overlay. How can I move up the page and then click?
Those dang overlays!
Here, let's try and use JS to scroll into view and then click:
read_more_buttons = responses[0].find_elements_by_class_name("read-more")
if len(read_more_buttons) > 0:
driver.execute_script("arguments[0].scrollIntoView(true);", read_more_buttons[0])
driver.execute_script("arguments[0].click()", read_more_buttons[0])