I have the problem with scroll page with selenium and, to be precise, with scrolling to the bottom of the page. This two line of code does not work for me. Who can explain, why?:
browser.execute_script("window.scrollBy(0, document.body.scrollHeight)")
browser.execute_script("window.scrollTo(0, document.body.scrollHeight)")
This is my properties for WebDriver:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
But this options works, but I need to scroll page without connect to some element:
flag = browser.find_element(By.XPATH, ".....")
browser.execute_script("arguments[0].scrollIntoView();", flag)
You can find the body and try:
body = driver.find_element_by_xpath('/html/body')
body.send_keys(Keys.PAGE_DOWN)
You can find more information, e.g. here:
enter link description here
Related
I'm using selenium to get to YouTube and write something on the search bar and then press the button or press the enter key.
Both clicking or pressing a key does sometimes work, but sometimes it does not.
I tried to wait with WebDriverWait, and I even changed the waiting time from 10 to 20 seconds, but it didn't make any difference.
And if I add anything (like printing the new page title), it only shows me the first page title and not the title after the search.
Here is my code and what I tried:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def get_driver():
firefox_options = Options()
# firefox_options.add_argument("--headless")
driver = webdriver.Firefox(executable_path=r"C:\Program Files\Mozilla Firefox\geckodriver.exe", options=firefox_options)
driver.implicitly_wait(9)
return driver
driver = get_driver()
driver.get('https://www.youtube.com/')
search = driver.find_element(By.XPATH, '//input[#id="search"]')
search.send_keys("python")
# search.send_keys(Keys.ENTER) #using the enter key # If I add nothing after this line it work
# searchbutton = driver.find_element(By.XPATH,'//*[#id="search-icon-legacy"]') # This also dose doesn't work
# searchbutton.click() # using the click method() #also dose not work
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="search-icon-legacy"]'))).click() # Sometimes work
# driver.implicitly_wait(10)
# print(driver.title) # This show me only the title of the first page not the one after the search
Is it because I use the Firefox webdriver (should I change to Chrome)?
Or is it because of my internet connection?
To make this working you need to click the search field input first, then add a short delay and then send the Keys.ENTER or click search-icon-legacy element.
So, this is not your fault, this is how YouTube webpage works. You may even call it a kind of bug. But since this webpage it built for human users it works good since human will never click on the input field and insert the search value there within zero time.
Anyway, the 2 following codes are working:
First.
import time
from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
options.add_argument('--disable-notifications')
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.youtube.com/"
driver.get(url)
search = wait.until(EC.element_to_be_clickable((By.XPATH, '//input[#id="search"]')))
search.click()
time.sleep(0.2)
search.send_keys("python")
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="search-icon-legacy"]'))).click()
Second.
import time
from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
options.add_argument('--disable-notifications')
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.youtube.com/"
driver.get(url)
search = wait.until(EC.element_to_be_clickable((By.XPATH, '//input[#id="search"]')))
search.click()
time.sleep(0.2)
search.send_keys("python" + Keys.ENTER)
I'm having a strange issue with Python Selenium when trying to click on the next page link in Curry's PC World Televisions page. See the next button on (https://www.currys.co.uk/tv-and-audio/televisions/tvs)
I am testing in non-headless mode with chrome, so i can see the outline of the next button being highlighted, when actioning the click, but it is not navigating to the next page url as expected.
Here is what i have tried on the page above:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.common.exceptions import NoSuchElementException
import time
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
driver.set_window_size(1920, 1080)
driver.maximize_window()
driver.get('https://www.currys.co.uk/tv-and-audio/televisions/tvs')
next_page_link = WebDriverWait(driver, 10).until(
ec.element_to_be_clickable((By.CSS_SELECTOR,
'ul.pagination '
'li.pagination-arrrow-next a.page-next')))
next_page_link.click()
time.sleep(10)
I have also tried using execute script, (which does not work either):
driver.execute_script('arguments[0].click()', next_page_link)
I am able to extract the href from the link element (so i know its the right element):
href = next_page_link.get_attribute('href')
print("href is: {}".format(href))
# href is: https://www.currys.co.uk/tv-and-audio/televisions/tvs?start=20&sz=20
I'm a bit lost as to what the issue might be here, would appreciate any help!
I see the following issues here:
You have to close the accept cookies pop-up.
You need to scroll the page down to the pager elements.
Selenium indeed doesn't make the page to go to the next page BUT even when I click the Next button manually or even any other pagination button there manually the page is not changed. So, the problem is with the web page itself, not with Selenium.
This is the code I used:
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = 'https://www.currys.co.uk/tv-and-audio/televisions/tvs'
driver.get(url)
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
next_page_link = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR,'ul.pagination li.pagination-arrrow-next a.page-next')))
next_page_link.location_once_scrolled_into_view
time.sleep(1)
next_page_link.click()
I just need to click the load more button once to reveal a bunch more information so that I can scrape more HTML than what is loaded.
The following "should" go to github.com/topics and find the one and only button element and click it one time.
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Edge()
driver.get("https://github.com/topics")
time.sleep(5)
btn = driver.find_element(By.TAG_NAME, "button")
btn.click()
time.sleep(3)
driver.quit()
I'm told Message: element not interactable so I'm obviously doing something wrong but I'm not sure what.
use
btn = driver.findElementsByXPath("//button[contains(text(),'Load more')]");
You are not finding the right element. This is the reason why it is not "interactable"
There are several issues with your code:
The "Load more" button is initially out of the view, so you have to scroll the page in order to click it.
Your locator is bad.
You need to wait for elements to appear on the page before accessing them. WebDriverWait expected_conditions explicit waits should be used for that, not hardcoded sleeps.
The following code works, it scrolls the page and clicks "Load more" 1 time.
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 20)
url = "https://github.com/topics"
driver.get(url)
load_more = wait.until(EC.presence_of_element_located((By.XPATH, "//button[contains(.,'Load more')]")))
load_more.location_once_scrolled_into_view
time.sleep(1)
load_more.click()
UPD
You can simply modify the above code to make it clicking Load more button while it presented.
I implemented this with infinite while loop making a break if Load more button not found. This code works.
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)
url = "https://github.com/topics"
driver.get(url)
while True:
try:
load_more = wait.until(EC.presence_of_element_located((By.XPATH, "//button[contains(.,'Load more')]")))
load_more.location_once_scrolled_into_view
time.sleep(1)
load_more.click()
except:
break
I would like to scroll a container within a webpage - not the webpage itself - using selenium within Python. Would anyone have any solutions to this?
I have tried the standard:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
Of course, this is fine for traversing the full web page, but not a container itself.
EDIT: The webpage is https://www.mixcloud.com/
The container shows when you use the search bar at the top of the page.
This is one way to scroll the page (it will stop once it goes over 300 songs):
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchShadowRootException, NoSuchElementException
import time as t
import pandas as pd
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,720")
webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
actions = ActionChains(browser)
wait = WebDriverWait(browser, 20)
url = "https://www.mixcloud.com/"
browser.get(url)
s_box = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'input[name="mixcloud_query"]')))
s_box.click()
s_box.send_keys('house')
s_box.send_keys(Keys.ENTER)
music_div = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'body[class="rebrand"]')))
music_div.click()
while True:
t.sleep(5)
music_div.send_keys(Keys.END)
print('scrolled to bottom')
songs = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'div[class^="SearchAudioCard__SearchAudioCardContainer"]')))
print('songs:', len(songs))
print('______________-')
if len(songs) > 300:
print('got over 300 songs, stopping')
break
browser.quit()
Selenium documentation can be found at https://www.selenium.dev/documentation/
I know that this question was asked many times on stackoverflow. I tried different solutions but did not get it work. Here is a simple MWE to automate the search on Youtube. Any body familiar with this can help explain the raison ?
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
options = Options()
options.add_argument('--disable-extensions')
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--remote-debugging-port=9515')
options.add_argument('--disable-setuid-sandbox')
options.add_argument("--start-maximized")
driver = webdriver.Chrome(service=Service("/usr/bin/chromedriver"), options=options)
url = "https://www.youtube.com/"
driver.get(url)
search_area = driver.find_element(By.XPATH, '//*[#id="search"]')
driver.implicitly_wait(10)
search_area.send_keys('Lionel Messi', Keys.ENTER)
print(search_area.text)
There are 2 problems with your code:
You should use unique locator.
The locator you are using here matching 5 elements on that page, so Selenium returns you the first element on the page matching the locator you passing here while you need the second match.
This locator will work better //input[#id="search"]
You need to add a delay.
Selenium returns you search_area element at the moment that element is found on the page while it is still may be not ready to be interacted with.
The best way to overcome this issue is to use Explicit Waits.
So, this code should work better:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
options = Options()
options.add_argument('--disable-extensions')
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--remote-debugging-port=9515')
options.add_argument('--disable-setuid-sandbox')
options.add_argument("--start-maximized")
driver = webdriver.Chrome(service=Service("/usr/bin/chromedriver"), options=options)
wait = WebDriverWait(driver, 20)
url = "https://www.youtube.com/"
driver.get(url)
search_area = wait.until(EC.visibility_of_element_located((By.XPATH, "//input[#id='search']")))
search_area.send_keys('Lionel Messi', Keys.ENTER)
print(search_area.text)
P.S.
driver.implicitly_wait(10) is NOT a delay command, this will not put a delay of 10 seconds on the line where you put it. This only defines the timeout for Selenium to way for element presence. Also, you put if AFTER locating the element, so this could not affect the previously located element.