I am trying to scrape this:
https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393
And this is my code:
wait = WebDriverWait(d, 10)
close = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#id='closeButton']")))
close.click()
time.sleep(5)
chart = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(*,'Size Guide')][#class='size-chart-link']")))
chart.click()
It first closes the pop up and then clicks the size guide, However, it always gives timeout exception and works only a couple of times.
The PARTIAL_LINK_TEXT Size Guide is pretty much unique within the page so would be your best bet would be to:
Induce WebDriverWait for invisibility_of_element() for the wrapper element
Induce WebDriverWait for the element_to_be_clickable() for the desired element
You can use the following Locator Strategy:
Code Block (using XPATH and PARTIAL_LINK_TEXT):
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#id='closeButton']"))).click()
WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.XPATH, "//div[#id='tinymask']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Size Guide"))).click()
Code Block (using CSS_SELECTOR and PARTIAL_LINK_TEXT):
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#closeButton"))).click()
WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.CSS_SELECTOR, "div#tinymask")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Size Guide"))).click()
Browser Snapshot:
Use JavaScript Executor to click on the element.Seems like selenium webdriver unable to click on the element.Use the below xpath
d.get("https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393")
wait = WebDriverWait(d, 10)
close = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#id='closeButton']")))
close.click()
chart = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='size-chart-link']/a[contains(.,'Size Guide')]")))
d.execute_script("arguments[0].click();", chart)
Browser snapshot:
Related
I want to find out the "Accept All" button xpath for click accept cookies.
Code trials:
from ast import Pass
import time
from selenium import webdriver
driver = driver = webdriver.Chrome(executable_path=r'C:\Users\Nahid\Desktop\Python_code\Jobsite\chromedriver.exe') # Optional argument, if not specified will search path.
driver.get('http://jobsite.co.uk/')
driver.maximize_window()
time.sleep(1)
#find out XPath in div tag but there has another span tag
cookie = driver.find_element_by_xpath('//div[#class="privacy-prompt-button primary-button ccmgt_accept_button "]/span')
cookie.click()
The desired element:
<div id="ccmgt_explicit_accept" class="privacy-prompt-button primary-button ccmgt_accept_button ">
<span>Accept All</span>
</div>
is a <span> tag having an ancestor <div>.
Solution
To click on the clickable element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.privacy-prompt-button.primary-button.ccmgt_accept_button>span"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Accept All']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Your XPath looks correct but if can be improved.
Also you should use WebDriverWait expected conditions instead of hardcoded sleeps.
As following:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("--start-maximized")
s = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=s)
url = 'http://jobsite.co.uk/'
wait = WebDriverWait(driver, 10)
driver.get(url)
wait.until(EC.element_to_be_clickable((By.ID, "ccmgt_explicit_accept"))).click()
I would like to scrape data from this page: https://www.investing.com/equities/nvidia-corp-financial-summary.
There are two buttons that I'd like to click:
Accept button.
Checking the XPath of the button:
XPath = //*[#id="onetrust-accept-btn-handler"]
Replicating the steps performed here: Clicking a button with selenium using Xpath doesn't work
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
wait = WebDriverWait(driver, 5)
link= wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id="onetrust-accept-btn-handler")))
I got the error: SyntaxError: invalid syntax
Annual button
there is a toggle between Annual and Quarterly (default is quarterly)
XPath is //*[#id="leftColumn"]/div[9]/a[1]
wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id="leftColumn"]/div[9]/a[1]")))
also returned invalid Syntax.
Updated Code
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
company = 'nvidia-corp'
driver = webdriver.Chrome(path)
driver.get(f"https://www.investing.com/equities/{company}-financial-summary")
wait = WebDriverWait(driver, 2)
accept_link= wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="onetrust-accept-btn-handler"]')))
accept_link.click()
scrollDown = "window.scrollBy(0,500);"
driver.execute_script(scrollDown)
#scroll down to get the page data below the first scroll
driver.maximize_window()
time.sleep(10)
wait = WebDriverWait(driver, 2)
scrollDown = "window.scrollBy(0,4000);"
driver.execute_script(scrollDown)
#scroll down to get the page data below the first scroll
try:
close_popup_link= wait.until(EC.element_to_be_clickable((By.XPATH,'//*[#id="PromoteSignUpPopUp"]/div[2]/i')))
close_popup_link.click()
except NoSuchElementException:
print('No such element')
wait = WebDriverWait(driver, 3)
try:
annual_link = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="leftColumn"]/div[9]/a[1]')))
annual_link()
# break
except NoSuchElementException:
print('No element of that id present!')
The first accept button was successfully clicked,
but clicking the Annual button returns Timeout Exception error.
Annual button
At least for me I saw we need to use another locator to access that element.
I used scrolling until I can click that element.
The following code works for me:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("--start-maximized")
s = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=s)
company = 'nvidia-corp'
wait = WebDriverWait(driver, 5)
driver.get(f"https://www.investing.com/equities/{company}-financial-summary")
try:
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="PromoteSignUpPopUp"]/div[2]/i'))).click()
except:
pass
while True:
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='float_lang_base_1']//a[#data-ptype='Annual']"))).click()
break
except:
driver.execute_script("window.scrollBy(0, arguments[0]);", 1000)
You need to take care of a couple of things here as follows:
If you are supplying the xpath within double qoutes, i.e. "..." then the attribute values needs to be within single quotes, i.e. '...'
Similarly, if you are supplying the xpath within single qoutes, i.e. '...' then the attribute values needs to be within double quotes, i.e. "..."
This take care of both the SyntaxError: invalid syntax
Effectively, the lines of code will be:
link= wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id='onetrust-accept-btn-handler')))
and
wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id='leftColumn']/div[9]/a[1]")))
Solution
To click on the clickable elements you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Clicking on I Accept:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='onetrust-accept-btn-handler']"))).click()
Clicking on Annual:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[data-ptype='Annual']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#data-ptype='Annual']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I've tried using the elements linktext, value and xpath. I cant seem to make it click on the button with anything. What am I doing wrong?
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:/Users/yongs/Downloads/chromedriver_win32/chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://ttsfree.com/")
textbox = driver.find_element("id", "input_text")
textbox.send_keys("Text to convert")
driver.implicitly_wait(5)
button_xpath = "/html/body/section[2]/div[2]/form/div[2]/div[2]/a"
button = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, button_xpath)))
actions = ActionChains(driver)
actions.click(button)
actions.perform()
EDITED to include download solution as well
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.relative_locator import locate_with
import time as t
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
# chrome_options.add_argument("--headless")
webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
url = 'https://ttsfree.com/'
browser.get(url)
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//button/span[text()='AGREE']"))).click()
textbox = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.ID, "input_text")))
textbox.send_keys("Text to convert")
button = WebDriverWait(browser, 2000).until(EC.element_to_be_clickable((By.XPATH, "//a[text()='Convert Now']")))
button.click()
print('clicked!')
t.sleep(10)
browser.execute_script('document.getElementsByClassName("label_process text-left")[0].scrollIntoView();')
dl_button = WebDriverWait(browser, 2000).until(EC.element_to_be_clickable((By.CLASS_NAME, "fa-download")))
dl_button.click()
t.sleep(10)
browser.quit()
To click on the element Convert Now you need to induce WebDriverWait for the element_to_be_clickable() which automatically scrolls the element within view and you can use the following locator strategy:
Using LINK_TEXT:
driver.execute("get", {'url': 'https://ttsfree.com/'})
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "textarea#input_text"))).send_keys("Hello")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Convert Now"))).click()
Using CSS_SELECTOR:
driver.execute("get", {'url': 'https://ttsfree.com/'})
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "textarea#input_text"))).send_keys("Hello")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.convert-now"))).click()
Using XPATH:
driver.execute("get", {'url': 'https://ttsfree.com/'})
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "textarea#input_text"))).send_keys("Hello")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[text()='Convert Now']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I want to get the data-sitekey, but it is inside the iframe.
I only can get the element in class="container", can't find the element insde of it.
How can I get the data-sitekey?
driver.get(url)
driver.switch_to.frame("main-iframe")
container= driver.find_element(By.CLASS_NAME, 'container')
print(container)
time.sleep(2)
captcha = driver.find_element(By.CLASS_NAME, 'g-recaptcha')
print(captcha)
The reCAPTCHA element is within an <iframe>
Solution
To extract the value of the data-sitekey attribute you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the visibility_of_element_located.
You can use either of the following locator strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#main-iframe")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.g-recaptcha"))).get_attribute("data-sitekey"))
Using XPATH:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='main-iframe']"))).get_attribute("data-sitekey"))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#class='g-recaptcha']")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
This is how you get that information:
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.options import Options as Firefox_Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support import expected_conditions as EC
import time as t
firefox_options = Firefox_Options()
# firefox_options.add_argument("--width=1280")
# firefox_options.add_argument("--height=720")
# firefox_options.headless = True
driverService = Service('chromedriver/geckodriver')
browser = webdriver.Firefox(service=driverService, options=firefox_options)
url = 'https://premier.hkticketing.com/'
browser.get(url)
t.sleep(5)
WebDriverWait(browser, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//*[#id='main-iframe']")))
print('switched')
t.sleep(5)
element_x = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='g-recaptcha']")) )
print(element_x.get_attribute('data-sitekey'))
Result printed in terminal:
switched
6Ld38BkUAAAAAPATwit3FXvga1PI6iVTb6zgXw62
Setup is for linux/Firefox/geckodriver, but you can adapt it to your own system, just mind the imports, and the code after defining the browser.
Selenium docs: https://www.selenium.dev/documentation/
I don't know how to crawl the title of the page,below is my code(it's simple),but I have no idea where is wrong, if you have any idea please let me know,thank you.
from selenium import webdriver
url="https://sukebei.nyaa.si/?s=seeders&o=desc&p=1"
driver_path = "C:\\webdriver\\chromedriver.exe"
option = webdriver.ChromeOptions()
driver = webdriver.Chrome(driver_path, options=option)
driver.implicitly_wait(10)
driver.get(url)
print(driver.find_element_by_xpath("/html/head/title").text)
To crawl the title of the page you have to induce WebDriverWait for the visibility_of_element_located() for the <table> with torrent-list and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get('https://sukebei.nyaa.si/?s=seeders&o=desc&p=1')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.torrent-list")))
print(driver.title)
Using XPATH:
driver.get('https://sukebei.nyaa.si/?s=seeders&o=desc&p=1')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[contains(#class, 'torrent-list')]")))
print(driver.title)
Console Output:
Browse :: Sukebei
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
url="https://sukebei.nyaa.si/?s=seeders&o=desc&p=1"
driver_path = "C:\\webdriver\\chromedriver.exe"
option = webdriver.ChromeOptions()
driver = webdriver.Chrome(driver_path, options=option)
driver.implicitly_wait(10)
driver.get(url)
print(driver.title)