I'm just a beginner with programming and I'm trying to learn how to automate web procedures.
I am working based on https://www.coeweb.istat.it/.
I need to find elements of the page and I already tried
with driver.find_element_by_xpath
but since I'm not very familiar with html code, I'm not even sure to be using the right piece of code as argument for that method.
Can someone please send me as an example of code to click on the link qui. in bold right in the middle of the page please?
This is what I've tried:
from selenium import webdriver
from selenium.webdriver.common.by import By
url = "https://www.coeweb.istat.it/"
driver = webdriver.Chrome()
driver.get(url)
link = driver.find_element_by_xpath('//[#id="AutoNumber1"]/tbody/tr[3]/td[2]/p[3]/font/a/strong')
link.click()
Still, I get the error as:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="AutoNumber1"]/tbody/tr[3]/td[2]/p[3]/font/a/strong"}
You have to wait until it will be clickable:
# I have fixed xpath
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//*[#id='AutoNumber1']//a[contains(., 'qui.')]"))
)
element.click()
Note: you have to do some imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
There is also possible to use CSS Selector to locate the element:
#AutoNumber1 > tbody > tr:nth-child(3) > td:nth-child(2) > p:nth-child(3) > font > a
and
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "#AutoNumber1 > tbody > tr:nth-child(3) > td:nth-child(2) > p:nth-child(3) > font > a"))
)
element.click()
EDIT: you have to switch to the frame before locate your element like this:
driver.switch_to.frame(driver.find_element_by_xpath("//frame[#name = 'principale']"))
So the full code will be like this:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = "https://www.coeweb.istat.it/"
driver = webdriver.Chrome()
driver.get(url)
driver.switch_to.frame(driver.find_element_by_xpath("//frame[#name = 'principale']")) # switches to frame
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//*[#id='AutoNumber1']//a[contains(., 'qui.')]"))
)
element.click()
driver.switch_to.default_content() # switch back to default content
Explanation: you cannot interact with the elements which are in iframe or in a frame. To be able to do this, you have to find a frame, switch to it, do stuff and then switch back to default content
As per the URL you have shared the link with text as qui. is withan a <frame>. So you have to switch to the desired frame inducing WebDriverWait and then again induce WebDriverWait while looking out for the element as follows:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
url = "https://www.coeweb.istat.it/"
driver.get(url)
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"principale")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//table[#id='AutoNumber1']//tr//td//p//a[#href='lnkdati.htm' and .//strong[contains(.,'qui.')]]"))).click()
Browser Snapshot:
Related
I would like to scrape data from this page: https://www.investing.com/equities/nvidia-corp-financial-summary.
There are two buttons that I'd like to click:
Accept button.
Checking the XPath of the button:
XPath = //*[#id="onetrust-accept-btn-handler"]
Replicating the steps performed here: Clicking a button with selenium using Xpath doesn't work
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
wait = WebDriverWait(driver, 5)
link= wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id="onetrust-accept-btn-handler")))
I got the error: SyntaxError: invalid syntax
Annual button
there is a toggle between Annual and Quarterly (default is quarterly)
XPath is //*[#id="leftColumn"]/div[9]/a[1]
wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id="leftColumn"]/div[9]/a[1]")))
also returned invalid Syntax.
Updated Code
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
company = 'nvidia-corp'
driver = webdriver.Chrome(path)
driver.get(f"https://www.investing.com/equities/{company}-financial-summary")
wait = WebDriverWait(driver, 2)
accept_link= wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="onetrust-accept-btn-handler"]')))
accept_link.click()
scrollDown = "window.scrollBy(0,500);"
driver.execute_script(scrollDown)
#scroll down to get the page data below the first scroll
driver.maximize_window()
time.sleep(10)
wait = WebDriverWait(driver, 2)
scrollDown = "window.scrollBy(0,4000);"
driver.execute_script(scrollDown)
#scroll down to get the page data below the first scroll
try:
close_popup_link= wait.until(EC.element_to_be_clickable((By.XPATH,'//*[#id="PromoteSignUpPopUp"]/div[2]/i')))
close_popup_link.click()
except NoSuchElementException:
print('No such element')
wait = WebDriverWait(driver, 3)
try:
annual_link = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="leftColumn"]/div[9]/a[1]')))
annual_link()
# break
except NoSuchElementException:
print('No element of that id present!')
The first accept button was successfully clicked,
but clicking the Annual button returns Timeout Exception error.
Annual button
At least for me I saw we need to use another locator to access that element.
I used scrolling until I can click that element.
The following code works for me:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("--start-maximized")
s = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=s)
company = 'nvidia-corp'
wait = WebDriverWait(driver, 5)
driver.get(f"https://www.investing.com/equities/{company}-financial-summary")
try:
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="PromoteSignUpPopUp"]/div[2]/i'))).click()
except:
pass
while True:
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='float_lang_base_1']//a[#data-ptype='Annual']"))).click()
break
except:
driver.execute_script("window.scrollBy(0, arguments[0]);", 1000)
You need to take care of a couple of things here as follows:
If you are supplying the xpath within double qoutes, i.e. "..." then the attribute values needs to be within single quotes, i.e. '...'
Similarly, if you are supplying the xpath within single qoutes, i.e. '...' then the attribute values needs to be within double quotes, i.e. "..."
This take care of both the SyntaxError: invalid syntax
Effectively, the lines of code will be:
link= wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id='onetrust-accept-btn-handler')))
and
wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id='leftColumn']/div[9]/a[1]")))
Solution
To click on the clickable elements you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Clicking on I Accept:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='onetrust-accept-btn-handler']"))).click()
Clicking on Annual:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[data-ptype='Annual']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#data-ptype='Annual']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I wrote this short programm to login in to my postfield:
import time
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome()
driver.maximize_window()
driver.get('https://www.web.de/')
time.sleep(2)
frame = driver.find_element_by_name('landingpage')
# do something with frame ...
driver.switch_to.frame(frame)
innerFrame = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(innerFrame)
driver.find_element_by_id("save-all-conditionally").click()
time.sleep(2)
driver.switch_to.default_content()
time.sleep(3)
inputemail = driver.find_element_by_xpath('/html/body/div[2]/div/div[2]/div[2]/section[1]/div/form/div[2]/div/input')
inputpwd = driver.find_element_by_xpath('/html/body/div[2]/div/div[2]/div[2]/section[1]/div/form/div[3]/div/input')
buttonsend = driver.find_element_by_xpath('/html/body/div[2]/div/div[2]/div[2]/section[1]/div/form/button')
inputemail.send_keys('xxx#web.de')
inputpwd.send_keys('xxx')
buttonsend.click()
time.sleep(3)
frame2 = driver.find_element_by_name('home')
driver.switch_to.frame(frame2)
linkwrite = driver.find_element_by_xpath('/html/body/div[3]/div/div[3]/div[2]/div[1]/div[6]/div[1]/ul/li/section/div[3]/div/div[1]/a')
linkwrite.click()
This part is working fine with the iframes there. My next goal was then to fill out an input field. The Code of the page which opened after the sign in progress is the one on the picture: https://www.transfernow.net/dl/20210919Porfd5N7
But the Code for filling:
frame3 = driver.find_element_by_xpath('...')
driver.switch_to.frame(frame3)
fillin = driver.find_element_by_xpath('/html/body/div[3]/div[3]/div[3]/div[1]/div[1]/div/form/div[2]/div[1]/div[2]/div[1]/div[1]/div[2]/div/div/ul/li/input')
fillin.send_keys('hello')
has resulted in:
"Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/nav/div[2]/div[1]/div[2]/a[1]"}"
Where's my mistake?
Please help me!
Few things
There are nested iframe, first one is iframe[name='landingpage'] (located by css selector) second one is iframe[sandbox*='allow-popups'][style^='display'] and then you can click on Zustimmen und weiter button.
The xpath that you are using is absolute, try using relative xpath, if possible switch to css_selector.
Use Explicit waits.
Code :-
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
wait = WebDriverWait(driver, 20)
driver.get("https://web.de/consent-management/")
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[name='landingpage']")))
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[sandbox*='allow-popups'][style^='display']")))
wait.until(EC.element_to_be_clickable((By.ID, "save-all-conditionally"))).click()
driver.switch_to.default_content()
inputemail = wait.until(EC.element_to_be_clickable((By.ID, "freemailLoginUsername")))
inputpwd = wait.until(EC.element_to_be_clickable((By.ID, "freemailLoginPassword")))
buttonsend = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']")))
inputemail.send_keys('xxx#web.de')
inputpwd.send_keys('xxx')
buttonsend.click()
time.sleep(3)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Additionally you can use the below code :
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.pos-input input"))).send_keys('WEB.DE E-Mail-Adresse')
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.pos-input input[name^='password']"))).send_keys('password here')
to fill details on Bitte erneut einloggen page.
On this page 'https://www.nj.gov/health/cd/topics/covid2019_dashboard.shtml', I try to get the number of New Jersey Positives (upper right of the dashboard).
positiveCount = driver.find_elements_by_xpath("//text[#style='stroke-width: 2; font-size: 160px; line-height: normal;']")
print len(positiveCount)
it always show 0.
What did I do wrong? Thanks.
The data is wrapped in iframe so you have to switch to the iframe first and then get the details.
driver.switch_to.frame(0)
driver.find_element_by_css_selector('text').text
Result:
'6,876'
Screenshot:
Please find solution
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(executable_path=r"C:\New folder\chromedriver.exe")
url = 'https://www.nj.gov/health/cd/topics/covid2019_dashboard.shtml'
driver.get(url)
driver.maximize_window()
iframe = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.TAG_NAME, 'iframe')))
driver.switch_to.frame(iframe)
svg = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.CSS_SELECTOR, 'text')))
print svg.text
I am trying to scrape this:
https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393
And this is my code:
wait = WebDriverWait(d, 10)
close = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#id='closeButton']")))
close.click()
time.sleep(5)
chart = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(*,'Size Guide')][#class='size-chart-link']")))
chart.click()
It first closes the pop up and then clicks the size guide, However, it always gives timeout exception and works only a couple of times.
The PARTIAL_LINK_TEXT Size Guide is pretty much unique within the page so would be your best bet would be to:
Induce WebDriverWait for invisibility_of_element() for the wrapper element
Induce WebDriverWait for the element_to_be_clickable() for the desired element
You can use the following Locator Strategy:
Code Block (using XPATH and PARTIAL_LINK_TEXT):
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#id='closeButton']"))).click()
WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.XPATH, "//div[#id='tinymask']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Size Guide"))).click()
Code Block (using CSS_SELECTOR and PARTIAL_LINK_TEXT):
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#closeButton"))).click()
WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.CSS_SELECTOR, "div#tinymask")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Size Guide"))).click()
Browser Snapshot:
Use JavaScript Executor to click on the element.Seems like selenium webdriver unable to click on the element.Use the below xpath
d.get("https://www.lanebryant.com/chiffon-faux-wrap-fit-flare-midi-dress/prd-355958#color/0000091393")
wait = WebDriverWait(d, 10)
close = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#id='closeButton']")))
close.click()
chart = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='size-chart-link']/a[contains(.,'Size Guide')]")))
d.execute_script("arguments[0].click();", chart)
Browser snapshot:
I am trying to press this "New Search" button. It appears on the top of the screen after entering a search on http://www.lexisnexis.com/hottopics/lnacademic/
I have looked at the Xpath and Unique Selector.
What I have tried:
browser.find_element_by_css_selector('#restoreButtons > a:nth-child(3)').click()
browser.find_element_by_xpath(id('restoreButtons')/x:a[3])
browser.find_element_by_xpath(/x:a[3])
For all three I get an "unable to locate element error"
To build on alecxes answer,
once you're inside the iframe, you can find the clickable using it's xpath:
new_search_xpath = '/html/body/div[2]/table/tbody/tr[2]/td[2]/div[1]/table/tbody/tr/td/span/a[3]'
new_search = driver.find_element(By.XPATH, new_search_xpath)
new_search.click()
You should consider installing firebug firefox addon to grab Xpaths:
https://addons.mozilla.org/en-US/firefox/addon/firebug/
This is because the element is inside an iframe. You have to be in the context to search elements inside. Use .switch_to.frame():
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
wait = WebDriverWait(driver, 10)
frame = wait.until(EC.presence_of_element_located((By.ID, "mainFrame")))
driver.switch_to.frame(frame)
FYI, here is the complete working code:
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('http://www.lexisnexis.com/hottopics/lnacademic/')
actions = ActionChains(driver)
wait = WebDriverWait(driver, 10)
frame = wait.until(EC.presence_of_element_located((By.ID, "mainFrame")))
driver.switch_to.frame(frame)
driver.find_element_by_id("terms").send_keys("Test")
driver.find_element_by_id("srchButt").click()