I`m a newbie to python.
Recently I got interested in Web Crawling.
Today I got stuck in NoSuchElementException
This is the webpage that i want to scrape.
When I click the username that i erased, it returns box like this.
Though I used the xpath that i copied from Chrome developer tool,
it returns me NoSuchElementException:
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="main-area"]/div[4]/table/tbody/tr[1]/td[2]/div/table/tbody/tr/td/a"}
(Session info: chrome=87.0.4280.88)
HTML is like this
이주연마인
My code is just like this,
driver.find_element_by_xpath("//*[#id=\"main-area\"]/div[4]/table/tbody/tr[1]/td[2]/div/table/tbody/tr/td/a")
I checked there is this xpath, but when I get it into .find_element_by_xpath() method it returns Error.
I do really share the webpage, but it needs to log-in to get there,
So i cannot share the webpage.
Could you guess what might cause this problem?
I checked time is not the problem.
I checked iframe is not the problem.
Thank you in advance
Have a great day!
To locate the element with text as 이주연마인 you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using LINK_TEXT:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "이주연마인")))
Using CSS_SELECTOR:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.m-tcol-c[onclick*='royaltina']")))
Using XPATH:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='m-tcol-c' and contains(#onclick, 'royaltina')][text()='이주연마인']")))
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
Related
I am writing a script to automate data collection and was having trouble clicking a link. The website is behind a login, but I navigated that successfully. I ran into problems when trying to navigate to the download page. This is in python using chrome webdriver.
I have tried using:
find_element_by_partial_link_text('stuff').click()
find_element_by_xpath('stuff').click()
#and a few others
I get iterations of following message when I try a few of the selector statements.
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"partial link text","selector":"download"}
(Session info: chrome=88.0.4324.182)
Html source I'm trying to use is:
<a routerlink="/download" title="Download" href="/itron-mvweb/download"><i class="fa fa-lg fa-fw fa-download"></i><span class="menu-item-parent">Download</span></a>
Thank you!
This is caused by a typo. Download is case-sensitive, make sure you capitalize the D!
To click on the element with text as Download you can use either of the following Locator Strategies:
Using css_selector:
driver.find_element(By.CSS_SELECTOR, "a[title='Download'][href='/itron-mvweb/download'] span.menu-item-parent").click()
Using xpath:
driver.find_element(By.XPATH, "//a[#title='Download' and #href='/itron-mvweb/download']//span[#class='menu-item-parent' and text()='Download']").click()
Ideally, to click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[title='Download'][href='/itron-mvweb/download'] span.menu-item-parent"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Download' and #href='/itron-mvweb/download']//span[#class='menu-item-parent' and text()='Download']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I am trying to click on the ARCGIS button on webpage https://gisapr.atco.com/portal/home/signin.html?returnUrl=https%3A%2F%2Fgisapr.atco.com%2Fportal%2Fhome%2F
But getting no element found or timeout errors.
I'm also unable to access inside login form contents but no luck.
Tried:
Implicit and explicit waits
Time sleep()
by all finds..
The element ARCGIS is within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get('https://gisapr.atco.com/portal/home/signin.html?returnUrl=https%3A%2F%2Fgisapr.atco.com%2Fportal%2Fhome%2F')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#oAuthFrame")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.btn.btn_wide#ago_Name"))).click()
Using XPATH:
driver.get('https://gisapr.atco.com/portal/home/signin.html?returnUrl=https%3A%2F%2Fgisapr.atco.com%2Fportal%2Fhome%2F')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='oAuthFrame']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#id='ago_Name' and text()='ArcGIS']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I am new to python and trying to do some webscraping but have some real issues. May be you can help me out.
HTML:
<input autocomplete="off" type="search" name="search-search-field" placeholder="150k companies worldwide" data-cy-id="search-search-field" class="sc-dnqmqq grpFhe" value="">
The first part of my code looks as follows and works good without having any issues:
driver.get("https:")
login = driver.find_element_by_xpath(email_xpath).send_keys(email)
login = driver.find_element_by_xpath(pwd_xpath).send_keys(pwd)
login = driver.find_element_by_xpath(continue_xpath)
login.click()
time.sleep(10)
email and pwd are variables including my login details. As I said that part works pretty fine.
The issues I have are with the following code line:
search = driver.find_element_by_xpath('/html/body/div[1]/div/div[1]/header/div/nav/div[1]/div/div/fieldset/input')
As a result I get this following error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[1]/header/div/nav/div[1]/div/div/fieldset/input"}
I tried and tried but could not solve the problem. I would appreciate it very much, if anyone could help me out. Thank you!
To locate the search field you can use either of the following Locator Strategies:
Using css_selector:
search = driver.find_element_by_css_selector("input[name='search-search-field'][data-cy-id='search-search-field']")
Using xpath:
search = driver.find_element_by_xpath("//input[#name='search-search-field' and #data-cy-id='search-search-field']")
Ideally, to locate the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='search-search-field'][data-cy-id='search-search-field']")))
Using XPATH:
search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='search-search-field' and #data-cy-id='search-search-field']")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
The following Xpath will work and is much simpler.
/html/body/div[1]//fieldset/input
do not use absolute xpath or css , always use relative as it is more stable
Absolute (Full) xpath will be depended on parent and so if parent changes the locator will fail to find the element
in xpath and css the locator can be used in the form:
//tagname[#attributename="attriubutevalue"] - Xpath
tagname[attributename="attriubutevalue"] - CSS
so you can use any attribute , type, name , id ,class, what ever attribute there in your element eg:
//input[#type="search"] - xpath
input[type="search"] - css
search = driver.find_element_by_xpath('//input[#type="search"]')
Try wait:
WebDriverWait(driver,15).until(EC.presence_of_element_located((By.XPATH, '//input[#type="search"]')))
I am trying to get data from a password protected website with Selenium.
However I get stuck right in the beginning at login with the following error message:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[name="username"]"}
(Session info: chrome=87.0.4280.66)
The name I use is correct for sure, I inspected the website. Including waiting time did not help either...
My code is:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from getpass import getpass
driver = webdriver.Chrome()
driver.get("https://clearing.apcs.at/emwebapcsem/startApp.do")
print(driver.title)
print(driver.current_url)
# create an object for searchbox
username=driver.find_element_by_name("username")
password=driver.find_element_by_name("password")
# typte the input
username.send_keys("XXXXXX")
password.send_keys("XXXXXX")
driver.find_element_by_name('login').click()
Any suggestions would be appreciated.
To send a character sequence to the Benutzer and Passwort field as the elements are within an <frame> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get('https://clearing.apcs.at/emwebapcsem/startApp.do')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[title='menu']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.loginContentBoxInput[name='username']"))).send_keys("Endre")
driver.find_element_by_css_selector("input.loginContentBoxInput[name='password']").send_keys("Endre")
Using XPATH:
driver.get('https://clearing.apcs.at/emwebapcsem/startApp.do')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//frame[#title='menu']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#class='loginContentBoxInput' and #name='username']"))).send_keys("Endre")
driver.find_element_by_xpath("//input[#class='loginContentBoxInput' and #name='password']").send_keys("Endre")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I used selenium IDE to trace my UI activity. I got this following code from IDE and i inspected in UI also,but while using find_element by id i'm getting css selector error.
driver.find_element(By.ID, "button-1034-btnIconEl").click()
error is
raise exception_class(message, screen, stacktrace)
NoSuchElementException: no such element: Unable to locate element:
{"method":"css selector","selector":"[id="button-1034-btnIconEl"]"}
(Session info: chrome=78.0.3904.108)
Please help me to debug this..
To click() on the button with text as Add Order you have to induce WebDriverWait for the element_to_be_clickable() you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[id^='button-'] > span.x-btn-wrap > span.x-btn-button > span.x-btn-inner.x-btn-inner-center"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[starts-with(#id, 'button-')]/span[#class='x-btn-wrap']/span[#class='x-btn-button']/span[#class='x-btn-inner x-btn-inner-center' and contains(., 'Add')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a detailed discussion on NoSuchElementException: no such element in Selenium “selenium.common.exceptions.NoSuchElementException” when using Chrome
The id seems to be dynamic one so you cannot use static id in the selector. You need to use a dynamic xpath for this.
You can use the below xpath:
driver.find_element(By.XPATH, "//span[contains(#id,'btnIconEl')]").click()
OR
You can find the element using its text as well in the xpath:
driver.find_element(By.XPATH, "//span[contains(text(),'Add Order')]").click()
Might not be your case, but I made a silly mistake that lead me to this error.
I forgot to get the actual web page using the line:
driver.get(my_link)
and thus, I got the same error