I am new to python and trying to do some webscraping but have some real issues. May be you can help me out.
HTML:
<input autocomplete="off" type="search" name="search-search-field" placeholder="150k companies worldwide" data-cy-id="search-search-field" class="sc-dnqmqq grpFhe" value="">
The first part of my code looks as follows and works good without having any issues:
driver.get("https:")
login = driver.find_element_by_xpath(email_xpath).send_keys(email)
login = driver.find_element_by_xpath(pwd_xpath).send_keys(pwd)
login = driver.find_element_by_xpath(continue_xpath)
login.click()
time.sleep(10)
email and pwd are variables including my login details. As I said that part works pretty fine.
The issues I have are with the following code line:
search = driver.find_element_by_xpath('/html/body/div[1]/div/div[1]/header/div/nav/div[1]/div/div/fieldset/input')
As a result I get this following error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[1]/header/div/nav/div[1]/div/div/fieldset/input"}
I tried and tried but could not solve the problem. I would appreciate it very much, if anyone could help me out. Thank you!
To locate the search field you can use either of the following Locator Strategies:
Using css_selector:
search = driver.find_element_by_css_selector("input[name='search-search-field'][data-cy-id='search-search-field']")
Using xpath:
search = driver.find_element_by_xpath("//input[#name='search-search-field' and #data-cy-id='search-search-field']")
Ideally, to locate the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='search-search-field'][data-cy-id='search-search-field']")))
Using XPATH:
search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='search-search-field' and #data-cy-id='search-search-field']")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
The following Xpath will work and is much simpler.
/html/body/div[1]//fieldset/input
do not use absolute xpath or css , always use relative as it is more stable
Absolute (Full) xpath will be depended on parent and so if parent changes the locator will fail to find the element
in xpath and css the locator can be used in the form:
//tagname[#attributename="attriubutevalue"] - Xpath
tagname[attributename="attriubutevalue"] - CSS
so you can use any attribute , type, name , id ,class, what ever attribute there in your element eg:
//input[#type="search"] - xpath
input[type="search"] - css
search = driver.find_element_by_xpath('//input[#type="search"]')
Try wait:
WebDriverWait(driver,15).until(EC.presence_of_element_located((By.XPATH, '//input[#type="search"]')))
Related
I currently have a Selenium script that is running through a list of part numbers on a website and capturing some information such as product name (pulled from the page title).
I have noticed that some of the product is identified as "DISCONTINUED" (through a span) and so I would like to be able to capture that information so that I can ignore all of those products.
On the website in general, they denote these products through:
<span data*="">DISCONTINUED</span>
Any other products that are valid will not have this information on it, and so in this case I want to ensure that the script doesn't crash and just captures a blank value.
I tried using:
driver.find_element(By.XPATH, '//html/body/div/div/section/div/section/div/div/section/div/div/div/div/div/span/span[text()="DISCONTINUED"]')
However I get this error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//html/body/div/div/section/div/section/div/div/section/div/div/div/div/div/span/span[text()="DISCONTINUED"]"}
I did make sure to use only a single part number that did indeed have a DISCONTINUED on it.
I also did load up the page in dev tools and searched for that specific xpath and it did indeed highlight the proper section:
Searched:
//html/body/div/div/section/div/section/div/div/section/div/div/div/div/div/span/span
Highlighted:
<span data*="">DISCONTINUED</span>
What is the best way to do this? Also, I need to check to see how to include a blank value by default so that it will not crash when a current product is retrieved.
Considering the HTML:
<span data*="">DISCONTINUED</span>
To locate the element with the text DISCONTINUED you can use either of the following locator strategies:
Using xpath and text():
element = driver.find_element(By.XPATH, "//span[text()='DISCONTINUED']")
Using xpath and contains():
element = driver.find_element(By.XPATH, "//span[contains(., 'DISCONTINUED')]")
Ideally to locate the element you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
Using xpath and text():
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[text()='DISCONTINUED']")))
Using xpath and contains():
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(., 'DISCONTINUED')]")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
This is what I used to make it work with error avoidance.
try:
discontinued = WebDriverWait(driver, 2).until(EC.visibility_of_element_located((By.XPATH, "//html/body/div/div/section/div/section/div/div/section/div/div/div/div/div/span/span[text()='ARCHIVED']")))
discontinued = "CURRENT"
except:
discontinued="DISCONTINUED"
Thanks again for the guidance on using waits instead of the find_element approach!
I am writing a script to automate data collection and was having trouble clicking a link. The website is behind a login, but I navigated that successfully. I ran into problems when trying to navigate to the download page. This is in python using chrome webdriver.
I have tried using:
find_element_by_partial_link_text('stuff').click()
find_element_by_xpath('stuff').click()
#and a few others
I get iterations of following message when I try a few of the selector statements.
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"partial link text","selector":"download"}
(Session info: chrome=88.0.4324.182)
Html source I'm trying to use is:
<a routerlink="/download" title="Download" href="/itron-mvweb/download"><i class="fa fa-lg fa-fw fa-download"></i><span class="menu-item-parent">Download</span></a>
Thank you!
This is caused by a typo. Download is case-sensitive, make sure you capitalize the D!
To click on the element with text as Download you can use either of the following Locator Strategies:
Using css_selector:
driver.find_element(By.CSS_SELECTOR, "a[title='Download'][href='/itron-mvweb/download'] span.menu-item-parent").click()
Using xpath:
driver.find_element(By.XPATH, "//a[#title='Download' and #href='/itron-mvweb/download']//span[#class='menu-item-parent' and text()='Download']").click()
Ideally, to click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[title='Download'][href='/itron-mvweb/download'] span.menu-item-parent"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Download' and #href='/itron-mvweb/download']//span[#class='menu-item-parent' and text()='Download']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I`m a newbie to python.
Recently I got interested in Web Crawling.
Today I got stuck in NoSuchElementException
This is the webpage that i want to scrape.
When I click the username that i erased, it returns box like this.
Though I used the xpath that i copied from Chrome developer tool,
it returns me NoSuchElementException:
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="main-area"]/div[4]/table/tbody/tr[1]/td[2]/div/table/tbody/tr/td/a"}
(Session info: chrome=87.0.4280.88)
HTML is like this
이주연마인
My code is just like this,
driver.find_element_by_xpath("//*[#id=\"main-area\"]/div[4]/table/tbody/tr[1]/td[2]/div/table/tbody/tr/td/a")
I checked there is this xpath, but when I get it into .find_element_by_xpath() method it returns Error.
I do really share the webpage, but it needs to log-in to get there,
So i cannot share the webpage.
Could you guess what might cause this problem?
I checked time is not the problem.
I checked iframe is not the problem.
Thank you in advance
Have a great day!
To locate the element with text as 이주연마인 you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using LINK_TEXT:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "이주연마인")))
Using CSS_SELECTOR:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.m-tcol-c[onclick*='royaltina']")))
Using XPATH:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='m-tcol-c' and contains(#onclick, 'royaltina')][text()='이주연마인']")))
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I would like to select this checkbox using selenium
<input type="checkbox" name="rg" onchange="onCheckReg(this)" id="beauty_perfume_screening__c" value="beauty_perfume_screening__c" tabindex="-1" aria-labelledby="check-button-Beauty Perfume Screening check-group-header">
I tried finding by xpath an by id but always get the error unable to locate element
wd.find_element_by_id("beauty_perfume_screening__c").click()
Any Ideas?
Use xpath
driver.find_element_by_xpath("//input[#name='rg']").click()
and recommended is to use one of the wait methods -
WebDriverWait(driver, 10).until(
EC.presence_of_element_located(By.XPATH, "//input[#name='rg']")
For selecting iframe -
root1 = driver.find_element_by_xpath("//iframe")
driver.switch_to.frame(root1)
To click on the element with text as save you can use either of the following Locator Strategies:
Using id:
wd.find_element_by_id("beauty_perfume_screening__c").click()
Using css_selector:
wd.find_element_by_css_selector("input#beauty_perfume_screening__c[name='rg'][value='beauty_perfume_screening__c']").click()
Using xpath:
wd.find_element_by_xpath("//input[#id='beauty_perfume_screening__c' and #name='rg'][#value='beauty_perfume_screening__c']").click()
Ideally, to click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using ID:
WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.ID, "beauty_perfume_screening__c"))).click()
Using CSS_SELECTOR:
WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#beauty_perfume_screening__c[name='rg'][value='beauty_perfume_screening__c']"))).click()
Using XPATH:
WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#id='beauty_perfume_screening__c' and #name='rg'][#value='beauty_perfume_screening__c']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Important: Incase the element is within an <iframe> you have to switch to the <iframe> first.
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I used selenium IDE to trace my UI activity. I got this following code from IDE and i inspected in UI also,but while using find_element by id i'm getting css selector error.
driver.find_element(By.ID, "button-1034-btnIconEl").click()
error is
raise exception_class(message, screen, stacktrace)
NoSuchElementException: no such element: Unable to locate element:
{"method":"css selector","selector":"[id="button-1034-btnIconEl"]"}
(Session info: chrome=78.0.3904.108)
Please help me to debug this..
To click() on the button with text as Add Order you have to induce WebDriverWait for the element_to_be_clickable() you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[id^='button-'] > span.x-btn-wrap > span.x-btn-button > span.x-btn-inner.x-btn-inner-center"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[starts-with(#id, 'button-')]/span[#class='x-btn-wrap']/span[#class='x-btn-button']/span[#class='x-btn-inner x-btn-inner-center' and contains(., 'Add')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a detailed discussion on NoSuchElementException: no such element in Selenium “selenium.common.exceptions.NoSuchElementException” when using Chrome
The id seems to be dynamic one so you cannot use static id in the selector. You need to use a dynamic xpath for this.
You can use the below xpath:
driver.find_element(By.XPATH, "//span[contains(#id,'btnIconEl')]").click()
OR
You can find the element using its text as well in the xpath:
driver.find_element(By.XPATH, "//span[contains(text(),'Add Order')]").click()
Might not be your case, but I made a silly mistake that lead me to this error.
I forgot to get the actual web page using the line:
driver.get(my_link)
and thus, I got the same error