I am trying to pull a table from wikipedia. When I try and pull it using the following driver.find_element_by_class_name(name) it will not work. However when going to the html source code I can explicitly see the class name that I am looking for.
I do realize there are other ways to pull this table and I have moved on to easier ways. I am curious as to why Selenium does not find the class when it is in the HTML.
from selenium import webdriver
driver = webdriver.Chrome(r"\chromedriver_win32\chromedriver.exe")
driver.get(r'https://en.wikipedia.org/wiki/List_of_airports_in_the_United_States')
driver.implicitly_wait(2)
driver.find_element_by_class_name(name='wikitable sortable jquery-tablesorter')
However, the error I get is
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".wikitable sortable jquery-tablesorter"}
(Session info: chrome=75.0.3770.142)
wikitable sortable jquery-tablesorter is 3 class names: wikitable, sortable, and jquery-tablesorter. .find_element_by_class_name() only takes a single parameter consisting of a single class name, e.g. .find_element_by_class_name("wikitable"). That may or may not find the element you want based on whether that class name uniquely locates the element that you want.
Another option would be to use a CSS selector so that you can use all three classes in a single locator, e.g.
.wikitable.sortable.jquery-tablesorter
where the . indicates a class name in CSS selector syntax. See the CSS selector references below for more info on CSS selectors and their syntax.
W3C Selectors Overview
Selenium Tips: CSS Selectors
Taming Advanced CSS Selectors
To handle dynamic element use WebdriverWait and visibility_of_element_located and following css selector.
WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,".wikitable.sortable.jquery-tablesorter")))
You need to import followings.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
If you want to print the value of table.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(r"\chromedriver_win32\chromedriver.exe")
driver.get(r'https://en.wikipedia.org/wiki/List_of_airports_in_the_United_States')
print(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,".wikitable.sortable.jquery-tablesorter"))).text)
Please use class name directly in function find_element_by_class_name(). So, instead of writing like:
driver.find_element_by_class_name(name='wikitable sortable jquery-tablesorter')
Please write like:
driver.find_element_by_class_name('wikitable sortable jquery-tablesorter')
Hope it helps :)
Related
This is how I get the website
from selenium import webdriver
url = '...'
driver = webdriver.Firefox()
driver.get(url)
Now I want to extract all elements with a certain classes into a list
<li class=foo foo-default cat bar/>
How would I get all the elements from the website with these classes?
There is something like
fruit = driver.find_element_by_css_selector("#fruits .tomatoes")
But when I do this (I tried without spaces between the selectors too)
elements = driver.find_element_by_css_selector(".foo .foo-default .cat .bar")
I get
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .foo .foo-default .cat .bar
Stacktrace:
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:183:5
NoSuchElementError#chrome://remote/content/shared/webdriver/Errors.jsm:395:5
element.find/</<#chrome://remote/content/marionette/element.js:300:16
These are the classes I copied from the DOM`s website though...
If this is just the HTML
<li class=foo foo-default cat bar/>
You can remove the space and put a . to make a CSS SELECTOR as a locator.
elements = driver.find_elements(By.CSS_SELECTOR, "li.foo.foo-default.cat.bar")
print(len(elements))
or my recommendation would be to use it with explicit waits:
elements_using_ec = WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "li.foo.foo-default.cat.bar")))
print(len(elements))
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Have you tried without spaces between class names?
fruit = driver.find_element_by_css_selector(".foo.foo-default.cat.bar")
There is an undocumented function
driver.find_elements_by_css_selector(".foo.foo-default.cat.bar")
^
This works.
When I use mechanize, selenium libraries to run a url ("www.maps.google.com" in this case),
Chrome opens with a note saying that "Chrome is being controlled by an automated testing software".
But when I try to inspect element, there is only one element: body. All other inputs and buttons are not showing.
I want to automate the process to find distance between two addresses, so how do I solve the above problem?
from selenium import webdriver
driver = webdriver.Chrome(executable_path='C:/chromedriver.exe')
# Go to your page url
driver.get('https://www.google.com/maps')
# Get button you are going to click by its id ( also you could us find_element_by_css_selector to get element by css selector)
button_element = driver.find_element_by_id('searchbox-directions')
button_element.click()
dest_add = driver.find_element_by_class_name("tactile-searchbox-input")
dest_add.send_keys("Agra")
start_add = driver.find_element_by_class_name("tactile-searchbox-input")
start_add.send_keys("Jaipur")
For example this doesn't work. Since, there are no input fields so naturally, no elements with the class name "tactile-searchbox-input".
Induce WebDriverWait() and wait for element_to_be_clickable() and following css selector
driver.get('https://www.google.com/maps')
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.ID,"searchbox-directions"))).click()
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"input[aria-label*='starting point']"))).send_keys("Agra")
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"input[aria-label*='destination']"))).send_keys("Jaipur")
You need to import below libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
To click on search button try this.
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"(//button[#aria-label='Search'])[last()]"))).click()
I'm working on Python, Selenium and Chrome driver. I have a dynamic website and every time site loads the IDs get changed, Hence I can't use Xpath. But I can use the class name. I can get to the class as below;
Following is the code of the site.
<a class="appButton registerItemSearch-tabs-criteriaAndButtons-buttonPad-search appSearchButton appPrimaryButton appButtonPrimary appSubmitButton appNotReadOnly appIndex2" id="nodeW830" href="#" onclick="return function(me){var retVal = false; if (catHtmlFragmentCallback('W830','buttonPush',null,{asyncUpdate:true,containerSelector:'#AsyncWrapperW814',containerNodeId:'W814',success:function(html){jQuery('#AsyncWrapperW814').empty().append(html);webuiAsyncOk('#AsyncWrapperW814');}}, me) == 'skip') retVal = true;return retVal;}(this)" tabindex="118"><span class="left"></span><span class="appReceiveFocus" tabindex="-1">Search</span><span class="right"></span></a>
However I can do the following; to get to the element.
xxx = driver.find_elements_by_class_name("appButton.registerItemSearch-tabs-criteriaAndButtons-buttonPad-search.appSearchButton.appButtonPrimary.appPrimaryButton.appSubmitButton.appNotReadOnly.appIndex2")
Now I want to execute onclick method associated with the class. I can do that as well like below;
driver.execute_script("return function(me){var retVal = false; if (catHtmlFragmentCallback('W830','buttonPush',null,{asyncUpdate:true,containerSelector:'#AsyncWrapperW814',containerNodeId:'W814',success:function(html){jQuery('#AsyncWrapperW814').empty().append(html);webuiAsyncOk('#AsyncWrapperW814');}}, me) == 'skip') retVal = true;return retVal;}(this)")
But as I mentioned, I can't hardcode the driver.execute_script values. I should get the values dynamically. Could you please help me?
If you can suggest a way, that I can find the element by class name and execute onclick method. or as I mentioned;
I can find the element by class name, if I can get the code (HTML)associated with the element then I can do driver.execute_script.
Can you please suggest a way forward.
Thank you
Induce WebDriverWait and element_to_be_clickable() and following locator.
Xpath:
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//a[.//span[text()='Search']]"))).click()
CSS selector:
WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"a.appButton.registerItemSearch-tabs-criteriaAndButtons-buttonPad-search.appSearchButton.appPrimaryButton.appButtonPrimary.appSubmitButton.appNotReadOnly.appIndex2"))).click()
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
The script is failing to found element if the class contains more values in the class.
For example this class:
<a class="a-link-normal s-access-detail-page s-color-twister-title-link a-text-normal">
I want to find this element only by using class -- s-access-detail-page.
By looking for an element like this, I'm getting an error that element is not found:
find_element_by_css_selector("a[class*='s-access-detail-page']")
Same thing if I'm looking for an element with a class that contains:
a-link-normal a-text-normal
class on the page:
Parsing URL is Amazon: https://www.amazon.com/s?k=smart+watches&page=1
need to get product URLs.
You can use just the following CSS Selector:
.s-access-detail-page
Hope it helps you!
Try either of this.This should work.
find_element_by_css_selector("a.a-link-normal")
OR
find_element_by_css_selector(".a-link-normal")
OR
find_element_by_css_selector("a.s-access-detail-page")
OR
find_element_by_css_selector("a.s-color-twister-title-link")
Ensure you have a wait in and you can use just a simple class selector
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = 'https://www.amazon.com/s?k=smart+watches&page=1'
d = webdriver.Chrome()
d.get(url)
links = WebDriverWait(d,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".s-access-detail-page")))
linkUrls = [link.get_attribute('href') for link in links]
print(linkUrls)
The following code, which extracts elements using css selector, works in the ipython3 terminal, but doesn't find the elements when run as script:
from selenium import webdriver
driver = webdriver.Chrome()
url = scrape_url + "&keywords=" + keyword
driver.get(url)
driver.find_elements_by_css_selector(".search-result.search-result__occluded-item.ember-view")
The complex class of the element:
"search-result search-result__occluded-item ember-view"
The following xpath worked in the terminal, but not as a script:
driver.find_elements_by_xpath("//li[contains(#class, 'search-result search-result__occluded-item')]")
This might be a timing issue: required element could be generated dynamically, so you need to wait some time until it appears in DOM:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
driver = webdriver.Chrome()
url = scrape_url + "&keywords=" + keyword
driver.get(url)
wait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//li[contains(#class, 'search-result search-result__occluded-item')]")))
Also some class names could be assigned dynamically. That's why using compound name as "search-result search-result__occluded-item ember-view" might not work without ExplicitWait
If you can't find any elements with selenium css selector, then can you always try to use xpath instead of the css selector.
More information about that can be found here.
Pass only partial class name like,
driver.find_elements_by_css_selector(".search-result__occluded-item")