I am going to scrape the comments of facebook video using selenium using xpath but it gives exception of no element found.
In this picture there is div with text "Comments" inside this <div> there is second <div> with <ul> and inside which there is <li> lists I want to fetch the text of those lists.
Codetrials:
from selenium.webdriver.common.by import By
comments = driver.find_element(by=By.XPATH, value='//div/h2[text()="Comments"]/div[2]/ul/li')
Snapshot:
As per the snapshot:
the ancestor <div> of the <ul> isn't a descendant of h2[text()="Comments"]. Hence you see exception of no element found
Solution
To print the comments you have to induce WebDriverWait for visibility_of_all_elements_located() and you can usethe following locator strategy:
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div/h2[contains(., 'Comments')]//following-sibling::div[2]/ul//li")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Related
I am trying to use selenium click to open dropdown menu and then click again to download a file. Common error I get is that the "element is not attached to the page document" Have tried different codes but here's one:
#button = driver.find_element_by_xpath('//ul[#class="dropdown-menu-right dropdown-menu"]')
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, "options-dropdown")))
element.click()
HTML snapshot:
Generally <ul> nodes along with it's decendent <li> nodes gets expanded clicking on <span> nodes.
Solution
To click and open the drop-down-menu i.e. to expand the <li> nodes you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.dropdown-toggle#options-dropdown"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#class='dropdown-toggle' and #id='options-dropdown']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am trying to get the following count of an Instagram account. I believe that the XPATH is correct an exists. Here's a screenshot showing it exists when I search for it:
This is my code:
wait = WebDriverWait(driver, 30)
followers = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body/div[1]/div/div/div/div[1]/div/div/div/div[1]/div[1]/section/main/div/ul/li[2]/button/div/span")))
print(followers.get_attribute("title"))
I have even looked at similar projects that find the following count and our code is almost exactly the same.
The desired element is a dynamic element, so to locate the element instead of presence_of_element_located() you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategy:
Using XPATH:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[contains(., 'followers')]//span[#class and #title and text()]"))).get_attribute("title"))
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Considering the HTML:
I want to select the paragraphs to the left with Selenium. Tried my hand at class_name and id but got NoSuchElementException. Why am I getting this error? I mean the elements are clearly there then why isn't Selenium recognizing those?
Methods I tried:
element = driver.find_element_by_xpath("//div[#id = 'mar-2019']//div[#class='report_data']").text
element = driver.find_element_by_id("mar-2019").text
element = driver.find_element_by_class_name("report_data").text
Where am I going wrong?
To handle dynamic element induce WebDriverWait() and wait for visibility_of_element_located()
element=WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.ID,"mar-2019"))).text
OR
element=WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"div#mar-2019"))).text
You need to import following libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
There are multiple child <p> elements within multiple parent/ancestor <div> elements. To extract the contents of the <p> elements within the parent <div id="mar-2019"> element you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".data-row")))])
Using XPATH and text attribute:
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[#id='mar-2019']//div[#class='report_data']//p")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I'm developing test automation and I want to click on element located in toolbar,I'm using python and selenium. Here is a code of website
<dl class="ToolbarDropdown_menu_2wY9X ToolbarDropdown_isHidden_3UaGr" tabindex="0" style=""
xpath="1"><dt><span class="ToolbarDropdown_title_1NBVn">New</span>
<span class="DropdownArrow_arrowIcon_dDzph DropdownArrow_down_3dlvo"></span>
</dt>
<dd class="ToolbarDropdown_item_nP-_M" style="">Dataset</dd>
<dd class="ToolbarDropdown_item_nP-_M">Project</dd>
</dl>
I need to click on element with text Dataset.
I locate element in this way
DATASET_BUTTON = (By.XPATH, '//dd[contains(text(),"Dataset")]')
And I want to perform action like that
self.driver.find_element(*VideosPageLocator.DATASET_BUTTON).click()
And there is no action but in terminal looks like my step has passed. Do you have any idea how to click on element in dropdown?
Maybe the action is done before DOM has loaded completely.
Try using WebDriverWait.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
web_driver_wait = WebDriverWait(self.driver, 10)
element = web_driver_wait.until(EC.element_to_be_clickable((By.XPATH,'//dd[contains(text(),"Dataset")]')))
element.click()
To invoke click() on the element with text as Dataset you have to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//dl[starts-with(#class, 'ToolbarDropdown')]//dd[starts-with(#class, 'ToolbarDropdown_item') and text()='Dataset']"))).click()
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
the html I'm digging (there is no iframe on it) is:
<div id="app" class="container-fluid">
I try to get the block of the DOM tree corresponding to the above div
this is my code:
from selenium.common.exceptions import NoSuchElementException
try:
app = driver.find_element_by_xpath('//div[#id="app"]')
print(4444444444444444, app)
is_vue_app_loaded = len(app.text) != 0
except NoSuchElementException as e:
print(f'e: {e}')
I get this error:
4444444444444444 <selenium.webdriver.firefox.webelement.FirefoxWebElement
(session="c8da2a89-63fd-4dce-bdb5-28fdf0e67b01",
element="b9be2828-115e-4965-8ebc-b75c09236193")>
e: Message: Unable to locate element: //div[#id='app']
Is there a way to ask the driver to return me the raw html so I can check that the DOM try I think I have is the one I have actualy ?
Check if the element is present inside any iframe.If there you need to switch it first.
If Not then try.
Induce WebDriverWait() and visibility_of_element_located()
WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.XPATH,"//div[#id='app' and #class='container-fluid']")))
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
If your usecase is to return me the raw html of the element you have to induce WebDriverWait for the visibility_of_element_located() inconjunction with get_attribute("outerHTML") and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.container-fluid#app"))).get_attribute("outerHTML"))
Using XPATH:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#class='container-fluid' and #id='app']"))).get_attribute("outerHTML"))
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC