Develop Reference

Can't find page elements using Selenium python

I am trying to extraxt the review text from this page.
Here's a condensed version of the html shown in my chrome browser inspector:
<div id="module_product_review" class="pdp-block module">
<div class="lazyload-wrapper ">
<div class="pdp-mod-review" data-spm="ratings_reviews" lazada_pdp_review="expose" itemid="1615006548" data-nosnippet="true" data-aplus-ae="x1_490e4591" data-spm-anchor-id="a2o42.pdp_revamp.0.ratings_reviews.508466b1OJjCoH">
<div>...</div>
<div>...</div>
<div>
<div class="mod-reviews">
<div class="item">
<div class="top">...</div>
<div class="middle">...</div>
<div class="item-content">
<div class="content" data-spm-anchor-id="a2o42.pdp_revamp.ratings_reviews.i3.508466b1OJjCoH">Slim and light. feel good. better if providing 16G version.</div>
<div class="review-image">...></div>
<div class="skuInfo">Color Family:MYSTIC SILVER</div>
<div class="bottom">...</div>
<div class="dialogs"></div>
</div>
<div class="seller-reply-wrapper">...</div>
<div class="item">...</div>
<div class="item">...</div>
<div class="item">...</div>
<div class="item">...</div>
</div>
</div>
</div>
</div>
</div>
I'm trying to extract the "Slim and light. feel good. better if providing 16G version." text from the class="content" element.
But when I try to retrieve the id="module_product_review" element using Selenium in python, this is what I get instead:
<div class="pdp-block module" id="module_product_review">
<div class="lazyload-wrapper">
<div class="lazy-load-placeholder">
<div class="lazy-load-skeleton">
</div>
</div>
</div>
</div>
This is my code:
op = webdriver.ChromeOptions()
op.add_argument('--headless')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=op)
driver.get("https://www.lazada.sg/products/huawei-matebook-d14-laptop-14-fullview-display-intel-i5-processor-8gb512gb-intel-uhd-graphics-i1615006548-s7594078907.html?spm=a2o42.searchlist.list.3.15064828Od60kh&search=1&freeshipping=1")
module_product_review = driver.find_element(By.ID, "module_product_review")
html = module_product_review.get_attribute("outerHTML")
soup = BeautifulSoup(html, 'lxml')
print(soup.prettify())
I thought it might have been because I was retrieving the element before it was fully loaded, so I tried to sleep the program for 30 seconds before calling find_element(), but I still get the same result. As far as I can tell, it's not an issue of iframes or shadow roots either.
Is there some other issue that I'm missing?

The element you are trying to access and to get it's text is initially out of the visible view. You have first to scroll that element into the view.
Also, since you are working in headless mode you should set the window size. The default window size in headless mode is much smaller than we normally use.
And you should use expected conditions explicit waits to access the elements only when they are ready for that.
This should work better:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
op = webdriver.ChromeOptions()
op.add_argument('--headless')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=op)
options.add_argument("window-size=1920,1080")
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
driver.get("https://www.lazada.sg/products/huawei-matebook-d14-laptop-14-fullview-display-intel-i5-processor-8gb512gb-intel-uhd-graphics-i1615006548-s7594078907.html?spm=a2o42.searchlist.list.3.15064828Od60kh&search=1&freeshipping=1")
element = wait.until(EC.presence_of_element_located((By.ID, "module_product_review")))
time.sleep(1)
actions.move_to_element(element).perform()
module_product_review = wait.until(EC.visibility_of_element_located((By.ID, "module_product_review")))
#now you can do what you want here
html = module_product_review.get_attribute("outerHTML")
Also, in order to find that specific element and get that specific text you could use something more precise, like this:
your_text = wait.until(EC.visibility_of_element_located((By.XPATH, "(//div[#id='module_product_review']//div[#class='item']//div[#class='content'])[1]"))).text
You can use this after scrolling, as mentioned above

Unable to Locate Element for Selenium Webdriver Path in Python

I've been trying to reference the two inputs in the following HTML code using Selenium's get_element_by_...() with the purpose of automatically entering a username and password into the fields. I'm running into trouble with accessing the proper HTML elements.
<body>
<div id="app">
<div>
<section id="root" class="page-login">
<section class="main">
<section class="auth-page register-fill">
<div class="page-center">
<a class="switch-lang" href="/cn/login">cn</a>
<a class="btn-back" href="en/login">
<span class="isvg loaded">
<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100" viewBox="0 0 24 24">
<path class="a" d="M0,0H24V24H0Z"></path>
<path class="b" d="M20,11H7.83l5.59-5.59L12,4,4,12l8,8,1.41-1.41L7.83,13H20Z"></path>
</svg>
</span>
</a>
<div class="auth-page-wrapper">
<h1 class="auth-page-title">Welcome Back</h1>
<h2 class="auth-page-subtitle">Login to get started</h2>
<div class="auth-page-fields">
<div class="custom-input">
<input type="text" placeholder="Email">
<label>Email</label>
</div>
<div class="custom-input custom-input-password">
<a class="forgot-link" href="en/forgot-password">Forgot Password?</a>
<input type="password" placeholder="Minimum 6 characters">
<label>Password</label>
</div>
</div>
Note: The HTML may have some mistakes, I just quickly formatted it, but the mistakes should not affect the question.
Tried:
I've tried using all the different functions of the 'form find_element_by_...' and referencing a variety of things, but all tend to return "no such element: unable to locate element ..."
I've done absolute xpaths, relative xpaths, css selectors, ids, and combinations of these.
Some examples:
username = browser.find_element_by_xpath("//input[1]")
username = browser.find_element_by_xpath("//html/body/div/div/section/.../input")
The error begins at this point in the path:
username = browser.find_element_by_xpath("//html/body/div/div/section")
I'm likely making a stupid mistake, so sorry if this question isn't great, but other StackOverflow answers all have ids and such that are referable on the input field. (eg. <input id=username ...>)

To identify the Email field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[placeholder='Email']")))
Using XPATH:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#placeholder='Email']")))
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

There are more than one way to do this. But, this should work browser.find_element_by_css_selector('[placeholder="Email"]')

How to get javascript generated html that I see by clicking "inspect element" in browser?

I'm trying to get the hours of the available time slots from this webpage (the boxes below the calendar):
https://magicescape.it/le-stanze/lo-studio-di-harry-houdini/
I've read other related questions and wrote this code
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup
url = 'https://magicescape.it/le-stanze/lo-studio-di-harry-houdini/'
wait_time = 10
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.get(url)
driver.switch_to.frame(0)
wait = WebDriverWait(driver, wait_time)
first_result = wait.until(presence_of_element_located((By.ID, "sb_main")))
soup = BeautifulSoup(driver.page_source, 'html.parser')
print(soup)
driver.quit()
After switching to the iframe containing the time slots, I get this from printing soup
<script id="time_slots_view" type="text/html"><div class="slots-view{{#ifCond (getThemeOption 'timeline_modern_display') '==' 'as_table'}} as-table{{/ifCond}}">
<div class="timeline-wrapper">
<div class="tab-pd">
<div class="container-caption">
{{_t 'available_services_on_this_day'}}
</div>
{{#if error_message}}
<div class="alert alert-danger alert-dismissible" role="alert">
{{error_message}}
</div>
{{/if}}
{{>emptyTimePart is_empty=is_empty is_loaded=is_loaded}}
<div id="sb_time_slots_container"></div>
{{> bookingTimeLegendPart legend="only_available" time_diff=0}}
</div>
</div>
</div></script>
<script id="time_slot_view" type="text/html"><div class="slot">
<a class="sb-cell free {{#ifPluginActive 'slots_count'}}{{#if available_slots}}has-available-slot{{/if}}{{/ifPluginActive}}" href="#{{bookingStepUrl time=time date=date}}">
{{formatDateTime datetime 'time' time_diff}}
{{#ifCond (getThemeOption 'timeline_show_end_time') '==' 1}}
-<span class="end-time">
{{formatDateTime end_datetime 'time' time_diff}}
</span>
{{/ifCond}}
{{#ifPluginActive 'slots_count'}}
{{#if available_slots}}
<span class="slot--available-slot">
{{available_slots}}
{{#ifConfigParam 'slots_count_show_total' '==' true}} / {{total_slots}} {{/ifConfigParam}}
</span>
{{/if}}
{{/ifPluginActive}}
</a>
</div></script>
while from right click > inspect element in the webpage I get this
<div class="slots-view">
<div class="timeline-wrapper">
<div class="tab-pd">
<div class="container-caption">
Orari d'inizio disponibili
</div>
<div id="sb_time_slots_container">
<div class="slot">
<a class="sb-cell free " href="#book/location/4/service/6/count/1/provider/6/date/2020-03-09/time/23:00:00/">
23:00
</a>
</div>
</div>
<div class="time-legend">
<div class="available">
<div class="circle">
</div>
- Disponibile
</div>
</div>
</div>
</div>
</div>
How can I get the hour of the available slots (23:00 in this example) using selenium?

To get the desired response you need to:
Correctly identify the iframe you want to switch to (and switch to it). You were trying to switch to frame[0] but needed frame[1]. The following code removes reliance on indexes and uses xpath instead.
Get the elements containing the time. Again this uses xpath to identify all child div's of an element with id=sb_time_slots_container.
We then iterate over these child div's and get the text property, which is nested within an <a> of these div's.
For both steps 1 & 2 you should also use wait.until so that the content can be loaded.
...
driver.get(url)
wait = WebDriverWait(driver, wait_time)
# Wait until the iframe exists then switch to it
iframe_element = wait.until(presence_of_element_located((By.XPATH, '//*[#id="prenota"]//iframe')))
driver.switch_to.frame(iframe_element)
# Wait until the times exist then get an array of them
wait.until(presence_of_element_located((By.XPATH, '//*[#id="sb_time_slots_container"]/div')))
all_time_elems = driver.find_elements_by_xpath('//*[#id="sb_time_slots_container"]/div')
# Iterate over each element and print the time out
for elem in all_time_elems:
print(elem.find_element_by_tag_name("a").text)
driver.quit()

is_displayed() method returning true even if WebElement is not visible

I'm working on a test which, at some point, records some datas displayed in a scrollable table:
<div class='table-body'>
<div class='scroll-wrapper>
<div class='row'>
<button class='button' type='button'></button>
<div class='inner-data></div>
</div>
<div class='row'>
<button class='button' type='button'></button>
<div class='inner-data></div>
</div>
<div class='row'>
<button class='button' type='button'></button>
<div class='inner-data></div>
</div>
</div>
</div>
The total number of line present in the table is displayed on the screen, allowing me to use a while loop in order to be sure to collect all the datas.
However, as you can see on the html code, each row has a button i'm clicking for every row. And here is my issue : at some point, my method find_elements_by_css_selector(div.row) finds a WebElement which is not visible on the window, and tries to click on its <button>.
Consequently, i get the following error:
ElementNotInteractableException: Message: Element <button class="btn" type="button"> could not be scrolled into view
I tried using is_displayed() method and is_enabled() to check if the element is visible on the screen, unfortunately they are always returning True.
Do you guys have any solutions ?

This error message...
ElementNotInteractableException: Message: Element <button class="btn type="button"> could not be scrolled into view
...implies that the desired element could not be scrolled into view and is not interactable.
If you observe the relevant HTML the <button> tags are:
<button class='button' type='button'></button>
which are within the parent <div> tag as follows:
<div class='row'>
You have targetted the <div> tag with your Locator Strategy:
find_elements_by_css_selector(div.row)
But the error ElementNotInteractableException is returned for some other element:
<button class="btn type="button">
observe the ^^^class^^^ attribute value
Which is of coarse not your desired element. Hence you see the error.
Solution
As a solution to target the <button> tags you need to dig deeper one step further inducing induce WebDriverWait for the visibility_of_all_elements_located() and click each of the elements and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
for element in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "div.table-body > div.scroll-wrapper div.row > button.button[type='button']"))):
WebDriverWait(driver, 20).until(EC.visibility_of(element)).click()
Using XPATH:
for element in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[#class='table-body']/div[#class='scroll-wrapper']//div[#class='row']/button[#class='button' and #type='button']"))):
WebDriverWait(driver, 20).until(EC.visibility_of(element)).click()
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Thanks for your answer #DebanjanB. Your solution works well:
try:
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS-SELECTOR, "button"))):
element.click()
except TimeoutException:
# The button is not clickable, I scroll down
However, i know that the button is not clickable (and not visible) only when it's in the DOM but not visible on my window. And even if your solution works, it's quite time consuming, considering that i've a timeout after 20s. I know that I can decrease the time parameter in the WebDriverWait(driver, time) solution, but I found another solution:
try:
self.driver.find_element_by_css_selector('button').click()
except ElementNotInteractableException:
# The button is not clickable, I scroll down
Anyway, thanks for you help, and I hope this trick can help others ;)

Selenium with Python: First instance of the element is identified but the next occurance is ElementNotVisibleException

I have the following Selenium Test for Python/Django application:
class EmailRecordsTest(StaticLiveServerTestCase):
def test_can_store_email_and_retrieve_it_later(self):
self.browser.get(self.live_server_url)
emailbox = self.browser.find_element_by_xpath("//form[#class='pma-subscribe-form']/input[1]")
self.assertEqual(emailbox.get_attribute("placeholder"), 'Enter your Email')
print("tested until here")
print("The placeholder: ", emailbox.get_attribute("placeholder"))
print(emailbox)
emailbox.send_keys('vio#mesmerizing.com')
First occurance of emailbox is clearly identified as seen from the print runs and assert Equal for placeholder. The last instance of emailbox.send_keys throws following error:
selenium.common.exceptions.ElementNotVisibleException: Message:
Element is not currently visible and so may not be interacted with
Cannot find why the same element become Not Visible when using with send_keys.
The Html code being tested is as below:
<!-- Start footer -->
<footer id="pma-footer">
<!-- start footer top -->
<div class="pma-footer-top">
<div class="container">
<div class="pma-footer-top-area">
<div class="row">
<div class="col-lg-3 col-md-3 col-sm-3">
<div class="pma-footer-widget">
<h4>News letter</h4>
<p>Get latest update, news & offers</p>
<form class="pma-subscribe-form">
<input id="subscribe-email" type="email" placeholder="Enter your Email">
<button class="btn btn-danger btn-md" type="submit">Subscribe!</button>
</form>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- end footer top -->
Kindly help.

Actually find_element returns element which would be present on the DOM no matter it's visible or not and you can get attribute of this element as well but send_keys does an action on element and selenium does action only visible element, So you need to be sure before doing action on element that it's visible using WebDriverWait as below :-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
emailbox = wait.until(EC.visibility_of_element_located((By.ID, "subscribe-email")))
#do your all stuff before send keys
# now use send_keys
emailbox.send_keys('vio#mesmerizing.com')
Edited :- If you are still unable to interact with element try using execute_script() to set value as below :-
emailbox = wait.until(EC.presence_of_element_located((By.ID, "subscribe-email")))
#do your all stuff before send keys
# now use execute_script
driver.execute_script("arguments[0].value = 'vio#mesmerizing.com'", emailbox)

Another option which worked in this case is that you scroll to the specific element (which was at the bottom of the page)and then use send_keys it works.
emailbox = self.browser.find_element_by_xpath("//form[#class='mu-subscribe-form']/input[1]")
self.browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
emailbox.send_keys('vio#mesmerizing.com')