Develop Reference

Can't find page elements using Selenium python

I am trying to extraxt the review text from this page.
Here's a condensed version of the html shown in my chrome browser inspector:
<div id="module_product_review" class="pdp-block module">
<div class="lazyload-wrapper ">
<div class="pdp-mod-review" data-spm="ratings_reviews" lazada_pdp_review="expose" itemid="1615006548" data-nosnippet="true" data-aplus-ae="x1_490e4591" data-spm-anchor-id="a2o42.pdp_revamp.0.ratings_reviews.508466b1OJjCoH">
<div>...</div>
<div>...</div>
<div>
<div class="mod-reviews">
<div class="item">
<div class="top">...</div>
<div class="middle">...</div>
<div class="item-content">
<div class="content" data-spm-anchor-id="a2o42.pdp_revamp.ratings_reviews.i3.508466b1OJjCoH">Slim and light. feel good. better if providing 16G version.</div>
<div class="review-image">...></div>
<div class="skuInfo">Color Family:MYSTIC SILVER</div>
<div class="bottom">...</div>
<div class="dialogs"></div>
</div>
<div class="seller-reply-wrapper">...</div>
<div class="item">...</div>
<div class="item">...</div>
<div class="item">...</div>
<div class="item">...</div>
</div>
</div>
</div>
</div>
</div>
I'm trying to extract the "Slim and light. feel good. better if providing 16G version." text from the class="content" element.
But when I try to retrieve the id="module_product_review" element using Selenium in python, this is what I get instead:
<div class="pdp-block module" id="module_product_review">
<div class="lazyload-wrapper">
<div class="lazy-load-placeholder">
<div class="lazy-load-skeleton">
</div>
</div>
</div>
</div>
This is my code:
op = webdriver.ChromeOptions()
op.add_argument('--headless')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=op)
driver.get("https://www.lazada.sg/products/huawei-matebook-d14-laptop-14-fullview-display-intel-i5-processor-8gb512gb-intel-uhd-graphics-i1615006548-s7594078907.html?spm=a2o42.searchlist.list.3.15064828Od60kh&search=1&freeshipping=1")
module_product_review = driver.find_element(By.ID, "module_product_review")
html = module_product_review.get_attribute("outerHTML")
soup = BeautifulSoup(html, 'lxml')
print(soup.prettify())
I thought it might have been because I was retrieving the element before it was fully loaded, so I tried to sleep the program for 30 seconds before calling find_element(), but I still get the same result. As far as I can tell, it's not an issue of iframes or shadow roots either.
Is there some other issue that I'm missing?

The element you are trying to access and to get it's text is initially out of the visible view. You have first to scroll that element into the view.
Also, since you are working in headless mode you should set the window size. The default window size in headless mode is much smaller than we normally use.
And you should use expected conditions explicit waits to access the elements only when they are ready for that.
This should work better:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
op = webdriver.ChromeOptions()
op.add_argument('--headless')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=op)
options.add_argument("window-size=1920,1080")
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
driver.get("https://www.lazada.sg/products/huawei-matebook-d14-laptop-14-fullview-display-intel-i5-processor-8gb512gb-intel-uhd-graphics-i1615006548-s7594078907.html?spm=a2o42.searchlist.list.3.15064828Od60kh&search=1&freeshipping=1")
element = wait.until(EC.presence_of_element_located((By.ID, "module_product_review")))
time.sleep(1)
actions.move_to_element(element).perform()
module_product_review = wait.until(EC.visibility_of_element_located((By.ID, "module_product_review")))
#now you can do what you want here
html = module_product_review.get_attribute("outerHTML")
Also, in order to find that specific element and get that specific text you could use something more precise, like this:
your_text = wait.until(EC.visibility_of_element_located((By.XPATH, "(//div[#id='module_product_review']//div[#class='item']//div[#class='content'])[1]"))).text
You can use this after scrolling, as mentioned above

Cannot click on radio button in fieldset

I am using python with selenium to automate some process but I am having problems to click a radio button. This is the situation:
Code:
<div class="cart">
<form method="post" id="pmntFrm" name="chsPmntRt" action="/cart" class="form-validate">
<fieldset>
<div class="payment-field">
<input onchange="load('payment', 1)" type="radio" name="paymentmethod_id" id="payment_id_1" value="123">
<label for="payment_id_1" class="">
//foo
</label>
<input onchange="load('payment', 2)" type="radio" name="paymentmethod_id" id="payment_id_2" value="456">
<label for="payment_id_2" class="">
//foo
</label>
</div>
</fieldset>
</form>
</div>
I have tried click on it using:
driver.find_element_by_xpath(".//input[#type='radio' and #value='1']").click()
and I receive this error Message=Message: element not interactable
Also I read in other thread that maybe the problem is that the fieldset is in a different but is not.
I really apreciate if someone can help me with this topic.
Thank you and have a nice day!

To wait for the element to be clickable induce a webdriver wait and then click. Also don't use . it's for child elements not root driver.
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[#type='radio' and #value='1']"))).click()
Or the label
wait.until(EC.element_to_be_clickable((By.XPATH,"//label[#for='payment_id_1']"))).click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Unable to Locate Element for Selenium Webdriver Path in Python

I've been trying to reference the two inputs in the following HTML code using Selenium's get_element_by_...() with the purpose of automatically entering a username and password into the fields. I'm running into trouble with accessing the proper HTML elements.
<body>
<div id="app">
<div>
<section id="root" class="page-login">
<section class="main">
<section class="auth-page register-fill">
<div class="page-center">
<a class="switch-lang" href="/cn/login">cn</a>
<a class="btn-back" href="en/login">
<span class="isvg loaded">
<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100" viewBox="0 0 24 24">
<path class="a" d="M0,0H24V24H0Z"></path>
<path class="b" d="M20,11H7.83l5.59-5.59L12,4,4,12l8,8,1.41-1.41L7.83,13H20Z"></path>
</svg>
</span>
</a>
<div class="auth-page-wrapper">
<h1 class="auth-page-title">Welcome Back</h1>
<h2 class="auth-page-subtitle">Login to get started</h2>
<div class="auth-page-fields">
<div class="custom-input">
<input type="text" placeholder="Email">
<label>Email</label>
</div>
<div class="custom-input custom-input-password">
<a class="forgot-link" href="en/forgot-password">Forgot Password?</a>
<input type="password" placeholder="Minimum 6 characters">
<label>Password</label>
</div>
</div>
Note: The HTML may have some mistakes, I just quickly formatted it, but the mistakes should not affect the question.
Tried:
I've tried using all the different functions of the 'form find_element_by_...' and referencing a variety of things, but all tend to return "no such element: unable to locate element ..."
I've done absolute xpaths, relative xpaths, css selectors, ids, and combinations of these.
Some examples:
username = browser.find_element_by_xpath("//input[1]")
username = browser.find_element_by_xpath("//html/body/div/div/section/.../input")
The error begins at this point in the path:
username = browser.find_element_by_xpath("//html/body/div/div/section")
I'm likely making a stupid mistake, so sorry if this question isn't great, but other StackOverflow answers all have ids and such that are referable on the input field. (eg. <input id=username ...>)

To identify the Email field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[placeholder='Email']")))
Using XPATH:
username = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#placeholder='Email']")))
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

There are more than one way to do this. But, this should work browser.find_element_by_css_selector('[placeholder="Email"]')

Can find element within an iframe using Firefox but not Chrome

So I'm trying to write a test for a webpage which has some elements within an iframe. I've been able to successfully run the test using webdriver.Firefox() without any problems but if I switch it over to webdriver.Chrome() I get a timeout exception on the following lines of code:
self.driver.switch_to.frame(0)
self.activity_status = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#overview > div.details.w-66 > div > div.duration-and-status > span.status.stat_incomplete#')))
It'd be great to get a solution to this as I'm all out of ideas.
Thanks for your help.
edit, partial html for the page:
<iframe id="iframe_course_details" allowfullscreen="" src="../Course/Details.aspx?HidePageNav=true&IsInIframe=true"></iframe>
Close
Edit (Inactive)
Edit
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_favourite" class="favourite button tooltipstered" style="display: none;">Favourite</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_basket_dull" class="add-to-basket button delete tooltipstered" style="display: none;">Enrolled (Remove From Enrolments)</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_basket" class="add-to-basket button tooltipstered">Add to Enrolments</span>
<span id="ctl00_cph_main_content_area_ucCourseDetails_spn_print" class="print button tooltipstered">Print</span>
</div>
<section id="overview" style="opacity: 1;">
<div id="fullname" class="fullname w-100" style="display: none;">
</div>
<div class="image w-33" style="cursor: pointer;">
<div style="background-image:url(/App_Themes/MainTheme-responsive/Images/Course/webcast.jpg);"></div></div>
<div class="details w-66">
<div class="inner">
<h2>testing activity</h2>
<div class="star-rating-num-ratings">
<div class="star-rating">
<span></span><span></span><span></span><span></span><span></span>
</div>
<span class="num-of-ratings">0 Ratings</span>
</div>
<div class="duration-and-status">
<span class="duration">
<label>
Duration:
</label>
<span>0</span>
</span>
<span class="status stat_incomplete">Started</span>
</div>
Edit 2:
So we've managed to find a solution to this and its even more confusing than the original problem
WebDriverWait(self.driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, 'iframe_course_details')))
time.sleep(0)
self.activity_status = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//*[#id="overview"]/div[3]/div/div[2]/span[2]')))
I'd be really curious to hear some theories on why this works, it times out without the 'time.sleep(0).

If you reference the iframe directly rather then an integer that will work between Firefox/Chrome.
self.driver.switch_to.frame(driver.find_element_by_name("iframe"))
You can find the iframe element any way you wish e.g by css/xpath etc

As the the desired element is within an <iframe> so to invoke click() on the element you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Code Block:
# as per your comment assuming -> there is only one frame on the page
WebDriverWait(self.driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME,"iframe")))
self.element = self.activity_status = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#overview > div.details.w-66 > div > div.duration-and-status > span.status.stat_incomplete#')))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a relevant detailed discussion in:
Ways to deal with #document under iframe

Can't find element selenium python with xpath

I'm a relative newcomer to selenium so this might be something incredibly simple but I can't seem to access an element even though it appears on the page. I don't think it can be that it hasn't loaded yet because I can reference other elements. The line of code I am trying to use and the html is below.
max_questions = driver.find_element_by_xpath(xpath="//span[contains(#class, 'total-questions')]")
<div data-v-404a90e7="" data-v-084771db="" class="header animated fadeInDown anim-300-duration">
<div data-v-404a90e7="" class="left-section half-width">
<div data-v-404a90e7="" flow="right" class="menu-icon animated fadeIn anim-300-duration">
<div data-v-404a90e7="" class="menu-icon-image"></div>
</div>
<div data-v-404a90e7="" class="question-number-wrapper text-unselectable animated fadeIn anim-300-duration">
<span data-v-404a90e7="" class="current-question">1</span>
<span data-v-404a90e7="" class="total-questions">/10</span>
</div>
</div>
<div data-v-404a90e7="" class="right-section half-width">
<div data-v-404a90e7="" class="room-code animated fadeIn anim-300-duration">712851</div>
<div data-v-404a90e7="" flow="left" class="exit-game-btn-wrapper animated fadeIn anim-300-duration">
<div data-v-404a90e7="" class="exit-game-icon"></div>
</div>
</div>
</div>

You can use WebDriverWait with expected_conditions:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('d:\\chromedriver\\chromedriver.exe')
driver.get(url)
wait = WebDriverWait(driver, 10)
max_questions = wait.until(EC.element_to_be_clickable((By.XPATH, "//*[contains(#class, 'total-questions')]")))
print(max_questions.text)