Develop Reference

With Selenium python, do I need to refresh the page when waiting for hidden btn to appear and be clickable?

I'm trying to make a small program that looks at a web page at a hidden button (uses hide in the class) and waits for it to be clickable before clicking it. The code is below. I'm wondering if the WebDriverWait and element_to_be_clickable functions will already by refreshing things or if I would have to manually refresh the page.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.common.exceptions import WebDriverException
driver = webdriver.Firefox()
driver.get(<URL>)
print("beginning 120s wait")
time.sleep(120)
print("finished 120s wait")
try:
element = WebDriverWait(driver, 1000).until(
EC.element_to_be_clickable((By.CLASS_NAME, "btn add"))
)
print("It went through")
element.click()
driver.execute_script("alert('It went through!');")
finally:
driver.execute_script("alert('Did it work?');")
First of all, I am not really sure if just searching by the class name minus the "hide" part will actually find the correct element, but the larger issue is that I do not know if the button will only be visible after refreshing the page. If I need to refresh, then it gets annoying because most sites throw up additional captchas for both Firefox or Chrome when they figure out a bot is accessing the site. (That's why I have the initial sleep: so that I can finish any captcha manually first)
So, do I need to have a refresh in my code, or will it be fine without it? If I do need it, how do I implement it? Do I just add it like:
try:
element = WebDriverWait(driver, 1000).until(
drive.refresh()
EC.element_to_be_clickable((By.CLASS_NAME, "btn add"))
)
And sorry if this has been answered elsewhere, I searched a bunch, but I have not quite found the answer on this site.

First, you shouldn't use sleep the WebDriverWait with the correct EC will do the trick.
As for the EC.element_to_be_clickable this is the code behind the function:
def element_to_be_clickable(locator):
""" An Expectation for checking an element is visible and enabled such that
you can click it."""
def _predicate(driver):
element = visibility_of_element_located(locator)(driver)
if element and element.is_enabled():
return element
else:
return False
return _predicate
As you can see the EC.element_to_be_clickable function does not refresh the browser.
If you insist you need the refresh the correct way to implement it will be:
try:
element = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.CLASS_NAME, "btn add"))
except (NoSuchElementException, StaleElementReferenceException):
driver.refresh()
element = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.CLASS_NAME, "btn add"))
I don't think the refresh will help with the hidden element...

Selenium not blocking for element

I am facing inconsistencies in Selenium execution.
Last line in the code snippet I pasted below doesn't execute consistently. Sometimes it works, sometimes it throws an error saying that element is not found. Doesn't Selenium "block" for the element to appear before attempting to execute the click? I generated it using Selenium IDE. What I am missing here?
self.driver.find_element(By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)").click()
self.driver.switch_to.default_content()
self.driver.find_element(By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText").click()

Selenium may not find elements if they happen to be loaded dynamically by JS and if you search for them before they are loaded.
You can try either an implicit wait or an explicit wait.
In case of implicit waiting, the docs say:
An implicit wait tells WebDriver to poll the DOM for a certain amount of time when trying to find any element (or elements) not immediately available.
You could do with something like:
from selenium import webdriver
driver = webdriver.Chrome()
driver.implicitly_wait(10) #wait and poll for 10 seconds
Whereas the explicit waiting means to explicitly specify the element which is to be waited for it to be available. As per the docs:
An explicit wait is a code you define to wait for a certain condition to occur before proceeding further in the code.
You can do this with something like:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
element1 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)")))
element1.click()
self.driver.switch_to.default_content()
element2 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText")))
element2.click()

As you are using the line of code:
self.driver.switch_to.default_content()
Presumably you are switching Selenium's focus from a frame or iframe to the Top Level Content. Hence you need to induce WebDriverWait for the desired element to be clickable and you can use the following Locator Strategy:
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
Note:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant detailed discussions in:
How to send text to the Password field within https://mail.protonmail.com registration page?
How to switch between iframes using Selenium and Python?

Wait for the element to be loaded
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)"))).click()
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
The number is how long the driver should spend looking for the element before moving on.

python selenium can't interact with input element? [duplicate]

This question already has answers here:
ElementNotVisibleException: Message: element not interactable error while trying to click a button through Selenium and Python
(2 answers)
Closed 3 years ago.
I'm trying to put my name in an input field. It seems like a simple thing that selenium is built to do, but I cannot figure out what I'm doing wrong.
name = driver.find_element_by_xpath('//input[#id="signUpName16"]')
name.send_keys('Josh')
I know the driver works because I've been able to click other elements. I know the xpath is right because I copied it from chrome inspector. The error I get is
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
I've seen people say to try clicking or clearing elements so I've tried that too, but that still failed.
name = driver.find_element_by_xpath('//input[#id="signUpName16"]')
name.click()
name.send_keys('Josh')
yields this for the name.click() line
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable

There's a few different things that can be going wrong here. If the input is not fully loaded, then it will throw this exception if you try to send_keys before it is ready. We can invoke WebDriverWait on the input element to ensure it is fully loaded before sending keys to it:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
input = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[contains(#id, 'signUpName')]")))
input.send_keys("Josh")
If this still throws the exception, we can instead try to set the input value through Javascript:
input = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[contains(#id, 'signUpName')]")))
driver.execute_script("arguments[0].value = 'Josh';", input)
If neither of these solutions work, we may need to see some of the HTML on the page you are working with to see if there's any other issue happening here.

ElementNotInteractableException occurs when
Element is not displayed,
Element is out of screen ,
Some time element is hidden or
Behind to another element
Please refer below code to solve this issue:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as Wait
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path=r"C:\New folder\chromedriver.exe")
driver.set_page_load_timeout("10")
driver.get("your url")
actionChains = ActionChains(driver)
element = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.XPATH, "//input[#id='signUpName16']")))
actionChains.move_to_element(element).click().perform()
Solution 2:
element = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.XPATH, "//input[starts-with(#id,signUpName')]"))) # if your signUpName16 element is dynamic then use contains method to locate your element
actionChains.move_to_element(element).click().perform()

Trying to use selenium to automate signups. Running into a problem

Currently trying to automate signups on 'mail.com' using Selenium. So far i've managed to get the program to go to the URL. The problem i'm having is that even when I copied the full XPATH of "Sign Up" i'm getting an:
"selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/table/tbody/tr[114]/td[2]"}"
error
Here is the code i'm working with so far:
import selenium
import time
from selenium.webdriver.common.by import By
driver = selenium.webdriver.Chrome(executable_path='pathtochromedriver')
driver.get('https://www.mail.com/')
driver.maximize_window()
# Delay added to allow elements to load on webpage
time.sleep(30)
# Find the signup element
sign_up = driver.find_element_by_xpath('/html/body/table/tbody/tr[114]/td[2]')

Try using ActionsChains to scroll to ensure the element is in view.
from selenium.webdriver.common.action_chains import ActionChains
some_page_item = driver.find_element_by_class_name('some_class')
ActionsChains(driver).move_to_element(some_page_item).click(some_page_item).perform()
Also another tip... instead of simply using time.sleep() to wait for an element to appear, instead use WebDriverWait
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait_for_item = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CLASS_NAME ,"some_class_name")))
30 is the amount of seconds that it will wait until the item appears; however if it appears before 30 seconds then it will immediately continue execution. If 30 seconds passes and the item doesn't appear a timeout error will occur.

Can't get rid of hardcoded delay even when Explicit Wait is already there

I've written some code in python in combination with selenium to parse the different questions from quora.com. My scraper is doing it's job at this moment. The thing is I've used here hardcoded delay for the scraper to work, even when Explicit Wait has already been defined. As the page is an infinite scrolling one, i tried to make the scrolling process to a limited number. Now, I have got two questions:
Why wait.until(EC.staleness_of(page)) is not working within my scraper. It is commented out now.
If i use something else instead of page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link"))) the scraper throws an error: can't focus element.
Btw, I do not wish to go for page = driver.find_element_by_tag_name('body') this option.
Here is what I've written so far:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.quora.com/topic/C-programming-language")
wait = WebDriverWait(driver, 10)
page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link")))
for scroll in range(10):
page.send_keys(Keys.PAGE_DOWN)
time.sleep(2)
# wait.until(EC.staleness_of(page))
for item in wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "rendered_qtext"))):
print(item.text)
driver.quit()

You can try below code to get as much XHR as possible and then parse the page:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome()
driver.get("https://www.quora.com/topic/C-programming-language")
wait = WebDriverWait(driver, 10)
page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link")))
links_counter = len(wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "question_link"))))
while True:
page.send_keys(Keys.END)
try:
wait.until(lambda driver: len(driver.find_elements_by_class_name("question_link")) > links_counter)
links_counter = len(driver.find_elements_by_class_name("question_link"))
except TimeoutException:
break
for item in wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "rendered_qtext"))):
print(item.text)
driver.quit()
Here we scroll page down and wait up to 10 seconds for more links to be loaded or break the while loop if the number of links remains the same
As for your questions:
wait.until(EC.staleness_of(page)) is not working because when you scroll page down you don't get the new DOM - you just make XHR which adds more links into existed DOM, so the first link (page) will not be stale in this case
(I'm not quite confident about this, but...) I guess you can send keys only to nodes that can be focused (user can set focus manually), e.g. links, input fields, textareas, buttons..., but not content division (div), paragraphs (p), etc