Develop Reference

How to refresh a page until an element is available using python (Selenium)?

I want to design a code where it tends to refresh the page until the particular element is visible in on the webpage using selenium. I have designed the following code but it gave me an error.
Error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"partial link text","selector":"BUY NOW"}
Code:
while True:
if driver.find_element_by_partial_link_text('BUY NOW'):
break
driver.refresh()
buy = driver.find_element_by_partial_link_text('BUY NOW')
buy.click()
Can anyone help?

selenium raises exceptions if it failed to find single element.
so your loop (while True) cannot be a successful loop, the code "if driver.find_element_by_partial_link_text('BUY NOW')" may break the script.
there are two ways to do it:
use 'WebDriverWait' to wait the element explicitly. (suggest to use WebDriverWait)
example:
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 30)
wait.until(EC.presence_of_element_located(("partial link text","BUY NOW")))
driver.refresh()
use find_elements instead.
example:
from selenium.webdriver.common.by import By
while True:
if len(driver.find_elements(By.PARTIAL_LINK_TEXT, 'BUY NOW')) > 0:
break
driver.refresh()
buy = driver.find_element_by_partial_link_text('BUY NOW')
buy.click()

You should give some time for the element to show up. Try using waits:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
while True:
wait = WebDriverWait(driver, 1)
wait.until(EC.element_to_be_clickable(driver.find_element_by_partial_link_text('BUY NOW'))
if driver.find_element_by_partial_link_text('BUY NOW'):
break
driver.refresh()

Selenium not blocking for element

I am facing inconsistencies in Selenium execution.
Last line in the code snippet I pasted below doesn't execute consistently. Sometimes it works, sometimes it throws an error saying that element is not found. Doesn't Selenium "block" for the element to appear before attempting to execute the click? I generated it using Selenium IDE. What I am missing here?
self.driver.find_element(By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)").click()
self.driver.switch_to.default_content()
self.driver.find_element(By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText").click()

Selenium may not find elements if they happen to be loaded dynamically by JS and if you search for them before they are loaded.
You can try either an implicit wait or an explicit wait.
In case of implicit waiting, the docs say:
An implicit wait tells WebDriver to poll the DOM for a certain amount of time when trying to find any element (or elements) not immediately available.
You could do with something like:
from selenium import webdriver
driver = webdriver.Chrome()
driver.implicitly_wait(10) #wait and poll for 10 seconds
Whereas the explicit waiting means to explicitly specify the element which is to be waited for it to be available. As per the docs:
An explicit wait is a code you define to wait for a certain condition to occur before proceeding further in the code.
You can do this with something like:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
element1 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)")))
element1.click()
self.driver.switch_to.default_content()
element2 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText")))
element2.click()

As you are using the line of code:
self.driver.switch_to.default_content()
Presumably you are switching Selenium's focus from a frame or iframe to the Top Level Content. Hence you need to induce WebDriverWait for the desired element to be clickable and you can use the following Locator Strategy:
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
Note:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant detailed discussions in:
How to send text to the Password field within https://mail.protonmail.com registration page?
How to switch between iframes using Selenium and Python?

Wait for the element to be loaded
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)"))).click()
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
The number is how long the driver should spend looking for the element before moving on.

Is there a way to bypass the 500ms delay on explicit waits with selenium and python

For explicit waits in the selenium python docs it says:
By default, WebDriverWait calls the ExpectedCondition every 500 milliseconds until it returns success.
I notice the by default at the start of the sentance. Is there a way to bypass this 500ms delay? Can i set it to something like 100ms? Is this possible?
Thanks for your help.

Please have a look at the official WebDriverWait function
You may simply set the poll_frequency to another value rather than 0.5, so for your case you may use poll_frequency=0.1.
This code will be an example with your wished frequency (taken from documentation)
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10, poll_frequency=0.1).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()

What is the best way to check URL change with Selenium in Python?

So, what's I want to do is to run a function on a specific webpage (which is a match with my regex).
Right now I'm checking it every second and it works, but I'm sure that there is a better way (as it's flooding that website with getting requests).
while flag:
time.sleep(1)
print(driver.current_url)
if driver.current_url == "mydesiredURL_by_Regex":
time.sleep(1)
myfunction()
I was thinking to do that somehow with WebDriverWait but not really sure how.

This is how I implemented it eventually. Works well for me:
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 5)
desired_url = "https://yourpageaddress"
def wait_for_correct_current_url(desired_url):
wait.until(
lambda driver: driver.current_url == desired_url)

I was thinking to do that somehow with WebDriverWait
Exactly. First of all, see if the built-in Expected Conditions may solve that:
title_is
title_contains
Sample usage:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
wait.until(EC.title_is("title"))
wait.until(EC.title_contains("part of title"))
If not, you can always create a custom Expected Condition to wait for url to match a desired regular expression.

To really know that the URL has changed, you need to know the old one. Using WebDriverWait the implementation in Java would be something like:
wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.not(ExpectedConditions.urlToBe(oldUrl)));
I know the question is for Python, but it's probably easy to translate.

Here is an example using WebdriverWait with expected_conditions:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://example.com/before'
changed_url = 'https://example.com/after'
driver = webdriver.Chrome()
driver.get(url)
# wait up to 10 secs for the url to change or else `TimeOutException` is raised.
WebDriverWait(driver, 10).until(EC.url_changes(changed_url))

Use url_matches Link to match a regex pattern with a url. It does re.search(pattern, url)
from selenium import webdriver
import re
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
pattern='https://www.example.com/'
driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)
wait.until(EC.url_matches(pattern))

Wait until loader disappears python selenium

<div id="loader-mid" style="position: absolute; top: 118.5px; left: 554px; display: none;">
<div class="a">Loading</div>
<div class="b">please wait...</div>
</div>
And want to wait until it disappears. I have following code but it wait sometimes too long and at some point of code it suddenly freeze all process and I don't know why.
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
self.wait = WebDriverWait(driver, 10)
self.wait.until(EC.invisibility_of_element_located((By.XPATH, "//*[#id='loader_mid'][contains(#style, 'display: block')]")))
and also I tried this one:
self.wait.until_not(EC.presence_of_element_located((By.XPATH, "//*[#id='loader_mid'][contains(#style, 'display: block')]")))
I don't know exactly how to check but maybe my element is always present on the page and selenium thought that it is there, the only thing that changes is parameter display changes from none to block. I think I can get attribute like string and check if there is word "block" but it is so wrong I thing... Help me please.

Reiterated your answer (with some error handling) to make it easier for people to find the solution :)
Importing required classes:
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
Configuration variables:
SHORT_TIMEOUT = 5 # give enough time for the loading element to appear
LONG_TIMEOUT = 30 # give enough time for loading to finish
LOADING_ELEMENT_XPATH = '//*[#id="xPath"]/xPath/To/The/Loading/Element'
Code solution:
try:
# wait for loading element to appear
# - required to prevent prematurely checking if element
# has disappeared, before it has had a chance to appear
WebDriverWait(driver, SHORT_TIMEOUT
).until(EC.presence_of_element_located((By.XPATH, LOADING_ELEMENT_XPATH)))
# then wait for the element to disappear
WebDriverWait(driver, LONG_TIMEOUT
).until_not(EC.presence_of_element_located((By.XPATH, LOADING_ELEMENT_XPATH)))
except TimeoutException:
# if timeout exception was raised - it may be safe to
# assume loading has finished, however this may not
# always be the case, use with caution, otherwise handle
# appropriately.
pass

Use expected condition : invisibility_of_element_located
This works fine for me.
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
WebDriverWait(driver, timeout).until(EC.invisibility_of_element_located((By.ID, "loader-mid")))

The following code creates an infinite loop until the element disappears:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
while True:
try:
WebDriverWait(driver, 1).until(EC.presence_of_element_located((By.XPATH, 'your_xpath')))
except TimeoutException:
break

Ok, here is how i solved this issue for my project,
imports
from selenium.common.exceptions import TimeoutException
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait as wait
Now the code part
browser = webdriver.Chrome(service=Service("./chromedriver.exe"))
browser.get("https://dashboard.hcaptcha.com/signup?type=accessibility")
try:
d = wait(browser, 10).until(EC.invisibility_of_element_located((By.ID, 'loader-mid')))
if d: # just a check you can ignore it.
print("yes")
sleep(3)
else:
print("F")
except TimeoutException:
print("timeout error occurred.")
pass
browser.quit()
Or you can use Implicit wait
driver.implicitly_wait(10) # seconds
# do something after...