Develop Reference

How to wait for the disappearance of the Google Loader Icon to continue test execution using Selenium and Python

My problem:
I was try crawl Google People Also Ask with selenium and my code write with python, but I have problem when internet slowly. When I click more question, it will show loader icon of Google with this HTML:
<g-loading-icon jsname="aZ2wEe" class="nhGGkb S3PB2d" style="height: 24px; width: 24px; display: none;"><img height="24" src="//www.gstatic.com/ui/v1/activityindicator/loading_24.gif" width="24" alt="Đang tải..." role="progressbar" data-atf="0" data-frt="0"></g-loading-icon>
Note this: It will show when click more result Google People Also Ask and internet slowly. When load complete g-loading-icon will hiden.
I was test and I think Xpath will change any time with structure of Google result. So I want code to wait until it loading complete to crawl not fail. Because if not waiting it load complete the code will have error: IndexError: string index out of range.
I don't want use time.sleep because I think it not best way.
Case 1: I was try catch with Xpath. But Xpath will change when structure of result Google change.
This is my code for case 1:
def click_more_gpaa(order):
# Click button question
# Condition for check load question
short_timeout = 10 # give enough time for the loading element to appear
long_timeout = 30 # give enough time for loading to finish
loading_element_xpath = '/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/g-loading-icon'
loading_element_css_selector = 'nhGGkb.S3PB2d'
try:
# Case 1: Test with Xpath
# wait for loading element to appear
# - required to prevent prematurely checking if element
# has disappeared, before it has had a chance to appear
is_gppa_show = WebDriverWait(driver, short_timeout).until(
EC.presence_of_element_located((By.XPATH, loading_element_xpath))
)
# then wait for the element to disappear
is_gppa_show = WebDriverWait(driver, long_timeout).until_not(
EC.presence_of_element_located((By.XPATH, loading_element_xpath)))
except TimeoutException:
# if timeout exception was raised - it may be safe to
# assume loading has finished, however this may not
# always be the case, use with caution, otherwise handle
# appropriately.
pass
Case 2: I was try catch with CSS Selector. But it not work.
This is my code for case 2:
def click_more_gpaa(order):
# Click button question
# Condition for check load question
short_timeout = 10 # give enough time for the loading element to appear
long_timeout = 30 # give enough time for loading to finish
loading_element_xpath = '/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/g-loading-icon'
loading_element_css_selector = 'nhGGkb.S3PB2d'
try:
# Case 2: Test with CSS_SELECTOR
# wait for loading element to appear
# - required to prevent prematurely checking if element
# has disappeared, before it has had a chance to appear
is_gppa_show = WebDriverWait(driver, short_timeout).until(
EC.presence_of_element_located((By.CSS_SELECTOR, loading_element_xpath))
)
# then wait for the element to disappear
is_gppa_show = WebDriverWait(driver, long_timeout).until_not(
EC.presence_of_element_located((By.CSS_SELECTOR, loading_element_xpath)))
except TimeoutException:
# if timeout exception was raised - it may be safe to
# assume loading has finished, however this may not
# always be the case, use with caution, otherwise handle
# appropriately.
pass
Have any way to do that or solution for this?
Or any document and link to read about wait in selenium and python?
I was try research but a lot of document about wait in selenium and python, I was confused about that.
Thanks you so much!

To wait for the <g-loading-icon> to disappear you need to induce WebDriverWait for the invisibility_of_element_located() and you can use either of the following Locator Strategies:
Using TAG_NAME:
WebDriverWait(driver, 20).until(EC.invisibility_of_element_located((By.TAG_NAME, "g-loading-icon")))
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "g-loading-icon > img[role='progressbar']")))
Using XPATH:
WebDriverWait(driver, 20).until(EC.invisibility_of_element_located((By.XPATH, "//g-loading-icon/img[#role='progressbar']")))
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant detailed discussions in:
How to click on the anchor element when the spinner obscures it using Selenium and Python?
Selenium C# label text click
Selenium invisibilityOf(element) method throwing NoSuchElementException + WebDriverWait.ignoring(NoSuchElementException.class) is not working

Having trouble referencing to a certain element on page with Selenium

I am having a terribly hard time referencing to a certain "next page" button on a website that I am trying to scrape links from [https://www.sreality.cz/adresar?strana=2]. If you scroll down you can see a red right arrow button that you can click to go to the next page and so the website load new dynamic content. Every approach seems to report the same exact error and I don't know how am I supposed to point to the element without running into it.
This is the code that I currently have :
from selenium import webdriver
chromedriver_path = "/home/user/Dokumenty/iCloud/RealityScraper/chromedriver"
driver = webdriver.Chrome(chromedriver_path)
print("WebDriver Successfully Initialized")
driver.get("https://www.sreality.cz/adresar?strana=2")
links = driver.find_elements_by_css_selector("h2.title a")
nextPage = driver.find_element_by_css_selector("li.paging-item a.btn-paging-pn.icof.icon-arr-right.paging-next")
for link in links:
print(link.get_attribute("href"))
nextPage.click()
The "nextPage" variable is holding a supposed value to be clicked on once the "links" variable search finishes scraping all the links from the company titles. However when I run this code I get an error :
selenium.common.exceptions.StaleElementReferenceException: Message:
stale element reference: element is not attached to the page document
I have been searching for various fixes online but none of them seemed to resolve the issue. I think that the issue at this point is not caused by the element not loading quickly enough but rather Selenium having trouble finding the element because of wrong reference.
Because of this I have tried using XPath to accurately point to the actual element and so I changed the "nextPage" variable to :
nextPage = driver.find_element_by_xpath("""/html/body/div[2]/div[1]/div[2]/div[2]/div[4]/div/div/div/div[2]/div/div[2]/ul[1]/li[12]/a""")
Which returns exactly the same error as stated above. I have been trying to find a solution to this for hours now and I can't understand where the issue lies. I would be grateful if anyone could explain to me what am I doing wrong. Thanks to anyone.

If you want to get all the ng-href tags from every page. Or you could look into their api.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep
driver.get("https://www.sreality.cz/adresar?strana=2")
wait = WebDriverWait(driver, 10)
while True:
try:
links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "h2.title > a")))
#print(len(links))
for link in links:
print(link.get_attribute("ng-href"))
nextPage = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.btn-paging-pn.icof.icon-arr-right.paging-next")))
nextPage.click()
time.sleep(10)
except Exception as e:
print(e)
break

First of all never use the absolute xpath it will breakdown easily, Use the relative xpath.
Secondly, i think the error you are getting is because after clicking "Next" button for the first time it loads a new page. Which has a different DOM structure and that's why you are not able to find that element.
You can try searching for the element after every new page load (after clicking "Next" button everytime.)
// imports
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
// initialize
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
action = ActionChains(driver)
// Try to use the below code and see if it works.
Next_btn = wait.until(EC.presence_of_element_located((By.XPATH, '(//li[#class="paging-item"])[2]')))
action.move_to_element(Next_btn).click().perform()

How to fix "Message: stale element reference: element is not attached to the page document"

Trying got automate a task via Selenium python i have the issue where the for each section does work only the first time, after that does not see the second variable. Also tried to add delays so the webpage would be fully loaded but the same issue.
I tested different scenarios that i found in the internet also so manual tests i did, but looks like the second div is not recognizable also the rest of the divs
for server in browser.find_elements_by_xpath("//*[starts-with(#id,'server-list-')]"):
#try:
print("Server Section----")
time.sleep(5)
#Print server name
print(server.text)
#clicn on button inside the server
server.click()
#back into the server listing
browser.back()
Basically the automation need to enter every server ( div starting with id server-list- ) click on it, after entering on that section click another button and than back to the main page.

A stale element reference exception is thrown because of below reason
element has been deleted entirely.
element is no longer attached to the DOM.
Please check your element is still present on UI with the same element XPath which you are using while interacting with it
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "//*[starts-with(#id,'server-list-')]"))
)
for i in range(len(element)):
element[i].click()
driver.back()
element = wait.until(EC.presence_of_element_located((By.XPATH, "//*[starts-with(#id,'server-list-')]")))
finally:
driver.quit()

With limited information provided, you can try code below:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome()
wait = WebDriverWait(browser, 10)
servers = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//*[starts-with(#id,'server-list-')]")))
servers_count = len(servers)
for i in range(servers_count):
print(servers[i].text)
servers[i].click()
browser.back()
servers = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//*[starts-with(#id,'server-list-')]")))

You capture the server list that no more exists after you have drilled into the particular list item. So when you get back to the list that old one does not exist any more hence all the items (including that one you expect to move next to) are stale.
You need to rework your logic so that you get the list of servers each time you get back from the server details and store somethere the flag that would let your script know which items you have already visited.

How to click on a element as per the HTML provided through Selenium?

I have a link I need to click on:
<a id="selectLink">...</a>
I do it like so:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.ID, "selectLink")))
but for some reason the link doesnt click, or it does and nothing happens. When I do it manually it works.
I even try to put it in a loop and click on it until something happens, but then it works at times and sometimes it doesn't:
while True:
try:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.ID, "selectLink"))).click()
except Exception:
break
I can't tell what is the problem.
For example:
while True:
try:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.ID, "selectLink"))).click()
print(len(browser.find_elements_by_id("selectLink")))
print('click')
except Exception:
print(len(browser.find_elements_by_id("selectLink")))
print('break')
break
It gives me:
1
click
1
click
1
break
And still nothing happens. My question is how come the loop breaks even tho the link is still accessible, since the length is still 1?

I think you were pretty close. Let us see what went wrong.
As per your code first code trial:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.ID, "selectLink")))
In this attempt you have considered the ID attribute of the element where as the element is a <a> tag. As a general practice when you need to interact with a <a> tag is it always a better idea to take help of the linkText present within the <a> tag.
As per your code second code trial:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.ID, "aaaa.Isu_Recherche_de_prmAvanceView.Button_Selectionner"))).click()
In this attempt the ID attribute which you have considered doesn't matches the ID of the element as per the HTML.
Hence you see errors.
Solution
As per the HTML you have shared to click on the desired element you need to induce WebDriverWait for the element to be clickable and can use either of the solutions:
LINK_TEXT:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.LINK_TEXT, "Sélectionner"))).click()
PARTIAL_LINK_TEXT:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Sélectionner"))).click()
CSS_SELECTOR:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.urBtnCntTxt"))).click()
XPATH:
WebDriverWait(browser, timeout).until(EC.element_to_be_clickable((By.XPATH, "//span[#class='urBtnCntTxt'][contains(.,'Sélectionner')]"))).click()
Note :
You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
While using CSS_SELECTOR or XPATH it is always better to construct them with the help of minimum two attributes amongst class, id, or others

You can try something like this :
button = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "text between anchor tags")))
button.click()

If I were to guess, it might be a timing issue. If you click the link, the link still exists on the page since the locator is the same. If you instead click the link, wait for the element to go stale (which indicates the page is reloading), then wait for it to be clickable (indicating the page has finished reloading), then click it again... that loop might work.
wait = WebDriverWait(browser, timeout)
while driver.find_elements_by_id("selectLink")
link = wait.until(EC.element_to_be_clickable((By.ID, "selectLink")))
link.click()
wait.until(EC.staleness_of(link))

Selenium Element not visible exception

I have been tasked with writing a parser to click a button on a website and I am having issues to click only one of the buttons. The following code works on every button except one.
Here's the html:
http://pastebin.com/6dLF5ru8
here's the source html:
http://pastebin.com/XhsedGLb
python code:
driver = webdriver.Firefox()
...
el = driver.find_element_by_id("-spel-nba")
actions.move_to_element(el)
actions.sleep(.1)
actions.click()
actions.perform()
I am getting this error.
ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
as per Saifur I just tried waits with the same element not visible exception:
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.XPATH, "//input[contains(#id,'spsel')][#value='nba']"))).click()

If you look at the page source, you'll understand that almost all of theSELECT, DIV elements are faked and created from JavaScript, that is why webdriver cannot SEE them.
There's a workaround though, by using ActionChains to open your developer console, and inject an artificial CLICK on the desired element, which in fact, is the Label triggering the NBA data loading... here's a working example:
from selenium import webdriver
from selenium.webdriver.common import action_chains, keys
import time
driver = webdriver.Firefox()
driver.get('Your URL here...')
assert 'NBA' in driver.page_source
action = action_chains.ActionChains(driver)
# open up the developer console, mine on MAC, yours may be diff key combo
action.send_keys(keys.Keys.COMMAND+keys.Keys.ALT+'i')
action.perform()
time.sleep(3)
# this below ENTER is to rid of the above "i"
action.send_keys(keys.Keys.ENTER)
# inject the JavaScript...
action.send_keys("document.querySelectorAll('label.boxed')[1].click()"+keys.Keys.ENTER)
action.perform()
Alternatively to replace all the ActionChains commands, you can simply run execute_script like this:
driver.execute_script("document.querySelectorAll('label.boxed')[1].click()")
There you go, at least on my local file anyway... Hope this helps!

What worked for me was to find the element just before the problematic one (that is, just before it in terms of tab order), then call Tab on that element.
from selenium.webdriver.common.keys import Keys
elem = br.find_element_by_name("username")
elem.send_keys(Keys.TAB) # tab over to not-visible element
After doing that, I was able to send actions to the element.

The actual solution of this thread did not work for me.
however,
this one did :
element = WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.XPATH, xpaths['your_xpath_path'])))
the trick is to use :
EC.visibility_of_element_located
the WebDriverWait
WebDriverWait
from this import :
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

I suggest you use xpath with explicit wait
//input[contains(#id,'spsel')][#value='nba']

if "Element is not currently visible" then make it VISIBLE
f.e.
>>> before is hidden top is outside of page
<input type="file" style="position: absolute;top:-999999" name="file_u">
>>> after move top on in page area
DRIVER.execute_script("document.getElementByName('file_u').style.top = 0;")
time.sleep(1); # give some time to render
DRIVER.find_element_by_name("file_u").send_keys("/tmp/img.png")

Instead of get_element_by_id() you can try elem = browser.find_element_by_css_selector('#elemId') (go to that webpage and the element, right click it and Copy CSS Selector, or something like that.) This is what i did and it works. You also try find_element_by_link_text(text), find_element_by_partial_link_text(text), find_element_by_tag_name(tagName_case_insensitive_here), find_element_by_name(name) etc. Something will work. After the id the CSS Selector is your best bet.

I ended up using #twasbrillig's solution, but instead of finding the previous element and sending a TAB keypress, I find the desired element, send a TAB keypress with that element, and then a SHIFT + TAB keypress to the driver:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
el = driver.find_element_by_id("-spel-nba")
el.send_keys(Keys.TAB)
webdriver.ActionChains(driver).key_down(Keys.SHIFT).send_keys(Keys.TAB).key_up(Keys.SHIFT)

I tried using the other methods but in the end found that the simplest way was to just try and click the button, and catch the error. This allows me to perform other actions based on if it worked (True) or didn't (False).
def click_button(html_object):
try:
html_object.click()
except:
return False #most likely because it is NotVisible object and can be ignored
return True
...
...
click_button(actions)

The way I solved this in python was:
try:
# the element you want to scroll to
element = driver.find_element_by_id("some_id")
ActionChains(driver).move_to_element(element).perform()
element.send_keys(Keys.TAB).key_up(Keys.SHIFT)
#element.click()
except Exception as e:
print(e)