Develop Reference

How to get selenium to wait on multiple elements to load

I am using the following code to wait on all 4 elements to be loaded before proceeding with the screen scrape; however, the code is not waiting on all 4 nor is throwing a timeout error -- it just proceeds and I get an error on elements that haven't yet been loaded.
What am I missing to get Selenium to wait until all four elements are present before proceeding?
CSSSelector1_toWaitOn = "#div1 table tbody tr td"
CSSSelector2_toWaitOn = "#div2 table tbody tr:nth-child(5) td"
CSSSelector3_toWaitOn = "#div3 table tbody tr:nth-child(5) td"
CSSSelector4_toWaitOn = "#div4 table tbody tr td"
browser.get(url)
browser_delay = 15 # seconds
try:
WebDriverWait(browser, browser_delay).until(expected_conditions and (
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector1_toWaitOn)) and
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector2_toWaitOn)) and
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector3_toWaitOn)) and
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector4_toWaitOn))))
except TimeoutException:
print("Selenium timeout")```

WebDriverWait.until expects callable object. This is an actual snippet from its source:
while True:
try:
value = method(self._driver)
if value:
return value
All expected_contiditions are callable objects. So in this case you need to compose them, something like following should work.
class composed_expected_conditions:
def __init__(self, expected_conditions):
self.expected_conditions = expected_conditions
def __call__(self, driver):
for expected_condition in self.expected_conditions:
if not expected_condition(driver):
return False
return True
And pass it to the until
conditions = [
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector1_toWaitOn)),
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector2_toWaitOn)),
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector3_toWaitOn)),
expected_conditions.presence_of_element_located((By.CSS_SELECTOR, CSSSelector4_toWaitOn)),
]
WebDriverWait(browser, browser_delay).until(composed_expected_conditions(conditions))

The method presence_of_element_located(locator) only checks that the element is present in the DOM. It does not mean that the element can be interacted with. Furthermore, the search process finds all the elements for given locator and returns first one.
Please check that the element is valid, available, and specific. In case there are multiple elements in the list, make sure your locator is specific enough to find the single element.

Rather than trying to combine them all into a single wait, you can have a separate wait for each.
...
try:
wait = WebDriverWait(browser, browser_delay)
wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, CSSSelector1_toWaitOn))
wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, CSSSelector2_toWaitOn))
wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, CSSSelector3_toWaitOn))
wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, CSSSelector4_toWaitOn))))
except TimeoutException:
print("Selenium timeout")
Just be aware, there are 3 levels of interaction within Selenium for elements:
present - the element is in the DOM. If you attempt to click on or get text from, etc. a present (but not visble) element, an ElementNotInteractable exception will be thrown.
visible - the element is in the DOM and visible (e.g. not invisible, display: none, etc.)
clickable - the element is visible and enabled. For most cases, this is just... is it visible? The special cases would be elements like an INPUT button that is marked as disabled. An element that is styled as disabled (greyed out) using CSS, is not considered disabled.

How can I check if an element exists on a page using Selenium XPath?

I'm writing a script in to do some webscraping on my Firebase for a few select users. After accessing the events page for a user, I want to check for the condition that no events have been logged by that user first.
For this, I am using Selenium and Python. Using XPath seems to work fine for locating links and navigation in all other parts of the script, except for accessing elements in a table. At first, I thought I might have been using the wrong XPath expression, so I copied the path directly from Chrome's inspection window, but still no luck.
As an alternative, I have tried to copy the page source and pass it into Beautiful Soup, and then parse it there to check for the element. No luck there either.
Here's some of the code, and some of the HTML I'm trying to parse. Where am I going wrong?
# Using WebDriver - always triggers an exception
def check_if_user_has_any_data():
try:
time.sleep(10)
element = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//*[#id="event-table"]/div/div/div[2]/mobile-table/md-whiteframe/div[1]/ga-no-data-table/div')))
print(type(element))
if element == True:
print("Found empty state by copying XPath expression directly. It is a bit risky, but it seems to have worked")
else:
print("didn’t find empty state")
except:
print("could not find the empty state element", EC)
# Using Beautiful Soup
def check_if_user_has_any_data#2():
time.sleep(10)
html = driver.execute_script("return document.documentElement.outerHTML")
soup = BeautifulSoup(html, 'html.parser')
print(soup.text[:500])
print(len(soup.findAll('div', {"class": "table-row-no-data ng-scope"})))
HTML
<div class="table-row-no-data ng-scope" ng-if="::config" ng-class="{overlay: config.isBuilderOpen()}">
<div class="no-data-content layout-align-center-center layout-row" layout="row" layout-align="center center">
<!-- ... -->
</div>
The first version triggers the exception and is expected to evaluate 'element' as True. Actual, the element is not found.
The second version prints the first 500 characters (correctly, as far as I can tell), but it returns '0'. It is expected to return '1' after inspecting the page source.

Use the following code:
elements = driver.find_elements_by_xpath("//*[#id='event-table']/div/div/div[2]/mobile-table/md-whiteframe/div[1]/ga-no-data-table/div")
size = len(elements)
if len(elements) > 0:
# Element is present. Do your action
else:
# Element is not present. Do alternative action
Note: find_elements will not generate or throw any exception

Here is the method that generally I use.
Imports
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
Method
def is_element_present(self, how, what):
try:
self.driver.find_element(by=how, value=what)
except NoSuchElementException as e:
return False
return True

Some things load dynamically. It is better to just set a timeout on a wait exception.

If you're using Python and Selenium, you can use this:
try:
driver.find_element_by_xpath("<Full XPath expression>") # Test the element if exist
# <Other code>
except:
# <Run these if element doesn't exist>

I've solved it. The page had a bunch of different iframe elements, and I didn't know that one had to switch between frames in Selenium to access those elements.
There was nothing wrong with the initial code, or the suggested solutions which also worked fine when I tested them.
Here's the code I used to test it:
# Time for the page to load
time.sleep(20)
# Find all iframes
iframes = driver.find_elements_by_tag_name("iframe")
# From inspecting page source, it looks like the index for the relevant iframe is [0]
x = len(iframes)
print("Found ", x, " iFrames") # Should return 5
driver.switch_to.frame(iframes[0])
print("switched to frame [0]")
if WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//*[#class="no-data-title ng-binding"]'))):
print("Found it in this frame!")

Check the length of the element you are retrieving with an if statement,
Example:
element = ('https://www.example.com').
if len(element) > 1:
# Do something.

web scraping contat list selenium python

How can i loop through contacts in group in Discord using selenium in Python?
I tried this code, and i have this error:
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
Problem is scroller and contacts are constantly updating...
I tried this code:
while True:
num=0
try:
users_list = driver.find_elements_by_css_selector("div.memberOnline-1CIh-0.member-3W1lQa")
for user in users_list:
num+=1
user.click()
driver.execute_script("arguments[0].scrollIntoView();",user)
print('User number {}'.format(num))
except StaleElementReferenceException and ElementClickInterceptedException:
print('bad')
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight",users_list)

From your given code, you only scroll element, so the reason of Stale exception is you not wait page load complete, or at least not wait the contacts not load complete.
For debug purpose, you can simple add a long sleep before the loop, like sleep(15), and replace to explicit wait if production code, like
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
Detail of Explicit Wait at here
If you call click() in loop, you need to find the elements again in loop:
while True:
num=0
try:
time.sleep(15)
users_list = driver
.find_elements_by_css_selector("div.memberOnline-1CIh-0.member-3W1lQa")
length = len(users_list)
for num in range(0, length):
user = users_list[num]
user.click()
time.sleep(15)
driver.execute_script("arguments[0].scrollIntoView();",user)
print('User number {}'.format(num+1))
// because the above `click` make page happen changes
// so selenium will treat it as a new page,
// those element reference found on `old` page, can not work on `new` page
// you need to find elements belongs to `old` page again on `new` page
// find users_list again from `new` page
users_list = driver
.find_elements_by_css_selector("div.memberOnline-1CIh-0.member-3W1lQa")
except StaleElementReferenceException and ElementClickInterceptedException:
print('bad')
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight",
users_list)

How to dismiss a method when except block is run in a try-except

I'm just working on a simple script that goes through real estate listings and collects agents personal websites. I ran into an issue when I came across a listing where the agent didn't have a website in which case the script stopped working. Now i've put in a try-except which works until the except block is run, when that happens the whole browser closes.
time.sleep(15)
for i in range(1,9):
listing_page = driver.find_element_by_xpath('//*[#id="m_property_lst_cnt_realtor_more_'+str(i)+'"]').click()
try:
realtor_url = driver.find_element_by_xpath('//*[#id="lblMediaLinks"]/a').click()
WebDriverWait(driver, 10).until(lambda d: len(driver.window_handles) == 2)
driver.switch_to_window(driver.window_handles[1])
WebDriverWait(driver, 10)
except:
print("Not found")
continue
driver.close()
driver.switch_to_window(driver.window_handles[0])
driver.get(home_page)
time.sleep(10)
Is there anyway I can revert back to the home page and start the loop again when the except block is run? Then if it's not; the loop runs as usual?

In my opinion, using exceptions as logic flow is not a good practice. Exceptions should be exceptional... they should indicate when something unexpectedly wrong happened.
Instead, use find_elements_* (plural) and check if the collection returned is empty. If it's not empty, that means that the link exists, click on it, etc. If the collection is empty, return to home and start the next loop.
for i in range(1,9):
listing_page = driver.find_element_by_xpath('//*[#id="m_property_lst_cnt_realtor_more_'+str(i)+'"]').click()
realtor_url = driver.find_elements_by_xpath('//*[#id="lblMediaLinks"]/a')
if (len(realtor_url)) > 0
realtor_url[0].click
WebDriverWait(driver, 10).until(lambda d: len(driver.window_handles) == 2)
driver.switch_to_window(driver.window_handles[1])
driver.close()
driver.switch_to_window(driver.window_handles[0])
driver.get(home_page)
BTW, .click() still doesn't return anything so assigning its return to a variable will never return anything but null so I removed that part.

Scraper: Try skips code in while loop (Python)

I am working on my first scraper and ran into an issue. My scraper accesses a website and saves links from the each result page. Now, I only want it to go through 10 pages. The problem comes when the search results has less than 10 pages. I tried using a while loop along with a try statement, but it does not seem to work. After the scraper goes through the first page of results, it does not return any links on the successive pages; however, it does not give me an error and stops once it reaches 10 pages or the exception.
Here is a snippet of my code:
links = []
page = 1
while(page <= 10):
try:
# Get information from the propertyInfo class
properties = WebDriverWait(driver, 10).until(lambda driver: driver.find_elements_by_xpath('//div[#class = "propertyInfo item"]'))
# For each listing
for p in properties:
# Find all elements with a tags
tmp_link = p.find_elements_by_xpath('.//a')
# Get the link from the second element to avoid error
links.append(tmp_link[1].get_attribute('href'))
page += 1
WebDriverWait(driver, 10).until(lambda driver: driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click())
except ElementNotVisibleException:
break
I really appreciate any pointers on how to fix this issue.

You are explicitely catching ElementNotVisibleException exception and stopping on it. This way you won't see any error message. The error is probably in this line:
WebDriverWait(driver, 10).until(lambda driver:
driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click())
I assume lambda here should be a test, which is run until succeeded. So it shouldn't make any action like click. I actually believe that you don't need to wait here at all, page should be already fully loaded so you can just click on the link:
driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click()
This will either pass to next page (and WebDriverWait at the start of the loop will wait for it) or raise exception if no next link is found.
Also, you better minimize try ... except scope, this way you won't capture something unintentionally. E.g. here you only want to surround next link finding code not the whole loop body:
# ...
while(page <= 10):
# Scrape this page
properties = WebDriverWait(driver, 10).until(...)
for p in properties:
# ...
page += 1
# Try to pass to next page
try:
driver.find_element_by_xpath('//*[#id="paginador_siguiente"]/a').click()
except ElementNotVisibleException:
# Break if no next link is found
break