Develop Reference

How to check all checkboxes on a webpage with Selenium in Python (using a loop)

I am attempting to write a python program that will check all checkboxes on a given webpage. Currently, I only have the ability to select a single checkbox and am unsure why I am not able to iterate through the loop of elements containing checkboxes on this example page. Here is my code so far:
from selenium import webdriver
driver = webdriver.Firefox()
driver.maximize_window()
driver.get(
"https://www.sugarcrm.com/request-demo/")
elements = []
elements = driver.find_elements_by_xpath("//*[#class='form-check-input']")
elemlength = len(elements)
for x in range(1, elemlength):
if driver.find_element_by_xpath("//input[#type='checkbox']").is_selected():
print("Already Selected")
else:
driver.find_element_by_xpath("//input[#type='checkbox']").click()

driver.find_element_by_xpath("//input[#type='checkbox']") will return the first checkbox it finds every time, so it will always be the same one.
You can use the list of elements returned by driver.find_elements_by_xpath("//*[#class='form-check-input']") instead.
elements = driver.find_elements_by_xpath("//*[#class='form-check-input']")
for element in elements:
if element.is_selected():
print("Already Selected")
else:
element.click()

Python for loop not looping through all the elements? It is only taking the 1st element

I am trying to scrape the data from naukri.com, here I am trying to scrape the location details for each recruiter visible on the page.
The code I am writing is :
def sol7(url):
driver = webdriver.Chrome(r'C:\Users\Rahul\Downloads\chromedriver_win32\chromedriver.exe')
driver.maximize_window()
driver.get(url)
#getting link of recruiters
recruiters_tag = driver.find_element_by_xpath("//ul[#class='midSec menu']/li[2]/a")
driver.get(recruiters_tag.get_attribute('href'))
#search and click for data scientist
driver.find_element_by_xpath("//input[#class='sugInp']").send_keys('Data science')
driver.find_element_by_xpath("//button[#id='qsbFormBtn']").click()
highlight_table_tag = driver.find_elements_by_xpath("//p[#class='highlightable']")
print(len(highlight_table_tag))
for i in highlight_table_tag:
try:
print((i.find_element_by_xpath("//small[#class='ellipsis']")).text)
except NoSuchElementException:
print('-')
1st I have extracted all recruiters details in list highlight_table_tag.
highlight_table_tag includes all the elements on the page however the loop only takes the 0th element of my list.
I want to scrape the location in such a way, if the location tag is not there in each element of highlight_table_tag then print '-' instead.
Please help!!!!

While xpathing child elements use . in front of the xpath otherwise you will be getting it from root node. Which ends up with 1 element everytime.
print((i.find_element_by_xpath(".//small[#class='ellipsis']")).text)

How can I check if an element exists on a page using Selenium XPath?

I'm writing a script in to do some webscraping on my Firebase for a few select users. After accessing the events page for a user, I want to check for the condition that no events have been logged by that user first.
For this, I am using Selenium and Python. Using XPath seems to work fine for locating links and navigation in all other parts of the script, except for accessing elements in a table. At first, I thought I might have been using the wrong XPath expression, so I copied the path directly from Chrome's inspection window, but still no luck.
As an alternative, I have tried to copy the page source and pass it into Beautiful Soup, and then parse it there to check for the element. No luck there either.
Here's some of the code, and some of the HTML I'm trying to parse. Where am I going wrong?
# Using WebDriver - always triggers an exception
def check_if_user_has_any_data():
try:
time.sleep(10)
element = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//*[#id="event-table"]/div/div/div[2]/mobile-table/md-whiteframe/div[1]/ga-no-data-table/div')))
print(type(element))
if element == True:
print("Found empty state by copying XPath expression directly. It is a bit risky, but it seems to have worked")
else:
print("didn’t find empty state")
except:
print("could not find the empty state element", EC)
# Using Beautiful Soup
def check_if_user_has_any_data#2():
time.sleep(10)
html = driver.execute_script("return document.documentElement.outerHTML")
soup = BeautifulSoup(html, 'html.parser')
print(soup.text[:500])
print(len(soup.findAll('div', {"class": "table-row-no-data ng-scope"})))
HTML
<div class="table-row-no-data ng-scope" ng-if="::config" ng-class="{overlay: config.isBuilderOpen()}">
<div class="no-data-content layout-align-center-center layout-row" layout="row" layout-align="center center">
<!-- ... -->
</div>
The first version triggers the exception and is expected to evaluate 'element' as True. Actual, the element is not found.
The second version prints the first 500 characters (correctly, as far as I can tell), but it returns '0'. It is expected to return '1' after inspecting the page source.

Use the following code:
elements = driver.find_elements_by_xpath("//*[#id='event-table']/div/div/div[2]/mobile-table/md-whiteframe/div[1]/ga-no-data-table/div")
size = len(elements)
if len(elements) > 0:
# Element is present. Do your action
else:
# Element is not present. Do alternative action
Note: find_elements will not generate or throw any exception

Here is the method that generally I use.
Imports
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
Method
def is_element_present(self, how, what):
try:
self.driver.find_element(by=how, value=what)
except NoSuchElementException as e:
return False
return True

Some things load dynamically. It is better to just set a timeout on a wait exception.

If you're using Python and Selenium, you can use this:
try:
driver.find_element_by_xpath("<Full XPath expression>") # Test the element if exist
# <Other code>
except:
# <Run these if element doesn't exist>

I've solved it. The page had a bunch of different iframe elements, and I didn't know that one had to switch between frames in Selenium to access those elements.
There was nothing wrong with the initial code, or the suggested solutions which also worked fine when I tested them.
Here's the code I used to test it:
# Time for the page to load
time.sleep(20)
# Find all iframes
iframes = driver.find_elements_by_tag_name("iframe")
# From inspecting page source, it looks like the index for the relevant iframe is [0]
x = len(iframes)
print("Found ", x, " iFrames") # Should return 5
driver.switch_to.frame(iframes[0])
print("switched to frame [0]")
if WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//*[#class="no-data-title ng-binding"]'))):
print("Found it in this frame!")

Check the length of the element you are retrieving with an if statement,
Example:
element = ('https://www.example.com').
if len(element) > 1:
# Do something.

Webscrape table from CME, including selection of desired table from drop down menu

The below code works. It returns data in the default loaded table (making use of answer provided here: link, but how to access the other tables (that can be found by clicking on the 'Contracts' button and selecting from the menu a different contract, eg. Mar 2019)?
driver.get("http://www.cmegroup.com/tools-information/quikstrike/treasury-analytics.html")
# Need to include some more time here for data in iframe to load?
driver.implicitly_wait(3)
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
soup = BeautifulSoup(driver.page_source, 'html.parser')
CMEtreasuryAnalytics._table = soup.select('table.grid')[0]
I tried this but get the following error returned: NoSuchFrameException: Message: no such frame: element is not a frame
driver.get("http://www.cmegroup.com/tools-nformation/quikstrike/treasury-analytics.html")
cDate = 'Dec 2018'
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
elements = driver.find_elements_by_class_name("square-corners ")
options = [element.get_attribute("innerText") for element in elements]
if cDate in options:
element = elements[options.index(cDate)]
else:
pass
driver.switch_to.frame(element)
I've also tryed to 'click()' but couldn't get that to work either. I'm new to selenium and would appreciate some pointers on how to access the said data. I'm using python and chrome webdriver.

OK. I think I worked it out. The menu lies within the iFrame,
so after getting the element details, then need to click() the menu, then element.click(), then scrape the displayed data. The final code follows, but I don't know if it's the most straightforward way to approach it.
driver.get("http://www.cmegroup.com/tools-nformation/quikstrike/treasury-analytics.html")
cDate = 'Jun 2019'
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
elements = driver.find_elements_by_class_name("square-corners ")
options = [element.get_attribute("innerText") for element in elements]
if cDate in options:
element = elements[options.index(cDate)]
else:
pass
# Click the dropdown menu labelled 'Contracts'
driver.find_element_by_xpath('//*[#id="ctl00_MainContent_ucViewControl_IntegratedStrikeAsYield_ucContractPicker_ucTrigger_lnkTrigger"]').click()
driver.implicitly_wait(1)
element.click()
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
soup = BeautifulSoup(driver.page_source, 'html.parser')
CMEtreasuryAnalytics._table = soup.select('table.grid')[0]
Update:
The above worked for a while but then started failing with the below message. So maybe this is the right track but I need better way to select an option from the drop down list labelled 'Contracts'. How to do that?
Message: unknown error: Element is not clickable at point (511, 475). Other element would receive the click: <

Continue when element is not found in selenium python

The following script follows a page in Instagram:
browser = webdriver.Chrome('./chromedriver')
# GO INSTAGRAM PAGE FOR LOGIN
browser.get('https://www.instagram.com/accounts/login/?hl=it')
sleep(2)
# ID AND PASSWORD
elem = browser.find_element_by_name("username").send_keys('test')
elem = browser.find_element_by_name("password").send_keys('passw')
# CLICK BUTTON AND OPEN INSTAGRAM
sleep(5)
good_elem = browser.find_element_by_xpath('//*[#id="react-root"]/section/main/div/article/div/div[1]/div/form/span/button').click()
sleep(5)
browser.get("https://www.instagram.com")
# GO TO PAGE FOR FOLLOW
browser.get("https://www.instagram.com/iam.ai4/")
sleep(28)
segui = browser.find_element_by_class_name('BY3EC').click()
If an element with class BY3EC isn't found I want the script to keep working.

When an element is not found it throws NoSuchElementException, so you can use try/except to avoid that, for example:
from selenium.common.exceptions import NoSuchElementException
try:
segui = browser.find_element_by_class_name('BY3EC').click()
except NoSuchElementException:
print('Element BY3EC not found') # or do something else here
You can take a look at selenium exceptions to get an idea of what each one of them is for.

surround it with try catches, than you can build a happy path and handle failures as well, so your test case will always work

Best practice is to not use Exceptions to control flow. Exceptions should be exceptional... rare and unexpected. The simple way to do this is to get a collection using the locator and then see if the collection is empty. If it is, you know the element doesn't exist.
In the example below we search the page for the element you wanted and check to see that the collection contains an element, if it does... click it.
segui = browser.find_elements_by_class_name('BY3EC')
if segui:
segui[0].click()