Develop Reference

Python Selenium - Identifying and Clicking Angular Elements

Utilizing Python and Selenium, I am attempting to click on the 'Matrix' and 'Transaction Desk' Angular buttons on a specific webpage (see screenshot). I have looked into this for hours now and have not found anything that has worked. It actually seems like there are a lot of unsolved questions still out there around this topic. The source code (see screenshot) does not appear to be within any iframe. The traceback just tells me that it could not find the specific code and times out.
My code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome("C:\\Users\\Matt\\Documents\\WebDriver\\chromedriver_win32\\chromedriver.exe")
driver.get("https://www.stellarmls.com/")
#Password authentication HERE, so you will be unable to access this site yourself.
driver.maximize_window()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="collapse911"]/app-links-panel-body/div/div/div[1]'))).click()
print(driver.page_source)
Traceback:
Traceback (most recent call last):
File "C:\Users\Matt\Documents\Splitt\ROI Property Finder.py", line 53, in <module>
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="collapse911"]/app-links-panel-body/div/div/div[1]'))).click()
File "C:\Users\Matt\Python3.9\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Supplementary screenshots:
iframe:

How to tell google that I accept cookies python

I am trying to accept cookies and search in google. But I am facing a problem that I never faced before.
Webdriver cant find the accept cookies button element. I looked everything. I tried to see if something changes the xpath(Is there any thing triggers any other element so the xpath will change) but I have zero in my hand. I tried using accept.send_keys(Keys.RETURN) and accept.click(). Nothing seems to work.
What I have for now;
def GoogleIt():
path = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(path)
driver.get("https://google.com")
wait(5)
accept = driver.find_element_by_xpath("/html/body/div/c-wiz/div[2]/div/div/div/div/div[2]/form/div/div[2]")
accept.send_keys(Keys.RETURN)
accept.click()
search = driver.find_element_by_xpath("/html/body/div/div[2]/form/div[2]/div[1]/div[1]/div/div[2]/input")
search.click()
search.send_keys(talk)
Note: wait() is a different name of time.sleep()
Error;
Traceback (most recent call last):
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\tkinter\__init__.py", line 1883, in __call__
return self.func(*args)
File "C:/Users/Teknoloji/Desktop/Phyton/Assistant V1/Assistant.py", line 20, in GoogleIt
accept = driver.find_element_by_xpath("/html/body/div/c-wiz/div[2]/div/div/div/div/div[2]/form/div/div[2]")
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Teknoloji\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/c-wiz/div[2]/div/div/div/div/div[2]/form/div/div[2]"}
(Session info: chrome=85.0.4183.121)
Process finished with exit code -1
If you need more information I am here.
Thanks...

To click on accept cookies button which is inside an iframe you need to switch to iframe first.
Induce WebDriverWait() and frame_to_be_available_and_switch_to_it() and following css selector.
Induce WebDriverWait() and element_to_be_clickable() and following xpath.
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src^='https://consent.google.com']")))
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[#id='introAgreeButton']"))).click()
You need to import following libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

In case KunduK's solution doesn't work for some of those reading this, use "By.ID" parameter instead of the "By.CSS_SELECTOR" in the frame_to_be_available_and_switch_to_it() method -
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "name_of_the_iframe")))

After the latest update of Selenium, the above solutions did not work for me.
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium import webdriver
from selenium.webdriver.common.by import By
firefox_binary = FirefoxBinary()
browser = webdriver.Firefox(firefox_binary=firefox_binary)
Let's say you want to get this page from Google:
browser.get('https://www.google.com/search?q=cats&source=lnms&tbm=isch')
To continue, you have to click on "Accept all"
Then, you use Selenium to click the button:
browser.find_element(By.XPATH,"//.[#aria-label='Accept all']").click()
Then, you can continue to Google page!

Selenium can't find id element in <div>

I'm trying to get an HTML text off a webpage using python selenium. But Selenium appears to be unable to locate the element. Not sure if I am doing anything wrong and is looking for some answers here.
Please see a part of my code below:
wait(driver, 10).until(EC.visibility_of_element_located((By.ID, 'current_address_name')))
full_address = driver.find_element_by_id('current_adress_name')
street_address = full_address.text.split('(S)')[0]
print('Street Address: ' + str(street_address))
all_street_address.append(street_address)
This is the error message (timeout). Supposedly because it cannot find the element:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/Volumes/GoogleDrive/My Drive/PycharmProjects/googlestuff/gsheet_test_mod.py", line 121, in web_extraction
wait(driver, 10).until(EC.visibility_of_element_located((By.ID, 'current_address_name')))
File "/Users/cadellteng/googstuff/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
And this is a part of the HTML code:
<div id="current_address_name" style="display:none">4 Third Avenue (S)266576</div>
A little background about this thing though. This part of the element doesn't seem to be a visible element on the page because when I put my mouse over the element, it does not highlight any part of the page. Is it because Selenium cannot extract elements that are not displayed? Or am I doing something wrong here? Please advise.
Just in case you need the full code, this is the page: Singapore Streetdirectory

The visibility_of_element_located method is for "checking that an element is present on the DOM of a page and visible.".
Clearly this element is not visible, as the element has style="display:none" set.
You could use presence_of_element_located instead, which will just check that the element is present in the DOM tree:
wait(driver, 10).until(EC.presence_of_element_located((By.ID, 'current_address_name')))
Fixing a few other issues with your code, the full minimal, reproducible example would be:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium import webdriver
all_street_address = []
driver = webdriver.Chrome()
wait = WebDriverWait
driver.get("https://www.streetdirectory.com/sg/4-third-avenue-266576/1_27869.html")
wait(driver, 10).until(EC.presence_of_element_located((By.ID, 'current_address_name')))
full_address = driver.find_element_by_id('current_address_name')
street_address = full_address.get_attribute('innerHTML').split('(S)')[0]
print('Street Address: ' + str(street_address))
all_street_address.append(street_address)

Can't perform a click on a search button because of some ad hiding it

I've written a script in python with selenium to perform a search in a webpage using search-box available there. However, When I run my script, It throws an error which I'm gonna paste below. The thing is when the webpage is loaded through my script, there is an advertisement pops up hiding the search-box. How can I get around that and fetch the search result? Thanks in advance.
Link to that site: webpage
Script I'm trying with:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("replace_with_above_site")
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input#q"))).send_keys("Camden Medical Centre, 1 Orchard Boulevard 248649")
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input#search_button"))).click() ##error thrown here
driver.quit()
Traceback I'm having:
Traceback (most recent call last):
File "C:\Users\WCS\AppData\Local\Programs\Python\Python36-32\demo.py", line 12, in <module>
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input#search_button"))).click()
File "C:\Users\WCS\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\WCS\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "C:\Users\WCS\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "C:\Users\WCS\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element <input type="button" id="search_button" onclick="submitSearch();"> is not clickable at point (868, 137). Other element would receive the click: <div id="splash_screen_overlay"></div>
This is the ad which hides the search box:
Btw, the search parameter is available within .send_keys() in my script. Anything from the populated result will suffice.

The simplest solution is to simulate exactly the same action user should do: close an ad in case it appeared:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("replace_with_above_site")
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.btn_close"))).click()
except NoSuchElementException:
pass
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input#q"))).send_keys("Camden Medical Centre, 1 Orchard Boulevard 248649")
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input#search_button"))).click()
driver.quit()

Use below code. This will click the element even if it is hidden beneath ad
((IJavaScriptExecutor)driver).ExecuteScript("arguments[0].scrollIntoView(true)", driver.GetElement(By.Id("someID")));

Selenium -python element not clickable

I am trying to browse facebook via selenium in python.
Here is my script so far.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
usr = ""# I have put 3 different accounts and tested it. Same error
pwd = ""
driver = webdriver.Chrome('E:\python_libs\chromedriver.exe')
driver.get("https://www.facebook.com")
assert "Facebook" in driver.title
elem = driver.find_element_by_id("email")
elem.send_keys(usr)
elem = driver.find_element_by_id("pass")
elem.send_keys(pwd)
elem.send_keys(Keys.RETURN)
elem = driver.find_element_by_css_selector(".input.textInput")
elem.send_keys("Posted using Python's Selenium WebDriver bindings!")
elem = driver.find_element_by_css_selector(".selected")
elem.click()
time.sleep(5)
driver.close()
This script when run on windows with chromedriver obtained and correctly installed returns an error.
Traceback (most recent call last):
File "C:\Users\Home\Desktop\facebook_post.py", line 28, in <module>
elem.click()
File "C:\Python27\lib\site-packages\selenium-2.42.0- py2.7.egg\selenium\webdriver\remote\webelement.py", line 60, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium-2.42.0- py2.7.egg\selenium\webdriver\remote\webelement.py", line 370, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium-2.42.0-py2.7.egg\selenium\webdriver\remote\webdriver.py", line 172, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium-2.42.0-py2.7.egg\selenium\webdriver\remote\errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: u'unknown error: Element is not clickable at point (481, 185). Other element would receive the click: <input type="file" class="_n _5f0v" title="Choose a file to upload" accept="image/*" name="file" id="js_0">\n (Session info: chrome=35.0.1916.114)\n (Driver info: chromedriver=2.10.267521,platform=Windows NT 6.1 x86)'
I am unable to make head or tail of this.
Any help would be appreciated.
The Graph API is not feasible here as I want to browse as myself and not as some application.
If however browsing as myself can be done by graph API or some other means, please do tell.

If you're doing this process several times, Selenium has some options in WebDriverWait to not waste too much time and check to see if elements are visible.
Here is the documentation on Waits in Selenium and
this is a section of the Python documentation of Selenium that talks about the Expected Conditions classes. It seems that "clickable" is one conditions that you can check for.
In my case, I wrote a simple function that took care of visibility and clicking and called it every time I needed to click on something dynamic.
My example code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Browser = webdriver.Chrome()
def wait_until_visible_then_click(element):
element = WebDriverWait(Browser,5,poll_frequency=.2).until(
EC.visibility_of(element))
element.click()
EDIT:
The links above seem to be nerfed. This is the new documentation on waits and here are the expected conditions docs.