Develop Reference

Selenium unable to locate xpath however the xpath is there in the browser

Im not sure if that was quite the best title but Im not sure how else to describe it, I am trying to use selenium to automate 2fa on a website so all I need to do is anwser the phone call and the script will take care of the rest however the button I am trying to get selenium to click keeps showing up as unable to locate even though its always in the same place and never changes here is my code in python
callMe = driver.find_element('xpath', '//*[#id="auth_methods"]/fieldset/div[2]/button')
callMe.click()
sleep(25)
this is one of three buttons that all have the same element information besides the xpaths here are all 3 button elements I am trying to grab the second one
<button tabindex="2" type="submit" class="positive auth-button"><!-- -->Send Me a Push </button>
<button tabindex="2" type="submit" class="positive auth-button"><!-- -->Call Me </button>
<button tabindex="2" type="submit" class="positive auth-button"><!-- -->Text Me </button>
I am not sure how else I can locate the second button besides using the xpath but that is not working and I dont know if I can or how I can search for the button based off the text that is inside it.

Did you try with By ?
from selenium.webdriver.common.by import By
callMe = driver.find_element(By.XPATH, '//*[#id="auth_methods"]/fieldset/div[2]/button')

Try using By.cssselector, get the css selector of the main body html for the button you want.
callMe = driver.find_element(By.css_selector, 'selectorofbodyhtml')
callme.click()

Try below xpath
//button[starts-with(text(),'Call Me')]

The desired element Call Me is a dynamic element, so to click on it you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using XPATH and contains():
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(., 'Call Me')]"))).click()
Using XPATH and starts-with():
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[starts-with(., 'Call Me')]"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Pull data from a div class Python Selenium

I'm trying to pull a specific number out of a div class in Python Selenium but can't figure out how to do it. I'd want to get the "post_parent" ID 947630 as long as it matches the "post_name" number starting 09007.
I'm looking to do this across multiple "post_name" classes, so I'd feed it something like this: search_text = "0900766b80090cb6", but there will be multiple in the future so it has to read the "post_name" first then pull the "post_parent" if that makes sense.
Appreciate any advice anyone has to offer.
<div class="hidden" id="inline_947631">
<div class="post_title">Interface Converter</div>
<div class="post_name">0900766b80090cb6</div>
<div class="post_author">28</div>
<div class="comment_status">closed</div>
<div class="ping_status">closed</div>
<div class="_status">inherit</div>
<div class="jj">06</div>
<div class="mm">07</div>
<div class="aa">2001</div>
<div class="hh">15</div>
<div class="mn">44</div>
<div class="ss">17</div>
<div class="post_password"></div>
<div class="post_parent">947630</div>
<div class="page_template">default</div>
<div class="tags_input" id="rs-language-code_947631">de</div>
</div>

If you see <div class="post_name">0900766b80090cb6</div> this and <div class="post_parent">947630</div> are siblings nodes to each other.
You can use xpath -> following-sibling like this:
Code:
search_text = "0900766b80090cb6"
post_parent_num = driver.find_element(By.XPATH, f"//div[#class='post_name' and text()='{search_text}']//following-sibling::div[#class='post_parent']").text
print(post_parent_num)
or Using ExplicitWait:
search_text = "0900766b80090cb6"
post_parent_num = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, f"//div[#class='post_name' and text()='{search_text}']//following-sibling::div[#class='post_parent']"))).get_attribute('innerText')
print(post_parent_num)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Update:
NoSuchElementException:
Please check in the dev tools (Google chrome) if we have unique entry in HTML-DOM or not.
xpath that you should check :
//div[#class='post_name' and text()='0900766b80090cb6']//following-sibling::div[#class='post_parent']
Steps to check:
Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.
If this is unique //div[#class='post_name' and text()='0900766b80090cb6']//following-sibling::div[#class='post_parent'] then you need to check for the below conditions as well.
Check if it's in any iframe/frame/frameset.
Solution: switch to iframe/frame/frameset first and then interact with this web element.
Check if it's in any shadow-root.
Solution: Use driver.execute_script('return document.querySelector to have returned a web element and then operates accordingly.
Make sure that the element is rendered properly before interacting with it. Put some hardcoded delay or Explicit wait and try again.
Solution: time.sleep(5) or
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#class='post_name' and text()='0900766b80090cb6']//following-sibling::div[#class='post_parent']"))).text
If you have redirected to a new tab/ or new windows and you have not switched to that particular new tab/new window, otherwise you will likely get NoSuchElement exception.
Solution: switch to the relevant window/tab first.
If you have switched to an iframe and the new desired element is not in the same iframe context then first switch to default content and then interact with it.
Solution: switch to default content and then switch to respective iframe.

I don't see any specific relation between "post_parent" ID 947630 and "post_name" number starting 09007. Moreover, the parent <div> is having class="hidden".
However, to pull the specific number you can use either of the following locator strategies:
Using css_selector:
print(driver.find_element(By.CSS_SELECTOR, "div[id^='inline'] div.post_parent").text)
Using xpath:
print(driver.find_element(By.XPATH, "//div[starts-with(#id, 'inline_')]//div[#class='post_parent']").text)
Ideally you need to induce WebDriverWait for the presence_of_element_located() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div[id^='inline'] div.post_parent"))).text)
Using XPATH:
print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//div[starts-with(#id, 'inline_')]//div[#class='post_parent']"))).text)
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can create a method and use the following xpath to get the post_parent text based on post_name text.
def getPostPatent(postname):
element=driver.find_element(By.XPATH,"//div[#class='post_name' and starts-with(text(),'{}')]/following-sibling::div[#class='post_parent']".format(postname))
print(element.get_attribute("textContent"))
getPostPatent('09007')
This will return value if it is matches the text starts-with('09007')
It seems parent class is hidden you need to use textContent to get the value.

How to click on the angular based checkbox as per the HTML provided through Selenium and Python?

I am trying to click on a checkbox using selenium python. I tried
buttons = driver.find_element_by_xpath("//*[contains(text(), 'Exact')]"); buttons.click()
I keep getting
"ElementNotVisibleException:"
<button type="button" data-ng-class="{iconCheck: event.locationExactness.isExact, inputBox:!event.locationExactness.isExact}" class="link icon locationButton inputBox" data-sfs-callout-visible="relativeExactnesses.length > 1" data-sfs-callout="sfs_-sfsLocationExactness-1-place-callout" data-sfs-callout-focus="sfs_-sfsLocationExactness-1-exact" data-ng-click="updateIsExact(relativeExactnesses.length > 1 ? true : !event.locationExactness.isExact)" data-autoname="NameAPlace_msypn_LocationExactButton"><!--
--><span class="locationLabel ng-binding">Exact</span><!--
--></button>
<<pseudo:before>></<pseudo:before>>
<!--
-->
<span class="locationLabel ng-binding">Exact</span>
<!--
-->

First of all you are locating one element, but are giving it plural name buttons. I would use singular name button.
Second thing is even in your code snippet there is two span which contain Exact. You have to change your locator so, that only one element(which you want to interact with) will be selected via selector.
For example, if you want the first Exact in your code snippet, you can use this xPath:
//button[#data-sfs-callout = 'sfs_-sfsLocationExactness-1-place-callout']/span
Note: as #RajKamal already mentioned, there can be multiple elements on the page with text Exact. You can check this in dev tools by pressing F12.

As per the HTML you have shared the element with text as Exact is a Angular element so to invoke click() on the desired element you have to induce WebDriverWait for the element to be clickable and you can use the following solution:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='link icon locationButton inputBox' and #data-autoname='NameAPlace_msypn_LocationExactButton']//span[#class='locationLabel ng-binding'][contains(.,'Exact')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Python Selenium: Getting dynamic content within iframe

I am trying to scrape the available apartment listings from the following webpage: https://3160599v2.onlineleasing.realpage.com/
I am using the Python implementation of Selenium, but so far I haven't found an effective solution to programmatically get the content. My most basic code is the following, which currently just returns the non-dynamic HTML source code:
from selenium import webdriver
driver = webdriver.Chrome('/path_to_driver')
driver.get('https://3160599v2.onlineleasing.realpage.com/')
html = driver.page_source
The returned html variable does not contain the apartment listings I need.
If I 'Inspect' the element using Chrome's built-in inspect tool, I can see that the content is within an un-classed iframe: <iframe frameborder="0" realpage-oll-widget="RealPage-OLL-Widget" style="width: 940px; border: none; overflow: hidden; height: 2251px;"></iframe>
Several children down within this iframe you can also see the div <div class="main-content"> which contains all the info I need.
Other solutions I have tried include implementing an explicit WebDriverWait:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CLASS_NAME, 'main-content')))
I get a TimeoutException with this method as the element is never found.
I also tried using the driver.switch_to.frame() method, with no success.
The only steps that have actually allowed me to get the apartment listings out of the webpage have been (using Chrome):
Manually right-click on an element of the listings within the webpage
Click Inspect
Find the div 'main-content'
Manually right-click on this div and select Copy -> Copy Element
This is not an effective solution since I'm seeking to automate this process.
How can I get this dynamically generated content out of the webpage in a programatic way?

Try to use below code to switch to iframe:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_xpath('//iframe[#realpage-oll-widget="RealPage-OLL-Widget"]')))
Also note that method that allows to switch to static iframe is switch_to.frame(), but not switch-to.frame()

You can not directly see the content which is in the iframe. You need to change frame. You can do this by firstly selecting 'iframe element' and then switching to it with driver.switch_to.frame() function.
iframe = driver.get_element_by_id('iframe')
driver.switch_to.frame(iframe)
After that you can access the iframe's content.
Alternatively, you can take the source attribute of iframe then going to that page with selenium. In the end, iframe content is another html page.

python selenium click on button

I am quite new to python selenium and I am trying to click on a button which has the following html structure:
<div class="b_div">
<div class="button c_button s_button" onclick="submitForm('mTF')">
<input class="very_small" type="button"></input>
<div class="s_image"></div>
<span>
Search
</span>
</div>
<div class="button c_button s_button" onclick="submitForm('rMTF')" style="margin-bottom: 30px;">
<input class="v_small" type="button"></input>
<span>
Reset
</span>
</div>
</div>
I would like to be able to click both the Search and Reset buttons above (obviously individually).
I have tried a couple of things, for example:
driver.find_element_by_css_selector('.button .c_button .s_button').click()
or,
driver.find_element_by_name('s_image').click()
or,
driver.find_element_by_class_name('s_image').click()
but, I seem to always end up with NoSuchElementException, for example:
selenium.common.exceptions.NoSuchElementException: Message: u'Unable to locate element: {"method":"name","selector":"s_image"}' ;
I am wondering if I can somehow use the onclick attributes of the HTML to make selenium click?
Any thoughts which can point me in the right direction would be great.
Thanks.

Remove space between classes in css selector:
driver.find_element_by_css_selector('.button .c_button .s_button').click()
# ^ ^
=>
driver.find_element_by_css_selector('.button.c_button.s_button').click()

try this:
download firefox, add the plugin "firebug" and "firepath"; after install them go to your webpage, start firebug and find the xpath of the element, it unique in the page so you can't make any mistake.
See picture:
browser.find_element_by_xpath('just copy and paste the Xpath').click()

For python, use the
from selenium.webdriver import ActionChains
and
ActionChains(browser).click(element).perform()

open a website https://adviserinfo.sec.gov/compilation and click on button to download the file and even i want to close the pop up if it comes using python selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.chrome.options import Options
#For Mac - If you use windows change the chromedriver location
chrome_path = '/usr/local/bin/chromedriver'
driver = webdriver.Chrome(chrome_path)
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-popup-blocking")
driver.maximize_window()
driver.get("https://adviserinfo.sec.gov/compilation")
# driver.get("https://adviserinfo.sec.gov/")
# tabName = driver.find_element_by_link_text("Investment Adviser Data")
# tabName.click()
time.sleep(3)
# report1 = driver.find_element_by_xpath("//div[#class='compilation-container ng-scope layout-column flex']//div[1]//div[1]//div[1]//div[2]//button[1]")
report1 = driver.find_element_by_xpath("//button[#analytics-label='IAPD - SEC Investment Adviser Report (GZIP)']")
# print(report1)
report1.click()
time.sleep(5)
driver.close()

I had the same problem using Phantomjs as browser, so I solved in the following way:
driver.find_element_by_css_selector('div.button.c_button.s_button').click()
Essentially I have added the name of the DIV tag into the quote.

The following debugging process helped me solve a similar issue.
with open("output_init.txt", "w") as text_file:
text_file.write(driver.page_source.encode('ascii','ignore'))
xpath1 = "the xpath of the link you want to click on"
destination_page_link = driver.find_element_by_xpath(xpath1)
destination_page_link.click()
with open("output_dest.txt", "w") as text_file:
text_file.write(driver.page_source.encode('ascii','ignore'))
You should then have two textfiles with the initial page you were on ('output_init.txt') and the page you were forwarded to after clicking the button ('output_dest.txt'). If they're the same, then yup, your code did not work. If they aren't, then your code worked, but you have another issue.
The issue for me seemed to be that the necessary javascript that transformed the content to produce my hook was not yet executed.
Your options as I see it:
Have the driver execute the javascript and then call your find
element code. Look for more detailed answers on this on
stackoverflow, as I didn't follow this approach.
Just find a comparable hook on the 'output_dest.txt' that will produce the same result, which is what I did.
Try waiting a bit before clicking anything:
xpath2 = "your xpath that you are going to click on"
WebDriverWait(driver, timeout=5).until(lambda x:
x.find_element_by_xpath(xpath2))
The xpath approach isn't necessarily better, I just prefer it, you can also use your selector approach.

I had the same problem and with Firefox, I got button element with the following steps:
right click button of interest and select "Inspect Accessibility Properties"
this opens the inspector. Right click the highlighted line and click "Print to JSON"
this opens a new tab. Look for nodeCssSelector and copy the value
This allowed me to accept cookies of the website Yahoo by using.
url = "https://yahoo.com"
driver = Firefox(executable_path="geckodriver.exe")
driver.get(url)
driver.find_element_by_css_selector("button.btn:nth-child(5)").click()
I tested this further and it allowed me to accept individual cookies with ease. Simply repeat the mentioned steps from before to get the button names.
url = "https://yahoo.com"
driver = Firefox(executable_path="geckodriver.exe")
driver.get(url)
driver.find_element_by_css_selector("a.btn").click()
driver.find_element_by_css_selector(".firstPartyAds > div:nth-child(2) > label:nth-child(1)").click()
driver.find_element_by_css_selector(".preciseGeolocation > div:nth-child(2) > label:nth-child(1)").click()
driver.find_element_by_css_selector("button.btn").click()
Another method is to
right click button of interest and select "Inspect"
right click the highlighted line and click "Copy -> CSS Selector" or whatever you need (there are multiple options, including XPath)
However, I think the second method may include whitespaces depending on what you copy, so you might need to manually remove (some of) them. The first method seems to be more foolproof, but I don't know if/how it works on other browsers than Firefox. The second method should work for all browsers.

Use This code To Click On Button
# finding the button using ID
button = driver.find_element_by_id(ID)
# clicking on the button
button.click()

e = driver.find_element(By.XPATH, 's_image').click()
sometime it does not work!
you can try:
e = driver.find_element(By.XPATH, 's_image') driver.execute_script("arguments[0].click();", e)