Develop Reference

Python Selenium Script Having Trouble Finding Button on Modal

Our developer that does python left and I've been assigned the task of learning how to use the language. On top of that we can't find his source code so I'm having to rebuild the code from scratch. I have no experience in python and have spent all morning trying to find the answer to this. I realize there's 100 other questions similar to this and I promise I've looked through most of them.
I'm scraping a site and after logging in a modal is popping up forcing me to click a button to exit it. The button doesn't have an ID so I'm trying to find it by text. I've tried using XPATH, LINK_TEXT, and a couple of other things. Below is my code and some screenshots that I hope help identify what I'm doing wrong.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I think this is a temporary modal so I'm trying to set up the code to look for my "real" button first. If it can't find it then throw the exception and look for the modal button.
try:
srch = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.ID, "1744")))
srch.send_keys(Keys.ENTER)
#NoSuchElementException thrown if not present
except:
btn_text = "Don't ask again"
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[text()=\"" + btn_text + "\"]"))).click()
This is giving me the following error while debugging:
I'm trying to select the Don't ask again button to close the modal. Here's what I see when I inspect it:
Curiously though, if I just start by inspecting the modal "from the beginning", when I drill down to where the button is it's properties don't come up, this is as far down as it goes:
And this is where I'm stuck. Does anybody have any suggestions?
EDIT: I'm adding the entire error that I'm getting below:

Without seeing the real page entire HTML we can only guess, so as a first guess I would suggest instead of this
btn_text = "Don't ask again"
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[text()=\"" + btn_text + "\"]"))).click()
I'd try this:
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[contains(.,'ask again')]"))).click()

python selenium: clicking buttons with various html element/css attributes

i'm trying to create python automation scripts using selenium to make appointment times, however i'm having some trouble with clicking on certain elements. to confirm an appointment the steps are going to the webpage > selecting the date > selecting the time > hitting confirm
this is what i've got so far:
driver = webdriver.Chrome()
driver.get('<url.date>') -- this works
driver.find_element(By.ID,'corresponding time id').click() -- this works
driver.find_element(By.DATA-TEST-ID,'order_summary_page-button-book').click() -- this is where i'm struggling
i've also tried doing it by the button class, using the css selector and looking for 'Reserve Now' text, and using the xpath
when i do it by xpath, i get the error NoSuchElementException
i tried adding a webdriver wait function but that didn't seem to work either.
the appointment times are really hard to get, so i'd rather not but any wait times into the code so it can execute and book as soon as slots open
here are the elements:
https://i.stack.imgur.com/S62kR.png
any help is appreciated!

Selenium doesn't have option to identify the element by DATA-TEST-ID locator.
In this case You have to use either by xpath or css selector
driver.find_element(By.XPATH,'//button[#data-test-id="order_summary_page-button-book"]').click()
OR
driver.find_element(By.CSS_SELECTOR,'button[data-test-id="order_summary_page-button-book"]').click()
Ideally you should use explicit wait to click on the element.
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//button[#data-test-id="order_summary_page-button-book"]'))).click()
you need following imports.
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

What else can you do if the inspect element doesnt work?

I have been trying to get an account balance text to print in python using selenium. I go to chrome, inspect the element, copy the xpath, put it in VS and it says cant find the xpath. I did the same thing for CSS selector, full xpath, tried to create my own xpath, tried by class name, etc. Ive tried every option it has. nothing is working. What do you do when you run into this situation? It is there on the website, its gotta be accessible in the background somewhere. The has to be a way to grab it. I have been able to get other text on the site to display properly.
This is just a stock simulator game, I have it open to the public if you want to log in and try your hand at getting the correct inspect element.
The xpath it generates:
/html/body/div[4]/div[3]/div[1]/div[2]/ul/li[1]/span
How I have it in VS:
driver.find_element_by_xpath('/html/body/div[4]/div[3]/div[1]/div[2]/ul/li[1]/span')
The error in VS:
Unable to locate element: {"method":"xpath","selector":"/html/body/div[4]/div[3]/div[1]/div[2]/ul/li[1]/span"}

To get the amount Net Worth Use WebDriverWait() and wait for visibility_of_element_located() and following xpath.
print(WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//div[./span[text()='Net Worth']]/following::span[1]"))).text)
OR
print(WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//ul[#class='list list--kv']/li[1]/span"))).text)
You need to import below libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Automate Login with Selenium and Python

I am trying to use Selenium to log into a printer and conduct some tests. I really do not have much experience with this process and have found it somewhat confusing.
First, to get values that I may have wanted I opened the printers web page in Chrome and did a right click "view page source" - This turned out to be not helpful. From here I can only see a bunch of <script> tags that call some .js scripts. I am assuming this is a big part of my problem.
Next I selected the "inspect" option after right clicking. From here I can see the actual HTML that is loaded. I logged into the site and recorded the process in Chrome. With this I was able to identify the variables which contain the Username and Password. I went to this part of the HTML did a right click and copied the Xpath. I then tried to use the Selenium find_element_by_xpath but still no luck. I have tried all the other methods to (find by ID, and name) however it returns an error that the element is not found.
I feel like there is something fundamental here that I am not understanding. Does anyone have any experience with this???
Note: I am using Python 3.7 and Selenium, however I am not opposed to trying something other than Selenium if there is a more graceful way to accomplish this.
My code looks something like this:
EDIT
Here is my updated code - I can confirm this is not just a time/wait issue. I have managed to successfully grab the first two outer elements but the second I go deeper it errors out.
def sel_test():
chromeOptions = Options()
chromeOptions.add_experimental_option("useAutomationExtension", False)
browser = webdriver.Chrome(chrome_options=chromeOptions)
url = 'http://<ip address>/'
browser.get(url)
try:
element = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="ccrx-root"]')))
finally: browser.quit()
The element that I want is buried in this tag - Maybe this has something to do with it? Maybe related to this post
<frame name="wlmframe" src="../startwlm/Start_Wlm.htm?arg11=">

As mentioned in this post you can only work with the current frame which is seen. You need to tell selenium to switch frames in order to access child frames.
For example:
browser.switch_to.frame('wlmframe')
This will then load the nested content so you can access the children

Your issue is most likely do to either the element not loading on the page until after your bot searches for it, or a pop-up changing the xpath of the element.
Try this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
delay = 3 # seconds
try:
elementUsername = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.xpath, 'element-xpath')))
element.send_keys('your username')
except TimeoutException:
print("Loading took too much time!")
you can find out more about this here

Finding xpaths on pages running script

Im trying to scrape a webpage using selenium. The xpaths suggested by inspecting the page and right clicking are of an unstable kind (/html/body/table[2]/tbody/tr[1]/td/form/table/tbody/tr[2]) . So I tried the following solution instead:
driver = webdriver.Chrome("path")
driver.get("https://www.bundesfinanzhof.de/entscheidungen/entscheidungen-online")
time.sleep(1)
links=driver.find_element_by_xpath('//tr[#class="SuchForm"]')
or even
links=driver.find_elements_by_xpath('//*[#class="SuchForm"]')
don't return any results. However earlier on in the page I can obtain:
links=driver.find_element_by_xpath('//iframe')
links.get_attribute('src')
It seems that after:
<script language="JavaScript" src="/rechtsprechung/jscript/list.js" type="text/javascript"></script>
I can no longer get to any of the elements.
How do I determine the correct XPath?
suggests that parts within a script are impossible to parse. However, the path I am after seems to me not to be within a path. Am I misinterpretting how scripts work on a page ?
For instance, later on there is a path:
/html/body/table[2]/tbody/tr[1]/td/script
I would expect this to create such a problem. I am by no means a programmer, so my understanding of this subject is limited. Can someone explain what the problem is and if possible a solution ?
Attempted using solutions from:
Find element text using xpath in selenium-python NOt Working
xpath does not work with this site, pls verify

The table is located inside an iframe, so you need to switch to that iframe before handling required tr:
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver.get("https://www.bundesfinanzhof.de/entscheidungen/entscheidungen-online")
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#src='https://juris.bundesfinanzhof.de/cgi-bin/rechtsprechung/list.py?Gericht=bfh&Art=en']")))
link = driver.find_element_by_xpath('//tr[#class="SuchForm"]')
Use driver.switch_to.default_content() to switch back from iframe