Develop Reference

Selenium cant find element on Javascript page

I am still quite new to python and selenium, However have managed to get quite far with what I am doing. But I appear to now be stuck. The page in question is an internal business page. I have tried using ID, name and XPATH with very little success.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
import time
PATH = r"C:\Users\p819364\Downloads\chromedriver.exe"
options = webdriver.ChromeOptions()
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--ignore-certificate-errors')
driver = webdriver.Chrome(PATH, options=options)
driver.get("https://10.47.31.102/3/Login")
driver.implicitly_wait(15)
print (driver.title)
username = driver.find_element(By.NAME, value='j_username')
username.send_keys("username")
password = driver.find_element(By.NAME, value='j_password')
password.send_keys("password")
password.send_keys(Keys.RETURN)
driver.implicitly_wait(20)
Settings = driver.find_element(By.XPATH, value="//*[#id='evo_widget_TBFisheyeItem_4']")
Settings.click()
driver.implicitly_wait(20)
Filter = driver.find_element(By.XPATH, value="//*[#id='tableForm:authenticationPolicyTable:tableActions']")
driver.implicitly_wait(20)
Filter.click()
The problem I am having is with the filter, I think it may be because this is a page within a page. I am sorry I cannot share the page as its internal. But I need to be able to click the filter and click and options
I keep getting the following error
Message: no such element; Unable to locate element: {"method":"css selector","selector":["//*[#id='tableForm:authenticationPolicyTable:tableActions']"
The XPATH is as follows:
//*[#id="tableForm:authenticationPolicyTable:tableActions"]
This is the full XPATH (which I have tried):
/html/body/form[3]/table/thead/tr[1]/th/table/tbody/tr/td[2]/select
I appreciate any help given and I also am not sure if its caused this because the field I want to select is not in view until I scroll. However I cannot seem to scroll as the element is within another as pictured
Thanks
Edit:
The iFrame was stopping it i had to switch to it first
iframe = driver.find_element_by_xpath("//*[#id='consoleCanvas']")
driver.switch_to.frame(iframe)

I think if the page is within page then you need to check if that element is within iframe. If iframe is there then you should first switch to frame and then click on filter. If you can post html code then it will be easy to understand issue.

Selenium: no such element: Unable to locate element:

I have a question about Selenium.
My idea:
My idea is to make a Python script that logs in to this website.
Selenium sends the username and password to the HTML input field and submits it.
Problem:
My code keeps saying:
Message: no such element: Unable to locate element:
I have tried this code with google.com for example and that works.
Why is this not working with this login page?
Can anybody help me please?
My Python code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
login_URL = ""
driver = webdriver.Chrome()
driver.get(login_URL)
time.sleep(5)
inputElement = driver.find_element_by_name('uname')
inputElement.send_keys(username)
time.sleep(20)
driver.close()

i don't know how it works in pyton, im use js
but try to use xpath
driver.find_element_by_xpath('your xpath') [maybe use 1 more click) ('xpath element').click() - element is active.
and after use send keys
******.send_keys('username')
driver.switchTo().frame(driver.findElement(By.xpath('xpath frame'))) - in js

As already explained, the element is in an iframe. Need to switch to frame to interact with the element.
It would be better apply Explicit waits.
# Imports required:
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver.get("https://nl.infothek-sptk.com/isps/infothek/?1043")
wait = WebDriverWait(driver,30)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"body_frame")))
wait.until(EC.element_to_be_clickable((By.NAME,"uname"))).send_keys("username#mail.com")
# Code to enter other fields.
# Switch back to default to interact with elements outside the iframe.
driver.switch_to.default_content()

Unable to Populate Search Field in Python Selenium

I am trying to search www.oddschecker.com using Python and Selenium (chromedriver) but am struggling to populate the search field with the search term.
The code below clicks the magnifying glass ok but the send_keys statement does not populate the expanded search box.
Can anyone see the issue?
driver.find_element_by_xpath('//*[#id="top-menu"]/li[8]/ul/li[1]/span/span').click()
sleep(5)
driver.find_element_by_xpath('/html/body/div[1]/header/div[1]/div/div[1]/div/ul/li[8]/ul/li[1]/div/div/div/form/fieldset/input').send_keys('Bicep')

xpath is brittle, please use relative xpath. see below.
Use explicit waits.
Close modal pop up may appear sometime, may not appear sometime so better to wrap them inside try and except block.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
driver.get("http://www.oddschecker.com/")
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.js-close-class"))).click()
print('Clicked on close button if it appears')
except:
pass
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[starts-with(#data-ng-click,'NavSearchCtrl')]"))).click()
search = wait.until(EC.visibility_of_element_located((By.ID, "search-input")))
search.send_keys('Bicep', Keys.RETURN)
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Selenium: Element Click Intercepted while submitting a form

So I'm trying to submit a form but something is either preventing me from accessing the box or I'm using a wrong element but I think I'm using the correct one.
Here is my code:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path = 'mypath/chromedriver.exe')
driver.maximize_window()
#driver.implicitly_wait(50)
driver.get("https://ai.fmcsa.dot.gov/SMS")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Close']"))).click()
wait = WebDriverWait(driver, 20)
driver.find_element_by_xpath('//*[#id="home-body"]/div[1]/div/div[1]/form/label').click()
driver.find_element_by_xpath('//*[#id="home-body"]/div[1]/div/div[1]/form/label').send_keys('1818437')
driver.find_element_by_xpath('/html/body/div[3]/div[2]/article/section[2]/div[1]/div/div[1]/form/input[2]').click();
What I'm getting on the output is
ElementClickInterceptedException: Message: element click intercepted:
Element ... is
not clickable at point (553, 728). Other element would receive the
click:
(Session info: chrome=93.0.4577.63)
What might be the issue?

Things to noted down in this scenario :-
When you define an explicit waits wait = WebDriverWait(driver, 20), you can always use wait reference in the scope. you do not need to create again and again in same class.
Try to avoid absolute xpath /html/body/div[3]/div[2]/article/section[2]/div[1]/div/div[1]/form/input[2], try with relative xpath/xpath axes.
When we try to send keys to some element, in general it should be a input tag, not label
You may have to scroll, may be not in this case but when you scroll manually to interact with elements in UI, same has to automated with Selenium as well.
Also I observed to this webapp that search and input tags are duplicated, so I have used xpath indexing [2] to handle.
Sample code :-
driver = webdriver.Chrome(executable_path = 'mypath/chromedriver.exe')
driver.maximize_window()
#driver.implicitly_wait(50)
driver.get("https://ai.fmcsa.dot.gov/SMS")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Close']"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "(//input[#name='MCSearch'])[2]"))).send_keys('1818437')
wait.until(EC.element_to_be_clickable((By.XPATH, "(//input[#name='search'])[2]"))).click()

You can use below xpaths too.
driver.find_element_by_xpath("//div[#class='sms-search-box']//input[1]").send_keys('1818437')
driver.find_element_by_xpath("//div[#class='sms-search-box']//input[2]").click()

Xpath you are using is not right. Your xpath for the input field should be like this.
driver.find_element_by_xpath("//input[#name='MCSearch' and #placeholder='Type Name or U.S. DOT#']").send_keys("1818437")
driver.find_element_by_xpath("//input[#placeholder='Type Name or U.S. DOT#']//following::input[#value='Search']").click();

How to execute all javascript content on webpage with selenium to find and send login form info on fully loaded webpage

I've been trying to make a Python script to login into a certain website, navigate through the menu, fill out a form and save the file it generates to a folder.
I've been using Selenium trying to make the website fully load so i can find the elements for the login, but i'm being unsucessful, maybe because the website does a lot of JavaScript content before it fully loads, but i can't make it fully load and show me the data i want.
I tried Robobrowser, Selenium, Requests and BeautifulSoup to get it done.
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
url = "https://directa.natal.rn.gov.br/"
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
driver.get(url)
html = driver.execute_script("return document.documentElement.outerHTML")
sel_soup = BeautifulSoup(html, 'html.parser')
senha = driver.find_element_by_xpath('//*[#id="senha"]')
senha.send_keys("123")
I expected to have filled the password (senha) field with "123" but i can't even find the element.

It seems like what's needed here is a little bit of a scroll, wait and switch, incase the login fields just aren't ready for input :) The below should work, whereby we actually scroll to the element, having switch to the iframe, before we interact with the rest of the login form. You're able to adjust the delay from 5 seconds to anything of your preference.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
""" Variables """
url = "https://directa.natal.rn.gov.br/"
delay = 5 # seconds
""" Initiate driver """
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
""" Go to url """
driver.get(url)
""" Iframe switch """
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
""" Attempt to get all our elements """
try:
""" Username """
usuario = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'usuario')))
""" Password """
senha = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'senha')))
print("All elements located!")
except TimeoutException:
print("Loading took too much time!")
exit(0)
"""Scroll to our element """
driver.execute_script("arguments[0].scrollIntoView();", usuario)
""" Input data into our fields """
usuario.send_keys("username")
senha.send_keys("password")
""" Locate our login element """
login = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'acessar')))
""" Click Login """
login.click()

To send the character sequence 123 to the password (senha) field, as the the desired elements are within a <frame> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://directa.natal.rn.gov.br/")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.input[name='usuario']"))).send_keys("Tads")
driver.find_element_by_css_selector("input.input[name='senha']").send_keys("123")
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe