Develop Reference

Why is this Xpath not working with selenium?

I'm trying to input an email address to test logging in, but I'm continuing to receive this error: no such element: Unable to locate element:
I have tried using the relative and absolute Xpath and receive the same error message.
Forgive me as I'm sure missing something simple, very new to this!
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
import time
driver = webdriver.Chrome()
url = 'https://soundcloud.com/signin'
driver.get(url)
time.sleep(2)
driver.find_element_by_xpath('//*[#id="sign_in_up_email"]').send_keys('test#test.com')

The reason its throwing error is because the element sign_in_up_email is present inside the iframe
Refer image
Check the link here for detail about how to switch to iframe
You will first need to switch to iframe and then enter value in the input
Note:- When you first open the page you might see the accept cookies popup from soundcloud you will have to accept that
Your solution would look like
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
import time
driver = webdriver.Chrome()
url = 'https://soundcloud.com/signin'
driver.get(url)
driver.maximize_window()
time.sleep(2)
# Accept cookie
driver.find_element_by_id('onetrust-accept-btn-handler').click()
# Switch to frame
driver.switch_to.frame(driver.find_element_by_class_name("webAuthContainer__iframe"))
driver.find_element_by_xpath('//*[#id="sign_in_up_email"]').send_keys('test#test.com')

Password disappears when sent by selenium

Ok, To start, Everything here is working. My issue right now is when the password sends the keys it disappears immediately. It's happening because the password input itself erases the password when you click the input again. My question is, is there a workaround to get the password injected without it disappearing?
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from random import randint
import pickle
import datetime
import os
import time
url = 'https://sef.clareityiam.net/idp/login'
current_time = datetime.datetime.now()
current_time.strftime("%m/%d/%Y")
chromedriver = "chromedriver.exe"
driver = webdriver.Chrome(chromedriver)
driver.get(url)
pickle.dump(driver.get_cookies() , open("cookies.pkl","wb"))
user = driver.find_element_by_id("clareity")
user.send_keys("user")
password = driver.find_element_by_id("security")
password.send_keys("Password")
button = driver.find_element_by_id("loginbtn")
button.click()

Click this field before entering data. This works for me. Also, use at least implicit wait.
Another issue in your code may be that the id for password is not unique. There are two such locators.
import pickle
import time
from selenium import webdriver
url = 'https://sef.clareityiam.net/idp/login'
driver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')
driver.get(url)
driver.implicitly_wait(10)
pickle.dump(driver.get_cookies(), open("cookies.pkl","wb"))
user = driver.find_element_by_id("clareity")
user.send_keys("user")
password = driver.find_element_by_xpath('//div[#data-ph="PASSWORD"]')
password.click()
password.send_keys("Password")
time.sleep(6) # Added temporary so you could see that password stays
button = driver.find_element_by_id("loginbtn")
button.click()

Alternatively, to send keys in the input tag you can use the execute_script method with javascript. For example
driver.execute_script('document.getElementById("security").setAttribute("value", "12345")')
Then you can submit this form by clicking on the button or submit via javascript like this
driver.execute_script('document.querySelector("form").submit()')

I found somewhat of a work around for it. Idk why i didn't think about this while i was writing this script but.
if you send the passwords send_keys twice it stays. Don't understand why, but it works.

How to execute all javascript content on webpage with selenium to find and send login form info on fully loaded webpage

I've been trying to make a Python script to login into a certain website, navigate through the menu, fill out a form and save the file it generates to a folder.
I've been using Selenium trying to make the website fully load so i can find the elements for the login, but i'm being unsucessful, maybe because the website does a lot of JavaScript content before it fully loads, but i can't make it fully load and show me the data i want.
I tried Robobrowser, Selenium, Requests and BeautifulSoup to get it done.
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
url = "https://directa.natal.rn.gov.br/"
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
driver.get(url)
html = driver.execute_script("return document.documentElement.outerHTML")
sel_soup = BeautifulSoup(html, 'html.parser')
senha = driver.find_element_by_xpath('//*[#id="senha"]')
senha.send_keys("123")
I expected to have filled the password (senha) field with "123" but i can't even find the element.

It seems like what's needed here is a little bit of a scroll, wait and switch, incase the login fields just aren't ready for input :) The below should work, whereby we actually scroll to the element, having switch to the iframe, before we interact with the rest of the login form. You're able to adjust the delay from 5 seconds to anything of your preference.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
""" Variables """
url = "https://directa.natal.rn.gov.br/"
delay = 5 # seconds
""" Initiate driver """
driver = webdriver.Chrome(executable_path="C:\\webdrivers\\chromedriver.exe")
""" Go to url """
driver.get(url)
""" Iframe switch """
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
""" Attempt to get all our elements """
try:
""" Username """
usuario = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'usuario')))
""" Password """
senha = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'senha')))
print("All elements located!")
except TimeoutException:
print("Loading took too much time!")
exit(0)
"""Scroll to our element """
driver.execute_script("arguments[0].scrollIntoView();", usuario)
""" Input data into our fields """
usuario.send_keys("username")
senha.send_keys("password")
""" Locate our login element """
login = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'acessar')))
""" Click Login """
login.click()

To send the character sequence 123 to the password (senha) field, as the the desired elements are within a <frame> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://directa.natal.rn.gov.br/")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.input[name='usuario']"))).send_keys("Tads")
driver.find_element_by_css_selector("input.input[name='senha']").send_keys("123")
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe

Scraper doesn't stop clicking on the next page button

I've written a script in python in combination with selenium to get some names and corresponding addresses displayed upon a search and the search keyword is "Saskatoon". However, the data, in this case, traverse multiple pages. My script almost does everything except for one thing.
It still runs even though there are no more pages to traverse. The last page also holds ">" sign for next page option and is not grayed out.
Here is the link: Page_link
Search_keyword: Saskatoon (in the city/town field).
Here is what I've written:
from selenium import webdriver; import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("above_link")
time.sleep(3)
search_input = driver.find_element_by_id("cityField")
search_input.clear()
search_input.send_keys("Saskatoon")
search_input.send_keys(Keys.ENTER)
while True:
try:
wait.until(EC.visibility_of_element_located((By.LINK_TEXT, "›"))).click()
time.sleep(2)
except:
break
driver.quit()
BTW, I've just taken out the name and address part form this script which I suppose is not relevant here. Thanks.

You can use class attribute of > button as on last page it is "ng-scope disabled" while on rest pages - "ng-scope":
wait.until(EC.visibility_of_element_located((By.XPATH, "//li[#class='ng-scope']/a[.='›']"))).click()

Python Selenium - Webdriver interacts with data from previous page

I am trying to create a piece of code that takes an input from the user and searches for that input in his gmail account and then checks all the boxes from the sender which is the same as the input.
Here is my code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
from selenium.webdriver import Remote
from selenium.webdriver.support.ui import WebDriverWait
import urllib
driver = webdriver.Chrome('D:/chromedriver_win32/chromedriver.exe')
driver.get("http://www.gmail.com")
elem = driver.find_element_by_id('Email')
elem.send_keys("****************")
elem2 = driver.find_element_by_id('next')
elem2.send_keys(Keys.ENTER)
driver.maximize_window()
driver.implicitly_wait(20)
elem3 = driver.find_element_by_name('Passwd')
elem3.send_keys("*************")
driver.find_element_by_id('signIn').send_keys(Keys.ENTER)
driver.implicitly_wait(20)
inp = 'randominput'
driver.find_element_by_name('q').send_keys(inp)
driver.find_element_by_css_selector('button.gbqfb').send_keys(Keys.ENTER)
x = driver.current_url
for i in driver.find_elements_by_css_selector('.zA.zE'):
print(i.find_element_by_class_name('zF').get_attribute('name'))
if(i.find_element_by_class_name('zF').get_attribute('name') == inp):
i.find_element_by_css_selector('.oZ-jc.T-Jo.J-J5-Ji').click()
This main problem is that although the webdriver shows the new page where it has searched for the query but when the code interacts with the page it interacts with the previous one.
I have tried putting implicit wait. And when I check for the current url it shows the new url.

The problem is that this is a single page app so selenium isn't going to wait for new data to load. You need to figure out a way to wait for the search results to come back. Whether it be based off of the 'Loading...' that appears at the top or even just waiting for the first result to change.

Grab an element off of the first page and wait for it to go stale. That will tell you that the page is loading. Then wait for the element you want. This is about the only way to ensure that the page has reloaded and you aren't referencing the original page with a dynamic page.
To wait for stale, see staleness_of on http://selenium-python.readthedocs.io/waits.html.