I'm working on making an automated web scraper run on Khan Academy to make offline backups of questions using selenium and a python scraper (to come later). I'm currently working on getting it to answer questions (right or wrong doesn't matter) to proceed through exercises. Unfortunately, selenium's .click() function doesn't actually select an answer. I think it has something to do with being pointed at the wrong object but I can't tell. It currently highlights the option, but doesn't select it.
HTML for a single option (out of 4)
I made some code to reproduce the error, and hooked it up to a test account for all your debugging needs. Thanks.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
# gets us to the SAT Math Exercise page
driver.get('https://www.khanacademy.org/mission/sat/tasks/5505307403747328')
# these next lines just automate logging in. Nothing special.
login = driver.find_element_by_name('identifier')
login.send_keys('stackflowtest') # look, I made a new account just for you guys
passw = driver.find_element_by_name('password')
passw.send_keys('stackoverflow')
button = driver.find_elements_by_xpath("//*[contains(text(), 'Sign in')]")
button[1].click()
driver.implicitly_wait(5) # wait for things to become visible
radio = driver.find_element_by_class_name('perseus-radio-option')
radio.click()
check = driver.find_element_by_xpath("//*[contains(text(), 'Check answer')]")
check.click()
After a trial and error process, I found that the actual click selection can be accomplished by pointing to the element with class "description"
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
# gets us to the SAT Math Exercise page
driver.get('https://www.khanacademy.org/mission/sat/tasks/5505307403747328')
# these next lines just automate logging in. Nothing special.
login = driver.find_element_by_name('identifier')
login.send_keys('stackflowtest') # look, I made a new account just for you guys
passw = driver.find_element_by_name('password')
passw.send_keys('stackoverflow')
button = driver.find_elements_by_xpath("//*[contains(text(), 'Sign in')]")
button[1].click()
driver.implicitly_wait(5) # wait for things to become visible
radio = driver.find_element_by_class_name('description')
radio.click()
check = driver.find_element_by_xpath("//*[contains(text(), 'Check answer')]")
check.click()
For people dealing with similar issues, I would recommend clicking on the very edge of the space that allows you to select and inspecting the element there. This prevents you from accidentally using one of the innermost tags.
Related
I am trying to make an automatic program for logging in to GitHub. I could find only the sign-in option. After that, I could not find the Username field. I have confirmed that the element is definitely not in a/an (i)frame. I have tried an alternative with css-selector.
Here is the code I tried:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
chrome_opt = Options()
chrome_opt.add_experimental_option("detach", True) # type: ignore[unknown]
auto = Chrome(options=chrome_opt)
auto.get("https://github.com")
signin_link = auto.find_element("link text", "Sign in")
signin_link.click()
username = auto.find_element("id", "login_field")
username.send_keys("ArnabRollin") # type: ignore[unknown]
# FIXME
The type-ignore comments are there because of 'strict mode' type checking in VS Code. Also, after 5 tries of running it, it finally worked, but when I ran it again it didn't.
now your code is looking for elements in the page https://github.com - the one used in method get()
instead of clicking element's link, get it with webdriver:
signin_link = auto.find_element("link text", "Sign in")
signin_link.click()
use
signin_link = auto.find_element("link text", "Sign in").get_attribute('href')
auto.get(signin_link)
auto.get(url2) will save new page context into driver. after sign in is complete, a new page context will be needed
Note, I'm not sure it's ethical scraping this website, and besides, they have Captcha.
You can use this CSS selector:
username = auto.find_element(By.CSS_SELECTOR, "input.js-login-field")
Additionally, when you go to github.com and click on login, the URL changes to /login: https://github.com/login
I've gone through a number of similar topics here but they all seem to vary in how the pop-up window is designed. I've tried a few different ways and here is the most recent. So before I enter the login info, I need to click that client login button to access the login form but I can't even get it to open, let alone entering login information.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome('C:\Login Automation\chromedriver.exe')
driver.get("https://www.datamyne.com/ ")
clientlogin = driver.find_element_by_xpath("//div[#id='holder']").click()
username = driver.find_element_by_xpath("//*[#id='user']").send_keys('myusername')
password = driver.find_element_by_xpath("//*[#id='pass']").send_keys('mypassword')
the error I'm getting here is "NoSuchWindowException: Message: no such window: target window already closed from unknown error: web view not found"
the element of the first button is this:
<a style="position: relative" href="javascript:showHide('dialog-login');" class="green-btn user-login top-right-5">Client Login</a>
and then the actual login button is another javascript line:
Login
Any tips as to how to approach this would be really appreciated!
I have inspect mention website login Form based on JavaScript. You can easily execute script through selenium. I have create a basic code snippet for you.
from selenium import webdriver
driver = webdriver.Chrome('chromedriver.exe')
driver.get("https://www.datamyne.com/")
##Javascript script execute using selenium
clientlogin = driver.execute_script("javascript:showHide('dialog-login');")
driver.implicitly_wait(5)
username = driver.find_element_by_xpath('//*[#id="User"]').send_keys('myusername')
password = driver.find_element_by_xpath('//*[#id="Pass"]').send_keys('mypassword')
save = driver.find_element_by_xpath('//*[#id="formLoginDM"]/div[1]/a').click()
clientlogin = driver.find_element_by_xpath("//div[#id='holder']").click()
The xpath is off. The holder isn't what you need to click.
I suspect you want:
clientlogin = driver.find_element_by_xpath("//a[text()='Client Login']").click()
Can you try this xpath once for the 'Client Login' pop-up modal
clientlogin = driver.find_element_by_xpath("//a[#class='green-btn user-login top-right-5']']").click()
I am new to Selenium and I am trying to mimic user actions on a site to fetch data from a built in html page on button click. I am able to populate all the field details, but button click is not working, it looks like js code not running.
I tried many options like adding wait time, Action chain etc but it didnt work, i am providing site and code i have written.
driver = webdriver.Chrome()
driver.get("https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm")
driver.implicitly_wait(10)
assigned values to all the other fields
driver.find_element_by_id('rdDateToDate').click()
Dfrom = driver.find_element_by_id('fromDate')
Dfrom.send_keys("02-Oct-2020")
Dto = driver.find_element_by_id('toDate')
Dto.send_keys("08-Oct-2020")
innerHTML = driver.execute_script("document.ready")
sleep(5)
getdata_btn = driver.find_element_by_id('getButton')
ActionChains(driver).move_to_element(getdata_btn).click().click().perform()
I recommend using a full xpath.
chrome.get("https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm")
time.sleep(2)
print("click")
fullxpath = "/html/body/div[2]/div[3]/div[2]/div[1]/div[3]/div/div[1]/div/form/div[19]/p[2]/input"
chrome.find_element_by_xpath(fullxpath).click()
I have tried the button clicking and it worked with XPath ... I though its because of someone used the ID twice on a website, but I can not find it ... so i have no idea whats going wrong there ...
Good luck :)
I am trying to automate the process for logging into Grofers - an Indian eCommerce grocery market.
My code is:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Firefox()
time.sleep(1)
driver.get('https://grofers.com/')
time.sleep(3)
login = driver.find_element_by_class_name('account-wrapper')
login.click()
time.sleep(3)
mobileNo = driver.find_element_by_xpath('//*[#id="phone-no-text-box"]')
mobileNo.send_keys('<Phone Number>')
time.sleep(3)
cont = driver.find_element_by_xpath('//*[#id="login-next-button"]')
cont.click()
When I run this script, I get the this error:
elenium.common.exceptions.ElementClickInterceptedException: Message: Element <div class="account-wrapper">
is not clickable at point (1136,57) because another element <div class="location__overlay"> obscures it
Couldn't get past it. Can anybody help?
Edit: I solved this issue. If anyone is facing the same problem, they can let me know, I'll share my code snippet.
The location selector on the page requires you to select a location before it closes.
Adding this code after the first time.sleep(3) should solve the problem, it just finds and selects the default location, Gurugram.
locationSelecter = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div[2]/header/div[2]/div[2]/div/div/div/div[1]/div[2]")
locationSelecter.click()
time.sleep(10)
I'm trying to automate a duolingo login with Selenium with the code posted below.
While everything seems to work as expected at first, I always get an "Wrong password" message on the website after the login button is clicked.
I have checked the password time and time again and even changed it to one without special characters, but still the login fails.
I have seen in other examples that there is sometimes an additional password input field, however I cannot find one while inspecting the html.
What could I be missing ?
(Side note: I'm also open to a completely different solution without a webdriver since I really only want to get to the duolingo.com/learn page to scrape some data, but as of yet I haven't found an alternative way to login)
The code used:
from selenium import webdriver
from time import sleep
url = "https://www.duolingo.com/"
def login():
driver = webdriver.Chrome()
driver.get(url)
sleep(2)
hve_acnt_btn = driver.find_element_by_xpath("/html/body/div/div/div/span[1]/div/div[1]/div[2]/div/div[2]/a")
hve_acnt_btn.click()
sleep(2)
email_input = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[2]/form/div[1]/div/label[1]/div/input")
email_input.send_keys("email#email.com")
sleep(2)
pwd_input = driver.find_element_by_css_selector("input[type=password]")
pwd_input.clear()
pwd_input.send_keys("password")
sleep(2)
login_btn = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[2]/form/div[1]/button")
login_btn.click()
sleep(5)
login()
I couldn't post the website's html because of the character limit, so here is the link to the duolingo page: Duolingo
Switch to Firefox or a browser which does not tell the page that you are visiting it automated. See my earlier answer for a very similar issue here: https://stackoverflow.com/a/57778034/8375783
Long story short: When you start Chrome it will run with navigator.webdriver=true. You can check it in console. Pages can detect that flag and block login or other actions, hence the invalid login. This is a read-only flag set by the browser during startup.
With Chrome I couldn't log in to Duolingo either. After I switched the driver to Firefox, the very same code just worked.
Also if I may recommend, try to use Xpath with attributes.
Instead of this:
hve_acnt_btn = driver.find_element_by_xpath("/html/body/div/div/div/span[1]/div/div[1]/div[2]/div/div[2]/a")
You can use:
hve_acnt_btn = driver.find_element_by_xpath('//*[#data-test="have-account"]')
Same goes for:
email_input = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[2]/form/div[1]/div/label[1]/div/input")
vs:
email_input = driver.find_element_by_xpath('//input[#data-test="email-input"]')