When I run the below code with headless chrome I'm facing issues while identifying the elements. The same code runs just runs fine by commenting the below lines with head full mode.
# chrome_options.add_argument('--headless')
# chrome_options.add_argument('--disable-gpu')
Test Details:
Operating system: Windows10
ChromeDriver: 75.0.3770.8
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()
Output:
"C:\Program Files (x86)\Python37-32\python.exe" C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py
Checking for win32 chromedriver:75.0.3770.8 in cache
Driver found in C:\Users\vishr\.wdm\chromedriver\75.0.3770.8\win32/chromedriver.exe
Home | Let's Kode It
Traceback (most recent call last):
File "C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py", line 15, in <module>
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
File "C:\Users\vishr\AppData\Roaming\Python\Python37\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Process finished with exit code 1
for headless browser you have to set the window size to fire on event.Because headless browser can't recognise where to click without window size.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('window-size=1920x1080')
driver = webdriver.Chrome(executable_path='path/to/chrome driver',options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()
Printed output on console on headless mode.
Home | Let's Kode It
Let's Kode It
You're clicking a button that loads another page.
You have to wait for the other page to load. This is tricky to get perfect.
The easiest solution:
sleep(3)
The better solution using implicit waiting:
driver.implicitly_wait(3)
An even better solution using explicit waiting:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.ID, "user_email")))
An even better better solution would be to catch exceptions, and layer all these solutions.
To click() on the element with text as Login you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following solutions:
Using PARTIAL_LINK_TEXT:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Login"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.navbar-link.fedora-navbar-link[href='/sign_in']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='navbar-link fedora-navbar-link' and #href='/sign_in']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
VISHVAMBRUTHJAVAGALTHIMMEGOWDA,
I tried your code and was getting the same exception, First I thought that username was in Frame or iframe but it is not.
Then I tried to introduce webdriver wait and it worked just fine :
Code :
wait = WebDriverWait(driver,10)
driver.maximize_window()
driver.get("https://learn.letskodeit.com/")
print(driver.title)
driver.find_element_by_xpath("//a[contains(text(),'Login')]").click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
#driver.find_element_by_id("user_email").
driver.find_element_by_id("user_password").send_keys("abcabc")
driver.find_element_by_name("commit").click()
print(driver.title)
Remember to imports these :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
EDIT 1 :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu') # Last I checked this was necessary.
driver = webdriver.Chrome("C:/Users/XXXX/Downloads/BrowserDriver/chromedriver_win32/chromedriver.exe", chrome_options=options)
wait = WebDriverWait(driver,10)
driver.get("https://learn.letskodeit.com/")
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
Related
This is my first attempt to login a website using selenium, i have written the below piece of code but not able to login the website, getting the following error message.
Message: no such element: Unable to locate element: {"method":"css selector","selector":"[name="txtUsername"]"}
(Session info: chrome=108.0.5359.95)
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
s=Service("C:\Program Files\Google\Chrome\Application\chromedriver.exe")
driver = webdriver.Chrome(service=s)
driver.maximize_window()
driver.get("https://opensource-demo.orangehrmlive.com")
wait = WebDriverWait(driver, 10)
driver.find_element(By.NAME,"txtUsername").send_keys("Admin")
driver.find_element(By.ID, "txtPassword").send_keys("admin123")
driver.quit()
Looking for some suggestions
There are 2 problems with your code:
You created the wait object but not used it...
Your locators are wrong.
The following code works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
wait = WebDriverWait(driver, 20)
url = "https://opensource-demo.orangehrmlive.com"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.NAME, "username"))).send_keys("Admin")
wait.until(EC.element_to_be_clickable((By.NAME, "password"))).send_keys("admin123")
wait.until(EC.element_to_be_clickable((By.TAG_NAME, "button"))).click()
The wait needs to be called:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
s=Service("C:\Program Files\Google\Chrome\Application\chromedriver.exe")
driver = webdriver.Chrome(service=s)
driver.maximize_window()
driver.get("https://opensource-demo.orangehrmlive.com")
wait = WebDriverWait(driver, 10)
user = wait.until(EC.element_to_be_clickable((By.XPATH, "//div")))
user.send_keys("Admin")
driver.find_element(By.ID, "txtPassword").send_keys("admin123")
driver.quit()
https://www.espncricinfo.com/player/aamer-jamal-793441
This is the URL and here i am trying to access Full Name "Aamer Jamal". with the help of selenium web driver. But I dont know why it gives
NoSuchElementException
`the code is written below:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium_firefox import Firefox
import time
import pandas as pd
driver = webdriver.Firefox()
#Reach to the Landing page
driver.get('https://www.espncricinfo.com/player/aamer-jamal-793441')
driver.maximize_window()
time.sleep(25)
not_now = driver.find_element(By.ID, 'wzrk-cancel')
not_now.click()
fullname = driver.find_element(By.XPATH, '/html/body/div[1]/section/section/div[4]/div[2]/div/div[1]/div/div/div[1]/div[1]/span/h5')
print(fullname.text)`
Error :
NoSuchElementException: Message: Unable to locate element: /html/body/div[1]/section/section/div[4]/div[2]/div/div[1]/div/div/div[1]/div[1]/span
You have to use WebDriverWait expected_conditions explicit waits, not a long hardcoded pauses. You also have to learn how to create correct locators. Long absolute XPaths and CSS Selectors are extremely breakable. The following code works:
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 60)
actions = ActionChains(driver)
url = "https://www.espncricinfo.com/player/aamer-jamal-793441"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.ID, 'wzrk-cancel')))
fullname = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'ds-text-title-l'))).text
print(fullname)
The output is:
Aamer Jamal
Hello I'm newbie of selenium.
and I got the problem. I want to scrap using XPATH and not working.
from __future__ import print_function
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver', options=chrome_options)
driver.set_window_size(1320, 550)
exchange_link = "https://icodrops.com/category/ended-ico/"
driver.get(exchange_link)
wait = WebDriverWait(driver, 30)
ico_links = driver.find_element(By.XPATH, "//div[#id='all']//div[#id='ajaxc']")
But the result is this
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#id='all']//div[#id='ajaxc']"}(Session info: headless chrome=97.0.4692.71)
I don`t know why it cant be work.
I just did by XPath.
image description here
my code link is this,
Scrap page is this
You are missing a wait / delay.
You can use expected conditions to wait for that element presence.
As following:
from __future__ import print_function
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver', options=chrome_options)
driver.set_window_size(1320, 550)
exchange_link = "https://icodrops.com/category/ended-ico/"
driver.get(exchange_link)
wait = WebDriverWait(driver, 30)
wait.until(EC.visibility_of_element_located((By.XPATH, "//h3/a[#rel]")))
time.sleep(1)
ico_links= driver.find_element_by_xpath('//h3/a[#rel]')
I've been trying to make a Python script to login into a certain website, navigate through the menu, fill out a form and save the file it generates to a folder.
I am having some problems to submit the form.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
options.add_argument("--disable-extensions")
browser = webdriver.Chrome()
browser.get("https://directa.natal.rn.gov.br/")
WebDriverWait(browser, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.input[name='usuario']"))).send_keys("11844691000126")
browser.find_element_by_css_selector("input.input[name='senha']").send_keys("Link2007")
browser.find_element_by_css_selector("button.btn[name='acessar']").click()
Once you fill up the required fields then to submit the form you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
# options.add_argument('disable-infobars')
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://directa.natal.rn.gov.br/")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame[name='mainsystem'][src^='main']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.input[name='usuario']"))).send_keys("11844691000126")
driver.find_element_by_css_selector("input.input[name='senha']").send_keys("Link2007")
driver.find_element_by_css_selector("button.btn#acessar").click()
Browser Snapshot:
You have to switch to the iframe, because your element is not in the main website, is inside iframe.
iframes = browser.find_elements_by_xpath("//iframe[#name='mainsystem']")
print("Iframes: %s" %len(iframes))
browser.switch_to_frame(iframes[0])
Headless Chrome does not load the page and gets stuck at:
wait = WebDriverWait(driver, 30)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".price-link:nth-child(1) .team-name")))
Why is it doing this? Is this a bug as it works perfectly in normal chrome and prints h below except in headless...
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("window-size=1400,600")
driver = webdriver.Chrome(chrome_options=options)
#driver = webdriver.Chrome()
driver.set_window_size(1024, 600)
driver.maximize_window()
driver.get('https://www.sportsbet.com.au/betting/soccer?LeftNav')
print('?')
wait = WebDriverWait(driver, 30)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".price-link:nth-child(1) .team-name")))
print('h')
Another example:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("window-size=1400,600")
driver = webdriver.Chrome(chrome_options=options)
#driver = webdriver.Chrome()
driver.set_window_size(1024, 600)
driver.maximize_window()
import time
driver.get('https://www.sportsbet.com.au/betting/soccer?LeftNav')
import time
time.sleep(10)
langs = driver.find_elements_by_css_selector(".price-link:nth-child(1) .team-name")
langs_text = []
for lang in langs:
print(lang.text)
langs_text.append(lang.text)
print('h')
Job prints with removed wait until though nothing is scraped. Page not loading is likely issue.
Try to use the following code:
element = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".price-link:nth-child(1) .team-name")))
driver.execute_script("arguments[0].scrollIntoView(true);", element)
Hope it helps you!