Develop Reference

Selecting jquery dropdown list using XPATH

Actually I am doing tasks from https://demo.seleniumeasy.com/jquery-dropdown-search-demo.html. But I found a problem - I can't find any element on this page using XPATH. For example I want to find "Select Country" using driver.find_element and XPATH:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("https://demo.seleniumeasy.com/jquery-dropdown-search-demo.html")
jquery_drop_list = driver.find_element(by=By.XPATH, value="//span[#class='select2-selection select2-selection--single']")
#jquery_drop_list = driver.find_element(by=By.XPATH, value="//span[#class='select2 select2-#container select2-container--default select2-container--above select2-container--focus']")
#jquery_drop_list = driver.find_element(by=By.XPATH, value="//span[#class='select2-hidden-#accessible']")
print(jquery_drop_list)
But none of the above searches works.
Could you advise me on what a proper selector should look like for similar problems? Maybe XPATH selector is not a good choice here?

There is a Select block here.
You need to utilize Selenium Select object for that.
This code is selecting Denmark:
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
url = "https://demo.seleniumeasy.com/jquery-dropdown-search-demo.html"
driver.get(url)
select_country = Select(wait.until(EC.element_to_be_clickable((By.ID, 'country'))))
select_country.select_by_value("Denmark")
But if you still want to open that drop down with regular click it is possible too. This XPath works:
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
url = "https://demo.seleniumeasy.com/jquery-dropdown-search-demo.html"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[#aria-labelledby='select2-country-container']"))).click()
Generally, XPath is most powerful way to select web elements with selenium.
Some people just not familiar with it :)
And sometimes some XPaths not properly supported by some webdrivers, but if you are using Chromedriver you will see no problems with XPaths.

Hover the Mouse to the tab and scrape the products

Hello I am trying to use selenium to automatically scrape the products titles and prices, i am using ActionChains and move_to_element, but somehow it gave me timeout exception, Is there a better way to do it?
titles in the tab
https://denago.com/collections/ebikes
#For Dynamic webpage, import selenium
!apt-get update
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
!pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#set up Chrome driver
options=webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
#Define web driver as a Chrome driver
driver=webdriver.Chrome('chromedriver',options=options)
driver.implicitly_wait(10)
driver.get('https://denago.com/collections/ebikes')
action = ActionChains(driver)
ourbike = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[6]/div/header/nav/ul/li[1]/a/span")))
ActionChains(driver).move_to_element(ourbike).perform()
Titles=driver.find_elements(By.CLASS_NAME,'mm-title')
for i in range(len(Titles)):
print(Titles[i].text)

There are a couple of problems:
Browser opens at the default size which is small and the element you are searching for to hover is absent on the page. So you need to set options.add_argument('window-size=1200,1980').
There is a message about cookies that overlaps elements on the page. It's better to close it: driver.find_element(By.ID, 'CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll').click()
The element you tried to hover could not be found by the XPATH you used. It can easily be found with (By.XPATH, '(//li[#itemid="m9RVB"])'), but there are two such elements on the page and the first is hidden. So you need to hover the second one, so add [2] to the locator: (By.XPATH, '(//li[#itemid="m9RVB"])[2]')
So, here is the code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#set up Chrome driver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('window-size=1200,1980')
#Define web driver as a Chrome driver
driver = webdriver.Chrome('chromedriver', options=options)
driver.implicitly_wait(10)
driver.get('https://denago.com/collections/ebikes')
driver.find_element(By.ID, 'CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll').click()
action = ActionChains(driver)
ourbike = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, '(//li[#itemid="m9RVB"])[2]')))
ActionChains(driver).move_to_element(ourbike).perform()
Titles = driver.find_elements(By.CLASS_NAME, 'mm-title')
for i in range(len(Titles)):
print(Titles[i].text)
driver.quit()

I think you are looking something like this:
# Needed libs
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# We create the driver
options=webdriver.ChromeOptions()
options.add_argument('--headless')
driver = webdriver.Chrome(options=options)
# We maximize the window, because if not the page will be different
driver.maximize_window()
# We navigate to the url
driver.get('https://denago.com/collections/ebikes')
# We wait for the first title, I think it is enough
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "(//h5)[1]")))
# We get all the titles elements
titles=driver.find_elements(By.XPATH,'//h5')
# For each title element we get the text and also we get the price
for i in range(0,len(titles)):
product_name = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, f'(//h5)[{i+1}]'))).text
product_price = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, f'(//div[#class="price"])[{i+1}]'))).text
print(f"Product {i+1}: {product_name} - Price: {product_price}")
driver.quit()

There are 5 bikes on that page. Here is a more pythonic (and more selenium..ish) way of getting those titles (and other info on each bike, if you want):
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,720")
webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
driver = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(driver, 25)
driver.get('https://denago.com/collections/ebikes')
try:
wait.until(EC.element_to_be_clickable((By.ID, "CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll"))).click()
print('accepted cookies')
except Exception as e:
print('no cookie button!')
bikes= wait.until(EC.presence_of_all_elements_located((By.XPATH, '//div[#class="grid-view-item product-card"]//h5/a')))
for bike in bikes:
print(bike.text.strip())
Printout in terminal:
accepted cookies
DENAGO CITY MODEL 1 STEP-THRU EBIKE
DENAGO CITY MODEL 1 TOP-TUBE EBIKE
DENAGO COMMUTE MODEL 1 STEP-THRU EBIKE
DENAGO FAT TIRE STEP-THRU EBIKE
DENAGO COMMUTE MODEL 1 TOP-TUBE EBIKE
Selenium docs: https://www.selenium.dev/documentation/

Selenium Web driver ( driver.find_element(By.XPATH, '')) IS NOT WORKING Python

https://www.espncricinfo.com/player/aamer-jamal-793441
This is the URL and here i am trying to access Full Name "Aamer Jamal". with the help of selenium web driver. But I dont know why it gives
NoSuchElementException
`the code is written below:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium_firefox import Firefox
import time
import pandas as pd
driver = webdriver.Firefox()
#Reach to the Landing page
driver.get('https://www.espncricinfo.com/player/aamer-jamal-793441')
driver.maximize_window()
time.sleep(25)
not_now = driver.find_element(By.ID, 'wzrk-cancel')
not_now.click()
fullname = driver.find_element(By.XPATH, '/html/body/div[1]/section/section/div[4]/div[2]/div/div[1]/div/div/div[1]/div[1]/span/h5')
print(fullname.text)`
Error :
NoSuchElementException: Message: Unable to locate element: /html/body/div[1]/section/section/div[4]/div[2]/div/div[1]/div/div/div[1]/div[1]/span

You have to use WebDriverWait expected_conditions explicit waits, not a long hardcoded pauses. You also have to learn how to create correct locators. Long absolute XPaths and CSS Selectors are extremely breakable. The following code works:
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 60)
actions = ActionChains(driver)
url = "https://www.espncricinfo.com/player/aamer-jamal-793441"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.ID, 'wzrk-cancel')))
fullname = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'ds-text-title-l'))).text
print(fullname)
The output is:
Aamer Jamal

Pycharm: Selenium with Python: Unable to find web elements using headless chrome

When I run the below code with headless chrome I'm facing issues while identifying the elements. The same code runs just runs fine by commenting the below lines with head full mode.
# chrome_options.add_argument('--headless')
# chrome_options.add_argument('--disable-gpu')
Test Details:
Operating system: Windows10
ChromeDriver: 75.0.3770.8
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()
Output:
"C:\Program Files (x86)\Python37-32\python.exe" C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py
Checking for win32 chromedriver:75.0.3770.8 in cache
Driver found in C:\Users\vishr\.wdm\chromedriver\75.0.3770.8\win32/chromedriver.exe
Home | Let's Kode It
Traceback (most recent call last):
File "C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py", line 15, in <module>
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
File "C:\Users\vishr\AppData\Roaming\Python\Python37\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Process finished with exit code 1

for headless browser you have to set the window size to fire on event.Because headless browser can't recognise where to click without window size.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('window-size=1920x1080')
driver = webdriver.Chrome(executable_path='path/to/chrome driver',options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()
Printed output on console on headless mode.
Home | Let's Kode It
Let's Kode It

You're clicking a button that loads another page.
You have to wait for the other page to load. This is tricky to get perfect.
The easiest solution:
sleep(3)
The better solution using implicit waiting:
driver.implicitly_wait(3)
An even better solution using explicit waiting:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.ID, "user_email")))
An even better better solution would be to catch exceptions, and layer all these solutions.

To click() on the element with text as Login you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following solutions:
Using PARTIAL_LINK_TEXT:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Login"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.navbar-link.fedora-navbar-link[href='/sign_in']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='navbar-link fedora-navbar-link' and #href='/sign_in']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

VISHVAMBRUTHJAVAGALTHIMMEGOWDA,
I tried your code and was getting the same exception, First I thought that username was in Frame or iframe but it is not.
Then I tried to introduce webdriver wait and it worked just fine :
Code :
wait = WebDriverWait(driver,10)
driver.maximize_window()
driver.get("https://learn.letskodeit.com/")
print(driver.title)
driver.find_element_by_xpath("//a[contains(text(),'Login')]").click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
#driver.find_element_by_id("user_email").
driver.find_element_by_id("user_password").send_keys("abcabc")
driver.find_element_by_name("commit").click()
print(driver.title)
Remember to imports these :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
EDIT 1 :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu') # Last I checked this was necessary.
driver = webdriver.Chrome("C:/Users/XXXX/Downloads/BrowserDriver/chromedriver_win32/chromedriver.exe", chrome_options=options)
wait = WebDriverWait(driver,10)
driver.get("https://learn.letskodeit.com/")
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test#email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)

How can i scrape the text from a located element using Selenium and Python

I am trying to run the following code
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)
driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
time.sleep(20) #bypass cloudflare
price = driver.find_element_by_xpath('//*[#id="battlepet-page"]/div[1]/table/tr[3]/td/span')
print (price)
so i can scrape the "Current Price" from the page . But this xpath location won't return the text value( i also tried the "text" varriant in the end with no success.
thanks in advance for any reply

First, use WebdriverWait to wait for the element instead of sleep.
Second, Your locator is not finding the element.
Try this,
driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
price = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.XPATH,"//div[#id='battlepet-page']/div/table/tr[#class='current-price']/td/span")))
print(price.text)
To use wait import the followings,
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

You should wait for visibility of element before getting text. Check WebDriverWait in example below:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
rom selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 20)
driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
current_price = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, ".current-price .price"))).text
print(current_price)

To scrape the value of Current Price from the webpage you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tr.current-price td>span"))).text)
Using XPATH:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//th[text()='Current Price']//following::td[1]/span"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC