Develop Reference

Failed to locate the webdriver element using Selenium |

Using selenium automation webdriver i'm unable to locate the text box element on a travel website using python. Using locator present in the webdriver such as Id/name/css_selector/class_name or xpath/full xpath.
Below is the screenshot of the python code:
[Code_1][1]
While the first one is located the second one isn't. The corresponding HTML code is
[text_box2][2]
How can i fill(automate) both fileds corresponding flight destinations i.e leaving and going
[1]: https://i.stack.imgur.com/gpoIr.jpg
[2]: https://i.stack.imgur.com/Q7mGs.jpg

Here is a slightly different approach for locating things. I am using waits borrowed from CruisePandey in this thread. I used firefox, but that is adaptable. Some notes of what was hard:
I had to make sure to be on the Flights tab.
I had to click in the from and to fields, which were buttons after all, and then wait to be able to type into the revealed input fields.
Finally, I had to choose the first element from the dropdown list that resulted.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox(executable_path='/usr/bin/geckodriver')
driver.maximize_window()
driver.get('https://www.expedia.co.in')
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Flights")))
driver.find_element_by_link_text('Flights').click()
wait.until(EC.presence_of_element_located((By.ID, "location-field-leg1-origin")))
driver.find_element(By.CLASS_NAME,'uitk-faux-input').click()
fieldInput = driver.find_element_by_id('location-field-leg1-origin')
wait.until(EC.visibility_of(fieldInput))
fieldInput.send_keys("SFO")
wait.until(EC.presence_of_element_located((By.TAG_NAME, "strong")))
driver.find_element_by_tag_name('strong').click()
driver.find_elements(By.CLASS_NAME,'uitk-faux-input')[1].click()
fieldInput = driver.find_element_by_id('location-field-leg1-destination')
wait.until(EC.visibility_of(fieldInput))
fieldInput.send_keys("BOS")
wait.until(EC.presence_of_element_located((By.XPATH,"//strong[contains(text(), 'Boston')]")))
driver.find_element_by_xpath("//strong[contains(text(), 'Boston')]").click()

You may want to use the below xpath for going To input field :
//label[text()='Going to']//following-sibling::input[#name]
Code :
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//label[text()='Going to']//following-sibling::input[#name]"))).send_keys('something')
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

if you want to automate from 1 place holder to another, i have use this set of code & it works for me
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[#id='location-field-leg1-origin-menu']/div[2]/ul/li[1]/button"))).click()
b = wait.until(EC.element_to_be_clickable((By.XPATH, ".//*[#id='location-field-leg1-destination-menu']/div[1]/button"))).send_keys("NYC")
b = wait.until(EC.element_to_be_clickable((By.XPATH,".//*[#id='location-field-leg1-destination-menu']/div[2]/ul/li[1]/button"))).click()
c = wait.until(EC.element_to_be_clickable((By.XPATH, ".//*[#id='d1-btn']"))).click()

Loop is not working properly for Selenium Python

i'm trying to run the following piece of code :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome('C:/Users/SoumyaPandey/Desktop/Galytix/Scrapers/data_ingestion/chromedriver.exe')
driver.get('https://www.cnhindustrial.com/en-us/media/press_releases/Pages/default.aspx')
years_urls = list()
#ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years --> id for the year filter
years_elements = driver.find_element_by_id('ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years').find_elements_by_tag_name('a')
for i in range(len(years_elements)):
years_urls.append(years_elements[i].get_attribute('href'))
newslinks = list()
for k in range(len(years_urls)):
url = years_urls[k]
driver.get(url)
#link-detailpage --> id for the newslinks in each year
news = driver.find_elements_by_class_name('link-detailpage')
for j in range(len(news)):
newslinks.append(news[j].find_element_by_tag_name('a').get_attribute('href'))
when I run this code, the newslinks list is empty at the end of execution. But if I run it line by line, by assigning the value of 'k' one by one, on my own, it runs successfully.
Where am I going wrong in the logic. Please help.

It seems there is too much redundant code. I would suggest use either linear xpath or css selector to identify the elements.
However some of the pages the new link not appeared you need to handle this using try..except.
Since you need to navigate each url I would suggest use explicit wait WebDriverWait()
Code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver=webdriver.Chrome("C:/Users/SoumyaPandey/Desktop/Galytix/Scrapers/data_ingestion/chromedriver.exe")
driver.get("https://www.cnhindustrial.com/en-us/media/press_releases/Pages/default.aspx")
allyears=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div#ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years a")))
yearsurl=[url.get_attribute("href") for url in allyears]
newslinks = list()
for yr in yearsurl:
driver.get(yr)
try:
for element in WebDriverWait(driver,5).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"div.link-detailpage >a"))):
newslinks.append(element.get_attribute("href"))
except:
continue
print(newslinks)
OutPut:
['https://www.cnhindustrial.com/en-us/media/press_releases/2021/march/Pages/a-problem-solved-at-a-rate-of-knots-the-latest-Top-Story-available-on-CNHIndustrial-com.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/march/Pages/CNH-Industrial-acquires-a-minority-stake-in-Augmenta.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/march/Pages/CNH-Industrial-presents-YOUNIVERSE.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/march/Pages/Calling-of-the-Annual-General-Meeting.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/march/Pages/CNH-Industrial-completes-minority-investment-in-Monarch-Tractor.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/February/Pages/CNH-Industrial-N-V--announces-the-extension-by-one-additional-year-to-March-2026-of-its-syndicated-credit-facility.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/February/Pages/Working-for-a-safer-future-with-World-Class-Manufacturing.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/February/Pages/Behind-the-Wheel-CNH-Industrial-supports-the-growing-hemp-industry-in-North-America.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/February/Pages/CNH-Industrial-employees-in-Italy-to-receive-contractual-bonus-for-2020-results.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/February/Pages/2020-Fourth-Quarter-and-Full-Year-Results.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/january/Pages/The-Iveco-Defence-Vehicles-plant-in-Sete-Lagoas,-Brazil-and-the-New-Holland-Agriculture-facility-in-Croix,-France.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/january/Pages/CNH-Industrial-to-announce-2020-Fourth-Quarter-and-Full-Year-financial-results-on-February-3-2021.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/january/Pages/CNH-Industrial-publishes-its-2021-Corporate-Calendar.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/january/Pages/Iveco-Defence-Vehicles-supplies-third-generation-protected-military-GTF8x8-(ZLK-15t)-trucks-to-the-German-Army.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/january/Pages/STEYR-New-Holland-Agriculture-CASE-Construction-Equipment-and-FPT-Industrial-win-prestigious-2020-Good-Design%C2%AE-Awards.aspx', 'https://www.cnhindustrial.com/en-us/media/press_releases/2021/january/Pages/CNH-Industrial-completes-the-acquisition-of-four-divisions-of-CEG-in-South-Africa.aspx',so on...]
Update:
If you don't want use webdriverwait which is best practice then use time.sleep() since page needs some time to load and element should be visible before interacting it.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome("C:/Users/SoumyaPandey/Desktop/Galytix/Scrapers/data_ingestion/chromedriver.exe")
driver.get('https://www.cnhindustrial.com/en-us/media/press_releases/Pages/default.aspx')
years_urls = list()
time.sleep(5)
#ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years --> id for the year filter
years_elements = driver.find_elements_by_xpath('//div[#id="ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years"]//a')
for i in range(len(years_elements)):
years_urls.append(years_elements[i].get_attribute('href'))
print(years_urls)
newslinks = list()
for k in range(len(years_urls)):
url = years_urls[k]
driver.get(url)
time.sleep(3)
news = driver.find_elements_by_xpath('//div[#class="link-detailpage"]/a')
for j in range(len(news)):
newslinks.append(news[j].get_attribute('href'))
print(newslinks)

There is a popup asking you to accept cookies that you need to click beforehand.
Add this to your script:
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "CybotCookiebotDialogBodyButtonAccept")))
driver.find_element_by_id("CybotCookiebotDialogBodyButtonAccept").click()
So the final result will be:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome('C:/Users/SoumyaPandey/Desktop/Galytix/Scrapers/data_ingestion/chromedriver.exe')
driver.get('https://www.cnhindustrial.com/en-us/media/press_releases/Pages/default.aspx')
# this part is added, together with the necessary imports
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "CybotCookiebotDialogBodyButtonAccept")))
driver.find_element_by_id("CybotCookiebotDialogBodyButtonAccept").click()
years_urls = list()
#ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years --> id for the year filter
# years_elements = driver.find_element_by_css_selector("#ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years")
years_elements = driver.find_element_by_id('ctl00_ctl33_g_8893c127_d0ad_40f2_9856_d85936172f35_years').find_elements_by_tag_name('a')
for i in range(len(years_elements)):
years_urls.append(years_elements[i].get_attribute('href'))
newslinks = list()
for k in range(len(years_urls)):
url = years_urls[k]
driver.get(url)
#link-detailpage --> id for the newslinks in each year
news = driver.find_elements_by_class_name('link-detailpage')
for j in range(len(news)):
newslinks.append(news[j].find_element_by_tag_name('a').get_attribute('href'))

Selenium clicks next button instead of back button

Trying to scrape this webpage for prices, and I need the prices to be in US dollars, so it is currency I understand. However, when I initially load the URL, it gives the prices in multiple seemingly random currencies. I found that I could change this by clicking the next button, and then the back button, but when I tried to automate this, it did not work. Instead, running this code clicks the next button twice, rather than clicking it once, waiting for five seconds, and then clicking the back button. Here is the code that I am currently using that can replicate this problem.
from selenium import webdriver
driver = webdriver.Chrome(r'C:\Users\Hank\Desktop\chromedriver_win32\chromedriver.exe')
driver.get('https://steamcommunity.com/market/listings/440/Unusual%20Old%20Guadalajara')
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
import time
time.sleep(5)
action = ActionChains(driver)
next_button=wait(driver, 10).until(EC.element_to_be_clickable((By.ID,'searchResults_btn_next')))
action.move_to_element(next_button).click().perform()
time.sleep(5)
back_button=wait(driver, 10).until(EC.element_to_be_clickable((By.ID,'searchResults_btn_prev')))
action.move_to_element(back_button).click().perform()
Thanks, your time and help is greatly appreciated. Please direct me to a relevant question if this one has already been answered somewhere else.

You don't need ActionChains class, it's works by .click() method.
Try following code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Chrome(r'C:\Users\Hank\Desktop\chromedriver_win32\chromedriver.exe')
driver.get('https://steamcommunity.com/market/listings/440/Unusual%20Old%20Guadalajara')
wait = WebDriverWait(driver, 20)
next_button = wait.until(EC.element_to_be_clickable((By.ID,'searchResults_btn_next')))
next_button.click()
time.sleep(5)
back_button = wait.until(EC.element_to_be_clickable((By.ID,'searchResults_btn_prev')))
back_button.click()
But note, time.sleep(5) is bad way, you can use other way, ex : wait until the second page element appear.
Or instead of time.sleep(...) in this case, you can use this code:
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR,'.pagebtn.disabled')))
The above is the disable previous button since you landing in the first pagination, and will gone when you arrieve on second pagination. Use .invisibility_of_element_located, it will more efficient.

Unable to get the right value of a certain amount from a webpage

I've written a script in python using selenium to get the converted value of a certain amount. The amount produces converted value when the earlier is made to put in a placeholder. The newly produced value is found adjacent to the amount. When I put any amount manually in that placeholder, I get a converted value accordingly but when I do the same programmatically, the value remains unchanged and as a result my scraper gets 0 as value. How can I make it work?
Link to that webpage: weblink
The script I've tried with:
from selenium.webdriver import Chrome
from contextlib import closing
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
with closing(Chrome()) as driver:
wait = WebDriverWait(driver, 10)
driver.get("find_the_link_above")
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".OrderForm_input-box_XkGmi input[name='amount']"))).send_keys(100)
item = wait.until(EC.presence_of_element_located((By.CLASS_NAME,"OrderForm_total_6EL8d"))).text
print(item)
When I put any amount to the placeholder manually, the change can be seen like below:
But, when I do the same using the script, this is how it looks like:
I've marked the valuees with black color to let you know what I meant.

Problem is that you are sending the value too early so that value is not reflecting after entering the amount value. Here i am waiting for EUR SPREAD element to load before setting the amount value.You can use the same element or any other of your chose but make sure page loads completely with that object and then send the amount value.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_path = r"path"
driver = webdriver.Chrome(chrome_path)
wait = WebDriverWait(driver, 10)
driver.get("https://www.gdax.com/trade/LTC-EUR")
wait.until(EC.presence_of_element_located((By.XPATH, "//div[#class='OrderBookPanel_text_33dbp']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//input[#name='amount']"))).send_keys(100)
item = wait.until(EC.presence_of_element_located((By.CLASS_NAME,"OrderForm_total_6EL8d"))).text
print(item)
Hope this will solve your problem.

Can't get rid of hardcoded delay even when Explicit Wait is already there

I've written some code in python in combination with selenium to parse the different questions from quora.com. My scraper is doing it's job at this moment. The thing is I've used here hardcoded delay for the scraper to work, even when Explicit Wait has already been defined. As the page is an infinite scrolling one, i tried to make the scrolling process to a limited number. Now, I have got two questions:
Why wait.until(EC.staleness_of(page)) is not working within my scraper. It is commented out now.
If i use something else instead of page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link"))) the scraper throws an error: can't focus element.
Btw, I do not wish to go for page = driver.find_element_by_tag_name('body') this option.
Here is what I've written so far:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.quora.com/topic/C-programming-language")
wait = WebDriverWait(driver, 10)
page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link")))
for scroll in range(10):
page.send_keys(Keys.PAGE_DOWN)
time.sleep(2)
# wait.until(EC.staleness_of(page))
for item in wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "rendered_qtext"))):
print(item.text)
driver.quit()

You can try below code to get as much XHR as possible and then parse the page:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome()
driver.get("https://www.quora.com/topic/C-programming-language")
wait = WebDriverWait(driver, 10)
page = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "question_link")))
links_counter = len(wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "question_link"))))
while True:
page.send_keys(Keys.END)
try:
wait.until(lambda driver: len(driver.find_elements_by_class_name("question_link")) > links_counter)
links_counter = len(driver.find_elements_by_class_name("question_link"))
except TimeoutException:
break
for item in wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "rendered_qtext"))):
print(item.text)
driver.quit()
Here we scroll page down and wait up to 10 seconds for more links to be loaded or break the while loop if the number of links remains the same
As for your questions:
wait.until(EC.staleness_of(page)) is not working because when you scroll page down you don't get the new DOM - you just make XHR which adds more links into existed DOM, so the first link (page) will not be stale in this case
(I'm not quite confident about this, but...) I guess you can send keys only to nodes that can be focused (user can set focus manually), e.g. links, input fields, textareas, buttons..., but not content division (div), paragraphs (p), etc