Develop Reference

Job Applying Bot in Python facing IndexError: list index out of range

Disclaimer: I'm coming back to scripting after more than a decade (and I was a novice to begin with) so apologies if the question is mundane but help is much needed and appreciated.
I'm trying to modify a python script to log me into a job portal, search job vacancies based on attributes and then apply to said jobs.
Since the portal opens new vacancies on separate tabs, the code is supposed to go to the next tab and test the criteria on the job description
The code snippet is as below:
for i in range(1,6):
driver.get('https://www.naukri.com/'+role+'-jobs-in-'+location+'-'+str(i)+'?ctcFilter='+str(LL)+'to'+str(UL))
driver.switch_to.window(driver.window_handles[1])
url = driver.current_url
driver.get(url)
try:
test = driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[2]')
if all(word in test.text.lower() for word in Skillset):
driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[1]/div[1]/div[3]/div/button[2]').click()
time.sleep(2)
driver.close()
driver.switch_to.window(driver.window_handles[0])
else:
driver.close()
driver.switch_to.window(driver.window_handles[0])
except:
driver.close()
driver.switch_to.window(driver.window_handles[0])
However, when I run the script, it just logs me in to the portal and goes to the correct listings page but just stays there. Plus, it pops this error:
> line 43, in <module> driver.switch_to.window(driver.window_handles[1])
> IndexError: list index out of range
Not able to understand what list this is referring to and how to fix this code. Any help is appreciated.
The complete code for those interested:
import selenium
from selenium import webdriver as wb
import pandas as pd
import time
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = wb.Chrome("/Users/shashank/Desktop/Naukri Auto Apply/chromedriver")
Skillset = ['Marketing','Communications','Sales']
Skillset = [x.lower() for x in Skillset]
LL = 15 #lower limit of expected CTC
UL = 25 #upper limit of expected CTC
location = 'Bangalore'
location = location.lower().replace(" ","-")
role = 'Marketing Manager'
role = role.lower().replace(" ","-")
driver.get("https://www.naukri.com")
driver.find_element_by_xpath('//*[#id="login_Layer"]/div').click()
time.sleep(5)
driver.find_element_by_xpath('//*[#id="root"]/div[3]/div[2]/div/form/div[2]/input').send_keys("email_address_here")
driver.find_element_by_xpath('//*[#id="root"]/div[3]/div[2]/div/form/div[3]/input').send_keys("password_here")
time.sleep(5)
driver.find_element_by_xpath('//*[#id="root"]/div[3]/div[2]/div/form/div[6]/button').click()
time.sleep(20)
driver.find_element_by_xpath('/html/body/div[3]/div/div[1]/div[1]/div').click()
for i in range(1,6):
driver.get('https://www.naukri.com/'+role+'-jobs-in-'+location+'-'+str(i)+'?ctcFilter='+str(LL)+'to'+str(UL))
driver.switch_to.window(driver.window_handles[1])
url = driver.current_url
driver.get(url)
try:
test = driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[2]')
if all(word in test.text.lower() for word in Skillset):
driver.find_element_by_xpath('//*[#id="root"]/main/div[2]/div[2]/section[1]/div[1]/div[3]/div/button[2]').click()
time.sleep(2)
driver.close()
driver.switch_to.window(driver.window_handles[0])
else:
driver.close()
driver.switch_to.window(driver.window_handles[0])
except:
driver.close()
driver.switch_to.window(driver.window_handles[0])
Thanks in advance for answering! It will really help with the job search!

No need to switch to another window
When you are opening the URL with that specific details in the for loop, the page is getting loaded one by one in the same window. Switch to window when there is a New window tab opened. Link to Refer
Choose Explicit waits instead of time.sleep(). You have refined WebdriverWait but never used it.
Try to come up with good locators. Go for Relative Xpath instead of Absolute Xpath.Link to Refer
Not sure what you are trying to do in try block. The locators does not highlight any elements in the page.
Refer below code:
# Imports
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Chrome(service= Service("path to chromedriver.exe"))
driver.maximize_window()
driver.get("https://www.naukri.com")
wait = WebDriverWait(driver,30)
login_btn = wait.until(EC.element_to_be_clickable((By.XPATH,"//div[text()='Login']")))
login_btn.click()
email = wait.until(EC.element_to_be_clickable((By.XPATH,"//form[#name='login-form']//input[contains(#placeholder,'Email')]")))
email.send_keys("abc#gmail.com")
password = wait.until(EC.element_to_be_clickable((By.XPATH,"//form[#name='login-form']//input[contains(#placeholder,'password')]")))
password.send_keys("password")
password.submit()
role = "marketing-manager"
loc = "bangalore"
llimt = "15"
ulimit = "25"
for i in range(1,6):
driver.get(f"https://www.naukri.com/{role}-jobs-in-{loc}-{i}?ctcFilter={llimt}to{ulimit}")
time.sleep(2)
# Update for Chat bot
try:
chatbot_close = wait.until(EC.element_to_be_clickable((By.XPATH,"//div[#class='chatbot_Nav']/div")))
chatbot_close.click()
except:
print("Chat bot did not appear")

How Do I Search And Select First Option From website in selenium python

I have this code which goes to https://xiaomifirmwareupdater.com/miui/ , searches query and selects first element and downloads the rom, but its always selecting the same element despite of different query, I first thought website was giving same top result, but I checked with my browser but its giving different results, How can I fix / do this?
My Code :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
import asyncio
GOOGLE_CHROME_BIN = 'path here'
CHROME_DRIVER = 'driver path here'
async def bruh(query="Redmi Note 8 Pro China"):
url = "https://xiaomifirmwareupdater.com/miui"
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.binary_location = GOOGLE_CHROME_BIN
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-gpu")
driver = webdriver.Chrome(executable_path=CHROME_DRIVER, options=chrome_options)
driver.get(url)
await asyncio.sleep(10)
w = WebDriverWait(driver, 20)
search_xpath = '/html/body/div[3]/section/div[2]/div[3]/div[1]/div/div[2]/div[1]/div[2]/div/label/input'
next_page_url_xpath = '/html/body/div[3]/section/div[2]/div[3]/div[1]/div/div[2]/div[2]/div/table/tbody/tr[1]/td[8]/a'
version_xpath = '/html/body/div[3]/section/div[2]/div[2]/div[2]/div[1]/div/ul/li[3]/h5'
name_xpath = '/html/body/div[3]/section/div[2]/div[2]/div[2]/div[1]/div/ul/li[8]/h5/span'
w.until(expected_conditions.presence_of_element_located((By.XPATH, search_xpath)))
elem = driver.find_element_by_xpath(search_xpath)
elem.send_keys(query)
await asyncio.sleep(20)
next_page_elem = driver.find_element_by_xpath(next_page_url_xpath)
nextm = next_page_elem.get_attribute('href')
driver.get(nextm)
await asyncio.sleep(10)
version_elem = driver.find_element_by_xpath(version_xpath).text
name_elem = driver.find_element_by_xpath(name_xpath).text
version_elem = version_elem.replace("Version: ", "")
print(version_elem)
print(name_elem)
url = f"https://bigota.d.miui.com/{version_elem}/{name_elem}"
print(url)
driver.close()
I want to visit website, send my query and select first option and convert it to download able url. can anyone help? Thank You

You are doing a lot of extra stuff that I don't quite follow.
A few pieces of feedback...
Waiting for presence just means that the element is in the DOM, not that it's visible or interactable. If you are going to use click or send keys, you need to wait until clickable or visible, respectively, or you may get an exception.
You don't need all those sleeps, especially when you are using WebDriverWait. Best practice is to avoid sleeps (they make your script slower and less reliable), instead use WebDriverWait.
The wait for an element will return that element so you don't need to wait, then find, then click... you can just wait.until(...).click().
You were getting the href from a link and then navigating to it... just click the link.
Instead of replace(), I just used split()... either is fine. I think split() is less likely to break, e.g. if they change the labels.
Updating your code based on my feedback above, this should work.
driver.get(url)
wait = WebDriverWait(driver, 20)
wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, "input"))).send_keys(query)
wait.until(expected_conditions.element_to_be_clickable((By.CSS_SELECTOR, "#miui td > a"))).click()
version = wait.until(expected_conditions.visibility_of_element_located((By.XPATH, "//h5[./b[text()='Version: ']]"))).text.split()[1]
package_name = wait.until(expected_conditions.visibility_of_element_located((By.ID, "filename"))).text
print(version)
print(package_name)
url = f"https://bigota.d.miui.com/{version}/{package_name}"
print(url)
driver.close()

Using Selenium to get past dropdown (md-select)

I'm stuck with a dropdown that I can't get pass in Selenium.
I'm trying to collect some price data using Selenium from this link:
https://xxx. In this link, you need to click on a button (Next), then select any option in the subsequent dropdown, then press (Next) again to advance into the information page that I wanted to collect some information. I'm stuck at the dropdown - I am unable to select any option.
This is my code thus far:
browser.get("https://xxx/#/pricePlans/step1")
wait = WebDriverWait(browser, 10)
while True:
try:
button = browser.find_element_by_css_selector('body > div.md-dialog-container.ng-scope > md-dialog > md-dialog-actions > div > button')
except TimeoutException:
break
button.click()
options_box= browser.find_element_by_class_name('bullet-content-title')
wait = WebDriverWait(browser, 5)
options_box.click()
The issue lies with the dropdown options (It has options like HDB 1-room, HDB 2-room etc). I tried to reference the option box by XPATH, CSS selector, class_name (as seen above) but with the snippet above, Spyder issues time-out. Other snippets I tried included:
ui.WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, "bullet-content-title")))
using XPATH, class_name but no luck.
I'm a newbie at web scraping who got thus far by searching the SO, but I am unable to find much solutions regarding (md-select) dropdowns.
I also attempted to use
ActionChains(driver).move_to_element(options_box).click(options_box)
but I did not see any clicking nor mouse movements so i'm stumped.
I appreciate any advice at this point of time. Thank you so much!
Edit:
Code Snippets and Responses:
from selenium import webdriver
from selenium.webdriver.support import ui
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.action_chains import ActionChains
option = webdriver.ChromeOptions()
option.add_argument('--incognito')
browser = webdriver.Chrome(executable_path='C:\\ChromeDriver\\chromedriver.exe', options=option)
browser.get("https://xxx")
wait = WebDriverWait(browser, 10)
while True:
try:
button = browser.find_element_by_css_selector('body > div.md-dialog-container.ng-scope > md-dialog > md-dialog-actions > div > button')
except TimeoutException:
break
button.click()
options_box = browser.find_element_by_class_name('bullet-content-title')
wait = WebDriverWait(browser, 5)
options_box.click()
This returns "StaleElementReferenceException: stale element reference: element is not attached to the page document"
Which I assume it is due to the presence of the second "Next" Button which is inert at the moment.
options_box = ui.WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, "bullet-content-title")))
options_box.click()
Does nothing. Spyder eventually returned me TimeOut Error.

#AndrewRay answer is good for getting the value but not for selecting the options. you can do this to select the options.
#browser.get("https://......")
wait = WebDriverWait(browser, 10)
try:
browser.find_element_by_css_selector('button.green-btn').click()
# wait until dialog dissapear
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, 'md-dialog[aria-describedby="dialogContent_0"]')))
# click the dropdown
browser.find_element_by_css_selector('md-input-container').click()
# select the option element
setOptionElement = browser.find_element_by_css_selector('md-option[value="HDB Executive"]')
# need to scrollIntoView if the option in the bottom
# or you get error the element not clickable
browser.execute_script('arguments[0].scrollIntoView();arguments[0].click()', setOptionElement)
except Exception as ex:
print(ex)

driver.get('https://compare.openelectricitymarket.sg/#/pricePlans/step1')
time.sleep(5)
next_btn = driver.find_element_by_css_selector('button.green-btn')
next_btn.click()
dropdown = driver.find_element_by_id('select_4')
options = dropdown.find_elements_by_tag_name('md-option')
for option in options:
print option.get_attribute('value')
Hope this helps. Use the .get_attribute method to find the value of the option and click that option if matches the desired value. :)

Can't get rid of "stale element" error while running my scrpt

I've written a script in python with selenium. The script is supposed to click on some links in a webpage. When I run my script, It does click on the first link and then throws an error stale element reference: element is not attached to the page document instead of chasing for the next link. I searched a lot for the last few hours to find any solution to get rid of this error but no luck.
I'm not interested in their data so any solution other than the perocess of clicking is not what I'm looking for. How can I click on the links until the last link?
This is my attempt so far:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def click_links(driver,url):
driver.get(url)
for olink in wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "result-row__item-hover-visualizer"))):
olink.click()
time.sleep(3)
if __name__ == '__main__':
weblink = "https://www.hitta.se/s%C3%B6k?vad=Markiser+%26+Persienner"
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
try:
click_links(driver,weblink)
finally:
driver.quit()

You can try below code:
def click_links(driver,url):
driver.get(url)
links_len = len(wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "result-row__item-hover-visualizer"))))
for index in range(links_len):
cookies_bar = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '[data-bind="visible: showCookieDialog"]')))
driver.execute_script("arguments[0].hidden='true';", cookies_bar)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button[data-track="click-show-more"]'))).click()
entry = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "result-row__item-hover-visualizer")))[index]
entry.click()
time.sleep(3)
driver.back()

Click on link using selenium webdriver

I am trying to click on a link and can't seem to get it to work. I click all the way up to the page I need, but then it won't click the last link. The code is as follows:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
import time
from bs4 import BeautifulSoup
import requests
import pandas as pd
import openpyxl
from password import DKpassword
#import SendKeys
beginningTime = time.time()
browser = webdriver.Chrome()
browser.get('https://www.draftkings.com/lobby')
browser.maximize_window()
time.sleep(5)
signinLink = browser.find_element_by_xpath("""//*[#id="react-mobile-home"]/section/section[2]/div[2]/div[3]/div/input""")
signinLink.click()
signinLink.send_keys("abcdefg")
signinLink.send_keys(Keys.TAB)
passwordLink = browser.find_element_by_xpath("""//*[#id="react-mobile-home"]/section/section[2]/div[2]/div[4]/div/input""")
passwordLink.send_keys(DKpassword)
passwordLink.send_keys(Keys.ENTER)
time.sleep(5)
if browser.current_url == "https://www.draftkings.com/account/sitelogin/false?returnurl=%2Flobby1":
signin = browser.find_element_by_partial_link_text("SIGN IN")
signin.click()
elif browser.current_url == "https://www.draftkings.com/lobby#/featured":
mlbLink = browser.find_element_by_partial_link_text("MLB")
mlbLink.click()
else:
print("error")
time.sleep(5)
featuredGame = browser.find_element_by_class_name("GameSetTile_tag")
featuredGame.click()
time.sleep(5)
firstContest = browser.find_element_by_partial_link_text("Enter")
firstContest.click()
I receive the error message that the element is not clickable at point... and another element would receive the click. Any help would be greatly appreciated. I don't care which contest is clicked on as long as it is on the featured page which the previous code directs it too.

There can be multiple reasons for that.
1. You might have to scroll down or might have to perform some action so that it'll be visible to script.
for scroll down you can use this code :
browser.execute_script("window.scrollTo(0, Y)")
where Y is the height (on a fullhd monitor it's 1080)
2. There can be multiple web element present , in this case you have to use unique element from dom , you can check that by right click > inspect > under element section > CTRL+F > write your locator (in your case xpath) to check how many entries are present.
You can try with this xpath also :
//a[contains(text(),'Enter') and contains(#href,'/contest/draftteam') and contains(#class,'dk-btn-dark')]
Code :
WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "//a[contains(text(),'Enter') and contains(#href,'/contest/draftteam') and contains(#class,'dk-btn-dark')]"))
firstContest = browser.find_element_by_xpath("//a[contains(text(),'Enter') and contains(#href,'/contest/draftteam') and contains(#class,'dk-btn-dark')]")
firstContest.click()

Replace click event with action class, which will solve this Exception
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(driver)
actions.move_to_element(firstContest).click().perform()

I was able to overcome this exception by using Click through JavaScript executor instead of regular Element.click(), as below-
WebElement element = webDriver.findElement(By.xpath(webElemXpath));
try {
JavascriptExecutor ex = (JavascriptExecutor) webDriver;
ex.executeScript("arguments[0].click();", element);
logger.info(elementName was + " clicked");
}
catch (NoSuchElementException | TimeoutException e) {
logger.error(e.getMessage());
}