Develop Reference

How to first write in search box and then select the first index in Listview by Selenium in Python

I am going to scrap glassdoor review for a list of companies. To do this, I need to first login in order to get access all reviews. Then, my approach is entering the name of the 1st company and then scarp its review.....do the same for all companies in the list....
To do this when I go to this URL "https://www.glassdoor.co.in/member/home/index.htm", I should insert the name of company in "search text-box" and then select first index from list view and finally enter search button to go the next link which is review of that comapny..My challenge is with selecting 1st index from "search textbox".. Actually, the last line of code where I going to send cursor on "search button" . I have this error "AttributeError: 'NoneType' object has no attribute 'send_keys'"
I would appreciate if you could help me!
I used the following code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, WebDriverException
from selenium.webdriver.chrome.options import Options
import time
import pandas as pd
driver_path= r"C:\Users\TMaghsoudi\Desktop\chromedriver_win32.exe"
# chrome options
options = webdriver.ChromeOptions()
# options.add_argument("--start-maximized")
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(driver_path, chrome_options=options)
# set driver
driver = webdriver.Chrome(driver_path, chrome_options=options)
# get url
url = "https://www.glassdoor.co.in/Job/index.htm"
driver.get(url)
time.sleep(3)
driver.find_element(By.CLASS_NAME, "HeaderStyles__signInButton").click()
# singin = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "SignInButton")))
# singin.click()
time.sleep(5)
Enter_email= driver.find_element(By.ID, "modalUserEmail")
Enter_email.send_keys("XXXXXX")
Enter_email.send_keys(Keys.ENTER)
Enter_pass= driver.find_element(By.ID,"modalUserPassword")
Enter_pass.send_keys("XXXXX")
SingIn= WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[#class='d-flex align-items-center flex-column']/button[#class='gd-ui-button mt-std minWidthBtn css-1dqhu4c evpplnh0']")))
SingIn.click()
time.sleep(5)
driver.set_window_size(1120, 1000)
driver.find_element(By.CLASS_NAME,"siteHeader__HeaderStyles__navigationItem:nth-child(2)").click()
Company=driver.find_element(By.ID,"sc\.keyword").send_keys("Amazon")
Company.send_keys(Keys.ENTER) **#the error is here!!!!!!**

By running
Company = driver.find_element(By.ID,"sc\.keyword").send_keys("Amazon")
you are assigning the value None to Company, this is because .send_keys() returns the value None if it executed without errors.
So you have to run
Company=driver.find_element(By.ID,"sc\.keyword")
Company.send_keys("Amazon")
Company.send_keys(Keys.ENTER)
You can also run the last two commands in one line
Company.send_keys("Amazon" + Keys.ENTER)

To first write within the search box and then instead of selecting the first index in Listview, you could send ENTER back to back using the following locator strategies:
Code block:
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://www.glassdoor.co.in/Job/index.htm')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='d-none d-lg-block']//button[text()='Sign In']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#modalUserEmail"))).send_keys("debanjan.selenium#gmail.com")
driver.find_element(By.CSS_SELECTOR, "input#modalUserPassword").send_keys("dev_anjan")
driver.find_element(By.XPATH, "//span[text()='Sign In']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[aria-label='Search Keyword']"))).send_keys("Amazon" + Keys.RETURN)
Browser Snapshot:

Trying to locate an element in a webpage but getting NoSuchElementException

I am trying to get the webdriver to click a button on the site random.org The button is a generator that generates a random integer between 1 and 100. It looks like this:
After inspecting the webpage, I found the corresponding element on the webpage looks something like this:
It is inside an iframe and someone suggested that I should first switch over to that iframe to locate the element, so I incorporated that in my code but I am constantly getting NoSuchElementException error. I have attached my code and the error measage below for your reference. I can't understand why it cannot locate the button element despite referencing the ID, which is supposed to unique in the entire document.
The code:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Edge()
driver.get("https://www.random.org/")
driver.implicitly_wait(15)
driver.switch_to.frame(driver.find_element(By.TAG_NAME, "iframe"))
button = driver.find_element(By.CSS_SELECTOR, "input[id='hnbzsqjufzxezy-button']")
button.click()
The error message:

Make sure that there are no more Iframes on the page. If there are a few an not only one do this:
iframes = driver.find_elements(By.CSS, 'iframe')
// try switching to each iframe:
driver.switch_to.frame(iframes[0])
driver.switch_to.frame(iframes[1])
You can't find the button because its name contain random letters. Every time you will refresh the page you can see that the name value will change. So, do this:
button = driver.findElement(By.CSS, 'input[type="button"][value="Generate"]')
button.click()

There are several issues with your code:
First you need to close the cookies banner
The locator of the button is wrong. It's id is dynamic.
You need to use WebDriverWait to wait for elements clickability.
The following code works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = 'https://www.random.org/'
driver.get(url)
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[onclick*='all']"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.TAG_NAME, "iframe")))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[id*='button']"))).click()

why element in html like (drop down, button, textbox) do not reachabel with selenium in python?

I want to change or at least make any small effect in the korean custom website but it seems they are not accessible by my code! Maybe they are in the internal iframe or not, I don't know. I want to change dropdown, write something in textbox and click search button, but I cannot.
May anyone help me?
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
webdriver_service = Service('C:\Webdriver\chromedriver.exe')
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
url = 'http://www.kita.org/kStat/byCom_AllCount.do'
browser.get(url)
time.sleep(5)
select = Select(WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, '/html/body/table/tbody/tr/td/table/tbody/tr/td/form/table/tbody/tr/td/table[3]/tbody/tr/td/table/tbody/tr/td/table/tbody/tr[2]/td[2]/select')))) # select dropdown
select.select_by_index(1)
browser.find_element(By.XPATH,'/html/body/table/tbody/tr/td/table/tbody/tr/td/form/table/tbody/tr/td/table[2]/tbody/tr/td[2]/table/tbody/tr/td[2]/a/img').click() #click seach Button
time.sleep(5)

There is an iframe there.
You need to switch to it first in order to access elements in it.
Also, you should improve your locators.
And insert some text into the item input.
The following code works
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = "http://www.kita.org/kStat/byCom_AllCount.do"
driver.get(url)
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID,"iframe_stat")))
select = Select(WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[name='cond_choosefield']"))))
select.select_by_index(1)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[name='cond_prdt_cd']"))).send_keys("kuku")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='searchForm']"))).click()

My Selenium webdriver doesn’t work on cTrader

Why is my Selenium webdriver not working?
I would like to log in automatically on https://ct.spotware.com/. But Selenium can't find the HTML class for the login box.
For this, I wrote this little script:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome("./chromedriver")
driver.get("https://ct.spotware.com/")
time.sleep(10)
Login = driver.find_element(By.CLASS_NAME,"_a _b _gc _gw _dq _dx _gd _cw _em _cy _gx _fu _gy _fv _fy _fw _fx _db _ge _gf _gz _gg _gh _gi _gj _gk _gl _gm _gn")
Ctrader HTM class reference
The error message is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"._a _b _gc _gw _dq _dx _gd _cw _em _cy _gx _fu _gy _fv _fy _fw _fx _db _ge _gf _gz _gg _gh _gi _gj _gk _gl _gm _gn"}
Somehow the whole site doesn't work with Selenium. On other sites, like Wikipedia, my script works perfectly. Just not on cTrader.
Is there a solution?

There are several issues here:
All these class name values _a _b _gc _gw _dq _dx _gd _cw _em _cy _gx _fu _gy _fv _fy _fw _fx _db _ge _gf _gz _gg _gh _gi _gj _gk _gl _gm _gn are multiple separate class names. To use them you need to use CSS Selector or XPath.
The sequence of all the above class names looks to be fragile. You should use another, more stable and more clear locator.
Instead of hardcoded sleep you should use WebdriverWait explicit waits.
You need to close the cookies banner
And insert the user name and passwords
Anyway, the code below clicks the login button itself.
Please see the code below:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("--start-maximized")
s = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=s)
wait = WebDriverWait(driver, 20)
driver.get("https://ct.spotware.com/")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[type='submit']"))).click()

The spaces in your class name are not handled by Selenium. The following may help.
Login = driver.find_element(By.CSS_SELECTOR, "._a._b._gc._gw._dq._dx._gd _cw._em._cy._gx._fu._gy._fv._fy._fw._fx._db._ge._gf._gz._gg._gh._gi._gj._gk._gl._gm._gn")
However, upon examining your site, I'd recommend using a CSS selector such as this:
'input[placeholder="Enter your email or cTrader ID"]'

This is one way to correctly select the elements and login:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time as t
import pandas as pd
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
# chrome_options.add_argument("--headless")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1920,1080")
webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
actions = ActionChains(browser)
wait = WebDriverWait(browser, 20)
url = 'https://ct.spotware.com/'
browser.get(url)
login_field = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'input[placeholder="Enter your email or cTrader ID"]')))
pass_field = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'input[placeholder="Enter your password"]')))
submit_button = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[text() = "Log In"]')))
login_field.send_keys('username')
pass_field.send_keys('bad_pass')
submit_button.click()
print('clicked')
Selenium documentation can be found at https://www.selenium.dev/documentation/

How to get the result value of the URL shorten page?

I am trying to get automatically the result of the URL shorten.
This is the page what I am using: url shortener site
This is the code I made (URLS list contains links):
driver.get("http://paylinx.pw/linx/")
for i in URLS:
driver.find_element_by_xpath('//*[#id="url"]').click()
time.sleep(2)
driver.find_element_by_xpath('//*[#id="url"]').send_keys(i)
time.sleep(2)
driver.find_element_by_xpath('//*[#id="invisibleCaptchaShort"]').click()
time.sleep(2)
After this I get the shortened url. I would need a little help to get it somehow.

Use WebDriverWait to wait for short url result and get the value.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
with driver:
driver.get("http://paylinx.pw/linx/")
for url in URLS:
driver.find_element_by_id("url").send_keys(url, Keys.ENTER)
short_url = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".add-link-result .input-lg"))).get_attribute("value")
print(short_url, url)

You need use page.source, because it translate you page on need code, for chromedriver, some like lifehuck )) or you can use get_attribute('innerHTML') - you can access everything from the page.
Python WebDriver how to print whole page source (html)

To extract the result value of the URL shortener automatically you need to induce WebDriverWait for the visibility_of_element_located() and you can use the following Locator Strategies:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
URLS = ['https://selenium.dev/downloads/','https://selenium.dev/documentation/en/']
for i in URLS:
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("http://paylinx.pw/linx/")
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#url")))
element.clear()
element.send_keys(i)
driver.find_element_by_css_selector("button.btn-captcha#invisibleCaptchaShort").click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.input-group>input.form-control.input-lg"))).get_attribute("value"))
driver.quit()
Console Output:
http://paylinx.pw/linx/Uksheqw8
http://paylinx.pw/linx/s0DA44C