Develop Reference

My Python selenium code cannot select a select option

I cannot select a select option using below Python code..
I have tried to refer many Q&A such as select, execute_script... but they are still not working.
enter image description here
import time
from selenium import webdriver
browser = webdriver.Chrome()
seoul_url = 'http://kras.seoul.go.kr/land_info/info/landprice/landprice.do'
browser.get(seoul_url)
time.sleep(1)
browser.find_element_by_xpath('//*[#id="sggnm"]/option[11]').click()

You have first click on dropdown and then click on the value that you wanted to select like below
Import
import org.openqa.selenium.JavascriptExecutor
import org.openqa.selenium.WebElement
select the desire value
WebElement element = browser.find_element_by_xpath("//option[contains(text(),'동대문구')]"); // you can choose the value you want
JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", element);

My guess would be you cannot click the element as it is not visible.
To make it visible you must click the dropdown menu first as suggested by #Akzy
driver = webdriver.Chrome()
url: str = "http://kras.seoul.go.kr/land_info/info/landprice/landprice.do"
driver.get(url)
dropdown_element = driver.find_element_by_id("sggnm")
# Open the dropdown menu first
dropdown_element.click()
option_element = driver.find_element_by_xpath('//*[#id="sggnm"]/option[11]')
# Now the options are visible we want to click the option in the list
option_element.click()
This is untested (for this page) but should work for a generic webpage.
There are other ways of doing this such as sending the down key n times to the dropdown element get the nth option element.

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
import pyautogui
driver = webdriver.Chrome()
url: str = "http://kras.gb.go.kr/land_info/info/landprice/landprice.do"
driver.get(url)
time.sleep(1)
def option_click(city):
driver.find_element(By.CSS_SELECTOR, city).click()
time.sleep(1)
def option_select(pos):
i = 0
for i in range(pos):
pyautogui.press("down")
i += 1
pyautogui.press("enter")
time.sleep(1)
option_click('#sggnm')
option_select(10)
option_click('#umdnm')
option_select(3)

NEW
Or you can use ActionChains. In the code pos is the position of the option you want to select
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
import time
actions = ActionChains(driver)
def select_option(id_, pos):
driver.find_element(By.CSS_SELECTOR, id_).click()
actions.send_keys(Keys.DOWN * pos).send_keys(Keys.ENTER).perform()
select_option('#sggnm', 10)
time.sleep(1)
select_option('#umdnm', 3)
OLD
With this code you can change the currently selected option to the desired one without doing a click.
# webelement containing the currently selected option
option1 = driver.find_element(By.XPATH, '//*[#id="sggnm"]/option[1]')
# string with the text of the 11th option
option11 = driver.find_element(By.XPATH, '//*[#id="sggnm"]/option[11]').text
# replace the current option with the 11th option
driver.execute_script("var el = arguments[0]; arguments[1].innerText = el", option11, option1)

Failed to obtain response using Selenium python to automate visiting a website

Basically what I want to do is to automate visiting
this website. If you visit using Chrome, manually picking report type and date, a download link will appear. I tried to automate this process using python + selenium by
following the suggestions in the linked question. But selenium clicking the Get Data button didn't work. I don't know why. Please help.
Please see this link https://stackoverflow.com/a/68598528/12469120 for more context about how to use DatePicker.
Her is my code.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome()
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source":
"const newProto = navigator.__proto__;"
"delete newProto.webdriver;"
"navigator.__proto__ = newProto;"
})
driver.get("https://www.nseindia.com/products/content/derivatives/equities/archieve_fo.htm")
time.sleep(5)
datepicker = driver.find_element_by_id("date")
datepicker.click()
selectMonth = driver.find_element_by_xpath('//select[#class="ui-datepicker-month"]')
for option in selectMonth.find_elements_by_tag_name('option'):
if option.text == 'Mar':
option.click()
break
selectYear = driver.find_element_by_xpath('//select[#class="ui-datepicker-year"]')
for option in selectYear.find_elements_by_tag_name('option'):
if option.text == '2018':
option.click()
break
days = driver.find_elements_by_xpath('//a[#class="ui-state-default"]')
days[4].click()
time.sleep(2)
select = Select(driver.find_element_by_id('h_filetype'))
select.select_by_visible_text('Market Activity Report')
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()
time.sleep(20)
Update
The site can detect Selenium driven browsers. A workaround is added in the code. The download link can be obtained.

The issue is that, there is no file available for the specified date.
No file found for specified date. Try another date.
Though, there were some tweak needed, I have made some changes, see below :-
driver = webdriver.Chrome()
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.nseindia.com/products/content/derivatives/equities/archieve_fo.htm")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[id='date']"))).click()
select_month = Select(wait.until(EC.visibility_of_element_located((By.XPATH, "//select[#class='ui-datepicker-month']"))))
select_month.select_by_visible_text('Mar')
select_year = Select(wait.until(EC.visibility_of_element_located((By.XPATH, "//select[#class='ui-datepicker-year']"))))
select_year.select_by_visible_text('2017')
days = driver.find_elements_by_xpath("//a[#class='ui-state-default']")
days[3].click()
time.sleep(2)
select = Select(wait.until(EC.element_to_be_clickable((By.ID, "h_filetype"))))
select.select_by_value('fomkt')
element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()
Update 1:
look, this css selector represent the 'file not available' message
div#spanDisplayBox td
it's actually working manually, but hangs with automation (chrome browser). I have faced this issue earlier as well with the same site. However, I have added a method that could help us in following way :
element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()
try:
if(len(driver.find_elements(By.CSS_SELECTOR, "div#spanDisplayBox td"))) > 0:
print("Response is present")
print(wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#spanDisplayBox td"))).text)
else:
print("There is no response")
except:
print("something went wrong")
pass
However, when I ran this I got this output :-
There is no response
I would recommend you to check this with Firefox, (cross browser, cross platform and see if that helps).

Selenium button click in loop fails after first try

Currently I am working on a web crawler that should be able to download text of a dutch newspaper bank. The first link is working correctly but suddenly the second link creates an error of which I do not know how to fix this.
It seems that selenium is unable to click the button in the second link while it succeeds doing so in the first link.
Do you know what causes the second link (telegraaf page) to fail?
UPDATE CODE:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
import pandas as pd
import numpy as np
import re
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.action_chains import ActionChains
#Set up the path to the chrome driver
driver = webdriver.Chrome()
html = driver.find_element_by_tag_name('html')
all_details = []
for c in range(1,2):
try:
driver.get("https://www.delpher.nl/nl/kranten/results?query=kernenergie&facets%5Bpapertitle%5D%5B%5D=Algemeen+Dagblad&facets%5Bpapertitle%5D%5B%5D=De+Volkskrant&facets%5Bpapertitle%5D%5B%5D=De+Telegraaf&facets%5Bpapertitle%5D%5B%5D=Trouw&page={}&sortfield=date&cql%5B%5D=(date+_gte_+%2201-01-1970%22)&cql%5B%5D=(date+_lte_+%2201-01-2018%22)&coll=ddd".format(c))
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
incategory = driver.find_elements_by_class_name("search-result")
print(driver.current_url)
links = [ i.find_element_by_class_name("thumbnail search-result__thumbnail").get_attribute("href") for i in incategory]
# Lets loop through each link to acces the page of each book
for link in links:
# get one book url
driver.get(link)
# newspaper
newspaper = driver.find_element_by_xpath("//*[#id='content']/div[2]/div/div[2]/header/h1/span[2]")
# date of the article
date = driver.find_element_by_xpath("//*[#id='content']/div[2]/div/div[2]/header/div/ul/li[1]")
#click button and find title
div_element = WebDriverWait(driver, 60).until(expected_conditions.presence_of_element_located((By.XPATH,'//*[#id="object"]/div/div/div')))
hover = ActionChains(driver).move_to_element(div_element)
hover.perform()
div_element.click()
button = WebDriverWait(driver, 90).until(expected_conditions.presence_of_element_located((By.XPATH, '//*[#id="object-viewer__ocr-button"]')))
hover = ActionChains(driver).move_to_element(button)
hover.perform()
button.click()
element = driver.find_element_by_css_selector(".object-viewer__ocr-panel-results")
driver.execute_script("$(arguments[0]).click();", element)
# content of article
try:
content = driver.find_elements_by_xpath("//*[contains(text(), 'kernenergie')]").text
except:
content = None
# Define a dictionary with details we need
r = {
"1Newspaper":newspaper.text,
"2Date":date.text,
"3Content":content,
}
# append r to all details
all_details.append(r)
except Exception as e:
print(str(e))
pass
# save the information into a CSV file
df = pd.DataFrame(all_details)
df = df.to_string()
time.sleep(3)
driver.close()

So you have some problems.
driver.implicitly_wait(10)
Should only be used once
links = [ i.find_element_by_class_name("search-result__thumbnail-link").get_attribute("href") for i in incategory]
Is a more useful way to get all links
print(driver.current_url)
Could replace
print("https://www.delpher.nl/nl/kranten/results?query=kernenergie&facets%5Bpapertitle%5D%5B%5D=Algemeen+Dagblad&facets%5Bpapertitle%5D%5B%5D=De+Volkskrant&facets%5Bpapertitle%5D%5B%5D=De+Telegraaf&facets%5Bpapertitle%5D%5B%5D=Trouw&page={}&sortfield=date&cql%5B%5D=(date+_gte_+%2201-01-1970%22)&cql%5B%5D=(date+_lte_+%2201-01-2018%22)&coll=ddd".format(c))
No need for url=link
for link in links:
driver.get(link)
Your title actually doesn't get on the second page. Use something like this for all values.
try:
content = driver.find_element_by_xpath('//*[#id="object-viewer__ocr-panel"]/div[2]/div[5]').text
except:
content = None
# Define a dictionary
r = {
"1Newspaper":newspaper,
"2Date":date,
"3Title": title,
"4Content": content,
}
You can replace your exception with to figure out the line of problem.
except Exception as e:
print(str(e))
pass

it might be that the button you are trying to reach is inside an iframe, which means you have to access that one before searching for the XPATH:
iframe = driver.find_elements_by_tag_name('iframe')
driver.switch_to.frame(iframe)
Also there might be a possibility the object youre trying to click is not visible yet, which could be solved by an timeout

Using Selenium to get past dropdown (md-select)

I'm stuck with a dropdown that I can't get pass in Selenium.
I'm trying to collect some price data using Selenium from this link:
https://xxx. In this link, you need to click on a button (Next), then select any option in the subsequent dropdown, then press (Next) again to advance into the information page that I wanted to collect some information. I'm stuck at the dropdown - I am unable to select any option.
This is my code thus far:
browser.get("https://xxx/#/pricePlans/step1")
wait = WebDriverWait(browser, 10)
while True:
try:
button = browser.find_element_by_css_selector('body > div.md-dialog-container.ng-scope > md-dialog > md-dialog-actions > div > button')
except TimeoutException:
break
button.click()
options_box= browser.find_element_by_class_name('bullet-content-title')
wait = WebDriverWait(browser, 5)
options_box.click()
The issue lies with the dropdown options (It has options like HDB 1-room, HDB 2-room etc). I tried to reference the option box by XPATH, CSS selector, class_name (as seen above) but with the snippet above, Spyder issues time-out. Other snippets I tried included:
ui.WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, "bullet-content-title")))
using XPATH, class_name but no luck.
I'm a newbie at web scraping who got thus far by searching the SO, but I am unable to find much solutions regarding (md-select) dropdowns.
I also attempted to use
ActionChains(driver).move_to_element(options_box).click(options_box)
but I did not see any clicking nor mouse movements so i'm stumped.
I appreciate any advice at this point of time. Thank you so much!
Edit:
Code Snippets and Responses:
from selenium import webdriver
from selenium.webdriver.support import ui
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.action_chains import ActionChains
option = webdriver.ChromeOptions()
option.add_argument('--incognito')
browser = webdriver.Chrome(executable_path='C:\\ChromeDriver\\chromedriver.exe', options=option)
browser.get("https://xxx")
wait = WebDriverWait(browser, 10)
while True:
try:
button = browser.find_element_by_css_selector('body > div.md-dialog-container.ng-scope > md-dialog > md-dialog-actions > div > button')
except TimeoutException:
break
button.click()
options_box = browser.find_element_by_class_name('bullet-content-title')
wait = WebDriverWait(browser, 5)
options_box.click()
This returns "StaleElementReferenceException: stale element reference: element is not attached to the page document"
Which I assume it is due to the presence of the second "Next" Button which is inert at the moment.
options_box = ui.WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, "bullet-content-title")))
options_box.click()
Does nothing. Spyder eventually returned me TimeOut Error.

#AndrewRay answer is good for getting the value but not for selecting the options. you can do this to select the options.
#browser.get("https://......")
wait = WebDriverWait(browser, 10)
try:
browser.find_element_by_css_selector('button.green-btn').click()
# wait until dialog dissapear
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, 'md-dialog[aria-describedby="dialogContent_0"]')))
# click the dropdown
browser.find_element_by_css_selector('md-input-container').click()
# select the option element
setOptionElement = browser.find_element_by_css_selector('md-option[value="HDB Executive"]')
# need to scrollIntoView if the option in the bottom
# or you get error the element not clickable
browser.execute_script('arguments[0].scrollIntoView();arguments[0].click()', setOptionElement)
except Exception as ex:
print(ex)

driver.get('https://compare.openelectricitymarket.sg/#/pricePlans/step1')
time.sleep(5)
next_btn = driver.find_element_by_css_selector('button.green-btn')
next_btn.click()
dropdown = driver.find_element_by_id('select_4')
options = dropdown.find_elements_by_tag_name('md-option')
for option in options:
print option.get_attribute('value')
Hope this helps. Use the .get_attribute method to find the value of the option and click that option if matches the desired value. :)

Selenium Element not visible exception

I have been tasked with writing a parser to click a button on a website and I am having issues to click only one of the buttons. The following code works on every button except one.
Here's the html:
http://pastebin.com/6dLF5ru8
here's the source html:
http://pastebin.com/XhsedGLb
python code:
driver = webdriver.Firefox()
...
el = driver.find_element_by_id("-spel-nba")
actions.move_to_element(el)
actions.sleep(.1)
actions.click()
actions.perform()
I am getting this error.
ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
as per Saifur I just tried waits with the same element not visible exception:
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.XPATH, "//input[contains(#id,'spsel')][#value='nba']"))).click()

If you look at the page source, you'll understand that almost all of theSELECT, DIV elements are faked and created from JavaScript, that is why webdriver cannot SEE them.
There's a workaround though, by using ActionChains to open your developer console, and inject an artificial CLICK on the desired element, which in fact, is the Label triggering the NBA data loading... here's a working example:
from selenium import webdriver
from selenium.webdriver.common import action_chains, keys
import time
driver = webdriver.Firefox()
driver.get('Your URL here...')
assert 'NBA' in driver.page_source
action = action_chains.ActionChains(driver)
# open up the developer console, mine on MAC, yours may be diff key combo
action.send_keys(keys.Keys.COMMAND+keys.Keys.ALT+'i')
action.perform()
time.sleep(3)
# this below ENTER is to rid of the above "i"
action.send_keys(keys.Keys.ENTER)
# inject the JavaScript...
action.send_keys("document.querySelectorAll('label.boxed')[1].click()"+keys.Keys.ENTER)
action.perform()
Alternatively to replace all the ActionChains commands, you can simply run execute_script like this:
driver.execute_script("document.querySelectorAll('label.boxed')[1].click()")
There you go, at least on my local file anyway... Hope this helps!

What worked for me was to find the element just before the problematic one (that is, just before it in terms of tab order), then call Tab on that element.
from selenium.webdriver.common.keys import Keys
elem = br.find_element_by_name("username")
elem.send_keys(Keys.TAB) # tab over to not-visible element
After doing that, I was able to send actions to the element.

The actual solution of this thread did not work for me.
however,
this one did :
element = WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.XPATH, xpaths['your_xpath_path'])))
the trick is to use :
EC.visibility_of_element_located
the WebDriverWait
WebDriverWait
from this import :
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

I suggest you use xpath with explicit wait
//input[contains(#id,'spsel')][#value='nba']

if "Element is not currently visible" then make it VISIBLE
f.e.
>>> before is hidden top is outside of page
<input type="file" style="position: absolute;top:-999999" name="file_u">
>>> after move top on in page area
DRIVER.execute_script("document.getElementByName('file_u').style.top = 0;")
time.sleep(1); # give some time to render
DRIVER.find_element_by_name("file_u").send_keys("/tmp/img.png")

Instead of get_element_by_id() you can try elem = browser.find_element_by_css_selector('#elemId') (go to that webpage and the element, right click it and Copy CSS Selector, or something like that.) This is what i did and it works. You also try find_element_by_link_text(text), find_element_by_partial_link_text(text), find_element_by_tag_name(tagName_case_insensitive_here), find_element_by_name(name) etc. Something will work. After the id the CSS Selector is your best bet.

I ended up using #twasbrillig's solution, but instead of finding the previous element and sending a TAB keypress, I find the desired element, send a TAB keypress with that element, and then a SHIFT + TAB keypress to the driver:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
el = driver.find_element_by_id("-spel-nba")
el.send_keys(Keys.TAB)
webdriver.ActionChains(driver).key_down(Keys.SHIFT).send_keys(Keys.TAB).key_up(Keys.SHIFT)

I tried using the other methods but in the end found that the simplest way was to just try and click the button, and catch the error. This allows me to perform other actions based on if it worked (True) or didn't (False).
def click_button(html_object):
try:
html_object.click()
except:
return False #most likely because it is NotVisible object and can be ignored
return True
...
...
click_button(actions)

The way I solved this in python was:
try:
# the element you want to scroll to
element = driver.find_element_by_id("some_id")
ActionChains(driver).move_to_element(element).perform()
element.send_keys(Keys.TAB).key_up(Keys.SHIFT)
#element.click()
except Exception as e:
print(e)