Develop Reference

Python/Selenium: Select all identical classes of several dropdown menus, select a value, refresh browser and restart script

I work with a CMS where I have to change a user status from "Pending" to "Confirmed". If I do it manually I have to click on a dropdown value. There are 12 dropdowns on the page the dropdown value is always the same. Then I have to refresh the browser and click through the next 12 dropdowns.
I wrote this Python script with Selenium:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(options=options)
driver.get("https://url.com")
select = Select(driver.find_element_by_class_name('address-status'))
select.select_by_value('1')
time.sleep(1)
driver.close()
This is the html for the dropdown:
<select name="data[status]" class="address-status" data-id="44980" id="status">
<option value="2">Post card sent</option>
<option value="1">Confirmed</option>
<option value="0" selected="selected">Pending</option>
</select>
My problems which I can't figure out on my own are:
The script only selects the first class of 'adress-status' and selects the value I want but then it stops. But there are 11 more identical classes/dropdown menus where the (same) value has to be selected.
I don't know how to refresh the page to get the next set of dropdowns and restart the selection process/script.
I figured out that I may have to loop parts of the script but I learned and used Python for the first time yesterday and I just can't figure it out in time for this task.

This is my solution
Find all drop down element. It will return a list of all web element have class name = "address-status"
Using loop to select confirmed value
# find all dropdown element
list_drop_down = driver.find_elements_by_class_name("address-status")
# select confirmed value for each dropdown
for drop_down_element in list_drop_down:
select = Select(drop_down_element)
select.select_by_value('1')

I had similar issue. You need to select address-status element by index.
Find no of address-status elements
n_elements = len(driver.find_elements_by_class_name('address-status'))
after each refresh find all the elements again, but choose the one at different index
for idx in range(n_elements):
element = driver.find_elements_by_class_name('address-status')[idx]
# do what you need with the element
...
# refresh the page
...

After learning more about Python and adding all I've learned to the script I finally achieved every step I mentioned in my intial posting.
The steps are:
Open Chrome
I'm opening webdriver.Chrome with my user data because I need to stay logged in to the CMS I'm working with. I'm achieving this by adding an argument to options
Find all identical classes
This particular class represents a dropdown menu
Loop through all dropdowns
Select value
Time sleep of 1s
For some reason the script sometimes won't select all values so this helped. This might have to do with the closing of the browser in the next step but I'm not sure.
Close Chrome
In my case this is necessary to save the selected values to the CMS
Restart script
I defined a while loop of 40 repetitions. Be aware of IndentationError meaning to add a tab to the looped elements (I know this is basic stuff but might be useful for a newbie like me). Otherwise the script won't work. Also see
Nested for loop IndentationError: expected an indented block when using calendar module
Here is my code:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
import time
count = 0
while count < 40:
print(count)
options = Options()
options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(options=options)
driver.get(
"https://URL")
# find all dropdown element
list_drop_down = driver.find_elements_by_class_name("address-status")
# select confirmed value for each dropdown
for drop_down_element in list_drop_down:
select = Select(drop_down_element)
select.select_by_value('1')
time.sleep(1)
driver.close()
count = count+1
Thanks to the user Trinh Phat who gave me the necessary push. My work place loves me now ;)

How to stop the automatic scroll-down when user scroll up?

I want to stop the automatic scroll down when user scroll up by mouse. I have made a function that scroll's down the webpage automatically after every 2 second but I want to stop that automatic scroll when user scroll up by mouse
My code:
from selenium import webdriver
import time
driver=webdriver.Chrome()
driver.get("https://YouTube.com")
driver. maximize_window()
a=input('press y if you want to scroll down')
if a=='y':
while(True) :
driver. execute_script("window.scrollBy(0, 150) ", "")
time.sleep(2)
else:
Print(" Ok")

One of the ways in which you can solve this is using the window.pageYOffset property.
From the Docs
The read-only Window property pageYOffset is an alias for scrollY; as such, it returns the number of pixels the document is currently scrolled along the vertical axis (that is, up or down) with a value of 0.0, indicating that the top edge of the Document is currently aligned with the top edge of the window's content area.
So what you will have to do is each time you scroll in the loop keep a count of the increase in y and for the condition of your loop, you should check that the pageYOffset == YScrolled. If it equals there's no user input, otherwise there is.
There is a good example here

Can't make webdriver.click() or .submit() work in loop (python)

I want to scrape data from an HTML table for different combinations of drop-down values via looping over those combinations. After a combination is chosen, the changes need to be submitted. This is, however, causing an error since it refreshes the page.
This is what I've done so far:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
browser.get('https://daten.ktbl.de/feldarbeit/entry.html')
# Selecting the constant values of some of the drop downs:
fertilizer = Select(browser.find_element_by_name("hgId"))
fertilizer.select_by_value("2")
fertilizer = Select(browser.find_element_by_name("gId"))
fertilizer.select_by_value("193")
fertilizer = Select(browser.find_element_by_name("avId"))
fertilizer.select_by_value("383")
fertilizer = Select(browser.find_element_by_name("hofID"))
fertilizer.select_by_value("2")
# Looping over different combinations of plot size and amount of fertilizer:
size = Select(browser.find_element_by_name("flaecheID"))
for size_values in size.options:
size.select_by_value(size_values.get_attribute("value"))
time.sleep(1)
amount= Select(browser.find_element_by_name("mengeID"))
for amount_values in amount.options:
amount.select_by_value(amount_values.get_attribute("value"))
time.sleep(1)
#Refreshing the page after the two variable values are chosen:
button = browser.find_element_by_xpath("//*[#type='submit']")
button.click()
time.sleep(5)
This leads to the error: selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <option> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed.
Obviously the issue is that I did indeed refresh the document.
I tried it with .submit():
button = browser.find_element_by_xpath("//*[#type='submit']")
button.submit()
This leads to the error: selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: ./ancestor-or-self::form
Time seems not to be the issue here (adding time.sleep or b.implicit.wait didn't change anything).
How can I update the page without breaking from the loop?
EDIT since it was marked as duplicate: I believe that this question does not answer my problem posed here, at least not on my level of python skills, even though the error message is similar. In the other question the problem was solved via added a wait(driver, x) and an expected condition if I'm not mistaken. Here on the other hand the issue was indeed that I did not call the relevant drop-down inside AND outside the loop, as correctly stated by Zaraki Kenpachi.

You need to initiate size select outside the loop to collect sizes and inside loop to grab current page state.
size = Select(browser.find_element_by_name("flaecheID"))
for size in [item.get_attribute("value") for item in size.options]:
size_select = Select(browser.find_element_by_name("flaecheID"))
size_select.select_by_value(str(size))
button = browser.find_element_by_xpath("//input[#type='submit']")
button.click()

Selenium Python Chrome: Last element in drop down list is not visible in Headless mode

I want to use Selenium to click an element in a menu drop down list. Unfortunately the element I want to click is all the way at the bottom of the list. My Selenium Python Chrome code runs perfectly in foreground mode, but when I run it in background / headless mode I get a timeout, or not clickable error (see notes below). Here is the code, using the actual public url, so you can test it out if you want:
# define chrome options
options = webdriver.ChromeOptions()
# make sure browser is maximized so all elements are visible
options.add_argument('--start-maximized')
# define driver
driver = webdriver.Chrome(chrome_options=options)
# define function used to get button
def getbutton(driver, xpath, waittime=10):
return WebDriverWait(driver, waittime).until(EC.element_to_be_clickable((By.XPATH, xpath)))
# navigate to url
driver.get('http://oasis.caiso.com/mrioasis/logon.do')
# another two different ways to make sure browser window is maximized
# I'm going through this trouble just in case selenium needs the window
# maximized to see the element
driver.fullscreen_window()
driver.maximize_window()
# choose dropdown menu
button1 = getbutton(driver, "//span[contains(#id, 'LBL_')][text()='ATLAS REFERENCE']", waittime=120)
hover = ActionChains(driver).move_to_element(button1)
hover.perform()
time.sleep(5)
# see about navigating to the second to last button in the drop down
# menu list - not what I want, but a test
testing = True
if testing:
# move to second to last button - interestingly this works in foregraound AND background / headless modes
button1a = getbutton(driver, "//span[contains(#id, 'LBL_')][text()='Intertie Scheduling Limit and Tie Mapping']")
hover = ActionChains(driver).move_to_element(button1a)
# THIS WORKS in foreground AND backgroubd modes
hover.perform()
time.sleep(10)
# Now try and click the button I really want - the LAST button in the drop down menu list
button2 = getbutton(driver, "//span[contains(#id, 'LBL_')][text()='Master Control Area Generating Capability List']", waittime=120)
button2.click()
button2 = getbutton fails with the following error message:
selenium.common.exceptions.TimeoutException
Interestingly, if I turn testing to False, I get the following error message:
selenium.common.exceptions.WebDriverException: Message: unknown error:
Element <span ...></table>">Master Control Area Generating Capability List</span>
is not clickable at point (128, 599). Other element would receive the click
It seems like selenium is unable to see the element in background / headless mode, even though neither of these exceptions occur when running in foreground mode.
I'm running Python in Windows Server 2012 R2 for what it's worth.
Thanks for any help!

I've changed a few things in your code and I no longer get the errors. I'm not sure if it's loading the correct webpage though since it's headless and everything seems to be dynamically loaded.
# another two different ways to make sure browser window is maximized
# I'm going through this trouble just in case selenium needs the window
# maximized to see the element
#driver.fullscreen_window()
#driver.maximize_window()
# choose dropdown menu
button1 = driver.find_element_by_xpath("//span[contains(#id, 'LBL_')][text()='ATLAS REFERENCE']")
hover = ActionChains(driver)
hover.move_to_element(button1)
# see about navigating to the second to last button in the drop down
# menu list - not what I want, but a test
testing = True
if testing:
# move to second to last button - interestingly this works in foregraound AND background / headless modes
button1a = driver.find_element_by_xpath("//span[contains(#id, 'LBL_')][text()='Intertie Scheduling Limit and Tie Mapping']")
hover.move_to_element(button1a)
# THIS WORKS in foreground AND backgroubd modes
# Now try and click the button I really want - the LAST button in the drop down menu list
button2 = driver.find_element_by_xpath("//span[contains(#id, 'LBL_')][text()='Master Control Area Generating Capability List']")
hover.click(button2)
hover.perform()
I'm selecting the elements with .find_element_by_xpath() and I'm queuing up all the ActionChains to execute them all at the end with hover.perform().
After further analysis, I'm unable to locate the "masterControlAreaGenCapGrid_GRID_LABEL" element with driver.find_element_by_id("masterControlAreaGenCapGrid_GRID_LABEL").text
It works when the browser isn't headless though...

So I finally figured out what the problem is: Running Windows task scheduler with option 'run whether user is logged on or not' only opens a small browser (1024x768) that CANNOT be resized, even with all the great suggestions being offered here. So I simply cannot see the element I want to click.
See the same issue resolved here: screen resolution in mode "Run whether user is logged on or not", in windows task scheduler
So the less than ideal workaround is to only run when user is logged on.
Thanks for all your help!

Selenium - Click at certain position

Using the Python version of Selenium, is it possible to click some element in the DOM and to specify the coordinates where you want to click it?
The Java version has the method clickAt, which actually does exactly what I am looking for, but can't find the equivalent in Python.

This should do it! Namely you need to use action chains from webdriver. Once you have an instance of that, you simply register a bunch of actions and then call perform() to perform them.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com")
el=driver.find_elements_by_xpath("//button[contains(string(), 'Lucky')]")[0]
action = webdriver.common.action_chains.ActionChains(driver)
action.move_to_element_with_offset(el, 5, 5)
action.click()
action.perform()
This will move the mouse 5 pixels down and 5 pixels right from the upper-left corner of the button I feel lucky. Then it will click().
Notice that you must use perform(). Else nothing will happen.

The reason you are getting confused is clickAt is an old v1 (Selenium RC) method.
WebDriver has a slightly different concept, of 'Actions'.
Specifically, the 'Actions' builder for the Python bindings live here.
The idea of the clickAt command is to click at a certain position relative to a particular element.
The same is achievable within the WebDriver, using the 'Actions' builder.
Hopefully this updated documentation can help.

You can perform the task with the Action chains in the python specially with Edge browser:
from selenium.webdriver import ActionChains
actionChains = ActionChains(driver)
button_xpath = '//xapth...'
button = driver.find_element_by_xpath(button_xpath)
actionChains.move_to_element(button).click().perform()
But sometimes Action chain does not finds the DOM element. Hence better option to use execute scipt in following way:
button_xpath = '//xapth...'
button = driver.find_element_by_xpath(button_xpath)
driver.execute_script("arguments[0].click();", button)

I've not personally used this method, but looking through the source code of selenium.py I've found the following methods that look like they'd do what you want - They look to wrap clickAt:
def click_at(self,locator,coordString):
"""
Clicks on a link, button, checkbox or radio button. If the click action
causes a new page to load (like a link usually does), call
waitForPageToLoad.
'locator' is an element locator
'coordString' is specifies the x,y position (i.e. - 10,20) of the mouse event relative to the element returned by the locator.
"""
self.do_command("clickAt", [locator,coordString,])
def double_click_at(self,locator,coordString):
"""
Doubleclicks on a link, button, checkbox or radio button. If the action
causes a new page to load (like a link usually does), call
waitForPageToLoad.
'locator' is an element locator
'coordString' is specifies the x,y position (i.e. - 10,20) of the mouse event relative to the element returned by the locator.
"""
self.do_command("doubleClickAt", [locator,coordString,])
They appear in the selenium object and here is their online API documentation.