I have been trying for the past couple of days to select a dropdown and at least print out the options available, but I just cannot get it to work.
I am getting the this error when I run the module.
Traceback (most recent call last):
File "sel_test_elements2.py", line 20, in
print ([o.text for o in select_element.options])
AttributeError: 'FirefoxWebElement' object has no attribute 'options'
Currently my code looks like this.
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
# Define Global Variables
url = "https://games.pcaha.ca/teams/4329"
csv_file = "game_schedule_4329.csv"
games = []
# create a new Firefox session
driver = webdriver.Firefox()
driver.get(url)
driver.implicitly_wait(30)
# Locate the Sector and create a Select object
select_element = driver.find_element_by_css_selector(".team-filters")
# this will print out strings available for selection on select_element, used in visible text below
print ([o.text for o in select_element.options])```
The issue you face is the fact that this website is using react and doesn't use default Select and Options. They have a custom dropdown implemented, so the way to interact with it is the same as interaction with regular web elements, Select and Options won't work in this case.
I modified your code and it works for me in Chrome:
from selenium.webdriver import Chrome
from time import sleep
# Define Global Variables
url = "https://games.pcaha.ca/teams/4329"
csv_file = "game_schedule_4329.csv"
games = []
# create a new Chrome session
driver = Chrome()
driver.get(url)
driver.implicitly_wait(30)
sleep(3) # make sure svgs load before interaction
# Click on arrow down
arrow = driver.find_elements_by_css_selector(".team-filters svg")[1].click()
# Collect options
options = driver.find_elements_by_xpath("//div[contains(#id, 'react-select-2')]")
# Print text from options
print([o.text for o in options])
Note: when manually opening the dropdown in your browser and trying to use web inspector, it closes, so in order to get the html inside a dropdown, you can use something like:
dropdown = driver.find_element_by_css_selector("div.css-kj6f9i-menu")
dropdown_html = dropdown.get_attribute('innerHTML')
I hope it helped. Good luck!
I have used something similar in a small script i have written, may be it can give you an hint on how to go about
Approach 1 This is to select the last of the options available
Variable options in the code below gets be the option available for the dropdown
select_datebox = driver.find_element_by_id('jrnyDateSrchTxt') # Drop down selection, you have to change the id appropriately
select_datebox.click()
time.sleep(2)
options = select_datebox.find_elements_by_tag_name('option')
options[len(options)-1].click() #selecting the last option
Approach 1 entering the option via a variable
select = Select(driver.find_element_by_id("jrnyDateSrchTxt")) # Drop down selection, you have to change the id appropriately
time.sleep(1)
select.select_by_value(datadate) # Date selection
time.sleep(2)
Related
I am building a web scraper that has to try a combination of multiple drop-down menu options and gather data from each combination.
So basically there're 5 drop-downs. I have to gather data from all of the possible combinations of the drop-down options. For each combination, I have to press a button to pull up the page with all the data on it. I am storing all the data in a dictionary.
This is the website: http://siops.datasus.gov.br/filtro_rel_ges_covid_municipal.php?S=1&UF=12;&Municipio=120001;&Ano=2020&Periodo=20
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
# General Stuff about the website
path = '/Users/admin/desktop/projects/scraper/chromedriver'
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")
driver = webdriver.Chrome(options=options, executable_path=path)
website = 'http://siops.datasus.gov.br/filtro_rel_ges_covid_municipal.php'
driver.get(website)
# Initial Test: printing the title
print(driver.title)
print()
# Dictionary to Store stuff in
totals = {}
# Drop Down Menus
year_select = Select(driver.find_element(By.XPATH, '//*[#id="cmbAno"]'))
uf_select = Select(driver.find_element(By.XPATH, '//*[#id="cmbUF"]'))
### THIS IS WHERE THE ERROR IS OCCURING ###
# Choose from the drop down menus
uf_select.select_by_value('29')
year_select.select_by_value('2020')
# Submit button on the page
submit_button = driver.find_element(By.XPATH, '//*[#id="container"]/div[2]/form/div[2]/div/input[2]')
submit_button.click()
# Pulling data from the webpage
nameof = driver.find_element(By.XPATH, '//*[#id="arearelatorio"]/div[1]/div/table[1]/tbody/tr[2]').text
total_balance = driver.find_element(By.XPATH, '//*[#id="arearelatorio"]/div[1]/div/table[3]/tbody/tr[9]/td[2]').text
paid_expenses = driver.find_element(By.XPATH, '//*[#id="arearelatorio"]/div[1]/div/table[4]/tbody/tr[11]/td[4]').text
# Update Dictionary with the new info
totals.update({nameof: [total_balance, paid_expenses]})
totals.update({'this is a test': ['testing stuff']})
# Print the final Dictionary and quit
print(totals)
driver.quit()
For some reason, this code does not work when trying 1 possible combination (selecting value 29 from the UF drop-down, as well as value 2020 from the year_select drop-down). If I comment out of the two drop-down selections, then it works perfectly fine.
How do I try multiple combinations of drop-down options during a single iteration?
try this instead.
# Drop Down Menus
### THIS IS WHERE THE ERROR IS OCCURING ###
# Choose from the drop down menus
uf_select = Select(driver.find_element(By.XPATH, '//*[#id="cmbUF"]'))
uf_select.select_by_value('29')
year_select = Select(driver.find_element(By.XPATH, '//*[#id="cmbAno"]'))
year_select.select_by_value('2020')
This works for me. With your example i get a stale... error, means that the element disappears. Haven´t checked, but maybe the checkbox is somehow updated and looses reference when selecting the other one.
I am trying to select dropdown year of 2021 on (https://www.theknot.com/registry/couplesearch) and am unable to figure out how to use the dropdown.
#This code is working
typetextfirst = driver.find_element_by_id("couples-search-first-name")
typetextfirst.clear()
typetextfirst.send_keys(row["First"])
typetextlast = driver.find_element_by_id("couples-search-last-name")
typetextlast.clear()
typetextlast.send_keys(row["Last"])
typetextyear = driver.find_element_by_id("couples-search-year")
#None of these options work to populate the year
typetextyear.selectByIndex(1)
typetextyear.select_by_index(1)
typetextyear.selectByVisibleText("2021")
typetextyear.select_by_visible_text("2021")
#This code is working
typetextlast.send_keys(Keys.ENTER)
Page doesn't use standard dropdown widget but it uses button and ul to emulate dropdown.
This code works for me on Firefox and Chrome on Linux Mint.
First I click button to open dropdown created with ul and later I search li with expected text and click it.
Because it may have text 2021 with some spaces/tabs/enters (which browser doesn't show) so I prefer contains instead of =
from selenium import webdriver
url = 'https://www.theknot.com/registry/couplesearch'
driver = webdriver.Firefox()
#driver = webdriver.Chrome()
driver.get(url)
year_dropdown = driver.find_element_by_id("couples-search-year")
year_dropdown.click()
year = year_dropdown.find_element_by_xpath(".//li[contains(text(), '2021')]")
#year = year_dropdown.find_element_by_xpath(".//li[text()='2021']")
year.click()
My code runs fine and prints the title for all rows but the rows with dropdowns.
For example, row 4 has a dropdown if clicked. I implemented a try which would in theory initiate the dropdown, to then pull the titles.
But my click/scrape for the rows with these drop downs are not printing.
Expected output- Print all titles including the ones in dropdown.
from selenium import webdriver
from bs4 import BeautifulSoup
import time
driver = webdriver.Chrome()
driver.get('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')
time.sleep(4)
page_source = driver.page_source
soup = BeautifulSoup(page_source,'html.parser')
productlist=soup.find_all('div',class_='card item-container session')
for property in productlist:
sessiontitle=property.find('h4',class_='session-title card-title').text
print(sessiontitle)
try:
ifDropdown=driver.find_elements_by_class_name('item-expand-action expand')
ifDropdown.click()
time.sleep(4)
newTitle=driver.find_element_by_class_name('card-title').text
print(newTitle)
except:
newTitle='none'
There were a couple of issues. First, when you locate from the driver by class and there is more than one, you need to separate them by dots, not spaces, so that the driver knows it's dealing with another class.
Second, find_elements returns a list, and the list has no .click(), so you get an error, which your except catches but assumes means there was no link to click.
I rewrote it (without soup for now) so that it instead checks (With the dot replacing space) for a link to open within the session and then loops over the new ones that appeared.
Here is what I have and tested. Note at the end this only gets the sessions and subsessions in the view. You will need to add logic to scroll and get the rest.
# stuff to initialize driver is above here, I used firefox
# Open the website page
URL = "https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list"
driver.get(URL)
time.sleep(4)#time for page to populate
product_list=driver.find_elements_by_css_selector('div.card.item-container.session')
#above line gets all top level sessions
for product in product_list:
session_title=product.find_element_by_css_selector('h4.card-title').text
print(session_title)
dropdowns=product.find_elements_by_class_name('item-expand-action.expand')
#above line finds dropdown within this session, if any
if len(dropdowns)==0:#nothing more for this session
continue#move to next session
#still here, click on the dropdown, using execute because link can overlap chevron
driver.execute_script("arguments[0].scrollIntoView(true); arguments[0].click();",
dropdowns[0])
time.sleep(4)#wait for subsessions to appear
session_titles=product.find_elements_by_css_selector('h4.card-title')
session_index = 0#suppress reprinting title of master session
for session_title in session_titles:
if session_index > 0:
print(" " + session_title.text)#indent for clarity
session_index = session_index + 1
#still to do, deal with other sessions that only get paged into view when you scroll
#that is a different question
I'am new to mobile automation. I thought I got the gist, but I ran into a problem.
An error occurs when using the command send_keys()
selenium.common.exceptions.InvalidElementStateException: Message: Cannot set the element to '321785214'. Did you interact with the correct element?
I have a code
driver = webdriver.Remote("http://localhost:4723/wd/hub", desired_cap)
# Skip fingerprint
driver.find_element_by_xpath("//android.widget.Button[#bounds='[413,1802][668,1877]']").click()
# Click Continue
driver.find_element_by_xpath("//android.widget.Button[#content-desc='Continue']").click()
# Click to input phone number
input_phone_number = driver.find_element_by_xpath("//android.widget.EditText").click()
input_phone_number_edit = driver.find_element_by_class_name("android.widget.EditText")
# Enter phone number
input_phone_number_edit.send_keys("321785214")
And I have code from other mobile app and this code is working
driver = webdriver.Remote("http://localhost:4723/wd/hub", desired_cap)
# click to Set up later button
set_up_later_btn = driver.find_element_by_xpath('//android.widget.Button[#content-desc="Set up later"]').click()
input_phone_number = driver.find_element_by_xpath('//android.view.View[#content-desc="Phone number"]').click()
input_phone_number_edit = driver.find_element_by_class_name('android.widget.EditText')
driver.implicitly_wait(50)
input_phone_number_edit.send_keys('178546128')
driver.find_element_by_accessibility_id("Continue").click()
I don't know why first code isn't working. I need help. Thanks
Instead of using the default send_keys method, you can use the send_keys method provided by the ActionChains class
from selenium.webdriver import ActionChains
actions = ActionChains(self.driver)
actions.send_keys("321785214")
actions.perform()
I am trying to scrape the Google News page in the following way:
from selenium import webdriver
import time
from pprint import pprint
base_url = 'https://www.google.com/'
driver = webdriver.Chrome('/home/vincent/wintergreen/chromedriver') ## change here to your location of the chromedriver
driver.implicitly_wait(30)
driver.get(base_url)
input = driver.find_element_by_id('lst-ib')
input.send_keys("brexit key dates timetable schedule briefing")
click = driver.find_element_by_name('btnK')
click.click()
news = driver.find_element_by_link_text('News')
news.click()
tools = driver.find_element_by_link_text('Tools')
tools.click()
time.sleep(1)
recent = driver.find_element_by_css_selector('div.hdtb-mn-hd[aria-label=Recent]')
recent.click()
# custom = driver.find_element_by_link_text('Custom range...')
custom = driver.find_element_by_css_selector('li#cdr_opt span')
custom.click()
from_ = driver.find_element_by_css_selector('input#cdr_min')
from_.send_keys("9/1/2018")
to_ = driver.find_element_by_css_selector('input#cdr_max')
to_.send_keys("9/2/2018")
time.sleep(1)
go_ = driver.find_element_by_css_selector('form input[type="submit"]')
print(go_)
pprint(dir(go_))
pprint(go_.__dict__)
go_.click()
This script manage to enter search terms, switch to the news tab, open the custom time period tab, fill in start and end date, but fails to click on the 'Go' button after that point.
From the print and pprint statement at the end of the script, I can deduct that it does find the 'go' button succesfully, but is somehow unable to click on it. The error displays as selenium.common.exceptions.ElementNotVisibleException: Message: element not visible
Could anyone experienced with Selenium have a quick run at it and give me hints as why it returns such error?
Thx!
Evaluating the css using developer tools in chrome yields 4 elements.
Click here for the image
use the following css instead:
go_ = driver.find_element_by_css_selector('#cdr_frm > input.ksb.mini.cdr_go')