I am trying to use Selenium to extract dynamically loaded content. The content is on http://www.afl.com.au/stats
I am attempting to navigate to the 'Players' tab, then obtain a list of all the Seasons available. When I do this from the Teams tab, the following code works:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
driver = webdriver.Chrome(executable_path=r'D:\ChromeDriver\chromedriver.exe')
driver.get('http://www.afl.com.au/stats')
dropdown_menu = Select(driver.find_element_by_xpath('//*[#id="selTeamSeason"]'))
for option in dropdown_menu.options:
print(option.text)
which gives me a list of all the options available in the Seasons tab.
However, when I click to the 'Players' tab first, I am unable to get the same list with almost identical code:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
driver = webdriver.Chrome(executable_path=r'D:\ChromeDriver\chromedriver.exe')
driver.get('http://www.afl.com.au/stats')
driver.find_element_by_xpath('//*[#id="stats_tab"]/ul/li[2]').click()
time.sleep(3)
dropdown_menu = Select(driver.find_element_by_xpath('//*[#id="selTeamSeason"]'))
for option in dropdown_menu.options:
print(option.text)
The click successfully executes, I wait for the content to update, but instead of printing all the years (2001 to 2018), Selenium prints 18 instances of empty strings. I am thoroughly stumped. Any help at all would be appreciated.
In your first case , locator(//*[#id="selTeamSeason"]) is pointing to season dropdown of Teams tab and page has only one matching node at the time ,so its working for you.
But in the second case , for the same locator there are 2 matching nodes are available and in this case selenium automatically pick the first one(Its an hidden element in your case).
So try to build a unique xpath with can work in both the tabs.
You can try //div[#id='stats-player-stats']//select[#id='selTeamSeason'] locator for season dropdown in Players tab and //div[#id='stats-team-stats']//select[#id='selTeamSeason'] for season dropdown in Teams tab
Hope this will work for you
Instead of using Select just find the xpath and get all the option tags like below, tested and works.
element = driver.find_element_by_xpath('//*[#id="selTeamSeason"]')
all_options = element.find_elements_by_tag_name("option")
for option in all_options:
print(option.text)
In your first attempt the following Locator Strategy worked on the default TEAMS TAB :
dropdown_menu = Select(driver.find_element_by_xpath('//*[#id="selTeamSeason"]'))
As the first match as per the Locator Strategy was the Dropdown itself which is not the case when dealing with PLAYERS TAB. On PLAYERS TAB to list of all the options available in the Seasons Dropdown you can use the following code block :
dropdown_menu = Select(driver.find_element_by_xpath("//div[#id='stats-player-stats']//select[#id='selTeamSeason']"))
for option in dropdown_menu.options:
print(option.text)
Related
I'm having some problems in selecting values that are in a species of Dropdown Button. I've never seen a button that works in that way, it is equal to a dropdown menu, but it is classified in the website HTML as a button. So, selenium returns me an error when try to manipulate the button as if it were a menu.
Can you please help me to know what code should I run to select a value from the first dropdown menu of this Brazilian Central Bank website? The default value is REAL (BRL) and I want to use regular expressions to select the others.
edit:
df = pd.DataFrame();
selector = Select(driver.find_element_by_id("button-converter-de"))
options = selector.options
for index in range(0, len(options)-1):
df.append(pd.DataFrame.from_dict(eval(options[index])), ignore_index= True)
selector.select_by_index(df.loc[df.iloc[:,0].str.contains(str(moeda_origem))])
The error is:
"UnexpectedTagNameException: Select only works on select elements, not on button"
This page does not use default Select. Its dropdown is custom and, in order to work with it, do not use Selenium select and options, they won't work.
Try this instead:
driver = webdriver.Chrome()
driver.implicitly_wait(5)
driver.get('https://www.bcb.gov.br/conversao')
# Click to open the dropdown
driver.find_element_by_id("button-converter-de").click()
sleep(2) # Make sure dropdown opened
# Search for dropdown options by their selector
options = driver.find_elements_by_css_selector('#moedaBRL > li > a.dropdown-item')
print([o.text for o in options]) # this just prints all options, you can use your loop
I hope this helps, good luck!
I am new to Python and I am trying to use Selenium to select a value from a drop down menu on a Firefox browser.
This is what is am trying. Please let me know what I am doing wrong:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
browser = webdriver.Firefox()
browser.get('URL')
select = Select(browser.find_element_by_id('Yesterday'))
I need to add the following to my original post:
I have used Selenium IDE to try and identify the selector. Please image below. I would like to select "Yesterday" from the drop down list.
enter image description here
You can use like this, locate Select drop down then select the Custom value
from selenium.webdriver.support.ui import Select
select = Select(driver.find_element_by_name('locator of the select'))
select.select_by_value('Custom')
Also, you can use the index
select = Select(driver.find_element_by_name('locator of the select'))
select.select_by_index(6)
You can use browser.find_element_by_link_text("Yesterday") in place of browser.find_element_by_id. Since it does not contain an id tag it will not work.
[Please click here to view the Tags][1]
The following is one of the tables of the website I am scraping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scraping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?
Thank you.
Following is the code:
table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[#class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[#class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
You should be able to use a CSS selector to gather those for clicking
.btn-group.btn-group-xs button:first-child
The selector certainly works:
Not sure whether you need waits but maybe something like:
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
I am trying to retrieve all possible lists from this website
Model will only open when year is selected. Similarly Make will only open when Model is selected. I want to store all combination of Year, Model and Make for learning purpose.
However I am not able to click year field only. It seems it is hidden and without this I can't go ahead with rest of code.
from selenium import webdriver
import time
from selenium.webdriver.support.ui import Select
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
driver.get("https://www.osram-americas.com/en-us/applications/automotive-lighting-systems/Pages/lrgmain.aspx")
# Switch to new window opened
driver.switch_to.window(driver.window_handles[-1])
# Close the new window
driver.close()
# Switch back to original browser (first window)
driver.switch_to.window(driver.window_handles[0])
el = driver.find_element_by_id('sbToggle_27562807')
el.click()
It gives error :-
Message: no such element: Unable to locate element: {"method":"id","selector":"sbToggle_27562807"}
Message: no such element: Unable to locate element: {"method":"id","selector":"sbToggle_27562807"}
year element is not hidden, actually your locator to locate element using find_element_by_id('sbToggle_27562807') is not correct. In this element id attribute value is dynamically changing that's why you're unable to locate this element.
Instead of id locator you should try using some different locator. I would suggest, for better way try using find_element_by_css_selector() as below :-
el = driver.find_element_by_css_selector("div#fldYear a.sbToggle")
el.click()
Use this CSS selector to get to the arrow pointing downwards on the select year dropdown
"div[id='fldYear'] > div[class='sbHolder'] > a[class='sbToggle']"
or this xpath
"//div[#id='fldYear']/div[#class='sbHolder']/a[#class='sbToggle']"
Click on this to webelement to get the options
The element IDs change on each reload of the page, you'll have to find a different way to find the dropdown.
You can always find the <a> link with the "-- Select Year --" text, for example.
I'm using Selenium and coding with Python.
I'm trying to do the following: for a flight search website, under Flight 1's 'Enter routing code' text box, type 'AA'
This is the code that I have at the moment:
flight1_routing = driver.find_element_by_xpath(".//*[#id='ita_form_location_RouteLanguageTextBox_0']")
flight1_routing.clear()
flight1_origin.send_keys("AA")
But instead, I get this error message: "invalid element state: Element is not currently interactable and may not be manipulated". How can this be with a regular text field that is also not an autocomplete field, AFAIK?
if you get Element is not currently interactable check if the element is not disabled and its visible. if you want to hack it execute JS to enable it.
i visited the homepage id ita_form_location_RouteLanguageTextBox_0 doesnt exist also under flight one there's no Enter routing code. i can see the text box saying airport city or city name
Also if you have the id prefer to use find_element_by_id if not try to use css selector if you can rather than xpath. Its much cleaner.
Update
here's a working script:
As recomended above, the elements selected are not visible. what is actualy done, is that there's 5-6 different elements all hidden and when you click on show advanced route it picks 2 random ones and makes them visible.
So the id is not always the same. If you use the same id you will get a hidden element some times(because it picks random ids) so selenium is not able to deal with it. i made a selector that gets the 2 hidden elements
from selenium import webdriver
import selenium.webdriver.support.ui as ui
driver = webdriver.Firefox()
driver.get("http://matrix.itasoftware.com/")
#click on the multi tab
tab = driver.find_element_by_id("ita_layout_TabContainer_0_tablist_ita_form_multislice_MultiSliceForm_0").click()
#click on the advanced routes
advanced_routing=ui.WebDriverWait(driver, 10).until(
lambda driver : driver.find_element_by_id("sites_matrix_layout_RouteLanguageToggleLink_1")
)
advanced_routing.click()
#get all visible elements with id like ita_form_location_RouteLanguageTextBox. its similar to regex ita_form_location_RouteLanguageTextBox.*
element = ui.WebDriverWait(driver, 10).until(
lambda driver : driver.find_elements_by_css_selector("[id*=ita_form_multislice_MultiSliceRow] [id*=ita_form_location_RouteLanguageTextBox]")
)
element[0].send_keys("foo")
element[1].send_keys("bar")
import time
time.sleep(20)
Did you click into the correct tab first & enable advanced routing codes?? e.g.
#Go to right tab
driver.find_element_by_css_selector("div#ta_layout_TabContainer_0_tablist_ita_form_multislice_MultiSliceForm_0 > span").click()
#Enable routing
driver.find_element_by_css_selector("a.itaToggleLink").click()
#note I seem to get a different id to the one you're using, assuming its dynamic numbering so handling all cases
#if you know how the dynamic numbering works youmay be able to deduce a single id that will work for your test case
#Instead I'm going for finding all elements matching a pattern then searching through them, assuming only one will be visible
flight1_routings = driver.find_elements_by_css_selector("input[id^='ita_form_location_RouteLanguageTextBox_']")
#probably better finding it then using it separately, but I was feeling lazy sorry.
for route in flight1_routings:
if route.is_displayed():
route.clear()
route.send_keys("AA")
Also you can probably skip the .clear() call as it looks like the box starts with no text to overwrite.
Edit: Updated the enable routing toggling to handle not knowing the id, the class name stays the same, should work. Handling finding the input despite variable id as suggested by foo bar with the css selector, just then iterating over that list and checking if its on top