I am using Selenium and python for web scraping and the page that i am using for testing this link
But the problem is i am not able to handle the dynamic content of the drop down, here is the problem arises
While selecting the state, the city is loaded based on the state, Some Php and js are going in the back end as far as i know.
So, i searched the web and came with a solution to wait for the sometime please use this link as reference.
The following is a part of my code
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
chrome_path = r"E:\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("http://www.blooddonors.in")
select = Select(driver.find_element_by_xpath('/html/body/table[3]/tbody/tr/td[2]/table[1]/tbody/tr/td/form/table/tbody/tr[2]/td[1]/select'))
select.select_by_visible_text('Tamil Nadu')
driver.implicitly_wait(60)
drop = Select(driver.find_element_by_xpath('//*[#id="div_city"]/select'))
select.select_by_visible_text('Coimbotore')
I am using a windows sys and i tried using CMD. It doesn't need a wait function it works fine without it.
The error that i am facing is:
raise NoSuchElementException("Could not locate element with visible text: %s" % text)
selenium.common.exceptions.NoSuchElementException: Message: Could not locate element with visible text: Coimbotore
But their it is actually their.
If someone can help me resolve the issue it would be great and i can move on to the next.
Thanks
To select Tamilnadu and then select Coimbotore you can use the following code block :
driver.get("http://www.blooddonors.in")
select = Select(driver.find_element_by_name('select'))
select.select_by_visible_text('Tamil Nadu')
drop = Select(driver.find_element_by_name('city'))
city_option = WebDriverWait(driver, 5).until(lambda x: x.find_element_by_xpath("//select[#name='city']/option[text()='Coimbotore']"))
city_option.click()
Selenium provides a select class which can be used to grab elements from drop-down menu.
select = Select(driver.find_element_by_id('city'))
select.select_by_value('430') #search by value, coimbotore is 430
Your second drop-down defined as drop while you're still trying to handle first drop-down (select) which doesn't have "Coimbotore" option...
Just replace
drop = Select(driver.find_element_by_xpath('//*[#id="div_city"]/select'))
select.select_by_visible_text('Coimbotore')
with
drop = Select(driver.find_element_by_xpath('//select[#name="city" and count(option) > 1]'))
drop.select_by_visible_text('Coimbotore')
Related
driver = webdriver.Remote(service.service_url)
timeout = 50
driver.get("https://www.nseindia.com/option-chain")
selection = Select(driver.find_element_by_id('select_symbol'))
time.sleep(2)
selection.select_by_index(3)
time.sleep(2)
driver.quit()
I need to get the call option total from the table on the site for each option in the dropdown menu, but the selected option doesn't load and the table doesn't load. Pls help
Sometimes, Webapps do not show desired web element while handled by automation software/tool such as Selenium.
The workaround I have come to know know is that, in this kind of situation, we need to pass Arrow_down key using automated software and using JS (could be Select class also depends on the website), we can let the application show all the options.
Using Selenium this is how drop down looks :- (Basically it does not show the options).
Now when we use the below code that has Arrow down keys implemented :
Code :-
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.select import Select
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(50)
driver.get("https://www.nseindia.com/option-chain")
wait = WebDriverWait(driver, 20)
select = Select(wait.until(EC.presence_of_element_located((By.ID, "equity_optionchain_select"))))
select.select_by_visible_text("NIFTY")
ActionChains(driver).key_down(Keys.ARROW_DOWN).key_up(Keys.ARROW_UP).perform()
driver.execute_script("return document.getElementById('select_symbol').selectedIndex = '3';")
Using Selenium this is how it looks now :
how about using
driver.find_element_by_xpath('//*[#id="select_symbol"]/option[3]').click()
It is easy to only change option['number'].
can someone help me on how to automate the dropdown menu in google form using selenium in python.
the problem that I am facing is I am able to locate the drop down menu and select it but I am not able to select the options I have tried select_by_index but it doesn't work. thank you in advance.
(I am new to the forum so sorry if I have asked the question in a ambiguous way)
drop = Select(browser.find_element_by_class_name('quantumWizMenuPaperselectContent').click())
drop.select_by_index(0)
Although I think there may be better ways to do it, I manage to do it this way:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome("YOUR PATH HERE")
driver.get("YOUR SITE HERE")
driver.find_element(By.CSS_SELECTOR, ".isSelected").click()
# Item 1 in drop down has an ending Div of 3
# Item 2 ending div of 4
# I added the + 2 in the f string so that DROPDOWN_ITEM = 1 Means find for div 3
DROPDOWN_ITEM = 1
dropdown_item_1 = WebDriverWait(driver, 10).until(EC.presence_of_element_located
((By.XPATH, f'//*[#id="mG61Hd"]/div[2]/div/div[2]/div/div/div/div[2]/div/div[2]/div[{DROPDOWN_ITEM} + 2]')))
dropdown_item_1.click()
if you would just like to target the first item, can use something like this:
dropdown_item_1 = WebDriverWait(driver, 10).until(EC.presence_of_element_located
((By.XPATH, '//*[#id="mG61Hd"]/div[2]/div/div[2]/div/div/div/div[2]/div/div[2]/div[3]')))
dropdown_item_1.click()
Without using Select , here is how I managed to do it.
Since google forms are TAB friendly (ie, you can navigate top to bottom using the TAB key), you can select the previous field and do send_keys(Keys.TAB) to focus the target field ...
example:-
url = "https://docs.google.com/forms/d/e/1FAIpQLScosZjmDrvgUvh77tXsaAb24hKVgaBnjJfJz2BX1PvoqIO1Ow/viewform"
phField = #xpath of phone number field
now, focusing on the next field of phField, and sending the needed option.
driver.find_element_by_xpath(phField ).send_keys(Keys.TAB,"10.05")
don't forget to import Keys from selenium.webdriver.common.keys
I want to click on Select Year dropdown and select a year from it. Go to that page and fetch the HTML.
I've written this piece of code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome('C:/Users/SoumyaPandey/Desktop/Galytix/Scrapers/data_ingestion/chromedriver.exe')
driver.get('https://investors.aercap.com/results-and-events/financial-results')
driver.maximize_window()
driver.find_element_by_id('year_filter_chosen').click()
driver.find_element_by_class_name('active-result')
I'm just starting to work with selenium and got no clue how to proceed further.
I tried to look for the next class after clicking on the dropdown. I want to set the attribute value 'data-option-array-index' to 1 first, open the page, get html. Then keep on changing the value of this attribute.
Any help would be much appreciated!!
driver.get('https://investors.aercap.com/results-and-events/financial-results')
elem=driver.find_element_by_css_selector('#year-filter')
driver.execute_script("arguments[0].style.display = 'block';", elem)
selectYear=Select(elem)
selectYear.select_by_index(1)
Simply find the element and use Select after you change it's style to display block to access it's values.
Imports
from selenium.webdriver.support.select import Select
For tag in selenium there's great Class Select, example provided my colleague in the neighbor answer. But there's also a bit easier way to do it as well, like:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://investors.aercap.com/results-and-events/financial-results')
el = driver.find_element(By.CSS_SELECTOR, '#year-filter')
time.sleep(3)
driver.execute_script("arguments[0].style.display = 'block';", el)
el.send_keys('2018')
time.sleep(3)
driver.quit()
Hoping you can help. I'm relatively new to Python and Selenium. I'm trying to pull together a simple script that will automate news searching on various websites. The primary focus was football and to go and get me the latest Manchester United news from a couple of places and save the list of link titles and URLs for me. I could then look through the links myself and choose anything I wanted to review.
In trying the the independent newspaper (https://www.independent.co.uk/) I seem to have come up against a problem with element not interactable when using the following approaches:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('chromedriver')
driver.get('https://www.independent.co.uk')
time.sleep(3)
#accept the cookies/privacy bit
OK = driver.find_element_by_id('qcCmpButtons')
OK.click()
#wait a few seconds, just in case
time.sleep(5)
search_toggle = driver.find_element_by_class_name('icon-search.dropdown-toggle')
search_toggle.click()
This throws the selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable error
I've also tried with XPATH
search_toggle = driver.find_element_by_xpath('//*[#id="quick-search-toggle"]')
and I also tried ID.
I did a lot of reading on here and then also tried using WebDriverWait and execute_script methods:
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="quick-search-toggle"]')))
driver.execute_script("arguments[0].click();", element)
This didn't seem to error but the search box never appeared, i.e. the appropriate click didn't happen.
Any help you could give would be fantastic.
Thanks,
Pete
Your locator is //*[#id="quick-search-toggle"], there are 2 on the page. The first is invisible and the second is visible. By default selenium refers to the first element, sadly the element you mean is the second one, so you need another unique locator. Try this:
search_toggle = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//div[#class="row secondary"]//a[#id="quick-search-toggle"]')))
search_toggle.click()
First you need to open search box, then send search keys:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import os
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
browser = webdriver.Chrome(executable_path=os.path.abspath(os.getcwd()) + "/chromedriver", options=chrome_options)
link = 'https://www.independent.co.uk'
browser.get(link)
# accept privacy
button = browser.find_element_by_xpath('//*[#id="qcCmpButtons"]/button').click()
# open search box
li = browser.find_element_by_xpath('//*[#id="masthead"]/div[3]/nav[2]/ul/li[1]')
search_tab = li.find_element_by_tag_name('a').click()
# send keys to search box
search = browser.find_element_by_xpath('//*[#id="gsc-i-id1"]')
search.send_keys("python")
search.send_keys(Keys.RETURN)
Can you try with below steps
search_toggle = driver.find_element_by_xpath('//*[#class="row secondary"]/nav[2]/ul/li[1]/a')
search_toggle.click()
I came across a website where I am hoping to scrape some data from. But the site seems to be un-scrapable for my limited Python knowledge. When using driver.find_element_by_xpath, I usually run into timeout exceptions.
Using the code I provided below, I hope to click on the first result and go to a new page. On the new page, I want to scrape the product title, and pack size. But no matter how I try it, I cannot even get Python to click the right thing for me. Let alone scraping the data. Can someone help out?
My desired output is:
Tris(triphenylphosphine)rhodium(I) chloride, 98%
190420010
1 GR 87.60
5 GR 367.50
These are the codes I have so far:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = "http://www.acros.com/"
cas = "14694-95-2" # need to select for the appropriate one
driver = webdriver.Firefox()
driver.get(url)
country = driver.find_element_by_name("ddlLand")
for option in country.find_elements_by_tag_name("option"):
if option.text == "United States":
option.click()
driver.find_element_by_css_selector("input[type = submit]").click()
choice = driver.find_element_by_name("_ctl1:DesktopThreePanes1:ThreePanes:_ctl4:ddlType")
for option in choice.find_elements_by_tag_name("option"):
if option.text == "CAS registry number":
option.click()
inputElement = driver.find_element_by_id("_ctl1_DesktopThreePanes1_ThreePanes__ctl4_tbSearchString")
inputElement.send_keys(cas)
driver.find_element_by_id("_ctl1_DesktopThreePanes1_ThreePanes__ctl4_btnGo").click()
Your code as presented works fine for me, in that it directs the instance of Firefox to http://www.acros.com/DesktopModules/Acros_Search_Results/Acros_Search_Results.aspx?search_type=CAS&SearchString=14694-95-2 which shows the search results.
If you locate the iframe element on that page:
<iframe id="searchAllFrame" allowtransparency="" background-color="transparent" frameborder="0" width="1577" height="3000" scrolling="auto" src="http://newsearch.chemexper.com/misc/hosted/acrosPlugin/center.shtml?query=14694-95-2&searchType=cas¤cy=&country=NULL&language=EN&forGroupNames=AcrosOrganics,FisherSci,MaybridgeBB,BioReagents,FisherLCMS&server=www.acros.com"></iframe>
and use driver.switch_to.frame to switch to that frame then I think data you want should be scrapable from there, for example:
driver.switch_to.frame(driver.find_element_by_xpath("//iframe[#id='searchAllFrame']"))
You can then carry on using the driver as usual to find elements within that iframe. (I think switch_to_frame works similarly, but is deprecated.)
(I can't seem to find a decent link to docs for switch_to, this isn't all that helpful.