I am trying to pull data from following page:
https://www.lifeinscouncil.org/industry%20information/ListOfFundNAVs
I have to select the name of the company and select a date from the calendar and click get data button.
I am trying to achieve this using Selenium Web Driver using Chrome in Python, I am stuck how do i pass the date parameter to the page.
it seems the page is postback after selection of date from the calendar.
Date needs to be selected from the calendar else the data is not returned by the webpage.
I have tried using requests Post method as well but am not able to get the NAV data.
I need to iterate this for a period of 5 years on daily (Trading Days) basis.
PS: I am bad at understanding DOM elements and have basic knowledge of Python and coding. by profession I am a data analyst.
Thanks in Advance.
Kiran Jain
edit: adding current code below:
from selenium import webdriver
url='https://www.lifeinscouncil.org/industry%20information/ListOfFundNAVs'
opt = webdriver.ChromeOptions()
prefs = {"profile.managed_default_content_settings.images": 2}
opt.add_argument("--start-maximized")
# opt.add_argument("--headless")
opt.add_argument("--disable-notifications")
opt.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(options=opt)
driver.get('https://www.lifeinscouncil.org/industry%20information/ListOfFundNAVs');
insurer = driver.find_element_by_id("MainContent_drpselectinscompany")
nav_date=driver.find_element_by_id('MainContent_txtdateselect')
get_data_btn=driver.find_element_by_id('MainContent_btngetdetails')
options=insurer.find_elements_by_tag_name("option")
data=[]
row={'FirmName','SFIN','fundName','NAVDate','NAV'}
for option in options:
print('here')
print(option.get_attribute("value") + ' ' + option.text)
if(option.text!='--Select Insurer--'):
option.click()
driver.find_element_by_id("MainContent_imgbtncalender").click()#Calender Icon
driver.find_element_by_link_text("June").click()#Date
driver.find_element_by_link_text("25").click()#Date
get_data_btn=driver.find_element_by_id('MainContent_btngetdetails') #this is put here again because on clicking the date, the page is reloaded
get_data_btn.click()
print('clicked')
driver.quit()
The date is in "a" tag. You can try to do select the date using "link-text".
driver.find_element_by_id("MainContent_imgbtncalender").click()#Calender Icon
driver.find_element_by_link_text("27").click()#Date
As per your comment I tried to traverse through dates but it only worked for that particular month. I tried to use "send_keys()" to that text box and its not working. Below is the code to traverse it for a month.
driver.get("https://www.lifeinscouncil.org/industry%20information/ListOfFundNAVs")
driver.find_element_by_id("MainContent_drpselectinscompany").click()
driver.find_element_by_xpath("//option[starts-with(text(),'Aditya')]").click()
driver.find_element_by_id("MainContent_imgbtncalender").click()
driver.find_element_by_link_text("1").click()
driver.find_element_by_id("MainContent_btngetdetails").click()
dateval = 2
while True:
if dateval == 32:
break
try:
driver.find_element_by_id("MainContent_imgbtncalender").click()
driver.find_element_by_link_text(str(dateval)).click()
driver.find_element_by_id("MainContent_btngetdetails").click()
dateval+=1
time.sleep(2)
except:
driver.switch_to.default_content()
dateval+=1
time.sleep(2)
time.sleep(5)
driver.quit()
Related
Code that i am using:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
wd = webdriver.Chrome()
url = 'https://www.dailyfx.com/economic-calendar#next-seven-days'
wd.get(url)
time.sleep(20)
try:
wd.find_element(By.XPATH, "/html/body/div[7]/div/div/button/img").click()
except:
print('No Calendar Advertisement')
try:
wd.find_element(By.XPATH,"/html/body/div[1]/div[2]/div/div/div[2]/button").click()
except:
print('No Cookies Button')
time.sleep(3)
try:
wd.find_element(By.XPATH,"/html/body/div[1]/div[1]/div/div/div[1]/span").click()
except:
print('No App Advertisement')
#Clear calendar filter
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[1]/div[2]").click()
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[2]/div[2]/div[2]/div[1]/div[1]/label").click()
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[2]/div[2]/div[2]/div[1]/div[2]/label").click()
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[2]/div[2]/div[2]/div[1]/div[3]/label").click()
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[2]/div[2]/div[2]/div[1]/div[4]/label").click()
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[2]/div[2]/div[2]/div[1]/div[5]/label").click()
#Selecting only United States
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[2]/div[2]/div[2]/div[1]/div[1]/div/span").click()
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[2]/div[2]/div[2]/div[2]/div[1]/div/div/div[1]/label").click()
#Closing Calendar Filter
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[3]/div/div[1]/div[1]/div[2]").click()
#Working part:
wd.find_element(By.XPATH,"/html/body/div[5]/div/div[4]/div[2]/div/div/div[1]/div/div/div[4]/div[5]/table/tbody/tr[13]/td[1]/div/div[1]").click()
https://www.dailyfx.com/economic-calendar#next-seven-days
So, I am accessing this website and trying to click on a element. As you can see, the website shows some economic news and when you click on it, it shows a graphic with information, which is my goal - opening the graph. For some reason, I can only open the graphic when the table data (td\[1\]) is 1(that occurs just for the first economic news). When the table data(td\[3\]) change to 3(economic news that are more distant to happen), I can't open the graphic anymore. This code works:
wd.find_element(By.XPATH,"/html/body/div\[5\]/div/div\[4\]/div\[2\]/div/div/div\[1\]/div/div/div\[4\]/div\[5\]/table/tbody/tr\[13\]/td\[1\]/div/div\[1\]").click() When change to td\[3\], doesnt work: wd.find_element(By.XPATH,"/html/body/div\[5\]/div/div\[4\]/div\[2\]/div/div/div\[1\]/div/div/div\[4\]/div\[5\]/table/tbody/tr\[93\]/td\[3\]/div/div\[1\]").click()
I tried clicking on multiple different elements, but still doesn't work when trying to click on td\[3\] elements.
Tried to open a graphic of a economic news but only work when td\[1\], not for td\[3\].
I am attempting to choose date on a calendar on this website. On the first calendar (date from) I can choose the desired date using Selenium, however, I get the following error while clicking on the desired month even though the exact element is found.
ElementNotInteractableException:element not interactable
To me, it seems weird because I can click on the month manually.
Here is what I have tried so far
from selenium import webdriver
import time
year = 2019
month = 'JAN'
driver_path = 'pathtochromedriver\chromedriver.exe'
url = 'https://app.cpcbccr.com/ccr/#/caaqm-dashboard-all/caaqm-landing/data'
driver = webdriver.Chrome(driver_path)
driver.get(url)
time.sleep(8)
# find desired calendar
to_date = driver.find_element_by_xpath('//*[#id="date2"]/angular2-date-picker/div/div[1]/i')
to_date.click()
# Click on year dropdown
to_year = driver.find_element_by_xpath('//*[#id="date2"]/angular2-date-picker/div/div[2]/div[3]/div')
to_year.click()
driver.find_element_by_xpath('//*[#id="{}"]'.format(year)).click()
# Click on month dropdown
to_month = driver.find_element_by_xpath('//*[#id="date2"]/angular2-date-picker/div/div[2]/div[2]/div')
to_month.click()
mm = driver.find_element_by_xpath('//*[#id="{}"]'.format(month))
mm.click()
Mistake in code mm = driver.find_element_by_xpath('//*[#id="{}"]'.format(month)). You find first element in DOM(which combined with element data instead data2) and it is not visible yet.
There is workig code
mm = driver.find_element_by_id('date2').find_element_by_class_name('months-view').find_element_by_id(month)
mm.click()
Also good deal, to use WebDriverWait, because the site is very slow.
for example
to_date = WebDriverWait(driver, 10).until(
expected_conditions.presence_of_element_located(
(By.XPATH, '//*[#id="date2"]/angular2-date-picker/div/div[1]/i')))
I'm making a project which goes to my orders page on amazon and collects data like product name, price, delivery date using selenium (cuz there is no api for that, and cant do with bs4). I get login and get to orders page without any problem.But I'm stuck where i have to find the delivery date using find element by class( I chose class because all other delivery date text have same class), but selenium says it cannot find it.
No, its not in an iframe as i cant see the option for This Frame when i right click on that element.
here is the code -
import requests
from selenium import webdriver
import time
userid = #userid
passwd = #passwd
browser = webdriver.Chrome()
browser.get('https://www.amazon.in/gp/your-account/order-history?ref_=ya_d_c_yo')
email_input = browser.find_element_by_id('ap_email')
email_input.send_keys(userid)
email_input.submit()
passwd_input = browser.find_element_by_id('ap_password')
passwd_input.send_keys(passwd)
passwd_input.submit()
time.sleep(5)
date = browser.find_element_by_class_name('a-color-secondary value')
print(date.text)
Finding element by xpath seems to work, but fails to find the date for all orders as xpath is different for every element.
Any help is appreciated.
Thanks
Refers to this line:
date = browser.find_element_by_class_name('a-color-secondary value')
It seem like your element target having multiple class name, a-color-secondary and value. Sadly .find_element_by_class_name just for single class name.
Instead you can use .find_element_by_css_selector:
date = browser.find_element_by_css_selector('.a-color-secondary.value')
I am trying to scrape the Google News page in the following way:
from selenium import webdriver
import time
from pprint import pprint
base_url = 'https://www.google.com/'
driver = webdriver.Chrome('/home/vincent/wintergreen/chromedriver') ## change here to your location of the chromedriver
driver.implicitly_wait(30)
driver.get(base_url)
input = driver.find_element_by_id('lst-ib')
input.send_keys("brexit key dates timetable schedule briefing")
click = driver.find_element_by_name('btnK')
click.click()
news = driver.find_element_by_link_text('News')
news.click()
tools = driver.find_element_by_link_text('Tools')
tools.click()
time.sleep(1)
recent = driver.find_element_by_css_selector('div.hdtb-mn-hd[aria-label=Recent]')
recent.click()
# custom = driver.find_element_by_link_text('Custom range...')
custom = driver.find_element_by_css_selector('li#cdr_opt span')
custom.click()
from_ = driver.find_element_by_css_selector('input#cdr_min')
from_.send_keys("9/1/2018")
to_ = driver.find_element_by_css_selector('input#cdr_max')
to_.send_keys("9/2/2018")
time.sleep(1)
go_ = driver.find_element_by_css_selector('form input[type="submit"]')
print(go_)
pprint(dir(go_))
pprint(go_.__dict__)
go_.click()
This script manage to enter search terms, switch to the news tab, open the custom time period tab, fill in start and end date, but fails to click on the 'Go' button after that point.
From the print and pprint statement at the end of the script, I can deduct that it does find the 'go' button succesfully, but is somehow unable to click on it. The error displays as selenium.common.exceptions.ElementNotVisibleException: Message: element not visible
Could anyone experienced with Selenium have a quick run at it and give me hints as why it returns such error?
Thx!
Evaluating the css using developer tools in chrome yields 4 elements.
Click here for the image
use the following css instead:
go_ = driver.find_element_by_css_selector('#cdr_frm > input.ksb.mini.cdr_go')
I'd like to use python selenium to search at https://www.homeaway.com/
The following works:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://www.homeaway.com/")
driver.find_element_by_xpath("//input[#id='searchKeywords']").send_keys("Philadelphia, PA, USA")
But I'm running into an issue using the the calendar dropdown date picker, as it's not taking any of the values.
I've tried the following
Attempt 1 to enter a date into the start and end date field:
driver.find_element_by_xpath("//input[#id='stab-searchbox-start-date']").send_keys("02/01/2017")
driver.find_element_by_xpath("//input[#id='stab-searchbox-end-date']").send_keys("03/01/2017")
Note: It looks like homeaway website is completely ignoring the above commands unless you mannually click on the website with your mouse and then use the above selenium commands. In other words, the above commands are not working without a manual mouse click on the website first.
Attempt 2 to enter a date into the start and end date field:
driver.find_element_by_xpath("//div[#id='2017-02-01_2017-02']").click()
Attempt 3 to enter a date into the start and end date field
driver.execute_script("document.querySelectorAll('#stab-searchbox-start-date')[0].value = '02/01/2017'")
driver.execute_script("document.querySelectorAll('#stab-searchbox-end-date')[0].value = '03/01/2017'")
driver.find_element_by_xpath("//button[#class='btn btn-primary btn-lg searchbox-submit js-searchSubmit']").click()
Note: this looks like it works, but the dates are actually not registered when you click on search despite the dates being entered into the star and end date text boxes. It seems that homeaway will only register the dates if you use the calendar dropdown.
Note that you can not send_keys in these types when input text is not supported.
try this code, it will work:
from selenium import webdriver
url='http://www.homeaway.com/'
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)
driver.find_element_by_id("search-location").send_keys("Philadelphia, PA, USA")
driver.find_element_by_id('search-checkin').click()
driver.find_element_by_xpath('//*[#id="ui-datepicker-div"]//*[text()="30"]').click()
driver.find_element_by_id('search-checkout').click()
driver.find_element_by_xpath('//*[#id="ui-datepicker-div"]//*[text()="31"]').click()