I am trying to webscrape an iframe object containing a list of links. The ultimate objective is to perform a series of actions, same for each link, that leads to downloading some data in xls format.
For each of the links in the iframe, I want to automate the following actions:
click on the link
select from a drop down list the option for getting data since 1999
click on the excel download button.
My code works for the first of the links, but I am struggling with automating it to go through all the links in the iframe.
This is the code that works for the first link:
#this automates the log in
driver.find_element_by_id('user').send_keys('myusername')
driver.find_element_by_id('password').send_keys('mypassword')
driver.find_element_by_xpath('myxpath').click()
#this switches to iframe and clicks on all the links
driver.switch_to.frame("report-iframe")
driver.find_element_by_tag_name('a').click()
#this performs the required actions for the first link(select drop down, select excel download)
s= Select(driver.find_element_by_id('start-date'))
s.select_by_value('1999-01-01')
driver.find_element_by_xpath('//*[#id="report-excel-download"]')
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, 'report-excel-download')))
element.click()
I want to automate this process for all the links in the iframe (i.e. for each of the links in iframe, click on the link, select the drop down, click the data download button. Does anyone have any advice in how to do this?
I tried:
links=[link.get_attribute('href') for link in driver.find_elements_by_xpath('myxpath')]
for l in links:
#click the link
driver.find_element_by_tag_name('a').click()
#select all years
s= Select(driver.find_element_by_id('report-control-date-start'))
s.select_by_value('1999-01-01')
#click download
driver.find_element_by_xpath('//*[#id="report-excel-download"]')
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, 'report-excel-download')))
element.click()
But this only works for the first link in the iframe, it does not work across all.
Does anyone have any advice?
Thanks!
Related
I am unable to loop click the links. When I try loop click the links it keeps clicking the first link only.
From the html code, I need the element named "key" value as well. How to capture it.
html file copy in dropbox. Please click https://www.dropbox.com/sh/85rx13m8iqwax4b/AACNDq_YyOukLh22JNv76vjua?dl=0.
html code
https://pastebin.com/Cyg98W2C
Python code I tried
elem = WebDriverWait(browser, 200).until(EC.element_to_be_clickable((By.XPATH, "//DIV[#id='propertySummaryList']/DIV[#class='summaryListItem ']/DIV[1]/DIV[3]/DIV[1]/H2[1]/A[1]")))
elem.click()
browser.back()
Edit: Added dropbox link. Since the site is sign in only. I have made a copy of the page.
You can gather all the elements, then use a relative find to find the link you need. Be careful, this may cause stale elements if you don't open the click in a new window.
summaryList = driver.find_elements_by_xpath("//DIV[#id='propertySummaryList']/DIV[#class='summaryListItem ']")
for elements in summaryList:
link = elements.find_elements_by_xpath(".//h2//a")
link.text // or link.click() but need to open in a new window or will get staleElementReference
I would like to automate clicking on "Refresh" button (See image "Refreshbutton") using python and selenium. I have tried to locate the button using xpath, id, class name but each time, I get the NoSuchElement exception, i.e. it cannot find the element. How can I locate the Refresh button? I don't care if it is by xpath, id or any other means. Thanks in advance!
HTML
RefreshButton
Find by XPATH try
Find By ID try
Class Name try
Python Code Snippet
The problem might not be the id/xpath but the presence of the button on the page. If the click is executed before the HTML page is fully loaded, the element might not be here yet. To make sure, you can use the WebDriverWait as follow:
element_xpath = '//*[#id="btnRefreshCameraList"]'
wait_time = 15 # wait for page to load for max. 15 seconds
WebDriverWait(
driver, wait_time).until(
EC.presence_of_element_located((By.XPATH, element_xpath))
)
)
driver.find_element_by_xpath(element_xpath).click()
Didn't have access to your site, so I couldn't try it out.
I'm trying to do a simple Python Selenium automation where the script will click a link which opens a dialog box on top of the page (Instagram profile).
The said dialog box will display the list of followers but unfortunately the UL that contains the list will only display the first 12 followers (or LI). It is powered by AJAX to "load more" followers.
Anyway, to simulate loading more followers, i tried this code:
driver.find_element_by_xpath('/html/body/div[3]/div/div[2]/ul').send_keys(Keys.END)
or
driver.find_element_by_tag_name('body').send_keys(Keys.END)
unfortunately, it doesn't work. I was wondering if there is a correct way to do this (scrolling down focused on the active div or any page element)?
Image below is provided to show the html structure of the said page.
Appreciate your help on this, thank you very much!
Can you try something like this?. This will scroll to the div element that you have mentioned.
Python Code
element = driver.find_element_by_xpath("xpath_of_div_element")
driver.execute_script("arguments[0].scrollIntoView(true);", element);
Java sample:
WebElement element = driver.findElement(By.id("id_of_div_element"));
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", element);
Use this
liElement = driver.find_element_by_xpath("//div[#role='dialog']//ul//li[text()='required_li_element_visible_text']")
driver.execute_script("arguments[0].scrollIntoView(true);", liElement);
I am building a web scraper selenium but the element I want to click is actually searched in chrome inspection layout but can't be seen visually(like when you hit ctrl+F and search for an element by typing it in, it shows up in the DOM structure collapsing...)
So I can't proceed since I should make a "clicking" on it with my Python.
http://bitly.kr/Nypl88
is the link and the element I would like to click with selenium has an id called "cns_Tab21". When you search for it, the result total is 1 but can't be seen in the DOM.
Thank you for reading this post and for your answer in advance.
Your element is hidden inside an iframe.
To get to it you will need to switch to the iframe first:
driver.get("https://finance.naver.com/item/coinfo.nhn?code=255440");
WebElement iframe = driver.findElement(By.id("coinfo_cp"));
driver.switchTo().frame(iframe);
WebElement tabElement = driver.findElement(By.id("cns_Tab21"));
*Edit*
Added a Python implementation
driver = webdriver.Chrome()
driver.get("https://finance.naver.com/item/coinfo.nhn?code=255440")
iframe = driver.find_element_by_id("coinfo_cp")
driver.switch_to.frame(iframe)
tabElement = driver.find_element_by_id("cns_Tab21")
I am scraping a webpage using selenium. I first find the link I want and then click on it and download it(Link is pdf). What happens is sometimes I am able to do so, but sometimes selenium says that link not found. I suppose that it is due to the page not loading properly. What can I do about this and am I in the right direction?
This is my previous code:
for b in source_code_2.find_all('a', href=True):
if b.has_attr("title"):
if(b['title']=='Click here to download'):
urllib2.urlretrieve(full_url)
now i want to do it using selenium and element. How can I do this?
I think you should use explicit wait to tell selenium to wait until specific element loads properly , In python you can use explicit wait in following way :
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "yourElement"))
OR
element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.ID, "yourElement"))
element.click()
You just need to replace your element ID in above code and you can change 20 seconds to 30 ,40 as per your need. So meaning of above code is your webdriver will wait until 20 seconds to find that specific element.