This is my very first time trying to scrape data from a website using Selenium. Fortunately I have got Selenium and Chrome to coordinate and the desired website opens.Once it opens up, I want to tell Python to click 'SEARCH' leaving the empty box blank (next to contains) and then tell Python to export the results ' and save the xlsx file as result_file. I do not know why the snippet is blowing up. Please provide your kind assistance.
from selenium import webdriver
driver = webdriver.Chrome("C:\Python27\Scripts\chromedriver.exe")
driver.get("https://etrakit.friscotexas.gov/Search/permit.aspx")
number_option = driver.find_element_by_id("cplMain_btnSearch")
number_option.click()
search_button = driver.find_element_by_id("cplMain_btnExportToExcel")
search_button.click()
result_file = open("result_file.xlsx", "w")
driver.close()
result_file.close()
Looking at the source of that page, the ID of the search button is "cplMain_btnSearch" not "SEARCH". And the Export button is "cplMain_btnExportToExcel".
To expand the answer of Daniel Roseman, you also need to specify the download location
options.add_argument("download.default_directory=C:/Python27")
driver = webdriver.Chrome(chrome_options=options)
The file will then be stored in your python27 directory with the name RadGridExport.csv.
Related
I have permission to download some weather data from the following website:
https://www.meteobridel.com/messnetz/index3.php#
I was wondering is there is a possibility to automatically find the download URL behind the 'CSV' button and then download that csv file with Python.
I tried this, but it didn't work:
from selenium import webdriver
browser = webdriver.Safari()
url = 'https://meteobridel.lu/?page_id=5'
browser.get(url)
browser.find_element_by_xpath('//*[#id="CSV"]').click()
browser.close()
Thanks already!
Try
from selenium import webdriver
browser = webdriver.Safari()
url = 'https://meteobridel.lu/?page_id=5'
browser.get(url)
browser.find_element_by_xpath('//body/div[#id='main']/div[1]/div[1]/div[1]/a[4]').click()
browser.close()
Checking the page you provided I can't find an "CSV"-ID.
Maybe try getting the button by class:
browser.find_element_by_xpath(r"//a[contains(#class, 'buttons-csv')]").click()
element is inside iframe so you have to switch to it first , as the id of frame is unique you can switch like
browser.switch_to.frame("iframe")
browser.find_element_by_xpath('//span[contains(text(),"CSV")]/..').click()
I need to scrape the CSV result of the website (https://requestmap.webperf.tools/). But I can’t.
This is the process I need to do:
1- load the website (https://requestmap.webperf.tools/)
2- enter a new website as an input (for example, https://stackoverflow.com/)
3- click submit
4- in the new page which opens after the submit, download the csv file at the end of the page
But I think the driver has the main page and doesn’t switch to the new page. That's why I can’t download the CSV file.
Would you please tell me how to do it?
here is my code:
options = webdriver.ChromeOptions()
options.add_argument('-headless')
options.add_argument('-no-sandbox')
options.add_argument('-disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=options)
driver.get("https://requestmap.webperf.tools/")
driver.find_element_by_id("url").send_keys("https://stackoverflow.com/")
driver.find_element_by_id("submitter").click()
I have tested different ways to solve my issue:
First of all:
I got this code from here :
But it doesn't work.
window_after = driver.window_handles[1]
driver.switch_to.window(window_after)
I also tried this or this, but they are not working as well.
# wait to make sure there are two windows open
# it is not working
WebDriverWait(driver, 30).until(lambda d: len(d.window_handles) == 2)
# switch windows
# it is not working
driver.switch_to_window(driver.window_handles[1])
content = driver.page_source
soup = BeautifulSoup(content)
driver.find_elements_by_name("Download CSV")
So How can I get this issue solved?
Is there any other way in python to do so and switch to a new windows?
I want to press what I thought was a simple button on a website to download a CSV file.
However, I cannot find the button to press in the HTML code and, if I can, it looks like it carries some parameters (if it is the class "input-group-addon btn").
I've done similar but limited stuff like this before, but this looks different. I can find other buttons to press on this website but not this one that downloads the CSV file. It does not contain a address to the file and it is definitely not a "normal" button. I struggle to find some info online that is not about either a link to the file address or a normal button.
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://www.tennet.eu/electricity-market/transparency-pages/transparency-germany/network-figures/actual-and-forecast-wind-energy-feed-in/')
text_area1 = driver.find_element_by_id('daterange')
text_area1.send_keys("print('23.05.2019 - 23.05.2019')")
submit_button = driver.find_elements_by_xpath('SOMETHING IN HERE')
submit_button.click()
Goal for this part of the script is to open site, input today's date in a textfield then press a button to download CSV file.
Here is how you could do it :
import time
text_area1 = driver.find_element_by_id('daterange')
text_area1.clear()
text_area1.send_keys("11.05.2019 - 12.05.2019")
#Wait for page to update
time.sleep(1)
#Click submit button
driver.find_element_by_css_selector('.show-calendar .applyBtn').click()
time.sleep(1)
#Download file
driver.find_element_by_css_selector('.icon-download').click()
EDIT
Using css selectors instead of xpath
I'm browsing through a website using dryscrape in python and i need to upload a file to this site. But there is only one way of doing it, that is clicking in a button and browse into my files and select the one i want. How can i do it with python? i would appreciate if someone could help me using dryscrape too, but i'm accepting all answers.
heres the example image:
You can use Selenium. I tested this code and it works.
from selenium import webdriver
url = "https://example.com/"
driver = webdriver.Chrome("./chromedriver")
driver.get(url)
input_element = driver.find_element_by_css_selector("input[type=\"file\"]")
# absolute path to file
abs_file_path = "/Users/foo/Downloads/bar.png"
input_element.send_keys(abs_file_path)
sleep(5)
driver.quit()
Resources
Selenium python
Chrome driver download
For those who are searching for the answer in dryscrape i translated the selenium code to dryscrape:
element = sessin.at_xpath("xpath...") # Session is a dryscrape session
# The xpath is from the button like the one in the image "Browse..."
element.set("fullpath")
just as simple as it is.
Here is my problem: I'm trying to use selenium to access a webpage and the special about this page is it is an auto redirecting page (you open that page and after few seconds, it automatically redirect to another page). When i use driver = webdriver.Firefox(), my IDM catched that link just perfectly after few seconds.
And because i don't want the browser to come up so i use Phantomjs instead, ut it not working. My application just can get the loading page url (bitdl-1336...) but not the redirected link. Please help!
This is my code:
link = 'http://torrent.ajee.sh/hash.php?hash=' + self.global_hash_code
driver = webdriver.PhantomJS('phantomjs.exe')
driver.get(str(link))
element = driver.find_element_by_link_text('Download Zip')
element.click()
time.sleep(10)
msg = QMessageBox.information(self, QString('Thành công'),QString(driver.current_url))
And this is the result:
Please help!
Sorry about my english
Not exactly an answer to your PhantomJS-specific question, but a workaround to the problem.
And because i don't want the browser to come up so i use Phantomjs instead
You can continue using Firefox, but start it in a Virtual Display, see more information at:
How do I run Selenium in Xvfb?
You may also need to let the browser automatically save the archive in a specified directory, see:
How do I automatically download files from a pop up dialog using selenium-python
Access to file download dialog in Firefox