I am web scraping with selenium and whenever i try to download i file the firefox download/save as file pops up however, even If i apply profile.set_preference('browser.helperApps.neverAsk.saveToDisk', "application/csv"), it still doesnt work, I have tried everyt .csv related MIME but doesn't work, is it possible to either click save as radio button and then click ok on the dialog or disable it entirely.
you should do two things, first set these three preferences as follows (this is in Java but I guess you manage to translate that to python :-):
profile.setPreference("browser.download.dir", "c:/yourDownloadDir");
profile.setPreference("browser.download.folderList", 2);
profile.setPreference("browser.helperApps.neverAsk.saveToDisk", "application/csv, text/csv");
secondly, you should make sure the download file has the desired mime type. To do that, you can use the web developer tools and inspect the download.
EDIT:
To find out the MIME type open Chrome, press Ctrl+Shift+I (Cmd+Alt+I on Mac OS) change to the 'Network' tab and click your download link. You should see something like this:
Just an additional answer that might help someone, as comments to the accepted answer put me on the right track (thanks!). Another MIME type of CSV you might be dealing with is application/x-csv - that was my case and once I looked it up in the Network tab of the browser, I became a happier man :)
In C#
FirefoxOptions options = new FirefoxOptions();
options.SetPreference("browser.download.folderList", 2);
options.SetPreference("browser.download.manager.showWhenStarting", false);
options.SetPreference("browser.download.dir", "c:\\temp");
options.SetPreference("browser.helperApps.neverAsk.saveToDisk", "text/csv");
Related
I am trying to write a program that goes to the following site and downloads the Excel file that automatically downloads when clicking the XLS button on the bottom of the page. To be honest, I am quite new to programming.
Site: https://echa.europa.eu/assessment-regulatory-needs
At first I tried using Selenium to let the program click itself through the browser and really click on the button. However, I think the website detects the usage of automated software and I cannot bypass the disclaimer that appears when opening the website.
Then I read some answers on the same topic where it is possible by using the requests module. However, I could not get it to work.
One thing I think I understood from this anwer was that you need to get the site/server where the data is requested from by inspecting the button with F12 in the browser. I tried this, and thought I had it, however I cannot get the code to function. I learnt from this answer that you need to give the referer as well, bu I think the referer from this file ist only partially written out, as it is "https://echa.europa.eu/assessment-regulatory-needs".
This answer explained the network process more in detail, howver I am not able to recreate it. Also, to be honest i do not fully understand what the API is and how to search for it.
I also found this answer, but it does not work for me either.
So I am asking for help on this, as I think that my HTML, Java, Website, Python knowledge is too tiny to see what I have to change to be able to download the excel file.
I am automating some QA processes and I have some doubts along the way that do not allow me to move forward.
I have the following problems:
I need that when I open the Chromewebdriver, it automatically chooses the tab I want from the devtools, being Elements, Console, Recorder, Sources, etc.
What I could do so far is, when the browser is running, that the DevTools is open, but in its main tab, that would be elements
Extract information from the values of an extension (Is this possible?) The extension appears in the DevTools.
To add a kind of alert that does not allow you to advance if you do not click.
Take a screenshot (I could do this, but only the web page appears, and I would need that it appears besides the web, the DevTools).
I want to automatically download a .ics file (a calendar file) from a website using selenium with python. The goal is to disable the pop-up window that firefox open when you download a file. To do this, I use the following code :
#I set my preferences
profile = webdriver.FirefoxProfile()
profile.set_preference("browser.download.folderList", 2)#not using default folder for downloading
profile.set_preference("browser.download.manager.showWhenStarting", False)#dont show downloading process
profile.set_preference("browser.download.dir", 'C:/Users/UserName/Documents/rpi/some folder')#set the directory for download
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'text/calendar')#tell it to automaticaly download a file
#using the profile to access firefox
browser = webdriver.Firefox(executable_path='geckodriver',firefox_profile=profile)
I first thought that it was a MIME type problem in the line browser.helperApps.neverAsk.saveToDisk, but after changing the MIME type it's still not working.
From here I have no idea of what is going wrong because my MIME type seems right according to all internet resources I have been able to find.
Perhaps it is a problem of settings or something I haven't noticed...
Anyway thank you for reading this, ask me if you need a bit more code.
Hello from what I understand, downloading the file prompts a alert box asking if you want to download it, have you tried,
browser.switchtoalert.accept?
Sorry i write in VBA, but you should get the idea,
Thanks
So I did find a solution by avoiding the problem : i created a new firefox profile > launched firefox with this profile (all of this is done in about:profiles) > went to the web site from wich I wanted to download the file > downloaded the file and checked the "always do this with this type of file" box > launch my programm with this custom profile using the line :
profile= webdriver.FirefoxProfile("C:/Users/user/AppData/Roaming/Mozilla/Firefox/Profiles/bqpa3bzv.nameofprofile")
This seems to work fine and might be an efficient way of doing this.
Well done for figuring it out, I know the
.switchtoalert.accept works in chromedriver
Let me know if you want any help with anything 👍👍
I am currently trying to save a webpage as it appears on the website in html format. Approach I am using is prerssing Ctrl + S using autoit. On pressing that save as dialog box appears where I am asked to enter the name of the file to save. This is working fine. However, I want to save the file by pressing Ctrl + S instead of bring the dialog box in front. I read somewhere by using "set_preference" we can do that. CAn someone suggest how to set a preference. Below is the code I am using for Chrome broswer:
driver=Webdriver.Chrome()
driver.get('http://www.yahoo.com/')
autoit.send("{CTRL down}")
autoit.send("{CTRL down}")
autoit.send("{CTRL up}")
autoit.send("C:\\Users\\karanjuneja\\Downloads\\kj\\ABCD.mhtml")
autoit.send("{ENTER}")
Currently I am using the aboved code, however I want that on pressing Ctrl + S it saves the file in the desired location.
Thanks
Karan
Selenium isn't the designed for this, you could either:
Use getHtmlSource and parse the resulting HTML for references to external files, which you can then download and store outside of Selenium.
Use something other than Selenium to download and store an offline version of a website - I'm sure there are plenty of tools that could do this if you do a search. For example WGet can perform a recursive download (http://en.wikipedia.org/wiki/Wget#Recursive_download)
Is there any reason you want to use Selenium? Is this part of your testing strategy or are you just wanting to find a tool that will create an offline copy of a page?
I'm trying to use the Selenium module in python to generate a text list from one website, save it in a directory, and browse to that text list on another site to submit it.
I'm working on the script in two parts- 1. Get metadata and 2. Order data. I've successfully completed the script in part 1, except for the very last thing: Choosing to save the metadata file that was just generated. I left it alone to work on part 2, hoping I would stumble upon the answer, but I'm just reaching to same problem when the pop-up to choose file comes along.
In the documentation, I'm told that Selenium WebDriver has built-in support for handling popup dialog boxes and that after triggering a dialog box, if I call alert = driver.switch_to_alert() then I can "accept, dismiss, read its contents, or even type into a prompt."
However, it's not working. When I try alert.text('some text') or alert.send_keys(Keys.TAB), I keep getting the error NoAlertPresentException: Message: No alert is present and after adding the command to wait, I get the error TimeoutException: Message:
Are the popups I'm getting (screenshots attached) not recognized by Selenium? If so, how do I interact with them? It seems like using this to save and/or upload files is something that many people have to do, but I cannot find anything on Google. Specifically, I would like to choose 'Save File' then 'OK' for the first image and for the second I would like to browse to the file (i.e. enter the path into the file name field) and click 'Open.' I don't want to just change my Firefox settings to automatically save because this will eventually be run in a different environment, and that won't help solve my second problem.
Thanks!
EDIT:
I'm testing my script on windows but it will eventually be implemented on a linux cloud server. I thought I was going to have to switch to PhantomJS webdriver (which was probably going to make my problem worse) to do headless browsing but I found a way to keep firefox. I guess all this means is that I can't use AutoIT to fix my problem.
The popups you see are not regular popups that can be interacted with using switch_to. These popups are system dialogs and cannot be automated using selenium.
Usually people avoid having these dialogs shown in the first place by tweaking browser preferences, e.g.:
downloading file using selenium
Access to file download dialog in Firefox
How to download a file using Selenium's WebDriver?
For uploading, usually you can find the appropriate input element and send keys to it with a path to the file:
How to upload file ( picture ) with selenium, python
How to upload files into file inputs? (python-selenium docs)
Let me know if your case cannot be solved by using the answers in the links I've attached.
As for your first, "download file automatically" problem, you just need to set a correct content-type:
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', "application/xml,text/xml")
Second problem fix (upload part):
driver.find_element_by_name("input_product_list").send_keys(textpath)
driver.find_element_by_name('include_sr').click()
driver.find_element_by_id('submit').click()
Extremely simple implementation using AutoIT.
Below scripts and steps can hely you to click on 'Save>OK' on this window's pop-UP
Step 1: Download AutoIT package/tool here AutoIt You may select ZIP format (extract it)
Step 2: Open any text editor (say notepad) and copy below code and save with extension .au3
(e.g file.au3)
WinWait("[TITLE:Opening ; CLASS:MozillaDialogClass]","", 10)
If WinExists("[TITLE:Opening ; CLASS:MozillaDialogClass]") Then
WinActivate("[TITLE:Opening ; CLASS:MozillaDialogClass]")
Send("{DOWN}")
Sleep(20)
Send("{TAB}")
Sleep(20)
Send("{TAB}")
Sleep(20)
Send("{ENTER}")
EndIf
Step 3: From extracted zip (Step 1) look for folder named: Aut2Exe and open it
Step 4: Click Aut2exe_x64.exe if your OS is 64 bit otherwise click Aut2exe.exe
Step 5: Browse/Locate file created in Step2. (file saved as extension .a3)
AND Choose Destination (.exe/.a3x) and select .exe option (say file.exe)
AND then Click convert
Step 6: include this file.exe in your project folder and use it as per your requirement using below code (as it is in Eclipse):
driver.dwonload().click(); // it can be something else as per your flow
Runtime.getRuntime().exec("C:/*path_to_your_EXE_file(selected in step 6))*/file.exe");