I'm using Selenium, Chrome, and Python 3.
Here's what I'm doing to set everything up
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
chrome_options = Options()
options = webdriver.ChromeOptions()
prefs = {
"download.default_directory": r"C:\Download\Dir",
"download.directory_upgrade": "true",
"download.prompt_for_download": "false",
"disable-popup-blocking": "true",
'download.neverAsk.saveToDisk': 'application/octet-stream, application/json, jar, ' +
'text/comma-separated-values, text/csv, application/csv, ' +
'application/excel, application/vnd.ms-excel, ' +
'application/vnd.msexcel, text/anytext, text/plaintext, ' +
'image/png, image/pjpeg, image/jpeg, application/zip',
"safebrowsing.enabled": True
}
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(options=options)
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-software-rasterizer')
chrome_options.add_argument('--safebrowsing-disable-download-protection')
Then I have some credential secrets imported, and some website navigation to get to the file to download.
Afterwards, I click on the link to download the file:
# Click Upgrade Jar File
driver.find_element_by_xpath('//a[#id="accordionPanel:j_id_3a:1:j_id_3j"]').click()
The problem I have is here. Chrome asks me if I'd like to keep the jar file after I've downloaded it.
I've been reading through a lot of documentation and I don't understand how I can circumvent this. I believe that it should be added to prefs or as an chrome_options.add_argument but so far I've had no luck with the options I've found.
Update 01:
A little update, I was able to get it to work "silently", which allows you to bypass the "keep" button, but I haven't yet found a solution that will bypass the "keep" button when you're looking at the GUI.
options = webdriver.ChromeOptions()
prefs = {
"download.default_directory": r"C:\Download\Dir",
"download.directory_upgrade": "true",
"download.prompt_for_download": "false",
"disable-popup-blocking": "true",
"safebrowsing.enabled": False,
"default_content_settings": "contentSettings",
"download": "download"
}
options.add_experimental_option("prefs", prefs)
options.add_argument("--headless")
options.add_argument("--disable-notifications")
options.add_argument('--disable-gpu')
options.add_argument('--disable-software-rasterizer')
driver = webdriver.Chrome(options=options)
options.add_argument('--safebrowsing-disable-download-protection')
Update 02
Adapted code from a comment:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import time
import os
# Secrets
user_id = [grabs from a function that queries secrets]
user_password = [grabs from a function that queries secrets]
download_dir = "C:\Download\Dir"
options = Options()
options.add_argument("--disable-infobars")
options.add_argument("start-maximized")
options.add_argument("--disable-extensions")
options.add_argument("--disable-popup-blocking")
# disable the banner "Chrome is being controlled by automated test software"
options.add_experimental_option("useAutomationExtension", False)
options.add_experimental_option("excludeSwitches", ['enable-automation'])
prefs = {
"download.default_directory": r"C:\Download\Dir",
'download.prompt_for_download': False,
'download.extensions_to_open': 'jar',
'safebrowsing.enabled': True
}
capabilities = DesiredCapabilities().CHROME
options.add_experimental_option('prefs', prefs)
capabilities.update(options.to_capabilities())
driver = webdriver.Chrome('.\chromedriver.exe', options=options)
url = 'URL GOES HERE'
driver.get(url)
print("Navigating to", url)
driver.get(url)
# Define Text Boxes For Login
print('Logging into GoAnywhere Website')
username = driver.find_element_by_id('email')
password = driver.find_element_by_name('secret')
# Enter Username/Password
username.send_keys(user_id)
password.send_keys(user_password)
password.send_keys('\n')
# Wait for Page to load
....[Navigation Logic Here]....
# Click Upgrades
print("Selecting Upgrades")
time.sleep(1)
driver.find_element_by_xpath('//div[#id="accordionPanel"]/div[5]').click()
# CLick Upgrade Jar File
time.sleep(1)
print("Downloading upgrade jar to", download_dir)
driver.find_element_by_xpath('//a[#id="accordionPanel:j_id_3a:1:j_id_3j"]').click()
....[Logic to do stuff with file after downloaded]....
UPDATE 05-28-2021
After doing some more testing I have determined the code below works flawlessly on a Microsoft Windows platform that does not have any type of Chrome Browser policies enabled.
The code will likely fail when the Microsoft Windows platform is under corporate IT control. Google has policy templates that allow corporate IT to set Chrome politics that can override the selenium code within this answer.
For example this is one of the politics that can be set by corporate IT that will negate the selenium code within this answer.
If either SafeBrowsingProtectionLevel_StandardProtection or SafeBrowsingProtectionLevel_EnhancedProtection are enabled Google Chrome will Gives you warnings about potentially risky sites, downloads, and extensions.
CATEGORY !!SafeBrowsing_Category
POLICY !!SafeBrowsingExtendedReportingEnabled_Policy
#if version >= 4
SUPPORTED !!SUPPORTED_WIN7
#endif
EXPLAIN !!SafeBrowsingExtendedReportingEnabled_Explain
VALUENAME "SafeBrowsingExtendedReportingEnabled"
VALUEON NUMERIC 1
VALUEOFF NUMERIC 0
END POLICY
POLICY !!SafeBrowsingProtectionLevel_Policy
#if version >= 4
SUPPORTED !!SUPPORTED_WIN7
#endif
EXPLAIN !!SafeBrowsingProtectionLevel_Explain
PART !!SafeBrowsingProtectionLevel_Part DROPDOWNLIST
VALUENAME "SafeBrowsingProtectionLevel"
ITEMLIST
NAME !!SafeBrowsingProtectionLevel_NoProtection_DropDown VALUE NUMERIC 0
NAME !!SafeBrowsingProtectionLevel_StandardProtection_DropDown VALUE NUMERIC 1
NAME !!SafeBrowsingProtectionLevel_EnhancedProtection_DropDown VALUE NUMERIC 2
END ITEMLIST
END PART
END POLICY
I also noted this within Chrome's source code. This is why the message "This type of file can harm your computer" is thrown when downloading a .jar file
namespace download_util
static const struct Executables {
const char* extension;
DownloadDangerLevel level;
} g_executables[] = {
// Some files are dangerous on all platforms.
truncated...
// Java.
{ "class", DANGEROUS },
{ "jar", DANGEROUS },
{ "jnlp", DANGEROUS },
truncated...
Jar files are considered dangerous by Google Chrome. The verbiage below is from Chrome's GitHub repository.
Note the text below is taking about displaying warning messages in the User interface (UI) of the Chrome browser.
platform_settings.danger_level: (required) Controls how files should be handled by the UI in the absence of a better signal from the Safe Browsing ping. This applies to all file types where ping_setting is either SAMPLED_PING or NO_PING, and downloads where the Safe Browsing ping either fails, is disabled, or returns an UNKNOWN verdict. Exceptions are noted below.
The warning controlled here is a generic "This file may harm your computer." If the Safe Browsing verdict is UNCOMMON, POTENTIALLY_UNWANTED, DANGEROUS_HOST, or DANGEROUS, Chrome will show that more severe warning regardless of this setting.
This policy also affects also how subresources are handled for "Save As ..." downloads of complete web pages. If any subresource ends up with a file type that is considered DANGEROUS or ALLOW_ON_USER_GESTURE, then the filename will be changed to end in .download. This is done to prevent the file from being opened accidentally.
This policy also affects also how subresources are handled for "Save As ..." downloads of complete web pages. If any subresource ends up with a file type that is considered DANGEROUS or ALLOW_ON_USER_GESTURE, then the filename will be changed to end in .download. This is done to prevent the file from being opened accidentally.
NOT_DANGEROUS: Safe to download and open, even if the download was accidental. No additional warnings are necessary.
DANGEROUS: Always warn the user that this file may harm their computer. We let them continue or discard the file. If Safe Browsing returns a SAFE verdict, we still warn the user.
CONCLUSION
The OP of this question stated in the comments that he was using a corporate IT controlled computer. It is HIGHLY LIKELY that SafeBrowsingProtection was enabled on the OP's system. With this extra level of security protection enforced by corporate IT the selenium code within this answer could not suppress the warning message being displayed in the OP's Chrome browser UI.
The OP stated that he was able to bypass the "This type of file can harm your computer" message when using selenium and Chrome in headless mode. The reason the message was suppressed is because headless mode does not use a UI thus the warning message will not be raised. Chrome's own documentation/source validates that this warning message is only displayed in the UI. Ref: platform_settings.danger_level above.
ORIGINAL POST 05-12-2021
The code below allows me to click on a href link for a .jar file without receiving "This type of file can harm your computer" message.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
chrome_options = Options()
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("start-maximized")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")
# disable the banner "Chrome is being controlled by automated test software"
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])
prefs = {
'download.default_directory': 'download_directory',
'download.prompt_for_download': False,
'download.extensions_to_open': 'jar',
'safebrowsing.enabled': True
}
capabilities = DesiredCapabilities().CHROME
chrome_options.add_experimental_option('prefs', prefs)
capabilities.update(chrome_options.to_capabilities())
driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)
# I used this site in my testing, because it had JAR files
url_main = 'http://www.jgoodies.com/downloads/demos/'
driver.get(url_main)
driver.implicitly_wait(20)
# download a jar file
download_jar_file = driver.find_element_by_xpath('//*[#id="post-70"]/div/table/tbody/tr[2]/td[5]/a')
download_jar_file.click()
the preference download.extensions_to_open is linked to this policy in the Chrome's source code.
"AutoOpenFileTypes" : {
"os": ["win", "mac", "linux", "chromeos"],
"policy_pref_mapping_tests": [
{
"policies": { "AutoOpenFileTypes": ["exe", ".txt", "pdf"] },
"prefs": {
"download.extensions_to_open_by_policy": {"value" : ["exe", "pdf"] }
}
}
]
},
----------------------------------------
My system information
----------------------------------------
Platform: macOS
Python: 3.8.0
Selenium: 3.141.0
Chromedriver: 90.0.4430.24
----------------------------------------
Try adding chrome_options.add_argument('--safebrowsing-disable-download-protection') in your chrome options setup.
Edit:
Wait a second. You've defined options = webdriver.ChromeOptions(). Try setting the arguments like this?
options.add_argument('--disable-gpu')
options.add_argument('--disable-software-rasterizer')
options.add_argument('--safebrowsing-disable-download-protection')
Related
I'm do me code in Cromedrive in 'normal' mode and works fine. When I change to headless mode it don't download the file. I already try the code I found alround internet, but didn't work.
chrome_options = Options()
chrome_options.add_argument("--headless")
self.driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'{}/chromedriver'.format(os.getcwd()))
self.driver.set_window_size(1024, 768)
self.driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': os.getcwd()}}
self.driver.execute("send_command", params)
Anyone have any idea about how solve this problem?
PS: I don't need to use Chomedrive necessarily. If it works in another drive it's fine for me.
First the solution
Minimum Prerequisites:
Selenium client version: Selenium v3.141.59
Chrome version: Chrome v77.0
ChromeDriver version: ChromeDriver v77.0
To download the file clicking on the element with text as Download Data within this website you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe', service_args=["--log-path=./Logs/DubiousDan.log"])
print ("Headless Chrome Initialized")
params = {'behavior': 'allow', 'downloadPath': r'C:\Users\Debanjan.B\Downloads'}
driver.execute_cdp_cmd('Page.setDownloadBehavior', params)
driver.get("https://www.mockaroo.com/")
driver.execute_script("scroll(0, 250)");
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#download"))).click()
print ("Download button clicked")
#driver.quit()
Console Output:
Headless Chrome Initialized
Download button clicked
File Downloading snapshot:
Details
Downloading files through Headless Chromium was one of the most sought functionality since Headless Chrome was introduced.
Since then there were different work-arounds published by different contributors and some of them are:
Downloading with chrome headless and selenium
Python equivalent of a given wget command
Now the, the good news is Chromium team have officially announced the arrival of the functionality Downloading file through Headless Chromium.
In the discussion Headless mode doesn't save file downloads #eseckler mentioned:
Downloads in headless work a little differently. There's the Page.setDownloadBehavior devtools command to set a download folder. We're working on a way to use DevTools network interception to stream the downloaded file via DevTools as well.
A detailed discussion can be found at Issue 696481: Headless mode doesn't save file downloads
Finally, #bugdroid revision seems to have nailed the issue for us.
[ChromeDriver] Added support for headless mode to download files
Previously, Chromedriver running in headless mode would not properly download files due to the fact it sparsely parses the preference file given to it. Engineers from the headless chrome team recommended using DevTools's "Page.setDownloadBehavior" to fix this. This changelist implements this fix. Downloaded files default to the current directory and can be set using download_dir when instantiating a chromedriver instance. Also added tests to ensure proper download functionality.
Here is the revision and commit
From ChromeDriver v77.0.3865.40 (2019-08-20) release notes:
Resolved issue 2454: Headless mode doesn't save file downloads [Pri-2]
Solution
Update ChromeDriver to latest ChromeDriver v77.0 level.
Update Chrome to Chrome Version 77.0 level. (as per ChromeDriver v76.0 release notes)
Note: Chrome v77.0 is yet to be GAed/pushed for release so till then you can download and install a development build and test either from:
Chrome Canary
Latest build from the Dev Channel
Outro
However Mac OSX users have a wait for their pie as On Chromedriver, headless chrome crashes after sending Page.setDownloadBehavior on MacOSX.
Chomedriver Version: 95.0.4638.54
Chrome Version 95.0.4638.69
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("--start-maximized")
options.add_argument("--no-sandbox")
options.add_argument("--disable-extensions")
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--disable-gpu")
options.add_argument('--disable-software-rasterizer')
options.add_argument("user-agent=Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Microsoft; Lumia 640 XL LTE) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Mobile Safari/537.36 Edge/12.10166")
options.add_argument("--disable-notifications")
options.add_experimental_option("prefs", {
"download.default_directory": "C:\\link\\to\\folder",
"download.prompt_for_download": False,
"download.directory_upgrade": True,
"safebrowsing_for_trusted_sources_enabled": False,
"safebrowsing.enabled": False
}
)
What seemed to work was that I used "\\" instead of "/" for the address. The latter approach didn't throw any error, but didn't download any documents either. But, using double back slashes did the job.
For javascript use below code:
const chrome = require('selenium-webdriver/chrome');
let options = new chrome.Options();
options.addArguments('--headless --window-size=1500,1200');
options.setUserPreferences({ 'plugins.always_open_pdf_externally': true,
"profile.default_content_settings.popups": 0,
"download.default_directory": Download_File_Path });
driver = await new webdriver.Builder().setChromeOptions(options).forBrowser('chrome').build();
Then switch tabs as soon as you click the download button:
await driver.sleep(1000);
var Handle = await driver.getAllWindowHandles();
await driver.switchTo().window(Handle[1]);
import pathlib
from selenium.webdriver import Chrome
driver = Chrome()
driver.execute_cdp_cmd("Page.setDownloadBehavior", {
"behavior": "allow",
"downloadPath": str(pathlib.Path.home().joinpath("Downloads"))
})
This C# works for me
Note the new headless option https://www.selenium.dev/blog/2023/headless-is-going-away/
private IWebDriver StartBrowserChromeHeadlessDriver()
{
var chromeOptions = new ChromeOptions();
chromeOptions.AddArgument("--headless=new");
chromeOptions.AddArgument("--window-size=1920,1080");
chromeOptions.AddUserProfilePreference("download.default_directory", downloadFolder);
var chromeDownload = new Dictionary<string, object>
{
{ "behavior", "allow" },
{ "downloadPath", downloadFolder }
};
var driver = new ChromeDriver(driverFolder, chromeOptions, TimeSpan.FromSeconds(timeoutSecs));
driver.ExecuteCdpCommand("Browser.setDownloadBehavior", chromeDownload);
return driver;
}
I don't think you should be using the browser for downloading content, leave it to Chrome developers/testers.
I believe you should rather get href attribute of the element you want to download and obtain it using requests library
If your site requires authentication you could fetch cookies from the browser instance and pass them to requests.Session.
I'm do me code in Cromedrive in 'normal' mode and works fine. When I change to headless mode it don't download the file. I already try the code I found alround internet, but didn't work.
chrome_options = Options()
chrome_options.add_argument("--headless")
self.driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'{}/chromedriver'.format(os.getcwd()))
self.driver.set_window_size(1024, 768)
self.driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': os.getcwd()}}
self.driver.execute("send_command", params)
Anyone have any idea about how solve this problem?
PS: I don't need to use Chomedrive necessarily. If it works in another drive it's fine for me.
First the solution
Minimum Prerequisites:
Selenium client version: Selenium v3.141.59
Chrome version: Chrome v77.0
ChromeDriver version: ChromeDriver v77.0
To download the file clicking on the element with text as Download Data within this website you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe', service_args=["--log-path=./Logs/DubiousDan.log"])
print ("Headless Chrome Initialized")
params = {'behavior': 'allow', 'downloadPath': r'C:\Users\Debanjan.B\Downloads'}
driver.execute_cdp_cmd('Page.setDownloadBehavior', params)
driver.get("https://www.mockaroo.com/")
driver.execute_script("scroll(0, 250)");
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#download"))).click()
print ("Download button clicked")
#driver.quit()
Console Output:
Headless Chrome Initialized
Download button clicked
File Downloading snapshot:
Details
Downloading files through Headless Chromium was one of the most sought functionality since Headless Chrome was introduced.
Since then there were different work-arounds published by different contributors and some of them are:
Downloading with chrome headless and selenium
Python equivalent of a given wget command
Now the, the good news is Chromium team have officially announced the arrival of the functionality Downloading file through Headless Chromium.
In the discussion Headless mode doesn't save file downloads #eseckler mentioned:
Downloads in headless work a little differently. There's the Page.setDownloadBehavior devtools command to set a download folder. We're working on a way to use DevTools network interception to stream the downloaded file via DevTools as well.
A detailed discussion can be found at Issue 696481: Headless mode doesn't save file downloads
Finally, #bugdroid revision seems to have nailed the issue for us.
[ChromeDriver] Added support for headless mode to download files
Previously, Chromedriver running in headless mode would not properly download files due to the fact it sparsely parses the preference file given to it. Engineers from the headless chrome team recommended using DevTools's "Page.setDownloadBehavior" to fix this. This changelist implements this fix. Downloaded files default to the current directory and can be set using download_dir when instantiating a chromedriver instance. Also added tests to ensure proper download functionality.
Here is the revision and commit
From ChromeDriver v77.0.3865.40 (2019-08-20) release notes:
Resolved issue 2454: Headless mode doesn't save file downloads [Pri-2]
Solution
Update ChromeDriver to latest ChromeDriver v77.0 level.
Update Chrome to Chrome Version 77.0 level. (as per ChromeDriver v76.0 release notes)
Note: Chrome v77.0 is yet to be GAed/pushed for release so till then you can download and install a development build and test either from:
Chrome Canary
Latest build from the Dev Channel
Outro
However Mac OSX users have a wait for their pie as On Chromedriver, headless chrome crashes after sending Page.setDownloadBehavior on MacOSX.
Chomedriver Version: 95.0.4638.54
Chrome Version 95.0.4638.69
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("--start-maximized")
options.add_argument("--no-sandbox")
options.add_argument("--disable-extensions")
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--disable-gpu")
options.add_argument('--disable-software-rasterizer')
options.add_argument("user-agent=Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Microsoft; Lumia 640 XL LTE) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Mobile Safari/537.36 Edge/12.10166")
options.add_argument("--disable-notifications")
options.add_experimental_option("prefs", {
"download.default_directory": "C:\\link\\to\\folder",
"download.prompt_for_download": False,
"download.directory_upgrade": True,
"safebrowsing_for_trusted_sources_enabled": False,
"safebrowsing.enabled": False
}
)
What seemed to work was that I used "\\" instead of "/" for the address. The latter approach didn't throw any error, but didn't download any documents either. But, using double back slashes did the job.
For javascript use below code:
const chrome = require('selenium-webdriver/chrome');
let options = new chrome.Options();
options.addArguments('--headless --window-size=1500,1200');
options.setUserPreferences({ 'plugins.always_open_pdf_externally': true,
"profile.default_content_settings.popups": 0,
"download.default_directory": Download_File_Path });
driver = await new webdriver.Builder().setChromeOptions(options).forBrowser('chrome').build();
Then switch tabs as soon as you click the download button:
await driver.sleep(1000);
var Handle = await driver.getAllWindowHandles();
await driver.switchTo().window(Handle[1]);
import pathlib
from selenium.webdriver import Chrome
driver = Chrome()
driver.execute_cdp_cmd("Page.setDownloadBehavior", {
"behavior": "allow",
"downloadPath": str(pathlib.Path.home().joinpath("Downloads"))
})
This C# works for me
Note the new headless option https://www.selenium.dev/blog/2023/headless-is-going-away/
private IWebDriver StartBrowserChromeHeadlessDriver()
{
var chromeOptions = new ChromeOptions();
chromeOptions.AddArgument("--headless=new");
chromeOptions.AddArgument("--window-size=1920,1080");
chromeOptions.AddUserProfilePreference("download.default_directory", downloadFolder);
var chromeDownload = new Dictionary<string, object>
{
{ "behavior", "allow" },
{ "downloadPath", downloadFolder }
};
var driver = new ChromeDriver(driverFolder, chromeOptions, TimeSpan.FromSeconds(timeoutSecs));
driver.ExecuteCdpCommand("Browser.setDownloadBehavior", chromeDownload);
return driver;
}
I don't think you should be using the browser for downloading content, leave it to Chrome developers/testers.
I believe you should rather get href attribute of the element you want to download and obtain it using requests library
If your site requires authentication you could fetch cookies from the browser instance and pass them to requests.Session.
I'm do me code in Cromedrive in 'normal' mode and works fine. When I change to headless mode it don't download the file. I already try the code I found alround internet, but didn't work.
chrome_options = Options()
chrome_options.add_argument("--headless")
self.driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'{}/chromedriver'.format(os.getcwd()))
self.driver.set_window_size(1024, 768)
self.driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': os.getcwd()}}
self.driver.execute("send_command", params)
Anyone have any idea about how solve this problem?
PS: I don't need to use Chomedrive necessarily. If it works in another drive it's fine for me.
First the solution
Minimum Prerequisites:
Selenium client version: Selenium v3.141.59
Chrome version: Chrome v77.0
ChromeDriver version: ChromeDriver v77.0
To download the file clicking on the element with text as Download Data within this website you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe', service_args=["--log-path=./Logs/DubiousDan.log"])
print ("Headless Chrome Initialized")
params = {'behavior': 'allow', 'downloadPath': r'C:\Users\Debanjan.B\Downloads'}
driver.execute_cdp_cmd('Page.setDownloadBehavior', params)
driver.get("https://www.mockaroo.com/")
driver.execute_script("scroll(0, 250)");
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#download"))).click()
print ("Download button clicked")
#driver.quit()
Console Output:
Headless Chrome Initialized
Download button clicked
File Downloading snapshot:
Details
Downloading files through Headless Chromium was one of the most sought functionality since Headless Chrome was introduced.
Since then there were different work-arounds published by different contributors and some of them are:
Downloading with chrome headless and selenium
Python equivalent of a given wget command
Now the, the good news is Chromium team have officially announced the arrival of the functionality Downloading file through Headless Chromium.
In the discussion Headless mode doesn't save file downloads #eseckler mentioned:
Downloads in headless work a little differently. There's the Page.setDownloadBehavior devtools command to set a download folder. We're working on a way to use DevTools network interception to stream the downloaded file via DevTools as well.
A detailed discussion can be found at Issue 696481: Headless mode doesn't save file downloads
Finally, #bugdroid revision seems to have nailed the issue for us.
[ChromeDriver] Added support for headless mode to download files
Previously, Chromedriver running in headless mode would not properly download files due to the fact it sparsely parses the preference file given to it. Engineers from the headless chrome team recommended using DevTools's "Page.setDownloadBehavior" to fix this. This changelist implements this fix. Downloaded files default to the current directory and can be set using download_dir when instantiating a chromedriver instance. Also added tests to ensure proper download functionality.
Here is the revision and commit
From ChromeDriver v77.0.3865.40 (2019-08-20) release notes:
Resolved issue 2454: Headless mode doesn't save file downloads [Pri-2]
Solution
Update ChromeDriver to latest ChromeDriver v77.0 level.
Update Chrome to Chrome Version 77.0 level. (as per ChromeDriver v76.0 release notes)
Note: Chrome v77.0 is yet to be GAed/pushed for release so till then you can download and install a development build and test either from:
Chrome Canary
Latest build from the Dev Channel
Outro
However Mac OSX users have a wait for their pie as On Chromedriver, headless chrome crashes after sending Page.setDownloadBehavior on MacOSX.
Chomedriver Version: 95.0.4638.54
Chrome Version 95.0.4638.69
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument("--start-maximized")
options.add_argument("--no-sandbox")
options.add_argument("--disable-extensions")
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--disable-gpu")
options.add_argument('--disable-software-rasterizer')
options.add_argument("user-agent=Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Microsoft; Lumia 640 XL LTE) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Mobile Safari/537.36 Edge/12.10166")
options.add_argument("--disable-notifications")
options.add_experimental_option("prefs", {
"download.default_directory": "C:\\link\\to\\folder",
"download.prompt_for_download": False,
"download.directory_upgrade": True,
"safebrowsing_for_trusted_sources_enabled": False,
"safebrowsing.enabled": False
}
)
What seemed to work was that I used "\\" instead of "/" for the address. The latter approach didn't throw any error, but didn't download any documents either. But, using double back slashes did the job.
For javascript use below code:
const chrome = require('selenium-webdriver/chrome');
let options = new chrome.Options();
options.addArguments('--headless --window-size=1500,1200');
options.setUserPreferences({ 'plugins.always_open_pdf_externally': true,
"profile.default_content_settings.popups": 0,
"download.default_directory": Download_File_Path });
driver = await new webdriver.Builder().setChromeOptions(options).forBrowser('chrome').build();
Then switch tabs as soon as you click the download button:
await driver.sleep(1000);
var Handle = await driver.getAllWindowHandles();
await driver.switchTo().window(Handle[1]);
This C# works for me
Note the new headless option https://www.selenium.dev/blog/2023/headless-is-going-away/
private IWebDriver StartBrowserChromeHeadlessDriver()
{
var chromeOptions = new ChromeOptions();
chromeOptions.AddArgument("--headless=new");
chromeOptions.AddArgument("--window-size=1920,1080");
chromeOptions.AddUserProfilePreference("download.default_directory", downloadFolder);
var chromeDownload = new Dictionary<string, object>
{
{ "behavior", "allow" },
{ "downloadPath", downloadFolder }
};
var driver = new ChromeDriver(driverFolder, chromeOptions, TimeSpan.FromSeconds(timeoutSecs));
driver.ExecuteCdpCommand("Browser.setDownloadBehavior", chromeDownload);
return driver;
}
import pathlib
from selenium.webdriver import Chrome
driver = Chrome()
driver.execute_cdp_cmd("Page.setDownloadBehavior", {
"behavior": "allow",
"downloadPath": str(pathlib.Path.home().joinpath("Downloads"))
})
I don't think you should be using the browser for downloading content, leave it to Chrome developers/testers.
I believe you should rather get href attribute of the element you want to download and obtain it using requests library
If your site requires authentication you could fetch cookies from the browser instance and pass them to requests.Session.
I want to allow location and notifications on Chrome in incognito mode using Selenium.
Here is my code:
import selenium
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
option = Options()
option.add_argument("--incognito")
option.add_argument("--disable-infobars")
option.add_argument("start-maximized")
option.add_argument("--disable-extensions")
option.add_experimental_option("prefs", {
"profile.default_content_setting_values.notifications":1,
"profile.default_content_setting_values.geolocation": 1,
})
driver = webdriver.Chrome(options=option, executable_path="path/to/executable")
This code works (Notifications and Location are enabled) if option.add_argument("--incognito") isn't included (the browser is opened in normal mode). Why is this happening and how can I enable these along with incognito mode?
Note: You can check if the location is enabled by open a website that requests your location.
Browser's view in incognito:
Asil Açku based on your question and your comments, I don't have a firm handled on what you are trying to accomplish.
The code below opens a Chrome browser in incognito mode using Selenium. The website being accessed is Google Maps with an exact geolocation. Once the page loads Google automatically obtains my current location. It does this with no browser notifications.
If this isn't what you need please provide more details.
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium import webdriver
capabilities = DesiredCapabilities().CHROME
chrome_options = Options()
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("start-maximized")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")
prefs = {
'profile.default_content_setting_values':
{
'notifications': 1,
'geolocation': 1
},
'profile.managed_default_content_settings':
{
'geolocation': 1
},
}
chrome_options.add_experimental_option('prefs', prefs)
capabilities.update(chrome_options.to_capabilities())
driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)
url='https://www.google.com/maps/#48.1152117,-1.6634771,12.79z/data=!5m1!1e1'
driver.get(url)
# time used only for testing
time.sleep(10)
You mentioned something about "sharing your live location" in Google Maps. Since you want to open Chrome in incognito mode, this capability isn't available. In this cloaked mode your location history or shared location information will not be updated with anyone that you are are sharing this information with.
Google Maps Location Sharing in Chrome
Google Maps Location Sharing in incognito mode
I'm using python-selenium and Chrome 59 and trying to automate a simple download sequence. When I launch the browser normally, the download works, but when I do so in headless mode, the download doesn't work.
# Headless implementation
from selenium import webdriver
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("headless")
driver = webdriver.Chrome(chrome_options=chromeOptions)
driver.get('https://www.mockaroo.com/')
driver.find_element_by_id('download').click()
# ^^^ Download doesn't start
# Normal Mode
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.mockaroo.com/')
driver.find_element_by_id('download').click()
# ^^^ Download works normally
I've even tried adding a default path:
prefs = {"download.default_directory" : "/Users/Chetan/Desktop/"}
chromeOptions.add_argument("headless")
chromeOptions.add_experimental_option("prefs",prefs)
Adding a default path works in the normal implementation, but the same problem persists in the headless version.
How do I get the download to start in headless mode?
Yes, it's a "feature", for security. As mentioned before here is the bug discussion: https://bugs.chromium.org/p/chromium/issues/detail?id=696481
Support was added in chrome version 62.0.3196.0 or above to enable downloading.
Here is a python implementation. I had to add the command to the chromedriver commands. I will try to submit a PR so it is included in the library in the future.
def enable_download_in_headless_chrome(self, driver, download_dir):
# add missing support for chrome "send_command" to selenium webdriver
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': download_dir}}
command_result = driver.execute("send_command", params)
For reference here is a little repo to demonstrate how to use this:
https://github.com/shawnbutton/PythonHeadlessChrome
update 2020-05-01 There have been comments saying this is not working anymore. Given this patch is now over a year old it's quite possible they have changed the underlying library.
Here's a working example for Python based on Shawn Button's answer. I've tested this with Chromium 68.0.3440.75 & chromedriver 2.38
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("prefs", {
"download.default_directory": "/path/to/download/dir",
"download.prompt_for_download": False,
})
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': "/path/to/download/dir"}}
command_result = driver.execute("send_command", params)
driver.get('http://download-page.url/')
driver.find_element_by_css_selector("#download_link").click()
The Chromium developers recently added a 2nd headless mode (in 2021). See https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c36
They later renamed the option in 2023 for Chrome 109 -> https://github.com/chromium/chromium/commit/e9c516118e2e1923757ecb13e6d9fff36775d1f4
For Chrome 109 and above, the --headless=new flag will now allow you to get the full functionality of Chrome in the new headless mode, and you can even run extensions in it. (For Chrome versions 96 through 108, use --headless=chrome)
Usage: (Chrome 109 and above):
options.add_argument("--headless=new")
Usage: (Chrome 96 through Chrome 108):
options.add_argument("--headless=chrome")
If something works in regular Chrome, it should now work with the newer headless mode too.
This is a feature of Chrome to prevent from software to download files to your computer. There is a workaround though. Read more about it here.
What you need to do is enable it via DevTools, Something like that:
async function setDownload () {
const client = await CDP({tab: 'ws://localhost:9222/devtools/browser'});
const info = await client.send('Browser.setDownloadBehavior', {behavior : "allow", downloadPath: "/tmp/"});
await client.close();
}
This is the solution some one gave in the mentioned topic. Here is his comment.
UPDATED PYTHON SOLUTION -
TESTED Mar 4, 2021 on chromedriver v88 and v89
This will allow you to click to download files in headless mode.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
# Instantiate headless driver
chrome_options = Options()
# Windows path
chromedriver_location = 'C:\\path\\to\\chromedriver_win32\\chromedriver.exe'
# Mac path. May have to allow chromedriver developer in os system prefs
'/Users/path/to/chromedriver'
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_prefs = {"download.default_directory": r"C:\path\to\Downloads"} # (windows)
chrome_options.experimental_options["prefs"] = chrome_prefs
driver = webdriver.Chrome(chromedriver_location,options=chrome_options)
# Download your file
driver.get('https://www.mockaroo.com/')
driver.find_element_by_id('download').click()
Maybe the website that you handle returns different HTML pages for browsers, means the XPath or Id that you want maybe differently in headless browser.
Try to download pageSource in headless browser and open it as HTML page to see the Id or XPath that you want.
You can see this as c# example How to hide FirefoxDriver (using Selenium) without findElement function error in PhantomDriver? .
Usually it's redundant seeing the same thing just written in another language, but because this issue drove me crazy, I hope I'm saving someone else from the pain... so here's the C# version of Shawn Button's answer (tested with headless chrome=71.0.3578.98, chromedriver=2.45.615279, platform=Linux 4.9.125-linuxkit x86_64)):
var enableDownloadCommandParameters = new Dictionary<string, object>
{
{ "behavior", "allow" },
{ "downloadPath", downloadDirectoryPath }
};
var result = ((OpenQA.Selenium.Chrome.ChromeDriver)driver).ExecuteChromeCommandWithResult("Page.setDownloadBehavior", enableDownloadCommandParameters);
A full working example for JavaScript with selenium-cucumber-js / selenium-webdriver:
const chromedriver = require('chromedriver');
const selenium = require('selenium-webdriver');
const command = require('selenium-webdriver/lib/command');
const chrome = require('selenium-webdriver/chrome');
module.exports = function() {
const chromeOptions = new chrome.Options()
.addArguments('--no-sandbox', '--headless', '--start-maximized', '--ignore-certificate-errors')
.setUserPreferences({
'profile.default_content_settings.popups': 0, // disable download file dialog
'download.default_directory': '/tmp/downloads', // default file download location
"download.prompt_for_download": false,
'download.directory_upgrade': true,
'safebrowsing.enabled': false,
'plugins.always_open_pdf_externally': true,
'plugins.plugins_disabled': ["Chrome PDF Viewer"]
})
.windowSize({width: 1600, height: 1200});
const driver = new selenium.Builder()
.withCapabilities({
browserName: 'chrome',
javascriptEnabled: true,
acceptSslCerts: true,
path: chromedriver.path
})
.setChromeOptions(chromeOptions)
.build();
driver.manage().window().maximize();
driver.getSession()
.then(session => {
const cmd = new command.Command("SEND_COMMAND")
.setParameter("cmd", "Page.setDownloadBehavior")
.setParameter("params", {'behavior': 'allow', 'downloadPath': '/tmp/downloads'});
driver.getExecutor().defineCommand("SEND_COMMAND", "POST", `/session/${session.getId()}/chromium/send_command`);
return driver.execute(cmd);
});
return driver;
};
The key part is:
driver.getSession()
.then(session => {
const cmd = new command.Command("SEND_COMMAND")
.setParameter("cmd", "Page.setDownloadBehavior")
.setParameter("params", {'behavior': 'allow', 'downloadPath': '/tmp/downloads'});
driver.getExecutor().defineCommand("SEND_COMMAND", "POST", `/session/${session.getId()}/chromium/send_command`);
return driver.execute(cmd);
});
Tested with:
Chrome 67.0.3396.99
Chromedriver 2.36.540469
selenium-cucumber-js 1.5.12
selenium-webdriver 3.0.0
Following is the equivalent in Java, selenium, chromedriver and chrome v 71.x. The code in is the key to allow saving of downloads
Additional jars: com.fasterxml.jackson.core, com.fasterxml.jackson.annotation, com.fasterxml.jackson.databind
System.setProperty("webdriver.chrome.driver","C:\libraries\chromedriver.exe");
String downloadFilepath = "C:\\Download";
HashMap<String, Object> chromePreferences = new HashMap<String, Object>();
chromePreferences.put("profile.default_content_settings.popups", 0);
chromePreferences.put("download.prompt_for_download", "false");
chromePreferences.put("download.default_directory", downloadFilepath);
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setBinary("C:\\pathto\\Chrome SxS\\Application\\chrome.exe");
//ChromeOptions options = new ChromeOptions();
//chromeOptions.setExperimentalOption("prefs", chromePreferences);
chromeOptions.addArguments("start-maximized");
chromeOptions.addArguments("disable-infobars");
//HEADLESS CHROME
**chromeOptions.addArguments("headless");**
chromeOptions.setExperimentalOption("prefs", chromePreferences);
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
**ChromeDriverService driverService = ChromeDriverService.createDefaultService();
ChromeDriver driver = new ChromeDriver(driverService, chromeOptions);
Map<String, Object> commandParams = new HashMap<>();
commandParams.put("cmd", "Page.setDownloadBehavior");
Map<String, String> params = new HashMap<>();
params.put("behavior", "allow");
params.put("downloadPath", downloadFilepath);
commandParams.put("params", params);
ObjectMapper objectMapper = new ObjectMapper();
HttpClient httpClient = HttpClientBuilder.create().build();
String command = objectMapper.writeValueAsString(commandParams);
String u = driverService.getUrl().toString() + "/session/" + driver.getSessionId() + "/chromium/send_command";
HttpPost request = new HttpPost(u);
request.addHeader("content-type", "application/json");
request.setEntity(new StringEntity(command));**
try {
httpClient.execute(request);
} catch (IOException e2) {
// TODO Auto-generated catch block
e2.printStackTrace();
}**
//Continue using the driver for automation
driver.manage().window().maximize();
I solved this problem by using the workaround shared by #Shawn Button and using the full path for the 'downloadPath' parameter. Using a relative path did not work and give me the error.
Versions:
Chrome Version 75.0.3770.100 (Official Build) (32-bit)
ChromeDriver 75.0.3770.90
Using: google-chrome-stable amd64 86.0.4240.111-1,chromedriver 86.0.4240.22, selenium 3.141.0 python 3.8.3
Tried multiple proposed solutions, and nothing really worked for chrome headless, also my testing website opens a new blank tab and then the data is downloaded.
Finally gave up on headless and implemented pyvirtualdisplay and xvfd to emulate X server, something like:
from selenium.webdriver.chrome.options import Options # and other imports
import selenium.webdriver as webdriver
import tempfile
url = "https://really_badly_programmed_website.org"
tmp_dir = tempfile.mkdtemp(prefix="hamster_")
driver_path="/usr/bin/chromedriver"
chrome_options = Options()
chrome_options.binary_location = "/usr/bin/google-chrome"
prefs = {'download.default_directory': tmp_dir,}
chrome_options.add_experimental_option("prefs", prefs)
with Display(backend="xvfb",size=(1920,1080),color_depth=24) as disp:
driver = webdriver.Chrome(options=chrome_options, executable_path=driver_path)
driver.get(url)
At the end everything worked and had the dowload file on the tmp folder.
I finally got it to work by upgrading to Chromium 90! I previously had version 72-78, but I saw that it had been fixed recently: https://bugs.chromium.org/p/chromium/issues/detail?id=696481 so i decided to give it a shot.
So after upgrading, which took a while (home brew in MacOS is so slow...), I simply did, without setting options or anything (this is a JavaScript example):
await driver.findElement(By.className('download')).click();
And it worked! I saw the downloaded PDF in the same working folder that I had been trying to download for a long time...