Develop Reference

python selenium Access to script at https://sitesA.com from origin https://sitesB.com has been blocked by CORS policy only in headless mode

I'm building a python program that using selenium and chrome driver to download .xml and .pdf files from a website, it's running fine, but when i turn the driver into headless mode then "CORS policy occur", I already try adding "disable-web-security" and "--disable-site-isolation-trials" after spend alot of time searching on internet but still no luck,So anyone please tell me what am I missing? What am I doing wrong? this is how I implement chrome driver:
options = webdriver.ChromeOptions()
options.add_argument('no-sandbox')
options.add_argument('--disable-gpu')
options.add_argument('--disable-extensions')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument('--safebrowsing-disable-download-protection')
options.add_argument('--start-maximized')
options.headless = True
extset = ['enable-automation', 'ignore-certificate-errors']
options.add_experimental_option('excludeSwitches', extset)
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--ignore-certificate-errors')
options.add_argument('--allow-insecure-localhost')
options.add_argument('--safebrowsing-disable-download-protection')
options.add_argument('--disable-web-security')
options.add_argument('--disable-site-isolation-trials')
options.add_argument('safebrowsing-disable-extension-blacklist')
prefs = {
'download.default_directory' : DOWNLOAD_DIRECTORY, # set up download directory
'safebrowsing.enabled': True, # disable xml download asking
'profile.default_content_setting_values.automatic_downloads': 1, # allow multiple files download
'download.prompt_for_download': False
}
if rpa.get('options') == 'AUTO_DOWNLOAD_PDF':
prefs.update({ 'plugins.plugins_list': [{ 'enabled': False, 'name': 'Chrome PDF Viewer' }] })
prefs.update({ 'plugins.always_open_pdf_externally': True })
prefs.update({ 'browser.helperApps.neverAsk.saveToDisk': 'application/pdf,application/vnd.adobe.xfdf,application/vnd.fdf,application/vnd.adobe.xdp+xml' })
options.add_experimental_option('prefs', prefs)
driver = webdriver.Chrome(os.getcwd()+'\\webdriver\\chromedriver.exe', options=options)

I'm setting the download path option and that exception is appears only when I used chromdriver.v2 as uc

Exception: selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: cannot parse capability: goog:chromeOptions
from invalid argument: unrecognized chrome option: prefs
Chrome options I'm using:
import undetected_chromedriver.v2 as uc
chrome_options = uc.ChromeOptions()
chrome_options.add_argument("--window-size=800,800")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--start-maximized")
chrome_options.user_data_dir = project_settings.LOCAL_PROFILES_STORAGE_PATH + "\\" + phone number
download_path = "D://driver"
preferences = {"download.default_directory": download_path}
chrome_options.add_experimental_option('prefs', preferences)
chrome_options.to_capabilities()
chrome_options.add_argument('--no-first-run --no-service-autorun --password-store=basic')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--profile-directory=Default')
return uc.Chrome(options=chrome_options)

On Linux, you can use download.default_directory to specify a download folder for undetected_chromedriver. For example,
import undetected_chromedriver.v2 as uc
options.add_argument("--download.default_directory --your_download_folder_path")
driver = uc.Chrome(options=options)
If the method above doesn't work or you're on a Windows machine, you can try the following method:
options = uc.ChromeOptions()
driver = uc.Chrome(options=options)
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior',
'params': {'behavior': 'allow', 'downloadPath': your_download_folder_path}}
driver.execute("send_command", params)

How to Use Selenium Webdriver to download files via a list of URLs

I wrote a code that use Selenium Webdriver to download files via a list of URLs but for some reason it didn't download anything to my assignedn directory. The code works perfectly fine when I only download it one by one but when I use a for loop, it doesn't work.
This is an example URL: https://www.regulations.gov/contentStreamer?documentId=WHD-2020-0007-1730&attachmentNumber=1&contentType=pdf
Here is my code:
download_dir = '/Users/datawizard/files/'
for web in down_link:
try:
options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_experimental_option("prefs", {
"download.default_directory": '/Users/clinton/GRA_2021/scraping_project/pdf/',
"download.prompt_for_download": False,
"download.directory_upgrade": True,
# "safebrowsing.enabled": True,
"plugins.always_open_pdf_externally": True
})
driver = webdriver.Chrome(chrome_options=options)
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': download_dir}}
command_result = driver.execute("send_command", params)
driver.get(url)
except:
print(str(web)+"Link cannot be open")
I am wondering did I do something wrong with the code since it doesn't give me any error when I ran the code above.

You don't need Selenium to download files, you can download files easily using the request library
import requests
for web in down_link:
fileName = YOUR_DOWNLOAD_PATH + web.split("=")[1].split("&")[0] + ".pdf" #I created a filename
r = requests.get(web, stream=True)
with open(fileName, 'wb') as f:
for chunk in r.iter_content():
f.write(chunk)
Updated Answer based on Selenium
#replace the below value with your urls list
down_link = [
'https://www.regulations.gov/contentStreamer?documentId=WHD-2020-0007-1730&attachmentNumber=1&contentType=pdf',
'https://www.regulations.gov/contentStreamer?documentId=WHD-2020-0007-1730&attachmentNumber=1&contentType=pdf']
download_dir = "/Users/datawizard/files/"
options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_experimental_option("prefs", {
"download.default_directory": download_dir,
"download.prompt_for_download": False,
"download.directory_upgrade": True,
"plugins.always_open_pdf_externally": True
})
driver = webdriver.Chrome(chrome_options=options)
for web in down_link:
driver.get(web)
time.sleep(5) #wait for the download to end, a better handling it's to check if the file exists
driver.quit()
If your files don't have a unique file name - the above code will replace the existing file with the downloaded one.

File Not Saving While Downloading File in Headless chrome using Selenium in python

I am able to download file in normal chrome mode. where as, i am not able to see the download happening in headless chrome using selenium python.
I hope it is not saving the file downloaded
Tried with solutions provided by many users in internet but none of them works
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
prefs = {'download.default_directory' :'/Users/nrpss/Downloads'}
options.add_experimental_option('prefs', prefs)
download_path = '/Users/nrpss/Downloads'
browser = webdriver.Chrome('chromedriver.exe', options=options)
browser.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': download_path}}
command_result = browser.execute("send_command", params)
print ("Headless Chrome Initiated")
### Below is ID for the Download link on webpage
browser.find_element_by_id('downloadExportLink').click()
time.sleep(50)
def download_completed():
for i in os.listdir('/Users/nrpss/Downloads'):
if ".crdownload" in i:
time.sleep(1)
download_completed()
Expected result: File should be downloaded and saved in downloads folder.

Try adding download.prompt_for_download = False and download.directory_upgrade = True you car set safebrowsing_for_trusted_sources_enabled to False as well as safebrowsing.enabled.
try changing your prefs to:
prefs = {'download.default_directory' :'/Users/nrpss/Downloads',
"download.prompt_for_download": False,
"download.directory_upgrade": True,
"safebrowsing_for_trusted_sources_enabled": False,
"safebrowsing.enabled": False
}
options.add_experimental_option('prefs', prefs)
Hope this helps you!

to enable headless downloads in Python:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = Chrome(options=options)
params = {'behavior': 'allow', 'downloadPath': '/path/for/download'}
driver.execute_cdp_cmd('Page.setDownloadBehavior', params)

Downloading a file at a specified location through python and selenium using Chrome driver

I am trying to automatically download some links through selenium's click functionality and I am using a chrome webdriver and python as the programming language. How can I select the download directory through the python program so that it does not get downloaded in the default Downloads directory. I found a solution for firefox but there the download dialog keeps popping up every time it clicks on the link which does not happen in Chrome.

I found the accepted solution didn't work, however this slight change did:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
prefs = {'download.default_directory' : '/path/to/dir'}
chrome_options.add_experimental_option('prefs', prefs)
driver = webdriver.Chrome(chrome_options=chrome_options)

Update 2018:
Its not valid Chrome command line switch, see the source code use hoju answer below to set the Preferences.
Original:
You can create a profile for chrome and define the download location for the tests. Here is an example:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("download.default_directory=C:/Downloads")
driver = webdriver.Chrome(chrome_options=options)

the exact problem I also have faced while trying to do exactly same what you want to :)
For chrome:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--start-maximized")
prefs = {"profile.default_content_settings.popups": 0,
"download.default_directory":
r"C:\Users\user_dir\Desktop\\",#IMPORTANT - ENDING SLASH V IMPORTANT
"directory_upgrade": True}
options.add_experimental_option("prefs", prefs)
browser=webdriver.Chrome(<chromdriver.exe path>, options=options)
For Firefox:
follow this blog for the answer:
https://srirajeshsahoo.wordpress.com/2018/07/26/how-to-bypass-pop-up-during-download-in-firefox-using-selenium-and-python/
The blog says all about the pop-up and downloads dir and how to do

Using prefs solved my problem
path = os.path.dirname(os.path.abspath(__file__))
prefs = {"download.default_directory":path}
options = Options()
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome('../harveston/chromedriver.exe',options = options)

This worked for me on Chrome v81.0.4044.138
preferences = {
"profile.default_content_settings.popups": 0,
"download.default_directory": os.getcwd() + os.path.sep,
"directory_upgrade": True
}
chrome_options.add_experimental_option('prefs', preferences)
browser = webdriver.Chrome(executable_path="/usr/bin/chromedriver", options=chrome_options)

I see that many people have the same problem, just add the backslash at the end
op = webdriver.ChromeOptions()
prefs = {'download.default_directory' : 'C:\\Users\\SAJComputer\\PycharmProjects\\robot-test\\'}
op.add_experimental_option('prefs', prefs)
driver = webdriver.Chrome(executable_path=driver_path , options=op)

Update 2022:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
prefs = {"download.default_directory" : "C:\YourDirectory\Folder"}
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

To provide download directory and chrome's diver executable path use the following code.
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("download.default_directory=C:/Your_Directory")
driver = webdriver.Chrome(options=options ,executable_path='C:/chromedriver')
change the path in your code accordingly.

If you are using linux distribution
Use this code
prefs = {'download.prompt_for_download': False,
'download.directory_upgrade': True,
'safebrowsing.enabled': False,
'safebrowsing.disable_download_protection': True}
options.add_argument('--headless')
options.add_experimental_option('prefs', prefs)
driver = webdriver.Chrome('chromedriver.exe', chrome_options=options)
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
driver.desired_capabilities['browserName'] = 'ur mum'
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': r'C:\chickenbutt'}}
driver.execute("send_command", params)

Below code snippet holds good for Windows/linux/MacOs distro:
downloadDir = f"{os.getcwd()}//downloads//"
# Make sure path exists.
Path(downloadDir).mkdir(parents=True, exist_ok=True)
# Set Preferences.
preferences = {"download.default_directory": downloadDir,
"download.prompt_for_download": False,
"directory_upgrade": True,
"safebrowsing.enabled": True}
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("--window-size=1480x560")
chromeOptions.add_experimental_option("prefs", preferences)
driver = webdriver.Chrome(DRIVER_PATH, options=chromeOptions)
driver.get(url)
time.sleep(10)
driver.close()

This is non code level solution with no chrome profiling/options settings.
If you are using script only on your local machine then use this solution
Click on Menu -> Setting -> Show advanced settings... -> Downloads
Now uncheck
Ask where to save each file before downloading
Hope it will help you :)