Develop Reference

What is preventing my Chrome profile from working with Selenium in Python?

I am trying to use Python to automate a number of work processes that primarily take place in an online content management system, and to access the CMS, I need to be logged into my work profile on Chrome. While I could sign in once Chromedriver is open, I would have to approve the login on my authenticator each time, somewhat defeating the purpose of automation.
I am using the following code for opening Chromedriver and attempting to load my regular Chrome profile. However, while the rest of the code works as intended, it does not load my profile and I am unsure why.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--start-maximized")
options.add_argument("--user-data-dir=C:\\Users\\Saul\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
driver = webdriver.Chrome("C:\\Users\\Saul\\Downloads\\chromedriver_win32\\chromedriver.exe", options=options)
driver.get("https://myaccount.google.com/")
while(True):
pass
I do get the following error while running the code, but I don't believe this is affecting my issue.
C:\Users\Saul\PycharmProjects\OnlineAutomation\main.py:12: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome("C:\\Users\\Saul\\Downloads\\chromedriver_win32\\chromedriver.exe", options=options)

Your question is not related to the error you mentioned.
For the error - 'DeprecationWarning: executable_path has been deprecated, please pass in a Service object', you have to use Service:
from selenium.webdriver.chrome.service import Service
driver = webdriver.Chrome(service=Service("C:\\Users\\Saul\\Downloads\\chromedriver_win32\\chromedriver.exe"), options=options)
For the Chrome profile issue, you have to mention the profile path like below:
options.add_argument("--user-data-dir=C:\\Users\\Saul\\AppData\\Local\\Google\\Chrome\\User Data")
options.add_argument("--profile-directory=Default")

If your website has MFA enabled, you might be able to use SeleniumBase (a Python framework), which has methods for handling MFA Login:
📘📝 An example test with the BaseCase class. Runs with pytest or nosetests. (Learn more)
from seleniumbase import BaseCase
class TestMFALogin(BaseCase):
def test_mfa_login(self):
self.open("https://seleniumbase.io/realworld/login")
self.type("#username", "demo_user")
self.type("#password", "secret_pass")
self.enter_mfa_code("#totpcode", "GAXG2MTEOR3DMMDG") # 6-digit
self.assert_exact_text("Welcome!", "h1")
self.assert_element("img#image1")
self.click('a:contains("This Page")')
self.save_screenshot_to_logs()
📗📝 An example test with the sb pytest fixture. Runs with pytest.
def test_mfa_login(sb):
sb.open("https://seleniumbase.io/realworld/login")
sb.type("#username", "demo_user")
sb.type("#password", "secret_pass")
sb.enter_mfa_code("#totpcode", "GAXG2MTEOR3DMMDG") # 6-digit
sb.assert_exact_text("Welcome!", "h1")
sb.assert_element("img#image1")
sb.click('a:contains("This Page")')
sb.save_screenshot_to_logs()
📙📝 An example test with the SB Context Manager. Runs with pure python.
from seleniumbase import SB
with SB() as sb: # By default, browser="chrome" if not set.
sb.open("https://seleniumbase.github.io/realworld/login")
sb.type("#username", "demo_user")
sb.type("#password", "secret_pass")
sb.enter_mfa_code("#totpcode", "GAXG2MTEOR3DMMDG") # 6-digit
sb.assert_text("Welcome!", "h1")
sb.highlight("img#image1") # A fancier assert_element() call
sb.click('a:contains("This Page")') # Use :contains() on any tag
sb.click_link("Sign out") # Link must be "a" tag. Not "button".
sb.assert_element('a:contains("Sign in")')
sb.assert_exact_text("You have been signed out!", "#top_message")
There's a SeleniumBase pytest command-line option, --user-data-dir=DIR, which you can add to set a user data directory to use. Eg:
pytest --user-data-dir=DIR

Save PDF by using Selenium and IE Browser

To save PDF by using CHrome Browser does not cause any issues (I'm using these options):
options.add_experimental_option('prefs',{
'credentials_enable_service': False,
'plugins':{
'always_open_pdf_externally': True
},
'profile': {
'password_manager_enabled': False,
},
'download': {
'prompt_for_download': False,
'directory_upgrade': True,
'default_directory': ''
}
})
BUT .... How to save PDF by using webdriver.Ie() Internet Explorer Driver with Python + Selenium?
P.S. AFAIK Internet explorer can not be executed by using headless mode, but if someone will not the way to do it, will be amazing !!!

You can't use Selenium to deal with the download prompt in IE because that's an OS-level prompt. Selenium WebDriver has no capability to automate OS-level prompt window. You need to use some 3rd party tools to help you to download file in IE using Selenium.
Here I use Wget to bypass the download prompt and download file in IE. You can refer to this article about how to use Wget.
About using headless mode in IE in Selenium, you can also use a 3rd party tool called headless_ie_selenium. You can download this tool and use headless_ie_selenium.exe instead of IEDriverServer.exe to automate IE.
The sample code to download a pdf file is like below, please note to change the paths in the code to your owns:
from selenium import webdriver
import time
import os
url = "https://file-examples.com/index.php/sample-documents-download/sample-pdf-download/"
driver = webdriver.Ie('D:\\headless-selenium-for-win-v1-4\\headless_ie_selenium.exe')
driver.get(url)
time.sleep(3)
link = driver.find_elements_by_class_name("download-button")[0]
hrefurl = link.get_attribute("href")
os.system('cmd /c C:\\Wget\\wget.exe -P D:\\Download --no-check-certificate ' + hrefurl)
print("*******************")

Firefox Browser Extensions not loading: Selenium with Python 2.7

Using Firefox 54, Python 2.7 and Selenium 3.8.1 I'm trying to load some extensions using a Firefox Profile but Selenium fails to load them. The browser, however, does load with this code:
from selenium import webdriver
def setup_firefox_profile():
firefox_custom_profile = webdriver.FirefoxProfile(
r'C:\Users\Owner\AppData\Roaming\Mozilla\Firefox\Profiles\1hmz9w1k.GenericFootprintHardened2')
firefox_custom_profile.add_extension(r'C:\Users\Owner\AppData\Roaming\Mozilla\Firefox\Profiles\1hmz9w1k.GenericFootprintHardened2\extensions\bestproxyswitcher#bestproxyswitcher.com.xpi')
# C:\Users\Owner\AppData\Roaming\Mozilla\Firefox\Profiles\<profile folder>
# Adblock, Best Proxy Switcher, //BetterPrivacy, Clear Flash Cookies, Canvas Fingerprint Blocker, Cookie Quick Manager, Disable WebRTC, Ghostery, Privacy Badger,
firefox_custom_profile.set_preference("extensions.bestproxyswitcher.currentVersion", "5.4.2")
firefox_settings = webdriver.Firefox(firefox_profile=firefox_custom_profile)
return firefox_settings
driver = setup_firefox_profile()
driver.get('http://duckduckgo.com')
Attempting to load the profile with a manual session I am able to see the extensions load.
It appears the profile does not load correctly. I found instructions to go to "about:support" and then "Open Folder" under the Profile Folder heading. The directory did not match the profile in name or path ("C:\Users\Owner\AppData\Local\Temp\tmpp4lzxe")
Can anyone see why the profile is not being loaded?
Is the code for loading the extensions also wrong?

How to capture network traffic using selenium webdriver and browsermob proxy on Python?

I would like to capture network traffic by using Selenium Webdriver on Python. Therefore, I must use a proxy (like BrowserMobProxy)
When I use webdriver.Chrome:
from browsermobproxy import Server
server = Server("~/browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server={host}:{port}'.format(host='localhost', port=proxy.port))
driver = webdriver.Chrome(executable_path = "~/chromedriver", chrome_options=co)
proxy.new_har
driver.get(url)
proxy.har # returns a HAR
for ent in proxy.har['log']['entries']:
print ent['request']['url']
the webpage is loaded properly and all requests are available and accessible in the HAR file.
But when I use webdriver.Firefox:
# The same as above
# ...
from selenium import webdriver
profile = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=profile, proxy = proxy.selenium_proxy())
proxy.new_har
driver.get(url)
proxy.har # returns a HAR
for ent in proxy.har['log']['entries']:
print ent['request']['url']
The webpage cannot be loaded properly and the number of requests in the HAR file is smaller than the number of requests that should be.
Do you have any idea what the problem of proxy settings in the second code? How should I fix it to use webdriver.Firefox properly for my purpose?

Just stumbled across this project https://github.com/derekargueta/selenium-profiler. Spits out all network data for a URL. Shouldn't be hard to hack and integrate into whatever tests you're running.
Original source: https://www.openhub.net/p/selenium-profiler

For me, following code component works just fine.
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)

I am sharing my solution, this would not write any logs to any file
but you can collect all sort of messages such as Errors,
Warnings, Logs, Info, Debug , CSS, XHR as well as Requests(traffic)
1. We are going to create Firefox profile so that we can enable option of
"Persist Logs" on Firefox (you can try it to enable on your default browser and see
if it launches with "Persist Logs" without creating firefox profile )
2. we need to modify the Firefox initialize code
where this line will do magic : options.AddArgument("--jsconsole");
so complete Selenium Firefox code would be, this will open Browser Console
everytime you execute your automation :
else if (browser.Equals(Constant.Firefox))
{
var profileManager = new FirefoxProfileManager();
FirefoxProfile profile = profileManager.GetProfile("ConsoleLogs");
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(DrivePath);
service.FirefoxBinaryPath = DrivePath;
profile.SetPreference("security.sandbox.content.level", 5);
profile.SetPreference("dom.webnotifications.enabled", false);
profile.AcceptUntrustedCertificates = true;
FirefoxOptions options = new FirefoxOptions();
options.AddArgument("--jsconsole");
options.AcceptInsecureCertificates = true;
options.Profile = profile;
options.SetPreference("browser.popups.showPopupBlocker", false);
driver = new FirefoxDriver(service.FirefoxBinaryPath, options, TimeSpan.FromSeconds(100));
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);
}
3. Now you can write your logic since you have traffic/ logging window open so don't
go to next execution if test fails. That way Browser Console will keep your errors
messages and help you to troubleshoot further
Browser : Firefox v 61
How can you launch Browser Console for firefox:
1. open firefox (and give any URL )
2. Press Ctrl+Shift+J (or Cmd+Shift+J on a Mac)
Link : https://developer.mozilla.org/en-US/docs/Tools/Browser_Console

Firefox + Selenium WebDriver and download a csv file automatically

I have problem with Selenium WebDriver and Firefox. I want to download csv file without confirmation in dialog window and I have code like this:
fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.folderList",2)
fp.set_preference("browser.download.dir", download_dir)
fp.set_preference("browser.download.manager.showWhenStarting",False)
fp.set_preference("browser.helperApps.neverAsk.saveToDisk","text/csv")
but it seems not working.
I tried many combination with browser.helperApps.neverAsk.saveToDisk
fp.set_preference("browser.helperApps.neverAsk.saveToDisk","text/csv,application/csv,text/plan,text/comma-separated-values")
or
fp.set_preference("browser.helperApps.neverAsk.saveToDisk","application/csv")
fp.set_preference("browser.helperApps.neverAsk.saveToDisk","text/plain")
fp.set_preference("browser.helperApps.neverAsk.saveToDisk","text/comma-separated-values")
but there's no difference and Firefox won't download automaticly.
How can I fix it?

Sometime the content type is not as you'd expect
Use HttpFox Firefox plugin (or similar) to find the real content type of the file and use it in your code
BTW, For me the content type was
fp.set_preference("browser.helperApps.neverAsk.openFile", "application/octet-stream");
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/octet-stream");

SetPreference("browser.helperApps.neverAsk.saveToDisk", "application/comma-separated-values ,text/csv"); //in java selenium
this will work for downloading all type of csv files...
thanks, enjoy....

Now (May 2016),
SetPreference("browser.helperApps.neverAsk.saveToDisk", "text/csv"); // C#
works for me