Develop Reference

python selenium get_log("performance") won't log webworker requests

following many article one can log XHR calls in an automated browser (using selenium) as bellow:
capabilities = DesiredCapabilities.CHROME
capabilities["loggingPrefs"] = {"performance": "ALL"} # newer: goog:loggingPrefs driver = webdriver.Chrome(
desired_capabilities=capabilities, executable_path="./chromedriver" )
...
logs_raw = driver.get_log("performance")
my probleme is the target request is performed by a "WebWorker", so it's not listed in the performance list of the browser main thread.
getting in chrome and manualy selecting the webworkers scope in the dev console "performance.getEntries()" gets me the request i want;
my question is how can someone perform such an action in selenium ? (python preferable).
no where in the doc of python selenium or Devtool Api have i found something similar
i'm so greatful in advance.
Edit: after some deggin i found that it has something to do with execution context of javascrip, i have no clue how to switch that in selenium

Find out the current proxy used by selenium

Following this answer, I changed the proxy of my driver like so:
from selenium import webdriver
def set_proxy(driver, http_addr='', http_port=0, ssl_addr='', ssl_port=0, socks_addr='', socks_port=0):
driver.execute("SET_CONTEXT", {"context": "chrome"})
try:
driver.execute_script("""
Services.prefs.setIntPref('network.proxy.type', 1);
Services.prefs.setCharPref("network.proxy.http", arguments[0]);
Services.prefs.setIntPref("network.proxy.http_port", arguments[1]);
Services.prefs.setCharPref("network.proxy.ssl", arguments[2]);
Services.prefs.setIntPref("network.proxy.ssl_port", arguments[3]);
Services.prefs.setCharPref('network.proxy.socks', arguments[4]);
Services.prefs.setIntPref('network.proxy.socks_port', arguments[5]);
""", http_addr, http_port, ssl_addr, ssl_port, socks_addr, socks_port)
finally:
driver.execute("SET_CONTEXT", {"context": "content"})
driver = webdriver.Firefox()
set_proxy(driver, http_addr="212.35.56.21", http_port=8080)
driver.get("https://google.fr/")
It ran, but I'm not sure if the firefox browser is using the proxy that I gave to it or not. I thus have two questions:
Is this the correct way to change the proxy?
How can I get the value of the current proxy used by the driver?

Cannot attach to an existing Selenium session via geckodriver

After upgrading to geckodriver I'm unable to reuse my Selenium's sessions. Here's my setup:
I have a start_browser.py script, which launches a Firefox instance and prints a port to connect to, like:
firefox_capabilities = DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
browser = webdriver.Firefox(capabilities=firefox_capabilities)
print browser.service.port
wait_forever()
... and another script, which tries to connect to the existing instance via Remote driver:
caps = DesiredCapabilities.FIREFOX
caps['marionette'] = True
driver = webdriver.Remote(
command_executor='http://localhost:{port}'.format(port=port),
desired_capabilities=caps)
But it seems to be trying to launch a new session, and failing with a message:
selenium.common.exceptions.WebDriverException: Message: Session is already started
Is there an ability to just attach to the existing session, like in previous versions of Selenium? Or is this an intended behaviour of geckodriver (hope not)?

Alright, so unless anyone comes up with more elegant solution, here's a quick dirty hack:
class SessionRemote(webdriver.Remote):
def start_session(self, desired_capabilities, browser_profile=None):
# Skip the NEW_SESSION command issued by the original driver
# and set only some required attributes
self.w3c = True
driver = SessionRemote(command_executor=url, desired_capabilities=caps)
driver.session_id = session_id
The bad thing is that it still doesn't work, complaining that it doesn't know the moveto command, but at least it connects to a launched browser.
Update: Well, geckodriver seems to lack some functionality at the moment, so if you guys are going to keep using Firefox, just downgrade it to a version which supports old webdriver (45 plays fine), and keep an eye on tickets like https://github.com/SeleniumHQ/selenium/issues/2285 .

You can reconnect to a session by using the session id.
firefox_capabilities = DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
browser = webdriver.Firefox(capabilities=firefox_capabilities)
print browser.service.port
wait_forever()
# get the ID and URL from the browser
url = browser.command_executor._url
session_id = browser.session_id
# Connect to the existing instance
driver = webdriver.Remote(command_executor=url,desired_capabilities={})
driver.session_id = session_id

Using Selenium to By-pass SSL Certification In Internet Explorer (Python)

I have done some searching on this issue and have not found any working solutions. Essentially, I am trying to write a script that utilizes Selenium to open a particular site and enter user login information. However, Internet Explorer returns a certificate warning as shown:
Here is the code I am using
class PipelinePilotControl:
user_id = str(input("Please enter your username.\n"))
user_password = getpass.getpass(prompt="Please enter your password.\n")
def pipelinepilot_login(self):
ie_browser_driver = webdriver.Ie()
ie_browser_driver.get("url to be accessed")
user_login = ie_browser_driver.find_element_by_id("txtUsername")
password_login = ie_browser_driver.find_element_by_id("txtPassword")
login_button = ie_browser_driver.find_element_by_id("btnLogin")
user_login.send_keys(self.user_id)
password_login.send_keys(self.user_password)
login_button.send_keys(Keys.ENTER)
I have tried using the html and the get_element_by_id functionality in selenium as well as capability controls but nothing has worked. Any suggestions?
Thanks,
Travis

SSL error page in IE isn't Shadow DOM, you can navigate on the page and just click on "Continue".
Try 'find_element' :
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = webdriver.DesiredCapabilities().INTERNETEXPLORER
caps['acceptSslCerts'] = True
driver = webdriver.Ie(capabilities=caps)
driver.get('https://yourwebsite.com/')
overrideLink = driver.find_element_by_id('overridelink')
overrideLink.click()

How to capture network traffic using selenium webdriver and browsermob proxy on Python?

I would like to capture network traffic by using Selenium Webdriver on Python. Therefore, I must use a proxy (like BrowserMobProxy)
When I use webdriver.Chrome:
from browsermobproxy import Server
server = Server("~/browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
co = webdriver.ChromeOptions()
co.add_argument('--proxy-server={host}:{port}'.format(host='localhost', port=proxy.port))
driver = webdriver.Chrome(executable_path = "~/chromedriver", chrome_options=co)
proxy.new_har
driver.get(url)
proxy.har # returns a HAR
for ent in proxy.har['log']['entries']:
print ent['request']['url']
the webpage is loaded properly and all requests are available and accessible in the HAR file.
But when I use webdriver.Firefox:
# The same as above
# ...
from selenium import webdriver
profile = webdriver.FirefoxProfile()
driver = webdriver.Firefox(firefox_profile=profile, proxy = proxy.selenium_proxy())
proxy.new_har
driver.get(url)
proxy.har # returns a HAR
for ent in proxy.har['log']['entries']:
print ent['request']['url']
The webpage cannot be loaded properly and the number of requests in the HAR file is smaller than the number of requests that should be.
Do you have any idea what the problem of proxy settings in the second code? How should I fix it to use webdriver.Firefox properly for my purpose?

Just stumbled across this project https://github.com/derekargueta/selenium-profiler. Spits out all network data for a URL. Shouldn't be hard to hack and integrate into whatever tests you're running.
Original source: https://www.openhub.net/p/selenium-profiler

For me, following code component works just fine.
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)

I am sharing my solution, this would not write any logs to any file
but you can collect all sort of messages such as Errors,
Warnings, Logs, Info, Debug , CSS, XHR as well as Requests(traffic)
1. We are going to create Firefox profile so that we can enable option of
"Persist Logs" on Firefox (you can try it to enable on your default browser and see
if it launches with "Persist Logs" without creating firefox profile )
2. we need to modify the Firefox initialize code
where this line will do magic : options.AddArgument("--jsconsole");
so complete Selenium Firefox code would be, this will open Browser Console
everytime you execute your automation :
else if (browser.Equals(Constant.Firefox))
{
var profileManager = new FirefoxProfileManager();
FirefoxProfile profile = profileManager.GetProfile("ConsoleLogs");
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(DrivePath);
service.FirefoxBinaryPath = DrivePath;
profile.SetPreference("security.sandbox.content.level", 5);
profile.SetPreference("dom.webnotifications.enabled", false);
profile.AcceptUntrustedCertificates = true;
FirefoxOptions options = new FirefoxOptions();
options.AddArgument("--jsconsole");
options.AcceptInsecureCertificates = true;
options.Profile = profile;
options.SetPreference("browser.popups.showPopupBlocker", false);
driver = new FirefoxDriver(service.FirefoxBinaryPath, options, TimeSpan.FromSeconds(100));
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);
}
3. Now you can write your logic since you have traffic/ logging window open so don't
go to next execution if test fails. That way Browser Console will keep your errors
messages and help you to troubleshoot further
Browser : Firefox v 61
How can you launch Browser Console for firefox:
1. open firefox (and give any URL )
2. Press Ctrl+Shift+J (or Cmd+Shift+J on a Mac)
Link : https://developer.mozilla.org/en-US/docs/Tools/Browser_Console