Selenium headless not working with custom profile selection - python

Below is the code I'm using - I've attempted to make it as concise as possible.
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import bs4
options=Options()
#options.add_argument('--headless') # Works while not headless?
options.add_argument('--disable-gpu') # Last I checked this was necessary.
options.add_argument("--user-data-dir=profiles\\") #Keeps login data
options.add_argument("--profile-directory=Profile 1")
driver=webdriver.Chrome(chrome_options=options)
driver.get("http://www.google.com")
html=driver.page_source
soup=bs4.BeautifulSoup(html, "html.parser")
print(soup)
The main problem stems from the --user-data-dir and --profile-directory parameter. In my test example a custom chrome profile (usually found at C:\Users\User\AppData\Local\Google\Chrome\User Data) is in the current directory to keep it separate from any currently running chrome sessions.
If the --headless parameter is enabled while using the above parameters the driver hangs (the CMD stays and does not produce output on the Python command line). However, when not enabled the window pops open and performs as expected.
However, --headless does work while using any profile within the above default directory.
This is the console output;
[0120/222514.611:ERROR:gpu_process_transport_factory.cc(967)] Lost UI shared context.
DevTools listening on ws://127.0.0.1:59961/devtools/browser/ee317ed6-93c7-47c2-b26d-63647980ba0d
[0120/222514.619:ERROR:devtools_http_handler.cc(289)] Error writing DevTools active port to file
[0120/222514.624:ERROR:cache_util_win.cc(19)] Unable to move the cache: 0
[0120/222514.625:ERROR:cache_util.cc(140)] Unable to move cache folder profiles\Default\GPUCache to profiles\Default\old_GPUCache_000
[0120/222514.625:ERROR:disk_cache.cc(184)] Unable to create cache
[0120/222514.625:ERROR:shader_disk_cache.cc(622)] Shader Cache Creation failed: -2
So, it looks like somewhere Chromium is assuming I'm using the Default profile when in fact I specify I am not.
Does anyone have any suggestions? Thanks in advance!

I was able to your code to run with the following option added.
options.add_argument('--remote-debugging-port=45447')
Without it the code was hanging for around a minute before the following error was thrown:
WebDriverException: Message: unknown error: DevToolsActivePort file doesn't exist
Comment excerpt which pointed me to a solution:
When you send --remote-debugging-port=0 to Chrome, devtools chooses
its own port and writes it to chrome::DIR_USER_DATA as a file
named "DevToolsActivePort".
Basically once the port is set to something other than the default of 0 the DevToolsActivePort file does not need to be checked.
The full comment and bug are here: chromedriver bug
Hope this helps!

I solved this error by closing the command prompt and reopening it in administrator mode (right-click on the cmd.exe and click Run as administrator).
Nothing else worked.

Related

Where is the configuration to open a specific web browser in python?

This is a Win10 machine.
I have a file on disk called "test.html". When I run this in a terminal window C:\>test.html my default browser opens, and that is chrome.
Now I do the same in python
C:\> copy con test.py
import webbrowser
webbrowser.open('http://www.python.org')
^Z
1 file(s) copied.
C:\>python test.py
and Chrome, the default browser opens
But when I take an HTML response from an API call, and do the same webbrowser.open(...) then Internet Explorer opens. Not Edge, not Chrome, but Internet explorer.
How? It's almost as if there is something in the response that tells it "open with IE" except that if do it manually from the command line with C:\>response.html it opens chrome too.
Where is this instruction to open Internet Explorer coming from?
It should use your default browser. Otherwise, it falls back to IE
It appears to use os.startfile() to detect file associations, so running os.startfile('response.html') apparently throws a OSError for you, and you have no other browser installed in the list. Surprisingly, "chrome" nor "edge" is listed there.
Related - https://github.com/python/cpython/pull/11327
Another option you could try is to use something like webbrowser.open("file://c/response.html")
You can try webbrowser.get().
For example, opening a new tab in Google Chrome:
webbrowser.get(using='google-chrome').open_new_tab('https://google.com')
But it is not always possible to get by with .get () alone, and in this case the .register () function comes to the rescue, for example:
import webbrowser
webbrowser.register('Chrome', None, webbrowser.BackgroundBrowser('C:\Program Files (x86)\Google\Chrome\Application\chrome.exe'))
webbrowser.get('Chrome').open_new_tab('google.com')
You can also check these links:
https://docs.python.org/3/library/webbrowser.html
http://pymotw.com/2/webbrowser/
https://discourse.world/h/2019/10/10/How-to-open-a-link-in-Python.Working-with-WebBrowser-and-solving-a-problem-with-Internet-Explorer

Python Selenium Chrome Webdrives closes unexpectedly after the execution of program

I have updated the version of chrome and accordingly webdriver. The chrome closes automatically after the execution of the last line. It was a simple code to open a chrome driver and click the links. After I run in command prompt it generated the error message "failed to load pepper module from internal-not-yet-present (error 126)".
Due to which the Pycharm generates the error "Oserror:[winerror 6] The handle is invalid"
Can anyone please suggest the solution for it?
I've experienced this (quite) a while ago. In my case an Adobe Flash Player update was required for Chrome: in chrome://components find the Adobe Flash Player entry and check for updates.
Alternatively, disable extensions:
options = webdriver.ChromeOptions()
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options)

"This is the initial start page for the WebDriver server" - Unable to resolve in Selenium/Python program using IE 11

After running my Selenium/Python program, browser opened with below message:
This is the initial start page for the WebDriver server
I have done below steps to resolve this:
In IE Options -> Security tab, Enable Protected Mode check box is ticked OFF in all zones: Internet,
Local Intranet, Trusted sites and Restricted sites. Also, in Advanced tab -> Security, ticked OFF the
check box: "Enable Enhanced Protected Mode" (Also, I tried with enabling this Protected Mode in all
zones and in Advanced tab too).
My IEdriver (version 3.1.4) and Selenium web driver (version 3.1.4) are compatible (both are on same
version)
I tried above two, still I am getting the same message.
I have added below content to ignore Protected mode:
caps = DesiredCapabilities.INTERNETEXPLORER
caps['ignoreProtectedModeSettings'] = True
driver = webdriver.Ie(executable_path='C:/Selenium/Drivers/IEDriverServer.exe',capabilities=caps)
Still, I am getting the same message after adding above code.
Any ideas? Please help.
This is as per design. When IEDriverServer.exe opens a new a new Browsing Context i.e. Internet Explorer browsing session it navigates first to this page.
Browser Snapshot
Once you initialize the browser through the line:
driver = webdriver.Ie(executable_path='C:/Selenium/Drivers/IEDriverServer.exe',capabilities=caps)
next you can invoke the get() command to access any url. As an example:
driver.get('https://www.google.com/')
Additional Consideration
Additionally you need to:
Upgrade Selenium to current levels Version 3.141.59.
Upgrade IEDriverServer to latest IEDriverServer v3.150.1 level.
Note: As per best practices as Selenium Client and InternetExplorerDriver are released in sync and you must try to use both the binaries from the same major release.
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
Execute your #Test.
Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.

Browsing context has been discarded using GeckoDriver Firefox through Selenium

I didn't make any changes to my python selenium program and it worked fine 3 days ago. Now when i try to use it i get:
Browsing context has been discarded
Failed to decode response from marionette
Any idea what could have caused this outside the code? (since no changes were made)
I'm using firefox and geckodriver. After i got these errors i updated firefox, geckodriver, and selenium, but it didn't help.
This error message...
Browsing context has been discarded
.
Failed to decode response from marionette
...implies that the communication between GeckoDriver and Marionette was broken.
Some more information regarding the binary version interms of:
Selenium Server/Client
GeckoDriver
Firefox
Additionally, your code block and the error stack trace would have given us some clues about whats wrong happening. However this issue can happen due to multiple factors as follows:
As per Hang when navigation request removes the current browsing context if you have used driver.navigate().back(); when Selenium's focus was within an <iframe> this error is observed.
As per Crash during command execution results in "Internal Server Error: Failed to decode response from marionette" this issue can also occur due to ctypes checks for NULL pointer derefs.
You can find the Selenium testcase here. Perhaps instead of panicking, it would have been better to handle this more gracefully by clearing any state and returning geckodriver to accept new connections again.
As per Failed to decode response from marionette - Error to open Webdriver using python this issue can also occur if you are not using the complient version of the binaries.
GeckoDriver, Selenium and Firefox Browser compatibility chart
Reference
You can find a relevant detailed discussion in:
“Failed to decode response from marionette” message in Python/Firefox headless scraping script
I experienced the same error on a particular site, after performing a successful login and when I was redirected to the next page.
While Inspecting The Source of the new page code in my Firefox browser, I I noticed some bad format/HTML quality details that went away after a manual refresh. (I suspect related to lack of quality of that site in particular).
What I did in order to remediate this was to start every next step on a new page with a refresh on my drive:
def my_next_step(driver):
driver.refresh()
time.sleep(10)
element=driver.switch_to_frame('iframe')
.......
This helped me overcome the site quality issues.
On Ubuntu 22.10 using (apt, not snap) Firefox and Selenium in Python I also got this error after:
driver.switch_to.alert.accept()
The solution for me was to switch back to the context with:
def upload_file(driver, filepath):
"""Uploads a Ublock Origin backup (template) .txt file into the Ublock
Origin extension."""
driver.find_element("id", "restoreFilePicker").send_keys(filepath)
time.sleep(1)
driver.switch_to.alert.accept()
time.sleep(1)
# Switch back to first tab (and reload/restore it).
new_window = driver.window_handles[0]
driver.switch_to.window(new_window)
This answer was given in this question.
I removed size of window it is working without this error

How can I get rid of the message "This connection is not secure..."?

I have an automated test which fills the User + Password fields and click a certain button to Login.
During the development session I managed to run this automation tens of times without any problem. Suddenly today I found out that the response to the automated test has changed and now I can't Login. I can say that the WEB under test didn't change. I can almost tell for sure that the FireFox that is run by the automated test has updated without my control (the browser that the automated test runs has an up-to-date version 54 while the browser that I run has a version 52).
I have tried to configure the Version 54 so that he won't pop-up the message (through the about:config) but my settings are not saved.
First of all I would like to know how can I manage to get rid of the pop-up message ?
Second thing I would like to know how can I prevent the updates of the browser version ?
Could it be that the geckodriver has its own FireFox settings and Version ?
Firstly, you can configure FirefoxProfile to accept untrusted connections, as shown below:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True
driver = webdriver.Firefox(firefox_profile=profile)
This answer contains configuration details about other browsers as well.
Secondly, in order to disable automatic updates for Firefox browser, you can follow following steps:
Launch Firefox and go to 'Tools->Options->Advanced'.
Click on 'Updates' tab.
Click on 'Never Check for updates' option button under 'Firefox updates' section.
Restart Firefox.
Let me know, if it resolves your issue.
You need to set acceptInsecureCerts to true in your capabilities.
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability(FirefoxDriver.MARIONETTE, true);
capabilities.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true);
WebDriver webDriver = new FirefoxDriver(capabilities);
Edit: Sorry about the java code. The thing is you need set that capability.

Categories

Resources