Develop Reference

Python selenium headless start with extensions and xfvb

I'm trying to run a headless Chrome process with an extension1. I'm using WSL.
For testing purposes I can use one of two browsers:
Ubuntu: Google Chrome 86.0.4240.183
Windows: Version 86.0.4240.183 (Official Build) (64-bit)
For production purposes the Ubuntu's Chrome is the only viable option2.
The rest of the stack:
Python 3.6.9
selenium: 3.141.0
In order to overcome Chrome not being able to start headlessly with extensions I use PyVirtualDisplay.
from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600), backend='xvfb')
display.start()
options = ChromeOptions()
options.add_extension('/home/plonca/selenium_draft/extensions.crx')
options.add_argument('--no-sandbox')
windows_chrome_driver = '/home/plonca/python_virtual_environments/selenium_jupyter_venv/bin/chromedriver.exe'
ubuntu_chrome_driver = '/home/plonca/python_virtual_environments/selenium_jupyter_venv/bin/chromedriver'
chrome_driver = webdriver.Chrome(executable_path=ubuntu_chrome_driver
, options=options
)
The problem occurs in the last line: if the windows_chrome_driver is set as executable_path everything is fine. If I set ubuntu_chrome_driver instead, I get an error:
WebDriverException: Message: unknown error: failed to wait for extension background page to load: chrome-extension://bmlddehkgnjdalnfaecgflmmeknlbohi/_generated_background_page.html
from tab crashed
I wonder what the cause of the error might be. How to make chrome_driver work in the headless environment?
1 The extension is needed so that I can authenticate via a pop-up window. The extension has been created according to instructions found here. Other options such as sending username and password in the URL or automating the click with AutoIt doesn't work.
2 Please note that the Chrome seems to be the only option since using Firefox is a matter of an unresolved issue.

Can't open web browser using selenium chromedriver on windows 10

I'm trying to open a web browser using Selenium Chromedriver on Windows 10 with python in jupyter notebooks through an ubuntu command prompt. I've read stack overflow posts and tried to solve based on their answers, but I'm stuck in a loop where I keep receiving the same 3 errors.
Here is what I have installed:
OS - Windows 10, 1709, 64-bit Selenium - 3.8.1 Chromedriver - 2.45
Chrome - Version 71.0.3578.98 Python - 3.5.2
I tried various websites. The goal is to eventually get to a social media login page, but i'm stuck at opening a new blank web browser.
Here is my starting code:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.imdb.com/")
WebDriverException: Message: 'chromedriver' executable may have wrong
permissions.
Then I tried the following:
from selenium import webdriver
chromedriver = "C:/Users/xxxx/AppData/Local/lxss/home/xxxx/chromedriver.exe"
browser = webdriver.Chrome(chromedriver)
browser.get('https://www.imdb.com/')
WebDriverException: Message: 'chromedriver.exe' executable needs to be
in PATH.
Here are the steps I have taken:
I added a PATH under environment variables to the folder - (C:\Users\xxxxx\AppData\Local\lxss\home\xxxx),
I tried using \, and /, and even \
Once I added the PATH, I tried the following two codes (and various versions) and received the same error:
A.
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx)
driver.get("https://www.imdb.com/")
B.
from selenium import webdriver
chromedriver = r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx\chromedriver.exe'
driver = webdriver.Chrome(chromedriver)
driver.get("https://www.imdb.com/")
WebDriverException: Message:
'C:\Users\xxxx\AppData\Local\lxss\home\xxxx' executable may have wrong
permissions.
Then I did the following:
- Went to file Properties, under General, took off Read-only (Windows permissions)
- Went to file Properties, under Security and changed permissions to Full Control
- In the C:\Users\xxxxx\AppData\Local\lxss\home\xxxx file, I changed the permissions
using chmod 777 -R in my command prompt. Then I tried the following code:
from selenium import webdriver
import os
chromedriver = r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx\chromedriver.exe'
driver = webdriver.Chrome(os.path.join(os.getcwd(), 'chromedriver.exe'))
driver.get("https://www.imdb.com/")
WebDriverException: Message: Service /home/ariggs/chromedriver.exe
unexpectedly exited. Status code was: 1
I am stuck between these three error messages. Does anyone have another suggestion for a beginner?

You can actually start Windows executables from a linux subsystem as it is described here https://learn.microsoft.com/en-us/windows/wsl/interop.
But you have to keep in mind that Selenium and ChromeDriver communicate over a network connection. Actually chromedriver starts its own http server and Selenium sends requests and receives responses over http. (see https://sqa.stackexchange.com/questions/28358/how-does-chromedriver-exe-work-on-a-core-and-fundamental-level)
According to Microsoft, WSL and Windows share the same IP address and network connections via localhost are supported. But in your case there seems to be a problem during the startup.
You can start a remote webdriver on windows with Python and connect to that.
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import subprocess
subprocess.run(["d:\\develop\\remotewebdriver.cmd", ""])
driver = webdriver.Remote(
command_executor='http://localhost:4444/wd/hub',
desired_capabilities=DesiredCapabilities.CHROME)
driver.get('http://www.google.in/')
driver.close()
You need a windows script remotewebdriver.cmd for the remote webdriver that is called from Python:
SET JAVA_HOME=D:\develop\Java\jdk-11.0.2
d:
cd \develop
start D:\develop\Java\jdk-11.0.2\bin\java -Dwebdriver.chrome.driver=d:\develop\chromedriver.exe -jar selenium-server-standalone-3.141.59.jar
You have to adapt the path to your own environment. This setup works for me.

You can do this way.
Step 1:
Download chrome driver from this link (download the specific version as chrome):
http://chromedriver.chromium.org/downloads
Important: check your chrome version first. Go to help -> about Google Chrome, to see the version of your chrome.
Step 2:
After downloading extract and save the chromedriver file in a specific folder like, C:\selenium
go to environment variable and add a new path, C:\selenium
Step 3
Double click on the chromedriver application and then restart your command prompt. (If you are using conda environment.)

Selenium Grid + Jenkins + Python -> Headless RemoteWebDriver?

I'm currently working with Jenkins + Selenium Plugin for Jenkins. I have a hub and some nodes. Both Hub and Nodes are in my localhost.
I realized that, when I run my tests locally (That means, using chrome, firefox, IE webdrivers), then the browsers will appear and the tests will be executed.
driver = webdriver.Chrome() # Open Chrome Browser
On the other hand, if I run the tests through a remote webdriver, then the browsers appear to be headless.
capabilities = {"platform" : "VISTA"}
capabilities["browserName"] = "chrome"
driver = webdriver.Remote(command_executor='http://localhost:4444/wd/hub', desired_capabilities=capabilities)
# Hub and Node are running locally, windows chrome won't open (Headless?)
Are these remote browsers (selenium grid) "headless" by default?
How could I verify if they're really running in headless mode?
If they are not headless, how can I make them headless? (jenkins +
selenium grid)
I've been trying to do some research but can't find any documentation that specifies this. I just found this post:
http://grokbase.com/t/gg/selenium-users/15b64b173p/selenium-grid-browser-appears-headless
Thanks!

RemoteWebDriver is not headless by default, it is running where you want it to run, it can run on user account as a normal browser or it can run in background on LocalSystem account.
Which OS are you using? If you are running jenkins as a windows service your test will run in background, it's normal behavior because of windows service 0 policy.

Xvfb on Windows

I'm using pyvirtualdisplay to run a test with a headless Firefox browser. This is the code I'm using :
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from pyvirtualdisplay import Display
display= Display(visible=0, size=(320, 240)).start() # visible=0
display.start()
driver = webdriver.Firefox()
driver.get("https://google.com")
display.quit()
And the traceback that I obtain :
easyprocess.EasyProcessCheckInstalledError: cmd=['Xvfb','-help']

You can't use pyvirtualdisplay on Windows.
It is just a wrapper that calls Xvfb. Xvfb is a headless display server for the X Window System. Windows does not use the X Window System.

For windows users you can use the free VNC utility.. For example if you are running docker you can do it in 3 steps:
Run a docker image that has the standalone firefox server (that has port 5900 exposed for VNC)
$ docker run -d -p 4444:4444 -p 5990:5990 selenium/standalone-firefox-debug
Open VNC and connect to that host localhost:5990, password is 'secret'
Now simply execute your selenium script and you will see what's happening in VNC window live as its happening. Just make sure the script is pointing to your docker standalone server like localhost:4444/wd/hub in order for it to work

py2app selenium without firefox install

I am wondering if there is a way within py2app to include the Firefox browser, or if there is a way for Selenium to use Firefox without having to install Firefox on the host machine.
I have created an app with py2app that uses Selenium, however, I have Firefox installed on my machine, but not everyone that will receive the app will have Firefox installed. I am looking for a way to either include Firefox in the distribution or go around this.

Script will not run if Firefox is not preinstalled.
You can test your script with other browser, for example Chrome. If it works on Chrome also, then you can edit script like this:
from selenium.common.exceptions import WebDriverException
try:
driver = webdriver.Firefox()
except WebDriverException:
driver = webdriver.Chrome()
You can add same for few more browsers (IE, Opera, Safari...) to be sure that script will run on users machine