I have been using Selenium and python to web scrape for a couple of weeks now. It has been working fairly good. Been running on a macOS and windows 7. However all the sudden the headless web driver has stopped working. I have been using chromedriver with the following settings:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
chrome_options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(chrome_options=options)
driver.get('url')
Initially I had to add the window, gpu and sandbox arguments to get it work and it did work up until now. However, when running the script now it gets stuck at driver.get('url'). It doesn't produce an error or anything just seems to run indefinitely. When I run without headless and simply run:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('url')
it works exactly as intended. This problem is also isolated to my windows machine. Where do I start?
Solved
For some reason the proxy setting was slowing it down. Therefore it got solved by adding:
options.add_argument(f'--proxy-server={None}')
I had exactly the same problem. It appeared randomly after the script has run fine for weeks. OP has led me to the right direction, but his solution doesnt worked for me. I had to add:
chrome_options.add_argument("--no-proxy-server")
chrome_options.add_argument("--proxy-server='direct://'");
chrome_options.add_argument("--proxy-bypass-list=*");
My complete code:
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--start-fullscreen")
chrome_options.add_argument("--no-proxy-server")
chrome_options.add_argument("--proxy-server='direct://'");
chrome_options.add_argument("--proxy-bypass-list=*");
chrome_options.binary_location = "C:\Program Files (x86)\Google\Chrome Dev\\Application\chrome.exe"
browser = webdriver.Chrome(options=chrome_options)
browser.set_window_size(2000, 1080)
please see also:
Headless chrome driver too slow
and:
Chrome webdriver produces timeout in selenium
I have a Selenium test suite that runs many tests and on each new test it opens a browser window on top of any other windows I have open. Very jarring while working in a local environment. Is there a way to tell Selenium or the OS (Mac) to open the windows in the background?
If you are using Selenium web driver with Python, you can use PyVirtualDisplay, a Python wrapper for Xvfb and Xephyr.
PyVirtualDisplay needs Xvfb as a dependency. On Ubuntu, first install Xvfb:
sudo apt-get install xvfb
Then install PyVirtualDisplay from PyPI:
pip install pyvirtualdisplay
Sample Selenium script in Python in a headless mode with PyVirtualDisplay:
#!/usr/bin/env python
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# Now Firefox will run in a virtual display.
# You will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()
EDIT
The initial answer was posted in 2014 and now we are at the cusp of 2018. Like everything else, browsers have also advanced. Chrome has a completely headless version now which eliminates the need to use any third-party libraries to hide the UI window. Sample code is as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
CHROME_PATH = '/usr/bin/google-chrome'
CHROMEDRIVER_PATH = '/usr/bin/chromedriver'
WINDOW_SIZE = "1920,1080"
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=%s" % WINDOW_SIZE)
chrome_options.binary_location = CHROME_PATH
driver = webdriver.Chrome(executable_path=CHROMEDRIVER_PATH,
chrome_options=chrome_options
)
driver.get("https://www.google.com")
driver.get_screenshot_as_file("capture.png")
driver.close()
There are a few ways, but it isn't a simple "set a configuration value". Unless you invest in a headless browser, which doesn't suit everyone's requirements, it is a little bit of a hack:
How to hide Firefox window (Selenium WebDriver)?
and
Is it possible to hide the browser in Selenium RC?
You can 'supposedly', pass in some parameters into Chrome, specifically: --no-startup-window
Note that for some browsers, especially Internet Explorer, it will hurt your tests to not have it run in focus.
You can also hack about a bit with AutoIt, to hide the window once it's opened.
Chrome 57 has an option to pass the --headless flag, which makes the window invisible.
This flag is different from the --no-startup-window as the last doesn't launch a window. It is used for hosting background apps, as this page says.
Java code to pass the flag to Selenium webdriver (ChromeDriver):
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
ChromeDriver chromeDriver = new ChromeDriver(options);
For running without any browser, you can run it in headless mode.
I show you one example in Python that is working for me right now
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("headless")
self.driver = webdriver.Chrome(executable_path='/Users/${userName}/Drivers/chromedriver', chrome_options=options)
I also add you a bit more of info about this in the official Google website https://developers.google.com/web/updates/2017/04/headless-chrome
I used this code for Firefox in Windows and got answer(reference here):
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
Options = Options()
Options.headless = True
Driver = webdriver.Firefox(options=Options, executable_path='geckodriver.exe')
Driver.get(...)
...
But I didn't test it for other browsers.
Since Chrome 57 you have the headless argument:
var options = new ChromeOptions();
options.AddArguments("headless");
using (IWebDriver driver = new ChromeDriver(options))
{
// The rest of your tests
}
The headless mode of Chrome performs 30.97% better than the UI version. The other headless driver PhantomJS delivers 34.92% better than the Chrome's headless mode.
PhantomJSDriver
using (IWebDriver driver = new PhantomJSDriver())
{
// The rest of your test
}
The headless mode of Mozilla Firefox performs 3.68% better than the UI version. This is a disappointment since the Chrome's headless mode achieves > 30% better time than the UI one. The other headless driver PhantomJS delivers 34.92% better than the Chrome's headless mode. Surprisingly for me, the Edge browser beats all of them.
var options = new FirefoxOptions();
options.AddArguments("--headless");
{
// The rest of your test
}
This is available from Firefox 57+
The headless mode of Mozilla Firefox performs 3.68% better than the UI version. This is a disappointment since the Chrome's headless mode achieves > 30% better time than the UI one. The other headless driver PhantomJS delivers 34.92% better than the Chrome's headless mode. Surprisingly for me, the Edge browser beats all of them.
Note: PhantomJS is not maintained any more!
On Windows you can use win32gui:
import win32gui
import win32con
import subprocess
class HideFox:
def __init__(self, exe='firefox.exe'):
self.exe = exe
self.get_hwnd()
def get_hwnd(self):
win_name = get_win_name(self.exe)
self.hwnd = win32gui.FindWindow(0,win_name)
def hide(self):
win32gui.ShowWindow(self.hwnd, win32con.SW_MINIMIZE)
win32gui.ShowWindow(self.hwnd, win32con.SW_HIDE)
def show(self):
win32gui.ShowWindow(self.hwnd, win32con.SW_SHOW)
win32gui.ShowWindow(self.hwnd, win32con.SW_MAXIMIZE)
def get_win_name(exe):
''' Simple function that gets the window name of the process with the given name'''
info = subprocess.STARTUPINFO()
info.dwFlags |= subprocess.STARTF_USESHOWWINDOW
raw = subprocess.check_output('tasklist /v /fo csv', startupinfo=info).split('\n')[1:-1]
for proc in raw:
try:
proc = eval('[' + proc + ']')
if proc[0] == exe:
return proc[8]
except:
pass
raise ValueError('Could not find a process with name ' + exe)
Example:
hider = HideFox('firefox.exe') # Can be anything, e.q., phantomjs.exe, notepad.exe, etc.
# To hide the window
hider.hide()
# To show again
hider.show()
However, there is one problem with this solution - using send_keys method makes the window show up. You can deal with it by using JavaScript which does not show a window:
def send_keys_without_opening_window(id_of_the_element, keys)
YourWebdriver.execute_script("document.getElementById('" + id_of_the_element + "').value = '" + keys + "';")
I suggest using PhantomJS. For more information, you may visit the Phantom Official Website.
As far as I know PhantomJS works only with Firefox...
After downloading PhantomJs.exe you need to import it to your project as you can see in the picture below PhantomJS.
I have placed mine inside: common → Library → phantomjs.exe
Now all you have to do inside your Selenium code is to change the line
browser = webdriver.Firefox()
To something like
import os
path2phantom = os.getcwd() + "\common\Library\phantomjs.exe"
browser = webdriver.PhantomJS(path2phantom)
The path to PhantomJS may be different... change as you like :)
This hack worked for me, and I'm pretty sure it will work for u too ;)
It may be in options. Here is the identical Java code.
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setHeadless(true);
WebDriver driver = new ChromeDriver(chromeOptions);
This is a simple Node.js solution that works in the new version 4.x (maybe also 3.x) of Selenium.
Chrome
const { Builder } = require('selenium-webdriver')
const chrome = require('selenium-webdriver/chrome');
let driver = await new Builder().forBrowser('chrome').setChromeOptions(new chrome.Options().headless()).build()
await driver.get('https://example.com')
Firefox
const { Builder } = require('selenium-webdriver')
const firefox = require('selenium-webdriver/firefox');
let driver = await new Builder().forBrowser('firefox').setFirefoxOptions(new firefox.Options().headless()).build()
await driver.get('https://example.com')
The whole thing just runs in the background. It is exactly what we want.
If you are using the Google Chrome driver, you can use this very simple code (it worked for me):
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome('chromedriver2_win32/chromedriver.exe', options=chrome_options)
driver.get('https://www.anywebsite.com')
On *nix, you can also run a headless X Window server like Xvfb and point the DISPLAY environment variable to it:
Fake X server for testing?
One way to achieve this is by running the browser in headless mode. Another advantage of this is that tests are executed faster.
Please find the code below to set headless mode in the Chrome browser.
package chrome;
public class HeadlessTesting {
public static void main(String[] args) throws IOException {
System.setProperty("webdriver.chrome.driver",
"ChromeDriverPath");
ChromeOptions options = new ChromeOptions();
options.addArguments("headless");
options.addArguments("window-size=1200x600");
WebDriver driver = new ChromeDriver(options);
driver.get("https://contentstack.built.io");
driver.get("https://www.google.co.in/");
System.out.println("title is: " + driver.getTitle());
File scrFile = ((TakesScreenshot) driver)
.getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(scrFile, new File("pathTOSaveFile"));
driver.quit();
}
}
If you are using Ubuntu (Gnome), one simple workaround is to install Gnome extension auto-move-window: https://extensions.gnome.org/extension/16/auto-move-windows/
Then set the browser (eg. Chrome) to another workspace (eg. Workspace 2). The browser will silently run in other workspace and not bother you anymore. You can still use Chrome in your workspace without any interruption.
Here is a .NET solution that worked for me:
Download PhantomJS at http://phantomjs.org/download.html.
Copy the .exe file from the bin folder in the download folder and paste it to the bin debug/release folder of your Visual Studio project.
Add this using
using OpenQA.Selenium.PhantomJS;
In your code, open the driver like this:
PhantomJSDriver driver = new PhantomJSDriver();
using (driver)
{
driver.Navigate().GoToUrl("http://testing-ground.scraping.pro/login");
// Your code here
}
I had the same problem with my chromedriver using Python and options.add_argument("headless") did not work for me, but then I realized how to fix it so I bring it in the code below:
opt = webdriver.ChromeOptions()
opt.arguments.append("headless")
Just add a simple "headless" option argument.
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome("PATH_TO_DRIVER", options=options)
Use it ...
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)
I'm using Chrome with Selenium in Python 2.7.
I've tried to run Chrome in headless mode but it slows down my tests significantly.
One workaround shall be to disable the proxy settings but I don't know how to do it in python.
This is my code so far:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument('headless')
chrome_options.add_argument('--hide-scrollbars')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('???') # which option?
self.driver = webdriver.Chrome("C:\Python27\Scripts\chromedriver.exe", chrome_options=chrome_options)
Does anyone know how to solve this?
Try this:
chrome_options.add_argument('--no-proxy-server')
I am using a script daily. It's a headless chrome that just checks a site every 5 minutes and suddenly devmode turned on and i can't seem to turn it off. This is my script:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome + 'E:\Chrome Downloads\chromedriver_win32\chromedriver.exe', chrome_options=options)
And the output is:
[0926/111600.894:ERROR:devtools_http_handler.cc(786)]
DevTools listening on 127.0.0.1:12107
[0926/111601.685:ERROR:browser_gpu_channel_host_factory.cc(103)] Failed to launch GPU process.
It also spews out the F12 developer console info everytime it connects to a new site. :c
I managed to fix it finally :D
options.add_argument('--log-level=3')
That was all it took.
I was running into the same issue and adding the log-level argument only did not work for me.
On Windows you obviously need to add the following option to your ChromeOptions as well:
options.add_experimental_option('excludeSwitches', ['enable-logging'])
As described here: https://bugs.chromium.org/p/chromedriver/issues/detail?id=2907#c3
I've developed a couple of Python scripts using Selenium and, at first, PhantomJS. While heading toward automated downloads, I switched for (headed) Firefox (which worked) and then Chrome with the headless option so I won't have the browser opening in front of me.
My first script, which accesses a page and a couple of HTML elements, works perfectly with headless Chrome.
The second one, however, works only with headed Chrome. If I add the "headless" option, it doesn't work anymore. When I try to print the HTML in headless mode to see why it cannot find the HTML element I'm looking for, all I have is :
<html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>
With the headed Chrome, I have a complete HTML printed.
This is how I start my headless Chrome :
options = webdriver.ChromeOptions()
options.add_argument("--ignore-certificate-errors")
options.add_argument("headless")
driver = webdriver.Chrome(chrome_options=options)
Again, note that this works in another of my script. The only difference here is that I need to log in to access the page, but even then, why would it work with the head? My script is made to log in automatically anyway by filling the form.
Python : 3.6.1, Chrome : 60.0.3112.78 (64 bits), Selenium : 3.4.3
Any idea?
Thanks.
** EDIT: Here is the beginning of the code**
url = 'https://10.11.227.21/tmui/'
driver.get(url + "login.jsp")
html_source = driver.page_source
print(html_source)
blocStatus = WebDriverWait(driver, TIMEOUT).until(EC.presence_of_element_located((By.ID, "username")))
inputElement = driver.find_element_by_id("username")
inputElement.send_keys('actualLogin')
inputElement = driver.find_element_by_id("passwd")
inputElement.send_keys('actualPassword')
inputElement.submit()
I had a same experience like you, and solved it by using xvfb and pyvirtualdisplay.
I use chromedrive=v2.3.1, chrome-browser=v60 and Selenium=3.4.3
In Headless chrome, some of script seems not to work as expected.
Please refer to vpassapera's comment in https://gist.github.com/addyosmani/5336747.
How about try it like below,
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
# Do Not use headless chrome option
# options.add_argument('headless')
url = 'https://10.11.227.21/tmui/'
driver.get(url + "login.jsp")
html_source = driver.page_source
print(html_source)
blocStatus = WebDriverWait(driver, TIMEOUT).until(EC.presence_of_element_located((By.ID, "username")))
inputElement = driver.find_element_by_id("username")
inputElement.send_keys('actualLogin')
inputElement = driver.find_element_by_id("passwd")
inputElement.send_keys('actualPassword')
inputElement.submit()
display.stop()
xvfb is required to use "pyvortualdisplay"
$ sudo apt-get install -y xvfb
Headless Chrome does not support insecure certificates and hence, websites with insecure certificates does not open living it blank. You need to add capabilities as follow:
from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
capabilities = DesiredCapabilities.CHROME.copy()
capabilities['acceptSslCerts'] = True
capabilities['acceptInsecureCerts'] = True
driver = webdriver.Chrome(chrome_options = chrome_options,executable_path='your path',desired_capabilities=capabilities)
driver.get("yourWebsite")
This will do the work.
I had the same issue and setting the window size in conftest.py solved it.
My code snippet:
#pytest.fixture
def chrome_options(chrome_options, pytestconfig):
if pytestconfig.getoption('headless'):
chrome_options.add_argument('--headless')
chrome_options.add_argument("window-size=1920,1080")
else:
chrome_options.add_argument("start-maximized");
return chrome_options
Headless chrome may be faster on same machine than headed, try adding some wait before locating password element.
For some if you remove the below, it would work.
driver.fullscreen_window()
refer https://github.com/SeleniumHQ/selenium/issues/4477
add below code
self.chrome_options = webdriver.ChromeOptions()
self.chrome_options.add_argument("--window-size=1920,1080")
self.chrome_options.add_argument("--disable-extensions")
self.chrome_options.add_argument("--proxy-server='direct://'")
self.chrome_options.add_argument("--proxy-bypass-list=*")
self.chrome_options.add_argument("--start-maximized")
self.chrome_options.add_argument('--headless')
self.chrome_options.add_argument('--disable-gpu')
self.chrome_options.add_argument('--disable-dev-shm-usage')
self.chrome_options.add_argument('--no-sandbox')
self.chrome_options.add_argument('--ignore-certificate-errors')
self.browser = webdriver.Chrome(options=self.chrome_options)
For my situation, headless did not work because I was behind a proxy. Apparently, Chrome is able to use the system proxy, but headless does not use the system proxy.
I simply needed to provide the proxy, then headless (as well as Chrome) worked.
options.add_argument('--proxy-server=http://myproxy:port')