I have been using Selenium and python to web scrape for a couple of weeks now. It has been working fairly good. Been running on a macOS and windows 7. However all the sudden the headless web driver has stopped working. I have been using chromedriver with the following settings:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
chrome_options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(chrome_options=options)
driver.get('url')
Initially I had to add the window, gpu and sandbox arguments to get it work and it did work up until now. However, when running the script now it gets stuck at driver.get('url'). It doesn't produce an error or anything just seems to run indefinitely. When I run without headless and simply run:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('url')
it works exactly as intended. This problem is also isolated to my windows machine. Where do I start?
Solved
For some reason the proxy setting was slowing it down. Therefore it got solved by adding:
options.add_argument(f'--proxy-server={None}')
I had exactly the same problem. It appeared randomly after the script has run fine for weeks. OP has led me to the right direction, but his solution doesnt worked for me. I had to add:
chrome_options.add_argument("--no-proxy-server")
chrome_options.add_argument("--proxy-server='direct://'");
chrome_options.add_argument("--proxy-bypass-list=*");
My complete code:
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--start-fullscreen")
chrome_options.add_argument("--no-proxy-server")
chrome_options.add_argument("--proxy-server='direct://'");
chrome_options.add_argument("--proxy-bypass-list=*");
chrome_options.binary_location = "C:\Program Files (x86)\Google\Chrome Dev\\Application\chrome.exe"
browser = webdriver.Chrome(options=chrome_options)
browser.set_window_size(2000, 1080)
please see also:
Headless chrome driver too slow
and:
Chrome webdriver produces timeout in selenium
Related
Running latest version of all features
(chrome version 66, selenium version 3.12, chromedriver version 2.39, python version 3.6.5)
I have tried all of the solutions that I have found online but nothing seems to be working. I have automated something using Selenium for Chrome and it does exactly what I need it to do.
The last thing I need to to is hide the browser because I do not need to see the UI. I have attempted to make the browser headless using the following code but when I do the program crashes.
I have also tried to alter the window size to 0x0 and even tried the command: options.set_headless(headless=True) instead but nothing seems to work.
NOTE: I am running on Windows and have also tried with the command:
options.add_argument('--disable-gpu')
Try to move browser over the monitor
driver = webdriver.Chrome()
driver.set_window_position(-2000,0) # if -20000 don't help put -10000
What is happening when you run, does it produce any error or just run as if you hadn't even added the headless argument?
Here's what I do for mine to run headlessly -
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
selenium_chrome_driver_path = "blah blah my path is here"
my_driver = webdriver.Chrome(chrome_options = options, executable_path = selenium_chrome_driver_path)
I'm running this code with python 3.6 with up to date selenium drivers on Windows 8 (unfortunately)
Got the solutions after combining a few sources. It seems that the libraries get often updated and eventually it becomes deprecated. Atm this one worked for me in Windows, using Python 3.4:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.headless = True
driver = webdriver.Chrome(executable_path="C:/Users/Admin/Documents/chromedriver_win32/chromedriver", options=options)
Try to use user agent in chrome_options
ua = UserAgent()
userAgent = ua.random
print(userAgent)
chrome_options.add_argument('user-agent={userAgent}')
I'm using Chrome with Selenium in Python 2.7.
I've tried to run Chrome in headless mode but it slows down my tests significantly.
One workaround shall be to disable the proxy settings but I don't know how to do it in python.
This is my code so far:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument('headless')
chrome_options.add_argument('--hide-scrollbars')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('???') # which option?
self.driver = webdriver.Chrome("C:\Python27\Scripts\chromedriver.exe", chrome_options=chrome_options)
Does anyone know how to solve this?
Try this:
chrome_options.add_argument('--no-proxy-server')
I am using a script daily. It's a headless chrome that just checks a site every 5 minutes and suddenly devmode turned on and i can't seem to turn it off. This is my script:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome + 'E:\Chrome Downloads\chromedriver_win32\chromedriver.exe', chrome_options=options)
And the output is:
[0926/111600.894:ERROR:devtools_http_handler.cc(786)]
DevTools listening on 127.0.0.1:12107
[0926/111601.685:ERROR:browser_gpu_channel_host_factory.cc(103)] Failed to launch GPU process.
It also spews out the F12 developer console info everytime it connects to a new site. :c
I managed to fix it finally :D
options.add_argument('--log-level=3')
That was all it took.
I was running into the same issue and adding the log-level argument only did not work for me.
On Windows you obviously need to add the following option to your ChromeOptions as well:
options.add_experimental_option('excludeSwitches', ['enable-logging'])
As described here: https://bugs.chromium.org/p/chromedriver/issues/detail?id=2907#c3
this is my code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("https://google.com/")
Yet the browser opens instead of going headless.
Some other arguments do seem to work like --start-maximized
This is redundant but, since I've been asked to do it, https://developers.google.com/web/updates/2017/04/headless-chrome says that headless won't be available on Chrome for Windows until version 60.
Headless won't be available on Windows untill Chrome 60.
I am currently trying to code a basic smartmirror for my coding II class in high school with python. One thing I'm trying to do is open new tabs in full screen (using chrome). I currently have it so I can open url's, but I am not getting them in full screen. Any ideas on code I can use to open chrome in full screen?
If you're using selenium, just code like below:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://google.com')
driver.maximize_window()
As suggested, selenium is a good way to accomplish your task.
In order to have it full-screen and not only maximized I would use:
chrome_options.add_argument("--start-fullscreen");
or
chrome_options.add_argument("--kiosk");
First option emulates the F11 pressure and you can exit pressing F11. The second one turns your chrome in "kiosk" mode and you can exit pressing ALT+F4.
Other interesting flags are:
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
Those will remove the top bar exposed by the chrome driver saying it is a dev chrome version.
The complete script is:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
# chrome_options.add_argument("--start-fullscreen");
chrome_options.add_argument("--kiosk");
driver = webdriver.Chrome(executable_path=rel("path/to/chromedriver"),
chrome_options=chrome_options)
driver.get('https://www.google.com')
"path/to/chromedriver" should point to the chrome driver compatible with your chrome version downloaded from here