Develop Reference

Chrome Driver crashes in headless

I am making a bot to scrap me some information from the web via chromedriver. But because this information is sometimes limited to an account (like instagram insights) i need to use several UserData-Folders to save all the logins etc for the chromedriver. To save some memory I made a function "init" which initializes a chromedriver window with the desired UserData-Folder.
the function:
def init(userdata):
global driver
warnings.filterwarnings("ignore", category=DeprecationWarning)
option = Options()
option.add_argument("--disable-infobars")
option.add_argument("window-size=fullscreen")
option.add_argument("--disable-extensions")
option.add_argument("--log-level=3")
option.add_argument("--headless")
option.add_argument(r'--user-data-dir=dir\to\UserData' + str(userdata))
option.add_experimental_option("prefs", {"profile.default_content_setting_values.notifications": 1})
driver = webdriver.Chrome(chrome_options=option, executable_path='C:\Windows\chromedriver.exe')
when running the code without headless, it works like a charm. However when using headless it gives the following error:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: was killed.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location C:\Program Files\Google\Chrome\Application\chrome.exe is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

chrome_options have been deprecated for quite some time now. Instead you have to use options as follows:
driver = webdriver.Chrome(options=option, executable_path='C:\Windows\chromedriver.exe')

Errors in ChromeDriver logs using a proxy through Selenium and Python

I am trying to test how to use proxies using selenium in python
Here's the code I am trying
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
PROXY = "164.68.123.119:9300"
opts = Options()
#opts.add_argument('--proxy-server=%s' % PROXY)
driver = webdriver.Chrome(executable_path="D:/Webdrivers/chromedriver.exe", options=opts)
print(type(driver))
driver.get("https://www.google.com")
myPageTitle=driver.title
print(myPageTitle)
assert "Google" in myPageTitle
time.sleep(10)
print(driver.quit)
driver.quit()
When running the code without relying on the proxy, I got error logs like that
DevTools listening on ws://127.0.0.1:1056/devtools/browser/3bd3f85f-6d2c-4273-87be-a2e108c07626
<class 'selenium.webdriver.chrome.webdriver.WebDriver'>
Google
[10116:7976:1029/180608.508:ERROR:chrome_browser_main_extra_parts_metrics.cc(230)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that this ends.
[10116:7976:1029/180608.509:ERROR:chrome_browser_main_extra_parts_metrics.cc(233)] crbug.com/1216328: Checking Bluetooth availability ended.
[10116:10692:1029/180608.527:ERROR:device_event_log_impl.cc(214)] [18:06:08.528] Bluetooth: bluetooth_adapter_winrt.cc:1073 Getting Default Adapter failed.
[10116:7976:1029/180608.527:ERROR:chrome_browser_main_extra_parts_metrics.cc(236)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that this ends.
[10116:7976:1029/180608.561:ERROR:chrome_browser_main_extra_parts_metrics.cc(240)] crbug.com/1216328: Checking default browser status ended.
[10116:10692:1029/180608.589:ERROR:usb_descriptors.cc(114)] Failed to parse configuration descriptor.
[10116:10692:1029/180608.725:ERROR:usb_descriptors.cc(100)] Failed to read all configuration descriptors. Expected 1, got 0.
[10116:10692:1029/180608.725:ERROR:device_event_log_impl.cc(214)] [18:06:08.726] USB: usb_device_win.cc:93 Failed to read descriptors from \\?\usb#vid_0000&pid_3825#5&50adea6&0&1#{a5dcbf10-6530-11d2-901f-00c04fb951ed}.
<bound method WebDriver.quit of <selenium.webdriver.chrome.webdriver.WebDriver (session="9dde593ef8cb9492d05b49ea42fc0d9f")>>
Any idea how to fix such errors?

To initiate Chrome browser using a proxy you can try the following solution:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
PROXY = "164.68.123.119:9300"
options = Options()
options.add_argument('--proxy-server={}'.format(PROXY))
options.add_argument("start-maximized")
driver = webdriver.Chrome(executable_path=r'C:\WebDriver\ChromeDriver\chromedriver.exe', options=options)
print(type(driver))
driver.get("https://www.google.com")
myPageTitle=driver.title
print(myPageTitle)
assert "Google" in myPageTitle
time.sleep(10)
driver.quit()
PS: chrome_browser_main_extra_parts_metrics.cc, device_event_log_impl.cc, usb_descriptors.cc, etc are the result of a generic bug due to Chrome/ChromeDriver compatibility which you can ignore as of now. For details check Parametrized tests are flaky due to a timeout in recording expensive metrics on startup.
Reference
You can find a relevant discussions in:
How to rotate Selenium webrowser IP address

When trying to use a custom profile in Chrome with selenium webdriver in Python I keep getting this error

WebDriverException: Message: Can not connect to the Service /usr/lib/chromium/chromium
One of the only places which seemed to show you how, but that might only work on Windows. That's where I got the code from. How to load default profile in chrome using Python Selenium Webdriver?
My code.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=/home/me/.config/chromium/Default") #Path to your chrome profile
w = webdriver.Chrome(executable_path="/usr/lib/chromium/chromium", chrome_options=options)
w.get("https://google.com")
The browser opens up, pauses, doesn't go to the URL, then gives me that error message.
This code stops giving me the error message, but my user data does not show up.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
config = "_config/"
chromedriver = config+"chromedriver"
options.add_argument("--profile-directory='Default'") #Path to your chrome profile
w = webdriver.Chrome(chromedriver, chrome_options=options)
w.get("https://google.com")
And if I used this executable path instead, with everything else being the same, it opens up the browser with all of the desired user data, but then gives this error. WebDriverException: Message: Service chromium unexpectedly exited. Status code was: 0 The browser stays open:
w = webdriver.Chrome(executable_path="chromium", chrome_options=options)

Selenium connection

I'm trying to get a webpage with Selenium with this code :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
IEdriver= 'C:\Program Files\Internet Explorer\iexplore.exe'
browser = webdriver.Ie(IEdriver)
browser.get('www.google.com')
When IE is open, it tries to connect to :
http://--port=60803/
And I can't connect to Google. Does anyone know why ?
EDIT:
The exception is :
WebDriverException("Can not connect to the Se
selenium.common.exceptions.WebDriverException: Message:
ervice C:\Program Files\Internet Explorer\iexplore.exe

You should add scheme (application layer protocol you want to use) to URL, so replace
browser.get('www.google.com')
with
browser.get('https://www.google.com')
Also there is another problem in your code:
IEdriver= 'C:\Program Files\Internet Explorer\iexplore.exe' points on IE browser binary file while webdriver.Ie() should get path to IEDriverServer.exe as value for executable_path parameter

Can not connect to Ghostdriver

I wish to extract some information from a dynamic website.
from selenium import webdriver
import time
driver = webdriver.PhantomJS(executable_path='/Users/xxx/anaconda/phantomjs-2.0.0-macosx/bin/phantomjs')
driver.get("http://pythonscraping.com/pages/javascript/ajaxDemo.html")
time.sleep(3)
print(driver.find_element_by_id("content").text)
driver.close()
When I run the above code, I get the following error message:
"selenium.common.exceptions.WebDriverException: Message:
Can not connect to GhostDriver on port 61121"
Any ideas why?