Develop Reference

Selenium Webdriver error in Google Colab on Mac Google Chrome

I'm following a scraping tutorial from GeeksForGeeks below:
https://colab.research.google.com/drive/14kXYQCSGVye4bBKKExtuHOeq32bc4xbW?usp=sharing#scrollTo=UjB8J1L8s6kT
I'm working on a Macbook Pro in Google Colab via Chrome.
When I get to the 4th command block where the path and driver are defined I get an error saying
<ipython-input-33-bb59bfe38590>:2: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome(PATH)
And
During handling of the above exception, another exception occurred:
WebDriverException Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/selenium/webdriver/common/service.py in _start_process(self, path)
211 except OSError as err:
212 if err.errno == errno.ENOENT:
--> 213 raise WebDriverException(
214 f"'{os.path.basename(self.path)}' executable needs to be in PATH. {self.start_error_message}"
215 )
WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://chromedriver.chromium.org/home
What does this mean? Do I need to link to webdriver on my local mac drive? Currently the command in the tutorial is:
PATH = "/Users/Edu/Desktop/VENV/chromedriver"
driver = webdriver.Chrome(PATH)
driver.get(url)
Do I need to replace this with the path to chromedriver in my files? I tried replacing the path above with the below but it doesn't change anything:
/Users/KD/Downloads/chromedriver
Tried the command below from the tutorial linked here: https://colab.research.google.com/drive/14kXYQCSGVye4bBKKExtuHOeq32bc4xbW?usp=sharing#scrollTo=UjB8J1L8s6kT
PATH = "/Users/Edu/Desktop/VENV/chromedriver"
driver = webdriver.Chrome(PATH)
driver.get(url)

You need to use Service.
The following works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = 'your_link'
driver.get(url)

SessionNotCreatedException: Expected browser binary location, but unable to find binary in default location accessing GeckoDriver from mount location

I am trying to use Selenium in Google Colab, but I get some errors when I try to run a Firefox instance. i followed this links:
Selenium Documentation here, I tried with the Driver Management Software but I got the error that says that was unable to find the binary location to Firefox. So I tried with the Hard Coded Location method, but I still have the same error, so I think that I must install firefox in google colab, but I don't know if this is a correct approach
Here is my code:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
service = FirefoxService(executable_path= '/content/drive/MyDrive/geckodriver')
driver = webdriver.Firefox(service=service)
And this is the entire error:
SessionNotCreatedException Traceback (most recent call last)
<ipython-input-7-51ea1584f592> in <module>()
1 service = FirefoxService(executable_path= '/content/drive/MyDrive/geckodriver')
----> 2 driver = webdriver.Firefox(service=service)
4 frames
/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
245 alert_text = value['alert'].get('text')
246 raise exception_class(message, screen, stacktrace, alert_text) # type: ignore[call-arg] # mypy is not smart enough here
--> 247 raise exception_class(message, screen, stacktrace)
248
249 def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT:
SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line
Edit
i tried installing firefox on Google colab, and i think all is correct, but when it tried to run firefox i get this output on geckodriver.log:
1643070515477 geckodriver INFO Listening on 127.0.0.1:43355
1643070515979 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "--headless" "--no-sandbox" "--disable-dev-shm-usage" "--remote-debugging-port" "43945" "-no-remote" "-profile" "/tmp/rust_mozprofileeDGKcf"
src/tcmalloc.cc:283] Attempt to free invalid pointer 0x7fe67e416250
Redirecting call to abort() to mozalloc_abort
Here is the code:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
service = FirefoxService(executable_path= '/content/drive/MyDrive/geckodriver')
options = webdriver.FirefoxOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Firefox(service= service, options= options)

Seems you are downloading and storing the GeckoDriver in a network location and then mounting it:
drive.mount('/content/drive', force_remount=True)
As per several discussions, GeckoDriver / FirefoxService can't be initiated from a network location.
You can find a couple of relevant discussions in:
CreatePlatformSocket() returned an error: An invalid argument was supplied. (0x2726) when trying to access chromedriver through network path
CreatePlatformSocket() failed: Address family not supported by protocol and cannot find chrome binary using WebDriverManager in AWS Lambda
Solution
So a generalized solution would be to access GeckoDriver / Firefox by putting them into the local drives of the host machine.

Selenium webdriver.get() method is not opening specified URL [duplicate]

This question already has answers here:
org.openqa.selenium.WebDriverException: chrome not reachable - when attempting to start a new session
(1 answer)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable error while using find_element_by_id Selenium with ChromeDriver
(3 answers)
Python selenium WebDriverException: chrome not reachable while opening ChromeDriver
(1 answer)
WebDriverException: Message: chrome not reachable after long time
(2 answers)
Closed 2 years ago.
I am trying to use chromedriver.exe to do some web scraping / automation -- my code is below
from selenium import webdriver
browser = webdriver.Chrome('MY_PATH')
browser.get("https://www.google.com")
browser.close()
Once I execute the "browser" variable then a blank google browser opens, but when I try to execute the .get() function this is the error I receive:
WebDriverException Traceback (most recent call last)
<ipython-input-36-0ba0932650bf> in <module>
----> 1 browser.get("https://www.google.com")
2 browser.close()
~\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in get(self, url)
331 Loads a web page in the current browser session.
332 """
--> 333 self.execute(Command.GET, {'url': url})
334
335 #property
~\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
~\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
WebDriverException: Message: chrome not reachable
(Session info: chrome=85.0.4183.102)
Any thoughts / feedback / suggestions would be greatly appreciated!

ChromeDriver and Chrome browser aren't compatible, maybe you might have the latest Chrome browser but you don't have the latest chromeDriver - here you can download it - it's ChromeDriver 86.0.4240.22

first of all, make sure of the followings:
make sure chromedriver exist in your 'PATH' environment variables
check out your chromedriver version matches the chrome version you are using
-you can check your chrome version from help--> about Google Chrome
then download compatible Chromedriver version from here

This is caused due to the wrong chrome driver to the certain chrome version you have.
You can find the chromedriver here Chrome Driver
Or you could let webdriver manager handle all your issues by doing
pip install webdriver-manager in command prompt
Than simply using.
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())

Try this
Remember to pass the path to the executable_path keyword argument
Have matching version of Chrome and Chrome Driver
from selenium import webdriver
chrome_driver_path = '/home/aahnik/Downloads/apps/chromedriver'
# path to where chrome driver executable is installed.
# make sure to have matching versions of Chrome and Chrome Driver
driver = webdriver.Chrome(executable_path=chrome_driver_path)
driver.get("https://aahnik.github.io/")

Getting error while opening a page using selenium webdriver [duplicate]

When I'm executing this code with Selenium using Python:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome(executable_path=r'/Users/qa/Documents/Python/chromedriver')
The error occurred:
Traceback (most recent call last):
File "/Users/qa/Documents/Python/try.py", line 4, in <module>
driver = webdriver.Chrome(executable_path=r'/Users/qa/Documents/Python/chromedriver')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
(Session info: chrome=71.0.3578.98)
(Driver info: chromedriver=2.44.609545 (c2f88692e98ce7233d2df7c724465ecacfe74df5),platform=Mac OS X 10.13.6 x86_64)
Can someone help me? Thanks.

I had a similar error, first getting the error message:
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited normally.
(unknown error: DevToolsActivePort file doesn't exist)
This was solved by adding options.add_argument("--remote-debugging-port=9230") to the ChromeOptions(). And the program runs once and I gained the same error message as above:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
(Session info: headless chrome=89.0.4389.114)
The problem here was that the chrome process does not close properly in the program so the process is still active on the debugging-port. To solve this problem close the active port sudo kill -9 $(sudo lsof -t -i:9230) and simply add the following lines to the end of the code:
driver.stop_client()
driver.close()
driver.quit()
Since I didn't find this answer anywhere, I hope it helps someone.

If you have options.add_argument("--remote-debugging-port=9222") change this to options.add_argument("--remote-debugging-port=9230")
or just simply Adding options.add_argument("--remote-debugging-port=9230") fixed in my case.

This error message...
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
...implies that the ChromeDriver was unable to initiate/spawn a new WebBrowser i.e. Chrome Browser session.
You need to consider a fact:
As you are using Mac OS X the Key executable_path must be supported with a Value as :
'/Users/qa/Documents/Python/chromedriver'
So line will be:
driver = webdriver.Chrome(executable_path='/Users/qa/Documents/Python/chromedriver')
Note: The path itself is a raw path so you don't need to add the switch r and drop it.
Additionally, ensure that /etc/hosts on your system contains the following entry :
127.0.0.1 localhost.localdomain localhost
#or
127.0.0.1 localhost loopback

I ran into this same issue on a Windows 10 machine. What I had to do to resolve the issue was to open the Task Manager and exit all Python.exe processes, along with all Chrome.exe processes. After doing this,

I am facing the same error and add the close code solve the problem, the code finnaly block look like this:
#staticmethod
def fetch_music_download_url(music_name: str):
chrome_driver_service = Service(ChromeDriverManager(chrome_type=ChromeType.GOOGLE).install())
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--remote-debugging-port=9230")
driver = webdriver.Chrome(service=chrome_driver_service, options=chrome_options)
try:
driver.maximize_window()
driver.get('http://example.cn/music/?page=audioPage&type=migu&name=' + music_name)
driver.implicitly_wait(5)
// do some logic
except Exception as e:
logger.error(e)
finally:
// add the close logic
driver.stop_client()
driver.close()
driver.quit()
chrome_driver_service.stop()
the key is you should close the chrome service after using it by add chrome_driver_service.stop().hope this code could help other people facing the same issue.

This worked for me on WINDOWS OS
from selenium import webdriver
import time
from bs4 import BeautifulSoup
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(options=options)
driver.get(url)
time.sleep(3)
page = driver.page_source
driver.quit()
soup = BeautifulSoup(page, 'html.parser')
Hope you'd find this useful. I also used it more comprehensively here.

Why chrome browser doesn't launch when running Selenium chrome driver on Ubuntu 18.04

I am getting various errors when I'm trying to run a python program using Selenium library and chromedriver. I follow the instructions on Selenium website to install all what I needed and then start programming my two first lines :
from selenium.webdriver import Chrome
driver = Chrome()
I got this error message :
WebDriverException Traceback (most recent call last)
<ipython-input-16-dce6fb94cc37> in <module>
1 from selenium.webdriver import Chrome
2
----> 3 driver = Chrome()
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py in __init__(self, executable_path, port, options, service_args, desired_capabilities, service_log_path, chrome_options, keep_alive)
79 remote_server_addr=self.service.service_url,
80 keep_alive=keep_alive),
---> 81 desired_capabilities=desired_capabilities)
82 except Exception:
83 self.quit()
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in __init__(self, command_executor, desired_capabilities, browser_profile, proxy, keep_alive, file_detector, options)
155 warnings.warn("Please use FirefoxOptions to set browser profile",
156 DeprecationWarning, stacklevel=2)
--> 157 self.start_session(capabilities, browser_profile)
158 self._switch_to = SwitchTo(self)
159 self._mobile = Mobile(self)
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in start_session(self, capabilities, browser_profile)
250 parameters = {"capabilities": w3c_caps,
251 "desiredCapabilities": capabilities}
--> 252 response = self.execute(Command.NEW_SESSION, parameters)
253 if 'sessionId' not in response:
254 response = response['value']
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
WebDriverException: Message: unknown error: Chrome failed to start: crashed
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
so I read many messages and found advice from people who recommend to add options attributes before. So I change my code to this one:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('https://google.com')
But I still don't have any browser windows launch and I can't understand what is happening ... and my log tells this :
/home/lclis/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:8: DeprecationWarning: use options instead of chrome_options
I'm looking for any advice or suggestion, I'm a little lost, thank you very much for your help and time to read my post !
PS: I'm on windows10 but using Ubuntu app to code and use Jupyter.

If I am not mistaken, since you added the headless option, the browser won't open any window.
Try to remove this line: chrome_options.add_argument('--headless')

This first error message...
WebDriverException Traceback (most recent call last)
<ipython-input-16-dce6fb94cc37> in <module>
1 from selenium.webdriver import Chrome
2
----> 3 driver = Chrome()
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py in __init__(self, executable_path, port, options, service_args, desired_capabilities, service_log_path, chrome_options, keep_alive)
79 remote_server_addr=self.service.service_url,
80 keep_alive=keep_alive),
---> 81 desired_capabilities=desired_capabilities)
82 except Exception:
83 self.quit()
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
Ideally your code block should have worked out of box, but as you are using anaconda3 with ipython you need to pass the Key executable_path along with the Value set to the absolute path of the ChromeDriver as follows:
from selenium.webdriver import Chrome
driver = Chrome(executable_path='/path/to/chromedriver')
However, in your second code trial, you were pretty close. This second error message...
/home/lclis/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:8: DeprecationWarning: use options instead of chrome_options
...implies that the ChromeDriver was again unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
Instead of chrome_options you should have used options as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-extensions')
driver = webdriver.Chrome(options=chrome_options)
driver.get('https://google.com')