I'm trying to open a web browser using Selenium Chromedriver on Windows 10 with python in jupyter notebooks through an ubuntu command prompt. I've read stack overflow posts and tried to solve based on their answers, but I'm stuck in a loop where I keep receiving the same 3 errors.
Here is what I have installed:
OS - Windows 10, 1709, 64-bit Selenium - 3.8.1 Chromedriver - 2.45
Chrome - Version 71.0.3578.98 Python - 3.5.2
I tried various websites. The goal is to eventually get to a social media login page, but i'm stuck at opening a new blank web browser.
Here is my starting code:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.imdb.com/")
WebDriverException: Message: 'chromedriver' executable may have wrong
permissions.
Then I tried the following:
from selenium import webdriver
chromedriver = "C:/Users/xxxx/AppData/Local/lxss/home/xxxx/chromedriver.exe"
browser = webdriver.Chrome(chromedriver)
browser.get('https://www.imdb.com/')
WebDriverException: Message: 'chromedriver.exe' executable needs to be
in PATH.
Here are the steps I have taken:
I added a PATH under environment variables to the folder - (C:\Users\xxxxx\AppData\Local\lxss\home\xxxx),
I tried using \, and /, and even \
Once I added the PATH, I tried the following two codes (and various versions) and received the same error:
A.
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx)
driver.get("https://www.imdb.com/")
B.
from selenium import webdriver
chromedriver = r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx\chromedriver.exe'
driver = webdriver.Chrome(chromedriver)
driver.get("https://www.imdb.com/")
WebDriverException: Message:
'C:\Users\xxxx\AppData\Local\lxss\home\xxxx' executable may have wrong
permissions.
Then I did the following:
- Went to file Properties, under General, took off Read-only (Windows permissions)
- Went to file Properties, under Security and changed permissions to Full Control
- In the C:\Users\xxxxx\AppData\Local\lxss\home\xxxx file, I changed the permissions
using chmod 777 -R in my command prompt. Then I tried the following code:
from selenium import webdriver
import os
chromedriver = r'C:\Users\xxxx\AppData\Local\lxss\home\xxxx\chromedriver.exe'
driver = webdriver.Chrome(os.path.join(os.getcwd(), 'chromedriver.exe'))
driver.get("https://www.imdb.com/")
WebDriverException: Message: Service /home/ariggs/chromedriver.exe
unexpectedly exited. Status code was: 1
I am stuck between these three error messages. Does anyone have another suggestion for a beginner?
You can actually start Windows executables from a linux subsystem as it is described here https://learn.microsoft.com/en-us/windows/wsl/interop.
But you have to keep in mind that Selenium and ChromeDriver communicate over a network connection. Actually chromedriver starts its own http server and Selenium sends requests and receives responses over http. (see https://sqa.stackexchange.com/questions/28358/how-does-chromedriver-exe-work-on-a-core-and-fundamental-level)
According to Microsoft, WSL and Windows share the same IP address and network connections via localhost are supported. But in your case there seems to be a problem during the startup.
You can start a remote webdriver on windows with Python and connect to that.
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import subprocess
subprocess.run(["d:\\develop\\remotewebdriver.cmd", ""])
driver = webdriver.Remote(
command_executor='http://localhost:4444/wd/hub',
desired_capabilities=DesiredCapabilities.CHROME)
driver.get('http://www.google.in/')
driver.close()
You need a windows script remotewebdriver.cmd for the remote webdriver that is called from Python:
SET JAVA_HOME=D:\develop\Java\jdk-11.0.2
d:
cd \develop
start D:\develop\Java\jdk-11.0.2\bin\java -Dwebdriver.chrome.driver=d:\develop\chromedriver.exe -jar selenium-server-standalone-3.141.59.jar
You have to adapt the path to your own environment. This setup works for me.
You can do this way.
Step 1:
Download chrome driver from this link (download the specific version as chrome):
http://chromedriver.chromium.org/downloads
Important: check your chrome version first. Go to help -> about Google Chrome, to see the version of your chrome.
Step 2:
After downloading extract and save the chromedriver file in a specific folder like, C:\selenium
go to environment variable and add a new path, C:\selenium
Step 3
Double click on the chromedriver application and then restart your command prompt. (If you are using conda environment.)
Related
I'm using selenium chromedriver with Google Chrome profiles on Mac OS and Windows. As I understand it, in the usual case the selenium script will not work if another profile in Google Chrome is open. This causes some inconvenience as a regular google chrome user - every time before I run the script I have to close the browser completely.
I would not be comfortable switching to the gecko driver because firefox does not allow me to install the extensions I need. Are there any other options on how to solve this problem?
I have a selenium script that I execute in another python program. This program will only execute when I am logged into the server using ssh as root but not executable by the www-data user because it returns with the error:
selenium.common.exceptions.WebDriverException: Message: Service /usr/bin/chromedriver unexpectedly exited. Status code was: 1
I run the script using this command:
os.system('python3 /var/website/webscraping.py' + str(VARIABLE))
Any help would be appriciated!
Ideally you should have been able to execute the program as www-data user. However this error message...
selenium.common.exceptions.WebDriverException: Message: Service /usr/bin/chromedriver unexpectedly exited. Status code was: 1
...implies that the ChromeDriver unexpectedly exited.
Thumb rule
A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and highly discouraged. You need to configure your environment to run Chrome as a regular user instead.
There can be numerous reasons behind this error. Your code trials and the complete error stacktrace would have given us some more visibility on what's wrong happening under the hood.
However, a couple of remedial steps are as follows:
Ensure that Chrome is updated to current chrome=96.0.4664.45 (as per chrome=96.0.4664.45 release notes).
Ensure that ChromeDriver is updated to current ChromeDriver v96.0 level.
Ensure that you have downloaded the exact format of the ChromeDriver binary from the download location pertaining to your underlying OS among:
chromedriver_win32: For Windows OS
chromedriver_mac64: For MAC OS X
chromedriver_linux64: For Linux OS
Using Selenium you need to pass the absolute path of the ChromeDriver binary through the argument executable_path and you need to mention the path within single quotes (i.e. '') seperated by a single forward slash (i.e. \) along with the raw switch (i.e. r) as follows:
from selenium import webdriver
driver = webdriver.Chrome(executable_path='/usr/bin/chromedriver')
driver.get(url)
Ensure that ChromeDriver binary have executable permission for the non-administrator user.
Execute your Test as a non-administrator user.
Another potential reason for the error can be due to missing the entry 127.0.0.1 localhost in /etc/hosts
Windows OS - Add 127.0.0.1 localhost to /etc/hosts
Mac OSX - Ensure the following entries:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost
References
As per the discussion in selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service geckodriver:
Selenium does not require 127.0.0.1 localhost to be explicitly set in the host file.
However it is mandatory requirement to map localhost to the IPv4 local loopback (127.0.0.1)
The mechanism of this mapping does not have to be through the hosts file always.
On Windows OS systems it is not mapped in the hosts file at all (resolving localhost is done by the DNS resolver).
TL;DR
How to reset the Hosts file back to the default
Update
As per your comment update:
chromedriver is at version 97 and google-chrome is at version 96
Your main issue seems to be the incompatibility between the version of the binaries you are using a there is a clear mismatch between chromedriver=97.0 and the chrome=96.0.4664.45
Solution
Ensure that:
ChromeDriver is updated to current ChromeDriver v96.0 level.
Chrome is updated to current chrome=96.0.4664.45 (as per chrome=96.0.4664.45 release notes).
You can find a relevant detailed discussion in WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally." (Driver info: chromedriver=97) using Selenium Python
Looks like your browser version is not matching with the browser binary provided by you in your path. You can implement WebDriverManager to resolve your issue.
Install WebDriverManager module for python
pip install webdriver-manager
Setup/initialize ChromeDriver service as below
//selenium 3
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
//selenium 4
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
More information below ->
https://cpsat.agiletestingalliance.org/2021/03/25/selenium-webdriver-manager-in-python/
I'm trying to run a headless Chrome process with an extension1. I'm using WSL.
For testing purposes I can use one of two browsers:
Ubuntu: Google Chrome 86.0.4240.183
Windows: Version 86.0.4240.183 (Official Build) (64-bit)
For production purposes the Ubuntu's Chrome is the only viable option2.
The rest of the stack:
Python 3.6.9
selenium: 3.141.0
In order to overcome Chrome not being able to start headlessly with extensions I use PyVirtualDisplay.
from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600), backend='xvfb')
display.start()
options = ChromeOptions()
options.add_extension('/home/plonca/selenium_draft/extensions.crx')
options.add_argument('--no-sandbox')
windows_chrome_driver = '/home/plonca/python_virtual_environments/selenium_jupyter_venv/bin/chromedriver.exe'
ubuntu_chrome_driver = '/home/plonca/python_virtual_environments/selenium_jupyter_venv/bin/chromedriver'
chrome_driver = webdriver.Chrome(executable_path=ubuntu_chrome_driver
, options=options
)
The problem occurs in the last line: if the windows_chrome_driver is set as executable_path everything is fine. If I set ubuntu_chrome_driver instead, I get an error:
WebDriverException: Message: unknown error: failed to wait for extension background page to load: chrome-extension://bmlddehkgnjdalnfaecgflmmeknlbohi/_generated_background_page.html
from tab crashed
I wonder what the cause of the error might be. How to make chrome_driver work in the headless environment?
1 The extension is needed so that I can authenticate via a pop-up window. The extension has been created according to instructions found here. Other options such as sending username and password in the URL or automating the click with AutoIt doesn't work.
2 Please note that the Chrome seems to be the only option since using Firefox is a matter of an unresolved issue.
So I have made a program on one computer using selenium and that worked, Now using it in another computer I get this error:
selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service chromedriver
Now this same issue was mention in:
Selenium python: Can not connect to the Service %s" % self.path
Selenium python: Can not connect to the Service %s" % self.path
Selenium and Python3 ChromeDriver raises Message: Can not connect to the Service chromedriver
however the solutions mentioned didnt work.
I am using chrome version 79 and have installed chromedriver 79, I tested writing chromedriver in command line and that works which means path is configured right, I have made sure 127.0.0.1 localhost is also in etc/hosts
Below is my code which works on my computer (so i doubt its an issue with the code):
chrome_options = Options()
chrome_options.add_argument("--headless")
with webdriver.Chrome(chrome_options=chrome_options) as driver:
driver.set_window_size(800, 460) # takes two arguments, width and height of the browser and it has to be called before using get()
driver.execute_script("document.body.style.zoom='150%'")
driver.get("file:\\"+url) # takes one argument, which is the url of the website you want to open
driver.find_element_by_tag_name('body').screenshot(output) # avoids scrollbar
In the last question I also tried this modification:
chrome_options = Options()
chrome_options.add_argument("--headless")
with webdriver.Chrome("C:\\chromedriver.exe",chrome_options=chrome_options) as driver:
driver.set_window_size(800, 460) # takes two arguments, width and height of the browser and it has to be called before using get()
driver.execute_script("document.body.style.zoom='150%'")
driver.get("file:\\"+url) # takes one argument, which is the url of the website you want to open
driver.find_element_by_tag_name('body').screenshot(output) # avoids scrollbar
which would instead give me this almost identical error message:
selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service C:\chromedriver.exe
I am unsure where the issue could lie.
I am using windows by the way.
EDIT: Things I tried and didn't work:
1- running everything as both admin and normal
2- re-installing chrome
3- using the beta-chroma 80 and webdriver 80
4- using normal chrome 79 and webdriver 79
5- Having both the script and the driver in the same directory (while using a
correct path)
6- Having an external path and have it setup as needed
7- Using it in the PATH folder.
8- Adding "127.0.0.1 localhost" to etc/hosts
9- Running service test
I have ensured in every test that everything was in it's correct placement, I have ran a reboot before every new test, and they always give me the same error, which also happens to occur if I had an incorrect path as well, but once I ran the code for a service and it gave me a different error as I had my webdriver in C:/ which required admin privilages, however re-running the test again with the correct privilages gave back the same error
Update the issue isn't exclusive to the chrome driver. Even following setup instructions for either the Firefox or edge drivers end up on the same issues. It makes me suspect that the connection is facing some issue. I have tried running the test codes provided by Mozilla for the setup and it didn't work.
Unsure if that does help much or at all.
This error message...
selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service chromedriver
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
You need to take care of a couple of things:
Ensure that you have downloaded the exact format of the ChromeDriver binary from the download location pertaining to your underlying OS among:
chromedriver_linux64.zip: For Linux OS
chromedriver_mac64.zip: For Mac OSX
chromedriver_win32.zip: For Windows OS
Ensure that /etc/hosts file contains the following entry:
127.0.0.1 localhost
Ensure that ChromeDriver binary have executable permission for the non-root user.
Ensure that you have passed the proper absolute path of ChromeDriver binary through the argument executable_path as follows:
with webdriver.Chrome(executable_path=r'C:\path\to\chromedriver.exe', chrome_options=chrome_options) as driver:
So your effective code block will be:
options = Options()
options.add_argument("--headless")
options.add_argument('--no-sandbox') # Bypass OS security model
options.add_argument('--disable-gpu') # applicable to windows os only
options.add_argument("--disable-dev-shm-usage") # overcome limited resource problems
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
with webdriver.Chrome(executable_path=r'C:\path\to\chromedriver.exe', options=options) as driver:
driver.set_window_size(800, 460) # takes two arguments, width and height of the browser and it has to be called before
driver.execute_script("document.body.style.zoom='150%'")
driver.get("file:\\"+url) # takes one argument, which is the url of the website you want to open
driver.find_element_by_tag_name('body').screenshot(output) # avoids scrollbar
Mandatory Considerations
Finally, to avoid incompatibility between the version of the binaries you are using ensure that:
Selenium is upgraded to current levels Version 3.141.59.
ChromeDriver is updated to current ChromeDriver v79.0.3945.36 level.
Chrome is updated to current Chrome Version 79.0 level. (as per ChromeDriver v79.0 release notes)
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.
Take a System Reboot.
Execute your #Test as non-root user.
References
You can find a couple of reference discussions in:
Python Selenium “Can not connect to the Service %s” % self.path in linux server
selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service chromedriver.exe while opening chrome browser
How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
I am wondering if there is a way within py2app to include the Firefox browser, or if there is a way for Selenium to use Firefox without having to install Firefox on the host machine.
I have created an app with py2app that uses Selenium, however, I have Firefox installed on my machine, but not everyone that will receive the app will have Firefox installed. I am looking for a way to either include Firefox in the distribution or go around this.
Script will not run if Firefox is not preinstalled.
You can test your script with other browser, for example Chrome. If it works on Chrome also, then you can edit script like this:
from selenium.common.exceptions import WebDriverException
try:
driver = webdriver.Firefox()
except WebDriverException:
driver = webdriver.Chrome()
You can add same for few more browsers (IE, Opera, Safari...) to be sure that script will run on users machine