Develop Reference

Unable to use chromedriver on linux server [Exec format error]

I have a raspberry-pi running linux-server as platform. Therefore there is no GUI and I execute all my tasks through terminal by SSH-ing into the Pi. Platform details:
uname -a
>> Linux ubuntu 5.4.0-1080-raspi #91-Ubuntu SMP PREEMPT Thu Jan 19 09:35:03 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Chromium [No issues here]
I have installed Chromium through snap.
chromium --version
>> Chromium 109.0.5414.119 snap
I am able to run chromium, navigate to a website, and take a snapshot
chromium --headless --disable-gpu --screenshot https://www.wikipedia.com
>> 0215/140750.965255:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.
[0215/140752.998408:WARNING:sandbox_linux.cc(385)] InitializeSandbox() called with multiple threads in process gpu-process.
[0215/140802.665622:INFO:headless_shell.cc(223)] 84646 bytes written to file screenshot.png
Chromedriver [Issues]
I downloaded chromedriver this way
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
And moved Chromedriver to the applications folder after unzipping
I get this error when trying to get chromedriver version, let alone run it
chromedriver --version
>> bash: /usr/local/bin/chromedriver: cannot execute binary file: Exec format error
My Python Script [Issues]
Here is the script I want to be able to run finally
import selenium
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(options=options)
driver.get("https://www.wikipedia.com")
driver.save_screenshot("proof.png")
This is the error I get when I try to run it
python3 test.py
>> OSError: [Errno 8] Exec format error: 'chromedriver'
What I've Tried already
Using chromedriver directly through ChromeDriverManager
import selenium
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(service=Service(ChromeDriverManager(path=".", chrome_type=ChromeType.CHROMIUM).install()), options=options)
driver.get("https://www.wikipedia.com")
driver.save_screenshot("proof.png")
The error
OSError: [Errno 8] Exec format error: './.wdm/drivers/chromedriver/linux64/109.0.5414/chromedriver'
Checking file permissions
Made sure file has execute permissions
ls -l /usr/local/bin/chromedriver
>> -rwxr-xr-x 1 ubuntu ubuntu 20427216 Sep 8 2021 /usr/local/bin/chromedriver

As you are having a raspberry-pi running linux-server as platform with Chromium 109.0.5414.119 you need to use ChromeDriver of:
Version 109.0
Linux compatible format i.e. chromedriver_linux64.tar.gz
Your effective line of code will be:
wget https://chromedriver.storage.googleapis.com/109.0.5414.74/chromedriver_linux64.tar.gz

How to properly use selenium with geckodriver and firefox with python on Ubuntu?

I am trying to use the geckodriver with firefox and selenium on my Ubuntu machine. This is the code I have so far:
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.keys import Keys
from selenium import webdriver
#path where browser is installed
binary = '/usr/bin/firefox'
options = webdriver.FirefoxOptions()
options.binary = binary
options.add_argument('start-maximized')
options.add_argument('--headless')
cap = DesiredCapabilities().FIREFOX
cap["marionette"] = False
path_to_driver = "/home/andrea/geckodriver"
# run firefox webdriver from executable path
driver = webdriver.Firefox(firefox_options=options, capabilities=cap, executable_path = path_to_driver)
#driver = webdriver.Firefox(capabilities=cap, executable_path = path_to_driver)
driver.get("https://www.amboss.com/us/account/login")
Despite that I am getting the following error:
selenium.common.exceptions.WebDriverException: Message: Can't load the profile.
Possible firefox version mismatch. You must use GeckoDriver instead for Firefox 48+. Profile Dir: /tmp/tmpuigrk9f7 If you specified a log_file in the FirefoxBinary constructor, check it for details.
The firefox version which I work with is:
Mozilla Firefox 68.0.2
Does anyone have any idea as to how I could go about fixing this?

Step1: Install Selenium
Type in Terminal(in Ubuntu) or in Command Prompt(in Windows)
$pip install selenium
Step2: Download Geckodriver
In order to work with Selenium there should be an executable called 'Gecko Driver' installed.
Download Gecko Driver from the following page:
https://github.com/mozilla/geckodriver/releases
Step3: Install Gecko Driver
Latest version for Windows:
https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-win64.zip
Latest version for Ubuntu:
https://github.com/mozilla/geckodriver/releases/download/v0.26.0/geckodriver-v0.26.0-linux64.tar.gz
Setup Gecko Driver For Windows:
Extract the zip file and move the geckodiver.exe executable file to any location which is already in Path variable(For Example you can move it to Python path location)
Unless add the path of 'geckodriver.exe' to the Path variable
Setup Gecko Driver For Ubuntu:
Open Terminal
Ctrl+Alt+T
move directory to the location where tar file is downloaded
Usually it will be in Downloads. so type $ cd Downloads
Unzip the tar file
eg:
$sudo tar -xvf filename.tar.gz
In my case it is:
$sudo tar -xvf geckodriver-v0.26.0-linux64.tar.gz
Move the geckodriver executable file to the '/usr/local/bin' location
$sudo mv geckodriver /usr/local/bin/
Move the directory to '/usr/local/bin/'
$cd /usr/local/bin/
Now make executable permission for 'geckodriver' executable file
$sudo chmod +x geckodriver
Now type 'geckodriver' in Terminal
geckodriver
If Gecko Driver is not working still then add its path
$export PATH=$PATH:/usr/local/bin/geckodriver
Now it is ready to work with selenium
Sample Code
Some sample codes are here:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import ui
driver = webdriver.Firefox()
driver.get('https://www.google.com/')
page_url=driver.find_elements_by_xpath("//a[#class='content']")
all_title = driver.find_elements_by_class_name("title")
title = [title.text for title in all_title]
print(title)

This error message...
selenium.common.exceptions.WebDriverException: Message: Can't load the profile.
Possible firefox version mismatch. You must use GeckoDriver instead for Firefox 48+. Profile Dir: /tmp/tmpuigrk9f7 If you specified a log_file in the FirefoxBinary constructor, check it for details.
...implies that there was a mismatch between the GeckoDriver and Firefox version while initiating/spawning a new WebBrowsing Session i.e. Firefox Browser session.
Your main issue is the incompatibility between the version of the binaries you are using as follows:
You are using Mozilla Firefox v68.0.2
Your Selenium Client version is is unknown to us.
Your GeckoDriver version is unknown to us.
However as you are using Mozilla Firefox v68.0.2, using GeckoDriver is mandatory and while you use GeckoDriver you can't set the capability marionette as False.
You can find a detailed discussion in How can Geckodriver/Firefox work without Marionette? (running python selenium 3 against FF 53)
Solution
Upgrade Selenium to current levels Version 3.141.59.
Upgrade GeckoDriver to current GeckoDriver v0.24.0 level.
GeckoDriver is present in the specified location.
GeckoDriver is having executable permission for non-root users.
Upgrade Firefox version to Firefox v68.0.2 levels.
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
Take a System Reboot.
Execute your Test as a non-root user.
Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
Outro
GeckoDriver, Selenium and Firefox Browser compatibility chart

If you want to use Firefox with Selenium, you need to import e Firefox Profile. You can use your own Profile through the following steps :
Locate the Firefox Profile directory
You have to specify the absolute path of the Firefox Profile directory when you initiate the webdriver.
from selenium import webdriver
profile = webdriver.FirefoxProfile(*path to your profile*)
driver = webdriver.Firefox(firefox_profile=profile)

Facing chromedriver unexpectedly exited. Status code was: 127 while using Headless chrome browser with docker & python 3.7 on AWS ec2

This is not a duplicate of other questions & I have tried the existing answers as well. On AWS ec2 I have created a docker container in which I am trying to open headless chrome. The docker file is
FROM python:3
# Install Chrome for Selenium
RUN curl https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb -o /chrome.deb
RUN dpkg -i /chrome.deb || apt-get install -yf
RUN rm /chrome.deb
RUN wget https://chromedriver.storage.googleapis.com/2.38/chromedriver_linux64.zip
RUN unzip chromedriver_linux64.zip -d chromedriver
RUN chmod +x /chromedriver/chromedriver
ENV PATH="/chromedriver:${PATH}"
RUN echo 'logging path'
RUN echo "${PATH}"
The code which I am using to open browser is:
options = Options()
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-gpu")
options.add_argument("--disable-extensions")
options.add_argument("--proxy-server='direct://'")
options.add_argument("--proxy-bypass-list=*")
options.add_argument("--start-maximized")
options.add_argument("--headless")
options.add_argument('--no-sandbox')
options.headless = True
driver = webdriver.Chrome(options=options, executable_path='/chromedriver/chromedriver')
driver.get(url)
The error which on ec2 is:
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
self.service.start()
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 98, in start
self.assert_process_still_running()
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 111, in assert_process_still_running
% (self.path, return_code)
selenium.common.exceptions.WebDriverException: Message: Service /chromedriver/chromedriver unexpectedly exited. Status code was: 127
I have tried Installing Chromium on Amazon Linux and other stackoverflow answers as well but unable to solve it.
Google-chrome version - Google Chrome 75.0.3770.100
driver version - 2.38
selenium - 3.141.0
The OS of AWS AMI is linux and not ubuntu thuscommand ldd chrome | grep not would help. Even though I have fixed that issue as well, still getting error. When logged the path of chromedriver it's logged as below
/usr/local/lib/python3.7/site-packages/chromedriver_binary:/chromedriver:

I had a similar error which was fixed by installing xvfb, this is required because you are running a headless session of chrome. Try doing apt-get install xvfb

Python Gmail auto login script [duplicate]

I have a script which uses selenium for testing. Now even opening a Google page using
driver.get(url) # url = Google homepage url
is giving me below error
driver.get("https://my.gumtree.com/login")
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 245, in get
self.execute(Command.GET, {'url': url})
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 233, in execute
self.error_handler.check_response(response)
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot determine loading status
from unknown error: missing or invalid 'entry.level'
(Session info: chrome=65.0.3315.3)
(Driver info: chromedriver=2.29.461591 (62ebf098771772160f391d75e589dc567915b233),platform=Windows NT 10.0.16299 x86_64)
I have Google chrome version 65, Chromedriver 2.35 and selenium 2.53.1
I tried different version combinations(mentioned in below table) as per solutions mentioned in other similar questions but nothing worked.
Selenium Chrome Chromedriver
2.53.0 63 2.33
2.53.1 65(latest) 2.34
3.6.0 2.35(latest)
3.7.0
3.8.0
3.8.1(latest)
EDIT 1: JDK version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

The error says it all :
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot determine loading status
from unknown error: missing or invalid 'entry.level'
Your main issue is the version compatibility among the binaries you are using as follows :
You are using chromedriver=2.29.461591 (which is as per the logs, though you mentioned Chromedriver 2.35 in your question)
Release Notes of chromedriver=2.29.461591 clearly mentions the following :
Supports Chrome v56-58
You are using chrome=65.0.3315.3
Release Notes of chromedriver=2.35 clearly mentions the following :
Supports Chrome v62-64
You are using Selenium Version 2.53.1.
Your JDK version is unknown to us.
Solution
Upgrade JDK to recent levels JDK Version 8 Update 151.
Upgrade ChromeDriver to ChromeDriver v2.35 level.
Keep Chrome to Chrome v64.x levels. (as per ChromeDriver v2.35 release notes)
Upgrade Selenium to current levels Version 3.8.1.
Clean the Project Workspace from your IDE & Rebuild All.
Run CCleaner tool to wipe off all the OS chores.
If your Chrome base version is too old, uninstall Chrome through Revo Uninstaller and install a recent GA Release version of Chrome.
Take a System Reboot.
Execute your Test.

Go to http://chromedriver.chromium.org/downloads
copy the download link according to your OS
wget -N paste_the_link_you_copied
unzip it using below command
unzip chromedriver_linux64.zip
Give the permission by the below command
chmod +x chromedriver
Then follow the below commands, if it says already exists (probabily old version) then go to that path (/usr/local/bin/chromedriver and /usr/bin/chromedriver) and delete chromedriver and run the commands again
sudo mv -f chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver
Hope this helps. Thanks

That error raised beacuse your chrome browser is not compatible with the web driver. If you are using Linux, then simply execute the following command.
sudo apt-get update

Recently I am facing the same problem and take me too much time to figure out what's going on, in my situation facing the problem, I did not close the chrome process after using it, so you should check the process exit or not when you exit the app, this is my last worked Python 3 code demo, hope it will help others:
#staticmethod
def fetch_music_download_url(music_name: str):
chrome_driver_service = Service(ChromeDriverManager(chrome_type=ChromeType.GOOGLE).install())
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--remote-debugging-port=9230")
driver = webdriver.Chrome(service=chrome_driver_service,
options=chrome_options,
executable_path="/usr/local/bin/chromedriver")
try:
driver.maximize_window()
driver.get('http://tool.example.cn/music/?page=audioPage&type=migu&name=' + music_name)
driver.implicitly_wait(5)
driver.find_element(By.CSS_SELECTOR, ".aplayer-list-download.iconfont.icon-xiazai").click()
urls = [a.get_attribute('href') for a in
driver.execute_script('return document.querySelectorAll(".modal-body a[href*=\'http\']")')]
for url in urls:
if "listenSong.do" in url:
logger.info("fetched url:" + url)
FetchMusic.do_save_music_download_url(url)
except Exception as e:
logger.error("scrapy impl error", e)
finally:
driver.stop_client()
driver.close()
driver.quit()
chrome_driver_service.stop()

Selenium WebDriverException: Message: unknown error: cannot determine loading status from unknown error: missing or invalid 'entry.level'

I have a script which uses selenium for testing. Now even opening a Google page using
driver.get(url) # url = Google homepage url
is giving me below error
driver.get("https://my.gumtree.com/login")
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 245, in get
self.execute(Command.GET, {'url': url})
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 233, in execute
self.error_handler.check_response(response)
File "C:\Python34\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot determine loading status
from unknown error: missing or invalid 'entry.level'
(Session info: chrome=65.0.3315.3)
(Driver info: chromedriver=2.29.461591 (62ebf098771772160f391d75e589dc567915b233),platform=Windows NT 10.0.16299 x86_64)
I have Google chrome version 65, Chromedriver 2.35 and selenium 2.53.1
I tried different version combinations(mentioned in below table) as per solutions mentioned in other similar questions but nothing worked.
Selenium Chrome Chromedriver
2.53.0 63 2.33
2.53.1 65(latest) 2.34
3.6.0 2.35(latest)
3.7.0
3.8.0
3.8.1(latest)
EDIT 1: JDK version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

The error says it all :
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot determine loading status
from unknown error: missing or invalid 'entry.level'
Your main issue is the version compatibility among the binaries you are using as follows :
You are using chromedriver=2.29.461591 (which is as per the logs, though you mentioned Chromedriver 2.35 in your question)
Release Notes of chromedriver=2.29.461591 clearly mentions the following :
Supports Chrome v56-58
You are using chrome=65.0.3315.3
Release Notes of chromedriver=2.35 clearly mentions the following :
Supports Chrome v62-64
You are using Selenium Version 2.53.1.
Your JDK version is unknown to us.
Solution
Upgrade JDK to recent levels JDK Version 8 Update 151.
Upgrade ChromeDriver to ChromeDriver v2.35 level.
Keep Chrome to Chrome v64.x levels. (as per ChromeDriver v2.35 release notes)
Upgrade Selenium to current levels Version 3.8.1.
Clean the Project Workspace from your IDE & Rebuild All.
Run CCleaner tool to wipe off all the OS chores.
If your Chrome base version is too old, uninstall Chrome through Revo Uninstaller and install a recent GA Release version of Chrome.
Take a System Reboot.
Execute your Test.

Go to http://chromedriver.chromium.org/downloads
copy the download link according to your OS
wget -N paste_the_link_you_copied
unzip it using below command
unzip chromedriver_linux64.zip
Give the permission by the below command
chmod +x chromedriver
Then follow the below commands, if it says already exists (probabily old version) then go to that path (/usr/local/bin/chromedriver and /usr/bin/chromedriver) and delete chromedriver and run the commands again
sudo mv -f chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver
Hope this helps. Thanks

That error raised beacuse your chrome browser is not compatible with the web driver. If you are using Linux, then simply execute the following command.
sudo apt-get update

Recently I am facing the same problem and take me too much time to figure out what's going on, in my situation facing the problem, I did not close the chrome process after using it, so you should check the process exit or not when you exit the app, this is my last worked Python 3 code demo, hope it will help others:
#staticmethod
def fetch_music_download_url(music_name: str):
chrome_driver_service = Service(ChromeDriverManager(chrome_type=ChromeType.GOOGLE).install())
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--remote-debugging-port=9230")
driver = webdriver.Chrome(service=chrome_driver_service,
options=chrome_options,
executable_path="/usr/local/bin/chromedriver")
try:
driver.maximize_window()
driver.get('http://tool.example.cn/music/?page=audioPage&type=migu&name=' + music_name)
driver.implicitly_wait(5)
driver.find_element(By.CSS_SELECTOR, ".aplayer-list-download.iconfont.icon-xiazai").click()
urls = [a.get_attribute('href') for a in
driver.execute_script('return document.querySelectorAll(".modal-body a[href*=\'http\']")')]
for url in urls:
if "listenSong.do" in url:
logger.info("fetched url:" + url)
FetchMusic.do_save_music_download_url(url)
except Exception as e:
logger.error("scrapy impl error", e)
finally:
driver.stop_client()
driver.close()
driver.quit()
chrome_driver_service.stop()