Develop Reference

Can't run Selenium RPA with Python on AWS EC2 instance [duplicate]

I'm trying to install and use Selenium chromedriver on my virtual ubuntu shell, I follow step by step various tutorial but it seems there is still something going wrong ... And after many research on the issue I can't find any answer.
Here is the little code I'm trying to run :
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('https://google.com')
I added the chrome_options thanks to the advice I found on various topic. However I don't really understand why is it necessary?
Unfortunately, my program send me the following error and here I am, stuck and I don't know what to do:
/home/lclis/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:7: DeprecationWarning: use options instead of chrome_options
import sys
---------------------------------------------------------------------------
WebDriverException Traceback (most recent call last)
<ipython-input-4-259faa721232> in <module>
5 chrome_options.add_argument('--no-sandbox')
6 chrome_options.add_argument('--disable-extensions')
----> 7 driver = webdriver.Chrome(chrome_options=chrome_options)
8 driver.get('https://google.com')
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py in __init__(self, executable_path, port, options, service_args, desired_capabilities, service_log_path, chrome_options, keep_alive)
79 remote_server_addr=self.service.service_url,
80 keep_alive=keep_alive),
---> 81 desired_capabilities=desired_capabilities)
82 except Exception:
83 self.quit()
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in __init__(self, command_executor, desired_capabilities, browser_profile, proxy, keep_alive, file_detector, options)
155 warnings.warn("Please use FirefoxOptions to set browser profile",
156 DeprecationWarning, stacklevel=2)
--> 157 self.start_session(capabilities, browser_profile)
158 self._switch_to = SwitchTo(self)
159 self._mobile = Mobile(self)
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in start_session(self, capabilities, browser_profile)
250 parameters = {"capabilities": w3c_caps,
251 "desiredCapabilities": capabilities}
--> 252 response = self.execute(Command.NEW_SESSION, parameters)
253 if 'sessionId' not in response:
254 response = response['value']
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Anyone already had this issue or have any suggestion on it?

Thumb rule
A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and highly discouraged. You need to configure your environment to run Chrome as a regular user instead.
This first error message...
/home/lclis/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:7: DeprecationWarning: use options instead of chrome_options
import sys
---------------------------------------------------------------------------
WebDriverException Traceback (most recent call last)
<ipython-input-4-259faa721232> in <module>
5 chrome_options.add_argument('--no-sandbox')
6 chrome_options.add_argument('--disable-extensions')
----> 7 driver = webdriver.Chrome(chrome_options=chrome_options)
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
As per the Release Notes of Selenium's python client v3.8.0 chrome_options is deprecated:
Browser option parameters are now standardized across drivers as options. firefox_options, chrome_options, and ie_options are now deprecated.
Solution
You need to use options instead of chrome_options while initializing ChromeDriver/Chrome session. So effectively your code block will be:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-extensions')
driver = webdriver.Chrome(executable_path='/path/to/chromedriver', options=chrome_options)
driver.get('https://google.com')

You can run a selenium crawler on a server (ubuntu with no GUI, no X window available in the box) with this code snippet :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
driver = webdriver.Chrome(executable_path='/usr/lib/chromium-browser/chromedriver', options=chrome_options)
driver.get('https://google.com')

I was googling for ruby error, but found this. For some it may be helpful.
I wrongly set capybara session (which relies on selenium on its own)
# WRONG
session = Capybara::Session.new(:selenium_chrome)
# CORRECT, see headless added
session = Capybara::Session.new(:selenium_chrome_headless)

SessionNotCreatedException: Expected browser binary location, but unable to find binary in default location accessing GeckoDriver from mount location

I am trying to use Selenium in Google Colab, but I get some errors when I try to run a Firefox instance. i followed this links:
Selenium Documentation here, I tried with the Driver Management Software but I got the error that says that was unable to find the binary location to Firefox. So I tried with the Hard Coded Location method, but I still have the same error, so I think that I must install firefox in google colab, but I don't know if this is a correct approach
Here is my code:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
service = FirefoxService(executable_path= '/content/drive/MyDrive/geckodriver')
driver = webdriver.Firefox(service=service)
And this is the entire error:
SessionNotCreatedException Traceback (most recent call last)
<ipython-input-7-51ea1584f592> in <module>()
1 service = FirefoxService(executable_path= '/content/drive/MyDrive/geckodriver')
----> 2 driver = webdriver.Firefox(service=service)
4 frames
/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
245 alert_text = value['alert'].get('text')
246 raise exception_class(message, screen, stacktrace, alert_text) # type: ignore[call-arg] # mypy is not smart enough here
--> 247 raise exception_class(message, screen, stacktrace)
248
249 def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT:
SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line
Edit
i tried installing firefox on Google colab, and i think all is correct, but when it tried to run firefox i get this output on geckodriver.log:
1643070515477 geckodriver INFO Listening on 127.0.0.1:43355
1643070515979 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "--headless" "--no-sandbox" "--disable-dev-shm-usage" "--remote-debugging-port" "43945" "-no-remote" "-profile" "/tmp/rust_mozprofileeDGKcf"
src/tcmalloc.cc:283] Attempt to free invalid pointer 0x7fe67e416250
Redirecting call to abort() to mozalloc_abort
Here is the code:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
service = FirefoxService(executable_path= '/content/drive/MyDrive/geckodriver')
options = webdriver.FirefoxOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Firefox(service= service, options= options)

Seems you are downloading and storing the GeckoDriver in a network location and then mounting it:
drive.mount('/content/drive', force_remount=True)
As per several discussions, GeckoDriver / FirefoxService can't be initiated from a network location.
You can find a couple of relevant discussions in:
CreatePlatformSocket() returned an error: An invalid argument was supplied. (0x2726) when trying to access chromedriver through network path
CreatePlatformSocket() failed: Address family not supported by protocol and cannot find chrome binary using WebDriverManager in AWS Lambda
Solution
So a generalized solution would be to access GeckoDriver / Firefox by putting them into the local drives of the host machine.

Getting error while opening a page using selenium webdriver [duplicate]

When I'm executing this code with Selenium using Python:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome(executable_path=r'/Users/qa/Documents/Python/chromedriver')
The error occurred:
Traceback (most recent call last):
File "/Users/qa/Documents/Python/try.py", line 4, in <module>
driver = webdriver.Chrome(executable_path=r'/Users/qa/Documents/Python/chromedriver')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
(Session info: chrome=71.0.3578.98)
(Driver info: chromedriver=2.44.609545 (c2f88692e98ce7233d2df7c724465ecacfe74df5),platform=Mac OS X 10.13.6 x86_64)
Can someone help me? Thanks.

I had a similar error, first getting the error message:
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited normally.
(unknown error: DevToolsActivePort file doesn't exist)
This was solved by adding options.add_argument("--remote-debugging-port=9230") to the ChromeOptions(). And the program runs once and I gained the same error message as above:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
(Session info: headless chrome=89.0.4389.114)
The problem here was that the chrome process does not close properly in the program so the process is still active on the debugging-port. To solve this problem close the active port sudo kill -9 $(sudo lsof -t -i:9230) and simply add the following lines to the end of the code:
driver.stop_client()
driver.close()
driver.quit()
Since I didn't find this answer anywhere, I hope it helps someone.

If you have options.add_argument("--remote-debugging-port=9222") change this to options.add_argument("--remote-debugging-port=9230")
or just simply Adding options.add_argument("--remote-debugging-port=9230") fixed in my case.

This error message...
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from disconnected: unable to connect to renderer
...implies that the ChromeDriver was unable to initiate/spawn a new WebBrowser i.e. Chrome Browser session.
You need to consider a fact:
As you are using Mac OS X the Key executable_path must be supported with a Value as :
'/Users/qa/Documents/Python/chromedriver'
So line will be:
driver = webdriver.Chrome(executable_path='/Users/qa/Documents/Python/chromedriver')
Note: The path itself is a raw path so you don't need to add the switch r and drop it.
Additionally, ensure that /etc/hosts on your system contains the following entry :
127.0.0.1 localhost.localdomain localhost
#or
127.0.0.1 localhost loopback

I ran into this same issue on a Windows 10 machine. What I had to do to resolve the issue was to open the Task Manager and exit all Python.exe processes, along with all Chrome.exe processes. After doing this,

I am facing the same error and add the close code solve the problem, the code finnaly block look like this:
#staticmethod
def fetch_music_download_url(music_name: str):
chrome_driver_service = Service(ChromeDriverManager(chrome_type=ChromeType.GOOGLE).install())
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--remote-debugging-port=9230")
driver = webdriver.Chrome(service=chrome_driver_service, options=chrome_options)
try:
driver.maximize_window()
driver.get('http://example.cn/music/?page=audioPage&type=migu&name=' + music_name)
driver.implicitly_wait(5)
// do some logic
except Exception as e:
logger.error(e)
finally:
// add the close logic
driver.stop_client()
driver.close()
driver.quit()
chrome_driver_service.stop()
the key is you should close the chrome service after using it by add chrome_driver_service.stop().hope this code could help other people facing the same issue.

This worked for me on WINDOWS OS
from selenium import webdriver
import time
from bs4 import BeautifulSoup
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(options=options)
driver.get(url)
time.sleep(3)
page = driver.page_source
driver.quit()
soup = BeautifulSoup(page, 'html.parser')
Hope you'd find this useful. I also used it more comprehensively here.

Why chrome browser doesn't launch when running Selenium chrome driver on Ubuntu 18.04

I am getting various errors when I'm trying to run a python program using Selenium library and chromedriver. I follow the instructions on Selenium website to install all what I needed and then start programming my two first lines :
from selenium.webdriver import Chrome
driver = Chrome()
I got this error message :
WebDriverException Traceback (most recent call last)
<ipython-input-16-dce6fb94cc37> in <module>
1 from selenium.webdriver import Chrome
2
----> 3 driver = Chrome()
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py in __init__(self, executable_path, port, options, service_args, desired_capabilities, service_log_path, chrome_options, keep_alive)
79 remote_server_addr=self.service.service_url,
80 keep_alive=keep_alive),
---> 81 desired_capabilities=desired_capabilities)
82 except Exception:
83 self.quit()
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in __init__(self, command_executor, desired_capabilities, browser_profile, proxy, keep_alive, file_detector, options)
155 warnings.warn("Please use FirefoxOptions to set browser profile",
156 DeprecationWarning, stacklevel=2)
--> 157 self.start_session(capabilities, browser_profile)
158 self._switch_to = SwitchTo(self)
159 self._mobile = Mobile(self)
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in start_session(self, capabilities, browser_profile)
250 parameters = {"capabilities": w3c_caps,
251 "desiredCapabilities": capabilities}
--> 252 response = self.execute(Command.NEW_SESSION, parameters)
253 if 'sessionId' not in response:
254 response = response['value']
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
WebDriverException: Message: unknown error: Chrome failed to start: crashed
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
so I read many messages and found advice from people who recommend to add options attributes before. So I change my code to this one:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('https://google.com')
But I still don't have any browser windows launch and I can't understand what is happening ... and my log tells this :
/home/lclis/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:8: DeprecationWarning: use options instead of chrome_options
I'm looking for any advice or suggestion, I'm a little lost, thank you very much for your help and time to read my post !
PS: I'm on windows10 but using Ubuntu app to code and use Jupyter.

If I am not mistaken, since you added the headless option, the browser won't open any window.
Try to remove this line: chrome_options.add_argument('--headless')

This first error message...
WebDriverException Traceback (most recent call last)
<ipython-input-16-dce6fb94cc37> in <module>
1 from selenium.webdriver import Chrome
2
----> 3 driver = Chrome()
~/anaconda3/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py in __init__(self, executable_path, port, options, service_args, desired_capabilities, service_log_path, chrome_options, keep_alive)
79 remote_server_addr=self.service.service_url,
80 keep_alive=keep_alive),
---> 81 desired_capabilities=desired_capabilities)
82 except Exception:
83 self.quit()
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
Ideally your code block should have worked out of box, but as you are using anaconda3 with ipython you need to pass the Key executable_path along with the Value set to the absolute path of the ChromeDriver as follows:
from selenium.webdriver import Chrome
driver = Chrome(executable_path='/path/to/chromedriver')
However, in your second code trial, you were pretty close. This second error message...
/home/lclis/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:8: DeprecationWarning: use options instead of chrome_options
...implies that the ChromeDriver was again unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
Instead of chrome_options you should have used options as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-extensions')
driver = webdriver.Chrome(options=chrome_options)
driver.get('https://google.com')

python selenium phantomjs execute_script

Why an error occurs when using phantomjs?
from selenium import webdriver
page_index = 1076
#driver = webdriver.Chrome() # no any errors.
driver = webdriver.PhantomJS() # will error.
driver.get('https://www.nexusmods.com/mods/')
driver.execute_script(f"window.RH_ModList.Send('page', '{page_index}');")
Result:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: {"errorMessage":"undefined is not an object (evaluating 'window.RH_ModList.Send')","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"118","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:57491","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"script\": \"window.RH_ModList.Send('page', '1076');\", \"args\": [], \"sessionId\": \"e7abf670-0132-11e9-ba9e-f19731f1c19f\"}","url":"/execute","urlParsed":{"anchor":"","query":"","file":"execute","directory":"/","path":"/execute","relative":"/execute","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/execute","queryKey":{},"chunks":["execute"]},"urlOriginal":"/session/e7abf670-0132-11e9-ba9e-f19731f1c19f/execute"}}
Screenshot: available via screen

Why an error occurs when using phantomjs?
It is because PhantomJS is not support ES6 Promise
See the issue in Github here. if you do debugging you will see this error for that page
ReferenceError: Can't find variable: Promise
phantomjs://platform/bootstrap.js:87 2
ReferenceError: Can't find variable: $
Because PhantomJS abandoned there are no easy fix.