I am running selenium via python 3 with the chromedriver browser on windows 10.
The program is very active and should run constantly. Means: lots of logging going on.
Problem: the crhomedriver logs get way too long after a few hours, and windows tends to crash.
Everything is working fine, the logs are just minor issues about which I do not really care.
Question:
How may I disable the chromedriver logs?
Notes:
I did my research and did not find any working solution yet.
I am forced to use chromedriver, hence no headless browser like phantomJS is a valid alternative.
Try this, not sure it will work. Pass the options to your chromedriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--disable-logging')
Related
My code worked perfectly until yesterday when I updated Google Chrome to version 110.0.5481.77. Now it's not working in headless mode:
options.add_argument("--headless")
I even tried adding options.add_argument("--window-size=1280,700") but still not working. Although if I remove the headless option it again works correctly!
Accroding to this answer and Google Chrome release notes you should add the headless mode option like below:
options.add_argument("--headless=new")
and no need to specify the window size
I'm using selenium chromedriver with Google Chrome profiles on Mac OS and Windows. As I understand it, in the usual case the selenium script will not work if another profile in Google Chrome is open. This causes some inconvenience as a regular google chrome user - every time before I run the script I have to close the browser completely.
I would not be comfortable switching to the gecko driver because firefox does not allow me to install the extensions I need. Are there any other options on how to solve this problem?
Does running a Python Web-automation script written in Selenium require modifications for running on a VPS (virtual private server)?
One of the problems you can face is the lack of graphical interface on most of VPS servers. You can easily manage using headless mode if you are using chromedriver, for example:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox') # required when running as root user. otherwise you would get no sandbox errors.
driver = webdriver.Chrome(driver_path='/home/dev/chromedriver', chrome_options=chrome_options,
service_args=['--verbose', '--log-path=/tmp/chromedriver.log'])
source
Also, check the anwser to a similar question: How to fix WebDriverException: The browser appears to have exited before we could connect?
I am using Selenium to open a certain website (for example YouTube) on the server, but it can't seem to open the website. However, the code works just fine with a different website. This code also works fine without any problems on my local PC.
I don't know if I have problems with my Chrome Driver or Selenium but it can't open youtube.com as it only outputs: "Before getting the website" and that's it. There are no exceptions/errors that are shown but the script still runs and I have to manually end stop it.
Why can't Selenium open certain URLs on the server, but it works fine on my PC?
options = webdriver.ChromeOptions()
options.add_argument("no-sandbox")
options.add_argument('--headless')
options.add_argument("--start-maximized")
PATH = "./chromedriver"
global driver
driver = webdriver.Chrome(PATH, chrome_options=options)
print("Before getting the website")
driver.get("https://youtube.com")
print("opened", driver.current_url)
I had a exactly same problem. Perhaps i don't know why this happened.
NOTE:
When you are scraping working on Ubuntu ec2 with no GUI you have to provide some GUI interface for chrome to run and for me Xvfb solved it.
"Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display (e.g., browser tests on a CI server) while also having the ability to take screenshots."
SOLUTION
Install Xvfb for ubuntu: sudo apt install xvfb
Now execute your script as: xvfb-run python[version] script.py
IMPORTANT NOTE:
If your program stuck during initialization does not show any output just make sure that you did not add chrome_option.add_argument("disable-dev-shm-usage").
If this argument is added in your chrome header then it would disable the /dev/shm. Not sure but it is some shared memory and xvfb is in-memory display server which i think need it.
This worked for me.
I am using chromedriver for web scraping by giving the binary path.
driver = webdriver.Chrome(r"C:\Program Files\JetBrains\PyCharm Community Edition 2017.3.3\bin\chromedriver.exe")
driver.get("https://www.example.com/")
This invoke chromedriver in GUI mode. How can I start chrome in headless mode?
Chrome headless is ideally much better than PhantomJS, whose owner decided to stop maintaining the project because the arrival of Chrome headless made it a bit less necessary. That being said, if you have a Chrome version which supports headless, you can do this:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(chrome_options=options)
As you see, headless is an argument, so if for some reason you want to run the same code but you need to see the GUI, remove that argument.
By the way, if you ever wanted to give the binary location, a nice way to do it is also with options:
options.binary_location = 'path to your chrome binary'
But if your installed version is recent enough, there should be no reason to do so.