Develop Reference

How to restart a Python script if it freezes and does not raise any Exception

I am running a python script where I use the WebDriver get() method, which is known to freeze sometimes, without raising any exception.
This is a well known issue of selenium module, as you can read below:
discussion 1
discussion 2
I tried many of the solutions on the web (eg. setting the set_page_load_timeout) but had no luck.
So, my only option is to restart the script when it gets stuck on the get() method for a given amount of time.
What is the best way of doing that?
Please find a simplified code below:
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver import Chrome, ChromeOptions
from selenium.webdriver.support.ui import WebDriverWait
TIMEOUT = 10
options = ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_argument(f'user-data-dir={DATA["user_data_dir"]}')
chrome_driver = ChromeDriverManager().install()
driver = Chrome(service=Service(chrome_driver), options=options)
# driver.set_page_load_timeout(TIMEOUT)
driver.maximize_window()
wait = WebDriverWait(driver, TIMEOUT)
driver.get('https://www.stackoverflow.com/')

Wait for cookie with Selenium python webdriver

I basically here the same question as here, only with Python.
I want to use Selenium to get the user to log in with a browser, then grab the session cookie that will contain the log in information, and finally close the browser. In an interactive session, it's easy to just wait until the authentication is complete before calling get_cookie(). But in run mode, I'm stuck.
From the Selenium documentation, I get that it is about defining a wait strategy. So I tried to repurpose their code example as follows:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
driver.get('https://examplecom/')
sessionId = WebDriverWait(driver, timeout=120).until(lambda d: d.get_cookie('session_cookie')['value'])
driver.quit()
Unfortunately, the following error is immediately raised:
TypeError: 'NoneType' object is not subscriptable
What should I do instead?

Duh! I was calling the value key as part of the wait function, thus even before it was available! This works:
sessionId = WebDriverWait(driver, timeout=120).until(lambda d: d.get_cookie('TK'))['value']

To save cookies in Selenium, you can do as follows:
import os
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
expanduser = os.path.expanduser('~')
options = webdriver.ChromeOptions()
and then add option of user-data-dir:
options.add_argument(f'--user-data-dir={expanduser}\\AppData\\Local\\Google\\Chrome\\any_Name_You_Want')
and then create your browser with these option:
browser = webdriver.Chrome(ChromeDriverManager().install(), options=options)

Not start Chromium with headless mode from selenium

Lately one of my python selenium script running on Raspi4 linux stops working properly. The same code worked well before and I'm trying to figure out the reason.
The example code below works without headless option, but it is stuck in starting Chromium. Does anyone encounter similar issue?
Chromium and chromedriver version is 97.0.4692.99.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
url = 'https://www.google.com'
options = Options()
options.add_argument('headless')
#options.add_argument('-headless')
#options.add_argument('--headless')
#options.headless = True
service = ChromeService(executable_path='C:\selenium\driver\chromedriver.exe')
chrome = webdriver.Chrome(service=service, options=options)
chrome.get(url)
chrome.save_screenshot('xx.png')
chrome.close()

Instead of using --headless argument, you need to use the headless property as follows:
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
Reference
You can find a couple of relevant detailed discussion in:
DeprecationWarning: use setter for headless property instead of set_headless opts.set_headless(headless=True) using Geckodriver and Selenium in Python
How to make Firefox headless programmatically in Selenium with Python?
How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?

Capture console.log outputs on Chrome console using selenium

I have a list of websites which I am doing some testing and experimentation on. I visit a website from the list using selenium and inject a piece of JS into some of the files using MITMproxy scripting. This injected code performs some test and output the results using console.log() in JS onto the chrome console, something like
console.log(results of the injected JS)
The injection is successful and the results which I desire do appear on the Chrome Console when I run my experiment. The issue that I am facing is when I try to capture the chrome console for console.log output, it is not successful. It will capture warning and error message from chrome console but not console.log output. Currently this how I am doing it.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException, TimeoutException, StaleElementReferenceException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.by import By
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
option = Options()
option.add_argument('start-maximized')
# Specify the proxy
option.add_argument('--proxy-server=%s:%s' % (proxy_host, proxy_port))
# enable browser logging
d = DesiredCapabilities.CHROME
# d['loggingPrefs'] = { 'browser':'ALL' }
d['goog:loggingPrefs'] = { 'browser':'ALL' }
# Launch Chrome.
driver = webdriver.Chrome(options=option, executable_path = './chromedriver', desired_capabilities=d, service_args=["--verbose", "--log-path=./js_inject/qc1.log"])
for url in list_urls:
# Navigate to the test page
driver.get(url)
sleep(15)
# in this 15 seconds, the MITMproxy will inject the code and the injected code will output on chrome console.
for entry in driver.get_log('browser'):
print(entry)
Can anyone point me what mistake I might be making or an alternate approach to perform this task. Thank you.
P.S Pardon me for grammatical errors.

options.add_experimental_option('excludeSwitches', ['enable-logging'])
dc = DesiredCapabilities.CHROME
dc["goog:loggingPrefs"] = {"browser":"INFO"}
self.driver = webdriver.Chrome(chrome_options=options, desired_capabilities=dc)
I managed to make it work a while ago, so not sure what exactly did it, but I think it was the experimental option of "anable-logging".

Python Selenium Can't locate google sheet element [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 2 years ago.
Improve this question
I have tried all ways possible to locate the bellow import file window elements from google sheets and yet haven't locate it. Here is the code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
import time
import pynput
from pynput.keyboard import Key, Controller
keyboard = Controller()
import os
import logging
import glob
#save chrome user data to chromedriver
options = webdriver.ChromeOptions()
options.add_argument("") #Path to your chrome profile
driver = webdriver.Chrome(executable_path="C:\\Program Files (x86)\\chromedriver.exe", chrome_options=options)
#open the google sheet in chrome
driver.get("https://docs.google.com/spreadsheets/d/")
wait3 = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "docs-file-menu")))
#click on file menue
wait3 = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "docs-file-menu")))
driver.find_element_by_id("docs-file-menu").click()
#click on import
wait3 = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.ID, ":4q")))
driver.find_element_by_id(":4q").click()
Google Sheet import Window

The Element which you are looking for is in the iframe.
First, you have to switch to that iframe and then start searching for your element.