Develop Reference

Unable to Open .htm link with Selenium

I cannot open the the link described in the picture with selenium.
I have tried to find element by css_selector, link, partial link, xpath. Still no success, program shows no error, but does not click the last link. Here is the picture from the inspect code from the sec website. Picture of Inspect Code. The line of code that wants to open this is in bold.
from bs4 import BeautifulSoup as soup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
PATH = "C:\Program Files (x86)\Misc Programs\chromedriver.exe"
stock = 'KO'
#stock = input("Enter stock ticker: ")
browser = webdriver.Chrome(PATH)
#First SEC search
sec_url = 'https://www.sec.gov/search/search.htm'
browser.get(sec_url)
tikSearch = browser.find_element_by_css_selector('#cik')
tikSearch.click()
tikSearch.send_keys(stock)
Sclick = browser.find_element_by_css_selector('#searchFormDiv > form > fieldset > span > input[type=submit]')
Sclick.click()
formDesc = browser.find_element_by_css_selector('#seriesDiv > table > tbody > tr:nth-child(2) > td:nth-child(1)')
print(formDesc)
doc = browser.find_element_by_css_selector('#documentsbutton')
doc.click()
##Cannot open file
**form = browser.find_element_by_xpath('//*[#id="formDiv"]/div/table/tbody/tr[2]/td[3]/a')
form.click()**
uClient = uReq(sec_url)
page_html = uClient.read()```

On Firefox this worked and got https://www.sec.gov/Archives/edgar/data/21344/000002134421000018/a20201231crithrifplan.htm
Pasting that into Chrome directly also works.
But in the script, it indeed did not open and left one stuck at:
https://www.sec.gov/Archives/edgar/data/21344/000002134421000018/0000021344-21-000018-index.htm
where, oddly, clicking on the link by hand works in the browser that Selenium launched.
It's better with a wait, but if I put time.sleep(5) before your
form = browser.find_element_by_xpath('//*[#id="formDiv"]/div/table/tbody/tr[2]/td[3]/a')
it opens in Chrome.
EDIT: And here it is done properly with no sleep:
wait = WebDriverWait(browser, 20)
wait.until(EC.presence_of_element_located((By.XPATH, '//*[#id="formDiv"]/div/table/tbody/tr[2]/td[3]/a'))).click()
This assumes you have the imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Possibly useful addition:
I am surprised there is no Selenium Test Helper out there with methods that wrap in some bulletproofing (or maybe there are and I do not know), like what Hetzner Cloud did in its Protractor Test Helper. So I wrote my own little wrapper method for the click (also for send keys, which calls this one). If it's useful to you or readers, enjoy. It could be enhanced to build in retries or take the wait time or whether to scroll the field into the top or bottom of the window (or at all) as parameters. It is working in my context as is.
def safe_click(driver, locate_method, locate_string):
"""
Parameters
----------
driver : webdriver
initialized browser object
locate_method : Locator
By.something
locate_string : string
how to find it
Returns
-------
WebElement
returns whatever click() does.
"""
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((locate_method, locate_string)))
driver.execute_script("arguments[0].scrollIntoView(false);",
driver.find_element(locate_method, locate_string))
return wait.until(EC.element_to_be_clickable((locate_method, locate_string))).click()
If you use it, then the call (which I just tested and it worked) would be:
safe_click(browser, By.XPATH, '//*[#id="formDiv"]/div/table/tbody/tr[2]/td[3]/a')
You could be using it elsewhere, too, but it does not seem like there is a need.

Capture console.log outputs on Chrome console using selenium

I have a list of websites which I am doing some testing and experimentation on. I visit a website from the list using selenium and inject a piece of JS into some of the files using MITMproxy scripting. This injected code performs some test and output the results using console.log() in JS onto the chrome console, something like
console.log(results of the injected JS)
The injection is successful and the results which I desire do appear on the Chrome Console when I run my experiment. The issue that I am facing is when I try to capture the chrome console for console.log output, it is not successful. It will capture warning and error message from chrome console but not console.log output. Currently this how I am doing it.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException, TimeoutException, StaleElementReferenceException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.by import By
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
option = Options()
option.add_argument('start-maximized')
# Specify the proxy
option.add_argument('--proxy-server=%s:%s' % (proxy_host, proxy_port))
# enable browser logging
d = DesiredCapabilities.CHROME
# d['loggingPrefs'] = { 'browser':'ALL' }
d['goog:loggingPrefs'] = { 'browser':'ALL' }
# Launch Chrome.
driver = webdriver.Chrome(options=option, executable_path = './chromedriver', desired_capabilities=d, service_args=["--verbose", "--log-path=./js_inject/qc1.log"])
for url in list_urls:
# Navigate to the test page
driver.get(url)
sleep(15)
# in this 15 seconds, the MITMproxy will inject the code and the injected code will output on chrome console.
for entry in driver.get_log('browser'):
print(entry)
Can anyone point me what mistake I might be making or an alternate approach to perform this task. Thank you.
P.S Pardon me for grammatical errors.

options.add_experimental_option('excludeSwitches', ['enable-logging'])
dc = DesiredCapabilities.CHROME
dc["goog:loggingPrefs"] = {"browser":"INFO"}
self.driver = webdriver.Chrome(chrome_options=options, desired_capabilities=dc)
I managed to make it work a while ago, so not sure what exactly did it, but I think it was the experimental option of "anable-logging".

no such element: Unable to locate element using chromedriver and Selenium in production environment

I have a problem with selenium chromedriver which I cannot figure out what's causing it. Some weeks ago everything was working OK, and suddenly this error started to show up.
The problem is coming from the following function.
def login_(browser):
try:
browser.get("some_url")
# user credentials
user = browser.find_element_by_xpath('//*[#id="username"]')
user.send_keys(config('user'))
password = browser.find_element_by_xpath('//*[#id="password"]')
password.send_keys(config('pass'))
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
login.send_keys("\n")
time.sleep(1)
sidebar = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/a')
sidebar.send_keys("\n")
app_submit = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/ul/li[1]/a')
app_submit.send_keys("\n")
except TimeoutException or NoSuchElementException:
raise LoginException
This function works with no problem in the development environment (macOS 10.11), but throws the following error in the production environment:
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="sidebar"]/ul/li[1]/a"}
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee97XXX),platform=Linux 4.4.0-116-generic x86_64)
I already updated both Chrome and chromedriver (v67 & 2.40, respectively) in each environment. I also gave it more time.sleep(15). But the problem persists. My latest guess is that maybe the initialization of the webdriver is not working properly:
def initiate_webdriver():
option = webdriver.ChromeOptions()
option.binary_location = config('GOOGLE_CHROME_BIN')
option.add_argument('--disable-gpu')
option.add_argument('window-size=1600,900')
option.add_argument('--no-sandbox')
if not config('DEBUG', cast=bool):
display = Display(visible=0, size=(1600, 900))
display.start()
option.add_argument("--headless")
else:
option.add_argument("--incognito")
return webdriver.Chrome(executable_path=config('CHROMEDRIVER_PATH'), chrome_options=option)
Because, if the Display is not working, then there may not be the mentioned sidebar but some other button.
So my questions are: does anybody have had a similar issue? Is there a way to know what is the page showing at the time the driver is looking for such an element?

It's report that the element not found error after you supplying the login , so I think the login failed and the page redirected to somewhere. You can use screenshot option to take a screenshot of the page and then see which page the driver load.
driver.save_screenshot("path to save screen.jpeg")
Also you can save the raw html code and inspect the same page.
Webdriver Screenshot
Using Selenium in Python to save a webpage on Firefox

A couple of things as per the login_(browser) method:
As you have identified the Login button through:
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
I would suggest rather invoking send_keys("\n") take help of the onclick() event through login.click() to mock the clicking of Login button as follows:
login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
login.click()
Next when you identify the sidebar induce WebDriverWait for the element to be clickable as follows:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="sidebar"]/ul/li[1]/a'))).click()
As you mentioned your code code block works perfect in macOS 10.11 environment but throws the following error in the production environment (Linux) it is highly possible that different browsers renders the HTML DOM differently in different OS architecture. So instead of absolute xpath you must use relative xpath as follows:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#attribute='value']"))).click()
A couple of things as per the initiate_webdriver() method:
As per Getting Started with Headless Chrome the argument --disable-gpu is applicable only for Windows but not a valid configuration for Linux OS. So need o remove:
option.add_argument('--disable-gpu')
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Whenever I encounter strange issues in Selenium like this, I prefer retrying to find the particular element which is causing intermittent troubles. One way is to wrap it around a try-except block:
try:
sidebar = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/a')
except NoSuchElementException:
time.sleep(10)
print("Unable to find element in first time, trying it again")
sidebar = browser.find_element_by_xpath('//*[#id="sidebar"]/ul/li[1]/a')
You could also put the try code in a loop with a suitable count variable to make the automation code work. (Check this). In my experience with JAVA, this idea has resolved multiple issues.

You need to wait until the element is visible or else you will get this error. Try something like this:
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support.expected_conditions import visibility_of_element_located
from selenium.webdriver.common.by import By
TIMEOUT = 5
...
xpath = '//*[#id="sidebar"]/ul/li[1]/a'
WebDriverWait(self.selenium, TIMEOUT).until(visibility_of_element_located((By.XPATH, xpath)))
browser.find_element_by_xpath(xpath)
...

Python - How to Disable an Extension, After Opening a Chrome Window, with Selenium

I am trying to disable AdBlock for a specific website only, but I can't find a way to do it. I tried looking in the selenium documentation, but I couldn't find any methods to disable extensions afterward. However, I am still pretty new at reading documentation, so I may have missed something. I also tried to automate the disabling of the AdBlock extension by using selenium but it didn't work. The plan was to go to the extension section of chrome(chrome://extensions/), get the "enabled" checkbox and click it without my intervention. Here is my attempt:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
def main():
opening = True
while opening:
try:
chrome_options = Options()
#Path to AdBlock
chrome_options.add_extension('/usr/local/bin/AdBlock_v.crx')
driver = webdriver.Chrome(chrome_options=chrome_options)
except:
print('An unkown error has occured. Trying again...')
else:
opening = False
disable_adblocker(driver)
def click_element(driver, xpath, index):
getting = True
not_found_times = 0
while getting:
try:
getting = False
element = WebDriverWait(driver, 5).until(
EC.presence_of_all_elements_located((By.XPATH,xpath)))[index]
element.click()
#driver.get(element.get_attribute("href"))
#In case the page does not load properly
except TimeoutException:
not_found_times += 1
if not_found_times < 2:
driver.refresh()
getting = True
else:
raise
#In case DOM updates which makes elements stale
except StaleElementReferenceException:
getting = True
def disable_adblocker(driver):
driver.get('chrome://extensions')
ad_blocker_xpath = '//div[#id="gighmmpiobklfepjocnamgkkbiglidom"]//div[#class="enable-controls"]//input'
click_element(driver,ad_blocker_xpath,0)
print('')
main()
The reason my attempt failed is because selenium couldn't use the xpath, I specified, to get the checkbox element. I believe the path is correct.
The only solution that I can think of is creating two chrome windows: one with AdBlock and another without AdBlock. However, I don't want two windows as this will make things more complicated.

It doesn't look like this is possible using any settings in selenium. However... You can automate adding the domain you wish to exclude after creating the driver.
Before your test actually starts, but after you've initialized the browser, navigate to chrome-extension://[your AdBlock extention ID]/options.html. AdBlock extension ID is unique to the crx file. So go into chrome and find the value in the extension manager. For example, mine is gighmmpiobklfepjocnamgkkbiglidom.
After you've navigated to that page, click 'Customize', then 'Show ads everywhere except for these domains...', then input the domain into the field, then click 'OK'. Boom! Now the domain is added and will show ads! Just make sure
I know its not the ideal quick, easy, one line of code solution... But it seems like the best option, unless you want to go digging in the local storage files and find where this data is added to...

When writing tests with selenium how can you view the resulting HTML?

Take for example the following selenium test in python:
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
def test_search_in_python_org(self):
driver = self.driver
driver.get("http://www.python.org")
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("selenium")
elem.send_keys(Keys.RETURN)
self.assertIn("Google", driver.title)
def tearDown(self):
self.driver.close()
if __name__ == "__main__":
unittest.main()
Taken from: http://selenium-python.readthedocs.org/en/latest/getting-started.html#id2
The resulting output is something like:
----------------------------------------------------------------------
Ran 1 test in 15.566s
OK
Is there any way to get selenium to output the html after it has executed its browser actions?
Basically, I am using Selenium IDE for Firefox to record actions on the browser. I want to play them back on python, get the resulting html, and based on that html take further action (e.g. the 1st selenium test might be to log on to a website and navigate somewhere. Based on what is there I want to conduct a second test (w. the user still logged on)). Is this possible using selenium?
Thanks in Advance!

It sounds as though your tests might end up being dependant on each other, which is a very very bad idea.
Nonetheless, the page_source function will return the full HTML of the current page the driver is looking at:
https://code.google.com/p/selenium/source/browse/py/selenium/webdriver/remote/webdriver.py#429