Take for example the following selenium test in python:
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
def test_search_in_python_org(self):
driver = self.driver
driver.get("http://www.python.org")
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("selenium")
elem.send_keys(Keys.RETURN)
self.assertIn("Google", driver.title)
def tearDown(self):
self.driver.close()
if __name__ == "__main__":
unittest.main()
Taken from: http://selenium-python.readthedocs.org/en/latest/getting-started.html#id2
The resulting output is something like:
----------------------------------------------------------------------
Ran 1 test in 15.566s
OK
Is there any way to get selenium to output the html after it has executed its browser actions?
Basically, I am using Selenium IDE for Firefox to record actions on the browser. I want to play them back on python, get the resulting html, and based on that html take further action (e.g. the 1st selenium test might be to log on to a website and navigate somewhere. Based on what is there I want to conduct a second test (w. the user still logged on)). Is this possible using selenium?
Thanks in Advance!
It sounds as though your tests might end up being dependant on each other, which is a very very bad idea.
Nonetheless, the page_source function will return the full HTML of the current page the driver is looking at:
https://code.google.com/p/selenium/source/browse/py/selenium/webdriver/remote/webdriver.py#429
Related
I have two functions using selenium webdriver. One logs in, and another that performs authenticated actions on the website.
from selenium import webdriver
def login(driver):
driver.get('website.com')
# login
return driver
def do_stuff(driver):
driver.get('website.com/things')
# do stuff
return driver
if __name__ == '__main__':
driver = webdriver.Firefox()
driver = login(driver)
driver = do_stuff(driver)
driver.close()
The first function successfully logs in, but when the second function runs, I get the following error:
selenium.common.exceptions.InvalidSessionIdException: Message: Tried to run command without establishing a connection
I checked the session_id of the driver at the end of the first function and at the beginning of the first function, and they are the same. Also, it should not be an issue with the specific commands in the script, because it works when not split into two functions. I can provide a more specific code sample if it helps.
What am I missing, is there a way to get this to work?
I took your code and made a couple of simple tweaks to adjust with my environment, added print() statement and executed your program and seems it runs perfecto.
selenium4 based code
Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
def login(driver):
driver.get('https://www.selenium.dev/')
print("Within login")
return driver
def do_stuff(driver):
driver.get('https://www.selenium.dev/documentation/')
print("Within do_stuff")
return driver
if __name__ == '__main__':
driver = webdriver.Firefox(service=Service('C:\\BrowserDrivers\\geckodriver.exe'))
driver = login(driver)
driver = do_stuff(driver)
driver.quit()
Console Output:
Within login
Within do_stuff
Update
If you are still facing the InvalidSessionIdException error you need to cross check the compatibility of the versions of the binaries you are using.
You can find a relevant detailed discussion in selenium.common.exceptions.InvalidSessionIdException using GeckoDriver Selenium Firefox in headless mode through Python
I'm running simple selenium test on GitLab CI (free tier).
Sometimes I get failing job because elements where not located (after taking screenshot it appears that page is displayed as white screen).
After running failed job couple of times it starts working. It is very hard to reproduce because it is quite rare.
How can I get more info what kind of issue it is? Some memory problems with runners or other?
Example selenium script:
import unittest
from selenium import webdriver
class SmokeTests(unittest.TestCase):
#classmethod
def setUpClass(self):
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
self.driver = webdriver.Chrome(options=chrome_options)
base_url = 'https://google.com'
self.driver.get(base_url)
#classmethod
def tearDownClass(self):
self.driver.quit()
def test_name(self):
expected_title = 'Google'
observed_title = self.driver.title
self.assertEqual(expected_title, observed_title)
EDIT
How it looks like when running job couple times
Logs contain standard info when title will be not found:
AssertionError: 'Google' != ''
- Google
+
----------------------------------------------------------------------
Ran 1 test in 1.193s
FAILED (failures=1)
tearDown
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1
I experienced this problem on different pages/servers/repositories. Possible issue is lack of memory on shared runners so maybe usage of below option can help:
options.add_argument('--disable-dev-shm-usage')
but is not possible to test it due to randomness of this fail.
CI script:
test staging ui:
image: jaktestowac/python-chromedriver:3.9-selenium
before_script:
- pip install -r tests/requirements.txt
script:
- python -m unittest tests/smoke_tests.py
Image used for testing: https://hub.docker.com/layers/jaktestowac/python-chromedriver/3.9-selenium/images/sha256-5feb585b9ebdac3f5191e7c24f29e8e78beb4f69c9bc734d7050157544f4653f?context=explore
EDIT2
Using wait is not a case here (tested for 20sec). With this random bug the Page is not loaded at all (title element is not available).
Currently I'm trying find some clues with logging :
d = DesiredCapabilities.CHROME
d['goog:loggingPrefs'] = { 'browser':'ALL' }
self.driver = webdriver.Chrome(desired_capabilities=d, options=self.chrome_options)
After taking screenshot it appears that page is displayed as white screen can be interpreted as the webpage didn't render completely when taking of the screenshot was attempted.
Ideally before validating the Page Title you need to induce WebDriverWait for either:
visibility_of_element_located() for any of the visible element within the DOM Tree. As an example, waiting for the Search Box to be visible as follows:
def test_name(self):
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.NAME, "q")))
expected_title = 'Google'
observed_title = self.driver.title
self.assertEqual(expected_title, observed_title)
element_to_be_clickable() for any of the clickable element within the HTML DOM. As an example, waiting for the Search Box to be clickable as follows:
def test_name(self):
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.NAME, "q")))
expected_title = 'Google'
observed_title = self.driver.title
self.assertEqual(expected_title, observed_title)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I would like to scrape job listings from a Dutch job listings website. However, when I try to open the page with selenium I run into a cookiewall (new GDPR rules). How do I bypass the cookiewall?
import selenium
#launch url
url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO"
# create a new Firefox session
driver = webdriver.Firefox()
driver.implicitly_wait(30)
driver.get(url)
Edit I tried something
import selenium
import pickle
url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO"
driver = webdriver.Firefox()
driver.set_page_load_timeout(20)
driver.get(start_url)
pickle.dump(driver.get_cookies() , open("NVBCookies.pkl","wb"))
after that loading the cookies did not work
for cookie in pickle.load(open("NVBCookies.pkl", "rb")):
driver.add_cookie(cookie)
InvalidCookieDomainException: Message: Cookies may only be set for the current domain (cookiewall.vnumediaonline.nl)
It looks like I don't get the cookies from the cookiewall, correct?
Instead of bypassing why don't you write code to check if it's present then accept it otherwise continue with next operation. Please find below code for more details
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Chrome(executable_path="C:\\Users\\USER\\Downloads\\New folder (2)\\chromedriver_win32\\chromedriver.exe")
def test_search_in_python_org(self):
driver = self.driver
driver.get("https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO")
elem = driver.find_element_by_xpath("//div[#class='article__button']//button[#id='form_save']")
elem.click()
def tearDown(self):
self.driver.close()
if __name__ == "__main__":
unittest.main()
driver.find_element_by_xpath('//*[#id="form_save"]').click()
ok I made selenium click the accept button. Also fine by me. Not sure if I'll run into cookiewalls later
I'm doing a "stupid" bot with Python and Selenium to automate some actions on web.telegram.org.
I want stay logged in after the first log on, but when I try to save cookie with function driver.get_cookies() they are empty (I tryed to print them and the output was "[]"). When I did the same thing with other website like youtube.com for example, it works! I tryed also using different webDrivers but I got the same results.
The code is:
from selenium import webdriver
import time
import pickle
driver = webdriver.Firefox()
driver.get('https://web.telegram.org')
time.sleep(4)
phone_number = driver.find_element_by_name("phone_number")
phone_number.send_keys("3478995060")
login_button = driver.find_element_by_class_name("login_head_submit_btn")
login_button.click()
time.sleep(2)
ok_button = driver.find_element_by_xpath("//span[#my-i18n='modal_ok']")
ok_button.click()
time.sleep(30)
all_cookies = driver.get_cookies()
print(all_cookies)
driver.quit()
I have written a small python script with selenium to search Google and open the first link but whenever I run this script, it opens a console and open a new Chrome window and run this script in that Chrome window.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import pyautogui
def main():
setup()
# open Chrome and open Google
def setup():
driver = webdriver.Chrome(r'C:\\python_programs'+
'(Starting_out_python)'+
'\\chromedriver.exe')
driver.get('https://www.google.com')
assert 'Google' in driver.title
mySearch(driver)
#Search keyword
def mySearch(driver):
search = driver.find_element_by_id("lst-ib")
search.clear()
search.send_keys("Beautiful Islam")
search.send_keys(Keys.RETURN)
first_link(driver)
#click first link
def first_link(driver):
link = driver.find_elements_by_class_name("r")
link1 = link[0]
link1.click()
main()
How can I open this in the same browser I am using?
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
def main():
setup()
# open Chrome and open Google
def setup():
driver = webdriver.Chrome()
driver.get('https://www.google.com')
assert 'Google' in driver.title
mySearch(driver)
#Search keyword
def mySearch(driver):
search = driver.find_element_by_id("lst-ib")
search.clear()
search.send_keys("test")
search.send_keys(Keys.RETURN)
first_link(driver)
#click first link
def first_link(driver):
link = driver.find_elements_by_xpath("//a[#href]")
# uncomment to see each href of the found links
# for i in link:
# print(i.get_attribute("href"))
first_link = link[0]
url = first_link.get_attribute("href")
driver.execute_script("window.open('about:blank', 'tab2');")
driver.switch_to.window("tab2")
driver.get(url)
# Do something else with this new tab now
main()
A few observation: the first link you get might not be the first link you want. In my case, the first link is the login to Google account. So you might want to do some more validation on it until you open it, like check it's href property, check it's text to see if it matches something etc.
Another observation is that there are easier ways of crawling google search results and using googles API directly or a thirdparty implementation like this: https://pypi.python.org/pypi/google or https://pypi.python.org/pypi/google-search
To my knowledge, there's no way to attach Selenium to an already-running browser.
More to the point, why do you want to do that? The only thing I can think of is if you're trying to set up something with the browser manually, and then having Selenium do things to it from that manually-set-up state. If you want your tests to run as consistently as possible, you shouldn't be relying on a human setting up the browser in a particular way; the script should do this itself.