Develop Reference

How to get cookies saved in browser with python and Selenium

I'm getting to use the cookie that is already in the browser and use it in Selenium, I know you can't use it using Selenium only, but is there any library I can use to save cookies in json in a variable and use it in Selenium? How can I extract the cookie saved in the browser with python? not only Chrome but others also preferably.
This is my code currently:
option = Options()
option.add_argument("--no-sandbox")
driver = webdriver.Chrome(options=option)
driver.get("https://google.com")
wait = WebDriverWait(driver, 5)
How can I get the cookie from the browser, save it in json format and use it with Selenium?

import pickle
import os
from selenium import webdriver
import time
option = webdriver.ChromeOptions()
option.add_argument("--no-sandbox")
driver = webdriver.Chrome(options=option)
driver.get("https://google.com")
time.sleep(5)
if os.path.exists('cookies.pkl'):
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
driver.add_cookie(cookie)
driver.refresh()
sleep(5)
pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb"))
pickle will help you save and add cookies. But, be sure to add them to the correct domain otherwise you might get errors.

bypass cookiewall selenium

I would like to scrape job listings from a Dutch job listings website. However, when I try to open the page with selenium I run into a cookiewall (new GDPR rules). How do I bypass the cookiewall?
import selenium
#launch url
url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO"
# create a new Firefox session
driver = webdriver.Firefox()
driver.implicitly_wait(30)
driver.get(url)
Edit I tried something
import selenium
import pickle
url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO"
driver = webdriver.Firefox()
driver.set_page_load_timeout(20)
driver.get(start_url)
pickle.dump(driver.get_cookies() , open("NVBCookies.pkl","wb"))
after that loading the cookies did not work
for cookie in pickle.load(open("NVBCookies.pkl", "rb")):
driver.add_cookie(cookie)
InvalidCookieDomainException: Message: Cookies may only be set for the current domain (cookiewall.vnumediaonline.nl)
It looks like I don't get the cookies from the cookiewall, correct?

Instead of bypassing why don't you write code to check if it's present then accept it otherwise continue with next operation. Please find below code for more details
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Chrome(executable_path="C:\\Users\\USER\\Downloads\\New folder (2)\\chromedriver_win32\\chromedriver.exe")
def test_search_in_python_org(self):
driver = self.driver
driver.get("https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO")
elem = driver.find_element_by_xpath("//div[#class='article__button']//button[#id='form_save']")
elem.click()
def tearDown(self):
self.driver.close()
if __name__ == "__main__":
unittest.main()

driver.find_element_by_xpath('//*[#id="form_save"]').click()
ok I made selenium click the accept button. Also fine by me. Not sure if I'll run into cookiewalls later

Selenium not storing cookies

I'm working on trying to automate a game I want to get ahead in called pokemon vortex and when I login using selenium it works just fine, however when I attempt to load a page that requires a user to be logged in I am sent right back to the login page (I have tried it outside of selenium with the same browser, chrome).
This is what I have
import time
from selenium import webdriver
from random import randint
driver = webdriver.Chrome(r'C:\Program Files (x86)\SeleniumDrivers\chromedriver.exe')
driver.get('https://zeta.pokemon-vortex.com/dashboard/');
time.sleep(5) # Let the user actually see something!
usernameLoc = driver.find_element_by_id('myusername')
passwordLoc = driver.find_element_by_id('mypassword')
usernameLoc.send_keys('mypassword')
passwordLoc.send_keys('12345')
submitButton = driver.find_element_by_id('submit')
submitButton.submit()
time.sleep(3)
driver.get('https://zeta.pokemon-vortex.com/map/10')
time.sleep(10)
I'm using python 3.6+ and I literally just installed selenium today so it's up to date, how do I force selenium to hold onto cookies?

Using a pre-defined user profile might solve your problem. This way your cache will be saved and will not be deleted.
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--user-data-dir=C:/Users/user_name/AppData/Local/Google/Chrome/User Data")
driver = webdriver.Chrome(options=options)
driver.get("xyz.com")

Trying to open a tab in my opened browser with selenium

I have written a small python script with selenium to search Google and open the first link but whenever I run this script, it opens a console and open a new Chrome window and run this script in that Chrome window.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import pyautogui
def main():
setup()
# open Chrome and open Google
def setup():
driver = webdriver.Chrome(r'C:\\python_programs'+
'(Starting_out_python)'+
'\\chromedriver.exe')
driver.get('https://www.google.com')
assert 'Google' in driver.title
mySearch(driver)
#Search keyword
def mySearch(driver):
search = driver.find_element_by_id("lst-ib")
search.clear()
search.send_keys("Beautiful Islam")
search.send_keys(Keys.RETURN)
first_link(driver)
#click first link
def first_link(driver):
link = driver.find_elements_by_class_name("r")
link1 = link[0]
link1.click()
main()
How can I open this in the same browser I am using?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
def main():
setup()
# open Chrome and open Google
def setup():
driver = webdriver.Chrome()
driver.get('https://www.google.com')
assert 'Google' in driver.title
mySearch(driver)
#Search keyword
def mySearch(driver):
search = driver.find_element_by_id("lst-ib")
search.clear()
search.send_keys("test")
search.send_keys(Keys.RETURN)
first_link(driver)
#click first link
def first_link(driver):
link = driver.find_elements_by_xpath("//a[#href]")
# uncomment to see each href of the found links
# for i in link:
# print(i.get_attribute("href"))
first_link = link[0]
url = first_link.get_attribute("href")
driver.execute_script("window.open('about:blank', 'tab2');")
driver.switch_to.window("tab2")
driver.get(url)
# Do something else with this new tab now
main()
A few observation: the first link you get might not be the first link you want. In my case, the first link is the login to Google account. So you might want to do some more validation on it until you open it, like check it's href property, check it's text to see if it matches something etc.
Another observation is that there are easier ways of crawling google search results and using googles API directly or a thirdparty implementation like this: https://pypi.python.org/pypi/google or https://pypi.python.org/pypi/google-search

To my knowledge, there's no way to attach Selenium to an already-running browser.
More to the point, why do you want to do that? The only thing I can think of is if you're trying to set up something with the browser manually, and then having Selenium do things to it from that manually-set-up state. If you want your tests to run as consistently as possible, you shouldn't be relying on a human setting up the browser in a particular way; the script should do this itself.

How to refresh an already opened web page

I just want to refresh an already opened web page with Selenium.
It always opens a new browser window.
What I'm doing wrong?
from selenium import webdriver
import urllib
import urllib2
driver = webdriver.Firefox()
driver.refresh()

I would suggest binding the driver element search to the tag body and use the refresh command of the browser.
In OSX for example
driver.find_element_by_tag_name('body').send_keys(Keys.COMMAND + 'r')
Documentation on keys here: http://selenium-python.readthedocs.org/en/latest/api.html
Update:
The following code, very similar to your one, works fine for me.
driver = webdriver.Firefox()
driver.get(response.url) #tested in combination with scrapy
time.sleep(3)
driver.refresh()
Are you sure you correctly load the web page with the driver before refreshing it ?

The problem is you are opening the webdriver and then trying to refresh when you have not specified a URL.
All you need to do is get your desired URL before refreshing:
from selenium import webdriver
import urllib
import urllib2
driver = webdriver.Firefox()
driver.get("Your desired URL goes here...")
#now you can refresh the page!
driver.refresh()

The following codes work for me
driver.get(driver.current_url)
sleep(2)
driver.refresh()
I use python 3.7.6, selenium 3.141.0

You are trying to refresh the page before it loads so u can use a sleep function
from time import sleep
sleep(1)
or you can wait for an XPath to load so
WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, xpath goes here)))

For me helped
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.get("URL")
time.sleep(5)
driver.refresh()

I got mine fixed by adding "browser.refresh()" the for loop or while loop.

You can try any one of the below methods for the same.
Method 1:
driver.findElement(By.name("s")).sendKeys(Keys.F5);
Method 2:
driver.get(driver.getCurrentUrl());
Method3:
driver.navigate().to(driver.getCurrentUrl());
Method4:
driver.findElement(By.name("s")).sendKeys("\uE035");