How can I save all cookies in Python's Selenium WebDriver to a .txt file, and then load them later?
The documentation doesn't say much of anything about the getCookies function.
You can save the current cookies as a Python object using pickle. For example:
import pickle
import selenium.webdriver
driver = selenium.webdriver.Firefox()
driver.get("http://www.google.com")
pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb"))
And later to add them back:
import pickle
import selenium.webdriver
driver = selenium.webdriver.Firefox()
driver.get("http://www.google.com")
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
driver.add_cookie(cookie)
When you need cookies from session to session, there is another way to do it. Use the Chrome options user-data-dir in order to use folders as profiles. I run:
# You need to: from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("www.google.com")
Here you can do the logins that check for human interaction. I do this and then the cookies I need now every time I start the Webdriver with that folder everything is in there. You can also manually install the Extensions and have them in every session.
The second time I run, all the cookies are there:
# You need to: from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("www.google.com") # Now you can see the cookies, the settings, extensions, etc., and the logins done in the previous session are present here.
The advantage is you can use multiple folders with different settings and cookies, Extensions without the need to load, unload cookies, install and uninstall Extensions, change settings, change logins via code, and thus no way to have the logic of the program break, etc.
Also, this is faster than having to do it all by code.
Remember, you can only add a cookie for the current domain.
If you want to add a cookie for your Google account, do
browser.get('http://google.com')
for cookie in cookies:
browser.add_cookie(cookie)
Just a slight modification for the code written by Roel Van de Paar, as all credit goes to him. I am using this in Windows and it is working perfectly, both for setting and adding cookies:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=chrome-data")
driver = webdriver.Chrome('chromedriver.exe',options=chrome_options)
driver.get('https://web.whatsapp.com') # Already authenticated
time.sleep(30)
Based on the answer by Eduard Florinescu, but with newer code and the missing imports added:
$ cat work-auth.py
#!/usr/bin/python3
# Setup:
# sudo apt-get install chromium-chromedriver
# sudo -H python3 -m pip install selenium
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=chrome-data")
driver = webdriver.Chrome('/usr/bin/chromedriver',options=chrome_options)
chrome_options.add_argument("user-data-dir=chrome-data")
driver.get('https://www.somedomainthatrequireslogin.com')
time.sleep(30) # Time to enter credentials
driver.quit()
$ cat work.py
#!/usr/bin/python3
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=chrome-data")
driver = webdriver.Chrome('/usr/bin/chromedriver',options=chrome_options)
driver.get('https://www.somedomainthatrequireslogin.com') # Already authenticated
time.sleep(10)
driver.quit()
Ideally it would be better to not copy the directory in the first place, but this is very hard, see
How to Prevent Selenium 3.0 (Geckodriver) from Creating Temporary Firefox Profiles?
how do I use an existing profile in-place with Selenium Webdriver?
Also
Can't use existing Firefox profile in Selenium WebDriver using C# (similar solution to the solution below)
This is a solution that saves the profile directory for Firefox (similar to the user-data-dir (user data directory) in Chrome) (it involves manually copying the directory around. I haven't been able to find another way):
It was tested on Linux.
Short version:
To save the profile
driver.execute_script("window.close()")
time.sleep(0.5)
currentProfilePath = driver.capabilities["moz:profile"]
profileStoragePath = "/tmp/abc"
shutil.copytree(currentProfilePath, profileStoragePath,
ignore_dangling_symlinks=True
)
To load the profile
driver = Firefox(executable_path="geckodriver-v0.28.0-linux64",
firefox_profile=FirefoxProfile(profileStoragePath)
)
Long version (with demonstration that it works and a lot of explanation -- see comments in the code)
The code uses localStorage for demonstration, but it works with cookies as well.
#initial imports
from selenium.webdriver import Firefox, FirefoxProfile
import shutil
import os.path
import time
# Create a new profile
driver = Firefox(executable_path="geckodriver-v0.28.0-linux64",
# * I'm using this particular version. If yours is
# named "geckodriver" and placed in system PATH
# then this is not necessary
)
# Navigate to an arbitrary page and set some local storage
driver.get("https://DuckDuckGo.com")
assert driver.execute_script(r"""{
const tmp = localStorage.a; localStorage.a="1";
return [tmp, localStorage.a]
}""") == [None, "1"]
# Make sure that the browser writes the data to profile directory.
# Choose one of the below methods
if 0:
# Wait for some time for Firefox to flush the local storage to disk.
# It's a long time. I tried 3 seconds and it doesn't work.
time.sleep(10)
elif 1:
# Alternatively:
driver.execute_script("window.close()")
# NOTE: It might not work if there are multiple windows!
# Wait for a bit for the browser to clean up
# (shutil.copytree might throw some weird error if the source directory changes while copying)
time.sleep(0.5)
else:
pass
# I haven't been able to find any other, more elegant way.
#`close()` and `quit()` both delete the profile directory
# Copy the profile directory (must be done BEFORE driver.quit()!)
currentProfilePath = driver.capabilities["moz:profile"]
assert os.path.isdir(currentProfilePath)
profileStoragePath = "/tmp/abc"
try:
shutil.rmtree(profileStoragePath)
except FileNotFoundError:
pass
shutil.copytree(currentProfilePath, profileStoragePath,
ignore_dangling_symlinks=True # There's a lock file in the
# profile directory that symlinks
# to some IP address + port
)
driver.quit()
assert not os.path.isdir(currentProfilePath)
# Selenium cleans up properly if driver.quit() is called,
# but not necessarily if the object is destructed
# Now reopen it with the old profile
driver=Firefox(executable_path="geckodriver-v0.28.0-linux64",
firefox_profile=FirefoxProfile(profileStoragePath)
)
# Note that the profile directory is **copied** -- see FirefoxProfile documentation
assert driver.profile.path!=profileStoragePath
assert driver.capabilities["moz:profile"]!=profileStoragePath
# Confusingly...
assert driver.profile.path!=driver.capabilities["moz:profile"]
# And only the latter is updated.
# To save it again, use the same method as previously mentioned
# Check the data is still there
driver.get("https://DuckDuckGo.com")
data = driver.execute_script(r"""return localStorage.a""")
assert data=="1", data
driver.quit()
assert not os.path.isdir(driver.capabilities["moz:profile"])
assert not os.path.isdir(driver.profile.path)
What doesn't work:
Initialize Firefox(capabilities={"moz:profile": "/path/to/directory"}) -- the driver will not be able to connect.
options=Options(); options.add_argument("profile"); options.add_argument("/path/to/directory"); Firefox(options=options) -- same as above.
Try this method:
import pickle
from selenium import webdriver
driver = webdriver.Chrome(executable_path="chromedriver.exe")
URL = "SITE URL"
driver.get(URL)
sleep(10)
if os.path.exists('cookies.pkl'):
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
driver.add_cookie(cookie)
driver.refresh()
sleep(5)
# check if still need login
# if yes:
# write login code
# when login success save cookies using
pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb"))
This is code I used in Windows. It works.
for item in COOKIES.split(';'):
name,value = item.split('=', 1)
name=name.replace(' ', '').replace('\r', '').replace('\n', '')
value = value.replace(' ', '').replace('\r', '').replace('\n', '')
cookie_dict={
'name':name,
'value':value,
"domain": "", # Google Chrome
"expires": "",
'path': '/',
'httpOnly': False,
'HostOnly': False,
'Secure': False
}
self.driver_.add_cookie(cookie_dict)
Use this code to store the login session of any website like google facebook etc
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import undetected_chromedriver as uc
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:/Users/salee/AppData/Local/Google/Chrome/User Data/Profile 1")
browser = uc.Chrome(use_subprocess=True,Options=options)
For my case, the accepted answer is almost there.
For people that have no luck with the answers above, you are welcome to try my way.
Before you start coding, make sure that the website is using cookies for authentication.
Step:
Open your browser (I am using chrome here), login to your website.
Go to this website in order to know how to check the value of the cookies
Open another browser in incognito mode and go to your website (at this stage, your website should still prompt you the login page)
Try to modify the cookies value with the first browser cookies's value accordingly (your first browser must authenticated to your website)
Refresh your incognito mode browser, it should by pass the login page
The steps above is how I used to make sure adding cookies can authenticate to my website.
Now is the coding part, it is almost the same as the accepted answer. The only problem for me with the accepted answer is that I ended up having double the numbers of my cookies.
The pickle.dump part has no issue for me, so I would straight to the add cookie part.
import pickle
import selenium.webdriver
driver = selenium.webdriver.Chrome()
driver.get("http://your.website.com")
cookies = pickle.load(open("cookies.pkl", "rb"))
# the reason that I delete the cookies is because I found duplicated cookies by inspect the cookies with browser like step 2
driver.delete_all_cookies()
for cookie in cookies:
driver.add_cookie(cookie)
driver.refresh()
You are able to use step 2 to check if the cookies you add with the code is working correctly.
Hope it helps.
Related
I want to open a browser that is logged in to my gmail account like my default browser in selenium. Is there a way to do this?
edit:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
w = webdriver.Chrome(executable_path="C:\\Users\\chromedriver.exe", chrome_options=options)
this method does not work for me.
Chrome Options
Chrome options are very particular on how you call them. Syntax needs to be followed with perfection. Correct your below syntax errors by changing your old code:
options.add_argument("user-data-dir=C:\\Path")
To the following code:
userdatadir = 'C:/Users/<user>/AppData/Local/Google/Chrome/User Data'
chromeOptions.add_argument(f"--user-data-dir={userdatadir}")
You can use your profile that you use to open browser:
from selenium import webdriver
profile = webdriver.FirefoxProfile('/home/user/.mozilla/firefox/xxxx-release/') # this work for linux, if you use windows, you can find this file in local AppData
browser = webdriver.Firefox(executable_path='./geckodriver', firefox_profile =profile)
This will use your standard profile that you use with your standard browser, including logins, cookies etc.
Edit:
I used firefox here, but the same principle works with chrome as well.
options = webdriver.ChromeOptions()
options.add_argument(f"user-data-dir={CHROME_USER_DIR}")
options.add_argument("profile-directory=Default")
This will open the Default Profile in Chrome.
The user-data-dir looks something like C:\Users\<username>\AppData\Local\Google\Chrome\User Data
To open a different profile, replace the Default with your profile directory
I'm trying to use Selenium (3.141.0) with ChromeDriver (87.0.4280) to access a page. When accessed manually, it brings me to a policy page (different URL) where you have to hit 'Ok' before continuing to the site. Edit This is using Win 10 and I have the folder with the chromedriver on PATH.
When using the following code, I'm able to get to the policy page with the ("--headless") option but without it I get a blank page with 'data:,' in the URL and nothing else loads. I've tried accessing straight from the policy page and the site URL but they both get stuck when the webdriver is created. Am I missing something? I'm open to any suggestions, thanks!
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver_path = 'D:\....\chromedriver.exe'
driver = webdriver.Chrome(executable_path= driver_path, options= chrome_options)
driver.get(...) # left out the url
This is the output page I get without using ("--headless")
Funny enough, I realized it was because my Chrome Developer tools had become disabled. Not sure how but when I re-enabled them, it worked perfectly again. Weird.
I am trying to acces whatsapp web with python without having to scan the QR code everytime I restart the program (because in my normal browser I also dont have to do that). But how can I do that? Where is the data stored that tells whatsapp web to connect to my phone? And how do I save this data and send it to the browser when I rerun the code?
I already tried this because someone told me I should save the cookies:
from selenium import webdriver
import time
browser = None
cookies = None
def init():
browser = webdriver.Firefox(executable_path=r"C:/Users/Pascal/Desktop/geckodriver.exe")
browser.get("https://web.whatsapp.com/")
time.sleep(5) # in this time I scanned the QR to see if there are cookies
cookies = browser.get_cookies()
print(len(cookies))
print(cookies)
init()
Unfortunately there were no cookies..
The output was 0 and [].
How do I fix this probblem?
As mentioned in the answer to this question, pass your Chrome profile to the Chromedriver in order to avoid this problem. You can do it like this:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
driver = webdriver.Chrome(executable_path="C:\\Users\\chromedriver.exe", options=options)
This one works for me, I just created a folder, on the home directory of the script and a little modifications and it works perfectly.
###########
E_PROFILE_PATH = "user-data-dir=C:\Users\Denoh\Documents\Project\WhatBOts\SessionSaver"
##################
This is the Config File that I will import later
##################
The main script starts here
##################
from selenium import webdriver
from config import E_PROFILE_PATH
options = webdriver.ChromeOptions()
options.add_argument(E_PROFILE_PATH)
driver = webdriver.Chrome(executable_path='chromedriver_win32_86.0.4240.22\chromedriver.exe', options=options)
driver.get('https://web.whatsapp.com/')
I'm working on trying to automate a game I want to get ahead in called pokemon vortex and when I login using selenium it works just fine, however when I attempt to load a page that requires a user to be logged in I am sent right back to the login page (I have tried it outside of selenium with the same browser, chrome).
This is what I have
import time
from selenium import webdriver
from random import randint
driver = webdriver.Chrome(r'C:\Program Files (x86)\SeleniumDrivers\chromedriver.exe')
driver.get('https://zeta.pokemon-vortex.com/dashboard/');
time.sleep(5) # Let the user actually see something!
usernameLoc = driver.find_element_by_id('myusername')
passwordLoc = driver.find_element_by_id('mypassword')
usernameLoc.send_keys('mypassword')
passwordLoc.send_keys('12345')
submitButton = driver.find_element_by_id('submit')
submitButton.submit()
time.sleep(3)
driver.get('https://zeta.pokemon-vortex.com/map/10')
time.sleep(10)
I'm using python 3.6+ and I literally just installed selenium today so it's up to date, how do I force selenium to hold onto cookies?
Using a pre-defined user profile might solve your problem. This way your cache will be saved and will not be deleted.
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--user-data-dir=C:/Users/user_name/AppData/Local/Google/Chrome/User Data")
driver = webdriver.Chrome(options=options)
driver.get("xyz.com")
I'm trying to make a bot in order to reset my router easily, so I'm using mechanize for this task.
import mechanize
br = mechanize.Browser()
br.set_handle_robots(False)
response = br.open("http://192.168.0.1/")
br.select_form(nr=0)
br.form['loginUsername']='support'
br.form['loginPassword']='71689637'
response=br.submit()
if(response.read().find("wifi")) != -1:
# ?????
If it finds the string 'wifi', it means the bot has logged in, but here's where I get stuck, because the restart button is in another tab (Another page, I guess that from the same object indicating the new URL it should be able to follow the redirection URL without logging off). However, the button from that tab is, well, a button but not a form.
Picture 1:
Picture 2:
And here's the source:
https://github.com/SharkiPy/Code-stackoverflow/blob/master/Source
Here is a start of code using Selenium, with hidden browser. You just have to add the actions you take when browsing through your router. I hope it can get you started!
import time
from selenium import webdriver
from selenium.common.exceptions import WebDriverException, NoSuchElementException,InvalidElementStateException,ElementNotInteractableException, StaleElementReferenceException, ElementNotVisibleException
from selenium.webdriver.common.keys import Keys
# There may be some unnecessary import above
from selenium.webdriver.chrome.options import Options
options_chrome = webdriver.ChromeOptions()
options_chrome.binary_location = 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe' # PATH to your chrome driver (you can also use firefox or any other browser, but options below will not be exactly the same
prefs = {"profile.default_content_setting_values.notifications" : 2} # disable notification by default
options_chrome.add_experimental_option("prefs",prefs)
#### below are options for headless chrome
#options_chrome.add_argument('headless')
#options_chrome.add_argument("disable-gpu")
#options_chrome.add_argument("--start-maximized")
#options_chrome.add_argument("--no-sandbox")
#options_chrome.add_argument("--disable-setuid-sandbox")
#### You should uncomment these lines when your code will be working
# starting browser :
browser = webdriver.Chrome( options=options_chrome)
# go to the router page :
browser.get("http://192.168.0.1/")
# connect
elem = browser.find_element_by_id("loginUsername")
elem.send_keys('support')
elem = browser.find_element_by_id("loginPassword")
elem.send_keys('71689637')
elem.send_keys(Keys.RETURN)
# here you need to find your button and click it
button = browser.find_element_by_[Whatever works for you]
button.click()