I am trying to automate some web scrapping in chrome
Previously I was trying it with a different method where the chrome used to open in guest mode and it was working, here is the previous method
url = 'https://stackoverflow.com/'#Example chrome = webdriver.Chrome(ChromeDriverManager().install())
chrome.get(url)
But when I shifted the method, where the chrome would open logged in by default, it is showing problems, there is no error, but driver.get(url) is not working, infact any lines after webdriver.Chrome... are not working
Here is the code after changes
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument(r"user-data-dir=C:\\Users\user\AppData\Local\Google\Chrome\User Data\Default")
url = "https://www.geeksforgeeks.org/"
driver = webdriver.Chrome(executable_path=r"C:\\Users\user\AppData\Local\Google\Chrome\Application\chrome.exe", options=options)
print("h")
driver.get(url)
This is where you made a mistake.
driver = webdriver.Chrome(executable_path=r"C:\\Users\user\AppData\Local\Google\Chrome\Application\chrome.exe", options=options)
you need to pass the PATH of the chrome driver Not the chrome browser installed path.
if you not sure path and your chrome browser compatibility then use below code.
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
Related
webdriver_manager site has a code to start webdriver with brave, but instead of brave it opens it with google chrome. My selenium version 4.6.0 gave the following code for selenium 4 (I also tried the given codes for selenium 3) as it can be seen on the site, but the webdriver still opens with chrome
# selenium 4
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as BraveService
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.utils import ChromeType
driver = webdriver.Chrome(service=BraveService(ChromeDriverManager(chrome_type=ChromeType.BRAVE).install()))
driver.get("https://pypi.org/project/webdriver-manager/")
Okay, so you just want to start a driver with Brave browser instead of chrome? Here is how I do that simplely, keep in mind I'm on Mac. You need the binaryPATH to the application.
add_block_ext = "Path to .crx extension"
driverPath = 'chromedriver'
binaryPath = '/Applications/Brave Browser.app/Contents/MacOS/Brave Browser'
options = webdriver.ChromeOptions()
options.binary_location = binaryPath
options.add_extension(add_block_ext)
browser = webdriver.Chrome(executable_path=binaryPath, chrome_options=options)
browser.get("https://www.google.com")
I am sucesfully able to load my chrome profile using the flags:
user-data-dir as well as profile-directory, yet once the profile is loaded and the chrome window is actually open, no webpage appears. It simply gets stuck on a blank screen.
When I remove the code for the profile it is actually able to open the webpage stored in the login-url variable.
Tried updating to latest version of chrome (94.0.4606.81) and I also used the exact steps listed here to ensure I have the right chrome driver version.
I also did the obvious like making sure there are not any instances of chrome running in the background.
Code is as follows:
import os
from os.path import exists
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
headless = False
login_url = "https://google.com)"
def startChrome():
global headless
try:
chrome_options = Options()
if headless:
chrome_options.add_argument("--headless")
chrome_options.add_argument("----user-data-dir=C:/Users/ERIK/AppData/Local/Google/Chrome/User Data")
chrome_options.add_argument("--profile-directory=Profile 1")
global driver
driver = webdriver.Chrome(path+"/chromedriver.exe", options=chrome_options)
except:
print("Failed to start Chrome!")
input()
exit()
startChrome()
driver.get(login_url)
input()
The following successfully opens google.com for me.
Selenium Version 3.141.0
ChromeDriver Version 94.0.4606.61
Chrome Version 94.0.4606.71
from os import path
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
def getDriver(profile_directory, headless = False):
chrome_options = Options()
if headless:
chrome_options.add_argument("--headless")
userDataDir = path.expandvars(r"%LOCALAPPDATA%\Google\Chrome\User Data")
chrome_options.add_argument(f"--user-data-dir={userDataDir}")
chrome_options.add_argument(f"--profile-directory={profile_directory}")
return webdriver.Chrome("./chromedriver.exe", options=chrome_options)
driver = getDriver("Profile 2")
driver.get("https://google.com")
This code works for Windows where it launches Chrome connected via Tor. Keep in mind you have to have Tor browser running beforehand. How can I enable the user-profile and start the browser logged in? I have tried the regular method. I have only 1 profile. Default. Doesn't seem to be working. Any clues?
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
tor_proxy = "127.0.0.1:9150"
chrome_options = Options()
'''chrome_options.add_argument("--test-type")'''
chrome_options.add_argument('--ignore-certificate-errors')
'''chrome_options.add_argument('--disable-extensions')'''
chrome_options.add_argument('disable-infobars')
'''chrome_options.add_argument("--incognito")'''
chrome_options.add_argument('--user-data=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data\\Default')
chrome_options.add_argument('--proxy-server=socks5://%s' % tor_proxy)
driver = webdriver.Chrome(executable_path='C:\\chromedriver.exe', options=chrome_options)
driver.get('https://www.gmail.com')
time.sleep(4)
driver.switch_to.frame(0)
driver.find_element_by_id("introAgreeButton").click()
Use this instead.
chrome_options.add_argument("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data")
you don't have to specify the profile directory (Default) as it is used by default if nothing is explicitly specified using below code. SO in your case use only the above line of code
chrome_options.add_argument("profile-directory=Profile 1")
Eg:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument(r"user-data-dir=C:\Users\prave\AppData\Local\Google\Chrome\User Data")
driver =webdriver.Chrome("./chromedriver.exe",options=options)
driver.get(url)
Output:
How can I make webdriver_manager.chrome to use a custom chrome user profile?
I know for selenium webdriver I can specify like this:
options = Options()
options.add_argument(f'user-data-dir={script_path}\\User Data\\profile')
driver = webdriver.Chrome(executable_path=f'{script_path}\\chromedriver.exe', options=options)
But since I want to let chromedriver install the correct version on its own (because I sell my program to non python users) I am using the webdriver_manager module which looks like this:
driver = webdriver.Chrome(ChromeDriverManager().install())
Is there any way I could load a custom profile so that my log in data on websites is getting saved in my profile while using webdriver_manager?
You can use webdriver_manager.chrome and custom chrome user profile simultanously using the following solution:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
options = Options()
options.add_argument(f'user-data-dir={script_path}\\User Data\\profile')
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options)
driver.get('https://www.google.com/')
I started having issues running a python script which uses selenium and chrome driver, so I want to disable extensions of chrome driver using python without losing the path where the driver is located.
Currently I have the below:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
path_to_Ie = 'C:\\Python34\\ChromeDriver\\ChromeDriver.exe'
browser = webdriver.Chrome(executable_path = path_to_Ie)
url = 'https://wwww.test.com/'
browser.get(url)
and I would like to add the below lines:
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
browser = webdriver.Chrome(chrome_options=chrome_options)
Any idea how I can merge both codes to make it work properly?
Thanks a lot.
Just provide multiple keyword arguments:
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
browser = webdriver.Chrome(executable_path=path_to_Ie, chrome_options=chrome_options)