Develop Reference

Selenium either Custom Profile or get() method not working

I was trying to control an already open instance of Chrome, but when that didn't work, I started to try to make a new instance of Chrome on my profile. I found two stackoverflow posts detailing the problem (How to use Chrome Profile in Selenium Webdriver Python 3 and Selenium: get() not working with custom google profile) but neither solution worked. I have two code blocks, both of which has its own problem.
The first thing I tried was this:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument(r"--user-data-dir=C:\\Users\\EliBa\\AppData\\Local\\Google\\Chrome\\User Data")
options.add_argument(r'--profile-directory=Default')
driver = webdriver.Chrome(executable_path=r'C:\\Selenium Drivers\\chromedriver.exe', chrome_options=options)
driver.get("https://google.com")
While this opened my profile, it did not open google like I asked it to. The second thing I tried was this:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("user-data-dir=C:\\Users\\EliBa\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
driver = webdriver.Chrome(executable_path='C:\\Selenium Drivers\\chromedriver.exe', options=options)
driver.get("https://www.google.com")
While this opened Google, it did not open my custom profile. Does anyone know what I should be putting so that the custom profile opens AND it loads the site I ask it to?

how to run Selenium ChromeDriver with cookies from actual Chrome installation?

I want to use Selenium(python) to log in to several sites and I need to get all my Chrome cookies,
How should I do this?

This is a duplicate question, anyways:
Do the following:
# Remember to import
from selenium.webdriver.chrome.options import Options
options = Options()
# Your user agent
options.add_argument("YOUR_USER_AGENT")
# Path to chrome application
options.add_argument("CHROME_APPLICATION_PATH")
# Path to chrome driver
driver = webdriver.Chrome(executable_path="CHROME_DRIVER_PATH", options=options)
You will need to do some research about your own system and see what exactly is your user-agent (there are a million site that tell you)
As well as finding the correct path for your chrome application.
Then, proceed normally, this will execute chrome as if you opened it yourself.

Selenium Chromedriver doesn't keep me logged in in some websites

I'm using Selenium and ChromeDriver to scrape data from a website.
I need to keep my account logged in after closing the Driver: for this purpose I use every time the default Chrome profile.
Here you can see my code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
urlpage = 'https://example.com/'
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\\MyName\\AppData\\Local\\Google\\Chrome\\User Data")
driver = webdriver.Chrome(options=options)
driver.get(urlpage)
The problem is that for some websites (e.g. https://projecteuler.net/) it works, so I'm logged in also the following session, but for other (like https://www.fundraiso.ch, the one I need) it doesn't, although in the "normal" browser I'm still logged in after I close the window.
Does anyone know how to fix this problem?
EDIT:
I didn't mention that I can't automate the login because the website has a maximum login number, and if I breach it the website will block my account.

Selenium is not loading TikTok pages

I'm implementing a TikTok crawler using selenium and scrapy
start_urls = ['https://www.tiktok.com/trending']
....
def parse(self, response):
options = webdriver.ChromeOptions()
from fake_useragent import UserAgent
ua = UserAgent()
user_agent = ua.random
options.add_argument(f'user-agent={user_agent}')
options.add_argument('window-size=800x841')
driver = webdriver.Chrome(chrome_options=options)
driver.get(response.url)
The crawler open Chrome but it does not load videos.
Image loading
The same problem happens also using Firefox
No loading page using Firefox
The same problem using a simple script using Selenium
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()
driver = webdriver.Chrome()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

Did u try to navigate further within the selenium browser window? If an error 404 appears on following sites, I have a solution that worked for me:
I simply changed my User-Agent to "Naverbot" which is "allowed" by the robots.txt file from Tik Tok
(Robots.txt)
After changing that all sites and videos loaded properly.
Other user-agents that are listed under the "allow" segment should work too, if you want to add a rotation.

You can use Windows IE. Instead of chrome or firefox
Videos will load in IE but IE's Layout of showing feed is somehow different from chrome and firefox.
Reasons, why your page, is not loading.
Few advance web apps check your browser history, profile data and cached to check the authentication of the user.
One other thing you can do is run your default profile within your selenium It would be helpfull.

Chrome Webdriver doesn't load the default profile

The Selenium Chrome Webdriver does not load my default Chrome profile.
I already tried many other Stack Overflow solutions including changing the path (and using the local Chrome App, which gives a 'permission denied' Error).
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("--user-data-dir=~/Library/Application Support/Google/Chrome/Default")
options.add_argument("--profile-directory=Default")
driver = webdriver.Chrome()
driver.get("https://tagmanager.google.com/#/home")
assert "Google Analytics" in driver.title
account_element = driver.find_element_by_css_selector("div.account.card div#149385038 table")
accountPublicId = driver.find_element_by_css_selector("div#149385038 table td.account__public-id")
The result is still the same; it loads only a 'naked' Chrome Webdriver instead of loading the local default profile (I use everyday for work).
Update:
I don't know how or why but now, when I quit Chrome and start Chrome through the Python Script, Google Chrome starts with my profile but does not use the cookies I have in that profile.
I will see if I can add the cookies "manually" with an options.add_arguments.

Try this:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\ProfilePath") ##profile path
w = webdriver.Chrome(executable_path="C:\\chromedriver.exe", chrome_options=options)

you may try this
options.add_argument("user-data-dir=C:\\Test")
did not add anything to the
options arguments (looking via debugger it was an empty list)
options.arguments.append("user-data-dir=C:/test/") managed to update the list of arguments
and it worked.