Develop Reference

Starting selenium program in a manually opened browser

I am using selenium and python in order to scrape data on a website.
The problem is I need to manually log in because there is a CAPTCHA after the login.
My question is the following : is there a way to start the program on a page that is already loaded ? (for example, here I would log to the website, solve the CAPTCHA manually, and then launch the program that would scrape the data)
Note: I have already been looking for an answer on SO but did not find it, might have missed it as it seems to be an obvious question.

don't open in headless mode. open in head mode.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.headless = False # Set false here
driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
driver.get("http://google.com/")
print ("Headless Chrome Initialized")
time.sleep(30) # wait 30 seconds, this should give enough time to manually do the capture
# do other code here
driver.quit()

Code stops executing after opening browser through webdriver using python

I have been trying to open multiple browser windows in internet explorer using webdriver in selenium. Once it reaches the get(url) line, it just halts there and eventually times out. I've added a print line, which does not execute. I've tried various methods and the one below is the Ie version of code I used to open multiple tabs in Chrome. Even if I remove the first 3 lines, it still only goes up to opening google.com. I've looked googled this issue and looked through other posts but nothing has helped. Would really appreciate any advice, thanks!
options = webdriver.IeOptions()
options.add_additional_option("detach", True)
driver = webdriver.Ie(options = options, executable_path=r'blahblah\IEDriverServer.exe')
driver.get("http://google.com")
print("syrfgf")
driver.execute_script("window.open('about:blank', 'tab2');")
driver.switch_to.window("tab2")
driver.get("http://yahoo.com")

You need to replace the url you have provided:
http://google.com
with a proper url as follows:
https://www.google.com/
Which should be represented as per the syntax diagram as follows:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome()
driver.get("https://github.com")
signin_link = driver.find_element(By.LINK_TEXT, "Sign in")
signin_link.click()
time.sleep(1)
user = driver.find_element(By.ID, "login_field")
user.send_keys("X")
passw = driver.find_element(By.ID, "password")
passw.send_keys("X")
passw.submit()
time.sleep(5)
driver.close()
I had this issue and writing this code seems to have made it work flawlessly. Adjust the sleep time as you want it. Putting my chromedriver.exe into my project folder also helped with some errors

Why is my webdriver.chrome() not working?

Before I get blasted in my comments on this question, I am well aware that this is a possible duplication of this link, but the answer provided is not helping me out with my instance of code, even after applying an adjusted version of the answer in my code. I have looked into many answers, including installing Chromedriver into my device, but to no avail.
My code is as follows:
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(executable_path = r'C:\Users\user\Downloads\chromedriver_win32')
driver.get('http://codepad.org')
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("This text is send using Python code.")
Every time I run the code, including the executable_path = r'C:\Users\user\Downloads\chromedriver_win32'
I keep getting a permission error message when I run the code with the executable path.
My code without the path is the same minus the executable_path which I replace with driver = webdriver.Chrome(options), but I get the error message argument of type 'Options' is not iterable.
Any help with this problem is greatly appreciated. Admittedly I am a bit new to Python, and coding, in general, and I am trying new ideas to learn the program better, but everything that I try to find an answer to just breaks my code in general.

Try adding the executable file name at the end of executable_path argument:
executable_path = r'C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe'
Code I used for testing:
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(executable_path = r'C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe')
driver.get('http://codepad.org')
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("This text is send using Python code.")

Selenium with Firefox or Chrome profile

I am trying to load a profile to selenium so that I don't have to keep log in to the website that selenium is about to visit. I am running it with Python on my Mac.
In the Firefox version, I use the below code:
def create_selenium_FF():
profile = webdriver.FirefoxProfile('/Users/Victor/Library/Application Support/Firefox/Profiles/z3ay0enb.default')
driver = webdriver.Firefox(profile)
return driver
It can successfully start Firefox, but it doesn't have the log in info of the website that it visits, however I check in the automated Firefox browser using about:profiles, it does recognise the profile that I feed it.
In the Chrome version, I use the below code, notice I make a local copy of the profile already.
def create_selenium_chrome():
DRIVER = 'chromedriver'
options = webdriver.ChromeOptions()
options.add_argument("--user-data-dir=/Users/Victor/Library/Application Support/Google/Chrome2")
options.add_argument("--profile-directory=Default")
driver = webdriver.Chrome(DRIVER, options=options)
return driver
It can also start Chrome, and looks like it has my profile, but it raises an error:
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
How can I get it working please?

I just solved the problem !
So here is my code :
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\\%username%\\AppData\\Local\\Google\\Chrome\\User Data 2")
driver = webdriver.Chrome(executable_path=path,options=options)
driver.get("https://www.google.com")
The thing is, you'll get this error if there already is chrome opened on your computer !
So, i just copy / paste the folder User Data and rename the pasted one into User Data 2 so my chrome works with user data and selenium with user data 2 I guess.
I know that you've been waiting for a long time and I don't know if u still need this but here you got !!

How can I use a Google Chrome extension with Selenium?

I am trying to scrape match information from the a page like this one (page is in the same format, but obviously has different values for different matches): https://csgolounge.com/match?m=8967
The problem is, the information that I want is only displayed if you are using the Chrome extension, "Lounge Destroyer"... After tons of trial and error, I finally figured out that in order to get that information, the python script I use has to have that extension "included in it" somehow. I have browsed other answers here and found this code from a different stackoverflow thread that demonstrates how to add an extension when using selenium:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('Adblock-Plus_v1.4.1.crx')
driver = webdriver.Chrome(chrome_options = chop)
I went to Chrome Extension Downloader to snag the .crx file for LoungeDestroyer, placed it in the chrome extension folder (getting the file address from "Get Info"), and modified the above code a little bit for my purposes to get the following:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('Users/Username_Here/Library/Application Support/Google/Chrome/Default/Extensions/ghahcnmfjfckcedfajbhekgknjdplfcl/LoungeDestroyer_v0.9.3.7.crx')
driver = webdriver.Chrome(chrome_options = chop)
matchID = raw_input("Enter match ID (four digit number in CSGL URL): ")
driver.get("https://csgolounge.com/match?m="+matchID)
The problem is, I don't think I've substituted the right thing where the 'Adblock-Plus_v1.4.1.crx' was in the original code.
Running my modified version returns the following error:
IOError: Path to the extension doesn't exist
Any help or advice is greatly, greatly appreciated.

The problem was that I didn't have chromedriver installed (http://chromedriver.storage.googleapis.com/index.html?path=2.21/). After installing that, I had to enter the path to the chromedriver executable in my code. All said and done, this was the code that worked:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('/Users/Username_Here/Library/Application Support/Google/Chrome/Default/Extensions/ghahcnmfjfckcedfajbhekgknjdplfcl/LoungeDestroyer_v0.9.3.7.crx')
driver = webdriver.Chrome(executable_path='/Users/Username_Here/Downloads/chromedriver', chrome_options = chop)
# go to the match page
matchID = raw_input("Enter match ID (four digit number in CSGL URL): ")
driver.get("https://csgolounge.com/match?m="+matchID)
Also, the reason I was getting that extension-path error was because I didn't have the forward slash in front of the word "Users" in the file address.