I am trying to scrape match information from the a page like this one (page is in the same format, but obviously has different values for different matches): https://csgolounge.com/match?m=8967
The problem is, the information that I want is only displayed if you are using the Chrome extension, "Lounge Destroyer"... After tons of trial and error, I finally figured out that in order to get that information, the python script I use has to have that extension "included in it" somehow. I have browsed other answers here and found this code from a different stackoverflow thread that demonstrates how to add an extension when using selenium:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('Adblock-Plus_v1.4.1.crx')
driver = webdriver.Chrome(chrome_options = chop)
I went to Chrome Extension Downloader to snag the .crx file for LoungeDestroyer, placed it in the chrome extension folder (getting the file address from "Get Info"), and modified the above code a little bit for my purposes to get the following:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('Users/Username_Here/Library/Application Support/Google/Chrome/Default/Extensions/ghahcnmfjfckcedfajbhekgknjdplfcl/LoungeDestroyer_v0.9.3.7.crx')
driver = webdriver.Chrome(chrome_options = chop)
matchID = raw_input("Enter match ID (four digit number in CSGL URL): ")
driver.get("https://csgolounge.com/match?m="+matchID)
The problem is, I don't think I've substituted the right thing where the 'Adblock-Plus_v1.4.1.crx' was in the original code.
Running my modified version returns the following error:
IOError: Path to the extension doesn't exist
Any help or advice is greatly, greatly appreciated.
The problem was that I didn't have chromedriver installed (http://chromedriver.storage.googleapis.com/index.html?path=2.21/). After installing that, I had to enter the path to the chromedriver executable in my code. All said and done, this was the code that worked:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('/Users/Username_Here/Library/Application Support/Google/Chrome/Default/Extensions/ghahcnmfjfckcedfajbhekgknjdplfcl/LoungeDestroyer_v0.9.3.7.crx')
driver = webdriver.Chrome(executable_path='/Users/Username_Here/Downloads/chromedriver', chrome_options = chop)
# go to the match page
matchID = raw_input("Enter match ID (four digit number in CSGL URL): ")
driver.get("https://csgolounge.com/match?m="+matchID)
Also, the reason I was getting that extension-path error was because I didn't have the forward slash in front of the word "Users" in the file address.
Related
I need to load my own Default chrome profile where all my saved login info is, instead of selenium logging me into my websites.
I have taken this code from stackoverflow, cannot find the exact user. My issue with this code is selenium opens chrome but does not load the correct profile. Every time I open run the code it creates a new "scoped_dir" folder and runs the profile "default" from there chrome version
(C:\Users\farja\AppData\Local\Temp\scoped_dir[bunch of numbers]\Default).
I have tried closing chrome and then running my code which doesn't work
I am thinking there is a big flaw in my code but do not know what it is or how to find it. A relevant answer for 2022 would be very very much appreciated as I have literally been stuck on this for a week now and have tried multiple answers on stackoverflow, the web and youtube but nearly all give me a deprecation error.
Thank you for taking the time to read.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
ser = Service(r'C:\Users\farja\Documents\Instagram Programmes\Scheduler 2\chromedriver')
op = webdriver.ChromeOptions()
s = webdriver.Chrome(service=ser, options=op)
op.add_argument(r"--user-data-dir=C:\\Users\\farja\\AppData\\Local\\Google\\Chrome\\User Data")
op.add_argument("--profile-directory=Default")
s.get("chrome://version/")
As your usecase is to load the Default Chrome Profile you can use the argument user-data-dir and remove the argument --profile-directory as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
op = webdriver.ChromeOptions()
op.add_argument(r"--user-data-dir=C:\\Users\\farja\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
ser = Service(r'C:\Users\farja\Documents\Instagram Programmes\Scheduler 2\chromedriver')
s = webdriver.Chrome(service=ser, options=op)
s.get("chrome://version/")
References
You can find a couple of relevant detailed discussions in:
How to open a Chrome Profile through Python
How to open a Chrome Profile through --user-data-dir argument of Selenium
I have been working on making a WhatsApp bot using selenium, but every time I run my code I have to rescan the Qr code to enter it. I found this solution in a youtube video to use options, but it doesn't work and returns an error which is selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
This is my code
from selenium import webdriver
import time
if __name__ == '__main__':
options = webdriver.ChromeOptions()
options.add_argument(r'--user-data-dir=C:\Users\Hill\AppData\Local\Google\Chrome\User Data\Default')
options.add_argument(r'--profile-directory=Default')
driver = webdriver.Chrome(executable_path=r'C:\Program Files (x86)\chromedriver.exe', chrome_options=options)
driver.get('https://web.whatsapp.com')
time.sleep(15)
You are basically giving your Chrome-Browser two profiles. The first one, which is your profile and the second one, which is a default profile. Following steps worked for me:
Open Chrome manually and go to https://web.whatsapp.com then scan the QR-Code. Close everything and implement following code (please delete the comments in your implementation, because their syntax will cause problems ;-)):
options = webdriver.ChromeOptions()
options.add_argument(r'--user-data-dir=C:\Users\Hill\AppData\Local\Google\Chrome\User Data\') #here is no <Default>!!
driver = webdriver.Chrome(executable_path=r'C:\Program Files (x86)\chromedriver.exe', options=options) #selenium 4 preferes "options" -> your code is more up to date :-)
driver.get('https://web.whatsapp.com')
time.sleep(15)
Before I get blasted in my comments on this question, I am well aware that this is a possible duplication of this link, but the answer provided is not helping me out with my instance of code, even after applying an adjusted version of the answer in my code. I have looked into many answers, including installing Chromedriver into my device, but to no avail.
My code is as follows:
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(executable_path = r'C:\Users\user\Downloads\chromedriver_win32')
driver.get('http://codepad.org')
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("This text is send using Python code.")
Every time I run the code, including the executable_path = r'C:\Users\user\Downloads\chromedriver_win32'
I keep getting a permission error message when I run the code with the executable path.
My code without the path is the same minus the executable_path which I replace with driver = webdriver.Chrome(options), but I get the error message argument of type 'Options' is not iterable.
Any help with this problem is greatly appreciated. Admittedly I am a bit new to Python, and coding, in general, and I am trying new ideas to learn the program better, but everything that I try to find an answer to just breaks my code in general.
Try adding the executable file name at the end of executable_path argument:
executable_path = r'C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe'
Code I used for testing:
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(executable_path = r'C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe')
driver.get('http://codepad.org')
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("This text is send using Python code.")
I try to use Python to log in a web, type in use id and password, and download some files. however, before logging the page, there is a error msg says
"Failed to load extension from
C:\User\xyz\AppData\Local\Temp\scoped_dir12188_17478\internal. Loading
of unpacked extensions is disabled by administrator"
. After I hit "Enter", the rest goes well.
I have searched here, and find one post is related to my question, Here. But the code is still not working for me. the pop up error windows is still showing up. Any ideas or suggestion is highly welcome. Thank you!
Here is my code:
import time
import urllib3
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
driverpath = 'C:\\Users\\xyz\\AppData\\Local\\Programs\\Python\\Python36\\chromedriver.exe'
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
browser = webdriver.Chrome(executable_path=driverpath, chrome_options=chrome_options)
url ='https://userid:password#weblink'
browser.get(url)
This is the code I use:
import time
from selenium import webdriver
driver = webdriver.Chrome('C:/Program Files (x86)/Google/Chrome/Application/chromedriver') # Optional argument, if not specified will search path.
driver.navigate().to("http://www.google.com")
The URL I get in Google is data; I have selenium installed and Chromium. I don't know if I missed something. I tried with get driver a lot and it doesn't work.
You need to use
driver.get(url)