Before I get blasted in my comments on this question, I am well aware that this is a possible duplication of this link, but the answer provided is not helping me out with my instance of code, even after applying an adjusted version of the answer in my code. I have looked into many answers, including installing Chromedriver into my device, but to no avail.
My code is as follows:
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(executable_path = r'C:\Users\user\Downloads\chromedriver_win32')
driver.get('http://codepad.org')
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("This text is send using Python code.")
Every time I run the code, including the executable_path = r'C:\Users\user\Downloads\chromedriver_win32'
I keep getting a permission error message when I run the code with the executable path.
My code without the path is the same minus the executable_path which I replace with driver = webdriver.Chrome(options), but I get the error message argument of type 'Options' is not iterable.
Any help with this problem is greatly appreciated. Admittedly I am a bit new to Python, and coding, in general, and I am trying new ideas to learn the program better, but everything that I try to find an answer to just breaks my code in general.
Try adding the executable file name at the end of executable_path argument:
executable_path = r'C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe'
Code I used for testing:
from selenium import webdriver
import time
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(executable_path = r'C:\Users\user\Downloads\chromedriver_win32\chromedriver.exe')
driver.get('http://codepad.org')
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("This text is send using Python code.")
Related
I'm making a price scraping program and have ran into the issue of antiscraping systems. I managed to get around these with the undetected_chromedriver but now I'm running into 2 issues
the first is that the UC is significantly slower than the standard chrome driver, through I need it for some sites, so I have some sites scraped with a normal driver and others with the UC
the second problem is that I have the standard Chrome driver install at the beginning of the program, but once I do that, the UC feels the need to install every time I open it?? this causes some sites to be scraped really slowly. can you help with why that is? and any other tips for running scraper faster would be appreciated.
I have this run at the beginning of the program as global variables:
chrome_path = Service(ChromeDriverManager().install())
options = webdriver.ChromeOptions()
options.headless = True
options.add_experimental_option('excludeSwitches', ['enable-logging'])
and this runs as a function every time I need a UC:
def start_uc():
options = webdriver.ChromeOptions()
# just some options passing in to skip annoying popups
options.add_argument('--no-first-run --no-service-autorun --password-store=basic')
driver = uc.Chrome(options=options)
driver.minimize_window()
return driver
My scraping functions just loop looking up the url and scrape the info, and restart the driver to clear the cookies if I run into a captcha .The scraping functions look like this (this is psuedo code to give you an idea):
driver = start_uc()
for url in url_list:
while true:
try:
driver.get(url)
#scrape info
break
except:
driver.close()
driver = start_uc()
I dont see why chrome_path would affect the UC? and are there any suggestions to make the scraping functions run more efficiently? Im not an expert on drivers and their intricacies so I could be doing something terribly wrong that I dont recognize.
thankyou in advance!
You can use https://github.com/seleniumbase/SeleniumBase to speed things up.
(It has a special undetected-chromedriver mode that works with headless mode.)
pip install -U seleniumbase
And then run the following with python:
from seleniumbase import Driver
from seleniumbase import page_actions
driver = Driver(headless=True, uc=True)
driver.get("https://nowsecure.nl")
page_actions.wait_for_text(driver, "OH YEAH, you passed!", "h1")
print(driver.find_element("css selector", "body").text)
screenshot_name = "now_secure_image.png"
driver.save_screenshot(screenshot_name)
print("\nScreenshot saved to: %s" % screenshot_name)
driver.quit()
How to describe the path to chrome on MAC?
I used this code:
import time
from selenium import webdriver
driver = webdriver.Chrome('../chromedriver') # Optional argument, if not specified will search path.
driver.get('http://www.google.com/');
time.sleep(5) # Let the user actually see something!
search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()
time.sleep(5) # Let the user actually see something!
driver.quit()```
I got error: **DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome('../chromedriver') # Optional argument, if not specified will search path.**
The "error" is just a warning; your code should work with it. Still, it is good to fix deprecation errors, because the code might stop working soon. This should not issue an error:
from selenium import webdriver
# modify if different
chrome_path = '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
chrome_service = webdriver.chrome.service.Service(chrome_path)
driver = webdriver.Chrome(service=chrome_service)
Of course, you can also download chromedriver. If you don't want to worry about where your Chrome or chromedriver is, you can use webdriver_manager which will automatically dowload the driver for you if necessary, and supply you with the path:
from webdriver_manager.chrome import ChromeDriverManager
chrome_path = ChromeDriverManager().install()
I have been working on making a WhatsApp bot using selenium, but every time I run my code I have to rescan the Qr code to enter it. I found this solution in a youtube video to use options, but it doesn't work and returns an error which is selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
This is my code
from selenium import webdriver
import time
if __name__ == '__main__':
options = webdriver.ChromeOptions()
options.add_argument(r'--user-data-dir=C:\Users\Hill\AppData\Local\Google\Chrome\User Data\Default')
options.add_argument(r'--profile-directory=Default')
driver = webdriver.Chrome(executable_path=r'C:\Program Files (x86)\chromedriver.exe', chrome_options=options)
driver.get('https://web.whatsapp.com')
time.sleep(15)
You are basically giving your Chrome-Browser two profiles. The first one, which is your profile and the second one, which is a default profile. Following steps worked for me:
Open Chrome manually and go to https://web.whatsapp.com then scan the QR-Code. Close everything and implement following code (please delete the comments in your implementation, because their syntax will cause problems ;-)):
options = webdriver.ChromeOptions()
options.add_argument(r'--user-data-dir=C:\Users\Hill\AppData\Local\Google\Chrome\User Data\') #here is no <Default>!!
driver = webdriver.Chrome(executable_path=r'C:\Program Files (x86)\chromedriver.exe', options=options) #selenium 4 preferes "options" -> your code is more up to date :-)
driver.get('https://web.whatsapp.com')
time.sleep(15)
I try to use Python to log in a web, type in use id and password, and download some files. however, before logging the page, there is a error msg says
"Failed to load extension from
C:\User\xyz\AppData\Local\Temp\scoped_dir12188_17478\internal. Loading
of unpacked extensions is disabled by administrator"
. After I hit "Enter", the rest goes well.
I have searched here, and find one post is related to my question, Here. But the code is still not working for me. the pop up error windows is still showing up. Any ideas or suggestion is highly welcome. Thank you!
Here is my code:
import time
import urllib3
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
driverpath = 'C:\\Users\\xyz\\AppData\\Local\\Programs\\Python\\Python36\\chromedriver.exe'
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
browser = webdriver.Chrome(executable_path=driverpath, chrome_options=chrome_options)
url ='https://userid:password#weblink'
browser.get(url)
I am trying to scrape match information from the a page like this one (page is in the same format, but obviously has different values for different matches): https://csgolounge.com/match?m=8967
The problem is, the information that I want is only displayed if you are using the Chrome extension, "Lounge Destroyer"... After tons of trial and error, I finally figured out that in order to get that information, the python script I use has to have that extension "included in it" somehow. I have browsed other answers here and found this code from a different stackoverflow thread that demonstrates how to add an extension when using selenium:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('Adblock-Plus_v1.4.1.crx')
driver = webdriver.Chrome(chrome_options = chop)
I went to Chrome Extension Downloader to snag the .crx file for LoungeDestroyer, placed it in the chrome extension folder (getting the file address from "Get Info"), and modified the above code a little bit for my purposes to get the following:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('Users/Username_Here/Library/Application Support/Google/Chrome/Default/Extensions/ghahcnmfjfckcedfajbhekgknjdplfcl/LoungeDestroyer_v0.9.3.7.crx')
driver = webdriver.Chrome(chrome_options = chop)
matchID = raw_input("Enter match ID (four digit number in CSGL URL): ")
driver.get("https://csgolounge.com/match?m="+matchID)
The problem is, I don't think I've substituted the right thing where the 'Adblock-Plus_v1.4.1.crx' was in the original code.
Running my modified version returns the following error:
IOError: Path to the extension doesn't exist
Any help or advice is greatly, greatly appreciated.
The problem was that I didn't have chromedriver installed (http://chromedriver.storage.googleapis.com/index.html?path=2.21/). After installing that, I had to enter the path to the chromedriver executable in my code. All said and done, this was the code that worked:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chop = webdriver.ChromeOptions()
chop.add_extension('/Users/Username_Here/Library/Application Support/Google/Chrome/Default/Extensions/ghahcnmfjfckcedfajbhekgknjdplfcl/LoungeDestroyer_v0.9.3.7.crx')
driver = webdriver.Chrome(executable_path='/Users/Username_Here/Downloads/chromedriver', chrome_options = chop)
# go to the match page
matchID = raw_input("Enter match ID (four digit number in CSGL URL): ")
driver.get("https://csgolounge.com/match?m="+matchID)
Also, the reason I was getting that extension-path error was because I didn't have the forward slash in front of the word "Users" in the file address.