I have been searching loads of forums for using a proxy in python with the selenium library to prevent "max number" timeout with web scraping through selenium.
I found the script below in many forums, but it just doesn't seem to work for me whatsoever... Could anyone please help me and give me some advice on how to implement proxy in chrome through python with selenium.
Thanks a lot!
SCRIPT:
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
chromedriver = directory....
PROXY = "177.202.59.58:8080"
chrome_options = Options()
chrome_options.add_argument('--proxy-server=%s' % PROXY)
chrome = webdriver.Chrome(chromedriver, options=chrome_options)
chrome.get("https://whatismyipaddress.com")
There's nothing wrong with your code. That proxy is just not available/not working anymore.
Try to find another proxy that a better uptime. Keep it mind that public proxies have a noticeable latency so the page will load pretty slow.
Related
I need to load my own Default chrome profile where all my saved login info is, instead of selenium logging me into my websites.
I have taken this code from stackoverflow, cannot find the exact user. My issue with this code is selenium opens chrome but does not load the correct profile. Every time I open run the code it creates a new "scoped_dir" folder and runs the profile "default" from there chrome version
(C:\Users\farja\AppData\Local\Temp\scoped_dir[bunch of numbers]\Default).
I have tried closing chrome and then running my code which doesn't work
I am thinking there is a big flaw in my code but do not know what it is or how to find it. A relevant answer for 2022 would be very very much appreciated as I have literally been stuck on this for a week now and have tried multiple answers on stackoverflow, the web and youtube but nearly all give me a deprecation error.
Thank you for taking the time to read.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
ser = Service(r'C:\Users\farja\Documents\Instagram Programmes\Scheduler 2\chromedriver')
op = webdriver.ChromeOptions()
s = webdriver.Chrome(service=ser, options=op)
op.add_argument(r"--user-data-dir=C:\\Users\\farja\\AppData\\Local\\Google\\Chrome\\User Data")
op.add_argument("--profile-directory=Default")
s.get("chrome://version/")
As your usecase is to load the Default Chrome Profile you can use the argument user-data-dir and remove the argument --profile-directory as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
op = webdriver.ChromeOptions()
op.add_argument(r"--user-data-dir=C:\\Users\\farja\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
ser = Service(r'C:\Users\farja\Documents\Instagram Programmes\Scheduler 2\chromedriver')
s = webdriver.Chrome(service=ser, options=op)
s.get("chrome://version/")
References
You can find a couple of relevant detailed discussions in:
How to open a Chrome Profile through Python
How to open a Chrome Profile through --user-data-dir argument of Selenium
When I run this code
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument(r"user-data-dir=C:\Users\micha\AppData\Local\Google\Chrome\User Data\Profile 1")
driver = webdriver.Chrome(executable_path=r'C:\Users\micha\Desktop\Visual_projects\chromedriver.exe', chrome_options = chrome_options)
driver.get("https://store.steampowered.com/")
This error pops up : [12216:1336:0411/232857.718:ERROR:browser_switcher_service.cc(238)] XXX Init()
Could someone please help me. I don't know what is wrong but the programm won't open the new profile I created. Any help would be appreciated.
I searched everywhere how to fix this error but I think the guides are outdated
Check your Chrome browser version and your Chrome webdriver version. If both are same then issue should be solved.
One thing to check would be, if your ChromeDriver-Executable matches the version of the Chrome you have installed (under Help/About Google Chrome in your browser). There are versions 79, 80 and 81 mostly in use. You can find the current ChromeDrivers for all three versions here: https://chromedriver.chromium.org/downloads.
When I try to use a proxy in firefox it doesn't work, the code doesn't give me any error, all works good but the ip doesn't change.
But when I do it with chrome it works.
Can you please tell me what I'm doing wrong?
from selenium import webdriver
PROXY = "ipproxy"
firefox_options = webdriver.FirefoxOptions()
firefox_options.add_argument ('--proxy-server=%s' % PROXY)
firefox = webdriver.Firefox(options=firefox_options)
firefox.get("https://www.youtube.com/")
I've been attempting to bypass using Spectron for End2End testing an electron application by leveraging my experience with Selenium Webdriver on Python.
Using a combination of the Chromedriver get started page, and several resources that seem to suggest its possible, this is what I came up with:
from selenium import webdriver
import selenium.webdriver.chrome.service as service
servicer = service.Service('C:\\browserDrivers\\chromedriver_win32\\chromedriver.exe')
servicer.start()
capabilities = {'chrome.binary': 'C:\\path\\to\\electron.exe'}
remote = webdriver.remote.webdriver.WebDriver(command_executor=servicer.service_url, desired_capabilities = capabilities, browser_profile=None, proxy=None, keep_alive=False
The issue is that instead of opening the electron application, it opens a standard instance of Chrome.
Most of resources I've seen have been several years old so something may have changed to make it no longer possible.
Does anyone know of a way to use Python Selenium WebDriver to test an Electron application?
Below works great for me
from selenium import webdriver
options = webdriver.ChromeOptions()
options.binary_location = "/Applications/Electron.app/Contents/MacOS/Electron"
driver = webdriver.Chrome(chrome_options=options)
driver.get("http://www.google.com")
driver.quit()
When using selenium to open a web page, it automatically deletes all cookies u had saved in browser which is inconvenient.
Find solution in this page java solution
but don't know how to solve the problem using Python.
Point the browser to a profile file (that's what the java example in the given link is doing)
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('user-data-dir=<path to chrome profile>')
browser = webdriver.Chrome(chrome_options=chrome_options)
On Linux the <path to chrome profile> is /home/<user>/.config/google-chrome.