When I try to use a proxy in firefox it doesn't work, the code doesn't give me any error, all works good but the ip doesn't change.
But when I do it with chrome it works.
Can you please tell me what I'm doing wrong?
from selenium import webdriver
PROXY = "ipproxy"
firefox_options = webdriver.FirefoxOptions()
firefox_options.add_argument ('--proxy-server=%s' % PROXY)
firefox = webdriver.Firefox(options=firefox_options)
firefox.get("https://www.youtube.com/")
Related
I have searched on stackoverflow as well as just googled how to use Proxies with Selenium. I found two different ways but none are working for me. Can you guys please help me figure out what I am doing wrong?
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType
proxy = "YYY.YYY.YYY.YY:XXXX"
prox = Proxy()
prox.proxy_type = ProxyType.MANUAL
prox.http_proxy = proxy
prox.https_proxy = proxy
capabilities = webdriver.DesiredCapabilities.CHROME
prox.add_to_capabilities(capabilities)
options = webdriver.ChromeOptions()
options.add_experimental_option('detach', True)
driver = webdriver.Chrome(desired_capabilities=capabilities, options=options)
Code Above did not work. The page would open but if I go to "Whatsmyip.com" I could see my home IP.
I then tried another method I found on this link:
https://www.browserstack.com/guide/set-proxy-in-selenium
proxy = "YYY.YYY.YYY.YY:XXXX"
options = webdriver.ChromeOptions()
options.add_experimental_option('detach', True)
options.add_argument("--proxy--server=%s" % proxy)
driver = webdriver.Chrome(options = options)
Same result as with the previous method. Browser will open but home IP.
Worth mentioning that I tried with USER:PASS proxies, as well as IP Authorized proxies. None worked!
In addition to helping me figure out how to use proxies, I would also like to understand why these methods are different. On the one hand, the Selenium documentation talks about a proxy class which you access via the "common.proxy" class, yet the second method is directly using Chrome's options and not Selenium's proxy class. I am confused as to why have two methods, and of course which one works more reliably.
Thanks
Authentificated proxies aren't supported for chrome by default. If you still need them, refer to Selenium-Profiles.
Your second code snippet should work as following:
proxy = "https://host_or_ip:port" # or "socks5://" or "http://"
options = webdriver.ChromeOptions()
options.add_argument("--proxy--server=%s" % proxy)
driver = webdriver.Chrome(options = options)
driver.get("http://lumtest.com/myip.json") # test proxy
input("Press ENTER to exit")
If it still doesn't work, check your proxy with curl or python requests.
I'm trying to use a Proxy in selenium python, but when i test the code on a web page that show my ip the proxy doesn't change my ip adress.
Here there is the code
import undetected_chromedriver as uc
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
option = Options()
driver = uc.Chrome(options = option)
PROXY = "139.162.214.146"
option.add_argument('--proxy-server=%s' % PROXY)
option.add_argument('--disable-extensions')
option.add_argument('--disable-notifications')
option.add_argument('--no-sandbox')
option.add_argument('--disable-dev-shm-usage')
option.add_argument('--disable-gpu')
option.add_argument('--mute-audio')
option.add_argument('--ignore-certificate-errors')
option.add_argument('--disable-logging')
driver.get("https://www.whatismyip.com/")
input ()
The code should change mi ip but this doesn't appen, i tried to change the proxy but the problem is the same.
I can see multiple issues here. Try using the following code:
import undetected_chromedriver as uc
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
option = Options()
PROXY = "139.162.214.146"
option.add_argument('--proxy-server=%s' % PROXY)
option.add_argument('--disable-extensions')
option.add_argument('--disable-notifications')
option.add_argument('--no-sandbox')
option.add_argument('--disable-dev-shm-usage')
option.add_argument('--disable-gpu')
option.add_argument('--mute-audio')
option.add_argument('--ignore-certificate-errors')
option.add_argument('--disable-logging')
driver = uc.Chrome(options = option)
driver.get("https://www.whatismyip.com/")
input ()
Also note, that you need to specify a port for your proxy as following: PROXY = "139.162.214.146:proxy_port"
I have been searching loads of forums for using a proxy in python with the selenium library to prevent "max number" timeout with web scraping through selenium.
I found the script below in many forums, but it just doesn't seem to work for me whatsoever... Could anyone please help me and give me some advice on how to implement proxy in chrome through python with selenium.
Thanks a lot!
SCRIPT:
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
chromedriver = directory....
PROXY = "177.202.59.58:8080"
chrome_options = Options()
chrome_options.add_argument('--proxy-server=%s' % PROXY)
chrome = webdriver.Chrome(chromedriver, options=chrome_options)
chrome.get("https://whatismyipaddress.com")
There's nothing wrong with your code. That proxy is just not available/not working anymore.
Try to find another proxy that a better uptime. Keep it mind that public proxies have a noticeable latency so the page will load pretty slow.
I'm using Selenium webdriver to open a webpage and I set up a proxy for the driver to use. The code is listed below:
PATH = "C:\Program Files (x86)\chromedriver.exe"
PROXY = "212.237.16.60:3128" # IP:PORT or HOST:PORT
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f'--proxy-server={PROXY}')
proxy = Proxy()
proxy.auto_detect = False
proxy.http_proxy = PROXY
proxy.sslProxy = PROXY
proxy.socks_proxy = PROXY
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
driver = webdriver.Chrome(PATH, chrome_options=chrome_options,desired_capabilities=capabilities)
driver.get("https://whatismyipaddress.com")
The problem is that the web driver is not using the given proxy and it accesses the page with my normal IP. I already tried every type of code I could find on the internet and it didn't work. I also tried to set a proxy directly in my pc settings and when I open a normal chrome page it works fine (it's not a proxy server problem then), but if I open a page with the driver it still uses my normal IP and somehow bypasses the proxy. I also tried changing the proxy settings of the IDE (pycharm) and still it's not working. I'm out of ideas, could someone help me?
This should work.
Code snippet-
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
PROXY = "212.237.16.60:3128"
#add proxy in chrome_options
chrome_options.add_argument(f'--proxy-server={PROXY}')
driver = webdriver.Chrome(PATH,options=chrome_options)
#to check new IP
driver.get("https://api.ipify.org/?format=json")
Note:- chrome_options is deprecated now, you have to use options instead
I am trying to connect to a Tor browser but get an error stating "proxyConnectFailure" any ideas I have tried multiple attempts to get into the basics of Tor browser to get it connected but all in vain if any could help life could be saved big time:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary(r"C:\Users\Admin\Desktop\Tor Browser\Browser\firefox.exe")
profile = FirefoxProfile(r"C:\Users\Admin\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default")
# Configured profile settings.
proxyIP = "127.0.0.1"
proxyPort = 9150
proxy_settings = {"network.proxy.type":1,
"network.proxy.socks": proxyIP,
"network.proxy.socks_port": proxyPort,
"network.proxy.socks_remote_dns": True,
}
driver = webdriver.Firefox(firefox_binary=binary,proxy=proxy_settings)
def interactWithSite(driver):
driver.get("https://www.google.com")
driver.save_screenshot("screenshot.png")
interactWithSite(driver)
To connect to a Tor Browser through a FirefoxProfile you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile = FirefoxProfile(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile= profile, executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("http://check.torproject.org")
Browser Snapshot:
You can find a relevant discussion in How to use Tor with Chrome browser through Selenium
I would like to expand on #DebanjanB answer by adding the Linux counterpart:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen('some/path/tor-browser_en-US/Browser/start-tor-browser')
# in my case, I installed it under a folder tor-browser_en-US after
# downloading and extracting it from
# https://www.torproject.org/download/ for linux
profile = FirefoxProfile(
'some/path/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = '/usr/bin/firefox'
# /usr/bin/firefox is default location of firefox - for me anyway
driver = webdriver.Firefox(
firefox_profile=profile, options=firefox_options,
executable_path='wherever/you/installed/geckodriver')
# I keep my geckodriver(s) in a special folder sorted by versions.
# Geckodriver downloadable here:
# https://github.com/mozilla/geckodriver/releases/
driver.get("http://check.torproject.org")
The verified answer does not work in case of opening dot onion sites(I believe that's something to do with tor network which is not allowing access to normal firefox).
As for the latest tor browser (from the tor browser bundle), starting it using selenium causes some error due to which the browser cannot start tor proxy itself causing proxy and timeout errors(doesn't matter if tor proxy is started by python or manually or not started at all). This could also be due to port 9050 or 9150 being used by tor proxy and not being available to browser's tor instance but this does not explain the error caused when no instance of tor proxy is running.
The solution i have found is to start the tor proxy as normal, manually or using os.popen("tor.exe") and configure tor browser to not start tor proxy.
here's the code:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
os.popen(r'e:\\bla\\bla\\bla\\tor\\Tor\\tor.exe')
binary=FirefoxBinary(r'e:\\bla\\bla\\bla\\Tor Browser\\Browser\\firefox.exe')
fp=FirefoxProfile(r'e:\\foo\\bar\\bla\\Tor Browser\\Browser\\TorBrowser\\Data\\Browser\\profile.default')
fp.set_preference('extensions.torlauncher.start_tor',False)#note this
fp.set_preference('network.proxy.type',1)
fp.set_preference('network.proxy.socks', '127.0.0.1')
fp.set_preference('network.proxy.socks_port', 9050)
fp.set_preference("network.proxy.socks_remote_dns", True)
fp.update_preferences()
driver = webdriver.Firefox(firefox_profile=fp,firefox_binary=binary)
driver.get("http://check.torproject.org")
driver.get('https://www.bbcnewsv2vjtpsuy.onion/')
*note fp.set_preference('extensions.torlauncher.start_tor',False) on line 10 is being used to configure tor to not start its own tor instance so that it uses the proxy config and tor instance started above.
lo and behold as the tbb starts working like normal firefox bot browser