I’ve got grey screen when was trying to open bet365 site using Chrome driver and Selenium.
var driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://www.bet365.it/");
I executed your usecase with a couple of tweaks and faced the same consequences. Here are the execution details:
Code Block [Python]:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.bet365.it/')
Browser Snapshot:
Deep Dive
When I checked the Terms and conditions it is clearly mentioned that:
Other
6.1 bet365 actively monitors traffic to and from its Site. Bet365 reserves the right to block access to the Site, at its discretion,
should it encounter any evidence of automated or robotized game
activity.
Conclusion
It seems Selenium driven ChromeDriver initiated google-chrome based browsing context is getting detected and the navigation is blocked.
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
# chrome_options.add_argument("--headless")
chrome_options.add_argument('--start-maximized')
driver = webdriver.Chrome(options=chrome_options, executable_path=r"chromedriver.exe")
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {"source":
"""Object.defineProperty(navigator,
'webdriver', {get: () => undefined})"""})
url = 'https://www.bet365.com/#/IP/B1'
driver.get(url)
sleep(1)
Related
I am trying to scrape the website. First of all, it is not working with Beautifulsoup but when I am trying to open it with selenium chrome driver it's not opening. It's opening with firefox but it's very slow and gives an error on element click. Here is my code:
from selenium import webdriver
opt = webdriver.ChromeOptions()
opt.add_argument("--disable-xss-auditor")
opt.add_argument("--disable-web-security")
opt.add_argument("--allow-running-insecure-content")
opt.add_argument("--no-sandbox")
opt.add_argument("--disable-setuid-sandbox")
opt.add_argument("--disable-webgl")
opt.add_argument("--disable-popup-blocking")
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(f"http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=25&tableName=TABLE25&title=%B9%FA%B2%FA%D2%A9%C6%B7&bcId=152904713761213296322795806604&CbSlDlH0=qGrYrAktn7.tn7.tnznJalIvVetjcXpaapSdKuqmmoVqqWL")
Possibly Selenium driven ChromeDriver initiated google-chrome Browsing Context is geting detected as bot and the arguments you have added can't bypass the bot detection mechanism effectively.
Solution
You can evade the detection by adding a few arguments and experimental_option as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get("http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=25&tableName=TABLE25&title=%B9%FA%B2%FA%D2%A9%C6%B7&bcId=152904713761213296322795806604&CbSlDlH0=qGrYrAktn7.tn7.tnznJalIvVetjcXpaapSdKuqmmoVqqWL")
This is what I have:
from selenium import webdriver
driver = webdriver.Firefox()
How can I let the geckodriver open minimized or hidden?
You have to set the firefox driver options to headless so that it opens minimized. Here is the code to do that.
fireFoxOptions = webdriver.FirefoxOptions()
fireFoxOptions.set_headless()
driver = webdriver.Firefox(firefox_options=fireFoxOptions)
If this doesn't work for you there are other methods that you can use. Check out this other SO question for those: How to make firefox headless programmatically in Selenium with python?
geckodriver
geckodriver in it's core form is a proxy for using W3C WebDriver-compatible clients to interact with Gecko-based browsers.
This program provides the HTTP API described by the WebDriver protocol to communicate with Gecko browsers, such as Firefox. It translates calls into the Firefox remote protocol by acting as a proxy between the local- and remote ends.
So more or less it acts like a service. So the question to minimize the GeckoDriver shouldn't arise.
Minimizing Mozilla Firefox browser
The Selenium driven GeckoDriver initiated firefox Browsing Context by default opens in semi maximized mode. Perhaps while executing tests with the browsing context being minimized would be against all the best practices as Selenium may loose the focus over the Browsing Context and an exception may raise during the Test Execution.
However, Selenium's python client does have a minimize_window() method which eventually minimizes the Chrome Browsing Context effectively.
Solution
You can use the following solution:
Firefox:
from selenium import webdriver
options = webdriver.FirefoxOptions()
options.binary_location = r'C:\Program Files\Mozilla Firefox\firefox.exe'
driver = webdriver.Firefox(firefox_options=options, executable_path=r'C:\WebDrivers\geckodriver.exe')
driver.get('https://www.google.co.in')
driver.minimize_window()
Chrome:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://www.google.co.in')
driver.minimize_window()
Reference
You can find a detailed relevant discussion in:
How to Minimize browser window in selenium webdriver 3
tl; dr
How to make firefox headless programmatically in Selenium with python?
I need help, selenium Chrome getting stuck at some pages while browsing,
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
#options.add_argument('--no-sandbox')
options.add_argument("disable-infobars")
options.add_argument('--disable-browser-side-navigation')
options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=options)
driver.set_page_load_timeout(10)
this how I make the driver, however it is definetly getting stuck on some pages for more than 300s (I have to Ctrl+C) to get it to keep moving (inside a for loop)
I am trying to run headless Chrome and it running normally without headless it just runs Chrome as normal. I have updated chrome recently so that's not the issue.
driver.maximize_window()
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('headless')
browser = webdriver.Chrome(chrome_options=options)
driver.get('https://www.youtube.com.au')
You need to make a couple of changes to your code block as follows :
You need to maximize the browser once the webdriver opens the browser instance.
As per best practices to maximize the Chrome Window use Options() Class.
To maximize the Chrome Window through Options() Class either use the start-maximized argument or use the window-size argument.
Here is the sample code block for your reference:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("--headless");
options.add_argument("window-size=1400,600");
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("http://google.com/")
print ("Headless Chrome Initialized")
I started having issues running a python script which uses selenium and chrome driver, so I want to disable extensions of chrome driver using python without losing the path where the driver is located.
Currently I have the below:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
path_to_Ie = 'C:\\Python34\\ChromeDriver\\ChromeDriver.exe'
browser = webdriver.Chrome(executable_path = path_to_Ie)
url = 'https://wwww.test.com/'
browser.get(url)
and I would like to add the below lines:
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
browser = webdriver.Chrome(chrome_options=chrome_options)
Any idea how I can merge both codes to make it work properly?
Thanks a lot.
Just provide multiple keyword arguments:
chrome_options = Options()
chrome_options.add_argument("--disable-extensions")
browser = webdriver.Chrome(executable_path=path_to_Ie, chrome_options=chrome_options)