Develop Reference

How to get rid of block while scraping with Selenium

I'm trying to scrape a website with Selenium, but I think it's blocking this access in many ways.
The error message shown is: "selenium.common.exceptions.NoSuchWindowException: Message: Browsing context has been discarded" but sometimes is shown an error saying that time for loading page had expired
Furthermore, Firefox is consuming a huge percent of CPU and Memory when loading this page.
I've already tried to change user-agent, or run it headlessly, but no results.
Below is the code:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://www.bet365.com/#/HO/')
matches = browser.find_elements_by_class_name('him-Fixture')
browser.quit()
Any tips to bypass it ?

Sometimes browsers loading late . So you add time.sleep() function in you code.
Example :
from selenium import webdriver
import time
browser = webdriver.Firefox()
browser.get('https://www.bet365.com/#/HO/')
time.sleep(5)
matches = browser.find_elements_by_class_name('him-Fixture')
browser.quit()

Code stops executing after opening browser through webdriver using python

I have been trying to open multiple browser windows in internet explorer using webdriver in selenium. Once it reaches the get(url) line, it just halts there and eventually times out. I've added a print line, which does not execute. I've tried various methods and the one below is the Ie version of code I used to open multiple tabs in Chrome. Even if I remove the first 3 lines, it still only goes up to opening google.com. I've looked googled this issue and looked through other posts but nothing has helped. Would really appreciate any advice, thanks!
options = webdriver.IeOptions()
options.add_additional_option("detach", True)
driver = webdriver.Ie(options = options, executable_path=r'blahblah\IEDriverServer.exe')
driver.get("http://google.com")
print("syrfgf")
driver.execute_script("window.open('about:blank', 'tab2');")
driver.switch_to.window("tab2")
driver.get("http://yahoo.com")

You need to replace the url you have provided:
http://google.com
with a proper url as follows:
https://www.google.com/
Which should be represented as per the syntax diagram as follows:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome()
driver.get("https://github.com")
signin_link = driver.find_element(By.LINK_TEXT, "Sign in")
signin_link.click()
time.sleep(1)
user = driver.find_element(By.ID, "login_field")
user.send_keys("X")
passw = driver.find_element(By.ID, "password")
passw.send_keys("X")
passw.submit()
time.sleep(5)
driver.close()
I had this issue and writing this code seems to have made it work flawlessly. Adjust the sleep time as you want it. Putting my chromedriver.exe into my project folder also helped with some errors

python selenium having difficulties with click

Currently using python and trying to have selenium click the "About" on google without using id. When I try to use .click() it does not execute, what is wrong with my code? I have looked at many videos and tutorials and it looks correct.
from selenium import webdriver
from time import sleep
browser = webdriver.Safari()
browser.get('http://google.com')
browser.maximize_window()
elm = browser.find_element_by_link_text('About')
browser.implicitly_wait(5)
elm.click()

I think, you can try using find_element_by_xpath.
First you will copy xpath of about link then you can try like below:
from selenium import webdriver
from time import sleep
browser = webdriver.Safari()
browser.get('http://google.com')
browser.maximize_window()
elm = browser.find_element_by_xpath('//*[#id="fsl"]/a[3]')
browser.implicitly_wait(5)
elm.click()

So the issue ended up being safari. For some reason the web driver safari was not allowing me to use .click. I switched to chrome and it worked.

Selenium not storing cookies

I'm working on trying to automate a game I want to get ahead in called pokemon vortex and when I login using selenium it works just fine, however when I attempt to load a page that requires a user to be logged in I am sent right back to the login page (I have tried it outside of selenium with the same browser, chrome).
This is what I have
import time
from selenium import webdriver
from random import randint
driver = webdriver.Chrome(r'C:\Program Files (x86)\SeleniumDrivers\chromedriver.exe')
driver.get('https://zeta.pokemon-vortex.com/dashboard/');
time.sleep(5) # Let the user actually see something!
usernameLoc = driver.find_element_by_id('myusername')
passwordLoc = driver.find_element_by_id('mypassword')
usernameLoc.send_keys('mypassword')
passwordLoc.send_keys('12345')
submitButton = driver.find_element_by_id('submit')
submitButton.submit()
time.sleep(3)
driver.get('https://zeta.pokemon-vortex.com/map/10')
time.sleep(10)
I'm using python 3.6+ and I literally just installed selenium today so it's up to date, how do I force selenium to hold onto cookies?

Using a pre-defined user profile might solve your problem. This way your cache will be saved and will not be deleted.
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--user-data-dir=C:/Users/user_name/AppData/Local/Google/Chrome/User Data")
driver = webdriver.Chrome(options=options)
driver.get("xyz.com")

Selenium Python - Webdriver

I am using selenium to crawl a javascript website, the issue is that, a Firefox browser opens up, but the call for the URL is not done. however, when I close the browser, it is then that call for URL is done and of course I get the missing driver exception. what do you think the issue comes from.
knowing that:
all programs are up-to-date
my solution works fine, in local, but when I try to deploy it on the server, I start having issues
Example: at my local machine, I run this script and everything goes smooth, however when I run it a server (Linux), only the browser opens up and no get URL is called
from selenium import webdriver
import time
geckodriver_path = r'.../geckodriver'
driver = webdriver.Firefox(executable_path= geckodriver_path)
time.sleep(3)
driver.get("http://www.stackoverflow.com")

I end up finding the solution :
from selenium import webdriver
import time
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
geckodriver_path = r'/path_to/geckodriver'
binary = FirefoxBinary(r'/usr/bin/firefox')
capabilities = webdriver.DesiredCapabilities().FIREFOX
capabilities["marionette"] = False
driver = webdriver.Firefox(firefox_binary=binary,
executable_path= geckodriver_path,
capabilities=capabilities)
time.sleep(3)
driver.get("https://stackoverflow.com/")
time.sleep(6)
driver.close()
# solution from:
# https://github.com/SeleniumHQ/selenium/issues/3884
# https://stackoverflow.com/questions/25713824/setting-path-to-firefox-binary-on-windows-with-selenium-webdriver