I am currently trying to crawl a website using selenium.
I have a table with elements and I want to click on each element, open the link in a new window/tab, process this window/tab, close it and click on the next element etc. So far I can click on the elements of the table, open the link and process the page.
Unfortunately I can not open the link in a new window or tab.
I can not send keyboard commands to the elements.
I have also tried
action = ActionChains(self.driver)
action.move_to_element(ele)\
.key_down(Keys.SHIFT)\
.click(ele)\
.key_up(Keys.SHIFT)\
.perform()
This somehow does only open the link in the current window (I dont't get a new window handle, still just one).
I would be very grateful for any help.
Edit: Also I am not able to open the link in a new window/tab manually with the browser.
To access the URL e.g. https://www.google.co.in then to click on an element e.g. link with text as Gmail to open the link in a new window/tab you can use action_chains class as per the solution below :
Code Block :
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://www.google.co.in')
print("Page Title is : %s" %driver.title)
mail_link = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Gmail")))
ActionChains(driver).key_down(Keys.CONTROL).click(mail_link).key_up(Keys.CONTROL).perform()
Browser Snapshot :
As Gmail is an 'a' tag, first get that a tag like below and perform an actionchain
from selenium.webdriver.common.action_chains import ActionChains
gmailtag = webdriver.find_element_by_xpath('path to gmail') # you can use any other methods to get it.
#Now you can run this to open the gmailtag in a new tab
ActionChains(webdriver).key_down(Keys.CONTROL).click(gmailtag).perform()
# this also worked for me
gmailtag.send_keys(Keys.CONTROL+Keys.ENTER)
It would open Gmail in a new tab. It worked for me.
Related
I have a python script, It look like this.
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from os import path
import time
# Tried this code
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)
browser = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=chrome_options)
links = ["https://www.henleyglobal.com/", "https://markets.ft.com/data"]
for link in links:
browser.get(link)
#WebDriverWait(browser, 20).until(EC.url_changes(link))
#How do I disable/Ignore/remove/escape this "Accept all cookie" popup and then access the website to scrape data?
browser.quit()
So each website in the links array displays an "Accept all cookie" popup after navigating to the site. check the below image.
I have tried many ways nothing works, Check the one after imports
How do I exit/pass/escape this popup and then access the website to scrape data?
If you open your page in a new browser you'll note the page fully loads, then, a moment later your popup appears. The default wait strategy in selenium is just that the page is loaded.
One way to handle this is to simply inspect the page and find the xpath of the popup window. The below code should work for that.
browser.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS)
if link == 'https://www.henleyglobal.com/':
browser.findElement(By.XPATH("/html/body/div[7]/div/div/div/div[2]/div/div[2]/button[2]")).click()
else:
browser.findElement(By.XPATH("/html/body/div[4]/div/div/div[2]/div[2]/a")).click()
The code is waiting until the element of the pop-up is clickable and then clicking it.
For unknown sites you could try:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-notifications")
webdriver.Chrome(os.path.join(path, 'chromedriver'), chrome_options=chrome_options)
generally, you can not use some universal locator that will match the "Accept cookies" buttons for each and every web site in the world.
Even here, you have 2 different sites and the elements you need to click are totally different on these sites.
For https://www.henleyglobal.com/ site the correct locator may be something like this CSS Selector .confirmation button.primary-btn while for https://markets.ft.com/data site I'd advise to use CSS Selector .o-cookie-message__actions a.o-cookie-message__button.
These 2 elements are totally different: the first one is button while the second is a, they have totally different class names and all other attributes.
You may thing about the Accept text. It seems to be common, so you could use this XPath //*[contains(text(),'Accept')] but even this will not work since on the first page it matches 2 elements while the accept cookies element is the second between them...
So, there is no General locators, you will have to define separate locators for each page.
Again, for https://www.henleyglobal.com/ I would prefer
driver.find_element(By.CSS_SELECTOR, ".confirmation button.primary-btn").click()
While for the second page https://markets.ft.com/data I would prefer this
driver.find_element(By.CSS_SELECTOR, ".o-cookie-message__actions a.o-cookie-message__button").click()
Also, generally we always use WebDriverWait expected_conditions explicit waits, so the code will be as following:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
# for the first page
wait.until(EC.element_to_be_clickable((By.XPATH, ".confirmation button.primary-btn"))).click()
# for the second page
wait.until(EC.element_to_be_clickable((By.XPATH, ".o-cookie-message__actions a.o-cookie-message__button"))).click()
There is a link embedded in a web element in the Main Tab, I want to open that link in a new tab in the same window using Selenium Webdriver and python.
Perform some tasks in the new tab and then close that tab and return back to the main tab.
Manually we will do this by right-clicking on the link and then select "open in new tab" to open that link in new tab.
I am new to Selenium. I am using Selenium and BeautifulSoup to web scrape. I only know how to click on a link. I have read through many posts but couldn't find a proper answer
url = "https://www.website.com"
path = r'path_to_chrome_driver'
drive = webdriver.Chrome(path)
drive.implicitly_wait(30)
drive.get(url)
source = drie.page_source
py_button = driver.find_element_by_css_selector("div[data-res-position = '1']")
py_button.click()
I expect the link in div[data-res-position = '1'] to open in a new tab
As there is a link embedded within in the webelement in the Parent Tab, to open the link in a New Tab in the same window using Selenium and Python you can use the following solution:
To demonstrate the workflow the url https://www.google.com/ was opened in the Parent Tab and then open in new tab functionalty is implemented through ActionChains methods key_down(), click() and key_up() methods.
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.google.com/")
link = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Gmail")))
ActionChains(driver).key_down(Keys.CONTROL).click(link).key_up(Keys.CONTROL).perform()
Note: You need to replace (By.LINK_TEXT, "Gmail") with your desired locator e.g. ("div[data-res-position = '1']")
Browser Snapshot:
You can find a relevant Java based solution in Opening a new tab using Ctrl + click combination in Selenium Webdriver
Update
To shift Selenium's focus to the newly opened tab you can find a detailed discussion in Open web in new tab Selenium + Python
This is trick can be achieve if your locator return a correct href.
Try this first:
href = driver.find_element_by_css_selector("div[data-res-position = '1']").get_attribute("href")
print(href)
If you get correct href, you can do:
#handle current tab
first_tab = driver.window_handles[0]
#open new tab with specific url
driver.execute_script("window.open('" +href +"');")
#hadle new tab
second_tab = driver.window_handles[1]
#switch to second tab
driver.switch_to.window(second_tab)
#switch to first tab
driver.switch_to.window(first_tab)
Hope this helps.
Following import:
selenium.webdriver.common.keys import Keys
You have to send key:
py_button.send_keys(Keys.CONTROL + 't')
I am trying to do a tutorial and learn Selenium in python however i cant seem to get Selenium to click the "Checkout" button using "element_to_be_clickable((By.XPATH".
I am using:
Python v3.9
Chrome v87
This is the URL i am practicing on:
https://www.aria.co.uk/myAria/ShoppingBasket
And this is my current code for the clicking:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time
# Open Chromedriver
driver = webdriver.Chrome(r"C:\Users\Ste1337\Desktop\chromedriver\chromedriver.exe")
# Open webpage
driver.get("https://www.aria.co.uk/SuperSpecials/Other+products/ASUS+ROG+Pugio+2+Wireless+Optical+RGB+Gaming+Mouse?productId=72427")
#https://www.aria.co.uk/Products/Components/Graphics+Cards/NVIDIA+GeForce/GeForce+RTX+3060+Ti/Palit+GeForce+RTX+3060+Ti+Dual+8GB+GPU?productId=73054
# Click "Add to Basket" or refresh page if out of stock
try:
element = WebDriverWait(driver, 1).until(EC.presence_of_element_located((By.XPATH, "Out of Stock!")))
time.sleep(5)
browser.refresh()
except:
button = driver.find_element_by_id("addQuantityButton")
button.click()
basket = WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.ID, "basketContent")))
basket.click()
checkout = WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH("//img[contains(#src,'/static/images/checkoutv2.png.png')]"))).click()
I can see your xpath is not correct.
Your Xpath should be.
//img[contains(#src,'/static/images/checkoutv2.png')]
Your code should be.
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//img[contains(#src,'/static/images/checkoutv2.png')]"))).click()
The link you provided contains hCaptcha, which is actually responsible to check whether you are a human being or a bot. I guess that it's also the reason, why you can't click any of the items on the page, because Selenium actually is nothing less than a bot.
You first have to pass the test by clicking on the images, which are asked for.
I'm having trouble with the Selenium webdriver's click() feature in Python when I'm trying to click on buttons that only appear after you click on a parent button.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
url = "https://law.lexmachina.com/cases/?pending-from=2000-01-01&pending-to=2000-02-01&filters=true&tab=summary&view=analytics&cols=475"
driver.get(url)
driver = webdriver.Chrome()
elem0 = driver.find_element_by_id('export-icon-container') # this works
all_children_by_css = elem0.find_elements_by_css_selector("*") # this works, but doesn't click on the sub-button (XLS) one successfully when I run below...
all_children_by_css[0].click() # this just makes the parent button's little window appear and disappear, the same as elem0.click() does.
>>> all_children_by_css[0] # this is the webElement that I thought was for the XLS button
<selenium.webdriver.remote.webelement.WebElement (session="6b4a559408fa4d512f8596759d81eaf7",
element="d83be2ca-c879-4706-85ef-db7120d345a3")>
Basically, I want to export the XLS file via the Webdriver, so that later on I can do this on a loop with URLs of filters of the data.
I included annotated screenshots below, detailing the buttons I'm trying to click & the inspected code associated with them.
I would recommend to see if there an API on the page to export directly. if not, You would probably need dynamic wait. you can try something like below.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
url = "https://law.lexmachina.com/cases/?pending-from=2000-01-01&pending-to=2000-02-01&filters=true&tab=summary&view=analytics&cols=475"
driver.get(url)
#Click on Export Icon
elem0 = driver.find_element_by_id('export-icon-container').click()
#Wait for XLS option to show up
wait = WebDriverWait(driver, 10)
ExportOption= wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#data-action='export-item'][contains(text(),'XLS')]")))
ExportOption.click()
There is a link embedded in a web element in the Main Tab, I want to open that link in a new tab in the same window using Selenium Webdriver and python.
Perform some tasks in the new tab and then close that tab and return back to the main tab.
Manually we will do this by right-clicking on the link and then select "open in new tab" to open that link in new tab.
I am new to Selenium. I am using Selenium and BeautifulSoup to web scrape. I only know how to click on a link. I have read through many posts but couldn't find a proper answer
url = "https://www.website.com"
path = r'path_to_chrome_driver'
drive = webdriver.Chrome(path)
drive.implicitly_wait(30)
drive.get(url)
source = drie.page_source
py_button = driver.find_element_by_css_selector("div[data-res-position = '1']")
py_button.click()
I expect the link in div[data-res-position = '1'] to open in a new tab
As there is a link embedded within in the webelement in the Parent Tab, to open the link in a New Tab in the same window using Selenium and Python you can use the following solution:
To demonstrate the workflow the url https://www.google.com/ was opened in the Parent Tab and then open in new tab functionalty is implemented through ActionChains methods key_down(), click() and key_up() methods.
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
import time
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.google.com/")
link = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Gmail")))
ActionChains(driver).key_down(Keys.CONTROL).click(link).key_up(Keys.CONTROL).perform()
Note: You need to replace (By.LINK_TEXT, "Gmail") with your desired locator e.g. ("div[data-res-position = '1']")
Browser Snapshot:
You can find a relevant Java based solution in Opening a new tab using Ctrl + click combination in Selenium Webdriver
Update
To shift Selenium's focus to the newly opened tab you can find a detailed discussion in Open web in new tab Selenium + Python
This is trick can be achieve if your locator return a correct href.
Try this first:
href = driver.find_element_by_css_selector("div[data-res-position = '1']").get_attribute("href")
print(href)
If you get correct href, you can do:
#handle current tab
first_tab = driver.window_handles[0]
#open new tab with specific url
driver.execute_script("window.open('" +href +"');")
#hadle new tab
second_tab = driver.window_handles[1]
#switch to second tab
driver.switch_to.window(second_tab)
#switch to first tab
driver.switch_to.window(first_tab)
Hope this helps.
Following import:
selenium.webdriver.common.keys import Keys
You have to send key:
py_button.send_keys(Keys.CONTROL + 't')