I tried to accept cookie which are located in an iframe. But none of what i tried work. I'm open to advice and correct way of doing it
Here is a screenshot of waht the site looks like :
Here is what my current code looks like :
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import time
import datetime
import re
def scrap_ouestfrance_immo(driver):
URL = 'https://immobilier.lefigaro.fr/annonces/annonce-50413740.html'
driver.get(URL)
wait = WebDriverWait(driver, 6)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[#style='border: 0px none; min-width: 100%; position: fixed; right: 0px; bottom: 0px; height: 100%; max-height: 100%; z-index: 2147483647']'")))
accept_all_button = driver.find_element(By.CSS_SELECTOR, 'button[class="sc-1epc5np-0 dnGUzk sc-f7uhhq-2 coEmEP button button--filled button__acceptAll"]')
accept_all_button.click()
driver.switch_to.parent_frame()
print('accepted cookies')
#Main
if __name__ == '__main__':
#opening browser
chrome_options = webdriver.ChromeOptions()
#chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox') # required when running as root user. otherwise you would get no sandbox errors.
driver = webdriver.Chrome(
# driver_path='/home/dev/chromedriver',
options=chrome_options,
service_args=[
'--verbose',
# '--log-path=/tmp/chromedriver.log'
]
)
scrap_ouestfrance_immo(driver)
#driver.quit()
I tried to use Xpath, CSS Selector to switch to the iframe but i can't manage to do it.
The iframe doesn't have specific class o Id. I only see the style to switch to it.
But i maybe miss something
Thanks everyone
The XPath expression you have used to locate and switch to the iframe is incorrect. Try the below XPath expression instead:
(//iframe)[2]
Code should look like below:
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "(//iframe)[2]")))
XPath expression explanation: Basically this XPath will find the second iframe in the DOM structure.
Also change the code which locates Accept All button to below:
accept_all_button = driver.find_element(By.XPATH, '//span[contains(text(),"TOUT ACCEPTER")]')
Use the following xpath to identify the iframe element and following css selector to cookie button.
After switching the frame wait for element_to_be_clickable()
driver.get("https://immobilier.lefigaro.fr/annonces/annonce-50413740.html")
wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//div[#id='appconsent']/iframe")))
accept_all_button =wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button.button--filled.button__acceptAll')))
accept_all_button.click()
Related
I'm having trouble performing a click operation on a website. I'm getting a error message NoSuchElementException, but I'm not sure why because I got the class name from the site.
What am I missing?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
s = Service('C:/Program Files (x86)/chromedriver.exe')
chromeOptions = Options()
chromeOptions.headless = False
driver = webdriver.Chrome(service=s, options=chromeOptions)
list_data = []
def initialize_browser():
driver.get("https://virtualracingschool.appspot.com/#/Home")
print("starting_Driver")
click_button = driver.find_element(By.CLASS_NAME, "white-text")
driver.implicitly_wait(15)
click_button.click()
initialize_browser()
Site & Code:
I tried referencing some documents from the selenium site and it mentions for a format
<p class="content">Site content goes here.</p>`
write the code:
content = driver.find_element(By.CLASS_NAME, 'content')`
I felt like I did this properly but my site has
<a class="white-text" style="" ...>
<span>Login</span>
</a>
format. Is the <a> and "style" element hindering my code?
Use the below XPath expression:
//span[text()='Login']
Your code should look like this:
click_button = driver.find_element(By.XPATH, "//span[text()='Login']")
Below is the inspect element by this XPath for your reference:
As per your code trials:
click_button = driver.find_element(By.CLASS_NAME, "white-text")
By.CLASS_NAME, "white-text" identifies four elements within the HTML DOM:
Hence you see the see the error.
Solution
To click on the clickable element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get('https://virtualracingschool.appspot.com/#/Home')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.white-text > span"))).click()
Using XPATH:
driver.get('https://virtualracingschool.appspot.com/#/Home')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='white-text']//span"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser snapshot:
The HTML is: <pre style="word-wrap: break-word; white-space: pre-wrap;">{ "name": "RATE_LIMIT_REACHED", "message": "Too many requests. Blocked due to rate limiting.", "debugId": "f518054e243d2" }</pre>
Can you please help me out with this one?
Try using this XPath locator:
"//*[name()='pre' and(contains(#style,'RATE_LIMIT_REACHED'))]"
or
"//*[name()='pre'][contains(#style,'RATE_LIMIT_REACHED')]"
The above XPath expressions are actually the same.
I think we do not need to use name() here, so this should work as well
"//pre[contains(#style,'RATE_LIMIT_REACHED')]"
UPD
The first issue here resolved by this code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("--start-maximized")
s = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=s)
wait = WebDriverWait(driver, 10)
recaptchaTitle = "//*[contains(text(),'セキュリティチェック')]"
driver.get("https://www.paypal.com/signin?country.x=JP&locale.x=ja_JP")
wait.until(EC.element_to_be_clickable((By.ID, "email"))).send_keys("jleonhard-bassek#t-online.de")
try:
wait.until(EC.visibility_of_element_located((By.XPATH, recaptchaTitle)))
print("Recaptcha detected!")
except:
pass
wait.until(EC.element_to_be_clickable((By.ID, "btnNext"))).click()
try:
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "[name='recaptcha']")))
print("Recaptcha detected!!!")
except:
pass
wait.until(EC.element_to_be_clickable((By.ID, "password"))).send_keys("mistery69")
wait.until(EC.element_to_be_clickable((By.ID, "btnLogin"))).click()
try:
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "[name='recaptcha']")))
print("Recaptcha detected!!!!!!!!!")
except:
pass
Well, that depends on where the element is situated in relation to the rest of the document. If it's the only element with a pre tag, you can simply use this:
from selenium.webdriver.common.by import By
# ...
elem = driver.find_element(By.TAG_NAME, "pre")
If the page is more complex than that, you can use an XPath such as the one suggested by Prophet.
#duyanhhz you can try the following XPath. Since the "RATE_LIMIT_REACHED" is part of visible text and not part of the style attribute of pre HTML tag:
//pre[contains(text(), 'RATE_LIMIT_REACHED')]
So, I'm not very familiar with Python nor HTML
But I made this code so I could vote in a poll on a website, as far as I'm concerned there is nothing morally wrong with this idea because it is encouraged to refresh your page so you can vote for the same person over and over again. I want to use the code to click a button.
Below is the code I've got so far
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.nfl.com/voting/rookies/")
So I've gotten as far as going onto the website but I'm not sure if the person I am going to vote for has an HTML ID/Class/Element, I'm not really sure how to find it with inspect element or if it even has an HTML ID/Class/Element. If I want to vote for someone like Justin Herbert of the LAC, how should my code look? How would I click the white button next to Justin with Python? Thank you!
From the driver.get we get a popup, an iframe to handle and we need to click on the div tag that responds to a player. You should be able to translate these steps to your player.
Popup:
<div class="slidedown-footer" id="slidedown-footer"><button class="align-right primary slidedown-button" id="onesignal-slidedown-allow-button">Allow</button><button class="align-right secondary slidedown-button" id="onesignal-slidedown-cancel-button">No Thanks</button><div class="clearfix"></div></div>
Iframe:
<iframe title="Content from Riddle" style="border: none; width: 100%; position: static; opacity: 1; height: 539px !important;" src="https://www.riddle.com/a/288390"></iframe>
Player:
<span class="list-title ng-binding" ng-bind-html="item.title">Salvon Ahmed, MIA</span>
For this we use the following:
driver.get("https://www.nfl.com/voting/rookies/")
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#onesignal-slidedown-cancel-button"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[#title='Content from Riddle']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Salvon Ahmed, MIA']/parent::div/parent::div/div[2]/div"))).click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I am trying to get selenium to scroll down to a button in a website that has the text 'Try it out!' inside the button.
My problem is that there are no uniquely ID'd elements around the button to which I could scroll the view to. In addition, when I inspect the website with dev tools and search from the text 'Try it out!' in the HTML I get 72 results. I figured out that I need the 18th button but I am unable to get the browser to scroll to the button. Instead I get an error saying "The provided double value is non-finite".
Could you please look at the code below and give me an explanation to why I the browser is not scrolling down to the button?
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.action_chains import ActionChains
import pathlib
# Get path to chromedriver
file_path = pathlib.Path(__file__).parent.absolute()
chromedriver_path = str(file_path)+"\\chromedriver.exe"
class Scraper:
def __init__(self):
# Open website
self.driver = webdriver.Chrome(chromedriver_path)
print(self.driver)
self.driver.get(
"https://flespi.io/gw/#/tags/!/devices/get_devices_dev_selector_messages")
sleep(5)
# Get the 18th button that says 'Try it out!'. Position()=17 because starts with 0.
element = self.driver.find_element_by_xpath(
'(//input[#value="Try it out!"])[position()=17]')
# Scroll to the button and click it
actions = ActionChains(self.driver)
actions.move_to_element(element).perform()
element.click()
sleep(5)
Scraper()
To grab the Try it out button and click on it first create a webdriver wait to wait for the element to be clickable.
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[#id='devices_get_devices_dev_selector_messages_content']/form/div[3]/input")))
element.click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Please try the below code.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
action = ActionChains(driver)
driver.get('https://flespi.io/gw/#/tags/!/devices/get_devices_dev_selector_messages')
Try_btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#devices_get_devices_dev_selector_messages_content > form > div.sandbox_header > input')))
action.move_to_element(Try_btn).click().perform()
This code works for me.
This error message...
Failed to execute 'elementsFromPoint' on 'Document': The provided double value is non-finite
...implies that the WebDriver instance was unable to find the element for one or the other reasons:
The element haven't loaded properly when you tried to interact with it.
Element is within an <iframe> / <frame>
The style attribute of the element contains display: none;
Element is within an shadow DOM
You can find a relevant detailed discussion in javascript error: Failed to execute 'elementsFromPoint' on 'Document': The provided double value is non-finite
This use-case
Among the 72 elements with text as Try it out! i.e.
<input class="submit" type="submit" value="Try it out!" data-sw-translate="">
barring the desired element all the other 71 elements have an ancestor with style attribute set as display: none; overflow: hidden;. Where as only the desired element is having style attribute set as overflow: hidden;.
to click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#devices_get_devices_dev_selector_messages_content input[value='Try it out!']"))).click()
Using XPATH:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='devices_get_devices_dev_selector_messages_content']//input[#value='Try it out!']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
On this page 'https://www.nj.gov/health/cd/topics/covid2019_dashboard.shtml', I try to get the number of New Jersey Positives (upper right of the dashboard).
positiveCount = driver.find_elements_by_xpath("//text[#style='stroke-width: 2; font-size: 160px; line-height: normal;']")
print len(positiveCount)
it always show 0.
What did I do wrong? Thanks.
The data is wrapped in iframe so you have to switch to the iframe first and then get the details.
driver.switch_to.frame(0)
driver.find_element_by_css_selector('text').text
Result:
'6,876'
Screenshot:
Please find solution
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(executable_path=r"C:\New folder\chromedriver.exe")
url = 'https://www.nj.gov/health/cd/topics/covid2019_dashboard.shtml'
driver.get(url)
driver.maximize_window()
iframe = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.TAG_NAME, 'iframe')))
driver.switch_to.frame(iframe)
svg = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.CSS_SELECTOR, 'text')))
print svg.text