Im trying to create automation for a cookie clicker website.
I need to click on elements (like the cursor element for example) on the website when they go from "blocked" to "unlocked" I have been trying for 2 days now and I have tried using the WebDriverWait but nothing is working no matter what my code does not detect when the element becomes available.
this is my code right now
import time
import ec as ec
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
Play = True
ser_obj = Service("\Progr\OneDrive\Documents\PythonFolder\chromedriver.exe")
driver = webdriver.Chrome(service=ser_obj)
driver.get(url="https://orteil.dashnet.org/cookieclicker/")
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.cc_btn.cc_btn_accept_all"))).click()
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.langSelectButton.title#langSelect-EN"))).click()
time.sleep(1)
Cookie = driver.find_element(By.CSS_SELECTOR, "#cookieAnchor #bigCookie")
while Play:
Cookie.click()
Cookie_number = (driver.find_element(By.XPATH,'//*[#id="cookies"]').text)
print(Cookie_number)
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="product0"]'))).click()
and for whatever reason I cannot click on the cookie unless I have a time.sleep() method called and I do not know why. I have tried using WebDriverWait to wait when the cookie becomes avaible to click, but nope, it wont run without the time.sleep().
Any help would be great.
I have tried using if statments with the .isDisplayed() function.
I have tried using "try-except" methods.
I have tried giving Play a value and then saying when that value reaches 0, check to see if the cursor is available to click.
I have tried using CSS Selectors and Xpath
Related
I have a python script, It look like this.
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from os import path
import time
# Tried this code
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)
browser = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=chrome_options)
links = ["https://www.henleyglobal.com/", "https://markets.ft.com/data"]
for link in links:
browser.get(link)
#WebDriverWait(browser, 20).until(EC.url_changes(link))
#How do I disable/Ignore/remove/escape this "Accept all cookie" popup and then access the website to scrape data?
browser.quit()
So each website in the links array displays an "Accept all cookie" popup after navigating to the site. check the below image.
I have tried many ways nothing works, Check the one after imports
How do I exit/pass/escape this popup and then access the website to scrape data?
If you open your page in a new browser you'll note the page fully loads, then, a moment later your popup appears. The default wait strategy in selenium is just that the page is loaded.
One way to handle this is to simply inspect the page and find the xpath of the popup window. The below code should work for that.
browser.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS)
if link == 'https://www.henleyglobal.com/':
browser.findElement(By.XPATH("/html/body/div[7]/div/div/div/div[2]/div/div[2]/button[2]")).click()
else:
browser.findElement(By.XPATH("/html/body/div[4]/div/div/div[2]/div[2]/a")).click()
The code is waiting until the element of the pop-up is clickable and then clicking it.
For unknown sites you could try:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--disable-notifications")
webdriver.Chrome(os.path.join(path, 'chromedriver'), chrome_options=chrome_options)
generally, you can not use some universal locator that will match the "Accept cookies" buttons for each and every web site in the world.
Even here, you have 2 different sites and the elements you need to click are totally different on these sites.
For https://www.henleyglobal.com/ site the correct locator may be something like this CSS Selector .confirmation button.primary-btn while for https://markets.ft.com/data site I'd advise to use CSS Selector .o-cookie-message__actions a.o-cookie-message__button.
These 2 elements are totally different: the first one is button while the second is a, they have totally different class names and all other attributes.
You may thing about the Accept text. It seems to be common, so you could use this XPath //*[contains(text(),'Accept')] but even this will not work since on the first page it matches 2 elements while the accept cookies element is the second between them...
So, there is no General locators, you will have to define separate locators for each page.
Again, for https://www.henleyglobal.com/ I would prefer
driver.find_element(By.CSS_SELECTOR, ".confirmation button.primary-btn").click()
While for the second page https://markets.ft.com/data I would prefer this
driver.find_element(By.CSS_SELECTOR, ".o-cookie-message__actions a.o-cookie-message__button").click()
Also, generally we always use WebDriverWait expected_conditions explicit waits, so the code will be as following:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
# for the first page
wait.until(EC.element_to_be_clickable((By.XPATH, ".confirmation button.primary-btn"))).click()
# for the second page
wait.until(EC.element_to_be_clickable((By.XPATH, ".o-cookie-message__actions a.o-cookie-message__button"))).click()
I am facing interesting issue with my **Web Scraping** use case. I need to get newest **Google Maps reviews**. I want to sort reviews by latest date. And all tutorials I am watching is in English, but in my native language the UI is different than in those tutorials.
I am able to click on the button using **Selenium** and button's **XPATH**, but I don't know how to change the sorting options from the visible drop menu.
# Click the sort button
driver.find_element_by_xpath('//*[#id="pane"]/div/div[1]/div/div/div[2]/div[8]/button').click()
select_by_visible_text() and select_by_value() doesn't work for me as I cannot select the button and doesn't work on div.
URL I am using : Link
To see my UI change to LITHUANIAN language.
First of all you have to learn how to create correct XPath locators.
Long XPath expressions are too fragile.
The "Sort Reviews" button locator instead of
//*[#id="pane"]/div/div[1]/div/div/div[2]/div[8]/button can be
//button[#aria-label='Sort reviews'] or
//button[#data-value='Sort']
After clicking this button, to sort reviews by latest date you can click this element: //li[#data-index='1']
So basically this will work:
driver.find_element_by_xpath("//li[#data-index='1']").click()
But since you need to wait for dialog to open after clicking the Sort button you need to utilize Expected Condition wait, as following:
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//li[#data-index='1']"))).click()
You will need the following imports for this:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
This code should work.
I added a webdriverwait after clicking on the 'Sort' button to wait until the all the options in the dropdown are visible, then clicked on 'Highest rating'.
//li[#role='menuitemradio'])[3] refers to the 3rd element in the dropdown which is the 'Higest rating'. I tried using the text to be specific and not count on element index, but somehow it is not working. But the below code does sort the reviews.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
import time
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get("https://www.google.com/maps/place/Senukai/#54.6832836,25.183965,12z/data=!4m11!1m2!2m1!1svilnius+senukai!3m7!1s0x46dd94055529fabf:0xb1132b0ad981d43b!8m2!3d54.7098368!4d25.2999662!9m1!1b1!15sCg92aWxuaXVzIHNlbnVrYWkiA4gBAVoRIg92aWxuaXVzIHNlbnVrYWmSARhidWlsZGluZ19tYXRlcmlhbHNfc3RvcmU")
print(driver.title)
time.sleep(5)
driver.find_element(By.XPATH, "//button[#data-value='Sort']").click()
WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//li[#role='menuitemradio']")))
driver.find_element(By.XPATH, "(//li[#role='menuitemradio'])[3]").click()
time.sleep(2)
driver.quit()
P.S I used time.sleep occasionally to make a quick code, but it would be a good practice to use WebdriverWait in lieu of time.sleep
I want to download the image at this site https://imginn.com/p/CXVmwujLqbV/ via the button. but i always fail.
this is the code i use.
driver.find_element_by_xpath('/html/body/div[2]/div[5]/a').click()
Well,
Check this post for downloading resource. Picture has 'src' attribute in 'img' tag, that holds it.
Also, (though it just might be simplification just for this question), do not hardcode your xpath. Learn to code nicely using "Page Object Pattern".
There are several possible problems here:
You need to add a delay / wait before accessing this element.
You have to scroll the page since the element you wish to click is initially out of the view.
You should improve you locator. It's highly NOT recommended to use absolute XPaths.
This should work:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time
driver = webdriver.Chrome(executable_path='chromedriver.exe')
driver.set_window_size(1920,1080)
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
driver.get("https://imginn.com/p/CXVmwujLqbV/")
button = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.downloads a")))
time.sleep(0.5)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(1)
#actions.move_to_element(button).perform()
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.downloads a"))).click()
I am trying to do a tutorial and learn Selenium in python however i cant seem to get Selenium to click the "Checkout" button using "element_to_be_clickable((By.XPATH".
I am using:
Python v3.9
Chrome v87
This is the URL i am practicing on:
https://www.aria.co.uk/myAria/ShoppingBasket
And this is my current code for the clicking:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time
# Open Chromedriver
driver = webdriver.Chrome(r"C:\Users\Ste1337\Desktop\chromedriver\chromedriver.exe")
# Open webpage
driver.get("https://www.aria.co.uk/SuperSpecials/Other+products/ASUS+ROG+Pugio+2+Wireless+Optical+RGB+Gaming+Mouse?productId=72427")
#https://www.aria.co.uk/Products/Components/Graphics+Cards/NVIDIA+GeForce/GeForce+RTX+3060+Ti/Palit+GeForce+RTX+3060+Ti+Dual+8GB+GPU?productId=73054
# Click "Add to Basket" or refresh page if out of stock
try:
element = WebDriverWait(driver, 1).until(EC.presence_of_element_located((By.XPATH, "Out of Stock!")))
time.sleep(5)
browser.refresh()
except:
button = driver.find_element_by_id("addQuantityButton")
button.click()
basket = WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.ID, "basketContent")))
basket.click()
checkout = WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH("//img[contains(#src,'/static/images/checkoutv2.png.png')]"))).click()
I can see your xpath is not correct.
Your Xpath should be.
//img[contains(#src,'/static/images/checkoutv2.png')]
Your code should be.
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//img[contains(#src,'/static/images/checkoutv2.png')]"))).click()
The link you provided contains hCaptcha, which is actually responsible to check whether you are a human being or a bot. I guess that it's also the reason, why you can't click any of the items on the page, because Selenium actually is nothing less than a bot.
You first have to pass the test by clicking on the images, which are asked for.
I am facing inconsistencies in Selenium execution.
Last line in the code snippet I pasted below doesn't execute consistently. Sometimes it works, sometimes it throws an error saying that element is not found. Doesn't Selenium "block" for the element to appear before attempting to execute the click? I generated it using Selenium IDE. What I am missing here?
self.driver.find_element(By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)").click()
self.driver.switch_to.default_content()
self.driver.find_element(By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText").click()
Selenium may not find elements if they happen to be loaded dynamically by JS and if you search for them before they are loaded.
You can try either an implicit wait or an explicit wait.
In case of implicit waiting, the docs say:
An implicit wait tells WebDriver to poll the DOM for a certain amount of time when trying to find any element (or elements) not immediately available.
You could do with something like:
from selenium import webdriver
driver = webdriver.Chrome()
driver.implicitly_wait(10) #wait and poll for 10 seconds
Whereas the explicit waiting means to explicitly specify the element which is to be waited for it to be available. As per the docs:
An explicit wait is a code you define to wait for a certain condition to occur before proceeding further in the code.
You can do this with something like:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
element1 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)")))
element1.click()
self.driver.switch_to.default_content()
element2 = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText")))
element2.click()
As you are using the line of code:
self.driver.switch_to.default_content()
Presumably you are switching Selenium's focus from a frame or iframe to the Top Level Content. Hence you need to induce WebDriverWait for the desired element to be clickable and you can use the following Locator Strategy:
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
Note:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant detailed discussions in:
How to send text to the Password field within https://mail.protonmail.com registration page?
How to switch between iframes using Selenium and Python?
Wait for the element to be loaded
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".dx-ellipsis:nth-child(2)"))).click()
self.driver.switch_to.default_content()
WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#PageContentPlaceHolder_TimeControlSplitter_TimeControlContent_TimesheetEntrySplitter_TimesheetDetailsMenu_DXI0_T > .dxm-contentText"))).click()
The number is how long the driver should spend looking for the element before moving on.