I am facing interesting issue with my **Web Scraping** use case. I need to get newest **Google Maps reviews**. I want to sort reviews by latest date. And all tutorials I am watching is in English, but in my native language the UI is different than in those tutorials.
I am able to click on the button using **Selenium** and button's **XPATH**, but I don't know how to change the sorting options from the visible drop menu.
# Click the sort button
driver.find_element_by_xpath('//*[#id="pane"]/div/div[1]/div/div/div[2]/div[8]/button').click()
select_by_visible_text() and select_by_value() doesn't work for me as I cannot select the button and doesn't work on div.
URL I am using : Link
To see my UI change to LITHUANIAN language.
First of all you have to learn how to create correct XPath locators.
Long XPath expressions are too fragile.
The "Sort Reviews" button locator instead of
//*[#id="pane"]/div/div[1]/div/div/div[2]/div[8]/button can be
//button[#aria-label='Sort reviews'] or
//button[#data-value='Sort']
After clicking this button, to sort reviews by latest date you can click this element: //li[#data-index='1']
So basically this will work:
driver.find_element_by_xpath("//li[#data-index='1']").click()
But since you need to wait for dialog to open after clicking the Sort button you need to utilize Expected Condition wait, as following:
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//li[#data-index='1']"))).click()
You will need the following imports for this:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
This code should work.
I added a webdriverwait after clicking on the 'Sort' button to wait until the all the options in the dropdown are visible, then clicked on 'Highest rating'.
//li[#role='menuitemradio'])[3] refers to the 3rd element in the dropdown which is the 'Higest rating'. I tried using the text to be specific and not count on element index, but somehow it is not working. But the below code does sort the reviews.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
import time
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get("https://www.google.com/maps/place/Senukai/#54.6832836,25.183965,12z/data=!4m11!1m2!2m1!1svilnius+senukai!3m7!1s0x46dd94055529fabf:0xb1132b0ad981d43b!8m2!3d54.7098368!4d25.2999662!9m1!1b1!15sCg92aWxuaXVzIHNlbnVrYWkiA4gBAVoRIg92aWxuaXVzIHNlbnVrYWmSARhidWlsZGluZ19tYXRlcmlhbHNfc3RvcmU")
print(driver.title)
time.sleep(5)
driver.find_element(By.XPATH, "//button[#data-value='Sort']").click()
WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//li[#role='menuitemradio']")))
driver.find_element(By.XPATH, "(//li[#role='menuitemradio'])[3]").click()
time.sleep(2)
driver.quit()
P.S I used time.sleep occasionally to make a quick code, but it would be a good practice to use WebdriverWait in lieu of time.sleep
Related
Im trying to create automation for a cookie clicker website.
I need to click on elements (like the cursor element for example) on the website when they go from "blocked" to "unlocked" I have been trying for 2 days now and I have tried using the WebDriverWait but nothing is working no matter what my code does not detect when the element becomes available.
this is my code right now
import time
import ec as ec
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
Play = True
ser_obj = Service("\Progr\OneDrive\Documents\PythonFolder\chromedriver.exe")
driver = webdriver.Chrome(service=ser_obj)
driver.get(url="https://orteil.dashnet.org/cookieclicker/")
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.cc_btn.cc_btn_accept_all"))).click()
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.langSelectButton.title#langSelect-EN"))).click()
time.sleep(1)
Cookie = driver.find_element(By.CSS_SELECTOR, "#cookieAnchor #bigCookie")
while Play:
Cookie.click()
Cookie_number = (driver.find_element(By.XPATH,'//*[#id="cookies"]').text)
print(Cookie_number)
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="product0"]'))).click()
and for whatever reason I cannot click on the cookie unless I have a time.sleep() method called and I do not know why. I have tried using WebDriverWait to wait when the cookie becomes avaible to click, but nope, it wont run without the time.sleep().
Any help would be great.
I have tried using if statments with the .isDisplayed() function.
I have tried using "try-except" methods.
I have tried giving Play a value and then saying when that value reaches 0, check to see if the cursor is available to click.
I have tried using CSS Selectors and Xpath
Trying to automate one thing for my work, which was choosing one option from the dropdown list on the website below:
https://interparking-pl.my.site.com/abonament/s/?id=a0A58000000D7pZ
The Selenium automation didn't work in that case. After writing such code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_driver_path = r"C:/Users/.../Projects/chromedriver.exe"
service = Service(executable_path=chrome_driver_path)
driver = webdriver.Chrome(service=service)
driver.get("https://interparking-pl.my.site.com/abonament/s/?id=a0A58000000D7pZ")
wait = WebDriverWait(driver, 10)
abo_button = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="combobox-button-53"]')))
After executing I've got a message:
TimeoutException
In case of finding element by tag name or any other options, the following message pops up:
Message: no such element: Unable to locate element
The list is built on button tags and has a structure of lightning-basecombobox. It looks like there is no possibility to click on the dropdown list and choose the required option automatically.
Is it needed to do something different with such stuff?
What I expect is to use Selenium to choose between the options in the list.
The combobox is a <button> element.
Additionally the id i.e. combobox-button-53 is dynamically generated and is bound to chage sooner/later. They may change next time you access the application afresh or even while next application startup. So can't be used in locators.
Solution
To click element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get("https://interparking-pl.my.site.com/abonament/s/?id=a0A58000000D7pZ")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[name='subscriptionsTypeId']"))).click()
Using XPATH:
driver.get("https://interparking-pl.my.site.com/abonament/s/?id=a0A58000000D7pZ")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#name='subscriptionsTypeId']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot;
When I load the page, I get the element ID (XPATH) as following. //*[#id="combobox-button-51"]
Maybe the number 51 isn't always the same? In that case, try:
//*[starts-with(#id,'combobox-button-5')]
Or just use //*[#name="subscriptionsTypeId"] , as another answer here allready mentioned.
I currently wish to select a value from a drop-down menu in python with selenium.
The site is the following: https://www.palyazat.gov.hu/tamogatott_projektkereso?fbclid=IwAR3rmPVj-YAVoMTs2Vodj7JKTVIAZkbTiZ9z4b0j04mq2ThECw5kQOI1p7M
My current code
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Safari(executable_path = '/usr/bin/safaridriver')
wait = WebDriverWait(driver, 20)
driver.get("https://www.palyazat.gov.hu/tamogatott_projektkereso?fbclid=IwAR3rmPVj-YAVoMTs2Vodj7JKTVIAZkbTiZ9z4b0j04mq2ThECw5kQOI1p7M")
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[#class='form-group' and .//label[#for='programok']]//div[contains(#class,'css-1wy0on6')]"))).click()
time.sleep(1)
current_program = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(#class,'css-26l3qy-menu')]")))
current_program.click()
My issue is that I am not able to find out the needed xpath code for the pop-upped drop-down menu, since I can see it in developer view only if the list is clicked, but I can see then only that "//div[contains(#class,'css-26l3qy-menu')]" is a suitable xpath for the menu. (see the enclosed picture)
But it chooses a random value. How specify which value I want to select?
I'm trying to click on the drop down menu but with no luck .
the menu is activated by javascript .
I tried to click on the link inside the parent div but nothing happens
here is some code :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=options)
driver.get("https://www.energisa.com.br/Paginas/login.aspx")
select_element = Select(driver.find_element_by_id('ddlEstado'))
select_element.select_by_value('MG')
# select by visible text
select_element.select_by_visible_text('MG')
As per the your question the website https://www.energisa.com.br/Paginas/login.aspx the dropdown menu is not with in a Select tag, so Select class won't work here.
Once the url is accessed, you need to induce WebDriverWait for the desired element to be clickable and you can use the following solution:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.energisa.com.br/Paginas/login.aspx")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[#class='estado']/div[#class='select2-container' and #id='s2id_ddlEstado']"))).click()
driver.find_element_by_xpath("//ul[#class='select2-results' and #id='select2-results-1']//li/div[normalize-space()='MG']").click()
The dropdown you are trying to click is not actually a SELECT element so you can't use the Select class. The SELECT you are trying to click is just a backing element but it's invisible so you can't interact with it.
To make this work, you will need to click the dropdown element to expose the options and then click the desired option.
driver.find_element_by_css_selector("#s2id_ddlEstado > a").click()
driver.find_element_by_xpath("//ul[#id='select2-results-1']/li[.='MG']").click()
This is untested code, so you may need to add a wait...
The one which appears like a select_list is not a select_list, the purpose of this kind of select_list is, we can write into the text_field to pick up the elements from the huge list, if you type 'M', then all the options which has M will be displayed.
Write the following code, it would work.
WebDriverWait(driver,10).until(EC.invisibility_of_element_located((By.ID,"loadingContent")))
driver.find_element_by_id("s2id_ddlEstado").click
driver.find_element_by_xpath("//ul[#id='select2-results-1']//div[text()='MG']").click
I am working with this code code:
I am trying to find text "Branch Selection" within a web page to click on it.
So I do this:
driver.find_elements_by_xpath("//*[contains(text(), 'Branch Selection')]").click()
It doesn't throw an error, but the click doesn't happen. What am I doing wrong?
If it clicks but nothing happens, first try adding a sleep like sleep(5) to help debug before clicking to see if its because Selenium thought the page loaded when it actually didn't finish loading. If you can click after sleep of 5 seconds then you need to use WebDriverWait and EC as #DebanjanB had shown. In worst case scenario you'll have to use a sleep in your code but try to get the sleep to as short as possible.
Otherwise you might have multiple elements on the page with Branch Selection text such as in the meta tags. Try using XPATH example below to isolate the XPATH look up:
"//button[.//*[contains(text(), 'Branch Selection')]]"
or if there's more than one phrase on page containing the text, use following to select exact text
"//button[.//*[text()='Branch Selection']"
This selects the button element that has a child element with the text you're looking for. More XPATH details here: https://devhints.io/xpath
As per the HTML provided to click on the element with text as Branch Selection you need to induce WebDriverWait for the element to be clickable as follows :
CSS_SELECTOR :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# lines of code
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.item.item-block.item-md div.input-wrapper>ion-label.label.label-md[id^=lbl-]"))).click()
XPATH :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# lines of code
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='item item-block item-md']//div[#class='input-wrapper']/ion-label[#class='label label-md' and starts-with(#id,'lbl-') and contains(.,'Branch Selection')]"))).click()