Develop Reference

Q: Can't keep clicking button by chorme web driver in python

Question:
How can I show all reviews by clicking "Show more reviews" button?
What did I do:
To scrape all reviews, I decided to Keep clicking until that button disappears.
But some new reviews didn't appear after clicking for 8 times (using while below).
I already checked xpath of the button but it didn't changed.
If I click the button manually without the driver, I can see new reviews.
import time, random
from selenium import webdriver
from latest_user_agents import get_random_user_agent
### ser user_agent
user_agent = get_random_user_agent()
options = webdriver.ChromeOptions()
options.add_argument("--user-agent=" + user_agent)
driverpath = "C:/Users/~~/chromedriver.exe"
driver = webdriver.Chrome(chrome_options=options,executable_path=driverpath)
### target url
url = "https://www.sony.co.uk/electronics/truly-wireless/wf-l900/reviews-ratings"
### open URL
driver.get(url)
time.sleep(5)
# accept cookies
driver.find_element_by_xpath('//*[#id="onetrust-accept-btn-handler"]').click()
# show all reviews
try:
while True:
driver.execute_script('window.scroll(0,1000000);')
driver.find_element_by_xpath('//*[#id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button').click()
time.sleep(random.randrange(2, 5, 1))
except:
print("---------------- finish showing all reviews ----------------")

Try going to the end of the webpage and then do it as the button might not been loaded fully.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
You might also try to press the button only when it is available.
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "myElement")))

You can try javascript click:
element = driver.find_element_by_xpath('//*[#id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button')
driver.execute_script("arguments[0].click();", element)
In this method, the scrolling shouldn't be necessary. See more info about the subject here.

Locating the Submit Button URL with Selenium Python

I'm trying to get to the URL behind the submit button:
from selenium import webdriver
browser = webdriver.Safari(executable_path = '/usr/bin/safaridriver')
#Fill in the required field
default_input = '111'
browser.get('https://trackapkg.com/aramex-tracking-number')
field = browser.find_element_by_xpath('//*[#id="ShipmentNumber"]')
field.send_keys(default_input)
url = browser.current_url
#Click submit button to get the new URL
browser.find_element_by_xpath("//span[#class='input-group-btn']").click();
while url == browser.current_url:
time.sleep(5)
url = browser.current_url
print(url)
Submitting works (though inconsistently), but apparently there's an issue with Xpath to the button itself so it's not clicked and the URL can't be caught. I've tried multiple variants:
browser.find_element_by_xpath("//*[#id='ShipmentNumber']/input[#class='btn btn-success']").click()
Or this:
browser.find_element_by_xpath("//input[#class='btn btn-success']").click()
But still can find the solution. I will appreciate your advice

Things to be noted down.
Use Explicit waits for more stability.
Use relative xpath, prefer css over xpath. Also I see the id and 'Name' is unique so use id and 'Name' rather than xpath. Please see illustration below.
Launch browser in full screen mode.
When we click on GO, it opens a new tab, in Selenium we have to switch to new tab first before accessing anything in the new tab.
Sample code:
from selenium import webdriver
browser = webdriver.Safari(executable_path = '/usr/bin/safaridriver')
browser.maximize_window()
browser.implicitly_wait(50)
wait = WebDriverWait(browser, 20)
driver.get("https://trackapkg.com/aramex-tracking-number")
default_input = '111'
field = wait.until(EC.visibility_of_element_located((By.ID, "ShipmentNumber")))
field.send_keys(default_input)
url = browser.current_url
print('current url', url)
#Click submit button to get the new URL
wait.until(EC.element_to_be_clickable((By.NAME, "track"))).click()
time.sleep(5)
#When we click on GO, it opens a new tab, in Selenium we have to switch to new tab first before accessing anything in the new tab.
all_handles = browser.window_handles
browser.switch_to.window(all_handles[1])
new_url = browser.current_url
print('new url', new_url)
Output :
current url https://trackapkg.com/aramex-tracking-number
new url https://www.aramex.com/in/en/track/shipment-details?q=c2hpcG1lbnRJZD0xMTEmcHJvZHVjdEdyb3VwPUZSVCZzaGlwbWVudFR5cGU9Y2FyZ293aXNlJnNlYXJjaE1vZGU9YnlJZCY%3d-nc4VTy84z6w%3d

The fault might be not waiting until the element is clickable. You could use
from selenium.webdriver.support.ui import WebDriverWait
submit = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//input[#class='btn btn-success']")))
submit.click()

If clicks on the button is intermittent on safari browser. Then here is one workaround. You can use enter key to click on the search box.
default_input = '111'
browser.get('https://trackapkg.com/aramex-tracking-number')
field = browser.find_element_by_xpath('//*[#id="ShipmentNumber"]')
field.send_keys(default_input)
field.send_keys(Keys.ENTER)
You need to import following library.
from selenium.webdriver.common.keys import Keys
Therefore your code block would be like this
default_input = '111'
browser.get('https://trackapkg.com/aramex-tracking-number')
field = browser.find_element_by_xpath('//*[#id="ShipmentNumber"]')
field.send_keys(default_input)
url = browser.current_url
field.send_keys(Keys.ENTER)# press enter key for search
time.sleep(1) # wait for 1 second to check
while url == browser.current_url:
time.sleep(5)
url = browser.current_url
print(url)

Is there a way to click a list of multiple buttons, one by one, in selenium?

I am a student working on a scraping project with selenium on python. I am trying to scrape data from multiple profiles, that are all formatted the same. There is a directory website with buttons that lead to all of the profiles, but the problem that I am having is being able to click every button one at a time because they are all formatted the same. My goal is to be able to open the 1st link in a new tab, scrape the data from that profile, then close the first profile's tab and move on too the second. I hope to make this process repeatable. Here is what I have so far:
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
import time
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path="MY PATH TO MY CHROME DRIVER")
driver.implicitly_wait(5)
driver.get("http://directory.bcsp.org")
buttons = driver.find_elements_by_link_text('View Profile')
Please let me know if you have any solutions to my problem. Thank you :)

"body" for to go down on the page
"profile_count" for get link
".get_attribute('href')" for to scrape the link from the "View Profile" button
"temp" the information in the old window does not switch to the new window so we need temp
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
driver = webdriver.Edge()
driver.get("https://directory.bcsp.org/")
count = int(input("Count : "))
body = driver.find_element_by_xpath("//body") #
profile_count = driver.find_elements_by_xpath("//div[#align='right']/a")
while len(profile_count) < count: # Get links up to "count"
body.send_keys(Keys.END)
sleep(1)
profile_count = driver.find_elements_by_xpath("//div[#align='right']/a")
for link in profile_count: # Calling up links
temp = link.get_attribute('href') # temp for
driver.execute_script("window.open('');") # open new tab
driver.switch_to.window(driver.window_handles[1]) # focus new tab
driver.get(temp)
# you can do
# what you want
# to do here
driver.close()
driver.switch_to.window(driver.window_handles[0])
driver.close()

Selenium Problem: New searches keep grabbing the element from an old search

I am running a loop that performs a search and grabs an element. The element on each search page appears to have the same CSS selector. However, it always prints the element associated with one search, the search I first began testing the script with. Not sure if this a CSS selector issue? Or a cookie issue perhaps?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import presence_of_element_located
EXE_PATH = r'C:\\geckodriver.exe'
tickers = ["bitcoin", "ethereum", "litecoin"]
for t in tickers:
with webdriver.Firefox(executable_path = EXE_PATH) as driver:
wait = WebDriverWait(driver, 10)
driver.get("https://coingecko.com/en")
driver.find_element_by_css_selector(".px-2").send_keys(t + Keys.RETURN)
first_result = wait.until(presence_of_element_located((By.CSS_SELECTOR, "div.text-3xl > span:nth-child(1)")))
price = first_result.get_attribute("innerHTML")
print(price)

I found the root cause of that issue my friend. The thing is that when you are sending the ticker in search field, it is taking some time to load the options as search field is an auto suggestive dropdown. But as per your script as soon as you are sending the ticker to the search field, you are hitting the enter button and the thing which is happening in the background is it is selecting bitcoin because if you will see in the trending bitcoin is at rank 1 and because of the lack of delay in between sending the ticker and hitting the enter, it is selecting bitcoin by default. I have modified a script you can view it below. If you don't want to use sleep then add web driver wait and wait for the desired option to be displayed in the search field drop down. Hope so that helps you. Please mark it is as accepted if you are happy with my answer.
tickers = ["bitcoin", "ethereum", "litecoin"]
for t in tickers:
with webdriver.Chrome(executable_path = EXE_PATH) as driver:
wait = WebDriverWait(driver, 10)
driver.get("https://coingecko.com/en")
driver.find_element_by_css_selector(".px-2").send_keys(t)
time.sleep(5)
driver.find_element_by_css_selector(".px-2").send_keys(Keys.RETURN);
first_result = wait.until(presence_of_element_located((By.CSS_SELECTOR, "div.text-3xl > span:nth-child(1)")))
price = first_result.get_attribute("innerHTML")
print(pric

Python - Selenium going through array

I would like to do automated script. I define an array in start of the program.
Later program is opening browser and search in google some specific word (for example apple), next program is clicking in first of string from array and close browser. Later is doing the same but it will click in second of word from array.
My code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome("C:/Users/Daniel/Desktop/chromedriver.exe")
driver.implicitly_wait(30)
driver.maximize_window()
hasla = ["ispot","myapple"]
for slogan in hasla:
driver.get("http://www.google.com")
search_field = driver.find_element_by_id("lst-ib")
search_field.clear()
search_field.send_keys("apple")
search_field.submit()
name = driver.find_element_by_link_text(slogan)
name.click()
driver.quit()
driver.implicitly_wait(10)
When Im starting this program from console in windows.
Program is opening browser, looking for apple click in ispot and clsoe browser but its not opening new browser and it not doing the same for next string in array. Any solution ?
In console i have this:
screen

You're exiting the browser in your for loop, so the second iteration can't do anything because there's not a browser open. You could try opening a new tab and closing the old one if you need to start fresh each time. Try this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome("C:/Users/Daniel/Desktop/chromedriver.exe")
driver.implicitly_wait(30)
driver.maximize_window()
hasla = ["ispot","myapple"]
for slogan in hasla:
driver.get("http://www.google.com")
search_field = driver.find_element_by_id("lst-ib")
search_field.clear()
search_field.send_keys("apple")
search_field.submit()
name = driver.find_element_by_link_text(slogan)
name.click()
# Save the current tab id
old_handle = driver.current_window_handle
# Execute JavaScript to open a new tab and save its id
driver.execute_script("window.open('');")
new_handle = driver.window_handles[-1]
# Switch to the old tab and close it
driver.switch_to.window(old_handle)
driver.close()
# Switch focus to the new tab
driver.switch_to.window(new_handle)
If you are closing the tab you won't be able to see the results. You might want to keep it open and just go to the new tab. In which case, just remove driver.close().
Alternatively, if you really want to completely close out the browser each time and reopen it, you just need to include the first three lines in your for-loop.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
hasla = ["ispot","myapple"]
for slogan in hasla:
driver = webdriver.Chrome("C:/Users/Daniel/Desktop/chromedriver.exe")
driver.implicitly_wait(30)
driver.maximize_window()
driver.get("http://www.google.com")
search_field = driver.find_element_by_id("lst-ib")
search_field.clear()
search_field.send_keys("apple")
search_field.submit()
name = driver.find_element_by_link_text(slogan)
name.click()
driver.quit()
To answer your second question:
First, import NoSuchElementException:
from selenium.common.exceptions import NoSuchElementException
Then replace your try/except with this:
try:
name = driver.find_element_by_link_text(slogan)
name.click()
except NoSuchElementException:
print('No such element')
driver.quit()
It's still going to close the browser and go to the next iteration if the element is found or not.