Develop Reference

Q: Can't keep clicking button by chorme web driver in python

Question:
How can I show all reviews by clicking "Show more reviews" button?
What did I do:
To scrape all reviews, I decided to Keep clicking until that button disappears.
But some new reviews didn't appear after clicking for 8 times (using while below).
I already checked xpath of the button but it didn't changed.
If I click the button manually without the driver, I can see new reviews.
import time, random
from selenium import webdriver
from latest_user_agents import get_random_user_agent
### ser user_agent
user_agent = get_random_user_agent()
options = webdriver.ChromeOptions()
options.add_argument("--user-agent=" + user_agent)
driverpath = "C:/Users/~~/chromedriver.exe"
driver = webdriver.Chrome(chrome_options=options,executable_path=driverpath)
### target url
url = "https://www.sony.co.uk/electronics/truly-wireless/wf-l900/reviews-ratings"
### open URL
driver.get(url)
time.sleep(5)
# accept cookies
driver.find_element_by_xpath('//*[#id="onetrust-accept-btn-handler"]').click()
# show all reviews
try:
while True:
driver.execute_script('window.scroll(0,1000000);')
driver.find_element_by_xpath('//*[#id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button').click()
time.sleep(random.randrange(2, 5, 1))
except:
print("---------------- finish showing all reviews ----------------")

Try going to the end of the webpage and then do it as the button might not been loaded fully.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
You might also try to press the button only when it is available.
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "myElement")))

You can try javascript click:
element = driver.find_element_by_xpath('//*[#id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button')
driver.execute_script("arguments[0].click();", element)
In this method, the scrolling shouldn't be necessary. See more info about the subject here.

Opening element(url) in new tabs in Python Selenium

Objective:
Need to open a webpage, find urls by class and then open each url in a new tab in chrome
What I tried:
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path="C:\\Users\xxx\Downloads\chromedriver_win32\chromedriver.exe")
driver.maximize_window()
driver.get("https://xxxxxxxx.com")
elements = driver.find_elements_by_class_name("post-comments")
for e in elements:
e.click()
driver.quit()
current output:
It's just opening the first url in the same parent window then stop there.. focus is not going to base url so it is not able to open second and other urls in new tab.
Please let me know your suggestions.
Update1: when I search for relative xpath, I came to know that I need to use regexp so that I can find all links containing "respond" in it.
For that I tried to use
driver.find_element_by_css_selector("a[href*='respond']]")
but it errored there is no find_element.
Update 2:
After some trials, I am able to achieve to the most output that I require with below code. But below code is working on first page only, when below code completes for first page, then there will be 'NextPage' button , I need to click on that to open that page, that will become new main page and I need to run the below piece of code again on that page. How to tell below piece of code once elements on this page done, click on 'NextPage' button and run below logic for that page. Please suggest
from selenium import webdriver
import time
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path="C:\\Users\xxx\Downloads\chromedriver_win32\chromedriver.exe")
driver.maximize_window()
driver.get("https://xxxxxxxx.com")
current_windows_handle = driver.current_window_handle
elements = driver.find_elements_by_css_selector('h2.post-title.entry-title>a')
for ele in (elements):
url = [ele.get_attribute("href")]
for x in url:
time.sleep(2)
driver.execute_script("window.open('');")
handles = driver.window_handles
driver.switch_to.window(handles[1])
driver.get(x)
driver.find_element_by_xpath("//span[#class='mb-text'][contains(.,'Posts')]").click()
time.sleep(2)
driver.close()
driver.switch_to.window(driver.window_handles[1])
time.sleep(5)
actualurl = driver.current_url
with open(r'c:\test\new.txt', 'a') as text_file:
text_file.write(actualurl + '\n')
driver.close()
time.sleep(5)
driver.switch_to.window(current_windows_handle)
driver.close()
driver.quit()

The error is valid, cause
driver.find_element_by_css_selector("a[href*='respond']]")
this is syntax error, but since it is wrapped inside "", compiler won't show it.
try this instead :
elements = driver.find_elements_by_css_selector("a[href*='respond']")
and then print the length like this :-
print(len(elements))
I am assuming that it will have some elements in it. and you can iterate it as well. Pleas see below, it should help you figure out how to simulate those actions.
current_windows_handle = driver.current_window_handle
elements = driver.find_elements_by_css_selector("a[href*='respond']")
for ele in elements:
time.sleep(2)
driver.execute_script("window.open('');") # open a new tab.
handles = driver.window_handles
driver.switch_to.window(handles[1])
driver.get(ele.get_attribute('href'))
time.sleep(2)
# Now it would have loaded a new url in new tab.
# do whatever you wanna do.
# Now close the new tab
driver.close()
time.sleep(2)
driver.switch_to.window(current_windows_handle)
updated 1 :
current_windows_handle = driver.current_window_handle
elements = driver.find_elements_by_css_selector("a[href*='respond']")
j = 0
for ele in range(len(elements)):
time.sleep(2)
elements = driver.find_elements_by_css_selector("a[href*='respond']")
driver.execute_script("window.open('');") # open a new tab.
handles = driver.window_handles
driver.switch_to.window(handles[1])
driver.get(elements[j].get_attribute('href'))
time.sleep(2)
j = j + 1
# Now it would have loaded a new url in new tab.
# do whatever you wanna do.
# Now close the new tab
driver.close()
time.sleep(2)
driver.switch_to.window(current_windows_handle)

Can't locate element by xpath in selenium?

I am learning python and selenium right now by making a small script that posts birthday comments to my facebook friends. I am able to successfully login and navigate to the "friends' birthday" modal, but then I am unable to comment in the textbox.
Here is the error I am getting:
selenium.common.exceptions.NoSuchElementException:
Message: no such element: Unable to locate element:
{"method":"xpath","selector":"//*[#id="u_i_6"]"}
I haven't had any issues finding other page elements via XPath so I am not sure what is causing the issue. I also tried finding the textbox to comment via classname, but had the same result. Can anyone offer some thoughts?
EDIT: forgot to paste code
from selenium import webdriver
from pynput.keyboard import Key, Controller
import time
import config
# fetch facebook password from separate file in the same directory
pw = config.secret['pw']
keyboard = Controller()
driver = webdriver.Chrome()
options = webdriver.ChromeOptions()
options.add_argument('--disable-extensions')
options.add_argument('--disable-notifications')
# navigate to facebook url
driver.get('https://facebook.com')
# input email address
email_input = driver.find_element_by_xpath('//*[#id="email"]')
email_input.send_keys(<omitted>)
# input pw
password_input = driver.find_element_by_xpath('//*[#id="pass"]')
password_input.send_keys(pw)
# click login button element
login_button = driver.find_element_by_xpath('//*[#id="u_0_b"]')
login_button.click()
home_button = driver.find_element_by_xpath('//*[#id="u_0_c"]/a')
# wait 8 seconds for browser popup before pressing esc to get out
time.sleep(8)
keyboard.press(Key.esc)
# check if there are any birthdays today
birthday_section = driver.find_element_by_xpath('//*[#id="home_birthdays"]/div/div/div/div/a/div/div/span/span[1]')
birthday_section.click()
first_comment = driver.find_element_by_xpath('//*[#id="u_i_6"]')
first_comment.send_keys('happy birthday!')
# click post button to post comment
# close browser window after actions are complete
time.sleep(5)
driver.close()

It is hard to answer without the code, but here are a couple of ideas:
Make sure you are using some sort of Selenium wait, implicit or explicit, so your code does not search for an element before this element appears on the page. You can add this code after driver.get() for implicit wait:
driver.implicitly_wait(5)
Also, it looks like IDs are dynamic on that page, I get a different one on my FB page. Try using this xpath to find the textarea:
"//form[contains(#action, 'birthday')]//textarea"
Hope this is helpful, good luck!

Python Selenium Webdriver Select Dropdown Value

I'm trying to automate the search process in this website: https://www.bcbsga.com/health-insurance/provider-directory/searchcriteria
The process involves clicking on the "Continue" button to search under the 'guest' mode. The next page has got a list of drop-down items to refine the search criteria. My code either produces the "Element not visible" exception (which I corrected by using a wait) or times out. Please help.
Here's my code:
# navigate to the desired page
driver.get("https://www.bcbsga.com/health-insurance/provider-directory/searchcriteria")
# get the guest button
btnGuest = driver.find_element_by_id("btnGuestContinue")
#click the guest button
btnGuest.click()
wait = WebDriverWait(driver,10)
#Find a Doctor Search Criteria page
element = wait.until(EC.visibility_of_element_located((By.ID,"ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance")))
lstGetInsurance = Select(element)
lstGetInsurance.select_by_value("BuyMyself$14States")
# close the browser window
#driver.quit()

you can use input search and key.return:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
divID = 'ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance_chosen'
inputID = 'ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance_chosen_input'
inputValue = 'I buy it myself (or plan to buy it myself)'
driver = webdriver.Chrome()
driver.get("https://www.bcbsga.com/health-insurance/provider-directory/searchcriteria")
driver.find_element_by_id("btnGuestContinue").click()
driver.implicitly_wait(10)
driver.find_element_by_id(divID).click()
driver.find_element_by_id(inputID).send_keys(inputValue)
driver.find_element_by_id(inputID).send_keys(Keys.RETURN)
time.sleep(6)
driver.close()

Click and view more pages with Selenium in Python

Problem
I need to go into all the top user profile in this page using Selenium.
The Top users profile are located in the right of the page.
What i've done
self.driver.get(response.url)
user_list = self.driver.find_elements_by_xpath('//table[contains(#class,"W-100 Bc-c")]/tbody/tr')
for single_user in user_list:
single_user.find_element_by_xpath('.//td/a').click()
time.sleep(3)
But I get this error message:
WebDriverException: Message: unknown error: Element is not clickable
at point (865, 685). Other element would receive the click:
<div id="MouseoverMask" class="End-0 Start-0 T-0 B-0"></div>
Info
Python 2.7.10
Selenium 2.48
Pycharm
EDIT+
I try to make a print of the name and it works:
print(str( single_user.find_element_by_xpath('.//td/a').text ) )
But the click() no.

if you sure that the object you get is the right one, often the problem is:
The object is not visible
Page was not fully loaded when you try to click on the object.
So just take a look on the Wait method provided by Selenium and be sure your object is visible
In order to wait an element to be clickable :
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID,'someid')))
In your case, you can try to find each element with the id you got and click on it :
self.driver.get(response.url)
user_list = self.driver.find_elements_by_xpath('//table[contains(#class,"W-100 Bc-c")]/tbody/tr')
for single_user in user_list:
id = single_user.find_element_by_xpath('.//td/a').get_attribute("id")
self.driver.find_elements_by_id(id).click()
time.sleep(3)

I don't see any error at my end but after first click web elements are changed so you will not get the next web element as captured earlier in xpath. By the way try below code-
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.get('https://answers.yahoo.com/dir/index/discover?sid=396545663')
user_list = driver.find_elements_by_xpath('//table[contains(#class,"W-100 Bc-c")]/tbody/tr')
lnks = [i.find_element_by_xpath('.//td/a').get_attribute('href') for i in user_list]
for single_user in lnks:
driver.get(single_user)
time.sleep(3)