Develop Reference

Selenium only locates element when I open developer tools in chrome [duplicate]

This question already has answers here:
Click on ember.js enabled element using Selenium
(5 answers)
Closed 1 year ago.
At this stage, all the script aims to do is to locate the search bar in a digital library, send the name of the resource I am looking for, click on it and get the current url. The first bit (sending keys to search bar) works fine. But I have noticed that the second try/finally block only executes when I right click and inspect any element on the page. If I don't, I get the following error message :
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="ember767"]"}
My code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
PATH = "/Applications/chromedriver"
ser = Service(PATH)
driver = webdriver.Chrome(service=ser)
driver.maximize_window()
driver.implicitly_wait(20)
driver.get("https://browzine.com/libraries/1374/subjects")
print("Enter targeted Journal name:")
targeted_journal = input()
wait = WebDriverWait(driver, 10)
try:
button = driver.find_element(By.ID, "ember648")
driver.implicitly_wait(10)
ActionChains(driver).move_to_element(button).click(button).perform()
button.send_keys(targeted_journal)
button.send_keys(Keys.RETURN)
finally:
pass
# step 2 : click targeted journal and get current url
try:
driver.implicitly_wait(20)
click_journal = driver.find_element(By.ID, "ember767")
ActionChains(driver).move_to_element(click_journal).click(click_journal).perform()
targeted_url = driver.current_url
finally:
pass
print(targeted_url)
driver.quit()
Update - answer found
The ID initially collected for the targeted element was a dynamic one. It was only as such when the developer tools window was open. Therefore the script was only able to detect that particular ID when "Inspect element" was clicked as it was the only case in which the targeted element was assigned this particular ID ("ember767" in my case).
To avoid this issue, an Xpath was used to locate the element instead. See Mayank Shukla's answer below.

In some cases, element's ID or other attributes changes or attribute's value appears on mouse hover. So it's preferred to use xpath for locating such elements
In this scenario, I think the element's id is changing after you are hovering the mouse. Try using xpath and let me know if that works. Because might be after browser refresh this is happening i.e. element's id is changing.

Python web scraping with Selenium on Dynamic Page - Issue with looping to next element

I am hoping someone can please help me out and put me out of my misery. I have recently started to learn Python and wanted to challenge myself with some web-scraping.
Over the past couple of days I have been trying to web-scrape this website (https://ebn.eu/?p=members). On the website, I am interesting in:
Clicking on each logo image which brings up a pop-up
From the pop-up scrape the link which is behind the text "VIEW FULL PROFILE"
Move to the next logo and do the same for each
I have managed to get Selenium up and running but the issue is that it keeps opening the first logo and copying the same link as opposed to moving to the next one. I have tried in various different ways but came up against a brick wall.
My code so far:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
PATH = "/home/ed/Documents/python/chromedriver" # location of the webdriver - amend as requried
url = "https://ebn.eu/?p=members"
driver = webdriver.Chrome(PATH)
driver.get(url)
member_list = []
# Flow to open page and click on link to extract href
members = driver.find_elements_by_xpath('//*[#class="projectImage"]') # Looking for the required class - the image which on click brings up the info
for member in members:
print(member.text) # to see that loop went to next iteration
member.find_element_by_xpath('//*[#class="projectImage"]').click()
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'VIEW FULL PROFILE')))
links = driver.find_element_by_partial_link_text("VIEW FULL PROFILE")
href = links.get_attribute("href")
member_list.append(href)
member.find_element_by_xpath("/html/body/div[5]/div[1]/button").click()
print(member_list)
driver.quit()
P.S: I have tried changing the member.find to:
member.find_element_by_xpath('.//*[#class="projectImage"]').click() But then I get "Unable to find element"
Any help is very much appreciated.
Thanks

If you study the HTML of the page, they have the onclick script which basically triggers the JS and renders the pop-up. You can make use of it.
You can find the onclick script in the child element img.
So your logic should be like (1)Get the child element (2)go to first child element (which is img always for your case) (3)Get the onclick script text (4)execute the script.
child element
for member in members:
print(member.text) # to see that loop went to next iteration
# member.find_element_by_xpath('//*[#class="projectImage"]').click()
#Begin of modification
child_elems = member.find_elements_by_css_selector("*") #Get the child elems
onclick_script = child_elems[0].get_attribute('onclick')#Get the img's onclick value
driver.execute_script(onclick_script) #Execute the JS
time.sleep(5) #Wait for some time
#end of modification
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'VIEW FULL PROFILE')))
links = driver.find_element_by_partial_link_text("VIEW FULL PROFILE")
href = links.get_attribute("href")
member_list.append(href)
member.find_element_by_xpath("/html/body/div[5]/div[1]/button").click()
print(member_list)
You need to import time module. I prefer time.sleep over wait.until, it's more easier to use when you are starting with web scraping.

Having trouble referencing to a certain element on page with Selenium

I am having a terribly hard time referencing to a certain "next page" button on a website that I am trying to scrape links from [https://www.sreality.cz/adresar?strana=2]. If you scroll down you can see a red right arrow button that you can click to go to the next page and so the website load new dynamic content. Every approach seems to report the same exact error and I don't know how am I supposed to point to the element without running into it.
This is the code that I currently have :
from selenium import webdriver
chromedriver_path = "/home/user/Dokumenty/iCloud/RealityScraper/chromedriver"
driver = webdriver.Chrome(chromedriver_path)
print("WebDriver Successfully Initialized")
driver.get("https://www.sreality.cz/adresar?strana=2")
links = driver.find_elements_by_css_selector("h2.title a")
nextPage = driver.find_element_by_css_selector("li.paging-item a.btn-paging-pn.icof.icon-arr-right.paging-next")
for link in links:
print(link.get_attribute("href"))
nextPage.click()
The "nextPage" variable is holding a supposed value to be clicked on once the "links" variable search finishes scraping all the links from the company titles. However when I run this code I get an error :
selenium.common.exceptions.StaleElementReferenceException: Message:
stale element reference: element is not attached to the page document
I have been searching for various fixes online but none of them seemed to resolve the issue. I think that the issue at this point is not caused by the element not loading quickly enough but rather Selenium having trouble finding the element because of wrong reference.
Because of this I have tried using XPath to accurately point to the actual element and so I changed the "nextPage" variable to :
nextPage = driver.find_element_by_xpath("""/html/body/div[2]/div[1]/div[2]/div[2]/div[4]/div/div/div/div[2]/div/div[2]/ul[1]/li[12]/a""")
Which returns exactly the same error as stated above. I have been trying to find a solution to this for hours now and I can't understand where the issue lies. I would be grateful if anyone could explain to me what am I doing wrong. Thanks to anyone.

If you want to get all the ng-href tags from every page. Or you could look into their api.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep
driver.get("https://www.sreality.cz/adresar?strana=2")
wait = WebDriverWait(driver, 10)
while True:
try:
links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "h2.title > a")))
#print(len(links))
for link in links:
print(link.get_attribute("ng-href"))
nextPage = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.btn-paging-pn.icof.icon-arr-right.paging-next")))
nextPage.click()
time.sleep(10)
except Exception as e:
print(e)
break

First of all never use the absolute xpath it will breakdown easily, Use the relative xpath.
Secondly, i think the error you are getting is because after clicking "Next" button for the first time it loads a new page. Which has a different DOM structure and that's why you are not able to find that element.
You can try searching for the element after every new page load (after clicking "Next" button everytime.)
// imports
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
// initialize
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
action = ActionChains(driver)
// Try to use the below code and see if it works.
Next_btn = wait.until(EC.presence_of_element_located((By.XPATH, '(//li[#class="paging-item"])[2]')))
action.move_to_element(Next_btn).click().perform()

Can't locate element by xpath in selenium?

I am learning python and selenium right now by making a small script that posts birthday comments to my facebook friends. I am able to successfully login and navigate to the "friends' birthday" modal, but then I am unable to comment in the textbox.
Here is the error I am getting:
selenium.common.exceptions.NoSuchElementException:
Message: no such element: Unable to locate element:
{"method":"xpath","selector":"//*[#id="u_i_6"]"}
I haven't had any issues finding other page elements via XPath so I am not sure what is causing the issue. I also tried finding the textbox to comment via classname, but had the same result. Can anyone offer some thoughts?
EDIT: forgot to paste code
from selenium import webdriver
from pynput.keyboard import Key, Controller
import time
import config
# fetch facebook password from separate file in the same directory
pw = config.secret['pw']
keyboard = Controller()
driver = webdriver.Chrome()
options = webdriver.ChromeOptions()
options.add_argument('--disable-extensions')
options.add_argument('--disable-notifications')
# navigate to facebook url
driver.get('https://facebook.com')
# input email address
email_input = driver.find_element_by_xpath('//*[#id="email"]')
email_input.send_keys(<omitted>)
# input pw
password_input = driver.find_element_by_xpath('//*[#id="pass"]')
password_input.send_keys(pw)
# click login button element
login_button = driver.find_element_by_xpath('//*[#id="u_0_b"]')
login_button.click()
home_button = driver.find_element_by_xpath('//*[#id="u_0_c"]/a')
# wait 8 seconds for browser popup before pressing esc to get out
time.sleep(8)
keyboard.press(Key.esc)
# check if there are any birthdays today
birthday_section = driver.find_element_by_xpath('//*[#id="home_birthdays"]/div/div/div/div/a/div/div/span/span[1]')
birthday_section.click()
first_comment = driver.find_element_by_xpath('//*[#id="u_i_6"]')
first_comment.send_keys('happy birthday!')
# click post button to post comment
# close browser window after actions are complete
time.sleep(5)
driver.close()

It is hard to answer without the code, but here are a couple of ideas:
Make sure you are using some sort of Selenium wait, implicit or explicit, so your code does not search for an element before this element appears on the page. You can add this code after driver.get() for implicit wait:
driver.implicitly_wait(5)
Also, it looks like IDs are dynamic on that page, I get a different one on my FB page. Try using this xpath to find the textarea:
"//form[contains(#action, 'birthday')]//textarea"
Hope this is helpful, good luck!

Python + selenium throws an error while clicking on the last next button

I've written some code in python with selenium to parse the name from a site. The site has got "next" button to get to its' next page. I've tried to manage this to run my script flawlessly. However, I'm facing two issues at this moment:
Upon execution the scraper gets to the next page and parses from there leaving the starting page unscraped because i could not fix the logic.
When it doesn't find the last grayed out next button it throws an error breaking the code.
Here is what I've tried so far:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=San%20Francisco%2C%20CA&page=10")
while True:
wait.until(EC.visibility_of_element_located((By.XPATH, '//li/a[contains(#class,"next")]')))
item = driver.find_element_by_xpath('//li/a[contains(#class,"next")]')
if not driver.find_element_by_xpath('//li/a[contains(#class,"next")]'):
break
item.click()
wait.until(EC.visibility_of_element_located((By.XPATH, '//div[#class="info"]')))
for items in driver.find_elements_by_xpath('//div[#class="info"]'):
name = items.find_element_by_xpath('.//span[#itemprop="name"]').text
print(name)
driver.quit()
Here is the element for grayed out next button:
<div class="pagination"><p><span>Showing</span>361-388
of 388<span>results</span></p><ul><li>Previous</li><li>9</li><li>10</li><li>11</li><li>12</li><li><span class="disabled">13</span></li></ul></div>

Obviously you should try to switch scraping page and clicking 'Next' button. Also you might use try/except to avoid braking code:
while True:
# Scraping required elements first
items = wait.until(EC.visibility_of_all_elements_located((By.XPATH, '//div[#class="info"]')))
for item in items:
name = item.find_element_by_xpath('.//span[#itemprop="name"]').text
print(name)
# ...and then try to click 'Next' button
try:
driver.find_element_by_xpath('//li/a[contains(#class,"next")]').click()
except:
break