This question already has answers here:
Click on ember.js enabled element using Selenium
(5 answers)
Closed 1 year ago.
At this stage, all the script aims to do is to locate the search bar in a digital library, send the name of the resource I am looking for, click on it and get the current url. The first bit (sending keys to search bar) works fine. But I have noticed that the second try/finally block only executes when I right click and inspect any element on the page. If I don't, I get the following error message :
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="ember767"]"}
My code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
PATH = "/Applications/chromedriver"
ser = Service(PATH)
driver = webdriver.Chrome(service=ser)
driver.maximize_window()
driver.implicitly_wait(20)
driver.get("https://browzine.com/libraries/1374/subjects")
print("Enter targeted Journal name:")
targeted_journal = input()
wait = WebDriverWait(driver, 10)
try:
button = driver.find_element(By.ID, "ember648")
driver.implicitly_wait(10)
ActionChains(driver).move_to_element(button).click(button).perform()
button.send_keys(targeted_journal)
button.send_keys(Keys.RETURN)
finally:
pass
# step 2 : click targeted journal and get current url
try:
driver.implicitly_wait(20)
click_journal = driver.find_element(By.ID, "ember767")
ActionChains(driver).move_to_element(click_journal).click(click_journal).perform()
targeted_url = driver.current_url
finally:
pass
print(targeted_url)
driver.quit()
Update - answer found
The ID initially collected for the targeted element was a dynamic one. It was only as such when the developer tools window was open. Therefore the script was only able to detect that particular ID when "Inspect element" was clicked as it was the only case in which the targeted element was assigned this particular ID ("ember767" in my case).
To avoid this issue, an Xpath was used to locate the element instead. See Mayank Shukla's answer below.
In some cases, element's ID or other attributes changes or attribute's value appears on mouse hover. So it's preferred to use xpath for locating such elements
In this scenario, I think the element's id is changing after you are hovering the mouse. Try using xpath and let me know if that works. Because might be after browser refresh this is happening i.e. element's id is changing.
Related
I am trying to create an automation program. I want to click on the "Accept Cookies" shadowbox on the given website.
Here's how I have tried to achieve this:
driver = webdriver.Chrome('chromedriver')
driver.get(r'https://www.studydrive.net/')
script = '''return document.querySelector('#usercentrics-root').shadowRoot.querySelector('button[aria-label="Accept All"]')'''
accept_all_btn = driver.execute_script(script)
accept_all_btn.click()
Here's the error that I get after following this approach:
AttributeError: 'NoneType' object has no attribute 'click'
I don't know, what I am doing wrong here. Any help is appreciated. Thank you in advance.
wait=WebDriverWait(driver, 60)
driver.get("https://www.studydrive.net/")
elem = wait.until(EC.presence_of_element_located((By.ID,"usercentrics-root")))
script = '''return document.querySelector('#usercentrics-root').shadowRoot.querySelector('button[data-testid="uc-accept-all-button"]')'''
accept_all_btn = driver.execute_script(script)
accept_all_btn.click()
Simply wait for the element and then proceed to click the accept all button. No aria-label was specified so I used another attribute.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
The element Accept all button is within a #shadow-root (open)
Solution
Tto click() on the desired element you need to use shadowRoot.querySelector()
You can use the following Locator Strategy:
driver = webdriver.Chrome(service=s, options=options)
driver.get("https://www.studydrive.net/")
time.sleep(5)
accept_all = driver.execute_script('''return document.querySelector("#usercentrics-root").shadowRoot.querySelector("button[data-testid='uc-accept-all-button']")''')
accept_all.click()
PS: The cookie popup surfaces on the screen after significant amount of time, so you may have to induce some waits
References
You can find a couple of relevant detailed discussions in:
How to locate the First name field within shadow-root (open) within the website https://www.virustotal.com using Selenium and Python
How to get past a cookie agreement page using Python and Selenium?
Unable to locate the Sign In element within #shadow-root (open) using Selenium and Python
I am having a terribly hard time referencing to a certain "next page" button on a website that I am trying to scrape links from [https://www.sreality.cz/adresar?strana=2]. If you scroll down you can see a red right arrow button that you can click to go to the next page and so the website load new dynamic content. Every approach seems to report the same exact error and I don't know how am I supposed to point to the element without running into it.
This is the code that I currently have :
from selenium import webdriver
chromedriver_path = "/home/user/Dokumenty/iCloud/RealityScraper/chromedriver"
driver = webdriver.Chrome(chromedriver_path)
print("WebDriver Successfully Initialized")
driver.get("https://www.sreality.cz/adresar?strana=2")
links = driver.find_elements_by_css_selector("h2.title a")
nextPage = driver.find_element_by_css_selector("li.paging-item a.btn-paging-pn.icof.icon-arr-right.paging-next")
for link in links:
print(link.get_attribute("href"))
nextPage.click()
The "nextPage" variable is holding a supposed value to be clicked on once the "links" variable search finishes scraping all the links from the company titles. However when I run this code I get an error :
selenium.common.exceptions.StaleElementReferenceException: Message:
stale element reference: element is not attached to the page document
I have been searching for various fixes online but none of them seemed to resolve the issue. I think that the issue at this point is not caused by the element not loading quickly enough but rather Selenium having trouble finding the element because of wrong reference.
Because of this I have tried using XPath to accurately point to the actual element and so I changed the "nextPage" variable to :
nextPage = driver.find_element_by_xpath("""/html/body/div[2]/div[1]/div[2]/div[2]/div[4]/div/div/div/div[2]/div/div[2]/ul[1]/li[12]/a""")
Which returns exactly the same error as stated above. I have been trying to find a solution to this for hours now and I can't understand where the issue lies. I would be grateful if anyone could explain to me what am I doing wrong. Thanks to anyone.
If you want to get all the ng-href tags from every page. Or you could look into their api.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep
driver.get("https://www.sreality.cz/adresar?strana=2")
wait = WebDriverWait(driver, 10)
while True:
try:
links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "h2.title > a")))
#print(len(links))
for link in links:
print(link.get_attribute("ng-href"))
nextPage = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.btn-paging-pn.icof.icon-arr-right.paging-next")))
nextPage.click()
time.sleep(10)
except Exception as e:
print(e)
break
First of all never use the absolute xpath it will breakdown easily, Use the relative xpath.
Secondly, i think the error you are getting is because after clicking "Next" button for the first time it loads a new page. Which has a different DOM structure and that's why you are not able to find that element.
You can try searching for the element after every new page load (after clicking "Next" button everytime.)
// imports
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
// initialize
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
action = ActionChains(driver)
// Try to use the below code and see if it works.
Next_btn = wait.until(EC.presence_of_element_located((By.XPATH, '(//li[#class="paging-item"])[2]')))
action.move_to_element(Next_btn).click().perform()
I am trying to retrieve an element that I would like to click on. Here's the opening of the website with Selenium in Python:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--dns-prefetch-disable')
driver = webdriver.Chrome("./chromedriver", options=chrome_options)
website = "https://www.agronet.gov.co/estadistica/Paginas/home.aspx?cod=4"
driver.get(website) # loads the page
Then, I look for the element I'm interested in:
driver.find_element_by_xpath('//*[#id="cmbDepartamentos"]')
which raises a NoSuchElementException error. When looking at the html source (driver.page_source), indeed "cmbDepartamentos" does not exist! and the text of the dropdown menu I am trying to locate which is "Departamentos:" does not exist either. How can I deal with this?
This should work:
iframe=driver.find_element_by_xpath('//div[#class="iframe"]//iframe')
driver.switch_to.frame(iframe)
driver.find_element_by_xpath('//*[#id="cmbDepartamentos"]').click()
Notes:
The reason for NoSuchElementException error is that the element is
inside an iframe. Unless you switch your driver to that iframe,
the identification will not work.
CTRL + F in the Dev Tools panel, then search for the xpath you
defined in your script is always a good way to rule out issues with
your xpath definition, as cause for NoSuchElementException error (and in your case, the xpath is correct)
You might want to consider adding a WebdriverWait for a complete load of the search area/iframe before attempting to find the "Departamentos" field
I want to download user data on Google analytics by using crawler so I write some code using selenium. However, I cannot click the "export" button. It always shows the error "no such element". I tried to use find_element_by_xpath, by_name and by_id.
I upload inspect of GA page below.
I TRIED:
driver.find_element_by_xpath("//*[#class='download-link']").click()
driver.find_element_by_xpath('//*[#id="ID-activity-userActivityTable"]/div/div[2]/span[6]/button')
driver.find_element_by_xpath('//*[#class='_GAD.W_DECORATE_ELEMENT.C_USER_ACTIVITY_TABLE_CONTROL_ITEM_DOWNLOAD']')
Python Code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome('/Users/parkjunhong/Downloads/chromedriver')
driver.implicitly_wait(3)
usrid = '1021'
url = 'https://analytics.google.com/analytics/web/#/report/app-visitors-user-activity/a113876882w169675624p197020837/_u.date00=20190703&_u.date01=20190906&_r.userId='+usrid+'&_r.userListReportStates=%3F_u.date00=20190703%2526_u.date01=20190906%2526explorer-
table.plotKeys=%5B%5D%2526explorer-table.rowStart=0%2526explorer-
table.rowCount=1000&_r.userListReportId=app-visitors-user-id'
driver.get(url)
driver.find_element_by_name('identifier').send_keys('ID')
idlogin = driver.find_element_by_xpath('//*[#id="identifierNext"]/span/span')
idlogin.click()
driver.find_element_by_name('password').send_keys('PASSWD')
element = driver.find_element_by_id('passwordNext')
driver.execute_script("arguments[0].click();", element)
#login
driver.find_element_by_xpath("//*[#class='download-link']").click()
#click the download button
ERROR:
Message: no such element: Unable to locate element
inspection of GA
your click element is in an iFrame (iFrame id="galaxyIframe" ...). Therefore, you need to tell the driver to switch from the "main" page to said iFrame. If you add this line of code after your #login it should work:
driver.switch_to.frame(galaxyIframe)
(If the frame did not have a name, you would use: iframe = driver.find_element_by_xpath("xpath-to-frame") and then driver.switch_to.frame(iframe)
To get back to your default frame, use:
driver.switch_to.default_content()
Crawling GA is generally a pain. Not just because you have these iFrames everywhere.
Apart from that, I would recommend looking into puppeteer, the new kid on the crawler block. Even though the prospect of switching to javascript from python may be daunting, it is worth it! Once you get into it, selenium will have felt super clunky.
You can try with the text:
If you want to click on 'Export'-
//button[contains(text(),'Export')]
I have been tasked with writing a parser to click a button on a website and I am having issues to click only one of the buttons. The following code works on every button except one.
Here's the html:
http://pastebin.com/6dLF5ru8
here's the source html:
http://pastebin.com/XhsedGLb
python code:
driver = webdriver.Firefox()
...
el = driver.find_element_by_id("-spel-nba")
actions.move_to_element(el)
actions.sleep(.1)
actions.click()
actions.perform()
I am getting this error.
ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
as per Saifur I just tried waits with the same element not visible exception:
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.XPATH, "//input[contains(#id,'spsel')][#value='nba']"))).click()
If you look at the page source, you'll understand that almost all of theSELECT, DIV elements are faked and created from JavaScript, that is why webdriver cannot SEE them.
There's a workaround though, by using ActionChains to open your developer console, and inject an artificial CLICK on the desired element, which in fact, is the Label triggering the NBA data loading... here's a working example:
from selenium import webdriver
from selenium.webdriver.common import action_chains, keys
import time
driver = webdriver.Firefox()
driver.get('Your URL here...')
assert 'NBA' in driver.page_source
action = action_chains.ActionChains(driver)
# open up the developer console, mine on MAC, yours may be diff key combo
action.send_keys(keys.Keys.COMMAND+keys.Keys.ALT+'i')
action.perform()
time.sleep(3)
# this below ENTER is to rid of the above "i"
action.send_keys(keys.Keys.ENTER)
# inject the JavaScript...
action.send_keys("document.querySelectorAll('label.boxed')[1].click()"+keys.Keys.ENTER)
action.perform()
Alternatively to replace all the ActionChains commands, you can simply run execute_script like this:
driver.execute_script("document.querySelectorAll('label.boxed')[1].click()")
There you go, at least on my local file anyway... Hope this helps!
What worked for me was to find the element just before the problematic one (that is, just before it in terms of tab order), then call Tab on that element.
from selenium.webdriver.common.keys import Keys
elem = br.find_element_by_name("username")
elem.send_keys(Keys.TAB) # tab over to not-visible element
After doing that, I was able to send actions to the element.
The actual solution of this thread did not work for me.
however,
this one did :
element = WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.XPATH, xpaths['your_xpath_path'])))
the trick is to use :
EC.visibility_of_element_located
the WebDriverWait
WebDriverWait
from this import :
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
I suggest you use xpath with explicit wait
//input[contains(#id,'spsel')][#value='nba']
if "Element is not currently visible" then make it VISIBLE
f.e.
>>> before is hidden top is outside of page
<input type="file" style="position: absolute;top:-999999" name="file_u">
>>> after move top on in page area
DRIVER.execute_script("document.getElementByName('file_u').style.top = 0;")
time.sleep(1); # give some time to render
DRIVER.find_element_by_name("file_u").send_keys("/tmp/img.png")
Instead of get_element_by_id() you can try elem = browser.find_element_by_css_selector('#elemId') (go to that webpage and the element, right click it and Copy CSS Selector, or something like that.) This is what i did and it works. You also try find_element_by_link_text(text), find_element_by_partial_link_text(text), find_element_by_tag_name(tagName_case_insensitive_here), find_element_by_name(name) etc. Something will work. After the id the CSS Selector is your best bet.
I ended up using #twasbrillig's solution, but instead of finding the previous element and sending a TAB keypress, I find the desired element, send a TAB keypress with that element, and then a SHIFT + TAB keypress to the driver:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
el = driver.find_element_by_id("-spel-nba")
el.send_keys(Keys.TAB)
webdriver.ActionChains(driver).key_down(Keys.SHIFT).send_keys(Keys.TAB).key_up(Keys.SHIFT)
I tried using the other methods but in the end found that the simplest way was to just try and click the button, and catch the error. This allows me to perform other actions based on if it worked (True) or didn't (False).
def click_button(html_object):
try:
html_object.click()
except:
return False #most likely because it is NotVisible object and can be ignored
return True
...
...
click_button(actions)
The way I solved this in python was:
try:
# the element you want to scroll to
element = driver.find_element_by_id("some_id")
ActionChains(driver).move_to_element(element).perform()
element.send_keys(Keys.TAB).key_up(Keys.SHIFT)
#element.click()
except Exception as e:
print(e)