driver = webdriver.Chrome(driver_path, options=chrome_options)
wait = WebDriverWait(driver, 20)
driver.get('https://%s/' % asset_id)
wait.until(EC.presence_of_element_located((By.XPATH, "//*[#id='dev_diaginfo_fid']")))
print(driver.find_element_by_xpath('//*[#id='dev_diaginfo_fid']").get_attribute=("innerHTML"))
I'm able to log into the website and Selenium returns the WebElement but it is not consistent when returning the text from that WebElement. Sometimes it returns it and other times it seems like it isn't loading fast enough (slow network where this is being utilized) and returns no data at all but I can still see the WebElement itself just not the data. The data is dynamically loaded via JS. Probably not relevant but I am using send_keys to pass the credentials needed to login and then the page with the version is loaded.
Is there a way to use an ExpectedCondition (EC) to wait until it sees text before moving on? I'm attempting to pull the firmware version from a network device and it finds the Firmware element but it is not consistent when returning the actual firmware version. As stated before, there are issues with network speeds occasionally so my suspicion is that it's moving on before loading the firmware number. This device does not have internet access so I can't share the URL. I can confirm that I have pulled the firmware version but it's just not consistent.
I have tried passing it to beautifulsoup and can verify that it sees Firmware Version: but the inner tags are empty.
Edit: I have tried EC.visibility_of_all_elements and EC.visibility_of_element as well with no luck.
Here's an idea.
Try a while loop until you see the text.
counter = 0
elem = driver.find_element_by_xpath("//*[#id='dev_diaginfo_fid']").get_attribute=("innerHTML")
while "" in elem:
pause(500)
elem = driver.find_element_by_xpath("//*[#id='dev_diaginfo_fid']").get_attribute=("innerHTML")
if "" not in elem:
print("Success! The text is: " + elem)
break
if counter > 20:
print("Text still not found!")
break
counter += 1
Obviously, adjust the loop to suit your needs.
Related
firebug
console
I have a project that I chose Selenium to open 1-5 links. It's stopping at the 3rd link. I've followed the same methods for the previously successful requests. I've allowed 17 seconds and watched as I can see the page load, before the script continues to run in my console. I'm just not sure why it can't find this link, and I hope it's something I'm simply overlooking...
from selenium import *
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import csv
import time
username = "xxxxxxx"
password = "xxxxxxx"
driver = webdriver.Firefox()
driver.get("https://tm.login.trendmicro.com/simplesaml/saml2/idp/SSOService.php")
assert "Trend" in driver.title
elem1 = driver.find_element_by_class_name("input_username")
elem2 = driver.find_element_by_class_name("input_password")
elem3 = driver.find_element_by_id("btn_logon")
elem1.send_keys(username)
elem2.send_keys(password)
elem3.send_keys(Keys.RETURN)
time.sleep(7)
assert "No results found." not in driver.page_source
elem4 = driver.find_element_by_css_selector("a.float-right.open-console")
elem4.send_keys(Keys.RETURN)
time.sleep(17)
elem5 = driver.find_element_by_tag_name("a.btn_left")
elem5.send_keys(Keys.RETURN)
Well one of the reasons is elem5 is looking for the element by tag name, but you are passing it a css tag. "a.btn_left" is not an html tag name and so your script will never actually find it, because it simply doesn't exist in the dom.
You either need to find it by css_selector or better yet by Xpath. If you want to make this as reliable possible and more future proof I always try and find elements on a page with at least 2 descriptors using Xpath if possible.
Change this:
elem5 = driver.find_element_by_tag_name("a.btn_left")
To this:
elem5 = driver.find_element_by_css_selector("a.btn_left")
You will almost never use tag_name, mostly because it will always retrieve the first tag you pass to it, so "a" will always find the first link and click it, yours however does not exist.
I wound up solving it with this code. I increased time to 20 secs, believe it or not, I did try the find by css, I actually left the a.btn_left, and cycled through all the elements, and none of them worked, fortunately, I could access by tab and key functions so that worked for now.
time.sleep(20)
driver.get("https://wfbs-svc-nabu.trendmicro.com/wfbs-svc/portal/en/view/cm")
elem5 = driver.find_element_by_link_text("Devices")
elem5.send_keys(Keys.ENTER)
I am using Python bindings for Selenium with PhantomJS to scrape the contents of a website, like so.
The element I want to access is in the DOM but not in the HTML source. I understand that if I want to access elements in the DOM itself, I need to use the PhantomJS evaluate() function. (e.g. http://www.crmarsh.com/phantomjs/ ; http://phantomjs.org/quick-start.html)
How can I do this from within Selenium?
Here is part of my code (which is currently not able to access the element using a PhantomJS driver):
time.sleep(60)
driver.set_window_size(1024,768)
todays_points = driver.find_elements_by_xpath("//div/a[contains(text(),'Today')]/preceding-sibling::span")
total = 0
for today in todays_points:
driver.set_window_size(1024,768)
points = today.find_elements_by_class_name("stream_total_points")[0].text
points = int(points[:-4])
total += points
driver.close()
print total
I realise that this question is very similar to this, and other SO questions. However, I've played around with the screen size, and also the wait times, and that has not fixed the problem.
I am
starting a driver
opening a website and logging in with Selenium
scraping specific values from the home page
This is the code that isn't working in PhantomJS (but works fine if I use chromedriver):
time.sleep(60)
driver.set_window_size(1024,768)
todays_points = driver.find_elements_by_xpath("//div/a[contains(text(),'Today')]/preceding-sibling::span")
total = 0
for today in todays_points:
driver.set_window_size(1024,768)
points = today.find_elements_by_class_name("stream_total_points")[0].text
points = int(points[:-4])
total += points
driver.close()
print total
The HTML that I'm trying to access is inside a div element:
<span class="stream-type">tracked 7 Minutes-Recode for <span class="stream_total_points">349 pts</span></span>
<a class="action_time gray_link" href="/entry/42350781/">Today</a>
I want to grab the '349 pts' text. However, with PhantomJS the value returned is always 0, so I think it's not finding that element.
EDIT: When I print the html source using print(driver.page_source) I get the correct page outputted but the element is not visible. Checking on chrome using the view source tool, I can't see the element here either (but I can with inspect element). Could this be why PhantomJS cannot access the element?
I use this great solution for waiting until the page loads completely.
But for one page it's don't work:
from selenium import webdriver
driver = webdriver.FireFox()
driver.get("https://vodafone.taleo.net/careersection/2a/jobsearch.ftl")
element = driver.find_element_by_xpath(".//*[#id='currentPageInfo']")
print element.id, element.text
driver.find_element_by_xpath(".//a[#id='next']").click()
element = driver.find_element_by_xpath(".//*[#id='currentPageInfo']")
print element.id, element.text
Output:
{52ce3a9f-0efb-49e1-be86-70446760e422} 1 - 25 of 1715
{52ce3a9f-0efb-49e1-be86-70446760e422} 26 - 50 of 1715
How to explain this behavior?
P.S.
With PhantomJS occurs the same thing
Selenium lib version 2.47.1
Edit
It's ajax calls on page.
This solution is used in tasks similar to that described in this article
Without the HTML one can only guess:
Reading the linked answer, the behaviour you observe is most probably because clicking the "next" button is not loading the whole page again but is only making an ajax call or something and filling an already existing table with new values.
This means that the whole page stays "the same" and thus also the "current page info" element still has the same id (just some java-script that changed its text value)
To check this you can do the following:
write another test-method identical to the one you got, but this time you replace this line:
driver.find_element_by_xpath(".//a[#id='next']").click()
with this line:
driver.navigate().refresh();
If it now gives you different ids, then I'm pretty sure, my guess is correct.
I'm trying to find a specific text from a page https://play.google.com/store/apps/details?id=org.codein.filemanager&hl=en using selenium python. I'm looking for the element name - current version from the above url. I used the below code
browser = webdriver.Firefox() # Get local session of firefox
browser.get(sampleURL) # Load page
elem = browser.find_elements_by_clsss_name("Current Version") # Find the query box
print elem;
time.sleep(2) # Let the page load, will be added to the API
browser.close()
I don't seem to get the output printed. Am I doing anything wrong here?
There is no class with name "Current Version". If you want to capture the version number that is below the "Current Version" text, the you can use this xpath expression:
browser.find_element_by_xpath("//div[#itemprop='softwareVersion']")