I use this great solution for waiting until the page loads completely.
But for one page it's don't work:
from selenium import webdriver
driver = webdriver.FireFox()
driver.get("https://vodafone.taleo.net/careersection/2a/jobsearch.ftl")
element = driver.find_element_by_xpath(".//*[#id='currentPageInfo']")
print element.id, element.text
driver.find_element_by_xpath(".//a[#id='next']").click()
element = driver.find_element_by_xpath(".//*[#id='currentPageInfo']")
print element.id, element.text
Output:
{52ce3a9f-0efb-49e1-be86-70446760e422} 1 - 25 of 1715
{52ce3a9f-0efb-49e1-be86-70446760e422} 26 - 50 of 1715
How to explain this behavior?
P.S.
With PhantomJS occurs the same thing
Selenium lib version 2.47.1
Edit
It's ajax calls on page.
This solution is used in tasks similar to that described in this article
Without the HTML one can only guess:
Reading the linked answer, the behaviour you observe is most probably because clicking the "next" button is not loading the whole page again but is only making an ajax call or something and filling an already existing table with new values.
This means that the whole page stays "the same" and thus also the "current page info" element still has the same id (just some java-script that changed its text value)
To check this you can do the following:
write another test-method identical to the one you got, but this time you replace this line:
driver.find_element_by_xpath(".//a[#id='next']").click()
with this line:
driver.navigate().refresh();
If it now gives you different ids, then I'm pretty sure, my guess is correct.
Related
Today I'm having troubles due to a "a href" button that does not have any ID to be identified, then let's explain a little bit more about the problem... I have an structure like this one(let's assume XXX is an anonymous path):
wait = WebDriverWait(driver, 5)
el=wait.until(EC.presence_of_element_located((By.ID, 'XXX1')))
entries = el.find_elements_by_tag_name('tr')
for i in range(len(entries)):
if(entries[i].find_element_by_xpath(XXX2).text==compare):
el = wait.until(EC.element_to_be_clickable((By.ID,XXX3)))
el.click()
el=wait.until(EC.presence_of_element_located((By.ID, XXX4)))
entries2 = el.find_elements_by_tag_name('tr')
for j in range(len(entries2)):
#Some statements...
xpath = ".../a"
your_element=WebDriverWait(driver,10)\
.until(EC.element_to_be_clickable((By.XPATH,xpath)))##Here the problem
your_element.click()
Then, I'm getting information from an hibrid page (dynamic and static) using as driver a ChromeDriver one, once I get a big table, inside every row there is a button that shows more info, then I need to click it for open it too, the main problem is when this operation iterates, that error is shown by the output. This driver is a ChromeDriver one. In summary, first I search something and click on search button, then I get a table where every row(at the end in the last column) has a button that once is open, shows more information, consequently I need to open it and close it due to the next row, it works with the first row, but with the second one, it crashes. I would really appreciate any advice of how to handle this problem.
Thanks in advance!
The problem is that you change with the click the dom within your loop. For mr this never worked.
One solution is, to try to re-query within the loop to make sure your at the correct position. Your third line:
entries = el.find_elements_by_tag_name('tr')
should be executed every time and with a counter make sure you are at the correct position of your <tr> entries.
Context
This is a repost of Get a page with Selenium but wait for element value to not be empty, which was Closed without any validity so far as I can tell.
The linked answers in the closure reasoning both rely on knowing what the expected text value will be. In each answer, it explicitly shows the expected text hardcoded into the WebDriverWait call. Furthermore, neither of the linked answers even remotely touch upon the final part of my question:
[whether the expected conditions] come before or after the page Get
"Duplicate" Questions
How to extract data from the following html?
Assert if text within an element contains specific partial text
Original Question
I'm grabbing a web page using Selenium, but I need to wait for a certain value to load. I don't know what the value will be, only what element it will be present in.
It seems that using the expected condition text_to_be_present_in_element_value or text_to_be_present_in_element is the most likely way forward, but I'm having difficulty finding any actual documentation on how to use these and I don't know if they come before or after the page Get:
webdriver.get(url)
Rephrase
How do I get a page using Selenium but wait for an unknown text value to populate an element's text or value before continuing?
I'm sure that my answer is not the best one but, here is a part of my own code, which helped me with similar to your question.
In my case I had trouble with loading time of the DOM. Sometimes it took 5 sec sometimes 1 sec and so on.
url = 'www.somesite.com'
browser.get(url)
Because in my case browser.implicitly_wait(7) was not enought. I made a simple for loop to check if the content is loaded.
some code...
for try_html in range(7):
""" Make 7 tries to check if the element is loaded """
browser.implicitly_wait(7)
html = browser.page_source
soup = BeautifulSoup(html, 'lxml')
raw_data = soup.find_all('script', type='application/ld+json')
"""if SKU in not found in the html page we skip
for another loop, else we break the
tryes and scrape the page"""
if 'sku' not in html:
continue
else:
scrape(raw_data)
break
It's not perfect, but you can try it.
recently I tried scraping, so this time i wanted to go from page to page until I get the final destination I want. Here's my code:
sub_categories = browser.find_elements_by_class_name("ty-menu__submenu-link")
for sub_category in sub_categories:
sub_category = str(sub_category.get_attribute("href"))
if(sub_category is not 'http://www.lsbags.co.uk/all-bags/view-all-handbags-en/' and sub_category is not "None"):
browser.get(sub_category)
print("Entered: " + sub_category)
product_titles = browser.find_elements_by_class_name("product-title")
for product_title in product_titles:
final_link = product_title.get_attribute("href")
if(str(final_link) is not "None"):
browser.get(str(final_link))
print("Entered: " + str(final_link))
#DO STUFF
I already tried doing the wait and the wrapper(the try and exception one) solutions from here, but I do not get why its happening, I have an idea why this s happening, because it the browser gets lost right? when it finishes one item?
I don't know how should I express this idea. In my mind I imagine it would be like this:
TIMELINE:
*PAGE 1 is within a loop, ALL THE URLS WITHIN IT IS PROCESSED ONE BY ONE
*The first url of PAGE 1 is caught. Thus do browser.get page turn to PAGE 2
*PAGE 2 has the final list of links I want to evaluate, so another loop here
to get that url, and within that url #DO STUFF
*After #DO STUFF get to the second url of PAGE 2 and #DO STUFF again.
*Let's assume PAGE 2 has only two urls, so it finished looping, so it goes back to PAGE 1
*The second url of PAGE 1 is caught...
and so on... I think I have expressed my idea in some poitn of my code, I dont know what part is not working thus returning the exception.
Any help is appreciated, please help. Thanks!
Problem is that after navigating to the next page but before reaching this page Selenium finds the elements where you are waiting for but this are the elements of the page where you are coming from, after loading the next page this elements are not connected to the Dom anymore but replaced by the ones of the new page but Selenium is going to interact with the elements of the former page wich are no longer attached to the Dom giving a StaleElement exception.
After you pressed on the link for the next page you have to wait till the next page is completly loaded before you start your loop again.
So you have to find something on your page, not being the elements you are going to interact with, that tells you that the next page is loaded.
I've never studied HTML seriously, so probably what I'm going to say is not right.
While I was writing my selenium code, I noticed that some buttons on some webpages does not redirect to other pages, but they change the structure of the the first one. From what I've understood, this happens because there's some JavaScript code that modifies it.
So, when I wanna get some data which is not present on the first page loading, I just have to click the right sequence of button to obtain it, rigth?
The page I wanted to load is https://watch.nba.com/, and what I want to get is the match list of a given day. The fastest path to get it is to open the calendary:
calendary = driver.find_element_by_class_name("calendar-btn")
calendary.click()
and click the selected day:
page = calendary.find_element_by_xpath("//*[contains(text(), '" + time.strftime("%d") + "')]")
page.click()
running this code, I get this error:
selenium.common.exceptions.ElementNotVisibleException
I read somewhere that the problem is that the page is not correctly loaded, or the element is not visible/clickable, so I tried with this:
wait = WebDriverWait(driver, 10)
page = wait.until(EC.visibility_of_element_located((By.XPATH, "//*[contains(text(), '" + time.stfrtime("%d") + "')]")))
page.click()
But this time I get this error:
selenium.common.exceptions.TimeoutException
Can you help me to solve at least one of these two problems?
The reason you are getting such behavior is because this page is loaded with iFrames (I can see 15 on the main page) once you click the calendar button, you will need to switch your context to either be on the defaultContext or a specific iframe where the calendar resides. There is tons of code outthere that shows you how to switch into and out of iframe. Hope this helps.
I'm having trouble selecting a button in my Splinter script using the find_by_css method. The documentation is sparse at best, and I haven't found a lot of good articles out there with examples.
br.find_by_css('div#edit-field-download-files-und-0 a.button.launcher').first.click()
...where br is my browser instance.
I've tried a few different ways of writing it. I'm really not sure how I'm supposed to do it because the documentation doesn't give any hard examples of the syntax.
Here's a screenshot of the element.
Sorry the screenshot kind of sucks.
Does anyone have any experience with this?
The css selector looks alright, just that i am not sure from where have you got find_by_css as a method?
How about this :-
br.find_element_by_css_selector("div#edit-field-download-files-und-0 a.button.launcher").click()
Selenium provides the following methods to locate elements in a page:
find_element_by_id
find_element_by_name
find_element_by_xpath
find_element_by_link_text
find_element_by_partial_link_text
find_element_by_tag_name
find_element_by_class_name
find_element_by_css_selector
To find multiple elements (these methods will return a list):
find_elements_by_name
find_elements_by_xpath
find_elements_by_link_text
find_elements_by_partial_link_text
find_elements_by_tag_name
find_elements_by_class_name
find_elements_by_css_selector
I'm working on something similar where I'm trying to click stuff on a webpage. The documentation for find_by_css() is very poor and you need to type the css path to the element you want to click.
Say we want to go to the about tab on python.org
from splinter import Browser
from time import sleep
with Browser() as browser: #<--Create browser instance (firefox default driver)
browser.visit('http://www.python.org') #<--Visits url string
browser.find_by_css('#about > a').click()
# ^--Put css path here in quotes
sleep(5)
If your connection is good you might not get the chance to see the about tab getting clicked but you should end up on the about page.
The hard part is figuring out the css path to an element. However once you have it, the find_by_css() method looks pretty easy
I like the W3Schools reference for CSS selection parameters: http://www.w3schools.com/cssref/css_selectors.asp
As for your code... I recommend breaking this down into a few steps, at least during debug. The call to br.find_by_css('css_string') returns a list of elements. So you can grab that list and check the count.
elems = br.find_by_css('div#edit-field-download-files-und-0 a.button.launcher')
if len(elems) == 1:
elems.first.click()
If you don't check the length of the returned list and call '.first' on an empty list, you'll get an exception. If len > 1, you're probably getting things you don't expect.
Each id on a page is unique, and you can daisy-chain searches, so you can use a few different statements to make this happen:
id_elems = br.find_by_id('edit-field-download-files-und-0')
if id_elems:
id_elem = id_elems.first
a_elems = id_elem.find_by_tag("a")
for e in a_elems:
if e.has_class("button launcher"):
print('Found it!')
e.click()
This is, of course, just one of many ways to do this.
Lastly, Splinter is a wrapper around Selenium and other webdrivers. It's possible that, even after you find the element to click, the actual click won't do anything. If this happens, you can also try clicking on the wrapped Selenium object, available as e._element. So you could try e._element.click() if necessary.