Develop Reference

Selenium wait until attribute appears

I'm using Selenium to download several files. The process is simple: I look for a reference, wait until results are ready and download the associated files.
I've a problem with the part "wait until results are ready". The website uses an AJAX table which loads the results. During the update of this table, an attribute appears in the HTML code and when results are ready it dissapears.
The object is always present, it's only the attribute that changes. If I do the next loop just right after clicking the button of search:
for i in range(0,10):
print(self.driver.find_element(By.ID, "gridpoOverview").get_attribute("aria-busy"))
time.sleep(0.05)
It returns (so I know how to detect it):
none
true
true
none
none
none
none
none
none
none
I want to do it using an EC, but the next code doesn't work (Timeout exception):
WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '//div[#id="gridpoOverview" and #aria-busy="true"]')))

Seems you are close enough.
As returned by get_attribute("aria-busy") the value of aria-busy attribute turns True.
So you need to replace true with True as follows:
WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '//div[#id="gridpoOverview" and #aria-busy="True"]')))
Update
As per your updated question you can use:
WebDriverWait(driver, 10).until(EC.staleness_of(driver.find_element_by_xpath("//div[#id="gridpoOverview" and not(#aria-busy)]")))
Or
WebDriverWait(driver, 10).until(EC.staleness_of(driver.find_element(By.XPATH, "//div[#id="gridpoOverview" and not(#aria-busy)]")))

I've found a solution that works, but I don't know why.
The attribute 'aria-disabled' from the same element is always present. If I include
this line before, it will work.
WebDriverWait(self.driver, 20).until(EC.element_attribute_to_include((By.XPATH, '//div[#id="gridpoOverview"]'),'aria-disabled'))
WebDriverWait(self.driver, 20).until(EC.element_attribute_to_include((By.XPATH, '//div[#id="gridpoOverview"]'),'aria-busy'))
If the first line if not present, it does not work (I've the Timeout exception).
Does anyone knows why ?

Selenium / Python: Why isn't my Selenium find_element_by finding elements anymore after finding the first one in my for loop iterations?

Do you see something wrong with this setup?
(selenium, etc. imported earlier on)
It iterates through table_rows until it finds the first row where the “try” is successful, then comes back from the getinfo() function ran from the “try” (which clicks a link, goes to a page, gets info, and then clicks the back button back to the original page), and then keeps iterating through the rest of table_rows.
The correct number of table_rows iterations are performed by the end, and the “try” function is being triggered again (the print() before current_cell works), but the find_element_by_class doesn’t seem to be picking up any more “a0” in the subsequent table_rows iterations, even though there are definitely some there that should be being found (the print() after current_cell never prints after the very first time).
Thank you for your help. I'm new to coding and have learned a ton, but this is stumping me.
def getinfo(current_cell):
link_in_current_cell = current_cell.find_element_by_tag_name("a")
link_in_current_cell.click()
waitfortable2 = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.CLASS_NAME, "top-edit-table"))
)
print("Here is the info about a0.")
driver.back()
return True
for row in table_rows:
print("got the row!")
waitfortable = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.XPATH, "/html/body/table[3]/tbody/tr/td[2]/form/table/tbody/tr/td/table/tbody/tr[4]/td/table/tbody/tr[1]/td[1]"))
)
try:
print("we're trying!")
current_cell = row.find_element_by_class_name("a0")
print("we got an a0!")
getinfo(current_cell)
except:
print("Not an a0 cell.")
pass
continue
Here is more of the code from before "for row in table_rows:" if you need it, but I don't think that part is an issue, as it is still iterating through the rest of table_rows after it finds the first "a0" cell.
try:
WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.XPATH, "/html/body/table[3]/tbody/tr/td[2]/form/table/tbody/tr/td/table/tbody/tr[4]/td/table/tbody/tr[1]/td[1]"))
)
table = driver.find_element_by_xpath("/html/body/table[3]/tbody/tr/td[2]/form/table/tbody/tr/td/table/tbody/tr[4]/td/table")
table_rows = table.find_elements_by_tag_name("tr")
for row in table_rows:
print("got the row!")
....
....
....(see code box above)

Success! I found my own workaround!
FYI: I still could not see anything wrong with my existing code, as
it correctly found all the a0 cells when I commented out the function
part of that text (#getinfo(current_cell)... thank you #goalie1998 for
the suggestion). And, I didn't change anything in that function for
this new workaround, which works correctly. So, it must have something
to do with Selenium getting messed up when trying to iterate through a
loop that (1) tries to find_element_by something on the page (that
exists multiple times on the page, and that's why you're creating the
loop) and (2) clicks on a link within that loop, goes to a page, goes
back to a page, and then is supposed to keep running through the
iterations with the find_element_by "function" (probably wrong term
usage here) to get the next one that exists on the page. Not sure why
Selenium gets messed up from that, but it does. (More experienced
coders, feel free to elaborate).
Anyway, my workaround thought process, which may help some of you solve this issue for yourselves by doing something similarly, is:
(1) Find all of the links BEFORE clicking on any of them (and create a list of those links)
Instead of trying to find & click the links one-at-a-time as they show up, I decided to find all of them FIRST (BEFORE clicking on them). I changed the above code to this:
# this is where I'm storing all the links
text_link_list = []
for row in table_rows:
print("got the row!")
waitfortable = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.XPATH, "/html/body/table[3]/tbody/tr/td[2]/form/table/tbody/tr/td/table/tbody/tr[4]/td/table/tbody/tr[1]/td[1]"))
)
## Get a0
try:
print("we're trying!")
row.find_element_by_class_name("a0")
print("we got an a0!")
# this next part is just because there are also blank "a0" cells without
# text (aka a link) in them, and I don't care about those ones.
current_row_has_a0 = row.find_element_by_class_name("a0")
if str(current_row_has_a0.text) != "":
text_link_list += [current_row_has_a0.text]
print("text added to text_link_list!")
else:
print("wasn't a text cell!")
except:
pass
continue
(2) Iterate through that list of links, running your Selenium code that includes .click() and .back()
Now that I had my list of links, I could just iterate through that and do my .click() —> perform actions —> .back() function that I created ( getinfo() -- original code in question above).
## brand new for loop, after "for row in table_rows" loop
for text in text_link_list:
# waiting for page to load upon iterations
table = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "/html/body/table[3]/tbody/tr/td[2]/form/table/tbody/tr/td/table/tbody/tr[4]/td/table/tbody/tr[1]/td[1]"))
)
# this is my .click() --> perform actions --> .back() function
getinfo(text)
However, I just needed to make two small changes to my .getinfo() function.
One, I was now clicking on the links via their "link text", not the a0 class I was using before (need to use .find_element_by_link_text).
Two, I could now use my more basic driver.find_element_by instead of my original table.find_element_by ...."table" may have worked as well, but I was worried about the memory of getting to "table" being lost since I was now in my function running the .click() code. I decided to go with "driver" since it was more certain. (I'm still pretty new to coding, this may not have been necessary.)
def getinfo(text):
link_in_current_cell = driver.find_element_by_link_text(text)
link_in_current_cell.click()
waitfortable2 = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.CLASS_NAME, "top-edit-table"))
)
print("Here is the info from this temporary page.")
driver.back()
return True
I hope that this can all be helpful to someone. I was stoked when I did it and it worked! Let me know if it helped you! <3
PS. NOTE IF HAVING STALE ERRORS / StaleElementReferenceException:
If you are iterating through your loops and clicking a link via something like driver.find_element_by (instead of using .back()), you may run into a Stale error, especially if you're just trying to use .click() on a variable that you assigned earlier or outside of the loop. You can fix this (maybe not in the most beautiful way) by redefining your variable right at that point of the loop when you're wanting to click on the link, with code like this:
my_link = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.LINK_TEXT, "linktext"))
)
my_link.click()
continue
"continue" is only necessary if you're wanting to end the current iteration and begin the next one, and is also not likely necessary if this is at the end of your loop
you can change that red "10" number to whatever amount of time you'd like to give the code to find the element (aka the page to reload, most likely) before the script fails

send_keys of selenium in pyhon does not work

I am using python3, and want to send messages in Instagram using python, package selenium.
the code works great before. Today most of the time the send_keys() does not work.
I increase the sleep() time to be sure that it is not the internet speed problem. still, it does not work. Any idea? or any alternative ways to send text in a box (I am using python)?
this is the code:
webdriver.get('https://www.instagram.com/explore/tags/'+ hashtag_list[tag] + '/')
sleep(5)
check='//*[#id="react-root"]/section/main/article/div[2]/div/div[%d]/div[%d]/a/div' %(i,x)
sleep(10)
#'//*[#id=\"react-root\"]/section/main/article/div[1]/div/div/div[1]/div[%s]/a/div' % (x)
first_thumbnail = webdriver.find_element_by_xpath(check)
first_thumbnail.click()
sleep(10)
webdriver.find_element_by_xpath('/html/body/div[3]/div[2]/div/article/div[2]/section[1]/span[2]/button/span').click()
sleep(10)
comment_box = webdriver.find_element_by_xpath('/html/body/div[3]/div[2]/div/article/div[2]/section[3]/div/form/textarea')
sleep(20)
comment_box.send_keys("Hi") # this line does not work
sleep(10)
webdriver.find_element_by_xpath('/html/body/div[3]/div[2]/div/article/div[2]/section[3]/div/form/button').click()
sleep(10)
except:
print("Oops!")
sleep(10)
continue

Have you checked to make sure comment_box is actually a valid webelement by printing it? Just to make sure the selector is correct?
Because it should normally work, but perhaps Instagram is somehow doing something to block it.

Is there a better solution than Selenium.title?

I'm trying to make an If/Else condition on the title of a specific webpage.
If the title is not 'You have been blocked', my script continue.
I've got an issue with selenium in python with different environment, in the first one (OSX) everything works.
I can grab the title check if it contains specific string and continue or not my script.
In Debian and headless mode activated, I can't access to the title of the web page.
driver.get(url)
print(driver.title)
if driver.title != 'You have been blocked':
print('Ok have fun')
else:
print('blocked')
It seems this function .title is very unstable, is there a better way to achieve it ?
Thanks

I've found the right way to work with .title.
Now my script wait the title of the webpage and and I parse it when it is loaded.
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((By.TAG_NAME,"title")))
if (driver.title != 'You have been blocked'):
print('Ok Have Fun')
else:
print('Blocked')
Thanks

Python Selenium Using WebDriverWait and presence_of_element_located times out despite element being present

I have a test running with this in the setUp method of the class:
self.selenium.implicitly_wait(40)
And in the specific test
element = WebDriverWait(self.selenium, 20).until(
EC.presence_of_element_located((By.ID, "element_id"))
)
The element with that ID is a <button> element.
The test kept timing out at this point. But I had a hunch the element was actually there, so I added this:
container = self.selenium.find_element_by_id('element_id_parent')
print container.get_attribute('innerHTML')
element = WebDriverWait(self.selenium, 20).until(
EC.presence_of_element_located((By.ID, "element_id"))
)
And sure enough, the html printout shows the element with id element_id is there even before the WebDriverWait. So it should just find it and continue right?
Occasionally the test passes. More frequently in a local environment, almost never in the CI.
Has anyone run across this? Any solutions come to mind, or am I doing something wrong?