How to get updated html data without refreshing the page selenium - python

I am using selenium python, and I am trying to get a element that gets added when I scroll down the page, youtube comment box that gets loaded when you scroll down. However none of the method of finding the element works out, as it seems to me selenium is not aware of the updated data. I have tried using sleeps, implicit and explicit waits but that did not help either.
<textarea id="textarea" class="style-scope iron-autogrow-textarea" rows="1" autocomplete="off" required="" maxlength="10000"></textarea>
this is the code that gets added.
What are the ways to get selenium to not store the html until it gets added? or how to tell selenium that there is a new element on the page?
I am using chromedriver, if that helps.

from selenium.webdriver.support.wait import
WebDriverWaitWebDriverWait(driver, 60).until(lambda x: x.find_element_by_xpath('//*[#id="textarea"]'))

It's depend on your application. If your application using AXAX then you will get updated data on page and DOM. Selenium have only able to access on your present DOM so if your application is not having such AXAX calls then you have to referesh the page.

Related

I cannot find a "button onclick" element using Selenium and Python

I am automating a process using Selenium and python. Right now, I am trying to click on a button in a webpage (sorry I cannot share the link, since it requires credential to login), but there is no way my code can find this button element. I have tried every selector (by id, css selector, xpath, etc.) and done a lot of googling, but no success.
Here is the source content from the web page:
<button onclick="javascript: switchTabs('public');" aria-selected="false" itemcount="-1" type="button" title="Public Reports" dontactassubmit="false" id="public" aria-label="" class="col-xs-12 text-left list-group-item tabbing_class active"> Public Reports </button>
I also added a sleep command before this to make sure the page is fully loaded, but it does not work.
Can anyone help how to select this onclick button?
Please let me know if you need more info.
Edit: you can take a look at this picture to get more insight (https://ibb.co/cYXWkL0). The yellow arrow indicates the button I want to click on.
The element you trying to click is inside an iframe. So, you need to switch driver into the iframe content before accessing elements inside it.
I can't give you a specific code solution since you didn't share a link to that page, even not all that HTML block. You can see solutions for similar questions here or enter link description here. More results can be found with google search

Scraping through multiple pages when url doesn't change

I need to scrape through all the pages in the (link)[https://mahabocw.in/safety-kit-benefits-distribution/]. But the url doesn't change when I move to the next page. I tried to use selenium, but I am stuck as I don't know how to click the next page. Any help or suggestions would be really appreciated.
I have implemented the following code till now.
from selenium import webdriver
import time
url = "https://mahabocw.in/safety-kit-benefits-distribution/"
driver = webdriver.Chrome()
driver.get(url)
Below is the button element I need to click
<button type="button" class="ag-paging-button">Next</button>
Thanks a lot in advance.
[1]: https://mahabocw.in/safety-kit-benefits-distribution/
You need to tell selenium to click the next button. Add this to your code and see if it works.
next_button = '/html/body/div/div[6]/div/article/div/div/div/div/div[2]/div/div/div[2]/div/div[4]/span[2]/div[3]/button'
click_next = driver.find_element_by_xpath(next_button)
click_next.submit()
You might have to use click_next.click() instead of .submit() depending on the page. Also, to get the 'next_button' you just inspect elements on the page, find the item you want, and click copy as xpath.

Python/Selenium:Different ways to click this specific button

I am trying to understand Python in general as I just switched over from using VBA. I interested in the possible ways you could approach this single issue. I already went around it by just going to the link directly, but I need to understand and apply here.
from selenium import webdriver
chromedriver = r'C:\Users\dd\Desktop\chromedriver.exe'
browser = webdriver.Chrome(chromedriver)
url = 'https://www.fake.com/'
browser.get(url)
browser.find_element_by_id('txtLoginUserName').send_keys("Hello")
browser.find_element_by_id('txtLoginPassword').send_keys("There")
browser.find_element_by_id('btnLogin').click()
At this point, I am trying to navigate to a particular button/link.
Here is the info from the page/element
T-Mobile
Here are some of the things I tried:
for elem in browser.find_elements_by_xpath("//*[contains(text(), 'T-Mobile')]"):
elem.click
browser.execute_script("InitiateCallBack(187, True, T-Mobile, https://www.fake.com/, TMobile)")
I also attempted to look for tags and use css selector all of which I deleted out of frustration!
Specific questions
How do I utilize the innertext,"T-Mobile", to click the button?
How would I execute the onclick event?
I've tried to read the following links, but still have not succeeded incoming up with a different way. Part of it is probably because I don't understand the specific syntax yet. This is just some of the things I looked at. I spent about 3 hours trying various things before I came here!
selenium python onclick() gives StaleElementReferenceException
http://selenium-python.readthedocs.io/locating-elements.html
Python: Selenium to simulate onclick
https://stackoverflow.com/questions/43531654/simulate-a-onclick-with-selenium-https://stackoverflow.com/questions/45360707/python-selenium-using-onclick
Running javascript in Selenium using Python
How do I utilize the innertext,"T-Mobile", to click the button?
find_elements_by_link_text would be appropriate for this case.
elements = driver.find_elements_by_link_text('T-Mobile')
for elem in elements:
elem.click()
There's also a by_partial_link_text locator as well if you don't have the full exact text.
How would I execute the onclick event?
The simplest way would be to simply call .click() on the element as shown above and the event should, naturally, execute at that time.
Alternatively, you can retrieve the onclick attribute and use driver.execute_script to run the js.
for elem in elements:
script = elem.get_attribute('onlcick')
driver.execute_script(script)
Edit:
note that in your code you did element.click -- this does nothing. element.click() (note the parens) calls the click method.
is there a way to utilize browser.execute_script() for the onclick event
execute_script can fire the equivalent event, but there may be more listeners that you miss by doing this. Using the element click method is the most sound. There may very well be many implementation details of the site that may hinder your automation efforts, but those possibilities are endless. Without seeing the actual context, it's hard to say.
You can use JS methods to click an element or otherwise interact with the page, but you may miss certain event listeners that occur when using the site 'normally'; you want to emulate, more or less, the normal use as closely as possible.
As per the HTML you have shared it's pretty clear the website uses JavaScript. So to click() on the link with text as T-Mobile you have to induce WebDriverWait with expected_conditions clause as element_to_be_clickable and your can use the following code block :
WebDriverWait(driver, 20).until(expected_conditions.element_to_be_clickable((By.XPATH, "//a[contains(.,'T-Mobile')]"))).click()
you can use it
<div class="button c_button s_button" onclick="submitForm('rMTF')" style="margin-bottom: 30px;">
<input class="v_small" type="button"></input>
<span>
Reset
</span>

Can't find element by id using selenium

I am making a script in python that goes on a webpage https://www.realtor.ca/ and searches for a specific location. My problem is at the very beginning. When you open a page in the middle is a large search element. Html for that element is:
<input name="ctl00$elContent$ctl00$home_search_input"
maxlength="255" id="home_search_input" class="m_hme_srch_ipt_txtbox"
type="text" style="display: none;">
I am trying to access the element with find_element_by_id method but I always get the error:Message: Unable to locate element: [id="home_search_input"]
This is my code:
from selenium import webdriver as web
Url = "https://www.realtor.ca/"
browser = web.Firefox()
browser.get(Url)
TextField = browser.find_element_by_id("home_search_input")
Has anyone encountered a similar problem or does anyone know how to fix it?
When navigating to the page, the element with the id home_search_input isn't visibble at first. It seems that this one only gets visible once you click the "Where are you looking" placeholder (which disappears then). You'll need to do this explicitly in your test.
Additionally make sure to use either implicit or explicit wait statements to ensure that the elements you interact with are properly loaded and rendered.
Here's an example for your page using the Java client bindings - Python should be quite similar:
driver.get("https://www.realtor.ca/");
new WebDriverWait(driver, 5).until(ExpectedConditions.elementToBeClickable(By.id("m_hme_wherelooking_lnk"))).click();
driver.findElement(By.id("home_search_input")).sendKeys("demo");

Selenium PhantomJS click element with href="#"

Coding in Python 2.7.
I have a crawler I setup and it works perfectly fine when driven by FireFox, but breaks when driven by PhantomJS.
I'm trying to click on an element with href="#"
The crux of the issue is that when the FF driver clicks on this element with the # href, it performs the javascript action (in this case revealing a hidden part of a layer), but when PhantomJS does it, it either doesn't perform the click or it does click it but # just reloads the same page (I can't tell which one).
I've tried everything I can think of, including multiple ActionChains and clicking element by element all the way down to this one with the link. Nothing seems to work.
Any ideas?
href="#" just refreshes the page when you select the link, for example if it was href=#top" when you select the link you would be brought to the top of that same page.
You are probably doing it correctly, you can use driver.find_element_by_link_text('some text') or driver.find_element_by_partial_link_text('some text'), but clicking on that element is just routing you to that same page.

Categories

Resources