I am using Python and Selenium to scrape a webpage, in some cases, I can't get it to work,
*
I would like to access the element with text 'PInt', wich is the second link in the below code.
The xPath for it (copied from Developer console) is: //[#id="submenu1"]/a[2]
<div id="divTest" onscroll="SetDivPosition();" style="height: 861px;">
<div class="menuTitle" id="title1">
</div>
<div class="subtitle" id="submenu1">
<img src="images/spacer.gif" border="0" width="2px" height="12px">
Mov<br>
<img src="images/spacer.gif" border="0" width="2px" height="12px">
PInt<br>
<img src="images/spacer.gif" border="0" width="2px" height="12px">
SWAM / SWIF<br>
</div>
...
A snipped of my code is:
try:
res = driver.find_elements_by_link_text('PInt')
print("res1:{}".format(res))
res = driver.find_element(By.XPATH,'//*[#id="submenu1"]/a[3]')
print("res:{} type[0]:{}".format(res,res[0]))
itm1 = res[0]
itm1.click()
I get the error:
Unable to locate element:
{"method":"xpath","selector":"//*[#id="submenu1"]/a[2]"}
My question is, how can I get the right xPath of the element or any other way to access the element?
UPDATE:
This might be important, the issue with
Message: invalid selector: Unable to locate an element with the xpath expression (and I've tried all proposed solutions) might be that this is after authenticate in the webpage (User + Pwd) before, everything works.
I noticed that the url driver.current_url after login is static (asp page).
Also this part I am trying to access in a frameset and frame
html > frameset > frameset > frame:nth-child(1)
Thanks to #JeffC to point me in the right direction.
since the page has some frames, I manage to access the element first by switching to the right frame (using xPath)
and then access the element.
driver.switch_to.default_content()
driver.switch_to.frame(driver.find_element_by_xpath('html / frameset / frameset / frame[1]'))
driver.find_element_by_xpath("//a[contains(text(),'PInt')]").click()
BTW, in case you want to run the script from a crontab,you need to setup a display:
30 5 * * * export DISPLAY=:0; python /usr/.../main.py
To see a full list of all the ways of selecting elements using selenium, you can read all about it in the documentation.
Using xpath:
res = driver.find_element_by_xpath(u'//*[#id="submenu1"]/a[2]')
Using css selector:
res = driver.find_element_by_css_selector('#submenu1 a:nth-of-type(2)')
Try with any of the below xpath. Sometimes automatically generated xpath does not work.
//a[contains(text(),'PInt')]
or
//div[#id='submenu1']//a[contains(text(),'PInt')]
Also I would suggest you to set some wait time before clicking on above link in case if above xpath does not work
To find xPath in chrome:
Right click the element you want
Inspect, which will open the developer window and highlight the selected element.
Right click the highlighted element and choose copy > copy by xpath
Here is a list of all the different ways to locate an element Locating Elements
Related
I am including portions of the HTML below. I believe I have found the larger element using the command:
driver.find_element_by_xpath('//div[#id="day-number-4"]')
That div ID is unique on the web page, and the command above did not create an exception nor error.
The HTML code that defines that element is:
<div id="day-number-4" class="c-schedule-calendar__class-schedule-content tabcontent js-class-schedule-content u-is-none u-is-block" data-index="3">
Now, the hard part. Inside that div element are a number of "<li"'s that take the form of:
<li tabindex="0" class="row c-schedule-calendar__class-schedule-listitem-wrapper c-schedule-calendar__workout-schedule-list-item" data-index="0" data-workout-id="205687" data-club-id="229">
and then followed a clickable button in this format:
<button class="c-btn-outlined class-action" data-class-action="book-class" data-class-action-step="class-action-confirmation" data-workout-id="205687" data-club-id="229" data-waitlistable="true"><span class="c-btn__label">Join Waitlist</span></button>
I always want to click on the 3rd button inside that <div element. The only thing unique would be the data-index, which starts at 0 for the 1st <li", and 2 for the 3rd So I want to find the clickable button that will follow this HTML code:
<li tabindex="0" class="row c-schedule-calendar__class-schedule-listitem-wrapper c-schedule-calendar__workout-schedule-list-item" data-index="2" data-workout-id="206706" data-club-id="229">
I cannot search on data-index as "data-index="2"" appears many times on the web page.
How do I do this?
I answered my own question. I used multiple searches within the specific element using this code:
Day_of_Timeslot = driver.find_element_by_xpath('//div[#id="day-number-4"]')
Precise_Timeslot = Day_of_Timeslot.find_element_by_xpath(".//li[#data-index='1']")
Actual_Button = Precise_Timeslot.find_element_by_xpath(".//button[#data-class-action='book-class']").click()
I am trying to pull the price of the first listing on this website with the following code, but it's returning nothing but blanks. I am navigating to the website, hitting F12, and then copying the XPATH into the line of code below. Any thoughts on why this wouldn't work?
the following gives nothing:
/html/body/div[1]/div/section[6]/div/div[3]/div/div[1]/a/div/div[1]/div/div[2]/div/div[2]/span/sup
While this shows the listing seller successfully:
/html/body/div[1]/div/section[6]/div/div[3]/div/div[1]/a/div/div[1]/div/div[2]/div/div
driver.get('https://swappa.com/mobile/buy/apple-iphone-xs/t-mobile')
pricing = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[#id="listing_previews"]/div[1]/a/div/div[1]/div/div[2]/div/div[2]/span/sup'))).text
Use getattribute and switch to the span.
pricing = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[#id="listing_previews"]/div[1]/a/div/div[1]/div/div[2]/div/div[2]/span')))
print(pricing.get_attribute('textContent'))
Outputs
$379
The reason you are not getting values using element.text because span element is hidden by parent tag.
<div class="col-xs-3 col-sm-2 hidden-md hidden-lg text-right">
<span class="price"><sup>$</sup>379</span>
</div>
Instead of .text you need to use textContent attribute which retrieves the value from hidden nodes.
Use the following code block to retrieve all the products price.
driver.get('https://swappa.com/mobile/buy/apple-iphone-xs/t-mobile')
allproductdetails=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.listing_row")))
for prod in allproductdetails:
print(prod.find_element_by_css_selector("div.media-body span.price").get_attribute("textContent"))
I'm trying to click on an element (radio button) using Selenium (in Python), I can't disclose the URL because it's a private corporate intranet, but will share the relevants part of code.
So basically this element is within an iframe, thus, I've used the following code to get the element:
# Select the item on main DOM that will udpate the iframe contents
wait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[#id='sm20']"))).click()
# Don't sleep, but only WedDriverWait...
wait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#name='ifrInterior']")))
# Select the element inside the iframe and click
wait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='gestionesPropias_Equipo']"))).click()
The HTML code for the element is this:
<span class="W1">
<input type="radio" name="gestionesPropias_Equipo" value="1" onclick="javascript:indicadorPropiasEquipo(); ocultarPaginacion('1'); limpiarDatosCapaResultado();">
</span>
I'm interested in clicking this because when we click on it a drop-down is enabled:
If clicked then the dropdown is enabled:
The intersting HTML code for this is
<span id="filtroAsignado" class="W30">
<select name="nuumaAsignado" class="W84">
<option value="">[Todo mi equipo]</option></select>
</span>
Debugging a bit Selenium I can see that the elemtn is found:
And this is actually the base64 image of the element, which is the expected radio button
So I'm wondering why the element actually does not get clicked??
UPDATE: Based on request from #DebanjanB, I'm adding the HTML code from the iframe, which is enclosed inside a div in the main page:
<div id="contenido">
<iframe frameborder="0" id="ifrInterior" name="ifrInterior" src="Seguimiento.do?metodo=inicio" scrolling="auto" frameborder="0"></iframe>
</div>
Actually if I look for the word "iframe", there's only one...
Now checking the iframe source itself, has several iframes hidden but the element I need to interact with is in the iframe mentioned above, the only thing that I forgot to mention is that it's inside a form, but I guess that's not relevant? You can see the whole structure in the following image:
Great question. If you know that selenium found the element, you can use Javascript to click the element directly.
The syntax is:
driver.execute_script("arguments[0].click();", element)
You can also do this, which works the same way:
automate.execute_script("arguments[0].click();", wait.until(EC.element_to_be_clickable((By.XPATH, 'Your xpath here'))))
Essentially you are having Selenium run a javascript click on the element you have found which bypasses Selenium. Let me know if this helps!
I don't see any such issue with your code block either. Perhaps you can try out either of the following options:
Using ActionChains:
ActionChains(driver).move_to_element(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='gestionesPropias_Equipo' and #type='radio']")))).click().perform()
Using executeScript() method:
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='gestionesPropias_Equipo' and #type='radio']"))))
Using Python 3+ and Selenium > IE Driver
I am running a python script to automate filling out a form on a page. The reason for finding the parent element is because there are multiple elements with the same class and no other identifiers other then the text based title.
HTML:
<div class="pg_BoxContents"> /* Div I am trying to select */
<div class="Title">Replay Permissions:</div>
</div>
Python / Selenium:
replay_form = browser.find_element_by_xpath("//span[contains(text(),'Replay Permissions:')]/ancestor::div[contains(#class, 'pg_BoxContents')]")
Error:
selenium.common.exceptions.NoSuchElementException: Message: Unable to find element with xpath == //span[contains(text(),'Details:')]/ancestor::div[contains(#class, 'pg_BoxContents')]
Replace:
//span[contains(text(),'Replay Permissions:')]/ancestor::div[contains(#class, 'pg_BoxContents')]
with:
//div[contains(text(),'Replay Permissions:')]/ancestor::div[contains(#class, 'pg_BoxContents')]
You are using //span[... where there is no span in HTML you have provided. But there is a div.
I don't understand why when I go to this page
https://www.ryanair.com/it/it/voli-low-cost/?from=BGY&out-from-date=2018-09-01&out-to-date=2018-09-30&budget=60&trip-length-from=1&trip-length-to=3
or whatever page from that website, Selenium doesn't allow me on the script to run the next page button with the code:
next_page = browser.find_element_by_xpath('/html/body/div[2]/main/div/div[2]/farefinder-widget/div/div[2]/div[2]/div/div/core-pagination/div/div[2]/a')
next_page.click()
On the HTML code:
<a ng-class="{'disabled': $ctrl.isLastPage($ctrl.currentPageIndex)}" ng-click="$ctrl.handleNextClick()" class="core-link"><span class="nav-label">Successivo</span><core-icon icon-id="glyphs.chevron-right" class="icon-22 fill-ryanair-bright-blue"><div><svg tabindex="-1" focusable="false" role="img"><use xlink:href="/it/it/voli-low-cost/?from=BGY&out-from-date=2018-09-01&out-to-date=2018-09-30&budget=60&trip-length-from=1&trip-length-to=3#glyphs.chevron-right" ng-href="/it/it/voli-low-cost/?from=BGY&out-from-date=2018-09-01&out-to-date=2018-09-30&budget=60&trip-length-from=1&trip-length-to=3#glyphs.chevron-right"></use></svg></div></core-icon></a>
Whereas If I write it on the shell, oddly enough it does work, with the same exact lines. I've also tried to give it some seconds with time.sleep, but nothing happened.
Pagination appears to be disabled for this element, i.e. <a ng-class="{'disabled': $ctrl.isLastPage..... It works for me if I click on the parent div, like so:
browser.find_element_by_css_selector(".next.vertical-center").click()
Or, if you want use your xpath, you just need to remove the last 'a':
next_step = browser.find_element_by_xpath('/html/body/div[2]/main/div/div[2]/farefinder-widget/div/div[2]/div[2]/div/div/core-pagination/div/div[2]')
For readability, I recommend the CSS selector.
EDIT: You are probably getting an exception because the element is not visible. Just need to scroll the element into view.
element = browser.find_element_by_css_selector(".next.vertical-center")
browser.execute_script("arguments[0].scrollIntoView();", element)
element.click()