I am trying to pull the price of the first listing on this website with the following code, but it's returning nothing but blanks. I am navigating to the website, hitting F12, and then copying the XPATH into the line of code below. Any thoughts on why this wouldn't work?
the following gives nothing:
/html/body/div[1]/div/section[6]/div/div[3]/div/div[1]/a/div/div[1]/div/div[2]/div/div[2]/span/sup
While this shows the listing seller successfully:
/html/body/div[1]/div/section[6]/div/div[3]/div/div[1]/a/div/div[1]/div/div[2]/div/div
driver.get('https://swappa.com/mobile/buy/apple-iphone-xs/t-mobile')
pricing = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[#id="listing_previews"]/div[1]/a/div/div[1]/div/div[2]/div/div[2]/span/sup'))).text
Use getattribute and switch to the span.
pricing = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[#id="listing_previews"]/div[1]/a/div/div[1]/div/div[2]/div/div[2]/span')))
print(pricing.get_attribute('textContent'))
Outputs
$379
The reason you are not getting values using element.text because span element is hidden by parent tag.
<div class="col-xs-3 col-sm-2 hidden-md hidden-lg text-right">
<span class="price"><sup>$</sup>379</span>
</div>
Instead of .text you need to use textContent attribute which retrieves the value from hidden nodes.
Use the following code block to retrieve all the products price.
driver.get('https://swappa.com/mobile/buy/apple-iphone-xs/t-mobile')
allproductdetails=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.listing_row")))
for prod in allproductdetails:
print(prod.find_element_by_css_selector("div.media-body span.price").get_attribute("textContent"))
Related
I'm trying to get the content from a tag, but it raised NoSuchElement even though getting it from an another tag with the same level is successful.
This is the link to website: https://soundcloud.com/pastlivesonotherplanets/sets/spell-jars-from-mars
This is the html code that I access to:
<div class="fullHero__tracksSummary">
<div class="playlistTrackCount sc-font">
<div class="genericTrackCount sc-font large m-active" title="16 tracks">
<div class="genericTrackCount__title">16</div>
<div class="genericTrackCount__subtitle"> Tracks </div>
<div class="genericTrackCount__duration sc-type-small sc-text-body sc-type-light sc-text-secondary">56:07</div>
</div>
</div>
</div>
I'm trying to get the playlist's duration with this code:
try:
tmp=driver.find_element_by_xpath("//div[#class='fullHero__tracksSummary']")
duration=tmp.find_element_by_class_name("genericTrackCount__duration sc-type-small sc-text-body
sc-type-light sc-text-secondary").get_attribute('textContent')
print(duration)
except:
print("None")
It raised error NoSuchElement even though the other two tags was successful.
What is the problem and how can I fix it?
Thank your for your time.
I think you can try directly using xpath //div[contains(#class, 'duration')] OR
//div[contains(#class, 'playlistTrackCount')]/descendant::div[contains(#class, 'duration')]
Without looking at the page, you probably need to wait for the element to load.
You can use either time.sleep(5) 5 being the number of seconds to wait or WebDriverWait(driver, 20) with an expected condition
so your code would look like
import expected_conditions as EC
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CLASS_NAME, '"genericTrackCount__duration sc-type-small sc-text-body
sc-type-light sc-text-secondary"))).text
Also maybe the get_attribute('textContent') is failing, you can just use .text
I'm trying to click on an element (radio button) using Selenium (in Python), I can't disclose the URL because it's a private corporate intranet, but will share the relevants part of code.
So basically this element is within an iframe, thus, I've used the following code to get the element:
# Select the item on main DOM that will udpate the iframe contents
wait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[#id='sm20']"))).click()
# Don't sleep, but only WedDriverWait...
wait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#name='ifrInterior']")))
# Select the element inside the iframe and click
wait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='gestionesPropias_Equipo']"))).click()
The HTML code for the element is this:
<span class="W1">
<input type="radio" name="gestionesPropias_Equipo" value="1" onclick="javascript:indicadorPropiasEquipo(); ocultarPaginacion('1'); limpiarDatosCapaResultado();">
</span>
I'm interested in clicking this because when we click on it a drop-down is enabled:
If clicked then the dropdown is enabled:
The intersting HTML code for this is
<span id="filtroAsignado" class="W30">
<select name="nuumaAsignado" class="W84">
<option value="">[Todo mi equipo]</option></select>
</span>
Debugging a bit Selenium I can see that the elemtn is found:
And this is actually the base64 image of the element, which is the expected radio button
So I'm wondering why the element actually does not get clicked??
UPDATE: Based on request from #DebanjanB, I'm adding the HTML code from the iframe, which is enclosed inside a div in the main page:
<div id="contenido">
<iframe frameborder="0" id="ifrInterior" name="ifrInterior" src="Seguimiento.do?metodo=inicio" scrolling="auto" frameborder="0"></iframe>
</div>
Actually if I look for the word "iframe", there's only one...
Now checking the iframe source itself, has several iframes hidden but the element I need to interact with is in the iframe mentioned above, the only thing that I forgot to mention is that it's inside a form, but I guess that's not relevant? You can see the whole structure in the following image:
Great question. If you know that selenium found the element, you can use Javascript to click the element directly.
The syntax is:
driver.execute_script("arguments[0].click();", element)
You can also do this, which works the same way:
automate.execute_script("arguments[0].click();", wait.until(EC.element_to_be_clickable((By.XPATH, 'Your xpath here'))))
Essentially you are having Selenium run a javascript click on the element you have found which bypasses Selenium. Let me know if this helps!
I don't see any such issue with your code block either. Perhaps you can try out either of the following options:
Using ActionChains:
ActionChains(driver).move_to_element(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='gestionesPropias_Equipo' and #type='radio']")))).click().perform()
Using executeScript() method:
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='gestionesPropias_Equipo' and #type='radio']"))))
Using Selenium with Python.
I've got pages full of links with a class called item-title. I'm trying to iterate through the pages and compile a list of all the link texts and the href attributes that go with them. I want to output the titles and links to a csv file. Here is my code:
myLinks=driver.find_elements_by_class_name("item-title")
for link in myLinks:
out.write(link.text)
out.write (",")
out.write(link.get_attribute("href"))
out.write("\n")
The line that outputs the href value gives the following error:
TypeError: expected a character buffer object
Tried the following:
myLinks=driver.find_elements_by_class_name("item-title")
for link in myLinks:
out.write(link.text)
out.write (",")
out.write(str(link.get_attribute("href")))
out.write("\n")
Error went away, link text is coming through okay, but now the href is coming through as 'None'
Edit to add the HTML
<div class="item-title">
<span class="icons-pinned"></span>
<span class="icons-solved"></span>
<span class="icons-locked"></span>
<span class="icons-moved"></span>
<span class="icons-type"></span>
<span class="icons-reply"></span>
My title
</div>
I think I see the issue now. The is a child element of the div, I need to target that, don't I?
As per the HTML you have shared, the link texts and the href attributes are not within the node identified as find_elements_by_class_name("item-title"). Rather they are within the decendent <a> tag. Hence instead of using find_elements_by_class_name("item-title") we have to use either find_elements_by_xpath or find_elements_by_css_selector as follows :
Using find_elements_by_css_selector :
myLinks=driver.find_elements_by_css_selector("div.item-title > a")
for link in myLinks:
out.write(link.get_attribute("innerHTML"))
out.write (",")
out.write(link.get_attribute("href"))
out.write("\n")
Using find_elements_by_xpath :
myLinks=driver.find_elements_by_xpath("//div[#class='item-title']/a")
for link in myLinks:
out.write(link.get_attribute("innerHTML"))
out.write (",")
out.write(link.get_attribute("href"))
out.write("\n")
I am using Python and Selenium to scrape a webpage, in some cases, I can't get it to work,
*
I would like to access the element with text 'PInt', wich is the second link in the below code.
The xPath for it (copied from Developer console) is: //[#id="submenu1"]/a[2]
<div id="divTest" onscroll="SetDivPosition();" style="height: 861px;">
<div class="menuTitle" id="title1">
</div>
<div class="subtitle" id="submenu1">
<img src="images/spacer.gif" border="0" width="2px" height="12px">
Mov<br>
<img src="images/spacer.gif" border="0" width="2px" height="12px">
PInt<br>
<img src="images/spacer.gif" border="0" width="2px" height="12px">
SWAM / SWIF<br>
</div>
...
A snipped of my code is:
try:
res = driver.find_elements_by_link_text('PInt')
print("res1:{}".format(res))
res = driver.find_element(By.XPATH,'//*[#id="submenu1"]/a[3]')
print("res:{} type[0]:{}".format(res,res[0]))
itm1 = res[0]
itm1.click()
I get the error:
Unable to locate element:
{"method":"xpath","selector":"//*[#id="submenu1"]/a[2]"}
My question is, how can I get the right xPath of the element or any other way to access the element?
UPDATE:
This might be important, the issue with
Message: invalid selector: Unable to locate an element with the xpath expression (and I've tried all proposed solutions) might be that this is after authenticate in the webpage (User + Pwd) before, everything works.
I noticed that the url driver.current_url after login is static (asp page).
Also this part I am trying to access in a frameset and frame
html > frameset > frameset > frame:nth-child(1)
Thanks to #JeffC to point me in the right direction.
since the page has some frames, I manage to access the element first by switching to the right frame (using xPath)
and then access the element.
driver.switch_to.default_content()
driver.switch_to.frame(driver.find_element_by_xpath('html / frameset / frameset / frame[1]'))
driver.find_element_by_xpath("//a[contains(text(),'PInt')]").click()
BTW, in case you want to run the script from a crontab,you need to setup a display:
30 5 * * * export DISPLAY=:0; python /usr/.../main.py
To see a full list of all the ways of selecting elements using selenium, you can read all about it in the documentation.
Using xpath:
res = driver.find_element_by_xpath(u'//*[#id="submenu1"]/a[2]')
Using css selector:
res = driver.find_element_by_css_selector('#submenu1 a:nth-of-type(2)')
Try with any of the below xpath. Sometimes automatically generated xpath does not work.
//a[contains(text(),'PInt')]
or
//div[#id='submenu1']//a[contains(text(),'PInt')]
Also I would suggest you to set some wait time before clicking on above link in case if above xpath does not work
To find xPath in chrome:
Right click the element you want
Inspect, which will open the developer window and highlight the selected element.
Right click the highlighted element and choose copy > copy by xpath
Here is a list of all the different ways to locate an element Locating Elements
HTML information is :
<a title="Create an Account" class="button" href="http://demo.magentocommerce.com/customer/account/create/">
<span>
<span>Create an Account
</span>
</span>
</a>
Create an Account
I am trying with :
create_account_button = driver.find_element_by_xpath("//button[#title='Create an Account']")
create_account_button.click()
but it is not working
Actually assign the WebElement variable, u use the following code
WebElement button = driver.findElement(By.xpath("//a[#title='Create an Account']");
button.click();
Steps
Creating a WebElement variable button and assigning the value to it.
Performing click() on that webelement
For Java follow above steps, for Python use below
driver.find_element_by_xpath('//a[#title='Create an Account']').click()
Xpath you need To use be in <a> not in <button>
So try this Xpath //a[#title='Create an Account']