Develop Reference

XPATH target div and image in loop?

Here's the document struvture:
<div class="search-results-container">
<div>
<div class="feed-shared-update-v2">
<div class="update-components-actor">
<div class="update-components-actor__image">
<img class="presence-entity__image" src="https://www.testimage.com/test.jpg"/>
<span></span>
<span>test</span>
</div>
</div>
</div>
</div>
<div>
<div class="feed-shared-update-v2">
<div class="update-components-actor">
<div class="update-components-actor__image">
<img class="presence-entity__image" src="https://www.testimage.com/test.jpg"/>
<span></span>
<span>test</span>
</div>
</div>
</div>
</div>
</div>
not sure the best way to do this but hoping someone can help. I have a for loop that grabs all the divs that precede a div with class "feed-shared-update-v2". This works:
elements = driver.find_elements(By.XPATH, "//*[contains(#class, 'feed-shared-update-v2')]//preceding::div[1]");
I then run a for loop over it:
for card in elements:
however i'm having trouble trying to target the img and the second span in these for loops. I tried:
for card in elements:
profilePic = card.find_element(By.XPATH, ".//following::div[#class='update-components-actor']//following::img[1]").get_attribute('src')
text = card.find_element(By.XPATH, ".//following::div[#class='update-components-text']//following::span[2]").text
but this produces a error saying:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//following::div[#class='update-components-actor']//following::img[1]"}
so I'm hoping someone can point me in the right direction as to what i'm doing wrong. I know its my xpath syntax and i'm not allowed to chain "followings" (although even just trying .//following doesn't work, so is ".//" not the right syntax?) but i'm not sure what the right syntax should be, especially since the span does not have a class. :(
Thanks!

I guess you are overusing the following:: axis. Simply try the following (no pun intended):
For your first expression use
//*[contains(#class, 'feed-shared-update-v2')]/..
This will select the parent <div> of the <div class="feed-shared-update-v2">. So you will select the whole surrounding element.
To retrieve the children you want, use these XPaths: .//img/#src and .//span[2]. Full code is
for card in elements:
profilePic = card.find_element(By.XPATH, ".//img").get_attribute('src')
text = card.find_element(By.XPATH, ".//span[2]").text
That's all. Hope it helps.

It seems in the span that there is not such class of div called: update-components-text
did you mean: update-components-actor?
Im not such a fan of xpath, but when i copied your html and img selector, it did find me 2 img, maybe you are not waiting for the element to load, and then it fails?
try using implicit/explicit waits in your code.
I know you are using xpath, but concider using css
This might do the trick:
.feed-shared-update-v2 span:nth-of-type(2)
And if you want a css of the img:
.feed-shared-update-v2 img

Selenium + Python: Print the text attribute of an element

I would like to navigate through a website, find an element and print it.
Python version: 3.10; Selenium Webdriver: Firefox; IDE: PyCharm 2021.3.2 (CE);
OS: Fedora 35 VM
I am able to navigate to the appropriate page where the text is generated in a drop down menu.
When I locate the element by CSS Selector and attempt to print it, the output does print the text "None".
I would like it to print the Plan Name which in this case is "Dual Complete Plan 1".
The element is not always present so I also need to catch any exceptions.
The relevant HTML code of the element I am trying to print:
<span class="OSFillParent" data-expression="" style="font-size: 12px; margin-top: 5px;">Dual Complete Plan 1</span>
More of the HTML code of the element I am trying to print (element I am trying to capture is below the fourth div):
<td data-header="Plan Name">
<div id="b8-b40-l1_0-132_0-$b2" class="OSBlockWidget" data-block="Content.AccordionItem">
<div id="b8-b40-l1_0-132_0-b2-SectionItem" class="section-expandable open is--open small-accordion" data-container="" data-expanded="true" aria-expanded="true" aria-disabled="false" role="tab">
<div id="b8-b40-l1_0-132_0-b2-TitleWrapper" class="section-expandable-title" data-container="" style="cursor: pointer;" role+"button" aria-hidden="false" aria-expanmded="true" tabindex="0" aria-controls="b8-b40-l1_0-132_0-b2-Content" EVENT FLEX
<div id="b8-b40-l1_0-132_0-b2-Title" class="dividers full-width">
<span class="OSFillParent" data-expression="" style="font-size: 12px; margin-top: 5px;">Dual Complete Plan 1</span>
</div>
<div class="section-expandable-icon" data-container="" aria-hidden="true"
::after
</div>
</div>
<div id="b8-b40-l1_0-132_0-b2-ContentWrapper" class="section-expandable-content no-padding is--expanded" data-container="" tabindex="0" aria-hidden="false" aria-labelledby="b8-b40-l1_0-132_0-b2-TitleWrapper">
<div id="b8-b40-l1_0-132_0-b2-Content" role="tabpanel">
<a data-link="" href="https://www.communityplan.com" target="_blank" title="Click for more information"> EVENT
<span class="OSFillParent" data-expression="" style="font-size: 12px;">www.CommunityPlan.com</span>
</a>
<span class="OSFillParent" data-expression="" style="font-size: 12px:">Phone Number: 8005224700</span>
</div>
</div>
</div>
</div>
</td>
My relevant Selenium code:
# Find the Plan Name & if present set it to the variable "Advantage"
try:
Advantage = (WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "#b8-b40-l1_0-132_0-b2-Title > span:nth-child(1)"))).get_attribute("value"))
except:
pass
print('\033[91;46m', Advantage, '\033[0m')
I expect the output to be "Dual Complete Plan 1", which is what I see on the screen and in the HTML. Instead I get the following:
None
Apparently the "Advantage" variable is being set to "None".
Why?
I can see the text "Dual Complete Plan 1" that I want to print in the HTML code above.
What am I doing wrong?
I feel like I need a primer on "get attribute"?

To get the text Dual Complete Plan 1 you need to use
element.text
or
element.get_attribute("innerHTML")
or
element.get_attribute("textContent")
Instead of presence_of_element_located() use visibility_of_element_located()
and following css selector to identify
div[id*='Title'] > span.OSFillParent
Or
div.dividers.full-width > span.OSFillParent
Code:
try:
Advantage = WebDriverWait(driver, 5).until(
EC.visibility_of_element_located((By.CSS_SELECTOR, "div[id*='Title'] > span.OSFillParent"))).text
except:
pass
print(Advantage )

Python, Selenium: How to get text next to element

I'm fairly new to selenium and I'm trying to get the text of a cell next to a known element.
This is an excerpt of a webtable:
<div class="row">
<div class="cell">
text-to-copy
</div>
<div class="cell">
<input type="text" size="10" id="known_id" onchange="update(this.id);" onclick="setElementId(this.id);"/>
X
</div>
<div class="cell right">
<div id="some_id">?</div>
</div>
</div>
It looks something like this:
From this table I would like to get the text-to-copy with selenium. As the composition of the table can vary, there is no way to know that cells xpath. Therefore I can not use selenium_driver.find_element_by_xpath(). The only known thing is the id of the cell next to it (id=known_id).
The following pseudo code is to illustrate what I'm looking for:
element = selenium_driver.find_element_by_id("known_id")
result = element.get_visible_text_from_cell_before_element()
Is there a way to get the visible text (text-to-copy) with selenium?

I believe you can fairly use xpath, all other locators that Selenium supports would not work, becasue we have to traverse upward in DOM.
The below xpath is dependent on known_id
//input[contains(#id,'known_id')]/../preceding-sibling::div
You have to either use .text or .get_attribute etc to get the text.
Sample code :
time.sleep(5)
element = selenium_driver.find_element_by_xpath("//input[contains(#id,'known_id')]/../preceding-sibling::div").get_attribute('innerText')
print(element)

Alternative to time.sleep() in selenium using python while web scraping?

I need to scrape price of certain listed food items basis different locations in the country. There's an input text box that allows me to enter the name of the city & pressing "Enter" shows me the list of items available in that city.
Here's how I am trying to automate this:
driver.get("https://grofers.com/")
ele = driver.find_element_by_xpath("//input[#data-test-id='area-input-box']")`
ele.send_keys(area)
ele.send_keys(Keys.RETURN)
Here's the HTML I'm working with:
<div style="margin-left: 51px; height: 36px;">
<div style="display: flex; height: 100%;">
<button class="btn location-box mask-button">Detect my location</button>
<div class="oval-container">
<div class="oval">
<span class="separator-text">
<div class="or">OR</div>
</span>
</div>
</div>
<div style="width: 220px;">
<div class="modal-right__input-wrapper">
<div class="display--table full-width">
<div class="display--table-cell full-width">
<div id="map-canvas"></div>
<div class="Select location-search-input-v1 is-searchable Select--single">
<div class="Select-control">
<div class="Select-multi-value-wrapper" id="react-select-2--value">
<div class="Select-placeholder">Type your city Society/Colony/Area</div>
<div class="Select-input" style="display: inline-block;">**<input data-test-id="area-input-box" aria-activedescendant="react-select-2--value" aria-expanded="false" aria-haspopup="false" aria-owns="" role="combobox" value="">**</div>
</div>
<span class="Select-arrow-zone"><span class="Select-arrow"></span></span>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
The problem is - after send_keys, the website takes time to autofill the input box AFTER WHICH I need to press enter.
I tried using time.sleep(2) after send_keys but this leads to pop-up disappearing & a StaleElementException when I do Keys.RETURN.
Have been stuck on this for quite some time now. Any help/pointers would be appreciated.

Selenium actually has an article on this with Explicit and Implicit waits, I think this is the one you're looking for:
# Wait until an element with id='myNewInput' has class 'myCSSClass'
wait = WebDriverWait(driver, 10)
element = wait.until(element_has_css_class((By.ID, 'myNewInput'), "myCSSClass"))
https://selenium-python.readthedocs.io/waits.html That's the article

You can also create custom wait conditions when none of the previous convenience methods fit your requirements. A custom wait condition can be created using a class with call method which returns False when the condition doesn’t match.
class element_has_css_class(object):
"""An expectation for checking that an element has a particular css class.
locator - used to find the element
returns the WebElement once it has the particular css class
"""
def __init__(self, locator, css_class):
self.locator = locator
self.css_class = css_class
def __call__(self, driver):
element = driver.find_element(*self.locator) # Finding the referenced element
if self.css_class in element.get_attribute("class"):
return element
else:
return False
# Wait until an element with id='myNewInput' has class 'myCSSClass'
wait = WebDriverWait(driver, 10)
element = wait.until(element_has_css_class((By.ID, 'myNewInput'), "myCSSClass"))

Unable to locate element Selenium webdriver || Python

<div class="container-fluid ">
<div class="navbar-header">
<span id="problem_hide_search" class="nav navbar-left">
<span id="ca660735dba5d3003d7e5478dc9619b2_title" class="list_search_title navbar-text " style="float: left; display:inherit">Go to</span>
<div style="float: left; display:inherit">
<div class="input-group" style="width: 300px;">
<span class="input-group-addon input-group-select">
<label class="sr-only" for="ca660735dba5d3003d7e5478dc9619b2_text">Search</label>
<input id="ca660735dba5d3003d7e5478dc9619b2_text" class="form-control" name="ca660735dba5d3003d7e5478dc9619b2_text" style="width: 150px;" placeholder="Search"/>
</div>
</div>
<script data-comment="widget search load event">addLoadEvent(function () { new GlideWidgetSearch('ca660735dba5d3003d7e5478dc9619b2', 'problem', 'true'); });</script>
Am trying to locate the Search box by switching into iframe and selecting by
search_box = driver.find_element_by_xpath('//*#id="ca660735dba5d3003d7e5478dc9619b2_text"]')
But i get error unable to locate Message: no such element: Unable to locate element:
Even thought I find one matching node.

As you mentioned in your question that you are trying to locate the Search box by switching into iframe and selecting as per the best practices you should :
Induce WebDriverWait for the frame to be available to switch as follows :
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(By.ID,"id_of_iframe"))
Here you will find a detailed discussion How can I select a html element no matter what frame it is in in selenium?
While you look out for an element within an <iframe> tag induce WebDriverWait with proper expected_conditions. Considering the fact that you intend to send text to the element you can use the following line of code :
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='navbar-header']//input[#class='form-control' and contains(#id,'_text')]"))).send_keys("hello")