I've been trying to implement tests to check for field validation in forms. A check for specific field error messages was straightforward, but I've also tried a generic check to identify the parent element of a field for an error class. This however isn't working.
A field with an error has the following HTML;
<div class="field clearfix error ">
<div class="error">
<p>Please enter a value</p>
</div>
<label for="id_fromDate">
<input id="id_fromDate" type="text" value="" name="fromDate">
</div>
So to check for an error I've got the following function;
def assertValidationFail(self, field_id):
# Checks for a div.error sibling element
el = self.find(field_id)
try:
error_el = el.find_element_by_xpath('../div[#class="error"]')
except NoSuchElementException:
error_el = None
self.assertIsNotNone(error_el)
So el is the input field, but then the xpath always fails. I believed that ../ went up a level in the same way that command line navigation does - is this not the case?
Misunderstood your question earlier. You may try the following logic: find the parent div, then check if it contains class error, rather than find parent div.error and check NoSuchElementException.
Because .. is the way to go upper level, ../div means parent's children div.
// non-working code, only the logic
parent_div = el.find_element_by_xpath("..") # the parent div
self.assertTrue("error" in parent_div.get_attribute("class"))
When you're using a relative xpath (based on an existing element), it needs to start with ./ like this:
el.find_element_by_xpath('./../div[#class="error"]')
Only after the ./ can you start specifying xpath nodes etc.
Related
I am trying to crawl the realtime Bitcoin-HKD Currency from https://www.coinbase.com/pt-PT/price/ with python3.
The only way I found to locate it specificly in the HTML is by this tage a with href="/pt-PT/price/bitcoin"
<a href="/pt-PT/price/bitcoin" title="Visite a moeda Bitcoin" data-element-handle="asset-highlight-top-daily-volume" class="Link__A-eh4rrz-0 hfBqui AssetHighlight__StyledLink-sc-1srucyv-1 cbFcph" color="slate">
<h2 class="AssetHighlight__Title-sc-1srucyv-2 jmJxYl">Volume mais alto (24 h)</h2>
<div class="Flex-l69ttv-0 gaVUrq">
<img src="https://dynamic-assets.coinbase.com/e785e0181f1a23a30d9476038d9be91e9f6c63959b538eabbc51a1abc8898940383291eede695c3b8dfaa1829a9b57f5a2d0a16b0523580346c6b8fab67af14b/asset_icons/b57ac673f06a4b0338a596817eb0a50ce16e2059f327dc117744449a47915cb2.png" alt="Visite a moeda Bitcoin" aria-label="Visite a moeda Bitcoin" loading="lazy" class="AssetHighlight__AssetImage-sc-1srucyv-5 lcjcxh"/>
<div class="Flex-l69ttv-0 kvilOX">
<div class="Flex-l69ttv-0 gTbYCC">
<h3 class="AssetHighlight__SubTitle-sc-1srucyv-3 gdcBEE">Bitcoin</h3>
<p class="AssetHighlight__Price-sc-1srucyv-4 bUAWAG">460 728,81 HK$</p>
Here 460 728,81 HK$ is the data wanted.
Thus I applied the following codes:
import bs4
import urllib.request as req
url="https://www.coinbase.com/prthe ice/bitcoin/hkd"
request=req.Request(url,headers={
"user-agent":"..."
})
with req.urlopen(request) as response:
data=response.read().decode("utf-8")
root=bs4.BeautifulSoup(data,"html.parser")
secBitcoin=root.find('a',href="/pt-PT/price/bitcoin")
realtimeCurrency=secBitcoin.find('p')
print(realtimeCurrency.string)
However, it always returns secBitcoin = None. No result matches.
The find function works just fine when I search 'div' label with class parameter.
I have also tried format like
.find('a[href="/pt-PT/price/bitcoin"]')
But nothing works.
It's possible the page is loading the currency values after the initial page load. You could try hitting ctrl+s to save the full webpage and open that file instead of using requests. If that also doesn't work, then I'm not sure where the problem is.
And if that does work, then you'll probably need to use something like selenium to get what you need
href is an attribute of an element and hence I think you cannot find it that way.
def is_a_and_href_matching(element):
is_a = element.name == a
if is_a and element.has_attr(href):
if element['href'] == "/pt-PT/price/bitcoin":
return True
return False
secBitcoins=root.find_all(is_a_and_href_matching)
for secBitcoin in secBitcoins:
p = setBitcoin.find('p')
Is there a way to select the exact "div" in a source of a Beautiful Soup object? For example, let's say we have soup like this:
<div class="dialog-shadow" id="popupMenu1" onblur="hidePopup();" onmouseout="closePopup = contextMenuInputHasFocus() ? null : setTimeout('hidePopup()',500);" onmouseover="if(closePopup!=null){clearTimeout(closePopup);closePopup=null}"></div>
<div id="popupMenu2" onblur="hidePopup();" onmouseout="closePopup = contextMenuInputHasFocus() ? null : setTimeout('hidePopup()',500);" onmouseover="if(closePopup!=null){clearTimeout(closePopup);closePopup=null}"></div>
<div class="shadow" id="popupMenu3" onblur="hidePopup3();hidePopup();" onmouseout="closePopup = setTimeout('hidePopup();', 500); closePopup3 = setTimeout('hidePopup3()',500);" onmouseover="if(closePopup!=null){clearTimeout(closePopup);closePopup=null};if(closePopup3!=null){clearTimeout(closePopup3);closePopup3=null};"></div>
<div id="container">
<div class="background-menu-dark shadow" id="navHolder">
<span class="customBranding" id="logo" onclick="loadView(V_SUMMARY);" title="Özet Görünümü"><img height="40" src="Branding/SmallBanner.jpg?ts=20140403111116"/></span>
<div id="navigation">
<ul id="navigationLargeWidth">
<li id="mainInboxLink">
And I want to find the third div whose class is "shadow" in this piece of soup. But when I do something like this, it returns None:
soup.find('div',attrs={"class":"shadow"})
I know that it should be something like "ABC-->BC-->C" If i want to find C in the soup, but is there a way that I can find C just by knowing its unique class or ID?
(soup.select("div:nth-of-type(3))) is not what I'm looking for)
I see only 2 divs with that class. However, the reason your nth-of-type could be failing is due to you not including the class. Unless there is some reason (you haven't given) as to why nth-of-type itself is not acceptable.
div.shadow:nth-of-type(3)
without proper html to test with I cannot be sure of index or whether content is dynamically loaded (if from webpage)
If you are trying to dynamically construct the path then something like this?
For a div with a unique class
select_one('div.shadow')
I am trying to pull the text out of a tag that follows an element I'm starting with. The HTML looks like this, with multiple entries of the same structure:
<h5>
Title
</h5>
<div class="author">
<p>"Author A, Author B"</p>
</div>
<div id="abstract-more#####" class="collapse">
<p>
<strong>Abstract:</strong>
"Text here..."
</p>
<p>...</p>
So once I've isolated a given title element/node (stored as 'paper'), I want to store the author and abstract text. It works when I use this to get the author:
author = paper.find_element_by_xpath("./following::div[contains(#class, 'author')]/p").text
But is returning a blank output for 'abstract' when I use this:
abstract = paper.find_element_by_xpath("./following::div[contains(#id, 'abstract-more')]/p").text
Why does it work fine for the author but not for the abstract? I've tried using .// instead of ./ and other slight tweaks but to no avail. I also don't know why it's not giving an error out and saying it can't find the abstract element and is instead just returning a blank...
Try this:
//div[contains(#id, 'abstract-more')]/p[1]
Please use starts-with in xpath instead of contains.
XPath: .//div[starts-with(#id, 'abstract-more')]/p"
abstract = paper.find_element_by_xpath(".//div[starts-with(#id, 'abstract-more')]/p").text
You can try this xpath :
//div[#class="author"]/following-sibling::div[contains(#id,'abstract-more')]/p[1]
in code :
author = paper.find_element_by_xpath("//div[#class="author"]/following-sibling::div[contains(#id,'abstract-more'')]/p[1]")
print(author.text)
Let's say we have this website https://www.coinichiwa.com/ which has a BET AMOUNT input box. It's html is:
<input autocomplete="off" id="betFa" name="a" maxlength="20" value="0.00000000" class="betinput" style="">
I need to add some value into it. Here is my code:
browser = webdriver.Firefox()
browser.get('https://www.coinichiwa.com')
browser.find_element_by_id("betFa").send_keys("0.00000005")
print browser.find_element_by_xpath("//input[contains(#id,'betFa')]").text
But it's neither setting it's value to "0.00000005" nor it printing the value of input.
I'm not sure what's going wrong. Can you suggest?
Why it's not working?
You need to clear() the text input first:
bet_fa = browser.find_element_by_id("betFa")
bet_fa.clear()
bet_fa.send_keys("0.00000005")
As for the your second problem - this is an input and the value you enter into it is kept inside the value attribute, not the text. Use get_attribute() method:
browser.find_element_by_xpath("//input[contains(#id,'betFa')]").get_attribute('value')
I am using the Zope testbrowser which has been recommended in my last question. The problem that I am facing is that I can use the getControl function to control different objects like: password, username etc.
I am trying to submit the page to get to the next page but the submit button has no 'name' variable, just an 'id' variable. 'Submit' is written as follows:
<input type="submit" id="lgn_button" class="button" tabindex="3" accesskey="s" />
and the other objects are written as:
<input type="password" class="button" name="password" id="password" size="24" maxlength="20" accesskey="p" tabindex="2" value=""/></td>
I have no access to change this. The python zope code I am using to gain control of the 'password' object is:
browser.getControl(name='password')
The submit button doesn't have 'name' so I have written:
browser.getControl(id='lgn_button')
This prints out the error that 'id' is invalid:
TypeError: getControl() got an unexpected keyword argument 'id'
Is there any way to gain control of one of the other values in 'submit'.
Thanks for any help.
I assume that, for one reason or another, you can't add a 'name' attribute to your tag, but if it's only a 'name' that you can't add, you can explicitly set a 'value=Submit' (instead of relying on the default one, which is Submit) and then use browser.getControl('Submit')
Failing that, you can do something along the lines of
for form in browser.mech_browser.forms():
for control in form.controls:
if control.id == 'lgn_button':
return control
You might even want to extend Browser.getControl() with that and contribute it back to zope.testbrowser. ;)