I am trying to grab data from something that looks like this:
<html>
<table>
<tr>
<td>
<div>
<tr>
<td>
<div>
I think this is basically what I was looking at earlier, so I did $x('//table/tr/td/div'). This returned like 10 divs, all the ones I wanted. However, I couldn't find a way to navigate through these 10 divs. Wouldn't it just be $x('//table/tr/td/div[?]'), where ? can be 1-10? BTW this is a dynamic table, each div is a different module to navigate to a part of the page, so I am trying to automate logging in and going to a certain module, yet I can only select all 10 and not a single module. In this html, div 1 and div 2 would be something like "security module" and "ticketing module".
First just a heads up, you may want to try Firebug+Firepath on Firefox, which is a more handy tool than Chrome's developer tool.
For your question, you can do this either by xpath or Selenium
Using Xpath's index (starts from one), say you want the 2nd:
# second div's xpath: (//table/tr/td/div)[2]
second_div = driver.find_element_by_xpath("(//table/tr/td/div)[2]")
Using Selenium's index (i.e get all ten divs from your xpath, then index them starting from zero):
all_divs = driver.find_elements_by_xpath("//table/tr/td/div") # all_divs should have 10 elements
second_div = all_divs[1] # zero-based index
Related
I'm currently working with Selenium and Python to create a script that helps with booking free spots on a website for internships so you don't have to check it all the time.
I found the clickable element but now I don't know how to add the condition. I would like to make a text attribute the condition, which is under a td tag below the "button" but don't know how. It shows how many places are available to the total amount of places (x/y, so I'd make it 1/ ). Is it even possible?
Maybe I could do it by the green color but I can't find where it is anchored in the HTML document
browser.get('https://www.pj-portal.de/index_hro.php?PAGE_ID=101')
delay = 2
browser.find_elements_by_xpath("//*[starts-with(text(),'1/')]")
print("found")
Liste1 = browser.find_elements_by_xpath("//*[starts-with(text(),'1/')]")
print(len(Liste1))
B = browser.find_element_by_xpath("//img[#src='images /wunsch_setzen_button.svg']")
B.click()
print("button")
Part of HTML
<img class="aktion_tertial_wunsch_setzen " data-pj_traineeship_tertial_id="166037"
data-pj_general_traineeship_id="8487" data-changetype="24"
src="images/wunsch_setzen_button.svg" alt="+" title="Wunsch">
<td class="hinweise_leer verfuegbar buchungsphase "> </td>
<td class=" tertial_verfuegbarkeit verfuegbar buchungsphase ">1/3</td>
Thanks you in advance
Did you try to use position () method in xpath ?
for example:
// xpath using position() targeting the first element starts-with(text(),'1/'
res= browser.find_elements_by_xpath("//*[starts-with(text(),'1/')][position()=1]")
res[0].click()
I have been struggling with this for a while now.
I have tried various was of finding the xpath for the following highlighted HTML
I am trying to grab the dollar value listed under the highlighted Strong tag.
Here is what my last attempt looks like below:
try:
price = browser.find_element_by_xpath(".//table[#role='presentation']")
price.find_element_by_xpath(".//tbody")
price.find_element_by_xpath(".//tr")
price.find_element_by_xpath(".//td[#align='right']")
price.find_element_by_xpath(".//strong")
print(price.get_attribute("text"))
except:
print("Unable to find element text")
I attempted to access the table and all nested elements but I am still unable to access the highlighted portion. Using .text and get_attribute('text') also does not work.
Is there another way of accessing the nested element?
Or maybe I am not using XPath as it properly should be.
I have also tried the below:
price = browser.find_element_by_xpath("/html/body/div[4]")
UPDATE:
Here is the Full Code of the Site.
The Site I am using here is www.concursolutions.com
I am attempting to automate booking a flight using selenium.
When you reach the end of the process of booking and receive the price I am unable to print out the price based on the HTML.
It may have something to do with the HTML being a java script that is executed as you proceed.
Looking at the structure of the html, you could use this xpath expression:
//div[#id="gdsfarequote"]/center/table/tbody/tr[14]/td[2]/strong
Making it work
There are a few things keeping your code from working.
price.find_element_by_xpath(...) returns a new element.
Each time, you're not saving it to use with your next query. Thus, when you finally ask it for its text, you're still asking the <table> element—not the <strong> element.
Instead, you'll need to save each found element in order to use it as the scope for the next query:
table = browser.find_element_by_xpath(".//table[#role='presentation']")
tbody = table.find_element_by_xpath(".//tbody")
tr = tbody.find_element_by_xpath(".//tr")
td = tr.find_element_by_xpath(".//td[#align='right']")
strong = td.find_element_by_xpath(".//strong")
find_element_by_* returns the first matching element.
This means your call to tbody.find_element_by_xpath(".//tr") will return the first <tr> element in the <tbody>.
Instead, it looks like you want the third:
tr = tbody.find_element_by_xpath(".//tr[3]")
Note: XPath is 1-indexed.
get_attribute(...) returns HTML element attributes.
Therefore, get_attribute("text") will return the value of the text attribute on the element.
To return the text content of the element, use element.text:
strong.text
Cleaning it up
But even with the code working, there’s more that can be done to improve it.
You often don't need to specify every intermediate element.
Unless there is some ambiguity that needs to be resolved, you can ignore the <tbody> and <td> elements entirely:
table = browser.find_element_by_xpath(".//table[#role='presentation']")
tr = table.find_element_by_xpath(".//tr[3]")
strong = tr.find_element_by_xpath(".//strong")
XPath can be overkill.
If you're just looking for an element by its tag name, you can avoid XPath entirely:
strong = tr.find_element_by_tag_name("strong")
The fare row may change.
Instead of relying on a specific position, you can scope using a text search:
tr = table.find_element_by_xpath(".//tr[contains(text(), 'Base Fare')]")
Other <table> elements may be added to the page.
If the table had some header text, you could use the same text search approach as with the <tr>.
In this case, it would probably be more meaningful to scope to the #gdsfarequite <div> rather than something as ambiguous as a <table>:
farequote = browser.find_element_by_id("gdsfarequote")
tr = farequote.find_element_by_xpath(".//tr[contains(text(), 'Base Fare')]")
But even better, capybara-py provides a nice wrapper on top of Selenium, helping to make this even simpler and clearer:
fare_quote = page.find("#gdsfarequote")
base_fare_row = fare_quote.find("tr", text="Base Fare"):
base_fare = tr.find("strong").text
This question might be really specific. I am trying to extract the number of employees from the Wikipedia pages of companies such as https://en.wikipedia.org/wiki/3M.
I tried using the Wikipedia python API and some regex queries. However, I couldn't find anything solid that I could generalize for any company (not considering exceptions).
Also, because the table row does not have an id or a class I cannot directly access the value. Following is the source:
<tr>
<th scope="row" style="padding-right:0.5em;">
<div style="padding:0.1em 0;line-height:1.2em;">Number of employees</div>
</th>
<td style="line-height:1.35em;">89,800 (2015)<sup id="cite_ref-FY_1-5" class="reference">[1]</sup></td>
</tr>
So, even though I have the id of the table - infobox vcard so I couldn't figure out a way to scrape this information using beautifulSoup.
Is there a way to extract this information? It is present in the summary table on the right at the beginning of the page.
Using lxml.etree instead of BeautifulSoup, you can get what you want with an XPath expression:
>>> from lxml import etree
>>> import requests
>>> r = requests.get('https://en.wikipedia.org/wiki/3M')
>>> doc = etree.fromstring(r.text)
>>> e = doc.xpath('//table[#class="infobox vcard"]/tr[th/div/text()="Number of employees"]/td')
>>> e[0].text
'89,800 (2015)'
Let's take a closer look at that expression:
//table[#class="infobox vcard"]/tr[th/div/text()="Number of employees"]/td
That says:
Find all table elements that have attribute class set to infobox
vcard, and inside those elements look for tr elements that have a
child th element that has a child div element that contains the
text "Number of employees", and inside that tr element, get the
first td element.
Why reinvent the wheel?
DBpedia
has this information in RDF triples.
See e.g.
http://dbpedia.org/page/3M
I am trying to read in information from this table that changes periodically. The HTML looks like this:
<table class="the_table_im_reading">
<thead>...</thead>
<tbody>
<tr id="uc_6042339">
<td class="expansion">...</td>
<td>
<div id="card_6042339_68587" class="cb">
TEXT I NEED TO READ
</td>
<td>...</td>
more td's
</tr>
<tr id="uc_6194934">...</tr>
<td class="expansion">...</td>
similar as the first <tr id="uc...">
I was able to get to the table using:
table_xpath = "//*[#id="content-wrapper"]/div[5]/table"
table_element = driver.find_element_by_xpath(table_xpath)
And I am trying to read the TEXT I NEED TO READ part for each unique <tr id="uc_unique number">. The id=uc_unique number changes periodically, so I cannot use find element by id.
Is there a way reach that element and read that specific text?
Looks like you can search via the anchor-element link (href-attribute), since I guess this will not change.
via xpath:
yourText = table_element.find_element_by_xpath(.//a[#href='/blahsomelink']).text
UPDATE
OP mentioned that his link is also changing (with each call?), which means that the first approach is not for him.
if you want the text of the first row-element you can try this:
yourText = table_element.find_element_by_xpath(.//tr[1]//a[#class='cl']).text
if you know for example that the link element is always in the second data-element of the first row and there is only one link-element, then you can do this:
yourText = table_element.find_element_by_xpath(.//tr[1]/td[2]//a).text
Unless you provide more detailed requirements as to what you are really searching for, this will have to suffice so far...
Another UPDATE
OP gave more info regarding his requirement:
I am trying to get the text in each row.
Given there is only one anchor-element with class cl in each tr element you can do the following:
elements = table_element.find_elements_by_xpath(.//tr//a[#class='cl'])
for element in elements:
row_text = element.text
Now you can do whatever you need with all these texts...
It looks like you have a few options.
If all you want is the first A, it might be as simple as
table_element.find_element_by_css_selector("a.cl")).text
or the little more specific
table_element.find_element_by_css_selector("div.cb > a.cl")).text
If you want all the As, try the find_elements_* versions of the above.
I managed to find the elements I needed using .get_attribute("textContent") instead of .text , a tip from Get Text from Span returns empty string
I am able to log on and access my account page, here is a sample of the HTML (modified for brevity and to not exceed the URL limit):
<div class='table m_t_4'>
<table class='data' border=0 width=100% cellpadding=0 cellspacing=0>
<tr class='title'>
<td align='center' width='15'><a></a></td>
<td align='center' width='60'></td>
</tr>
<TR bgcolor=>
<td valign='top' align='center'>1</TD>
<td valign='top' align='left'><img src='/images/sale_small.png' alt='bogo sale' />Garden Escape Planters</TD>
<td valign='top' align='right'>13225</TD>
<td valign='top' align='center'>2012-01-17 11:34:32</TD>
<td valign='top' align='center'>FILLED</TD>
<td valign='top' align='center'><A HREF='https://www.daz3d.com/i/account/orderdetail?order=7886745'>7886745</A></TD>
<td valign='top' align='center'><A HREF='https://www.daz3d.com/i/account/req_dlreset?oi=18087292'>Reset</A>
</TR>
Note that the only item I really need is the first HREF with the "order=7886745'>7886745<"...
And there are several of the TR blocks that I need to read.
I am using the following xpath coding:
browser.get('https://www.daz3d.com/i/account/orderitem_hist?')
account_history = browser.find_element_by_xpath("//div[#class='table m_t_4']");
print account_history
product_block = account_history.find_element_by_xpath("//TR[contains(#bgcolor, '')]");
print product_block
product_link = product_block.find_element_by_xpath("//TR/td/A#HREF")
print product_link
I am using the Python FireFox version of webdriver.
When I run this, the account_history and product_block xpath's seem to work fine (they print as "none" so I assume they worked), but I get a "the expession is not a legal expression" error on the product_link.
I have 2 questions:
1: Why doesn't the "//TR/td/A#HREF" xpath work? It is supposed to be using the product_block - which it (should be) just the TR segment, so it should start with the TR, then look for the first td that has the HREF...correct?
I tried using the exact case used in the HTML, but I think it shouldn't matter...
2: What coding do I need to use to see the content (HTML/text) of the elements?
I need to be able to do this to get the URL I need for the next page to call.
I would also like to see for sure that the correct HTML is being read here...that should be a normal part of debugging, IMHO.
How is the element data stored? Is it in an array or table that I can read using Python? It has to be available somewhere, in order to be of any use in testing - doesn't it?
I apologize for being so confused, but I see a lot of info on this on the web, and yet much of it either doesn't do anything, or it causes an error.
There do not seem to be any "standard" coding rules available...and so I am a bit desperate here...
I really like what I have seen in Selenium up to this point, but I need to get past it in order to make this work!
Edited!
OK, after getting some sleep the first answer provided the clue - find_elements_by_xpath creates a list...so I used that to find all of the xpath("//a[contains(#href,'https://www.daz3d.com/i/account/orderdetail?order=')]"); elements in the entire history, then accessed the list it created...and write it to a file to be sure of what I was seeing.
The revised code:
links = open("listlinks.txt", "w")
browser.get('https://www.daz3d.com/i/account/orderitem_hist?')
account_history = browser.find_element_by_xpath("//div[#class='table m_t_4']");
print account_history.get_attribute("div")
product_links = []
product_links = account_history.find_elements_by_xpath("//a[contains(#href,'https://www.daz3d.com/i/account/orderdetail?order=')]");
print str(len(product_links)) + ' elements'
for index, item in enumerate(product_links):
link = item.get_attribute("href")
links.write(str(index) + '\t' + str(link) + '\n')
And this gives me the file with the links I need...
0 https://www.daz3d.com/i/account/orderdetail?order=7905687
1 https://www.daz3d.com/i/account/orderdetail?order=7886745
2 https://www.daz3d.com/i/account/orderdetail?order=7854456
3 https://www.daz3d.com/i/account/orderdetail?order=7812189
So simple I couldn't see it for tripping over it...
Thanks!
1: Why doesn't the "//TR/td/A#HREF" xpath work? It is supposed to be
using the product_block - which it (should be) just the TR segment, so
it should start with the TR, then look for the first td that has the
HREF...correct?
WebDriver only returns elements, not attributes of said elements, thus:
"//TR/td/A"
works, but
"//TR/td/A#HREF"
or
"//TR/td/A#ANYTHING"
does not.
2: What coding do I need to use to see the content (HTML/text) of the
elements?
To retrieve the innertext:
string innerValue = element.Text;
To retrieve the innerhtml:
This is a little harder, you would need to iterate through each of the child elements and reconstruct the html based on that - or you could process the html with a scraping tool.
To retrieve an attribute:
string hrefValue = element.GetAttribute("href");
(C#, hopefully you can make the translation to Python)
There are other ways too to access an element than browser.find_element_by_xpath.
You can access by for e.g. id, or class
browser.find_element_by_id
browser.find_element_by_link_text
browser.find_element
browser.find_element_by_class_name
browser.find_element_by_css_selector
browser.find_element_by_name
browser.find_element_by_partial_link_text
browser.find_element_by_xpath
browser.find_element_by_tag_name
Each of above has a similar function which returns a list(just replace element with elements
Note: I have separated top two rows as I think they might help you.