Develop Reference

Select a html a tag with specified display content

I'm new to scrapy and have been struggling for this problem for hours.
I need to scrape a page, with its source somehow looks like this:
<tr class="odd">
<td class="pfama_PF02816">Pfam</td>
<td>Alpha_kinase</td>
<td>1389</td>
<td>1590</td>
<td class="sh" style="display: none">21.30</td>
</tr>
I need to get the information of the tr.odd tag, if and only if the a tag has "Alpha_kinase" value
I can get all of those content (including "Alpha_kinase", 1389, 1590 and many other values) and then process the output to get "Alpha_kinase" only, but this approach will be significantly fragile and ugly. Currently I have to do that way:
positions = response.css('tr.odd td:not([class^="sh"]) td a::text').extract()
then do a for-loop to check.
Is there any condition (like td.not above) expression to put in response.css to solve my problem?
Thanks in advance. Any advice will be highly appreciated!

You can use another selector: response.xpath to select element from the html,
and filter the text with xpath contains function.
>>> response.xpath("//tr[#class='odd']/td/a[contains(text(),'Alpha_kinase')]")
[<Selector xpath="//tr[#class='odd']/td/a[contains(text(),'Alpha_kinase')]" data='<a href="http://pfam.xfam.org/family/Alp'>]

I assume there are multiple such tr elements on the page. If so, I would probably do something like:
# get only rows containing 'Alpha_kinase' in link text
for row in response.xpath('//tr[#class="odd" and contains(./td/a/text(), "Alpha_kinase")]'):
# extract all the information
item['link'] = row.xpath('./td[2]/a/#href').extract_first()
...
yield item

How to search for the text within an "a href" tag and click?

I've got a table with a bunch of links. The IDs are all unique but do not correspond to the actual text that is displayed so I'm having some trouble.
Ex.
<tr>
<td><a id="011" href="/link">Project 1</a></td>
</tr>
<tr>
<td><a id="235" href="/link">Project 2</a></td>
</tr>
<tr>
<td><a id="033" href="/link">Project 3</a></td>
</tr>
<tr>
<td><a id="805" href="/link">Project 4</a></td>
</tr>
I only know the text within the ahref (ie. Project 1) and I want to search for it and click it. I haven't been able to figure this out and I've been playing around with find_element_by_xpath for a while.
I've been using
selectproject = browser.find_element_by_xpath("//a[contains(.,projectname)]").click();
(projectname is a variable that changes every iteration)
I think it works to find the element since the script runs but it doesn't click. I think it's because I'm not actually searching for the ahref and just for the text?

Here is the Answer to your Question:
If you want to click the link with text Project 1 you can use the following line of code:
browser.find_element_by_xpath("//a[contains(text(),'Project 1')]").click()
or
browser.find_element_by_xpath("//a[#id="011"][contains(text(),'Project 1')]").click()
Update:
As you mentioned the Project 1 part is dynamic so you can try to construct a separate function() for clicking these links. Call the function with all the projectnames one by one as follows (the function is in Java consider to convert as per your required language binding):
public void clickProject(String projectName)
{
browser.findElement(By.xpath("//a[.='" + projectName + "']")).click();
}
Now you can call from your main() class as: clickProject(Project1)
Let me know if this Answers your Question.

If your requirement is to "click on the link Project 1", then you should use that as the locator. No need to mess around with XPath.
browser.find_element_by_linkText("Project 1").click();
// or the more flexible
browser.find_element_by_partialLinkText("Project 1").click();
The .find_element_by_partialLinkText() locator strategy should account for any extra whitespace padding due to the extra span element.
Note: I write Java, so the above Python syntax may be off. But those methods must exist.

Iteratively reading a specific element from a <table> with Selenium for Python

I am trying to read in information from this table that changes periodically. The HTML looks like this:
<table class="the_table_im_reading">
<thead>...</thead>
<tbody>
<tr id="uc_6042339">
<td class="expansion">...</td>
<td>
<div id="card_6042339_68587" class="cb">
TEXT I NEED TO READ
</td>
<td>...</td>
more td's
</tr>
<tr id="uc_6194934">...</tr>
<td class="expansion">...</td>
similar as the first <tr id="uc...">
I was able to get to the table using:
table_xpath = "//*[#id="content-wrapper"]/div[5]/table"
table_element = driver.find_element_by_xpath(table_xpath)
And I am trying to read the TEXT I NEED TO READ part for each unique <tr id="uc_unique number">. The id=uc_unique number changes periodically, so I cannot use find element by id.
Is there a way reach that element and read that specific text?

Looks like you can search via the anchor-element link (href-attribute), since I guess this will not change.
via xpath:
yourText = table_element.find_element_by_xpath(.//a[#href='/blahsomelink']).text
UPDATE
OP mentioned that his link is also changing (with each call?), which means that the first approach is not for him.
if you want the text of the first row-element you can try this:
yourText = table_element.find_element_by_xpath(.//tr[1]//a[#class='cl']).text
if you know for example that the link element is always in the second data-element of the first row and there is only one link-element, then you can do this:
yourText = table_element.find_element_by_xpath(.//tr[1]/td[2]//a).text
Unless you provide more detailed requirements as to what you are really searching for, this will have to suffice so far...
Another UPDATE
OP gave more info regarding his requirement:
I am trying to get the text in each row.
Given there is only one anchor-element with class cl in each tr element you can do the following:
elements = table_element.find_elements_by_xpath(.//tr//a[#class='cl'])
for element in elements:
row_text = element.text
Now you can do whatever you need with all these texts...

It looks like you have a few options.
If all you want is the first A, it might be as simple as
table_element.find_element_by_css_selector("a.cl")).text
or the little more specific
table_element.find_element_by_css_selector("div.cb > a.cl")).text
If you want all the As, try the find_elements_* versions of the above.

I managed to find the elements I needed using .get_attribute("textContent") instead of .text , a tip from Get Text from Span returns empty string

Count the number of elements found in selenium python

I have the following html page where I am trying to locate the word silver and keep count on how many were found.
This is just two showing here but it can generate more so I don't know the exact count.
<tr id="au_4" class="odd">
<td>Silver</td>
</tr>
<tr id="au_4" class="even">
<td>Silver</td>
</tr>
This is what I tried but no luck:
count = driver.find_elements_by_xpath("//td[text()='Silver']")

count is a list of all elements that were found. In order to find its length, you should:
len(count)
I highly recommend you to go through the docs to better understand how Selenium works.

It would be quite faster to retrieve count via execute_script then getting len from result of find_elements_by method call:
script = "return $(\"td:contains('{}')\").length".format(text="Silver")
count = driver.execute_script(script)
Sample above suits to you If you use jQuery, another ways of retrieving elements by text value can be found in How to get element by innerText SO question ...

How to make XPath select multiple table elements with identical id attributes?

I'm currently trying to extract information from a badly formatted web page. Specifically, the page has used the same id attribute for multiple table elements. The markup is equivalent to something like this:
<body>
<div id="random_div">
<p>Some content.</p>
<table id="table_1">
<tr>
<td>Important text 1.</td>
</tr>
</table>
<h4>Some heading in between</h4>
<table id="table_1">
<tr>
<td>Important text 2.</td>
<td>Important text 3.</td>
</tr>
</table>
<p>How about some more text here.</p>
<table id="table_1">
<tr>
<td>Important text 4.</td>
<td>Important text 5.</td>
</tr>
</table>
</div>
</body>
Clearly this is incorrectly formatted HTML, due to the multiple use of the same id for an element.
I'm using XPath to try and extract all the text in the various table elements, utilising the language through the Scrapy framework.
My call, looks something like this:
hxs.select('//div[contains(#id, "random_div")]//table[#id="table_1"]//text()').extract()
Thus the XPath expression is:
//div[contains(#id, "random_id")]//table[#id="table_1"]//text()
This returns: [u'Important text 1.'], i.e., the contents of the first table that matches the id value "table_1". It seems to me that once it has come across an element with a certain id it ignores any future occurrences in the markup. Can anyone confirm this?
UPDATE
Thanks for the fast responses below. I have tested my code on a page hosted locally, which has the same test format as above and the correct response is returned, i.e.,
`[u'Important text 1.', u'Important text 2.', . . . . ,u'Important text 5.']`
There is therefore nothing wrong with either the Xpath expression or the Python calls I'm making.
I guess this means that there is a problem on the webpage itself which is either screwing up XPath or the html parser, which is libxml2.
Does anyone have any advice as to how I can dig into this a bit more?
UPDATE 2
I have successfully isolated the problem. It is actually with the underlying parsing library, which is lxml (which provides Python bindings for the libxml2 C library.
The problem is that the parser is unable to deal with vertical tabs. I have no idea who coded up the site I am dealing with but it is full of vertical tabs. Web browser seem to be able to ignore these, which is why running the XPath queries from Firebug on the site in question, for example, are successful.
Further, because the above simplified example doesn't contain vertical tabs it works fine. For anyone who comes across this issue in Scrapy (or in python generally), the following fix worked for me, to remove vertical tabs from the html responses:
def parse_item(self, response):
# remove all vertical tabs from the html response
response.body = filter(lambda c: c != "\v", response.body)
hxs = HtmlXPathSelector(response)
items = hxs.select('//div[contains(#id, \"random_div\")]' \
'//table[#id="table_1"]//text()').extract()

With Firebug, this expression:
//table[#id='table_1']//td/text()
gives me this:
[<TextNode textContent="Important text 1.">,
<TextNode textContent="Important text 2.">,
<TextNode textContent="Important text 3.">,
<TextNode textContent="Important text 4.">,
<TextNode textContent="Important text 5.">]
I included the td filtering to give a nicer result, since otherwise, you would get the whitespace and newlines between the tags. But all in all, it seems to work.
What I noticed was that you query for //div[contains(#id, "random_id")], while your HTML snippet has a tag that reads <div id="random_div"> -- the _id and _div being different. I don't know Scrapy so I can't really say if that does something, but couldn't that be your issue as well?

count(//div[#id = "random_div"]/table[#id= "table_1"])
This xpath returns 3 for your sample input. So your problem is not with the xpath itself rather with the functions you use to extract the nodes.