Develop Reference

Best way to get XPATH and CSS SELECTORS for scraping with Selenium

What is the best way you guys know to get the XPATH AND CSS SELECTOR of a scraped website using selenium?
Someone suggested that I use these XPATH and CSS SELECTORS as parameters for an exercise I'm working on:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[placeholder='Search']"))).send_keys('Tech')
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Cancel']/.."))).click()
These parameters work very well for the exercise. However, I'm unsure on how to get (or "build") those parameters...
If I use Chrome's Inspect > right click > Copy XPATH or Copy Selector, I get some very different parameters that don't seem to work as well, and don't seem to be found by selenium.
#search-bar
//*[#id="app-container"]/div/section/div/div/div[2]/button
Is there a tool or a technique to get better XPATH or CSS SELECTORS as in my first example?

I like the resources shared by #JD2775. They are good to get you started understanding how to construct and understand xpaths and css selectors. When you are comfortable with that, you can work on your selector strategy. Hopefully you find at least some of the following helpful.
What makes a "good" xpath or css selector?
The selector should reliably and uniquely identify the targeted element.
For example, if an element's class occurs multiple times on the page, do not use only this class to identify the element. This is the most basic requirement for your selector
The selector should not be prone to "flakiness" -- ie, false failures that occur as a result of changes that are unrelated to the test.
Accomplish this by relying on as little of the DOM as possible to identify your element. For example, if both work to uniquely identify the element, //*[#id="app-container"]//button should be preferred over //*[#id="app-container"]/div/section/div/div/div[2]/button. Or, as you identify, "//button[text()='Cancel']/.." is the better choice.
Probably less important, but still worth considering: how easy is it to understand from the selector which element is being grabbed?
Some best practices
If you are working with a development team and thus have access to the source code of the application you are testing, implement a custom HTML attribute that is used ONLY for automation, and which has a value to uniquely identify and describe the element. In your test code you can then identify each of the elements you need with a line like this:
my_field = driver.find_element_by_css_selector('input[data-e2e-attribute="aCertainField"]')`
Organize your selection of elements into a Page Object Model, which abstracts the definition of webelements to one spot. So you can use these elements anywhere in your test without having to locate them, and it's easier to make changes to your selectors when necessary

You are correct that right-clicking and Copy Xpath is a bad way to get an Xpath. You are left with a long and brittle selector. It is much better to build your own. Once you get the hang of it, it is pretty simple to start building your own CSS and Xpath selectors. Some of them get complicated but if you keep practicing and searching for solutions you will get better and better.
The problem is it is very difficult to explain how to do it in a forum like this. Your best bet is to YouTube some videos on how to create Xpath and CSS selectors for Selenium. Here is a decent one I just found for Xpath:
https://www.youtube.com/watch?v=3uktjWgKrtI
This follows the approach I use in Chrome Dev Tools and using the built in Find window (no plugins)
Here is a good cheatsheet I have used in the past for Xpath and CSS Selectors
https://www.automatetheplanet.com/selenium-webdriver-locators-cheat-sheet/
Good luck

How to identify a change in a websites’ structure programmatically

Within the implementation of a Python Scrapy crawler I would like to add a robust mechanism for monitoring/detecting potential layout changes within a website.
These changes do not necessarily affect existing spider selectors - for example, a site adds a new HTML element to represent the number of visitors an item has received - an element I might now be interested in parsing.
Having said that, detecting selector issues (Xpath/CSS) would be also beneficial in case where they are removed/relocated.
Please note this is not about selector content change or a website refresh (if-modified-since or last-modified), but rather a modification in the structure / nodes / layout of a site.
Therefore, how would one implement logic to monitor such circumstances?

This is actually a topic for research as you can see on this paper but there are of course some implemented tools that you can check out:
https://github.com/matiskay/html-similarity
https://github.com/matiskay/html-cluster
https://github.com/TeamHG-Memex/page-compare
Basically the base for comparing (on the previous approaches) is to use the Tree Edit Distance of the html layout.

Scrapy Xpath not extraction data [duplicate]

This is a problem that I always have getting a specific XPath with my browser.
Assume that I want to extract all the images from some websites like Google Image Search or Pinterest. When I use Inspect element then use copy XPath to get the XPath for an image, it gives me some thing like following :
//*[#id="rg_s"]/div[13]/a/img
I got this from an image from Google Search. When I want to use it in my spider, I used Selector and HtmlXPathSelector with the following XPaths, but they all don't work!
//*[#id="rg_s"]/div/a/img
//div[#id="rg_s"]/div[13]/a/img
//[#class="rg_di rg_el"]/a/img #i change this based on the raw html of page
#hxs.select(xpath).extract()
#Selector(response).xpath('xpath')
.
.
I've read many questions, but I couldn't find a general answer to how I can use XPaths obtained from a web browser in Scrapy.

Usually it is not safe and reliable to blindly follow browser's suggestion about how to locate an element.
First of all, XPath expression that developer tools generate are usually absolute - starting from the the parent of all parents - html tag, which makes it being more dependant on the page structure (well, firebug can also make expressions based on id attributes).
Also, the HTML code you see in the browser can be pretty much different from what Scrapy receives due to asynchronous nature of the website page load and javascript being dynamically executed in the browser. Scrapy is not a browser and "sees" only the initial HTML code of a page, before the "dynamic" part.
Instead, inspect what Scrapy really has in the response: open up the Scrapy Shell, inspect the response and debug your XPath expressions and CSS selectors:
$ scrapy shell https://google.com
>>> response.xpath('//div[#id="myid"]')
...
Here is what I've got for the google image search:
$ scrapy shell "https://www.google.com/search?q=test&tbm=isch&qscrl=1"
In [1]: response.xpath('//*[#id="ires"]//img/#src').extract()
Out[1]:
[u'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcRO9ZkSuDqt0-CRhLrWhHAyeyt41Z5I8WhOhTkGCvjiHmRiTSvDBfHKYjx_',
u'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQpwyzbW_qsRenDw3d4wwpwwm8n99ukMtLCVaPiTJxyviyQVBQeRCglVaY',
u'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSrxtoY3-3QHwhjc5Ofx8090uDYI8VOUbi3gUrd9USxZ-Vb1D5pAbOzJLMS',
u'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTQO1A3dDJ07tIaFMHlXNOsOnpiY_srvHKJE1xOpsMZscjL3aKGxaGLOgru',
u'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQ71ukeTGCPLuClWd6MetTtQ0-0mwzo3rn1ug0MUnbpXmKnwNuuBnSWXHU',
u'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRZmWrYR9A4W97jpjhtIbyUM5Lj3vRL0vgCKG_xfylc5wKFAk6UB8jiiKA',
...
u'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRj08jK8sBjX90Tu1RO4BfZkKe5A59U0g1TpMWPFZlNnA70SQ5i5DMJkvV0']

The XPath generated from an insertion point in a browser is bound to be brittle because there are many different possible XPath expressions to reach any given node, JavaScript can modify the HTML, and the browser doesn't know your intentions.
For the example you gave,
//*[#id="rg_s"]/div[13]/a/img
the 13th div is particularly prone to breakage.
Try instead to find a uniquely identifying characteristic closer to your target. A unique #id attribute would be ideal, or a #class that uniquely identifies your target or a close ancestor of your target can work well too.
For example, for Google Image Search, something like the following XPath
//div[#id='rg_s']//img[#class='rg_i']"
will select all images of class rg_i within the div containing the search results.
If you're willing to abandon the copy-and-paste approach and learn enough XPath to generalize your selections, you'll get much better results. Of course, standard disclaimers apply about changes to presentation necessitating updating of scraping techniques too. Using a direct API call would be much more robust (and proper as well).

Convert the XPath gotten from browser to usable XPath for Scrapy

This is a problem that I always have getting a specific XPath with my browser.
Assume that I want to extract all the images from some websites like Google Image Search or Pinterest. When I use Inspect element then use copy XPath to get the XPath for an image, it gives me some thing like following :
//*[#id="rg_s"]/div[13]/a/img
I got this from an image from Google Search. When I want to use it in my spider, I used Selector and HtmlXPathSelector with the following XPaths, but they all don't work!
//*[#id="rg_s"]/div/a/img
//div[#id="rg_s"]/div[13]/a/img
//[#class="rg_di rg_el"]/a/img #i change this based on the raw html of page
#hxs.select(xpath).extract()
#Selector(response).xpath('xpath')
.
.
I've read many questions, but I couldn't find a general answer to how I can use XPaths obtained from a web browser in Scrapy.

Usually it is not safe and reliable to blindly follow browser's suggestion about how to locate an element.
First of all, XPath expression that developer tools generate are usually absolute - starting from the the parent of all parents - html tag, which makes it being more dependant on the page structure (well, firebug can also make expressions based on id attributes).
Also, the HTML code you see in the browser can be pretty much different from what Scrapy receives due to asynchronous nature of the website page load and javascript being dynamically executed in the browser. Scrapy is not a browser and "sees" only the initial HTML code of a page, before the "dynamic" part.
Instead, inspect what Scrapy really has in the response: open up the Scrapy Shell, inspect the response and debug your XPath expressions and CSS selectors:
$ scrapy shell https://google.com
>>> response.xpath('//div[#id="myid"]')
...
Here is what I've got for the google image search:
$ scrapy shell "https://www.google.com/search?q=test&tbm=isch&qscrl=1"
In [1]: response.xpath('//*[#id="ires"]//img/#src').extract()
Out[1]:
[u'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcRO9ZkSuDqt0-CRhLrWhHAyeyt41Z5I8WhOhTkGCvjiHmRiTSvDBfHKYjx_',
u'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQpwyzbW_qsRenDw3d4wwpwwm8n99ukMtLCVaPiTJxyviyQVBQeRCglVaY',
u'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSrxtoY3-3QHwhjc5Ofx8090uDYI8VOUbi3gUrd9USxZ-Vb1D5pAbOzJLMS',
u'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTQO1A3dDJ07tIaFMHlXNOsOnpiY_srvHKJE1xOpsMZscjL3aKGxaGLOgru',
u'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQ71ukeTGCPLuClWd6MetTtQ0-0mwzo3rn1ug0MUnbpXmKnwNuuBnSWXHU',
u'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRZmWrYR9A4W97jpjhtIbyUM5Lj3vRL0vgCKG_xfylc5wKFAk6UB8jiiKA',
...
u'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRj08jK8sBjX90Tu1RO4BfZkKe5A59U0g1TpMWPFZlNnA70SQ5i5DMJkvV0']

The XPath generated from an insertion point in a browser is bound to be brittle because there are many different possible XPath expressions to reach any given node, JavaScript can modify the HTML, and the browser doesn't know your intentions.
For the example you gave,
//*[#id="rg_s"]/div[13]/a/img
the 13th div is particularly prone to breakage.
Try instead to find a uniquely identifying characteristic closer to your target. A unique #id attribute would be ideal, or a #class that uniquely identifies your target or a close ancestor of your target can work well too.
For example, for Google Image Search, something like the following XPath
//div[#id='rg_s']//img[#class='rg_i']"
will select all images of class rg_i within the div containing the search results.
If you're willing to abandon the copy-and-paste approach and learn enough XPath to generalize your selections, you'll get much better results. Of course, standard disclaimers apply about changes to presentation necessitating updating of scraping techniques too. Using a direct API call would be much more robust (and proper as well).

Need to dump entire DOM tree with element id from selenium server

I have been using python selenium for web automation testing. The key part of automation is to find the right element for a user-visible object in a HTML page. The following API will work most of the time, but not all the time.
find_element_by_xxx, xxx can be id, name, xpath, tag_name etc.
When HTML page is too complicated, I would like to search the dom tree. Wonder if it's possible to ask the selenium server to serialize the entire DOM (with the element id that can be used to perform action on through webdriver server). Client side (python script) can do its own search algorithm to find the right element.
Note that python selenium can get the entire html page by
drv.page_source
However, parsing this doesn't give the internal element id from selenium server's point of view, hence not useful.
EDIT1:
Paraphrase it to make it more clear (thanks #alecxe): what's needed here is a serialized representation of all the DOM elements (with their DOM structure preserved) in the selenium server, this serialized representation can be sent to the client side (a python selenium test app) which can do its own search.

The Problem
Ok, so there may be cases where you need to perform some substantial processing of a page on the client (Python) side rather than on the server (browser) side. For instance, if you have some sort of machine learning system already written in Python and it needs to analyze the whole page before performing actions on them, then although it is possible to do it with a bunch of find_element calls, this gets very expensive because each call is a round-trip between the client and the server. And rewriting it to work in the browser may be too expensive.
Why Selenium's Identifiers wont' do it
However, I do not see an efficient way to get a serialization of the DOM together with Selenium's own identifiers. Selenium creates these identifiers on an as-needed basis, when you call find_element or when DOM nodes are returned from an execute_script call (or passed to the callback that execute_async_script gives to the script). But if you call find_element to get identifiers for each element, then you are back to square one. I could imagine decorating the DOM in the browser with the required information but there is no public API to request some sort of pre-assignment of WebElement ids. As a matter of fact, these identifiers are designed to be opaque so even if a solution managed somehow to get the required information, I'd be concerned about cross-browser viability and ongoing support.
A Solution
There is however a way to get an addressing system that would work on both sides: XPath. The idea is to parse the DOM serialization into a tree on the client side and then get the XPath of the nodes you are interested in and use this to get the corresponding WebElement. So if you'd have to perform dozens of client-server roundtrips to determine which single element you need to perform a click on, you'd be able so reduce this to an initial query of the page source plus a single find_element call with the XPath you need.
Here is a super simple proof of concept. It fetches the main input field of the Google front page.
from StringIO import StringIO
from selenium import webdriver
import lxml.etree
#
# Make sure that your chromedriver is in your PATH, and use the following line...
#
driver = webdriver.Chrome()
#
# ... or, you can put the path inside the call like this:
# driver = webdriver.Chrome("/path/to/chromedriver")
#
parser = lxml.etree.HTMLParser()
driver.get("http://google.com")
# We get this element only for the sake of illustration, for the tests later.
input_from_find = driver.find_element_by_id("gbqfq")
input_from_find.send_keys("foo")
html = driver.execute_script("return document.documentElement.outerHTML")
tree = lxml.etree.parse(StringIO(html), parser)
# Find our element in the tree.
field = tree.find("//*[#id='gbqfq']")
# Get the XPath that will uniquely select it.
path = tree.getpath(field)
# Use the XPath to get the element from the browser.
input_from_xpath = driver.find_element_by_xpath(path)
print "Equal?", input_from_xpath == input_from_find
# In JavaScript we would not call ``getAttribute`` but Selenium treats
# a query on the ``value`` attribute as special, so this works.
print "Value:", input_from_xpath.get_attribute("value")
driver.quit()
Notes:
The code above does not use driver.page_source because Selenium's documentation states that there is no guarantee as to the freshness of what it returns. It could be the state of the current DOM or the state of the DOM when the page was first loaded.
This solution suffers from the exact same problems that find_element suffers from regarding dynamic contents. If the DOM changes while the analysis is occurring, then you are working on a stale representation of the DOM.
If you have to generate JavaScript events while performing the analysis, and these events change the DOM, then you'd need fetch the DOM again. (This is similar to the previous point but a solution that uses find_element calls could conceivably avoid the problem I'm talking about in this point by ordering the sequence of calls carefully.)
lxml's tree could possibly differ structurally from the DOM tree in such a way that the XPath obtained from lxml does not address the corresponding element in the DOM. What lxml processes is the cleaned up serialized view that the browser has of the HTML passed to it. Therefore, so long as the code is written to prevent the problems I've mentioned in point 2 and 3, I do not see this as a likely scenario, but it is not impossible.

Try:
find_elements_by_xpath("//*")
That should match all elements in the document.
UPDATE (to match question refinements):
Use javascript and return the DOM as a string:
execute_script("return document.documentElement.outerHTML")

See my other answer for the issues regarding any attempts at getting Selenium's identifiers.
Again, the problem is to reduce a bunch of find_element calls so as to avoid the round-trips associated with them.
A different method from my other answer is to use execute_script to perform the search on the browser and then return all the elements needed. For instance, this code would require three round-trips but can be reduced to just one round-trip:
el, parent, text = driver.execute_script("""
var el = document.querySelector(arguments[0]);
return [el, el.parentNode, el.textContent];
""", selector)
This returns an element, the element's parent and the element's textual contents on the basis of whatever CSS selector I wish to pass. In a case where the page has jQuery loaded, I could use jQuery to perform the search. And the logic can get as complicated as needed.
This method takes care of the vast majority of cases where reducing round-trips is desirable but it does not take care of a scenario like the one I've given in illustration in my other answer.

You can try to utilize the page object pattern. That sounds closer to what you are looking for in this case. You might not change everything to that, but at least for this part you might want to consider that.
http://selenium-python.readthedocs.org/en/latest/test-design.html?highlight=page%20object
You can also loop through all the elements of the page and save them off one at a time, but there should be some library that can do that. I know for .Net there is htmlAgility. I'm not sure on python.
Update
I found this...perhaps it will help you.
Html Agility Pack for python

Two ways I know are :-
get_source = driver.page_source
Secondly using javascript :-
pageSource = driver.execute_script("return document.documentElement.outerHTML;")

Actually you can do this quite easily. Write output to a stream like var w = window.open... and then document.write...
recursively iterate through the document object returning JSON.Stringify returning each object. I suggest you throw in typeof as well.
var s =
recurse(obj) {
for(var i in obj) {
return typeof(i) + ":" + i.toString() + ":" + JSON.stringify(obj[i]);
}
}
I'd suggest adding some sort of filtering to remove properties that you don't want to see. Also I doubt would run as the browsers detect and escape out of recursive loops.
I found this question looking for something similar, but I was hoping for a DataTable object (I'm using .Net) that I could bind into some sort of debugging window, something better than chrome. Before I used firebug to do this, but that is sorta dead.
So you could also get this data but in real time using a debugger.