Develop Reference

Python/Selenium: Any way to wildcard the end of an xpath? Or search for a specifically formatted piece of an xpath?

I am using python / selenium to archive some posts. They are simple text + images. As the site requires a login, I'm using selenium to access it.
The problem is, the page shows all the posts, and they are only fully readable on clicking a text labeled "read more", which brings up a popup with the full text / images.
So I'm writing a script to scroll the page, click read more, scrape the post, close it, and move on to the next one.
The problem I'm running into, is that each read more button is an identical element:
read more
If I try to loop through them using XPaths, I run into the problem of them being formatted differently as well, for example:
//*[#id="page"]/div[2]/article[10]/div[2]/ul/li/a
//*[#id="page"]/div[2]/article[14]/div[2]/p[3]/a
I tried formatting my loop to just loop through the article numbers, but of course the xpath's terminate differently. Is there a way I can add a wildcard to the back half of my xpaths? Or search just by the article numbers?

/ is used to go for direct child, use // instead to go from <article> to the <a>
//*[#id="page"]/div[2]/article//a[.="read more"]
This will give you a list of elements you can iterate. You might be able to remove the [.="read more"], but it might catch unrelated <a> tags, depends on the rest of the html structure.
You can also try looking for the read more elements directly by text
//a[.="read more"]

I recommend using CSS Selectors over XPaths. CSS Selector provide faster, cleaner and simpler way to deal with these queries.
('a[href^="javascript"]')
This will selects every element whose href attribute value begins with "javascript" which is what you are looking for...
You can learn more about Locating Elements by CSS Selectors in selenium here.
readMore = driver.find_element(By.CSS_SELECTOR, 'a[href^="javascript"]')
And about Locating Hyperlinks by Link Text
readMore_link = driver.find_elements(By.LINK_TEXT, 'javascript')

How can I search for text in a specific part of a webpage in Selenium (Python) ? With pictures:

So I've been trying to search for specific keywords on a webpage using selenium in python, but can't seem to figure out how to search for specific text in a specific area. As shown in the picture, when I search the word "Sim" in chrome, several spots are highlighted. The red region is the only place I am looking for code in. I've using the xpaths to identify the text, as nothing else is available for them.
This is the code so far:
else:
print("Nothing here yet 1")
if driver.find_elements_by_xpath("//*[contains(text(), 'Sim')]"):
login_to_reply = driver.find_element_by_xpath("//body/div[#id='app']/main/div[#id='content']/div/div/div/div/div/div/article/header[1]")
login_to_reply.click()
time.sleep(5)
if anyone could help and let me know what I'm not understanding, I would really appreciate that, thank you.

Based on the additional information in the comments, to search for keywords in the FIRST POST on the page you can use the following xpath:
(//article)[1]//div[#class='Post-body'][contains(normalize-space(),'point')]
The key bit is: (//article)[1] - it's locking the further identifiers to within the first [1] located article tag. The rest just reduces repetition within the dom and finds your text whoever the nested tags are strucutred.
Based on the link provided, that matches the top post only. You can see searching for keyword "point" only gets 1 hit even though it's in multiple posts...
The text 'point' can be swapped out for 'sim' or whatever you want to filter.
Looking at you code, i have some more suggestions:
1/
Your second xpath for "login to reply" is not great. Long copied xpaths like that are typically flakey and troublesome. You can use the same technique as above to click the reply within that article box:
(//article)[1]//button[span[text()='Reply']]
2/
You also need to be aware that this line won't work as you expect.
if driver.find_elements_by_xpath("//*[contains(text(), 'Sim')]"):
If the element is not found, it does not return false - it returns a NoSuchElement exception and fails (and stops) the script.
You need this to be a try/except block
It would need to look like this:
try:
driver.find_elements_by_xpath("(//article)[1]//div[#class='Post-body'][contains(normalize-space(),'point')]"):
login_to_reply = driver.find_element_by_xpath("(//article)[1]//button[span[text()='Reply']]")
login_to_reply.click()
time.sleep(5)
except:
print("Text was not found")
I've not run this but if it doesn't work let me know and i'll look again.

How to find an element's position using XPath?

I've been trying to write a simple script in order to upload 200+ links to a website I'm working in (I have poor knowledge on python and even poorer in HTML, of course I wasn't working as a web developer, I just need to upload these links).
Well, the situation I'm in is the following: I am using Splinter(therefore, Python) in order to navigate in the website. Certain section titles of this website will be compared with values I have in a .csv table.
For instance, in this screenshot, I am looking for this link /admin/pages/5, and I would like to compare the link's title (Explorar subpáginas de 'MA111 - Cálculo I') with my .CSV table. The problem is the link's title doesn't appear in the website.
To find the link I would guess that I should use find_by_xpath(), but I don't know how to do it. I would guess it's something like this link.
I would appreciate any help! I hope I have made myself clear.

You first need to define how are you detecting that url, so for example, "it is always to the right of certain button", or "it is the second row in a table", that way you can build the respective xpath (which is a path to follow inside the DOM.
I am not entirely sure, but this could give you the solution
url = browser.find_by_xpath('//td[#class="children"]/a')[0]['href']
if you are finding a tag by the link name for example, try this:
url = browser.find_by_xpath('//a[contains(#title, "MA111 - Cálculo I")]')[0]['href']
If you check there, the xpath says "find in the entire DOM // a tag named a which contains "MA111 - Cálculo I" in the title attribute.

Python with Selenium - Cannot find and click this specific element due to randomization of it's location on the site

I've been creating a tool that plays through an online game using python 2.7 and selenium, and am very stuck on one particular element I need to select.
The UI looks as follows:
1 2 3
a d g
b e h
c f i
The numbers one two and three represent a drop down menu, which when clicked open up the letters. Each option represents a different outcome. The problem is, at the start of each game the positions of both the numbers and letters are randomized. In the code, each button's css selector is labeled as for example "#action-1 > button:nth-child(1)", as in the "first button", but the "first button" will be different every game.
I've tried finding by link text, xpath, and css selector to no success.
If it helps, pressing "copy outer html" gives this:
<button class="ng-binding" ng-click="subBtn($event)" ng-class="{disabled : !state.chapterStart || state.btns.indexOf(btn.action) != -1}" ng-disabled="!state.chapterStart || state.btns.indexOf(btn.action) != -1">Wait and See What They Do</button>
The "Wait and See What They Do" part of it is what the button says and is how you know what to click, but I can't seem to find the element by that.
Copying the xpath results in //*[#id="action-1"]/button, which again isn't really helpful because telling it to click that would just be like saying "click the button that is in the first position", and it changes every time. I've added long time.sleep() commands to test, the page is definitely loading all the way so that is not the issue.
Anyways, I've been trying to figure this one out for way too long and it has me stumped. I'd be very grateful for any input you all can give. Thank you!

If you are looking to locate an element by the text it contains you can use XPath. You stated that you tried XPath but no specifics were given. Did you try the below? It should work given the HTML you provided.
//button[text()='Wait and See What They Do']
To read this XPath... find any descendant // that is a BUTTON tag that has an attribute [] where the text contained in the element is equal to the search string, text()=''.
Another example might be the below which is an alternate way to find your button. The problem with this way is that there may be many buttons on the page with that class so it may not be specific enough.
//button[#class='ng-binding']
I've found (and others have also) that XPaths are generally slower than other location methods. Because of this, I generally prefer By.Id, when available, then By.CssSelector. They are both significantly faster than XPath. I save XPath for things like finding text in an element or finding relative elements (CSS can do this some but is no where near as powerful as XPath).
XPath Examples

How can I get the text from a dialogue using selenium-python?

I want to crawl the dialogue text in a popup window. The problem is that after I triggered the link the window appears but it seems that the selenium driver cannot handle it automatically as I learned from other questions on this site by entering driver.window_handles.
The source of the trigger:
The value of len(driver.window_handles) is 1. I thought I can get the window element and then get the text via the get_attributes, fortunately I succeeded getting the element by
wd = driver.find_element_by_css_selector('div[node-type="repeat_list"]')
selenium.webdriver.remote.webelement.WebElement (session="f810cbbe-db43-4e8d-b484-664559ec8efc", element="{dd00e689-7991-44e9-85d3-76c69e79218f}")
But the sad thing is I don't know how to get all the stuff out from it since I don't know their attributes.
I'm not certain if it's a dialogue, a front end engineer told me that it looks like an animation. Anyway this is the source snippet:
PS: the browser is Firefox.
I thought it may violate the site's Acceptable Use Policy to crawl then I should hide some information. Sorry.

Once you have your parent element :
wd = driver.find_element_by_css_selector('div[node-type="repeat_list"]')
you can continue calling methods on this object, and in this order reach the children elements, you can use find element_by_xpath, or find element_by_class name, for example:
wd = driver.find_element_by_css_selector('div[node-type="repeat_list"]')
wd.find_element_by_class_name("list_box").find_element_by_class_name("list_ul").find_elements_by_class_name("list_li S_line1 clearfix")
and so on until you reach the desired element down the hierarchy and extract it's content as you wish.
I hope this helps!