Develop Reference

How do you find a specific descendent from an element in selenium?

I'm automating the process of filling some forms online. The problem is that there are many individual elements whose children have basically the same ID of the stuff I want to find and fill. So my idea was to first find the parent I needed using Selenium and then go from there.
for range in cards:
cardID = driver.find_element(By.XPATH, "//a[contains(text(),'{cardis}/')]/ancestor::tr".format(cardis=allCards_NUMBER_List[range]))
cardREG_PRICE = cardID.find_element(By.XPATH, "input[contains(id(), 'txt_preco_')]")
But when I run this it only said that it can't find cardREG_PRICE. The ID name is correct, and from what I've read the XPATH structure should work. How can I fix this?

Your xpath is incorrect.That's why it is failing.
Instead of this
cardREG_PRICE = cardID.find_element(By.XPATH, "input[contains(id(), 'txt_preco_')]")
if should be like.
cardREG_PRICE = cardID.find_element(By.XPATH, ".//input[contains(#id, 'txt_preco_')]")
first thing id is an attribute and should pass with #, second thing // denote the node, Third thing . means intermediate child of the parent.

Looking for 2 things in 2 different Xpaths - Python Selenium

I am working on a bot for a website and it requires a color and keyword to find the item. I am using selenium to look for the item from the keyword and then pick a color option (some items on the website, provide the item in multiple colors). I am having trouble looking for both the keyword and color at the same time, and then after choosing the correct colored version of the item from the user's color and keyword input. I want it to select on that option.
Formula I am trying to make in Python:
If the first Xpath(keyword) is found and the 2nd Xpath(color) is found
Then select on the item that contains those 2 properties.
This is the current code I have:
Item = driver.find_element_by_xpath('//*[contains(text(), "MLK")]' and contains ("Black")]')
if (item != None):
actions.moveToElement(item).click()
I've tried the code above and it doesn't work.
Here are the 2 pieces of code that I want to merge to find the item:
driver.find_element_by_xpath('//a[contains(text(), "MLK")]')
driver.find_element_by_xpath('//a[contains(text(), "Black")]')
The keyword is called MLK
The Color is called Black
After Merging, I want to find that Exact Element (Called MLK, Color version = Black)
This combined item should be clicked on, I only know to use .click()
If a better way, please let me know.
The website I am using to make a bot for: supremenewyork.com
The item I am using as an example, to pick a certain color (It's the Sweatshirt with MLK on it): http://www.supremenewyork.com/shop/all/sweatshirts

It took me a second to realize that there are 3 A tags for each shirt... one for the image, one for the name of the shirt, and one for the color. Since the last two A tags are the ones you are wanting to text search, you can't look for both strings in the same A tag. I've tested the XPath below and it works.
//article[.//a[contains(.,'MLK')]][.//a[.='Black']]//a
ARTICLE is the container for the shirt. This XPath is looking for an ARTICLE tag that contains an A tag that contains 'MLK' and then another A tag that contains 'Black' then finds the A tags that are descendants of the ARTICLE tag. You can click on any of them, they are all the same link.
BTW, your code has a problem. The first line below will throw an exception if there is no match so the next line will never be reached to test for None.
Item = driver.find_element_by_xpath('//*[contains(text(), "MLK")]' and contains ("Black")]')
if (Item != None):
actions.moveToElement(item).click()
A better practice is to use .find_elements() (plural) and check for an empty list. If the list is empty, that means there was no element that matched the locator.
Putting the pieces together:
items = driver.find_elements_by_xpath("//article[.//a[contains(.,'MLK')]][.//a[.='Black']]//a")
if items:
items[0].click()
I'm assuming you will be calling this code repeatedly so I would suggest that you put this in a function and pass the two strings to be searched for. I'll let you take it from here...

Try union "|" operator to combine two xpath.
Example:-
//p[#id='para1'] | //p[#id='para2']
('//a[contains(text(), "MLK")]' | '//a[contains(text(), "Black")]')

You can use a full XPath to select the item you want based on two conditions, you just need to start from a parent node and then apply the conditions on the child nodes:
//div[contains(./h1/a/text(), "MLK") and contains(./p/a/text(), "Black")]/a/#href
You first need to select the element itself, after that you need to get the attribute #href from the element, something like this:
Item = driver.find_element_by_xpath('//div[contains(./h1/a/text(), "MLK") and contains(./p/a/text(), "Black")]/a')
href = Item .get_attribute("href")

How to locate an element by class name and its text in python selenium

Hi I am trying to locate an element by its class name and the text that it contains
<div class="fc-day-number">15</div>
there are a bunch of fc-day-number on the page with different values, I need the one with for example 15.
I do
driver.find_element_by_class_name("fc-day-content")
but I also need it to be equal to 15 and I am stuck here, please help.

You can use xpath:
driver.find_element_by_xpath("//div[#class='fc-day-content' and text()='15']")

fc_day_contents = driver.find_elements_by_class_name("fc-day-content")
the_one_you_want = [x for x in fc_day_contents if "15" == x.text][0]
first line puts all elements with class name "fc-day-content" in a list ( also notice how its elementSSSSSSSSS with an S, this returns a list of all elements by_class_name, by_name, by_id or wahtever)
second line, goes through each element and looks to see if it has the text "15" as its text, and returns it as a (probably smaller) list
the [0] at the end of it, returns the first item in the list (you can remove it, if you want a list of all the ones that are "15" )

For things like this, prefer to use JavaScript:
els = driver.execute_script("""
return Array.prototype.slice.call(document.getElementsByClassName("fc-day-content"))
.filter(function (x) { return x.textContent === "15"; });
""")
assert len(els) == 1
el = els[0]
What this does is get all elements that have the class fc-day-content as the class. (By the way, your question uses fc-day-content and fc-day-number. Unclear which one you're really looking for but it does not matter in the grand scheme of things.) The call to Array.prototype.slice creates an array from the return value of getElementsByClassName because this method returns an HTMLCollection and thus filter is not available on it. Once we have the array, run a filter to narrow it to the elements that have 15 for text. This array of elements is returned by the JavaScript code. The assert is to make sure we don't unwittingly get more than one element, which would be a surprise. And then the element is extracted from the list.
(If you care about IE compatibility, textContent is not available before IE 9. If you need support for IE 8 or earlier, you might be able to get by with innerText or innerHTML or you could check that the element holds a single text node with value 15.)
I prefer not to do it like TehTris does (find_elements_by_class_name plus a Python loop to find the one with the text) because that method takes 1 round-trip between Selenium client and Selenium server to get all the elements of class fc-day-content plus 1 round-trip per element that was found. So if you have 15 elements on your page with the class fc-day-content, that's 16 round-trips. If you run a test through Browser Stack or Sauce Labs, that's going to slow things down considerably.
And I prefer to avoid an XPath expression with #class='fc-day-content' because as soon as you add a new class to your element, this expression breaks. Maybe the element you care about has just one CSS class now but applications change. You could use XPath's contains() function but then you run into other complications, and once you take care of everything, it becomes a bit unwieldy. See this answer for how to use it robustly.

[Python]Get a XPath Value from Steam and print it

I want to get an XPATH-Value from a Steamstoresite, e.g. http://store.steampowered.com/app/234160/. On the right side are 2 boxes. The first one contains Title, Genre, Developer ... I just need the Genre here. There is a different count on every game. Some have 4 Genres, some just one. And then there is another block, where the gamefeatures are listet (like Singleplayer, Multiplayer, Coop, Gamepad, ...)
I need all those values.
Also sometimes there is an image between (PEGI/USK)
http://store.steampowered.com/app/233290.
import requests
from lxml import html
page = requests.get('http://store.steampowered.com/app/234160/')
tree = html.fromstring(page.text)
blockone = tree.xpath(".//*[#id='main_content']/div[4]/div[3]/div[2]/div/div[1]")
blocktwo = tree.xpath(".//*[#id='main_content']/div[4]/div[3]/div[2]/div/div[2]")
print "Detailblock:" , blockone
print "Featureblock:" , blocktwo
This is the code I have so far. When I try it it just prints:
Detailblock: [<Element div at 0x2ce5868>]
Featureblock: [<Element div at 0x2ce58b8>]
How do I make this work?

xpath returns a list of matching elements. You're just printing out that list.
If you want the first element, you need blockone[0]. If you want all elements, you have to loop over them (e.g., with a comprehension).
And meanwhile, what do you want to print for each element? The direct inner text? The HTML for the whole subtree rooted at that element? Something else? Whatever you want, you need to use the appropriate method on the Element type to get it; lxml can't read your mind and figure out what you want, and neither can we.
It sounds like what you really want is just some elements deeper in the tree. You could xpath your way there. (Instead of going through all of the elements one by one and relying on index as you did, I'm just going to write the simplest way to get to what I think you're asking for.)
genres = [a.text for a in blockone[0].xpath('.//a')]
Or, really, why even get that blockone in the first place? Why not just xpath directly to the elements you wanted in the first place?
gtags = tree.xpath(".//*[#id='main_content']/div[4]/div[3]/div[2]/div/div[1]//a")
genres = [a.text for a in gtags]
Also, you could make this a lot simpler—and a lot more robust—if you used the information in the tags instead of finding them by explicitly walking the structure:
gtags = tree.xpath(".//div[#class='glance_tags popular_tags']//a")
Or, since there don't seem to be any other app_tag items anywhere, just:
gtags = tree.xpath(".//a[#class='app_tag']")

Chosing next relative in Python BeautifulSoup with automation

First of all - I'm creating xml document with python BeautifulSoup.
Currently, what I'm trying to create, is very similar to this example.
<options>
<opt name='string'>ContentString</opt>
<opt name='string'>ContentString</opt>
<opt name='string'>ContentString</opt>
</options>
Notice, that there should be only one tag, called name.
As options can be much more in count, and different as well, I decided to create little python function, which could help me create such result.
array = ['FirstName','SecondName','ThirdName']
# This list will be guideline for function to let it know, how much options will be in result, and how option tags will be called.
def create_options(array):
soup.append(soup.new_tag('options'))
if len(array) > 0: # It's small error handling, so you could see, if given array isn't empty by any reason. Optional.
for i in range(len(array)):
soup.options.append(soup.new_tag('opt'))
# With beatifullsoup methods, we create opt tags inside options tag. Exact amount as in parsed array.
counter = 0
# There's option to use python range() method, but for testing purposes, current approach is sufficient enough.
for tag in soup.options.find_all():
soup.options.find('opt')['name'] = str(array[counter])
# Notice, that in this part tag name is assigned only to first opt element. We'll discuss this next.
counter += 1
print len(array), ' options were created.'
else:
print 'No options were created.'
You notice, that in function, tag assignment is handled by for loop, which, unfortunately, assigns all different tag names to first option in options element.
BeautifulSoup has .next_sibling and .previous_sibling, which can help me in this task.
As they describe by name, with them I can access next or previous sibling in element. So, by this example:
soup.options.find('opt').next_sibling['name'] = str(array[counter])
We can access second child of options element. So, if we add .next_sibling to each soup.items.find('opt'), we could then move from first element to next.
Problem is, that by finding option element in options with:
soup.options.find('opt')
each time we access first option. But my function is willing to access with each item in list, next option as well. So it means, as more items are in list, more .next_sibling methods it must add to first option.
In result, with logic I constructed, with 4th or further item in list, accessing relevant option for assigning it's appropriate tag, should look like this:
soup.options.find('opt').next_sibling.next_sibling.next_sibling.next_sibling['name'] = str(array[counter])
And now we are ready to my questions:
1st. As I didn't found any other kind of method, how to do it with Python BeautifulSoup methods, I'm not sure, that my approach still is only way. Is there any other method?
2st. How could I achieve result by this approach, if as my experiments show me, that I can't put variable inside method row? (So I could multiply methods)
#Like this
thirdoption = .next_sibling.next_sibling.next_sibling
#As well, it's not quite possible, but it's just example.
soup.options.find('opt').next_sibling.next_sibling.next_sibling['name'] = str(array[counter])
3st. May be I read BeautifulSoup documentation badly, and just didn't found method, which could help me in this task?

I managed to achieve result, ignoring BeatifulSoup methods.
Python has element tree methods, which were sufficient enough to work with.
So, let me show the example code, and explain it, what it does. Comments provide explanation more precisely.
"""
Before this code, there goes soup xml document generation. Except part, I mentioned in topic, we just create empty options tags in document, thus, creating almost done document.
Right after that, with this little script, we will use basic python provided element tree methods.
"""
import xml.etree.ElementTree as ET
ET_tree = ET.parse("exported_file.xml")
# Here we import exactly the same file, we opened with soup. Exporting can be done in different file, if you wish.
ET_root = ET_tree.getroot()
for position, opt in enumerate(item.find('options')):
# Position is pretty important, as this will remove 'counter' thing in for loop, I was using in code in first example. Position will be used for getting out exact items from array, which works like template for our option tag names.
opt.set('name', str(array[position]))
opt.text = 'text'
# Same way, with position, we can get data from relevant array, provided, that they are inherited or connected in same way.
tree = ET.ElementTree(ET_root).write('exported_file.xml',encoding="UTF-8",xml_declaration=True)
# This part was something, I researched before quite lot. This code will help save xml document with utf-8 encoding, which is very important.
This approach is pretty inefficient, as for achieving same result, I could use ET for everything.
Thought, BeatifulSoup prepares document in nice output, which in any way is very neat, as element-tree creates files for software friendly only look.