I am running a series of selenium tests with python. I have a navigation on the page I'm testing that has this structure:
<ul>
<li class="has-sub">
<a href="#">
<span> First nav </span>
</a>
<ul style="display:block">
<li>
<a href="#">
<span> First subnav </span>
</a>
</li>
<li>...</li>
<li>...</li>
<li>...</li>
</ul>
</li>
<li>...</li>
</ul>
Now I am clicking on the first subnav, that is the first span, but clicking on First nav to open up that list then first subnav. I implement a webdriverwait, to wait for the element to be visible and click on it via it's xpath,
//span[1]
I often got timeout exceptions waiting for the subnav span to be visible after clicking on the nav, which made me think something was wrong with clicking on the first nav to open up the list. So I changed the xpath of the first nav (//span[1]) to
//li[#class='has-sub']/descendant::span[text()='First subnav']
and I never get timeout exceptions when waiting for subnav span to be visible now. So seems like it's always clicking on the nav span every time to open it up and give me no timeout when trying to get to the subnav. Anyone have any idea why that is?
Here is my python code as well:
inside LoadCommPage class:
def click_element(self, by, locator):
try:
WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((by, locator)))
print "pressing element " + str(locator)
self.driver.find_element(by, locator).click()
except TimeoutException:
print "no clickable element in 10 sec"
print self.traceback.format_exc()
self.driver.close()
inside main test (load_comm_page is an instance of LoadCommPage, where click_clement is defined):
load_comm_page.click_element(*LoadCommPageLocators.sys_ops_tab)
And another class for the locators:
class LoadCommPageLocators(object):
firstnav_tab = (By.XPATH, "//li[#class='has-sub']/descendant::span[text()='First nav']")
Xpath indexes begin at one, not 0 so the Xpath
//span[1]
is looking for the first span element in the html. Whereas
//span[2]
will look for the second span.
Related
So I have been trying to use selenium to click an <li> button on an HTML which is in a <ul> tag and looks something like this:
...
<ul id="ulVisualization">
<li class="active" id="liMap">Map</li>
<li id="liBar" class="">Bar</li>
<li id="liLine">Line</li>
</ul>
...
I have been using the following command to get to the element using XPATH:
WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.XPATH,"//li[#id='liBar']"))).click()
Yet for some reason that I have not been able to identify, this command is not able to find/click that button and the command hits its time-out.
I have even tried:
driver.find_element_by_xpath('//li[#id="liBar"]').click()
But that too was of no avail, throwing an error message saying NoSuchElementException: Message:
I would appreciate any and all help and thank you very much in advance.
EDIT:
Additionally, I have noticed that when I click on the button manually the HTML modifies to:
...
<ul id="ulVisualization">
<li class="" id="liMap">Map</li>
<li id="liBar" class="Active">Bar</li>
<li id="liLine">Line</li>
</ul>
...
Revealing the data I am trying to acquire later in the HTML code, which was not available before I clicked the button manually.
Since you want to click on the Element, try element_to_be_clickable, instead of presence_of_element_located
WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,"//li[#id='liBar']"))).click()
Make sure the locator you are using is Unique, that is 1/1. Link to Refer
And also check if the Elements are in an iframe or in an shadow-root or Trying to find Element in a newly opened browser tab.
If none of them work, apply some time.sleep() and check if it works.
I use selenium to access all divs which contain roster information:
# this returns a <selenium.webdriver.firefox.webelement.FirefoxWebElement
divs = driver.find_elements_by_class_name('appointment-template')
These divs inside this element should look like this:
div class="appointment-template" id="appointment-3500508">
<p class="title">Mentoruur</p>
<p class="time">11:15<span class="time-end"> - 12:15</span></p>
<ul class="facility">
<li onclick=""
title="HIC_Online">ONLINE
</li>
</ul>
<ul class="docent">
<li onclick=""
title="HANSJE">HANSJE
</li>
</ul>
<ul class="group">
<li onclick=""
title="ASD123">ASD123
</li>
</ul>
The next thing I want to do is access values like the docent name and time values that lie within this div:
for div in divs:
print(div.find_element_by_class_name('title'))
print(div.find_element_by_class_name('time'))
This does not seem to work:
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .title
How can I use selenium to get the values like:
Mentoruur
11:15 - 12:15
Hansje
to get the Mentoruur, one should try the below css :
div.appointment-template p.title
use it like this :
title = driver.find_element(By.CSS_SELECTOR, "div.appointment-template p.title").text
print(title)
to get time :
div.appointment-template p.time
code :
time = driver.find_element(By.CSS_SELECTOR, "div.appointment-template p.time").text
print(time)
same way you can go ahead with others.
In order to locate element inside element it's better to use this technique:
for div in divs:
print(div.find_element_by_xpath('.//p[#class="title"]'))
print(div.find_element_by_xpath('.//p[#class="time"]'))
the dot . in front of xpath expression means "from here". This is what we need when searching inside specific parent element
EDIT:
So I found a way to do it by clicking on the Countries elements, see my answer.
Still have one question that would make this better:
When I execute the scrollIntoView(true) on a country <li> it goes under another element (<div class="sportList_subtitle">Desportos</div>) and is not clickable.
Is there some javascript or selenium function like "scrollIntoClickable"?
ORIGINAL:
I'm trying to scrape info from Betclic website with python and BeautifulSoup + Selenium.
Given the URL for each game has the structure: "domain"/"sports_url"/"competition_url"/"match_url"
Example: https://www.betclic.pt/futebol-s1/liga-dos-campeoes-c8/rennes-chelsea-m2695669
You can try it in your language, they translate the actual URL string but the structure and ID's are the same.
The only thing that's left is grabbing all the different "competition_url"
So my question now is from the "sports_url" (https://www.betclic.pt/futebol-s1) how can I get all sub "competition_url"?
The problem is with the "hidden" URL's under each country's name on the left panel. Those only appear after you click the arrow next to each country's name, like a drop-down list. The click event actually adds one class "is-active" to the <li> for that country and also
an <ul> at the end of that <li>. It's this added <ul> that has the URL's list I'm trying to get.
Code before click:
<!---->
<li class="sportList_item has-children ng-star-inserted" routerlinkactive="active-link" id="rziat-DE">
<div class="sportList_itemWrapper prebootFreeze">
<div class="sportlist_icon flagsIconBg is-DE"></div>
<div class="sportlist_name">Alemanha</div>
</div>
<!---->
</li>
Code after click (reduced for presentation):
<li class="sportList_item has-children ng-star-inserted is-active" routerlinkactive="active-link" id="rziat-DE">
<div class="sportList_itemWrapper prebootFreeze">
<div class="sportlist_icon flagsIconBg is-DE"></div>
<div class="sportlist_name">Alemanha</div>
</div>
<!---->
<ul class="sportList_listLv2 ng-star-inserted">
<!---->
<li class="sportList_item ng-star-inserted" routerlinkactive="active-link">
<a class="sportList_itemWrapper prebootFreeze" id="competition-link-5" href="/futebol-s1/alemanha-bundesliga-c5">
<div class="sportlist_icon"></div>
<div class="sportlist_name">Alemanha - Bundesliga</div>
</a>
</li>(...)
</li>(...)
</li>(...)
</li>
</ul>
</li>
In this example is that "/futebol-s1/alemanha-bundesliga-c5" that I'm looking for.
Is there a way to get all those URL's? Or the "hiden" <ul> for that matter?
Maybe a way to simulate the click and parse the HTML code again?
Thanks in advance!
So I found a way to do it by clicking on the Countries elements.
Still have one question that would make this better:
When I execute the scrollIntoView(true) on a country <li> it goes under another element (<div class="sportList_subtitle">Desportos</div>) and is not clickable.
Is there some javascript or selenium function like "scrollIntoClickable"?
How I'm doing it now:
driver = webdriver.Chrome(ChromeDriverManager().install())
url = "https://www.betclic.pt/"
driver.get(url)
link_set = set()
all_sports = driver.find_element_by_css_selector(
("body > app-desktop > div.layout > div > app-left-menu > div >"
" app-sports-nav-bar > div > div:nth-child(2) > ul")
).find_elements_by_tag_name("li")
try:
cookies = driver.find_element_by_css_selector("body > app-desktop > bc-gb-cookie-banner > div > div > button")
cookies.click()
except:
print("Cookie error or not found...")
for sport in all_sports:
sport.click()
has_container = driver.find_element_by_tag_name("app-block-ext").size.get('height')>0
if not has_container:
for competition in driver.find_elements_by_css_selector("a[id*='block-link-']"):
link_set.add(competition.get_attribute("href"))
driver.execute_script("arguments[0].scrollIntoView(true);", competition)
else:
driver.execute_script("arguments[0].scrollIntoView(true);", driver.find_element_by_tag_name("app-block-ext"))
all_countries = driver.find_elements_by_css_selector("li[id^='rziat']")
for country in all_countries:
country.click()
competitions = driver.find_elements_by_css_selector("a[id^='competition-link']")
for element in competitions:
link_set.add(element.get_attribute("href"))
driver.execute_script("arguments[0].scrollIntoView(true);", country)
for link in sorted(link_set):
print(link)
I want to click on "Page 1" for show list of sub menu
python code I used
driver.find_element_by_xpath("nav/div/ul/li[2]/a").click()
website code:
<ul class="nav navbar-nav">
<li class="active">Home</li>
<li class="dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#">Page 1
<span class="caret"></span></a>
<ul class="dropdown-menu">
<li>Page 1-1</li>
<li>Page 1-2</li>
<li>Page 1-3</li>
</ul>
</li>
<li>Page 2</li>
<li>Page 3</li>
</ul>
Image of nav
As your requirement is "I want to click on "Page 1" for show list of sub menu".
Instead of using absolute xpath , you can use linkText.
parent_page1 = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.LINK_TEXT, "Page 1")))
parent_page1.click()
if this linkText does not work out for you, you can use a relative xpath :
parent_page1 = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='dropdown-toggle' and contains(text(),'Page 1')]")))
parent_page1.click()
Note that this code will click on Page 1 , if and only if it is clickable.
If it is a hover action then you can go ahead with action chain.
Note : This is for parent Page1 . Not for child. and based on the xpath you have tried nav/div/ul/li[2]/a
Your x-path is not correct. any x-path starts with / or //. This may be the issue.
try with the x-path //nav/div/ul/li[2]/a
in code,
driver.find_element_by_xpath("//nav/div/ul/li[2]/a").click()
I guess, you are using w3school bootstrap navigation bar. If yes, the page is inside frame, so first you need to switch to the frame as given below.
driver.switch_to_frame("iframeResult")
driver.find_element_by_xpath("//nav/div/ul/li[2]/a").click()
Using python and selenium I need to locate and click a specific button on a webpage. While under normal circumstances this can be done with the commands
next = driver.find_element_by_css_selector(".next")
next.click()
that won't work in this case due to the webpages coding. Where both the button leading to the previous page and the one leading to the next share the same class name. Thus this code will only go back and forth between the first and second page
As far as I can tell the only way to tell the two buttons apart, is that the button leading to the previous page lies within
<li class="previous">
#button
</li>
and the one that leads to the next page lies within
<li class="next">
#button
</li>
Is there some way of having selenium only select and click the button that lies in the "next" li class?
The complete button code:
the previous button:
<li class="previous">
<a class="next" rel="nofollow" onclick="qc.pA('nrForm', 'f76', 'QClickEvent', '1', 'f28'); return false;" href="">
Previous
</a>
</li>
the next button:
<li class="next">
<a class="next" rel="nofollow" onclick="qc.pA('nrForm', 'f76', 'QClickEvent', '3', 'f28'); return false;" href="">
Next
</a>
</li>
I think you should use .find_element_by_xpath().
For example:
next = driver.find_element_by_xpath("//li[#class='next']/a")
prev = driver.find_element_by_xpath("//li[#class='previous']/a")
or:
next = driver.find_element_by_xpath("//a[text()='Next']")
prev = driver.find_element_by_xpath("//a[text()='Previous']")
With CSS selectors:
next = driver.find_element_by_css_selector('li.next>a')
CSS Selectors are cool and useful: http://www.w3schools.com/cssref/css_selectors.asp
XPATH, not so much - it should be used only as a last resort.