I'm trying to expand some collapsible content by taking all the elements that need to be expanded, and then clicking on them. Then once they're open, scrape the data shown. So far I'm grabbing a list of elements by their xpath, with this:
clicks = driver.find_elements_by_xpath("//div[contains(#class, 'collapsableContent') and contains(#class, 'empty')]")
and I've tried iterating with a simple for loop:
for item in clicks:
item.click()
but that doesn't seem to work. Any suggestions on where to look?
The specific page I'm trying to get this from is: https://sports.betway.com/en/sports/sct/esports/cs-go
Here is the code that you should use to open all the divs which have the collapsed empty class.
# click on close button in cookies policy (as this is the element which will overlap the element clicks)
driver.find_element_by_css_selector(".messageBoxCloseButton.icon-cross").click()
# get all the divs (collapsed divs)
links = driver.find_elements_by_xpath("//div[#class='collapsableContent empty']/preceding-sibling::div[#class='collapsableHeader']")
# click on each of those links
for link in links:
link.location_once_scrolled_into_view
link.click()
Related
Good time of the day!
Faced with a seemingly simple problem,
But it’s been a while, and I’m asking for your help.
I work with Selenium on Python and I need to curse about
20 items on google search page by random request.
And I’ll give you an example of the elements below, and the bottom line is, once the elements are revealed,
Google generates new such elements
Problem:
Cannot click on the element. I will need to click on existing and then on new, generated elements in this block: click for open see blocks with element
Tried to click on xpath, having collected all the elements:
xpath = '//*[#id="qmCCY_adG4Sj3QP025p4__16"]/div/div/div[1]/div[4]'
all_elements = driver.find_element(By.XPATH, value=xpath)
for element in all_elements:
element.click()
sleep(2)
Important note!
id xpath has constantly changing and is generated by another on the google side
Tried to click on the class
class="r21Kzd"
Tried to click on the selector:
#qmCCY_adG4Sj3QP025p4__16 > div > div > div > div.wWOJcd > div.r21Kzd
Errors
This is when I try to click using xpath:
Message: no such element: Unable to locate element: {"method":"xpath","selector"://*[#id="vU-CY7u3C8PIrgTuuJH4CQ_9"]/div/div[1]/div[4]}
In other cases, the story is almost the same, the driver does not find the element and cannot click on it. Below I apply a scratch tag on which I need to click
screenshot tags on google search
Thanks for the help!
In case iDjcJe IX9Lgd wwB5gf are a fixed class name values of that element all you need is to use CSS_SELECTOR instead of CLASS_NAME with a correct syntax of CSS Selectors.
So, instead of driver.find_element(By.CLASS_NAME, "iDjcJe IX9Lgd wwB5gf") try using this:
driver.find_element(By.CSS_SELECTOR, ".iDjcJe.IX9Lgd.wwB5gf")
(dots before each class name, no spaces between them)
I'm looking to scrape a label from an SVG that only arrives with a mouse hover.
I'm working with this link for the data contained with the [+] expand button to the right in each of the table rows. When you press [+] expand, an SVG table pops up that shows elements that contain elements. When you hover on each of the elements, a element appears called "Capacity Impact" with a value for each of the bars. These values are the values I want to scrape.
See a screenshot below.
So far, my code is successful in opening each of the [+] expand buttons, and identifying the polygons but I can't get to the labels using either XPATH or CSS Selectors. See code below.
driver.get(url)
table_button_xpath = "//table[#class='data-view-table redispatching dataTable']//tr//td[#class = 'button-column']//a[#class='openIcon pre-table-button operation-detail-expand small-button ui-button-light ui-button ui-widget ui-corner-all ui-button-text-only']"
driver.find_element(By.ID, "close-button").click()
driver.find_element(By.ID, "cookieconsent-button").click()
# open up all the "+" buttons
table_buttons = driver.find_element(By.XPATH, table_button_xpath)
for i in list(range(1, 10)):
driver.find_element(By.XPATH, table_button_xpath).click()
# find all the polygons
polygons = driver.find_elements(By.TAG_NAME, 'path')
label_xpath = "//*[name()='svg']//*[name()='g' and #id = 'ballons')]//*[name()='g']//*[name()='tspan']"
for polygon in polygons :
action.move_to_element(polygon)
labels_by_xpath = driver.find_elements(By.XPATH, label_xpath)
labels_by_css_selector = driver.find_elements(By.CSS_SELECTOR, "svg>#ballons>g>text>tspan")
Both labels_by_xpath and labels_by_css_selector return a list of 0 elements. I've tried many versions of both the xpath and css selector approach, along with using WebDriverWait, but I can't get it to return the capacity impact values.
HTML screenshot is also copied below (to be clear, the number I need to scrape is the "50" text in the tag.
Any help is appreciated! Thank you,
Sophie
The solution to your problem is with the locator.
Here is the updated locator to select the desired element.
CSS Selector :
svg>[id^='balloons']>g:nth-child(2)>text:nth-child(2)>tspan
try this to get the element Capacity 50
x = driver.find_elements(By.CSS_SELECTOR, "svg>[id^='balloons']>g>text>tspan")
HTML Image
I'm using selenium to get some information from a site, but the links that I need to click on require double clicking. Here's the code (I've included an attachment of the image of the HTML)
PATH = "C:\Program Files (x86)\msedgedriver.exe"
driver = webdriver.Edge(PATH)
action = ActionChains(driver)
from selenium import webdriver
from selenium.webdriver import ActionChains
frame = search_by_XPATH('//[#id="frmMain"]/table/tbody/tr/td[2]/div[2]').get_attribute('id').replace("_Div","_Frame")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID, frame)))
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"fraTree_Frame")))
rows = driver.find_elements(By.XPATH,'//*[#id="ctl00_treChart_RadTreeView"]/ul/li/div/span[2]')
for row in rows:
# row.click()
action.double_click(row).perform()
sleep(1)
I'm accessing different pages to get the links, and although each page has the same content the "id" of the frame changes with each new page I access. So that's why I use the 'get_attribute' function to get the id of the frame of the page.
I'm using action chains right now to perform the double click, but every time I do I get the stale element reference: element is not attached to the page document error. I know it's not an issue with the XPATH I'm using to access the element because I can perform a element.click() function on it just fine and the element receives the click. Nothing changes after the first click except the element gets highlighted. The element's xpath doesn't change and its still on the page. I also tried doing the action.click().perform() on the element but that produces the same error.
I've tried clicking on other parts of the element that could receive the click (the div, the span2,etc) but I always get the error. The 'span2' is the part that holds the text of the link (although its not an 'a' tag so its not really a link in that sense.)
I don't understand why the element would be able to receive the click for the .click() function but then be unable to receive the double click. I'm using the double_click() function on another element previously in my code and that works as expected.
I am scraping a website, where i need to select a value from dropdown. Then i need to click on search button. Which then populates a table with multiple pages of search results. For each result i want to click on it, which redirects to another page, extract data from that. Then come back and do the same for other search results.
select = Select(browser.find_element_by_id("dropdown id here"))
options = select.options
for index in range(0,len(options)):
select.select_by_index(index)
browser.find_element_by_id("checkbox").click()
time.sleep(5)
browser.find_element_by_id("click on search").click()
elems = browser.find_elements_by_xpath("need to enter all href for search results")
time.sleep(5)
for elem in elems:
#Need to enter each search result's href scrape within data and get back to search results again##
elem.get_attribute("href")
elem.click()
browser.find_element_by_id("for going back to search page again").click()
I get this error when i'm trying to iterate
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=87.0.4280.88)
This problem comes when the element is not able to find out as it has become stale
Try to bring the find elemenet logic inside for loop, that should solve your problem.
elems = browser.find_elements_by_xpath((xpath)[i])
length = elems.size();
Run your logic with number
for (int i=0;i<length; i++)
#Need to enter each search result's href scrape within data and get back to search results again##
elems = browser.find_elements_by_xpath("need to enter all href for search results"[i])
elem.get_attribute("href")
elem.click()
time.sleep(1)
you can read more about it here https://stackoverflow.com/a/12967602/2986279
If you are opening new windows, you will likely want to switch focus between newly opened windows/tabs and the original.
Switch to newly opened window.
windows = browser.window_handles
browser.switch_to.window(windows[-1])
Close newly opened window after finding what you need on the new page, then switch back to the previous window.
browser.close()
windows = browser.window_handles
browser.switch_to.window(windows[-1])
I'm trying to create a program extracting all persons I follow on Instagram. I'm using Python, Selenium and Chromedriver.
To do so, I first get the number of followed persons and click on the 'following' button : `
nb_abonnements = int(webdriver.find_element_by_xpath('/html/body/span[1]/section[1]/main/div[1]/header/section[1]/ul/li[3]/a/span').text)
sleep(randrange(1,3))
abonnements = webdriver.find_element_by_xpath('/html/body/span[1]/section[1]/main/div[1]/header/section[1]/ul/li[3]/a')
abonnements.click()
I then use the following code to get the followers and scroll the popup page in case I can't find one:
followers_panel = webdriver.find_element_by_xpath('/html/body/div[3]/div/div/div[2]')
while i < nb_abonnements:
try:
print(i)
followed = webdriver.find_element_by_xpath('/html/body/div[3]/div/div/div[2]/ul/div/li[{}]/div/div[2]/div/div/div/a'.format(i+1)).text
#the followeds are in an ul-list
i += 1
followed_list.append(followed)
except NoSuchElementException:
webdriver.execute_script(
"arguments[0].scrollBy(0,400)",followers_panel
)
sleep(7)
The problem is once i is at 12, the program raises the exception and scrolls. From there, he still can't find the next follower and is stuck in a loop where he does nothing but scroll. I've checked the source codeof the IG page, and it turns out the path is still good, but apparently I can't access the elements as I do anymore, probably because the ul-list in which I am accessing them has become to long (line 5 of the program).
I can't work out how to solve this. I hope you will be of some help.
UPDATE: the DOM looks like this:
html
body
span
script
...
div[3]
div
...
div
div
div[2]
ul
div
li
li
li
li
...
li
The ul is the list of the followers.
The lis contain the info i'm trying to extract (username). Even when I go go by myself on the webpage, open the popup window, scroll a little and let everything load, I can't find the element I'm looking for by typing the xpath in the search bar of the DOM manually. Although the path is correct, I can check it by looking at the DOM.
I've tried various webdrivers for selenium, currently I am using chromedriver 2.45.615291. I've also put an explicit wait to wait for the element to show (WebDriverWait(webdriver, 10).until(EC.presence_of_element_located((By.XPATH, '/html/body/div[3]/div/div/div[2]/ul/div/li[{}]/div/div[2]/div/div/div/a'.format(i+1))))), but I just get a timeout exception: selenium.common.exceptions.TimeoutException: Message:.
It just seems like once the ul list is too long (which is from the moment I've scrolled down enough to load new people), I can't access any element of the list by its XPATH, even the elements that were already loaded before I began scrolling.
Instead of using xpath for each of the child element... find the ul-list element then find all the child elements using something like : ul-list element.find_elements_by_tag_name(). Then iterate through each element in the collection & get the required text
I've foud a solution: i just access the element through the XPATH like this: find_element_by_xpath("(//*[#class='FPmhX notranslate _0imsa '])[{}]".format(i)). I don't know why it didn't work the other way, but like this it works just fine.