I have this code:
from selenium import webdriver
# Set url and path
url = 'https://osu.ppy.sh/beatmapsets?m=0'
driver = webdriver.Chrome(executable_path=r"C:\Users\Gabri\anaconda3\chromedriver.exe")
driver.get(url)
# Select the item that i want
try:
dif = driver.find_element_by_css_selector('div.beatmapset-panel__beatmap-dot')
dif.text
print(dif)
except:
print('not found')
I'm trying to select this map difficulty 'expert in purple ' --> https://imgur.com/a/G224rka but i can't continue with my code cuz the output is "<selenium.webdriver.remote.webelement.WebElement (session="3cdaf38d0673d0aebe49733d629eae5c", element="60d6241b-80f7-42c8-bf38-9fd2c8574b08")>" and i expected that would be a string like "expert" or "--bg:var(--diff-expert);" How i can translate or convert ? I did try to select with '[class*="beatmapset-panel__beatmap-dot"' and the output is same. Someone can help me?
You need to change your code as following to print the element text:
from selenium import webdriver
# Set url and path
url = 'https://osu.ppy.sh/beatmapsets?m=0'
driver = webdriver.Chrome(executable_path=r"C:\Users\Gabri\anaconda3\chromedriver.exe")
driver.get(url)
# Select the item that i want
try:
dif = driver.find_element_by_css_selector('div.beatmapset-panel__beatmap-dot')
print(dif.text)
except:
print('not found')
You need to hover to an element, then wait for data to appear and get its text:
Here is the snippet for the first game from the list. To get all games you will need another loop.
I am using ActionChains to hover on element. Finding locators for this site was not easy even for me.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')
driver.get("https://osu.ppy.sh/beatmapsets?m=0")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".beatmapsets__items-row:nth-of-type(1)>.beatmapsets__item:nth-of-type(1)")))
games = driver.find_element_by_css_selector(".beatmapsets__items-row:nth-of-type(1) .beatmapsets__item:nth-of-type(1) .beatmapset-panel__info-row--extra")
actions = ActionChains(driver)
actions.move_to_element(games).perform()
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".beatmaps-popup__group")))
levels = driver.find_elements_by_css_selector(".beatmaps-popup__group .beatmaps-popup-item__col.beatmaps-popup-item__col--name.u-ellipsis-overflow")
for level in levels:
print(level.text)
Output:
Hinsvar's Hard
Zelqurre's Insane
Amamir's Shining Stars
For iteration through a list of levels use this css selector:
.beatmapsets__items-row:nth-of-type(1) .beatmapsets__item:nth-of-type(1) .beatmapset-panel__info-row--extra
And iterate this locator:
.beatmapsets__items-row:nth-of-type(1) .beatmapsets__item:nth-of-type(1) .beatmapset-panel__info-row--extra,
.beatmapsets__items-row:nth-of-type(2) .beatmapsets__item:nth-of-type(1) .beatmapset-panel__info-row--extra
Update:
To get scores use:
scores= driver.find_elements_by_css_selector(".beatmaps-popup__group .beatmaps-popup-item__col.beatmaps-popup-item__col--difficulty")
for score in scores:
print(score.text)
The output will be:
2.58
3.46
4.55
4.90
5.97
Also, check this answer on how to put results in one list: Trouble retrieving elements and looping pages using next page button
And finally read here about css selectors: https://www.w3schools.com/cssref/css_selectors.asp
I usually prefer using them because they are shorter.
Related
I'm beginner on programming. I trying get my Instagram follower list but i have just 12 follower. I tried firstly click to box and scroll down but it didn't work.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
url= "https://www.instagram.com/"
driver.get(url)
time.sleep(1)
kullaniciAdiGir = driver.find_element(By.XPATH, "//*[#id='loginForm']/div/div[1]/div/label/input"")
kullaniciAdiGir.send_keys("USERNAME")
sifreGir = driver.find_element(By.XPATH, "//input[#name='password']")
sifreGir.send_keys("PASS")
girisButonu = driver.find_element(By.XPATH, "//*[#id='loginForm']/div/div[3]/button/div").click()
time.sleep(5)
driver.get(url="https://www.instagram.com/USERNAME/")
time.sleep(3)
kutucuk= driver.get(url="https://www.instagram.com/USERNAME/followers/")
time.sleep(5)
box =driver.find_element(By.XPATH, "//div[#class='xs83m0k xl56j7k x1iy3rx x1n2onr6 x1sy10c2 x1h5jrl4 xieb3on xmn8rco x1hfn5x7 x13wlyjk x1v7wizp x1l0w46t xa3vuyk xw8ag78']")
box.click()
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
takipciler = driver.find_elements(By.CSS_SELECTOR, "._ab8y._ab94._ab97._ab9f._ab9k._ab9p._abcm")
for takipci in takipciler:
print(takipci.text)
time.sleep(10)
How can i fix it? How can scroll down in box? Thanks
You can select multiple elements with this.
#get all followers
followers = driver.find_elements(By.CSS_SELECTOR, "._ab8y._ab94._ab97._ab9f._ab9k._ab9p._abcm")
# loop each follower
for user in followers:
#do something here.
Using css selectors, in my opinion, is much easier.
Also note I used find_elemets not find_element, as the latter only returns a single result.
As the data is loaded dynamically, you will, infact have to scroll to make the site load more results. Then compare what you have agaisnt whats loaded. Probably execute some javascrtipt like scroll into view on the last element in that container etc.
or take a look here for an alternative solution
https://www.folkstalk.com/2022/10/how-to-get-a-list-of-followers-on-instagram-python-with-code-examples.html
or look at instragrams API, probably something in there for getting your followers.
I'm trying to get the text from the searched result but it shows me "TypeError: 'WebElement' object is not iterable" and want compare it with expected result below is my code [Python+Selenium]
from selenium import webdriver
import time texts = []
driver = webdriver.Chrome(executable_path = "C:\\chromedriver.exe")
driver.maximize_window() driver.get("https://rahulshettyacademy.com/seleniumPractise/#/")
driver.find_element_by_css_selector("input.search-keyword").send_keys("ber")
texts = driver.find_elements_by_xpath("//div[#class='product']/h4") print(texts)
Here is the website: https://rahulshettyacademy.com/seleniumPractise/#/
Trying to search "ber" and want to get all the product names
Thanks for your help in Advacne!
There are several issues here.
First of all you have to add a delay before getting the resulted elements with driver.find_elements_by_xpath("//div[#class='product']/h4")
The best way to do that is to introduce an Expected Conditions explicit wait.
Also your texts object is a list of web elements.
You can't print it directly.
What you can do is to iterate over that list of web elements and extract a text from each web element and then print it.
The following should work better:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time texts = []
driver = webdriver.Chrome(executable_path = "C:\\chromedriver.exe")
wait = WebDriverWait(driver, 20)
driver.maximize_window() driver.get("https://rahulshettyacademy.com/seleniumPractise/#/")
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input.search-keyword")))
driver.find_element_by_css_selector("input.search-keyword").send_keys("ber")
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[#class='product']/h4")))
text_elements = driver.find_elements_by_xpath("//div[#class='product']/h4")
for text_element in text_elements:
el_text = text_elements.text
print(el_text)
Hoping you can help. I'm relatively new to Python and Selenium. I'm trying to pull together a simple script that will automate news searching on various websites. The primary focus was football and to go and get me the latest Manchester United news from a couple of places and save the list of link titles and URLs for me. I could then look through the links myself and choose anything I wanted to review.
In trying the the independent newspaper (https://www.independent.co.uk/) I seem to have come up against a problem with element not interactable when using the following approaches:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('chromedriver')
driver.get('https://www.independent.co.uk')
time.sleep(3)
#accept the cookies/privacy bit
OK = driver.find_element_by_id('qcCmpButtons')
OK.click()
#wait a few seconds, just in case
time.sleep(5)
search_toggle = driver.find_element_by_class_name('icon-search.dropdown-toggle')
search_toggle.click()
This throws the selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable error
I've also tried with XPATH
search_toggle = driver.find_element_by_xpath('//*[#id="quick-search-toggle"]')
and I also tried ID.
I did a lot of reading on here and then also tried using WebDriverWait and execute_script methods:
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="quick-search-toggle"]')))
driver.execute_script("arguments[0].click();", element)
This didn't seem to error but the search box never appeared, i.e. the appropriate click didn't happen.
Any help you could give would be fantastic.
Thanks,
Pete
Your locator is //*[#id="quick-search-toggle"], there are 2 on the page. The first is invisible and the second is visible. By default selenium refers to the first element, sadly the element you mean is the second one, so you need another unique locator. Try this:
search_toggle = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//div[#class="row secondary"]//a[#id="quick-search-toggle"]')))
search_toggle.click()
First you need to open search box, then send search keys:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import os
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
browser = webdriver.Chrome(executable_path=os.path.abspath(os.getcwd()) + "/chromedriver", options=chrome_options)
link = 'https://www.independent.co.uk'
browser.get(link)
# accept privacy
button = browser.find_element_by_xpath('//*[#id="qcCmpButtons"]/button').click()
# open search box
li = browser.find_element_by_xpath('//*[#id="masthead"]/div[3]/nav[2]/ul/li[1]')
search_tab = li.find_element_by_tag_name('a').click()
# send keys to search box
search = browser.find_element_by_xpath('//*[#id="gsc-i-id1"]')
search.send_keys("python")
search.send_keys(Keys.RETURN)
Can you try with below steps
search_toggle = driver.find_element_by_xpath('//*[#class="row secondary"]/nav[2]/ul/li[1]/a')
search_toggle.click()
I don't understand why the list I'm trying to extract the text from is returning blanks when I'm definitely using the correct Xpath. Here is my code:
driver = webdriver.Firefox()
driver.get("https://www.omegawatches.com/watch-omega-specialities-first-omega-wrist-chronograph-51652483004001")
betweenLugs = driver.find_elements(By.XPATH, "/html/body/div[2]/main/div[3]/div/div/div[2]/div/div[2]/div[3]/div/ul/li[1]")])
print(betweenLugs.text)
This should grab the first list item and measurement
Between lugs: 20 mm
I have also tried other methods, but the fact that Xpath doesn't pick it up tells me something is wrong and it doesn't matter how I do it, I won't be able to extract the text inside the lists. Does anyone know what am I doing wrong? This is the first time I've ran into this problem.
The xpath is wrong. It fails in /div[2], it doesn't match anything. This is an example why you shouldn't use absolute path.
The section has id attribute, use it
betweenLugs = driver.find_elements(By.XPATH, "//*[#id='product-info-data-5bea7fa7406d7']/ul/li[1]")[0]
You might also want to add some wait for the loading
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
betweenLugs = WebDriverWait(driver, 10).until(expected_conditions.visibility_of_element_located((By.XPATH, "//*[#id='product-info-data-5bea7fa7406d7']/ul/li[1]")))
OK, try this and see if it solves the problem:
between_lugs = driver.find_element_by_xpath("//*[contains(text(), 'Between lugs')]").get_attribute("innerHTML")
between_lugs_value = driver.find_element_by_xpath("//*[contains(text(), 'Between lugs')]/../span").get_attribute("innerHTML")
final_text = between_lugs + " " + between_lugs_value
That page already has jQuery on it so you can just:
driver.execute_script("return jQuery('li:contains(Between lugs)').text().trim().replace(/\s+/g, ' ')")
You can fiddle with selectors in the chrome selectors, it makes it much easier.
Another simpler approach might be the following one:
from contextlib import closing
from selenium import webdriver
from selenium.webdriver.support import ui
url = "https://www.omegawatches.com/watch-omega-specialities-first-omega-wrist-chronograph-51652483004001"
with closing(webdriver.Chrome()) as wd:
wait = ui.WebDriverWait(wd, 10)
wd.get(url)
item = wait.until(lambda wd: wd.find_element_by_xpath("//*[contains(#class,'technical-data')]//li")).get_attribute('textContent')
print(' '.join(item.split()))
Output:
Between lugs: 20 mm
Using a scroll down and a wait with a css selector to target the parent li
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
driver = webdriver.Chrome() #Firefox()
driver.get("https://www.omegawatches.com/watch-omega-specialities-first-omega-wrist-chronograph-51652483004001")
driver.execute_script("window.scrollTo(0, 2000)")
betweenLugs = WebDriverWait(driver, 10).until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, "#product-info-data-5beaf5497d916 > ul > li:nth-child(1)")))
print(betweenLugs.text)
What i am trying to do is make a simple program that lets me run and it basically goes to Torrentz and follows a few link to finally be able to download the file through uttorent. Below is what i have coded so far and i cant seem to make the variable linkElem work. And i also cant seem to make linkElem.find_elements_by_xpath go to the link necessary. If you think you know what is wrong, please do help.
Thanks.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Firefox()
browser.get('https://torrentz.eu/')
searchElem = browser.find_element_by_id('thesearchbox')
searchElem.send_keys('Limitless')
searchButton = browser.find_element_by_id('thesearchbutton')
searchButton.click()
linkElem = linkElem.find_elements_by_xpath("//div[#class='results']//a[#href='/9ea0c575520a3065d85b285c9474231192368db7']")
#wait = WebDriverWait(browser, 6)
#linkElem = wait.until(EC.visibility_of_element_located((By.href, "/9ea0c575520a3065d85b285c9474231192368db7")))
#linkElem.clear()
#linkElem = browser.find_element_by_link_text('S01E20 HDTV x264 LOL ettv')
#linkElem.click()
#SignIn = browser.find_elements_by_id('signIn')
#SignIn.click()
#passwordElem.submit()
I don't think you can and should rely on the href attribute value. Instead, get the links from under the dl elements inside the search results container. Also, add a wait:
# wait for search results to appear
wait = WebDriverWait(browser, 6)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.results dl")))
links = driver.find_elements_by_css_selector("div.results dl dt a")
links[0].click()
links in your case would contain all of the search results links, links[0] is the first link.