Unable to scrape comments from YouTube in selenium - python

I am writing a YouTube scraper in selenium. I am unable to scrape comments.
Following is my code:
driver.get("https://www.youtube.com/watch?v=b5jt2bhSeXs")
wait = WebDriverWait(driver, 20)
time.sleep(10)
# wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-comments")))
wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-item-section-renderer")))
get_comments_and_commentors(driver)
def get_comments_and_commentors(driver):
scroll_down()
main_div = driver.find_element_by_id("contents")
inner = main_div.find_element_by_class_name("style-scope.ytd-item-section-renderer")
here = inner.find_element_by_tag_name("ytd-comment-renderer")
print(here)
scroll_down function is to scroll till the end to load all the comments this function is working because I can see the scrolling of page when I run this script.
The issue I am facing is that selenium is unable to find this tag "ytd-comment-renderer" instead of tag I have also tried to use the class name its giving the same error.
I am keep getting this error for both the tag and class name:
"selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"ytd-comment-renderer"}"
To fix this I tried this:
wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-comment-renderer")))
But I was getting the TimeOutException.
What can I do to fix this error.

Related

python selenium html element select

Hello first. Im try to make a bot which can make a list about your followers and Im using selenium/python for this. Now Im tying to click followers button on my profile but I guess something goes wrong about xpath, how can I fix this.
def followers(self):
self.browser.get("https://www.instagram.com/" + instagramLoginInfo.username +"/")
sleep(1)
followerslink = self.browser.find_element_by_xpath('//*[#id="react-root"]/section/main/div/ul/li[2]/a')
followerslink.click()
sleep(5)
followers = self.browser.find_element_by_css_selector("div[role=dialog] ul").find_elements_by_css_selector("li")
for user in followers:
link = user.find_element_by_css_selector("a").get_attribure("href")
print(link)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="react-root"]/section/main/div/ul/li[2]/a"}
(Session info: chrome=94.0.4606.61)

Python Selenium upload file to website by class name

I am trying to use Python to upload files to a website. But when I looked at the Developer Inspection Tool, there is no id, and no name. I am trying to use class name, but it does not work.
Inspect window from Selenium:
Inspect window from website:
I've tried methods like these:
upload = driver.find_element_by_class_name("icomoon icon-upload2 toolbar-button fileinput-button")
upload = driver.find_element_by_class_name("fileupload")
upload = driver.find_element_by_class_name("btnContainer").
All failed.
the error message:
NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":".btnContainer"}
(Session info: chrome=91.0.4472.77)
Then I use:
upload = driver.find_element_by_css_selector('body > div.fileupload')
upload.send_keys(r'c\user\xxx\upload\excel.xlsx')
#or
upload = driver.find_element_by_xpath("//div[#class='fileupload']")
upload.send_keys(r'c\user\xxx\upload\excel.xlsx')
The error message:
ElementNotInteractableException: Message: element not interactable
(Session info: chrome=91.0.4472.77)
Can anyone help me with this?
Thanks,
JH.
If you want a quick solution you can always try getting by ccs selctor. Right click on the element in the inspect widow. click on copy > selector. And then in python write driver.find_element_by_css_selector('paste the selector here')
This is not an elegant solution but it should work.

Selenium - Unable to locate element

I'm trying to web scrape the list of restaurants from a website using Selenium, but I always get the error NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//*[#id="lista-encontrada"]/div[2]/div[1]/h4"}
I've searched for similar errors but it's always related to frame or iframe, but I can't seem to find them in this particular website. Here's the code:
from selenium import webdriver
url = 'https://www.restaurantemadero.com.br/pt/restaurante/sp/sao-paulo'
driver = webdriver.Chrome()
driver.get(url)
driver.maximize_window()
restaurants = driver.find_elements_by_class_name('blocos')
nome_loja = restaurants[0].find_element_by_xpath('.//*[#id="lista-encontrada"]/div[7]/div[1]/h4')
Any hint will be very helpful!
To get the restaurant name use following xpath. //div[#class='blocos']//h4
Following code will returns restaurant name in a list
value using text:
print([res.text for res in driver.find_elements_by_xpath("//div[#class='blocos']//h4")])
OR
value using textContent:
print([res.get_attribute("textContent") for res in driver.find_elements_by_xpath("//div[#class='blocos']//h4")])

Getting an error:Unable to locate element: .ng-star-inserted when locating an element called ng-star-inserted in selenium python

I'm running the following code and getting the following error:
select_survey = browser.find_element_by_class_name("ng-star-inserted").click()
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .ng-star-inserted
this is the entire code:
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser = webdriver.Firefox()
browser.get("https://www.legeropinion.com/app/todo")
email_element = browser.find_element_by_id("email")
email_element.clear()
email_element.send_keys("something#gmail.com")
password_element = browser.find_element_by_id("password")
password_element.send_keys("password")
sign_in = browser.find_element_by_class_name("submit-btn").click()
select_survey = browser.find_element_by_class_name("ng-star-inserted").click()
I am trying to automate signing into a website and completing a form but upon clicking on the form for which the element is class = ng-star-inserted i am getting that error
edit:
this is the relevant html info i believe:
<task-block _ngcontent-buv-c2="" _nghost-buv-c9="" class="ng-star-inserted"><!----><div _ngcontent-buv-c9="" class="task ng-star-inserted"><!----><a _ngcontent-buv-c9=""
Okay so after some a lot of trial and error I finally got it to work by fluke. I don't know how but it worked!!
I simply located the element by the ABSOLUTE xpath which I copied from the HTML and then before that line i had the time sleep function like so:
time.sleep(2)
select_survey = browser.find_element_by_xpath("/html/body/app-root/ion-app/ion- router-outlet/app-todo/ion-content/mat-expansion-panel[1]/div/div/div[2]/task-block[2] /div").click()

Python Selenium Webdriver: return error after returning few elements

I am trying to retrieve the information related to reviewers posted on google play store. I tried to extract the name of reviewers and rating given by reviewer. Code returns the first set of information then it returns the error. I tried to debug but couldnt fix it. Here is my code:
driver = webdriver.Chrome(driverLocation)
driver.get('https://play.google.com/store/apps/details?id=com.getsomeheadspace.android&showAllReviews=true')
driver.implicitly_wait(20)
parentElement = driver.find_element_by_xpath('//h3[contains(text(),"User reviews")]//parent::div/div[1]/div')
for child in parentElement.find_element_by_tag_name('div'):
name = child.find_element_by_class_name('X43Kjb').text
indvd_rating = child.find_element_by_xpath('//span[#class="nt2C1d"]//div[#aria-label]').get_attribute('aria-label')
print(name)
print(indvd_rating)
driver.quit()
Error is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".X43Kjb"}
Could anyone point out where am I making the mistake?

Categories

Resources