Develop Reference

Unable to scrape comments from YouTube in selenium

I am writing a YouTube scraper in selenium. I am unable to scrape comments.
Following is my code:
driver.get("https://www.youtube.com/watch?v=b5jt2bhSeXs")
wait = WebDriverWait(driver, 20)
time.sleep(10)
# wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-comments")))
wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-item-section-renderer")))
get_comments_and_commentors(driver)
def get_comments_and_commentors(driver):
scroll_down()
main_div = driver.find_element_by_id("contents")
inner = main_div.find_element_by_class_name("style-scope.ytd-item-section-renderer")
here = inner.find_element_by_tag_name("ytd-comment-renderer")
print(here)
scroll_down function is to scroll till the end to load all the comments this function is working because I can see the scrolling of page when I run this script.
The issue I am facing is that selenium is unable to find this tag "ytd-comment-renderer" instead of tag I have also tried to use the class name its giving the same error.
I am keep getting this error for both the tag and class name:
"selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"ytd-comment-renderer"}"
To fix this I tried this:
wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-comment-renderer")))
But I was getting the TimeOutException.
What can I do to fix this error.

Selenium - Unable to locate element

I'm trying to web scrape the list of restaurants from a website using Selenium, but I always get the error NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//*[#id="lista-encontrada"]/div[2]/div[1]/h4"}
I've searched for similar errors but it's always related to frame or iframe, but I can't seem to find them in this particular website. Here's the code:
from selenium import webdriver
url = 'https://www.restaurantemadero.com.br/pt/restaurante/sp/sao-paulo'
driver = webdriver.Chrome()
driver.get(url)
driver.maximize_window()
restaurants = driver.find_elements_by_class_name('blocos')
nome_loja = restaurants[0].find_element_by_xpath('.//*[#id="lista-encontrada"]/div[7]/div[1]/h4')
Any hint will be very helpful!

To get the restaurant name use following xpath. //div[#class='blocos']//h4
Following code will returns restaurant name in a list
value using text:
print([res.text for res in driver.find_elements_by_xpath("//div[#class='blocos']//h4")])
OR
value using textContent:
print([res.get_attribute("textContent") for res in driver.find_elements_by_xpath("//div[#class='blocos']//h4")])

Python Selenium Webdriver: return error after returning few elements

I am trying to retrieve the information related to reviewers posted on google play store. I tried to extract the name of reviewers and rating given by reviewer. Code returns the first set of information then it returns the error. I tried to debug but couldnt fix it. Here is my code:
driver = webdriver.Chrome(driverLocation)
driver.get('https://play.google.com/store/apps/details?id=com.getsomeheadspace.android&showAllReviews=true')
driver.implicitly_wait(20)
parentElement = driver.find_element_by_xpath('//h3[contains(text(),"User reviews")]//parent::div/div[1]/div')
for child in parentElement.find_element_by_tag_name('div'):
name = child.find_element_by_class_name('X43Kjb').text
indvd_rating = child.find_element_by_xpath('//span[#class="nt2C1d"]//div[#aria-label]').get_attribute('aria-label')
print(name)
print(indvd_rating)
driver.quit()
Error is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".X43Kjb"}
Could anyone point out where am I making the mistake?

While writing an automation script in python, page is taking too much time to load, and the elements cannot be found. How to fix this?

#with_setup(setup_module,teardown_module)
def test_checklist():
uname = Util.findElement("username")
if uname.is_displayed():
uname.send_keys("crussell")
pwd = Util.findElement("password")
if pwd.is_displayed():
pwd.send_keys("welcome")
loginelem = Util.findElement("login_btn")
if loginelem.is_displayed():
loginelem.click()
wbelmnts = Util.findElements("Left_Menu")
Util.clickElementFromList(wbelmnts, "Job Postings")
NoSuchElementException: Message: no such element: Unable to locate
element:
{"method":"xpath","selector":"//*[#class='loginForm']//div[#class='formGroup']/input[#name='login']"}

put time.sleep(5) it will wait for a page to load.

Can't find a button 'ctl00' with selenium

From this page:
https://permits.losgatosca.gov/CitizenAccess/default.aspx
I'm trying to push the 'Search Permits' button under Building Permits.
This is its xpath:
//*[#id="ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl"]/span
This is the code I'm using:
url = "https://permits.losgatosca.gov/CitizenAccess/default.aspx"
driver_1 = webdriver.Firefox()
driver_1.get(url)
NEXT_BUTTON_XPATH = '//*[#id="ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl"]/span'
# "//*[#id="ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl"]/span"
button = driver_1.find_element_by_xpath(NEXT_BUTTON_XPATH)
button.click()
But I get this message:
selenium.common.exceptions.NoSuchElementException: Message: Unable to
locate element:
{"method":"xpath","selector":"//*[#id=\"ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl\"]/span"}
Stacktrace:

There is a frame with id ACAFrame, you need to switch that frame first as below :-
driver_1.switch_to_frame("ACAFrame")
NEXT_BUTTON_XPATH = '//*[#id="ctl00_PlaceHolderMain_TabDataList_TabsDataList_ctl01_LinksDataList_ctl00_LinkItemUrl"]/span'
button = driver_1.find_element_by_xpath(NEXT_BUTTON_XPATH)
Edited..
After clicking Search Permits to perform action on opened form you should try as below :-
driver_1.switch_to.default_content()
driver_1.switch_to_frame("ACAFrame")
# now find your desire element
Hope it will work...:)