I'm trying to web scrape the list of restaurants from a website using Selenium, but I always get the error NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//*[#id="lista-encontrada"]/div[2]/div[1]/h4"}
I've searched for similar errors but it's always related to frame or iframe, but I can't seem to find them in this particular website. Here's the code:
from selenium import webdriver
url = 'https://www.restaurantemadero.com.br/pt/restaurante/sp/sao-paulo'
driver = webdriver.Chrome()
driver.get(url)
driver.maximize_window()
restaurants = driver.find_elements_by_class_name('blocos')
nome_loja = restaurants[0].find_element_by_xpath('.//*[#id="lista-encontrada"]/div[7]/div[1]/h4')
Any hint will be very helpful!
To get the restaurant name use following xpath. //div[#class='blocos']//h4
Following code will returns restaurant name in a list
value using text:
print([res.text for res in driver.find_elements_by_xpath("//div[#class='blocos']//h4")])
OR
value using textContent:
print([res.get_attribute("textContent") for res in driver.find_elements_by_xpath("//div[#class='blocos']//h4")])
Related
I am writing a YouTube scraper in selenium. I am unable to scrape comments.
Following is my code:
driver.get("https://www.youtube.com/watch?v=b5jt2bhSeXs")
wait = WebDriverWait(driver, 20)
time.sleep(10)
# wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-comments")))
wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-item-section-renderer")))
get_comments_and_commentors(driver)
def get_comments_and_commentors(driver):
scroll_down()
main_div = driver.find_element_by_id("contents")
inner = main_div.find_element_by_class_name("style-scope.ytd-item-section-renderer")
here = inner.find_element_by_tag_name("ytd-comment-renderer")
print(here)
scroll_down function is to scroll till the end to load all the comments this function is working because I can see the scrolling of page when I run this script.
The issue I am facing is that selenium is unable to find this tag "ytd-comment-renderer" instead of tag I have also tried to use the class name its giving the same error.
I am keep getting this error for both the tag and class name:
"selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"ytd-comment-renderer"}"
To fix this I tried this:
wait.until(EC.presence_of_element_located((By.TAG_NAME, "ytd-comment-renderer")))
But I was getting the TimeOutException.
What can I do to fix this error.
I'm trying to open a website using Selenium, find a specific search box within it and fill it with a company name (MAGAZINE LUIZA S.A. in the example) and then click the "Search" button right next to the search box, since it will not work if I just hit "enter".
nav = webdriver.Edge()
nav.implicitly_wait(10)
nav.get('http://www.b3.com.br/pt_br/produtos-e-servicos/negociacao/renda-variavel/empresas-listadas.htm')
nav.find_element_by_xpath("/html/body/form/div[3]/div[1]/div/div/div/div/div[3]/div[1]/div[1]/div[2]/div[1]/div/div[1]/label/span[1]/input[1]").send_keys('MAGAZINE LUIZA S.A.')
nav.find_elements_by_xpath('/html/body/form/div[3]/div[1]/div/div/div/div/div[3]/div[1]/div[1]/div[2]/div[1]/div/div[2]/input').click()
And I get the Xpath of the search bar and search button by inspecting them and "copying full Xpath" in Microsoft Edge.
Problem is I get this error:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/form/div[3]/div[1]/div/div/div/div/div[3]/div[1]/div[1]/div[2]/div[1]/div/div[1]/label/span[1]/input[1]"}
(Session info: MicrosoftEdge=91.0.864.54)
But I have verified that I have copied and pasted the correct Xpaths for both the commands. Can anyone help?
You need to handle iframe, that XPath is relative to iframe.
This is your modified code
nav = webdriver.Edge()
nav.implicitly_wait(10)
nav.get('http://www.b3.com.br/pt_br/produtos-e-servicos/negociacao/renda-variavel/empresas-listadas.htm')
iframe = nav.find_element_by_xpath('/html/body/main/div[4]/div/div[1]/div[1]/div/form/section/div/div/div/iframe')
nav.switch_to.frame(iframe)
inp = nav.find_element_by_xpath("/html/body/form/div[3]/div[1]/div/div/div/div/div[3]/div[1]/div[1]/div[2]/div[1]/div/div[1]/label/span[1]/input[1]")
inp.send_keys('MAGAZINE LUIZA S.A.')
nav.find_element_by_xpath('/html/body/form/div[3]/div[1]/div/div/div/div/div[3]/div[1]/div[1]/div[2]/div[1]/div/div[2]/input').click()
I'm running the following code and getting the following error:
select_survey = browser.find_element_by_class_name("ng-star-inserted").click()
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .ng-star-inserted
this is the entire code:
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser = webdriver.Firefox()
browser.get("https://www.legeropinion.com/app/todo")
email_element = browser.find_element_by_id("email")
email_element.clear()
email_element.send_keys("something#gmail.com")
password_element = browser.find_element_by_id("password")
password_element.send_keys("password")
sign_in = browser.find_element_by_class_name("submit-btn").click()
select_survey = browser.find_element_by_class_name("ng-star-inserted").click()
I am trying to automate signing into a website and completing a form but upon clicking on the form for which the element is class = ng-star-inserted i am getting that error
edit:
this is the relevant html info i believe:
<task-block _ngcontent-buv-c2="" _nghost-buv-c9="" class="ng-star-inserted"><!----><div _ngcontent-buv-c9="" class="task ng-star-inserted"><!----><a _ngcontent-buv-c9=""
Okay so after some a lot of trial and error I finally got it to work by fluke. I don't know how but it worked!!
I simply located the element by the ABSOLUTE xpath which I copied from the HTML and then before that line i had the time sleep function like so:
time.sleep(2)
select_survey = browser.find_element_by_xpath("/html/body/app-root/ion-app/ion- router-outlet/app-todo/ion-content/mat-expansion-panel[1]/div/div/div[2]/task-block[2] /div").click()
I am trying to retrieve the information related to reviewers posted on google play store. I tried to extract the name of reviewers and rating given by reviewer. Code returns the first set of information then it returns the error. I tried to debug but couldnt fix it. Here is my code:
driver = webdriver.Chrome(driverLocation)
driver.get('https://play.google.com/store/apps/details?id=com.getsomeheadspace.android&showAllReviews=true')
driver.implicitly_wait(20)
parentElement = driver.find_element_by_xpath('//h3[contains(text(),"User reviews")]//parent::div/div[1]/div')
for child in parentElement.find_element_by_tag_name('div'):
name = child.find_element_by_class_name('X43Kjb').text
indvd_rating = child.find_element_by_xpath('//span[#class="nt2C1d"]//div[#aria-label]').get_attribute('aria-label')
print(name)
print(indvd_rating)
driver.quit()
Error is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".X43Kjb"}
Could anyone point out where am I making the mistake?
I am trying to screen scrape a website (snippet below)
The website takes an input, navigates to a second page and takes more inputs and finally displays a table. I fail at this step:
driver.find_element_by_xpath("//select[#id='agencies']/option[#value='13156']").click()
The error I get is:
selenium.common.exceptions.NoSuchElementException: Message: 'Unable to locate element:
Which is strange because I do see the element (Commented out Display id). Any help/pointers, please?
(I tried requests/RoboBrowser -- can't seem to get the post to work but failed there as well)
from selenium import webdriver
from selenium import selenium
from bs4 import BeautifulSoup
driver = webdriver.Firefox()
url = 'http://www.ucrdatatool.gov/Search/Crime/Local/OneYearofData.cfm'
driver.get(url)
driver.find_element_by_xpath("//select[#id='state']/option[#value='1']").click()
#driver.find_element_by_xpath("//select[#id='groups']/option[#value='8']").click()
driver.find_element_by_xpath("//input[#type='submit' and #value='Next']").click()
driver.implicitly_wait(5) # seconds
# Display id tags
#elementsAll = driver.find_elements_by_xpath('//*[#id]')
#for elements in elementsAll:
# print("id: ", repr(elements))
# print("idName: ",elements.get_attribute("id"))
# driver.implicitly_wait(5) # seconds
driver.find_element_by_xpath("//select[#id='groups']/option[#value='2']").click()
driver.find_element_by_xpath("//select[#id='year']/option[#value=1986]").click()
driver.find_element_by_xpath("//select[#id='agencies']/option[#value='13156']").click()
Update -- the below works on Selenium. I intended to choose all options in the list box and save the query results...Thanks for the pointer, Alecxe!
select = Select(driver.find_element_by_id('agencies'))
for options in select.options:
select.select_by_visible_text(options.text)
select = Select(driver.find_element_by_id('groups'))
for options in select.options:
select.select_by_visible_text(options.text)
driver.find_element_by_xpath("//select[#id='year']/option[#value=1985]").click()
driver.find_element_by_xpath("//input[#type='submit' and #value='Get Table']").click()
There is no option with 13156 value in select with agencies id. There are values from 102 to 522, you can see them by printing:
[element.get_attribute('value') for element in driver.find_elements_by_xpath('//select[#id="agencies"]/option')]
Also, instead of finding options by value, use Select and get options by text:
from selenium.webdriver.support.ui import Select
select = Select(driver.find_element_by_id('agencies'))
print select.options
select.select_by_visible_text('Selma Police Dept')