Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
I want to access a website's particular div,
can I access it?
if we can pls share an example.
I have watched many videos on this but all videos are for tables only(mostly)
Yes, you can, using beautiful soup.
For example, you can refine your search to only find those divs with a given class:
mydivs = soup.find_all("div", {"class": "class_to_find"})
Take a look here :
https://beautiful-soup-4.readthedocs.io/en/latest/
Use Beautiful Soup, a python library which can work with HTML/XML files.
soup.find('div', class_='your_div_class')
soup.select('div.your_class_name')[0]
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed last year.
Improve this question
I used -
URL = "http://www.google.com"
r = requests.get(URL)
soup = BeautifulSoup(r.content, 'html5lib')
print(soup.prettify())
I was learning python and did not know that web scraping is not allowed.
Short answer: No
Google doesn't allow web scraping, but they don't take any legal action against it. They use a lot of defense systems, so even if you tried to do something bad it would just be blocked.
Just don't worry about it :)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I am trying to scrape this website with scrapy and I have had to search for each link extracting the information from each one, I would like to know if there is an API of the site that I can use (I don't know how to find it).
I would also like to know how I can obtain the latitude and longitude? Currently the map is shown but I do not know how to obtain the numbers
I appreciate any suggestions
The website may be loading the data dynamically using Javascript. Use your browser dev tools and look at the networking tab, look for any XHR calls which may be accessing an API. Then you can scrape from that directly.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I am writing a python program to evaluate stock prices. I'm using this page on Yahoo! finance to get my stock informaiton. I want to be able to get the top five listings' stock symbol on the top gainers page.
Can someone either provide me with an example of how to get the top five stock symbols or show me how I can find the symbol element(using the data-reactid or any other meathod) using selenium preferably.
Before this is flagged as a copy, I looked at the pages similar to this, but they did not solve my problem. Thanks in advance for any help!
I personally don't have a lot of experience with Selenium, but this sounds like a job that could be handled with either BeatifulSoup's find()/findall() methods, or with scrapy's Xpath/ CSS selectors.
For a beginner, I would recommend BeautifulSoup for a task like this. it makes it easy to target the page element you're looking for (in this case the stock symbol w/ data-reactid).
Hope this helped.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
Could you recommend me some ways to scrape data from a web page?
I have been trying to use Python but I am stuck with my code. I was thinking about using Octoparse. This is the webpage (http://www.mlsa.am/?page_id=368), it is a drop-down list where the selection of a previous case allows you to choose other options in the other cases.
You could use scrapy framework specially built for scraping purpose only.
As an starter you can start from official documentation & you will find everything you need from it.
https://docs.scrapy.org/en/latest/intro/tutorial.html
except scrapy you can use beautifulsoup also.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
I'm trying to capture a test case result where table content Search/filter output need to cross check each time when the test run. I have attached a table grid that I need to use to search/filter. I'm using python script for the automation.
Any suggestion?
You can use selenium to test. The table's inner HTML can be accessed using
table_content = element.get_attribute('innerHTML').
you can parse that HTML to cross check your results.
Have a look at this question for reference.
Get HTML Source of WebElement in Selenium WebDriver using Python