I was trying to create a function to get a link to another Wikipedia page from one. Link to All other wiki articles starts with the prefix "/wiki/". I tried a code to get one random link but my code was getting all the class. After that, I saw the following code on the internet.
allLinks = soup.find(id="bodyContent").find_all("a")
random.shuffle(allLinks)
linkToScrape = 0
for link in allLinks:
# We are only interested in other wiki articles
if link['href'].find("/wiki/") == -1:
continue
# Use this link to scrape
linkToScrape = link
break
This code chunk seems to work perfectly. However, I couldn't understand a part.
if link['href'].find("/wiki/") == -1:
I couldn't understand the use of -1. Moreover, can someone explain how the conditional in this line of code works and how the find function is used here?
For background, here is the page I found the code on: "https://www.freecodecamp.org/news/scraping-wikipedia-articles-with-python/"
s.find(sub) returns -1 if the substring sub is not found in the string s. So in this case it is saying "If we don't find /wiki/ in the link string then continue because it is not a wikipedia link".
The reason why it is a weird number like -1 is because find returns the index the substring is found at and that could be any positive number or 0. So -1 is used to signify not found at any index.
https://docs.python.org/3/library/stdtypes.html#str.find
Although it seems like startswith would be more appropriate in this case:
if not link['href'].starts("/wiki/"):
continue
https://docs.python.org/3/library/stdtypes.html#str.startswith
Related
I'm new to programming and I have been studying through FreeCodeCamp for a couple of weeks. I was reviewing a code available on their website and I have two questions.
Why is it necessary to shuffle the variable allLinks?
What does if link ['href'].find("/wiki/") == -1 really check?
I really appreciate your help.
import requests
from bs4 import BeautifulSoup
import random
response = requests.get(
url="https://en.wikipedia.org/wiki/Web_scraping",
)
soup = BeautifulSoup(response.content, 'html.parser')
title = soup.find(id="firstHeading")
print(title.content)
# Get all the links
allLinks = soup.find(id="bodyContent").find_all("a")
random.shuffle(allLinks)
linkToScrape = 0
for link in allLinks:
# We are only interested in other wiki articles
if link['href'].find("/wiki/") == -1:
continue
# Use this link to scrape
linkToScrape = link
break
print(linkToScrape)
Here is the output:
Eventbrite, Inc.
It's better to first explain question 2 and describe what the code is doing.
Recall that find() returns returns the index of the first occurence of the substring or -1 if it can't find the desired substring. So if link ['href'].find("/wiki/") == -1checks if the substring "/wiki/" is found in the string link ['href']. It executes the continue if the reference link does not contain "/wiki/". So the loop will continue through the links until it finds a link that contains /wiki/ or until it goes through all the links. Hence the value of linkToScrape at the end of the loop will be the first link containing /wiki/ or 0 if no links contain it.
Now for the first question, the shuffle is necessary for the code to do what it does, which is printing a random wikipidia article linked within the article https://en.wikipedia.org/wiki/Web_scraping. Without the shuffling the code would instead always print the first wiki article linked within the article https://en.wikipedia.org/wiki/Web_scraping.
Note: The intention expressed in the comments is to find only wiki articles, but a link such as https://google.com/wiki/ will also be printed without being a wiki article.
You can read the documentation for .find() here: https://docs.python.org/3/library/stdtypes.html#str.find
Basically, haystack.find(needle) == -1 is a roundabout way of saying needle not in haystack.
The point of calling random.shuffle() is that the author of this code evidently wanted to select a link from the page at random rather than using, say, the first one or the last one.
For "extra credit" in a beginners class in Python that I am taking I wanted to extract data out of a URL using regex. I know that there are other ways I could probably do this, but my regex desperately needs work so...
Given a URL to start at, find the xth occurrence of a href on the page, and use that link to go down a level. Rinse and repeat until I have found the required link on the page at the requested depth on the site.
I am using Python 3.7 and Beautiful Soup 4.
At the beginning of the program, after all of the house-keeping is done, I have:
starting_url = 'http://blah_blah_blah_by_Joe.html'
extracted_name = re.findall('(?<=by_)([a-zA-Z0-9]+)[^.html]*', starting_url)
selected_names.append(extracted_name)
# Just for testing purposes
print(selected_name) [['Joe']]
Hmm, a bit odd didn't expect a nested list, but I know how to flatten a list, so ok. Let's go on.
I work my way through a couple of loops, opening each url for the next level down by using:
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')
tags = soup('a')
Continue processing and, in the loop where the program should have found the href I want:
# Testing to check I have found the correct href
print(desired_link) <a href="http://blah_blah_blah_by_Mary.html">blah
blah</a>
type(desired_link) bs4.element.tag
Correct link, but a "type" new to me and not something I can use re.findall on. So more research and I have found:
for link in soup.find_all('a') :
tags = link.get('href')
type(tags) str
print(tags)
http://blah_blah_blah_by_George.html
http://blah_blah_blah_by_Bill.html
http://blah_blah_blah_by_Mary.html
etc.
Right type, but when I look at what printed, I think what I am looking at is maybe just one long string? And I need a way to just assign the third href in the string to a variable that I can use in re.findall('regex expression', desired_link).
Time to ask for help, I think.
And, while we are at it, any ideas about why I get the nested list the first time I used re.findall with the regex?
Please let me know how to improve this question so it is clearer what I've done and what I'm looking for (I KNOW you guys will, without me even asking).
You've printed every link on the page. But each time in the loop tags contains only one of them (you can print len(tags) to validate it easily).
Also I suggest replacing [a-zA-Z0-9]+ with \w+ - it will catch letters, numbers and underscores and is much cleaner.
What i need is to find all the links on the page that have some keyword inside the link itself. So , based on some stack topics i build my xpath as follows:
response.xpath('//a[contains(#href,'/Best-Sellers-Health-Personal-Care')]')
which should return a link like = "https://www.amazon.com/Best-Sellers-Health-Personal-Care-Tongue......"
But i get Invalid syntax error all the time. What am I mistaken?
So what i do now is just add the if contains check when iterating through the list. But hoped there were more elegant and fast solution.
This is because of inconsistent quotes usage.
Just replace
'//a[contains(#href,'/Best-Sellers-Health-Personal-Care')]'
with
'//a[contains(#href,"/Best-Sellers-Health-Personal-Care")]'
I asked my previous question here:
Xpath pulling number in table but nothing after next span
This worked and i managed to see the number i wanted in a firefox plugin called xpath checker. the results show below.
so I know i can find this number with this xpath, but when trying to run a python scrpit to find and save the number it says it cannot find it.
try:
views = browser.find_element_by_xpath("//div[#class='video-details-inside']/table//span[#class='added-time']/preceding-sibling::text()")
except NoSuchElementException:
print "NO views"
views = 'n/a'
pass
I no that pass is not best practice but i am just testing this at the moment trying to find the number. I'm wondering if i need to change something on the end of the xpath like .text as the xpath checker normally shows a results a little differently. Like below:
i needed to use the xpath i gave rather than the one used in the above picture because i only want the number and not the date. You can see part of the source in my previous question.
Thanks in advance! scratching my head here.
The xpath used in find_element_by_xpath() has to point to an element, not a text node and not an attribute. This is a critical thing here.
The easiest approach here would be to:
get the td's text (parent)
get the span's text (child)
remove child's text from parent's
Code:
span = browser.find_element_by_xpath("//div[#class='video-details-inside']/table//span[#class='added-time']")
td = span.find_element_by_xpath('..')
views = td.text.replace(span.text, '').strip()
I haven't used RegEx before, and everyone seems to agree that it's bad for webscraping and html in particular, but I'm not really sure how to solve my little challenge without.
I have a small Python scraper that opens 24 different webpages. In each webpage, there's links to other webpages. I want to make a simple solution that gets the links that I need and even though the webpages are somewhat similar, the links that I want are not.
The only common thing between the urls seems to be a specific string: 'uge' or 'Uge' (uge means week in Danish - and the week number changes every week, duh). It's not like the urls have a common ID or something like that I could use to target the correct ones each time.
I figure it would be possible using RegEx to go through the webpage and find all urls that has 'uge' or 'Uge' in them and then open them. But is there a way to do that using BS? And if I do it using RegEx, how would a possible solution look like?
For example, here are two of the urls I want to grab in different webpages:
http://www.domstol.dk/KobenhavnsByret/retslister/Pages/Uge45-Tvangsauktioner.aspx
http://www.domstol.dk/esbjerg/retslister/Pages/Straffesageruge32.aspx
This should work... The RegEx uge\d\d? tells it to find "uge" followed by a digit, and possibly another one.
import re
for item in listofurls:
l = re.findall("uge\d\d?", item, re.IGNORECASE):
if l:
print item #just do whatever you want to do when it finds it
Yes, you can do this with BeautifulSoup.
from BeautifulSoup import BeautifulSoup
soup = BeautifulSoup(html_string)
# To find just 'Uge##' or 'uge##', as specified in the question:
urls = [el["href"] for el in soup.findAll("a", href=re.compile("[Uu]ge\d+"))]
# To find without regard to case at all:
urls = [el["href"] for el in soup.findAll("a", href=re.compile("(?i)uge\d+"))]
Or just use a simple for loop:
list_of_urls = ["""LIST GOES HERE"""]
for url in list_of_urls:
if 'uge' in url.lower():
# Code to execute
The regex expression would look something like: uge\d\d