Develop Reference

BeautifulSoup trying to remove HTML data from list

As mentioned above, I am trying to remove HTML from the printed output to just get text and my dividing | and -. I get span information as well as others that I would like to remove. As it is part of the program that is a loop, I cannot search for the individual text information of the page as they change. The page architecture stays the same, which is why printing the items in the list stays the same. Wondering what would be the easiest way to clean the output. Here is the code section:
infoLink = driver.find_element_by_xpath("//a[contains(#href, '?tmpl=component&detail=true&parcel=')]").click()
driver.switch_to.window(driver.window_handles[1])
aInfo = driver.current_url
data = requests.get(aInfo)
src = data.text
soup = BeautifulSoup(src, "html.parser")
parsed = soup.find_all("td")
for item in parsed:
Original = (parsed[21])
Owner = parsed[13]
Address = parsed[17]
print (*Original, "|",*Owner, "-",*Address)
Example output is:
<span class="detail-text">123 Main St</span> | <span class="detail-text">Banner,Bruce</span> - <span class="detail-text">1313 Mockingbird Lane<br>Santa Monica, CA 90405</br></span>
Thank you!

To get the text between the tags just use get_text() but you should be aware, that there is always text between the tags to avoid errors:
for item in parsed:
Original = (parsed[21].get_text(strip=True))
Owner = parsed[13].get_text(strip=True)
Address = parsed[17].get_text(strip=True)

I wrote an algorithm recently that does something like this. It won't work if your target text has a < or a > in it, though.
def remove_html_tags(string):
data = string.replace(string[string.find("<"):string.find(">") + 1], '').strip()
if ">" in data or "<" in data:
return remove_html_tags(data)
else:
return str(data)
It recursively removes the text between < and >, inclusive.
Let me know if this works!

BeautifulSoup find_all('href') returns only part of the value

I'm attempting to scrape actor/actress IDs from an IMDB movie page. I only want actors and actresses (I don't want to get any of the crew), and this question is specifically about getting the person's internal ID. I already have peoples' names, so I don't need help getting those. I'm starting with this webpage (https://www.imdb.com/title/tt0084726/fullcredits?ref_=tt_cl_sm#cast) as a hard-coded url to get the code right.
On examination of the links I was able to find that the links for the actors look like this.
William Shatner
Leonard Nimoy
Nicholas Guest
while the ones for other contributors look like this
Nicholas Meyer
Gene Roddenberry
This should allow me to differentiate actors/actresses from crew like the director or writer by checking for the end of the href being "t[0-9]+$" rather than the same but with "dr" or "wr".
Here's the code I'm running.
import urllib.request
from bs4 import BeautifulSoup
import re
movieNumber = 'tt0084726'
url = 'https://www.imdb.com/title/' + movieNumber + '/fullcredits?ref_=tt_cl_sm#cast'
def clearLists(n):
return [[] for _ in range(n)]
def getSoupObject(urlInput):
page = urllib.request.urlopen(urlInput).read()
soup = BeautifulSoup(page, features="html.parser")
return(soup)
def getPeopleForMovie(soupObject):
listOfPeopleNames, listOfPeopleIDs, listOfMovieIDs = clearLists(3)
#get all the tags with links in them
link_tags = soupObject.find_all('a')
#get the ids of people
for linkTag in link_tags:
link = str(linkTag.get('href'))
#print(link)
p = re.compile('t[0-9]+$')
q = p.search(link)
if link.startswith('/name/') and q != None:
id = link[6:15]
#print(id)
listOfPeopleIDs.append(id)
#return the names and IDs
return listOfPeopleNames, listOfPeopleIDs
newSoupObject = getSoupObject(url)
pNames, pIds = getPeopleForMovie(newSoupObject)
The above code returns an empty list for the IDs, and if you uncomment the print statement you can see that it's because the value that gets put in the "link" variable ends up being what's below (with variations for the specific people)
/name/nm0583292/
/name/nm0000638/
That won't do. I want the IDs only for the actors and actresses so that I can use those IDs later.
I've tried to find other answers on stackoverflow; I haven't been able to find this particular issue.
This question (Beautifulsoup: parsing html – get part of href) is close to what I want to do, but it gets the info from the text part between tags rather than from the href part in the tag attribute.
How can I make sure I get only the name IDs that I want (just the actor ones) from the page?
(Also, feel free to offer suggestions to tighten up the code)

It appears that the links you are trying to match have either been modified by JavaScript after loading, or perhaps get loaded differently based on other variables than the URL alone (like cookies or headers).
However, since you're only after links of people in the cast, an easier way would be to simply match the ids of people in the cast section. This is actually fairly straightforward, since they are all in a single element, <table class="cast_list">
So:
import urllib.request
from bs4 import BeautifulSoup
import re
# it's Python, so use Python conventions, no uppercase in function or variable names
movie_number = 'tt0084726'
# The f-string is often more readable than a + concatenation
url = f'https://www.imdb.com/title/{movieNumber}/fullcredits?ref_=tt_cl_sm#cast'
# this is overly fancy for something as simple as initialising some variables
# how about:
# a, b, c = [], [], []
# def clearLists(n):
# return [[] for _ in range(n)]
# in an object-oriented program, assuming something is an object is the norm
def get_soup(url_input):
page = urllib.request.urlopen(url_input).read()
soup = BeautifulSoup(page, features="html.parser")
# removed needless parentheses - arguably, even `soup` is superfluous:
# return BeautifulSoup(page, features="html.parser")
return soup
# keep two empty lines between functions, it's standard and for good reason
# it's easier to spot where a function starts and stops
# try using an editor or IDE that highlights your PEP8 mistakes, like PyCharm
# (that's just my opinion there, other IDEs than PyCharm will do as well)
def get_people_for_movie(soup_object):
# removed unused variables, also 'list_of_people_ids' is needlessly verbose
# since they go together, why not return people as a list of tuples, or a dictionary?
# I'd prefer a dictionary as it automatically gets rid of duplicates as well
people = {}
# (put a space at the start of your comment blocks!)
# get all the anchors tags inside the `cast_list` table
link_tags = soup_object.find('table', class_='cast_list').find_all('a')
# the whole point of compiling the regex is to only have to do it once,
# so outside the loop
id_regex = re.compile(r'/name/nm(\d+)/')
# get the ids and names of people
for link_tag in link_tags:
# the href attributes is a strings, so casting with str() serves no purpose
href = link_tag.get('href')
# matching and extracting part of the match can all be done in one step:
match = id_regex.search(href)
if match:
# don't shadow Python keywords like `id` with variable names!
identifier = match.group(1)
name = link_tag.text.strip()
# just ignore the ones with no text, they're the thumbs
if name:
people[identifier] = name
# return the names and IDs
return people
def main():
# don't do stuff globally, it'll just cause problems when reusing names in functions
soup = get_soup(url)
people = get_people_for_movie(soup)
print(people)
# not needed here, but a good habit, allows you to import stuff without running the main
if __name__ == '__main__':
main()
Result:
{'0000638': 'William Shatner', '0000559': 'Leonard Nimoy', '0001420': 'DeForest Kelley', etc.
And the code with a few more tweaks and without the commentary on your code:
import urllib.request
from bs4 import BeautifulSoup
import re
def get_soup(url_input):
page = urllib.request.urlopen(url_input).read()
return BeautifulSoup(page, features="html.parser")
def get_people_for_movie(soup_object):
people = {}
link_tags = soup_object.find('table', class_='cast_list').find_all('a')
id_regex = re.compile(r'/name/nm(\d+)/')
# get the ids and names of the cast
for link_tag in link_tags:
match = id_regex.search(link_tag.get('href'))
if match:
name = link_tag.text.strip()
if name:
people[match.group(1)] = name
return people
def main():
movie_number = 'tt0084726'
url = f'https://www.imdb.com/title/{movie_number}/fullcredits?ref_=tt_cl_sm#cast'
people = get_people_for_movie(get_soup(url))
print(people)
if __name__ == '__main__':
main()

How to clean HTML removing repeated paragraphs?

I´m trying to clean an html file that has repeated paragraphs within body. Below I show the input file and expected output.
Input.html
https://jsfiddle.net/97ptc0Lh/4/
Output.html
https://jsfiddle.net/97ptc0Lh/1/
I've been trying with the following code using BeautifulSoup but I don´t know why is not working, since the resultant list CleanHtml contains the repeated elements (paragraphs) that I´d like to remove.
from bs4 import BeautifulSoup
fp = open("Input.html", "rb")
soup = BeautifulSoup(fp, "html5lib")
Uniques = set()
CleanHtml = []
for element in soup.html:
if element not in Uniques:
Uniques.add(element)
CleanHtml.append(element)
print (CleanHtml)
May someone help me to reach this goal please.

I think this should do it:
elms = []
for elem in soup.find_all('font'):
if elem not in elms:
elms.append(elem)
else:
target =elem.findParent().findParent()
target.decompose()
print(soup.html)
This should get you your the desired output.
Edit:
To remove only for those paragraphs that have don't size 4 or 5, change the else block to
else:
if elem.attrs['size'] != "4" and elem.attrs['size'] !="5":
target =elem.findParent().findParent()
target.decompose()

In Python, how do I search an html webpage for a set of strings in text file?

I'm trying to figure out a way to search an html webpage for a number of strings that I have written in a text file, each on its own line. The code I have so far is this:
def mangasearch():
r = requests.get("https://www.mangaupdates.com/releases.html")
soup = BeautifulSoup(r.text)
if soup.find(text=re.compile(line)):
print("New chapter found!")
print("")
else:
print("No new chapters")
print("")
def textsearch():
global line
with open("manga.txt") as file:
for line in file:
print(line)
mangasearch()
It's supposed to read manga.txt and search the webpage for each string separately, but it always returns "no new chapters". If I replace if soup.find(text=re.compile(line)): with if soup.find(text=re.compile("An actual string")): it works correctly, but for some reason it doesn't want to use the line variable. Any help would be appreciated!

The problem is that you are trying to search with a string that contains some special character, like ' ', and '\n'.
Note that str.strip() removes ' ' and other whitespace characters as well (e.g. tabs and newlines), so, update the following line:
if soup.find(text=re.compile(line.strip())):

def textsearch():
#manga_file = open("managa.txt").readlines()
##assuming your text file has 3 titles
manga_file = ["Karakuri Circus","Sun-ken Rock","Shaman King Flowers"]
manga_html = "https://www.mangaupdates.com/releases.html"
manga_page = urllib2.urlopen(manga_html)
found = 0
soup = BeautifulSoup(manga_page)
##use BeautifulSoup to parse the section of interest out
##use 'inspect element' in Chrome or Firefox to find the interesting table
##in this case the data you are targeting is in a <div class="alt"> element
chapters = soup.findAll("div", attrs={"class":"alt"})
##chapters now contains all of the sections under the dates on the page
##so we iterate through each date to get all of the titles in each
##<tr> element of each table
for dated_section in chapters:
rows = dated_section.findAll("tr")
for r in rows:
title = r.td.text
#print title
#url = r.td.a.href
if title in manga_file:
found +=1
print "New Chapter Found!"
print r
if found >0:
print "Found a total of %d title"%found
else:
print "No chapters found"
above is not meant to be optimized but it does a good job of showing how BeautifulSoup can be used to parse the specific elements that you are looking for. In this case I used Chrome, right-clicked on the table that contains the titles to "inspect-element' and looked for the element that contains them to direct BeautifulSoup's attention directly to there. The rest is explained in code. I don't know what your managa.txt file looks like so I just created a list of 3 titles to search for as an example.

*Update: How to parse html with python/ beautifulsoup

First, I'm pretty new to Python. I'm trying to scrape contact information from offline websites and output the info to a csv. I'd like to grab the page url(not sure how to do this from the html), email, phone, location data if possible, any names, any phone numbers and the tag line for the html site if it exists.
Updated #2 code:
import os, csv, re
from bs4 import BeautifulSoup
topdir = 'C:\\projects\\training\\html'
output = csv.writer(open("scrape.csv", "wb+"))
output.writerow(["headline", "name", "email", "phone", "location", "url"])
all_contacts = []
for root, dirs, files in os.walk(topdir):
for f in files:
if f.lower().endswith((".html", ".htm")):
soup = BeautifulSoup(f)
def mailto_link(soup):
if soup.name != 'a':
return None
for key, value in soup.attrs:
if key == 'href':
m = re.search('mailto:(.*)',value)
if m:
all_contacts.append(m)
return m.group(1)
return None
for ul in soup.findAll('ul'):
contact = []
for li in soup.findAll('li'):
s = li.find('span')
if not (s and s.string):
continue
if s.string == 'Email:':
a = li.find(mailto_link)
if a:
contact['email'] = mailto_link(a)
elif s.string == 'Website:':
a = li.find('a')
if a:
contact['website'] = a['href']
elif s.string == 'Phone:':
contact['phone'] = unicode(s.nextSibling).strip()
all_contacts.append(contact)
output.writerow([all_contacts])
print "Finished"
This output currently doesn't return anything other than the row headers. What am I missing here? This should be at least returning some info from the html file, which is this page: http://bendoeslife.tumblr.com/about

There are (at least) two problems here.
First, f is a filename, not the file contents, or the Soup made from those contents. So, f.find('h2') is going to find 'h2' within the filename, which isn't very useful.
Second, most find methods (including str.find, which is what you're calling) return an index, not a substring. Calling str on that index is just going to give you the string version of a number. For example:
>>> s = 'A string with an h2 in it'
>>> i = s.find('h2')
>>> str(i)
'17'
So, your code is doing something like this:
>>> f = 'C:\\python\\training\\offline\\somehtml.html'
>>> headline = f.find('h2')
>>> str(headline)
'-1'
You probably want to call methods on the soup object, rather than f. BeautifulSoup.find returns a "sub-tree" of the soup, which is exactly what you want to stringify here.
However, it's impossible to test that without your sample input, so I can't promise that's the only problem in your code.
Meanwhile, when you get stuck with something like this, you should try printing out intermediate values. Print out f, and headline, and headline2, and it will be much more obvious why headline3 is wrong.
Just replacing the f with soup in the find calls, and fixing your indentation error, running against your sample file http://bendoeslife.tumblr.com/about now works.
It doesn't do anything all that useful, however. Since there's no h2 tag anywhere in the file, headline ends up as None. And the same goes for most of the other fields. The only thing that does find anything is url, because you're asking it to find an empty string, which will find something arbitrary. With three different parsers, I get <p>about</p> or <html><body><p>about</p></body></html>, and <html><body></body></html>…
You need to actually understand the structure of the file you're trying to parse before you can do anything useful with it. In this case, for example, there is an email address, but it's in an <a> element with a title of "Email", with an <li> element with an id of "email". So, you need to write a find to locate it based on one of those criteria, or something else it actually matches.