Develop Reference

How can I use regular expressions to extract all words with at least one digit in text with Python

I am new to regular expressions and I have a text as follows. How can I use the RegEx to extract all words with at least one digit in it? Really appreciate it.
text = '''The start of the Civil War in 1861 followed by Tennessee’s secession from the Union and the lodging of
wounded Confederate soldiers on campus did not close East Tennessee University. By spring 1862 when the
trustees finally suspended operations, the majority of students had joined the military, President Joseph
Ridley had resigned, and two professors had left the university. Wounded Confederate soldiers were lodged
at university buildings after the January 1862 Battle of Mill Springs in Kentucky, known as the Battle of
Fishing Creek to the Confederacy. In the fall of 1863, Union troops forced the Confederates out of
Knoxville. On the Hill, the Union Army enclosed the three university buildings with an earthen
fortification they named Fort Byington in honor of an officer from Michigan who was killed in the defense
of Knoxville. They used the buildings for their headquarters, barracks, and a hospital for Black troops.
Despite a Confederate attempt to retake the city by siege—climaxed by a bloody, abortive attack on Fort
Sanders on November 29, 1863—the Union held and occupied Knoxville for the rest of the war. During the
battle, the Hill was hit with artillery fire from Confederate guns located in a trench at the site of
UT’s present-day Sorority Village. Campus also sustained a great deal of damage caused by the Union Army.
Troops denuded the grounds of trees, ruined the steward’s house, and destroyed the gymnasium with
misdirected cannon fire aimed at Confederate troops across the river. After the Civil War ended in 1865
and the Union Army left campus, Thomas Humes was elected university president. The university reopened in
1866 and operated for six months downtown in the Deaf and Dumb Asylum while repairs began at the damaged
campus. A petition to the federal war department for monetary compensation for campus damage done by the
Union Army undoubtedly received more favorable consideration because of Humes’s known Union loyalty
throughout the war. A Senate committee which considered the bill for damages also noted that East
Tennessee University was “particularly deserving of the favorable consideration of Congress” because it
was “the only educational institution of known loyalty…in any of the seceding states.” However in 1873,
President Ulysses S. Grant vetoed the bill that would have provided $18,500 to the university because he
felt it would set a bad precedent. The bill was redrafted specifying that the payment was compensation
for aid East Tennessee University gave to the Union during the war. On June 22, 1874, President Grant
signed the new bill and the trustees accepted the funds the same day with an agreement to release the
government from all claims. (More than a century and a half later, a buried Union trench was located in
2019 on the north side of the present-day McClung Museum with the use of ground-penetrating radar.)
'''

You could use this pattern:
'\w*\d+\w*'
How does it work:
\w* matches 0 or more characters (but not space)
\d+ matches 1 or more digits
\w* matches 0 or more characters again
Using re and findall we get:
re.findall('\w*\d+\w*',your_text)
we get:
['1861',
'1862',
'1862',
'1863',
'29',
'1863',
'1865',
'1866',
'1873',
'18',
'500',
'22',
'1874',
'2019']

Is this what you mean?
re.findall(r"\S*\d+\S*", text)
\S any character but a space,
\d any digit,
+ one or more occurrences,
* zero or more occurrences

Get rid of weird indents when getting description in beautiful soup

I have a bs4 program where I collect the descriptions of links. It first checks to see if there are any meta description tags and if there aren't any it gets the descriptions from tags.
This is the code:
from bs4 import BeautifulSoup
import requests
def find_title(url):
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
with open('descrip.txt', 'a', encoding='utf-8') as f:
description = soup.find('meta', attrs={'name':'og:description'}) or soup.find('meta', attrs={'property':'description'}) or soup.find('meta', attrs={'name':'description'})
if description:
desc = description["content"]
else:
desc = soup.find_all('p')[0].getText()
lengths = len(desc)
index = 0
while lengths == 1:
index = index + 1
desc = soup.find_all('p')[index].getText()
lengths = len(desc)
if lengths > 300:
desc = soup.find_all('p')[index].getText()[0:300]
elif lengths < 300:
desc = soup.find_all('p')[index].getText()[0:lengths]
print(desc)
f.write(desc + '\n')
find_title('https://en.wikipedia.org/wiki/Portal:The_arts')
find_title('https://en.wikipedia.org/wiki/Portal:Biography')
find_title('https://en.wikipedia.org/wiki/Portal:Geography')
find_title('https://en.wikipedia.org/wiki/November_15')
find_title('https://en.wikipedia.org/wiki/November_16')
find_title('https://en.wikipedia.org/wiki/Wikipedia:Selected_anniversaries/November')
find_title('https://lists.wikimedia.org/mailman/listinfo/daily-article-l')
find_title('https://en.wikipedia.org/wiki/List_of_days_of_the_year')
find_title('https://en.wikipedia.org/wiki/File:Proclama%C3%A7%C3%A3o_da_Rep%C3%BAblica_by_Benedito_Calixto_1893.jpg')
find_title('https://en.wikipedia.org/wiki/First_Brazilian_Republic')
find_title('https://en.wikipedia.org/wiki/Empire_of_Brazil')
find_title('https://en.wikipedia.org/wiki/Pedro_II_of_Brazil')
find_title('https://en.wikipedia.org/wiki/Benedito_Calixto')
find_title('https://en.wikipedia.org/wiki/Rio_de_Janeiro')
find_title('https://en.wikipedia.org/wiki/Deodoro_da_Fonseca')
But the output to descrip.txt has some weird indents and some descriptions go in for multiple lines and there are spaces between some
This is the output:
The arts refers to the theory, human application and physical expression of creativity found in human cultures and societies through skills and imagination in order to produce objects, environments and experiences. Major constituents of the arts include visual arts (including architecture, ceramics,
A biography, or simply bio, is a detailed description of a person's life. It involves more than just the basic facts like education, work, relationships, and death; it portrays a person's experience of these life events. Unlike a profile or curriculum vitae (résumé), a biography presents a subject's
Geography (from Greek: γεωγραφία, geographia, literally "earth description") is a field of science devoted to the study of the lands, features, inhabitants, and phenomena of the Earth and planets. The first person to use the word γεωγραφία was Eratosthenes (276–194 BC). Geography is an all-encompass
November 15 is the 319th day of the year (320th in leap years) in the Gregorian calendar. 46 days remain until the end of the year.
November 16 is the 320th day of the year (321st in leap years) in the Gregorian calendar. 45 days remain until the end of the year.
The arts refers to the theory, human application and physical expression of creativity found in human cultures and societies through skills and imagination in order to produce objects, environments and experiences. Major constituents of the arts include visual arts (including architecture, ceramics,
A biography, or simply bio, is a detailed description of a person's life. It involves more than just the basic facts like education, work, relationships, and death; it portrays a person's experience of these life events. Unlike a profile or curriculum vitae (résumé), a biography presents a subject's
Geography (from Greek: γεωγραφία, geographia, literally "earth description") is a field of science devoted to the study of the lands, features, inhabitants, and phenomena of the Earth and planets. The first person to use the word γεωγραφία was Eratosthenes (276–194 BC). Geography is an all-encompass
November 15 is the 319th day of the year (320th in leap years) in the Gregorian calendar. 46 days remain until the end of the year.
November 16 is the 320th day of the year (321st in leap years) in the Gregorian calendar. 45 days remain until the end of the year.
Selected anniversaries / On this day archive
All · January · February · March · April · May · June · July · August · September · October · November · December
The sum of all human knowledge. Delivered to your inbox every day.
The following pages list the historical events, births, deaths, and holidays and observances of the specified day of the year:
Original file ‎(5,799 × 3,574 pixels, file size: 15.11 MB, MIME type: image/jpeg)
The First Brazilian Republic or República Velha (Portuguese pronunciation: [ʁeˈpublikɐ ˈvɛʎɐ], "Old Republic"), officially the Republic of the United States of Brazil, refers to the period of Brazilian history from 1889 to 1930. The República Velha ended with the Brazilian Revolution of 1930 that installed Getúlio Vargas as a new president.
The Empire of Brazil was a 19th-century state that broadly comprised the territories which form modern Brazil and (until 1828) Uruguay. Its government was a representative parliamentary constitutional monarchy under the rule of Emperors Dom Pedro I and his son Dom Pedro II. A colony of the Kingdom of Portugal, Brazil became the seat of the Portuguese colonial Empire in 1808, when the Portuguese Prince regent, later King Dom João VI, fled from Napoleon's invasion of Portugal and established himself and his government in the Brazilian city of Rio de Janeiro. João VI later returned to Portugal, leaving his eldest son and heir, Pedro, to rule the Kingdom of Brazil as regent. On 7 September 1822, Pedro declared the independence of Brazil and, after waging a successful war against his father's kingdom, was acclaimed on 12 October as Pedro I, the first Emperor of Brazil. The new country was huge, sparsely populated and ethnically diverse.
Early life (1825–40)
Consolidation (1840–53)
Growth (1853–64)
Paraguayan War (1864–70)
Apogee (1870–81)
Decline and fall (1881–89)
Exile and death (1889–91)
Legacy
Benedito Calixto de Jesus (14 October 1853 – 31 May 1927) was a Brazilian painter.[1] His works usually depicted figures from Brazil and Brazilian culture, including a famous portrait of the bandeirante Domingos Jorge Velho in 1923,[2] and scenes from the coastline of São Paulo.[3] Unlike many artis
Rio de Janeiro (/ˈriːoʊ di ʒəˈnɛəroʊ, - deɪ -, - də -/; Portuguese: [ˈʁi.u d(ʒi) ʒɐˈne(j)ɾu] (listen);[3]), or simply Rio,[4] is anchor to the Rio de Janeiro metropolitan area and the second-most populous municipality in Brazil and the sixth-most populous in the Americas. Rio de Janeiro is the capit
Manuel Deodoro da Fonseca (Portuguese pronunciation: [mɐnuˈɛw deoˈdɔɾu da fõˈsekɐ]; 5 August 1827 – 23 August 1892) was a Brazilian politician and military officer who served as the first President of Brazil. He took office after heading a military coup that deposed Emperor Pedro II and proclaimed t
is there any way to fix this problem?

Add strip=True to getText() (note: it’s an alias of get_text()), and than add a space as a separator. For example:
get_text(strip=True, separator=' ')

How to improve NLTK sentence segmentation?

Given the paragraph from Wikipedia:
An ambitious campus expansion plan was proposed by Fr. Vernon F.
Gallagher in 1952. Assumption Hall, the first student dormitory, was
opened in 1954, and Rockwell Hall was dedicated in November 1958,
housing the schools of business and law. It was during the tenure of
F. Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to
action.
I run NLTK nltk.sent_tokenize to get the sentences. This returns:
['An ambitious campus expansion plan was proposed by Fr.',
'Vernon F. Gallagher in 1952.',
'Assumption Hall, the first student dormitory, was opened in 1954, and Rockwell Hall was dedicated in November 1958, housing the schools of business and law.',
'It was during the tenure of Fr.',
'Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to action.'
]
While NTLK could handle F. Henry J. McAnulty as one entity,
It failed for Fr. Vernon F. Gallagher, and this broke the sentence into two.
The correct tokenization should be:
[
'An ambitious campus expansion plan was proposed by Fr. Vernon F. Gallagher in 1952.',
'Assumption Hall, the first student dormitory, was opened in 1954, and Rockwell Hall was dedicated in November 1958, housing the schools of business and law.',
'It was during the tenure of Fr. Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to action.'
]
How can I improve the tokenizer performance?

The awesome-ness of Kiss and Strunk (2006) Punkt algorithm is that it's unsupervised. So given a new text, you should retrain the model and apply the model to your text, e.g.
>>> from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
>>> text = "An ambitious campus expansion plan was proposed by Fr. Vernon F. Gallagher in 1952. Assumption Hall, the first student dormitory, was opened in 1954, and Rockwell Hall was dedicated in November 1958, housing the schools of business and law. It was during the tenure of F. Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to action."
# Training a new model with the text.
>>> tokenizer = PunktSentenceTokenizer()
>>> tokenizer.train(text)
<nltk.tokenize.punkt.PunktParameters object at 0x106c5d828>
# It automatically learns the abbreviations.
>>> tokenizer._params.abbrev_types
{'f', 'fr', 'j'}
# Use the customized tokenizer.
>>> tokenizer.tokenize(text)
['An ambitious campus expansion plan was proposed by Fr. Vernon F. Gallagher in 1952.', 'Assumption Hall, the first student dormitory, was opened in 1954, and Rockwell Hall was dedicated in November 1958, housing the schools of business and law.', "It was during the tenure of F. Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to action."]
Where there's not enough data to generate good statistics when re-training the model, you can also put in a pre-determined list of abbreviations before training; see How to avoid NLTK's sentence tokenizer spliting on abbreviations?
>>> from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
>>> punkt_param = PunktParameters()
>>> abbreviation = ['f', 'fr', 'k']
>>> punkt_param.abbrev_types = set(abbreviation)
>>> tokenizer = PunktSentenceTokenizer(punkt_param)
>>> tokenizer.train(text)
<nltk.tokenize.punkt.PunktParameters object at 0x106c5d828>
>>> tokenizer.tokenize(text)
['An ambitious campus expansion plan was proposed by Fr. Vernon F. Gallagher in 1952.', 'Assumption Hall, the first student dormitory, was opened in 1954, and Rockwell Hall was dedicated in November 1958, housing the schools of business and law.', "It was during the tenure of F. Henry J. McAnulty that Fr. Gallagher's ambitious plans were put to action."]

Python, keep only sentence from html and remove all non-alphabet, number

I got html from the website and change it to txt.
However, how to clean the txt so that i keep only the sentences in the txt.
for example: I want to remove all irrelevent information such as 1990...himself,1987, the 59th ....
keep the sentences:
tom cruise is an american actor who has starred in many blockbuster movies and as of 2012 is the highest paid actor in hollywood. he is also a film producer and owns a production company. tom cruise has been the winner of three golden globe awards and has been nominated thrice for academy awards. apart from this, many of the movies cruise has starred in have been huge blockbusters on the box office.
after repeated success in many films, tom cruise kept going on with release of two mission impossible movies, war of the worlds which was a super duper box office hit and many more.
and so on.
1990
... himself
1987
the 59th annual academy awards
(tv special)
jack
/
maverick
/
vincent lauria
(uncredited)
related videos
none
none
none
see all 35 videos  »
#csm.csm_widget />
reality tv
the office
late night
sitcoms
music
rappers
action
religion
top paid
how much money does tom cruise make? (salary & net worth)
tom cruise is an american actor who has starred in many blockbuster movies and as of 2012 is the highest paid actor in hollywood. he is also a film producer and owns a production company. tom cruise has been the winner of three golden globe awards and has been nominated thrice for academy awards. apart from this, many of the movies cruise has starred in have been huge blockbusters on the box office.
history
thomas cruise mapother iv a.k.a tom cruise was born in syracuse, new york to mother mary lee and father thomas cruise mapother iii. cruise’s mother was a special education teacher and father was an electrical engineer. tom cruise is basically of irish, german and english origin. cruise’s family had the male domination of his abusive father whom cruise had once described as the merchant of chaos. he was often bullied and beaten by his father and cruise called him a coward. a part of tom cruise’s childhood was spent in canada. however, when cruise was in the sixth grade, his mother left his father and brought cruise and his siblings back to america.
acting career
acting career of tom cruise started quite early but with a small role in the movie endless love (1981). however, he got his big break as a supporting actor in the movie taps later that year. in 1983, his movies risky business and all the right moves along with top gun in 1986 paved the path for tom cruise as an established actor and a superstar. after this there was no looking back and tom cruise went to star in many super-successful movies like cocktail, rain man, days of thunder, interview with the vampire.
then in 1996, he starred as a superspy ethan hunt in the very popular and blockbuster movie which went on to be a series, mission: impossible. that same year he also was seen in the lead role of the movie jerry maguire and won a golden globe for the same. in 1999, his supporting role in the movie magnolia again won him his second golden globe.
after repeated success in many films, tom cruise kept going on with release of two mission impossible movies, war of the worlds which was a super duper box office hit and many more.
net worth
tom cruise’s films have gained $7.3 million worldwide as of 2013. however, the net worth of the highest paid actor in hollywood is $270 million and he still gets paychecks from his previous movies.
154 magazine cover photos
|
none »
official sites:
facebook
|
official site
|
none
»
alternate names:
tomu kurûzu
height:
5' 7" (1.7 m)
none
did you know?
personal quote:
(1992 quote) i really enjoy talking to other actors and directors. sometimes, if i see their movies, i'll call them up or write them a note saying, "i enjoyed it," or asking, "how did you do that? how did you make that work?". i just saw
html text is called: text
sentence = re.sub(' ', '\n', text)
sentence = re.sub('none', '', words)
print sentence
the result: the sentence is destroyed.
ethan
hunt
/
ray
ferrier
(uncredited)
2006
the
late
late
show
with
craig
ferguson
(tv
series)
himself
-
episode
#2.140
(2006)
...
himself
(uncredited)
2006
getaway
(tv
series)
himself
-
seven
wonders
of
the
world
(2006)
...
himself
2006
cmt
insider
(tv
series)
himself
-
episode
dated
29
april
2006
(2006)
...
himself
2005-2006
corazón
de...
(tv
series)
himself
-
episode
dated
19
january
2006
(2006)
...
himself
-
episode
dated
15
november
2005
(2005)
...
himself
-

Try this:
^(\s*?\S*){5}$
The code is currently set to select any line that has five words or less. You can increase/decrease the number of words by changing the value of {5}
Demo: https://regex101.com/r/z2qxrx/3

reformat unstructured text into single line after removing punctuation

I have an unstructured text which I want to convert into 1 line and remove all the punctuation marks.
For the punctuation marks i used the following solution Best way to strip punctuation from a string in Python
How can i reformat the unstructured text into 1 line by using python?
Example 1:
The Bourne Identity is a 2002 spy film loosely based on Robert
Ludlum's novel of the same name. It stars Matt Damon as Jason Bourne,
an amnesiac attempting to discover his true identity amidst a
clandestine conspiracy within the Central Intelligence Agency (CIA) to
track him down and arrest or kill him for inexplicably failing to
carry out an officially unsanctioned assassination and then failing to
report back in afterwards. Along the way he teams up with Marie,
played by Franka Potente, who assists him on the initial part of his
journey to learn about his past and regain his memories. The film also
stars Chris Cooper as Alexander Conklin, Clive Owen as The Professor,
Brian Cox as Ward Abbott, and Julia Stiles as Nicky Parsons.
The film was directed by Doug Liman and adapted for the screen by Tony
Gilroy and William Blake Herron from the novel of the same name
written by Robert Ludlum, who also produced the film alongside Frank
Marshall. Universal Studios released the film to theaters in the
United States on June 14, 2002 and it received a positive critical and
public reaction. The film was followed by a 2004 sequel, The Bourne
Supremacy, and a third part released in 2007 entitled The Bourne
Ultimatum.
Plot
Example 2:
12 (0) 0 4 (0) 38 (3) 0 3 (0) 0 1 (0)
Example 3:
Franklin Township is one of the eighteen townships of Monroe County, Ohio,
United States. The 2000 census found 453 people in the township, 367 of whom
lived in the unincorporated portions of the township.
Geography
Located in the western part of the county, it borders the following townships:
The village of Stafford lies in southwestern Franklin Township.
Name and history
It is one of twenty-one Franklin Townships statewide.
Government
The township is governed by a three-member board of trustees, who are elected in
November of odd-numbered years to a four-year term beginning on the following
January 1. Two are elected in the year after the presidential election and one
is elected in the year before it. There is also an elected township clerk, who
serves a four-year term beginning on April 1 of the year after the election,
which is held in November of the year before the presidential election.
Vacancies in the clerkship or on the board of trustees are filled by the
remaining trustees.
As you can see in the previous examples. The text have different formats. How can I turn every single text into 1 line?

This is pretty straight forward - basically, other than the punctuation, you are now also looking to eliminate the line endings.
So, you can simply do:
import string
exclude = set(string.punctuation + "\n\t\r")
print ''.join(ch for ch in input_string if ch not in exclude)
input_string = """The Bourne Identity is a 2002 spy film loosely based on Robert Ludlum's novel of the same name. It stars Matt Damon as Jason Bourne, an amnesiac attempting to discover his true identity amidst a clandestine conspiracy within the Central Intelligence Agency (CIA) to track him down and arrest or kill him for inexplicably failing to carry out an officially unsanctioned assassination and then failing to report back in afterwards. Along the way he teams up with Marie, played by Franka Potente, who assists him on the initial part of his journey to learn about his past and regain his memories. The film also stars Chris Cooper as Alexander Conklin, Clive Owen as The Professor, Brian Cox as Ward Abbott, and Julia Stiles as Nicky Parsons.
The film was directed by Doug Liman and adapted for the screen by Tony Gilroy and William Blake Herron from the novel of the same name written by Robert Ludlum, who also produced the film alongside Frank Marshall. Universal Studios released the film to theaters in the United States on June 14, 2002 and it received a positive critical and public reaction. The film was followed by a 2004 sequel, The Bourne Supremacy, and a third part released in 2007 entitled The Bourne Ultimatum."""
>>> print ''.join(ch for ch in input_string if ch not in exclude)
The Bourne Identity is a 2002 spy film loosely based on Robert Ludlums novel of the same name It stars Matt Damon as Jason Bourne an amnesiac attempting to discover his true identity amidst a clandestine conspiracy within the Central Intelligence Agency CIA to track him down and arrest or kill him for inexplicably failing to carry out an officially unsanctioned assassination and then failing to report back in afterwards Along the way he teams up with Marie played by Franka Potente who assists him on the initial part of his journey to learn about his past and regain his memories The film also stars Chris Cooper as Alexander Conklin Clive Owen as The Professor Brian Cox as Ward Abbott and Julia Stiles as Nicky ParsonsThe film was directed by Doug Liman and adapted for the screen by Tony Gilroy and William Blake Herron from the novel of the same name written by Robert Ludlum who also produced the film alongside Frank Marshall Universal Studios released the film to theaters in the United States on June 14 2002 and it received a positive critical and public reaction The film was followed by a 2004 sequel The Bourne Supremacy and a third part released in 2007 entitled The Bourne Ultimatum