Develop Reference

Joining output of iterative list in a while loop [duplicate]

This question already has answers here:
What is the purpose of the return statement? How is it different from printing?
(15 answers)
How can I use `return` to get back multiple values from a loop? Can I put them in a list?
(2 answers)
Closed 5 months ago.
I have a text file that I'm extracting text from using its punctuation and indentation patterns. The output should be a list of lists combining two lists; company_name and description
[[company,description],[company,description]]
To do that I'm running a while loop nested within a for loop to extract the description for each company. Here's my code
for line in file:
if not re.search(r" ", line, re.MULTILINE):
name = line.split(',', 1)[0]
companies.append(name)
print(companies)
companies = []
while re.search(r" ", line, re.MULTILINE):
desc.append(line)
print(desc)
desc = []
break
Sample from text file:
XYZ Group, a nearly nine-year-old, Copenhagen-based
company that has built a dual-purpose platform, providing both
accountancy software and a marketplace for small and medium
businesses to find accountants, has landed $73 million in growth funding from a single investor,
Lugard Road Capital. TechCrunch has more
here.
Black Lake, a nearly five-year-old, China-based software
platform for factory workers to log their daily tasks and managers
to oversee the plant floor, recently raised $77 million in funding,
including from Singapore’s sovereign wealth fund Temasek,
which led the round, as well as China
Renaissance and Lightspeed Venture
Partners. The outfit has now raised more than $100
million altogether, including from from GGV...
That's the output:
['XYZ Group']
[' company that has built a dual-purpose platform, providing both']
[' accountancy software and a marketplace for small and medium']
[' businesses to find accountants, has landed 73 million in growth funding from a single investor,']
[' Lugard Road Capital TechCrunch has more']
[' here']
['Black Lake']
[' platform for factory workers to log their daily tasks and managers']
[' to oversee the plant floor, recently raised 77 million in funding,']
[' including from Singapore’s sovereign wealth fund Temasek,']
[' which led the round, as well as China']
[' Renaissance and Lightspeed Venture']
[' Partners The outfit has now raised more than 100']
[' million altogether, including from from GGV']
[' Capital, Bertelsmann Asia Investments,']
[' GSR Ventures, ZhenFund']
[' and others TechCrunch has more']
[' here']
The goal is to join the output of desc list under company name into 1 list
Update
I put desc = [] outside of the while loop and I'm getting this:
['XYZ Group']
[' company that has built a dual-purpose platform, providing both']
[' company that has built a dual-purpose platform, providing both', ' accountancy software and a marketplace for small and medium']
[' company that has built a dual-purpose platform, providing both', ' accountancy software and a marketplace for small and medium', ' businesses to find accountants, has landed 73 million in growth funding from a single investor,']
[' company that has built a dual-purpose platform, providing both', ' accountancy software and a marketplace for small and medium', ' businesses to find accountants, has landed 73 million in growth funding from a single investor,', ' Lugard Road Capital TechCrunch has more']
[' company that has built a dual-purpose platform, providing both', ' accountancy software and a marketplace for small and medium', ' businesses to find accountants, has landed 73 million in growth funding from a single investor,', ' Lugard Road Capital TechCrunch has more', ' here']
I only need the last iteration though

Assuming the text is always following a <company_name>, <description> pattern, a very simple approach based on .split(). Simply split on the first , by limiting the number of splits with maxsplit=1 to get the name and full_description which can be prettified afterwards:
text = "XYZ Group, a nearly nine-year-old, Copenhagen-based company that has built a dual-purpose platform, providing both accountancy software and a marketplace for small and medium businesses to find accountants, has landed $73 million in growth funding from a single investor, Lugard Road Capital. TechCrunch has more here."
name, full_description = text.split(',', 1)
description = [s.strip() for s in full_description.split(',')]
output = [name, description]
print(output)
Output:
['XYZ Group', ['a nearly nine-year-old', 'Copenhagen-based company that has built a dual-purpose platform', 'providing both accountancy software and a marketplace for small and medium businesses to find accountants', 'has landed $73 million in growth funding from a single investor', 'Lugard Road Capital. TechCrunch has more here.']]
Alternatively, you could also use .split(" ") to split on the multiple occurring spaces and ignore any commas.

How can I use regular expressions to extract all words with at least one digit in text with Python

I am new to regular expressions and I have a text as follows. How can I use the RegEx to extract all words with at least one digit in it? Really appreciate it.
text = '''The start of the Civil War in 1861 followed by Tennessee’s secession from the Union and the lodging of
wounded Confederate soldiers on campus did not close East Tennessee University. By spring 1862 when the
trustees finally suspended operations, the majority of students had joined the military, President Joseph
Ridley had resigned, and two professors had left the university. Wounded Confederate soldiers were lodged
at university buildings after the January 1862 Battle of Mill Springs in Kentucky, known as the Battle of
Fishing Creek to the Confederacy. In the fall of 1863, Union troops forced the Confederates out of
Knoxville. On the Hill, the Union Army enclosed the three university buildings with an earthen
fortification they named Fort Byington in honor of an officer from Michigan who was killed in the defense
of Knoxville. They used the buildings for their headquarters, barracks, and a hospital for Black troops.
Despite a Confederate attempt to retake the city by siege—climaxed by a bloody, abortive attack on Fort
Sanders on November 29, 1863—the Union held and occupied Knoxville for the rest of the war. During the
battle, the Hill was hit with artillery fire from Confederate guns located in a trench at the site of
UT’s present-day Sorority Village. Campus also sustained a great deal of damage caused by the Union Army.
Troops denuded the grounds of trees, ruined the steward’s house, and destroyed the gymnasium with
misdirected cannon fire aimed at Confederate troops across the river. After the Civil War ended in 1865
and the Union Army left campus, Thomas Humes was elected university president. The university reopened in
1866 and operated for six months downtown in the Deaf and Dumb Asylum while repairs began at the damaged
campus. A petition to the federal war department for monetary compensation for campus damage done by the
Union Army undoubtedly received more favorable consideration because of Humes’s known Union loyalty
throughout the war. A Senate committee which considered the bill for damages also noted that East
Tennessee University was “particularly deserving of the favorable consideration of Congress” because it
was “the only educational institution of known loyalty…in any of the seceding states.” However in 1873,
President Ulysses S. Grant vetoed the bill that would have provided $18,500 to the university because he
felt it would set a bad precedent. The bill was redrafted specifying that the payment was compensation
for aid East Tennessee University gave to the Union during the war. On June 22, 1874, President Grant
signed the new bill and the trustees accepted the funds the same day with an agreement to release the
government from all claims. (More than a century and a half later, a buried Union trench was located in
2019 on the north side of the present-day McClung Museum with the use of ground-penetrating radar.)
'''

You could use this pattern:
'\w*\d+\w*'
How does it work:
\w* matches 0 or more characters (but not space)
\d+ matches 1 or more digits
\w* matches 0 or more characters again
Using re and findall we get:
re.findall('\w*\d+\w*',your_text)
we get:
['1861',
'1862',
'1862',
'1863',
'29',
'1863',
'1865',
'1866',
'1873',
'18',
'500',
'22',
'1874',
'2019']

Is this what you mean?
re.findall(r"\S*\d+\S*", text)
\S any character but a space,
\d any digit,
+ one or more occurrences,
* zero or more occurrences

Remove breaks in input strings

This is a scrapping from a selenium scrap dump:
['The Quest for Ethical Artificial Intelligence: A Conversation with
Timnit Gebru', 'Mindfulness Self-Care for Students of Color', 'GPA: The Geopolitical landscape of the Olympic and Paralympic Movements', 'Interfaith Discussions', 'Mind the Gap', 'First-Year Arts Board Open House', 'Self-Care Night with CARE and BGLTQ+ Specialty Proctors', 'Drawing Plants & Flowers - Sold Out']
I have to pass this to an algorithm but as you can see, although all of them are perfectly encased within quotes, as the sentence breaks after "conversation with" and this is affecting my input. I tried removing whitespaces, didn't work. Any help will be highly appreciated.

You can't type a string on multiple lines unless you're using triple quotation marks.
i.e. The first item (string) in your list is
['The Quest for Ethical Artificial Intelligence: A Conversation with
Timnit Gebru']
You should keep the string on one line as follows.
['The Quest for Ethical Artificial Intelligence: A Conversation with Timnit Gebru']
Or you could use a triple quotation, as I mentioned.
['''The Quest for Ethical Artificial Intelligence: A Conversation with
Timnit Gebru''']
If you'd like to make your list visible and neat aswell, I'd recommend breaking between commas in the list for example:
['The Quest for Ethical Artificial Intelligence: A Conversation with Timnit Gebru',
'Mindfulness Self-Care for Students of Color',
'GPA: The Geopolitical landscape of the Olympic and Paralympic Movements',
'Interfaith Discussions',
'Mind the Gap',
'First-Year Arts Board Open House',
'Self-Care Night with CARE and BGLTQ+ Specialty Proctors',
'Drawing Plants & Flowers - Sold Out']

You could try
string_list = ['The Quest for Ethical Artificial Intelligence: A Conversation with
Timnit Gebru', 'Mindfulness Self-Care for Students of Color', 'GPA: The Geopolitical landscape of the Olympic and Paralympic Movements', 'Interfaith Discussions', 'Mind the Gap', 'First-Year Arts Board Open House', 'Self-Care Night with CARE and BGLTQ+ Specialty Proctors', 'Drawing Plants & Flowers - Sold Out']
string_list = [str.replace("\n", " ") for str in string_list]

Using function objects to speed up the training

from datasets import load_dataset #Huggingface
from transformers import BertTokenizer #Huggingface:
def tokenized_dataset(dataset):
""" Method that tokenizes each document in the train, test and validation dataset
Args:
dataset (DatasetDict): dataset that will be tokenized (train, test, validation)
Returns:
dict: dataset once tokenized
"""
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
encode = lambda document: tokenizer(document, return_tensors='pt', padding=True, truncation=True)
train_articles = [encode(document) for document in dataset["train"]["article"]]
test_articles = [encode(document) for document in dataset["test"]["article"]]
val_articles = [encode(document) for document in dataset["val"]["article"]]
train_abstracts = [encode(document) for document in dataset["train"]["abstract"]]
test_abstracts = [encode(document) for document in dataset["test"]["abstract"]]
val_abstracts = [encode(document) for document in dataset["val"]["abstract"]]
return {"train": (train_articles, train_abstracts),
"test": (test_articles, test_abstracts),
"val": (val_articles, val_abstracts)}
if __name__ == "__main__":
dataset = load_data("./train/", "./test/", "./val/", "./.cache_dir")
tokenized_data = tokenized_dataset(dataset)
I would like to modify the function tokenized_dataset because it creates a very heavy dictionary. The dataset created by that function will be reused for a ML training. However, dragging that dictionary during the training will slow down the training a lot.
Be aware that document is similar to
[['eleven politicians from 7 parties made comments in letter to a newspaper .',
"said dpp alison saunders had ` damaged public confidence ' in justice .",
'ms saunders ruled lord janner unfit to stand trial over child abuse claims .',
'the cps has pursued at least 19 suspected paedophiles with dementia .'],
['an increasing number of surveys claim to reveal what makes us happiest .',
'but are these generic lists really of any use to us ?',
'janet street-porter makes her own list - of things making her unhappy !'],
["author of ` into the wild ' spoke to five rape victims in missoula , montana .",
"` missoula : rape and the justice system in a college town ' was released april 21 .",
"three of five victims profiled in the book sat down with abc 's nightline wednesday night .",
'kelsey belnap , allison huguet and hillary mclaughlin said they had been raped by university of montana football '
'players .',
"huguet and mclaughlin 's attacker , beau donaldson , pleaded guilty to rape in 2012 and was sentenced to 10 years .",
'belnap claimed four players gang-raped her in 2010 , but prosecutors never charged them citing lack of probable '
'cause .',
'mr krakauer wrote book after realizing close friend was a rape victim .'],
['tesco announced a record annual loss of £ 6.38 billion yesterday .',
'drop in sales , one-off costs and pensions blamed for financial loss .',
'supermarket giant now under pressure to close 200 stores nationwide .',
'here , retail industry veterans , plus mail writers , identify what went wrong .'],
...,
['snp leader said alex salmond did not field questions over his family .',
"said she was not ` moaning ' but also attacked criticism of women 's looks .",
'she made the remarks in latest programme profiling the main party leaders .',
'ms sturgeon also revealed her tv habits and recent image makeover .',
'she said she relaxed by eating steak and chips on a saturday night .']]
So in the dictionary, the keys are just strings, but the values are all list of lists of strings. Instead of having value = list of list of strings, I think it is better to create sort of a list of object function instead of having list of lists of strings. It will make the dictionary much lighter. How can I do that?
EDIT
For me, there is a difference between copying a list of lists of strings and copying a list of objects. Copying an object will simply copy the reference while copying a list of strings will copy everything. So it is much faster to copy the reference instead. This is the point of this question.

Python, keep only sentence from html and remove all non-alphabet, number

I got html from the website and change it to txt.
However, how to clean the txt so that i keep only the sentences in the txt.
for example: I want to remove all irrelevent information such as 1990...himself,1987, the 59th ....
keep the sentences:
tom cruise is an american actor who has starred in many blockbuster movies and as of 2012 is the highest paid actor in hollywood. he is also a film producer and owns a production company. tom cruise has been the winner of three golden globe awards and has been nominated thrice for academy awards. apart from this, many of the movies cruise has starred in have been huge blockbusters on the box office.
after repeated success in many films, tom cruise kept going on with release of two mission impossible movies, war of the worlds which was a super duper box office hit and many more.
and so on.
1990
... himself
1987
the 59th annual academy awards
(tv special)
jack
/
maverick
/
vincent lauria
(uncredited)
related videos
none
none
none
see all 35 videos  »
#csm.csm_widget />
reality tv
the office
late night
sitcoms
music
rappers
action
religion
top paid
how much money does tom cruise make? (salary & net worth)
tom cruise is an american actor who has starred in many blockbuster movies and as of 2012 is the highest paid actor in hollywood. he is also a film producer and owns a production company. tom cruise has been the winner of three golden globe awards and has been nominated thrice for academy awards. apart from this, many of the movies cruise has starred in have been huge blockbusters on the box office.
history
thomas cruise mapother iv a.k.a tom cruise was born in syracuse, new york to mother mary lee and father thomas cruise mapother iii. cruise’s mother was a special education teacher and father was an electrical engineer. tom cruise is basically of irish, german and english origin. cruise’s family had the male domination of his abusive father whom cruise had once described as the merchant of chaos. he was often bullied and beaten by his father and cruise called him a coward. a part of tom cruise’s childhood was spent in canada. however, when cruise was in the sixth grade, his mother left his father and brought cruise and his siblings back to america.
acting career
acting career of tom cruise started quite early but with a small role in the movie endless love (1981). however, he got his big break as a supporting actor in the movie taps later that year. in 1983, his movies risky business and all the right moves along with top gun in 1986 paved the path for tom cruise as an established actor and a superstar. after this there was no looking back and tom cruise went to star in many super-successful movies like cocktail, rain man, days of thunder, interview with the vampire.
then in 1996, he starred as a superspy ethan hunt in the very popular and blockbuster movie which went on to be a series, mission: impossible. that same year he also was seen in the lead role of the movie jerry maguire and won a golden globe for the same. in 1999, his supporting role in the movie magnolia again won him his second golden globe.
after repeated success in many films, tom cruise kept going on with release of two mission impossible movies, war of the worlds which was a super duper box office hit and many more.
net worth
tom cruise’s films have gained $7.3 million worldwide as of 2013. however, the net worth of the highest paid actor in hollywood is $270 million and he still gets paychecks from his previous movies.
154 magazine cover photos
|
none »
official sites:
facebook
|
official site
|
none
»
alternate names:
tomu kurûzu
height:
5' 7" (1.7 m)
none
did you know?
personal quote:
(1992 quote) i really enjoy talking to other actors and directors. sometimes, if i see their movies, i'll call them up or write them a note saying, "i enjoyed it," or asking, "how did you do that? how did you make that work?". i just saw
html text is called: text
sentence = re.sub(' ', '\n', text)
sentence = re.sub('none', '', words)
print sentence
the result: the sentence is destroyed.
ethan
hunt
/
ray
ferrier
(uncredited)
2006
the
late
late
show
with
craig
ferguson
(tv
series)
himself
-
episode
#2.140
(2006)
...
himself
(uncredited)
2006
getaway
(tv
series)
himself
-
seven
wonders
of
the
world
(2006)
...
himself
2006
cmt
insider
(tv
series)
himself
-
episode
dated
29
april
2006
(2006)
...
himself
2005-2006
corazón
de...
(tv
series)
himself
-
episode
dated
19
january
2006
(2006)
...
himself
-
episode
dated
15
november
2005
(2005)
...
himself
-

Try this:
^(\s*?\S*){5}$
The code is currently set to select any line that has five words or less. You can increase/decrease the number of words by changing the value of {5}
Demo: https://regex101.com/r/z2qxrx/3