Develop Reference

string.punctuation fails to remove certain characters from a string

My aim is to remove all punctuations from a string so that I can then get the frequency of each word in the string.
My string is:
WASHINGTON—Coming to the realization in front of millions of viewers
during the broadcast of his show, a horrified Tucker Carlson stated,
‘I…I am the mainstream media’ Wednesday as he began spiraling live on
air. “We’ve discovered evidence of rampant voter fraud, and the
president has every right to call for an investigation even if the
mainstream media thinks...” said Carlson, who trailed off, stared down
at his shaking hands, and felt a sudden ringing in his ears as he
looked back up and zeroed in on the production crew surrounding him.
“The media says…wait. Those liars on TV will try to tell you…oh God.
We’re the number-one program on cable news, aren’t we? Fox News…Fox
‘News.’ It’s the media. It’s me. This can’t be. No, no, no, no. Jesus
Christ, I make $6 million a year. Get that camera off me!” At press
time, Carlson had torn the microphone from his lapel and fled the set
in panic.
source: https://www.theonion.com/i-i-am-the-mainstream-media-realizes-horrified-tuc-1845646901
I want to remove all punctuations from it. I do that like this -
s.translate(str.maketrans('', '', string.punctuation))
This is the output -
WASHINGTON—Coming to the realization in front of millions of viewers
during the broadcast of his show a horrified Tucker Carlson stated
‘I…I am the mainstream media’ Wednesday as he began spiraling live on
air “We’ve discovered evidence of rampant voter fraud and the
president has every right to call for an investigation even if the
mainstream media thinks” said Carlson who trailed off stared down at
his shaking hands and felt a sudden ringing in his ears as he looked
back up and zeroed in on the production crew surrounding him “The
media says…wait Those liars on TV will try to tell you…oh God We’re
the numberone program on cable news aren’t we Fox News…Fox ‘News’ It’s
the media It’s me This can’t be No no no no Jesus Christ I make 6
million a year Get that camera off me” At press time Carlson had torn
the microphone from his lapel and fled the set in panic
As you can see that characters/ string like ", — and ... still exist. Am I incorrectly expecting them to be removed too? If the output is correct then how can I NOT differentiate between "`News`" and "News"?

>>> import string
>>> "“" in string.punctuation
False
>>> "—" in string.punctuation
False
Welcome to the wonderful world of Unicode where, among many other things, … is not three concatenated full stop periods and :
>>> import unicodedata
>>> unicodedata.name('—')
'EM DASH'
is not a hyphen.
How you want to handle the full scope of what could be considered 'punctuation' across the Unicode table is probably out of scope for this question, but you could either come up with your own ad-hoc list or use a third-party library designed for that type of text manipulation. Here is one such approach:
Best way to strip punctuation from a string

I added the list of characters you can remove from string by using your implementation.
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~'
You can check this implementation to remove all special characters and keep whitespaces
''.join(e for e in s if e.isalnum() or e == ' ')

It looks like the … and a couple of the other characters you are having trouble with are special Unicode characters. A workaround is to use string.isalpha(), which tells you whether the characters of a string are part of the alphabet or not.
result = ""
for x in string:
if x.isalpha() or x == " ":
result = result + x

Using regex to split text content into dictionary

I have a text file that follows this format.
LESTER HOLT (00:00:01): Breaking News Tonight: A deadly mass shooting
at the airport. A gunman opens fire at baggage claim in Fort
Lauderdale, witnesses describing scenes of sheer horror. A silent
killer shooting people in the head as they tried to run and hide.
Tonight, a storm of questions. Why did he do it? The suspect, a
passenger with a firearm in his checked bag. New concerns about
airport security before the checkpoint.
(00:00:25): Also breaking tonight the new report from U.S.
intelligence: Vladimir Putin himself ordered the effort to influence
the election, aimed at hurting Clinton and helping Trump win. What the
President-elect is saying after his top-secret briefing.
(00:00:39): And States of Emergency: Millions from coast to coast
paralyzed by a massive winter storm.
(00:00:45): NIGHTLY NEWS begins right now.
I am trying to parse this information into a Python Dictionary, where the speaker is a dictionary, of dictionaries, which has timecode keys, and the content is the value, I can't consistently split because of potential information before the timecode, (IE the first quote), as well as the fact that the split character : is also a character involved with the timecode itself 00:00:00.
Trying to split according to the regex.
for line in msg.get_payload().split('\n'):
regex = r'\d{2}:\d{2}:\d{2}'
test = re.split(regex, line)
print(test)
sleep(1)
Appears to work in splitting it properly, but it causes me to lose the value I am splitting on (timecode), which I intend to use as a key. How can I properly split the above content, get the speaker, and then get the timecode as a key, and the content as a value.It is possible he may be present later in the text as well, and it should append to the list of timecodes./
The output format I am targeting is something along the lines of
{speakers:{'Lester Holt': {'00:00:01':content..., '00:00:0025': content...},
'speaker2':{etc,etc,etc} }}
Ive tried using the split as mentioned above, but it removes my timecode variable.
Any thoughts and guidance is appreciated.

Don't bother with split. You're trying to get 2-3 pieces of information out of each line, so try the following:
for line in msg.get_payload().split('\n'):
match = re.search(r'^\s*([^(]*?)\s*\((\d{2}:\d{2}:\d{2})\):\s*(.*)$', line)
if match:
(speaker, time, message) = match.groups()
Speaker will be an empty string if none was present on that line.
Regex explanation:
^ # Start of line
\s* # Drop leading whitespace
([^(]*?) # Capture the speaker if present (non-paren characters)
\s* # Drop whitespace between name and time
\( # Drop literal open paren
(\d{2}:\d{2}:\d{2}) # Capture time
\):\s* # Drop literal close paren, colon and whitespace
(.*) # Capture the rest of the line
$ # End of line

Splitting message in lines when you need to split it in time-stamped paragraphs is a waste. re.split can easily save the tokens that it split on, if you only look at the documentation. Here's my solution:
toks = re.split(r"\((\d\d:\d\d:\d\d)\):", msg.get_payload())[1:]
answer = dict(zip(toks[::2], toks[1::2]))
This creates a dictionary of timestamps and paragraphs. Feel free to use the same approach to split by speaker as well.
Result:
{
'00:00:01': ' Breaking News Tonight: A .....',
'00:00:25': ' Also breaking tonight ......', ....
}

Python : Word by word Text Processing between two files

I'm new to NLP. I have two text files. First file has dialogues formatted properly like below .
RECEPTIONIST Can I help you?
LINCOLN Yes. Um, is this the State bank?
RECEPTIONIST If you have to ask, maybe you shouldn't be here.
SARAH I think this is the place.
RECEPTIONIST Fill in the query.
LINCOLN Thank-you. We'll be right back.
RECEPTIONIST Oh, take your time. I'll just finish my crossword puzzle.
oh, wait.
The Second text file has 7 columns . In 5th column I have the words sequence from the dialogues of like below .
Column 5
Can
I
help
you
?
yes
.
Um
,
The Full stop and commas are considered as words here and if it has 3 or more full stop's together like "..." then it should be considered as a single word. Also if the words "Thank-you" (because they don't have space in between them) should be considered as a single word.
Now I want to write a script in python to compare each word from the dialogues and then make a new column (Column 8) which should show " who speaks the word " . Like below
Column 5 Column 8
Can RECEPTIONIST
I RECEPTIONIST
help RECEPTIONIST
you RECEPTIONIST
? RECEPTIONIST
yes LINCOLN
. LINCOLN
Um LINCOLN
, LINCOLN
As I'm completely new to python environment. I dont know where to start .Please provide your suggestion and any tips to coding!
The first file has the dialogues and the second file has information about the dialogues

I suggest the following steps to perform:
Process text file 1
here you want to split the string like LEONARD Agreed, what's your pointinto
a set of tokens. A naive approach is to use split(" ") which will split the text based on spaces, however you also need to take in consideration punctuations.
I suggest to use NLTK, a python library for natural language processing. A basic example will show how this might help you:
import nltk
sentence = """Hi this is a test."""
tokens = nltk.word_tokenize(sentence)
# output: tokens
['Hi', 'this', "is", 'a', 'test', '.']
Once you have tokenised each sentence correctly, you will know how many lines it will have in the second text file.
Process text file 2
Now you will iterate over each line in the second text file, you check if the word matches the supposed token which you found in the first step. If this is the case you add the first token (the name of the person who said it) to the end of the line (column 8).
You can get the word from the string TheBigBangTheory.Season01.Episode01.en 1 59.160 0.070 you 0.990 lexby simply doing sentence.split(" ")[4], which returns youin this case.
I believe it will still need some tweaking, but I'll leave that to you. This might outline the general idea.
Goodluck, Bazinga!

Splitting individual sentence to list

I am asking on how to make individual lists. Not how to find a substring as marked duplicated for.
I have the following file
'Gentlemen do not read each others mail.' Henry Stinson
'The more corrupt the state, the more numerous the laws.' Tacitus
'The price of freedom is eternal vigilance.' Thomas Jefferson
'Few false ideas have more firmly gripped the minds of so many intelligent men than the one that, if they just tried, they could invent a cipher that no one could break.' David Kahn
'Who will watch the watchmen.' Juvenal
'Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin.' John Von Neumann
'They that give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.' Benjamin Franklin
'And so it often happens that an apparently ingenious idea is in fact a weakness which the scientific cryptographer seizes on for his solution.' Herbert Yardley
I am trying to convert each sentence to a list so that when I search for the word say "Gentlemen" it should print me the entire sentence.
I am able to get the lines to split but I am unable to convert them to individual list. I have tried a few things from the internet but nothing has helped so far.
here is what
def myFun(filename):
file = open(filename, "r")
c1 = [ line for line in file ]
for i in c1:
print(i)

you can use in to search a string or array, for example 7 in a_list or "I" in "where am I"
you can iterate directly over a file if you want
for line in open("my_file.txt")
although to ensure it closes people recommend using a context manager
with open("my_file.txt") as f:
for line in f:
that should probably at least get you going in the right direction
if you want to search case insensitive you can simply use str.lower()
term.lower() in search_string.lower() #case insensitive

Python strings have a split() method:
individual_words = 'This is my sentence.'.split()
print(len(individual_words)) # 4
Edit: As #ShadowRanger mentions below, running split() without an argument will take care of leading, trailing, and consecutive whitespace.

How to replace all new lines, tabs, and excess whitespace in a text file

I have a text file of a book and I want it read into my python program to split it into sentences by using open("book.txt").read().split(".").
The problem is the file has new line breaks and multiple spaces. I want the file to be only the words separated by a space and all new lines turned into just a single space.
My book.txt is currently like this (a snippet):
To Sherlock Holmes she is always the woman. I have seldom
heard him mention her under any other name. In his eyes she
eclipses and predominates the whole of her sex. It was not that
he felt any emotion akin to love for Irene Adler. All emotions,
and that one particularly, were abhorrent to his cold, precise but
admirably balanced mind. He was, I take it, the most perfect
reasoning and observing machine that the world has seen, but as
a lover he would have placed himself in a false position. He
never spoke of the softer passions, save with a gibe and a sneer.

It sounds like you just want to remove all line breaks and trailing white space...
maybe something like...
import re
sentences = [re.sub("^\s*|\s*$,"",re.sub("\n","",each)) for each in open("book.txt").read().split(".")]
or if tabs are also a problem...
sentences = [re.sub("^\s*|\s*$","",re.sub("\s+"," ",each)) for each in open("book.txt").read().split(".")]
to also split by ?,!, or . use...
sentences = [re.sub("^\s*|\s*$","",re.sub("\s+"," ",each)) for each in re.split("[\?\.!]",open("book.txt").read())]