I need to use regex to strip punctuation at the start and end of a word. It seems like regex would be the best option for this. I don't want punctuation removed from words like 'you're', which is why I'm not using .replace().
You don't need regular expression to do this task. Use str.strip with string.punctuation:
>>> import string
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~'
>>> '!Hello.'.strip(string.punctuation)
'Hello'
>>> ' '.join(word.strip(string.punctuation) for word in "Hello, world. I'm a boy, you're a girl.".split())
"Hello world I'm a boy you're a girl"
I think this function will be helpful and concise in removing punctuation:
import re
def remove_punct(text):
new_words = []
for word in text:
w = re.sub(r'[^\w\s]','',word) #remove everything except words and space
w = re.sub(r'_','',w) #how to remove underscore as well
new_words.append(w)
return new_words
If you persist in using Regex, I recommend this solution:
import re
import string
p = re.compile("[" + re.escape(string.punctuation) + "]")
print(p.sub("", "\"hello world!\", he's told me."))
### hello world hes told me
Note also that you can pass your own punctuation marks:
my_punct = ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '.',
'/', ':', ';', '<', '=', '>', '?', '#', '[', '\\', ']', '^', '_',
'`', '{', '|', '}', '~', '»', '«', '“', '”']
punct_pattern = re.compile("[" + re.escape("".join(my_punct)) + "]")
re.sub(punct_pattern, "", "I've been vaccinated against *covid-19*!") # the "-" symbol should remain
### Ive been vaccinated against covid-19
You can remove punctuation from a text file or a particular string file using regular expression as follows -
new_data=[]
with open('/home/rahul/align.txt','r') as f:
f1 = f.read()
f2 = f1.split()
all_words = f2
punctuations = '''!()-[]{};:'"\,<>./?##$%^&*_~'''
# You can add and remove punctuations as per your choice
#removing stop words in hungarian text and english text and
#display the unpunctuated string
# To remove from a string, replace new_data with new_str
# new_str = "My name$## is . rahul -~"
for word in all_words:
if word not in punctuations:
new_data.append(word)
print (new_data)
P.S. - Do the identation properly as per required.
Hope this helps!!
Related
I have punctuation array like this
punctuation_data = [ '=' '+' '_' '-' ')' '(' '*' '&' '^' '%'
'SSSS' 'AAAA' 'wwww' '!' '~' '،']
and i have text to remove punctuation of this text, i use this but its not working
list = [''.join(c for c in original_data if c not in punctuation_data)
for s in list]
Edit: Original post did not delete longer substrings. I included a function that loops through the punctuation data and deletes the substrings.
You need to separate your list by comma. Also, don't use predefined names like list.
This will work:
punctuation_data = [ '=', '+', '_', '-', ')', '(', '*', '&', '^', '%',
'SSSS', 'AAAA', 'wwww', '!', '~', '،']
orig_string = ['3+5=8']
def delete_substrings(orig_sub_string, punctuation_data):
for element_to_delete in punctuation_data:
orig_sub_string = orig_sub_string.replace(element_to_delete, "")
return orig_sub_string
lst = [''.join(c for c in orig_sub_string if c not in punctuation_data) for orig_sub_string in orig_string]
print(lst) #['358']
Since you're trying match a number of strings of varying lengths, it's best to use regex instead. Escape the strings with re.escape first so that they don't get interpreted as special characters in regex:
import re
punctuation_data = [ '=', '+', '_', '-', ')', '(', '*', '&', '^', '%', 'SSSS', 'AAAA', 'wwww', '!', '~', '،']
print(re.sub('|'.join(map(re.escape, punctuation_data)), '', 'abc*xyzAAAA123'))
This outputs:
abcxyz123
this is worked for me
original_data = 'What is hello'
punctuation_data = [ '=' '+' '_' '-' ')' '(' '*' '&' '^'
'%'
'SSSS' 'AAAA' 'wwww' '!' '~' '،']
original_data = original_data.split()
resultwords = [word for word in original_data if
word.lower() not in punctuation_data]
result = ' '.join(resultwords)
print result
Checking palindrome
I am new to python. But I did the debugging. But couldn't find the error.
import string
def is_palindrome(str_1, lowest_index, highest_index):
punct = set(string.punctuation)
print(punct)
#remove punctuations
no_punct = ""
for char in str_1:
if char not in punct:
no_punct = no_punct + char
print(no_punct)
# rmv_whtspc = no_punct.rstrip()
rmv_whtspc = no_punct.replace(' ','')
print(rmv_whtspc)
str_2 = rmv_whtspc.lower()
print(str_2)
if lowest_index > highest_index:
return True
else:
if str_2[lowest_index] == str_2[highest_index]:
return is_palindrome(str_2, lowest_index+1, highest_index-1)
else:
return False
Calling the function:
str_1 = "Madama I am adam"
lowest_index = 0
highest_index = len(str_1)-1
print(is_palindrome(str_1, lowest_index, highest_index))
The Output:
{'{', '<', '_', '$', '"', ',', '&', '\\', ']', '`', '%', "'", '#', '*', '+', '>', '/', '?', '=', '^', ')', '[', '(',
'~', '!', '#', '|', '}', ':', '.', ';', '-'}
Madama I am adam
MadamaIamadam
madamaiamadam
Traceback (most recent call last):
File "recursion_5_2problem.py", line 27, in <module>
print(is_palindrome(str_1, lowest_index, highest_index))
File "recursion_5_2problem.py", line 19, in is_palindrome
if str_2[lowest_index] == str_2[highest_index]:
IndexError: string index out of range
You are getting the lowest and highest index before you clean the string (removing punctuation and whitespace). So you are trying to access a character in the string that may now be out of bounds.
I'd suggest maybe cleaning the string before putting it through the palindrome function then getting the lowest and highest index in the function itself (aka. after all the punctuation and whitespace is removed).
def clean_string()
# remove punctuation
# remove whitespace
return clean_string
def is_palindrome()
# set high/low index
# do your thing
return result
to_check = "race car!"
cleaned = clean_string(to_check)
print(is_palindrome(cleaned))
Just pseudocode, but I'm sure you get the point!
Hope it helps! :)
the mistake you made is nicely described in Andrew Grass's answer.
here a suggestion how you could make all this a lot simpler:
for the cleanup you could use str.maketrans and str.translate; then you just compare the first half of the string to the second half (in reverse):
from string import punctuation, whitespace
repl_table = str.maketrans("", "", punctuation + whitespace)
def normalize(strg):
# remove all punctuation and whitespace and lowercase strg
return strg.translate(repl_table).lower()
def ispalindrome(strg):
n2 = len(strg) // 2
return strg[:n2] == "".join(reversed(strg))[0:n2]
you could use that then as:
strg = "Madama I am adam"
strg = normalize(strg) # madamaiamadam
print(ispalindrome(strg)) # True
This question already has answers here:
Remove punctuation from Unicode formatted strings
(4 answers)
Closed 6 years ago.
This is my string:
mystring = "How’s it going?"
This is what i did:
import string
exclude = set(string.punctuation)
def strip_punctuations(mystring):
for c in string.punctuation:
new_string=''.join(ch for ch in mystring if ch not in exclude)
new_string = chat_string.replace("\xe2\x80\x99","")
new_string = chat_string.replace("\xc2\xa0\xc2\xa0","")
return chat_string
OUTPUT:
If i did not include this line new_string = chat_string.replace("\xe2\x80\x99","") this will be the output:
'How\xe2\x80\x99s it going'
i realized
exclude does not have that weird looking apostrophe in the list:
print set(exclude)
set(['!', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', ';', ':', '=', '<', '?', '>', '#', '[', ']', '\\', '_', '^', '`', '{', '}', '|', '~'])
How do i ensure all such characters are taken out instead of manually replacing them in the future?
If you are working with long texts like news articles or web scraping, then you can either use "goose" or "NLTK" python libraries. These two are not pre-installed. Here are the links to the libraries. goose, NLTK
You can go through the document and learn how to do.
OR
if you don't want to use these libraries, you may want to create your own "exclude" list manually.
import re
toReplace = "how's it going?"
regex = re.compile('[!#%$\"&)\'(+*-/.;:=<?>#\[\]_^`\{\}|~"\\\\"]')
newVal = regex.sub('', toReplace)
print(newVal)
The regex matches all the characters you've set and it replaces them with empty whitespace.
I want to extract all punctuation tokens from a text with the re module. How can I do this?
>>> text = "ABCDEFGHIJKLMNOPQRSTUVWXYZ!##$%^&*()"
>>> from string import punctuation
>>> for p in punctuation:
... if p in text:
... print p
...
It will print all punctuation characters from text.
!
#
$
%
&
(
)
*
#
^
OR
>>> text = "ABCDEFGHIJKLMNOPQRSTUVWXYZ!##$%^&*()"
>>> [char for char in punctuation if char in text]
['!', '#', '$', '%', '&', '(', ')', '*', '#', '^']
I don't know how to do it with the re module, but you can use list-comprehension:
from string import punctuation
old_string = "This, by the way, has some punctuation!"
new_string = "".join(char for char in old_string if char in punctuation)
print(new_string)
#,,!
Creating a Python program that converts the string to a list, uses a loop to remove any punctuation and then converts the list back into a string and prints the sentence without punctuation.
punctuation=['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
str=input("Type in a line of text: ")
alist=[]
alist.extend(str)
print(alist)
#Use loop to remove any punctuation (that appears on the punctuation list) from the list
print(''.join(alist))
This is what I have so far. I tried using something like: alist.remove(punctuation) but I get an error saying something like list.remove(x): x not in list. I didn't read the question properly at first and realized that I needed to do this by using a loop so I added that in as a comment and now I'm stuck. I was, however, successful in converting it from a list back into a string.
import string
punct = set(string.punctuation)
''.join(x for x in 'a man, a plan, a canal' if x not in punct)
Out[7]: 'a man a plan a canal'
Explanation: string.punctuation is pre-defined as:
'!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~'
The rest is a straightforward comprehension. A set is used to speed up the filtering step.
I found a easy way to do it:
punctuation = ['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
str = raw_input("Type in a line of text: ")
for i in punctuation:
str = str.replace(i,"")
print str
With this way you will not get any error.
punctuation=['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
result = ""
for character in str:
if(character not in punctuation):
result += character
print result
Here is the answer of how to tokenize the given statements by using python. the python version I used is 3.4.4
Assume that I have text which is saved as one.txt. then I have saved my python program in the directory where my file is (i.e. one.txt). The following is my python program:
with open('one.txt','r')as myFile:
str1=myFile.read()
print(str1)# This is to print the given statements with punctuations(before removal of punctuations)
# The following is the list of punctuations that we need to remove, add any more if I forget
punctuation = ['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
for i in punctuation:
str1 = str1.replace(i," ") #to make empty the place where punctuation is there.
myList=[]
myList.extend(str1.split(" "))
print (str1) #this is to print the given statements without puctions(after Removal of punctuations)
for i in myList:
# print ("____________")
print(i,end='\n')
print ("____________")
==============next I will post for you how to remove stop words============
until that let you comment if it is useful.
Thank you