Develop Reference

remove all the special chars from a list [duplicate]

This question already has answers here:
Removing punctuation from a list in python
(2 answers)
Closed last year.
i have a list of strings with some strings being the special characters what would be the approach to exclude them in the resultant list
list = ['ben','kenny',',','=','Sean',100,'tag242']
expected output = ['ben','kenny','Sean',100,'tag242']
please guide me with the approach to achieve the same. Thanks

The string module has a list of punctuation marks that you can use and exclude from your list of words:
import string
punctuations = list(string.punctuation)
input_list = ['ben','kenny',',','=','Sean',100,'tag242']
output = [x for x in input_list if x not in punctuations]
print(output)
Output:
['ben', 'kenny', 'Sean', 100, 'tag242']
This list of punctuation marks includes the following characters:
['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '#', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~']

It can simply be done using the isalnum() string function. isalnum() returns true if the string contains only digits or letters, if a string contains any special character other than that, the function will return false. (no modules needed to be imported for isalnum() it is a default function)
code:
list = ['ben','kenny',',','=','Sean',100,'tag242']
olist = []
for a in list:
if str(a).isalnum():
olist.append(a)
print(olist)
output:
['ben', 'kenny', 'Sean', 100, 'tag242']

my_list = ['ben', 'kenny', ',' ,'=' ,'Sean', 100, 'tag242']
stop_words = [',', '=']
filtered_output = [i for i in my_list if i not in stop_words]
The list with stop words can be expanded if you need to remove other characters.

Python, Split the input string on elements of other list and remove digits from it

I have had some trouble with this problem, and I need your help.
I have to make a Python method (mySplit(x)) which takes an input list (which only has one string as element), split that element on the elements of other list and digits.
I use Python 3.6
So here is an example:
l=['I am learning']
l1=['____-----This4ex5ample---aint___ea5sy;782']
banned=['-', '+' , ',', '#', '.', '!', '?', ':', '_', ' ', ';']
The returned lists should be like this:
mySplit(l)=['I', 'am', 'learning']
mySplit(l1)=['This', 'ex', 'ample', 'aint', 'ea', 'sy']
I have tried the following, but I always get stuck:
def mySplit(x):
l=['-', '+' , ',', '#', '.', '!', '?', ':', '_', ';'] #Banned chars
l2=[i for i in x if i not in l] #Removing chars from input list
l2=",".join(l2)
l3=[i for i in l2 if not i.isdigit()] #Removes all the digits
l4=[i for i in l3 if i is not ',']
l5=[",".join(l4)]
l6=l5[0].split(' ')
return l6
and
mySplit(l1)
mySplit(l)
returns:
['T,h,i,s,e,x,a,m,p,l,e,a,i,n,t,e,a,s,y']
['I,', ',a,m,', ',l,e,a,r,n,i,n,g']

Use re.split() for this task:
import re
w_list = [i for i in re.split(r'[^a-zA-Z]',
'____-----This4ex5ample---aint___ea5sy;782') if i ]
Out[12]: ['This', 'ex', 'ample', 'aint', 'ea', 'sy']

I would import the punctuation marks from string and proceed with regular expressions as follows.
l=['I am learning']
l1=['____-----This4ex5ample---aint___ea5sy;782']
import re
from string import punctuation
punctuation # to see the punctuation marks.
>>> '!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~'
' '.join([re.sub('[!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~\d]',' ', w) for w in l]).split()
Here is the output:
>>> ['I', 'am', 'learning']
Notice the \d attached at the end of the punctuation marks to remove any digits.
Similarly,
' '.join([re.sub('[!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~\d]',' ', w) for w in l1]).split()
Yields
>>> ['This', 'ex', 'ample', 'aint', 'ea', 'sy']
You can also modify your function as follows:
def mySplit(x):
banned = ['-', '+' , ',', '#', '.', '!', '?', ':', '_', ';'] + list('0123456789')#Banned chars
return ''.join([word if not word in banned else ' ' for word in list(x[0]) ]).split()

remove certain symbols form a string array

I have a numpy.ndarray with Strings. I have created a character list, which I would like to use against the strings array, to remove all characters which appear in the character list. I want to put the symbol free strings in a new array. How can I do this?
Input:
symbols = string.printable[62:]
symbolsList = list(symbols)
symbolsList
Output:
['!',
'"',
'#',
'$',
'%',
'&',
"'",
'(',
')',
'*',
'+',
',',
'-',
'.',
'/',
':',
';',
'<',
'=',
'>',
'?',
'#',
'[',
'\\',
']',
'^',
'_',
'`',
'{',
'|',
'}',
'~',
' ',
'\t',
'\n',
'\r',
'\x0b',
'\x0c']
A sample output of the string_array:
array(['[KFC] CHicken_Gravy_Coke_Biscuit This is my Order!!!<lf><lf>', dtype=object)
I want it to look like this:
array(['KFC CHicken Gravy Coke Biscuit This is my Order lf lf', dtype=object)
I tried:
cleanData = []
for i in string_array:
cleanData.append(string_array[i].replace(symbolsList[i], " "))
and:
cleanData = []
for i in summary_data:
cleanData = summary_data[i].replace(symbolsList[i], " ")
Both give same Output:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
But does not work :( How to make this work? Or do what I want?

This is one way.
import re, string, numpy as np
def remove_chars_re(x):
x = re.sub('[' + re.escape(''.join(string.printable[62:])) + ']', ' ', x)
return re.sub(' +', ' ', x).strip()
arr = np.array(['[KFC] CHicken_Gravy_Coke_Biscuit This is my Order!!!<lf><lf>'], dtype=object)
list(map(remove_chars_re, arr))
# ['KFC CHicken Gravy Coke Biscuit This is my Order lf lf']
Explanation
The first re.sub removes unwanted characters with a single space.
The second re.sub removes double spaces.
strip() removes whitespace from start and end of the string.

Here is how I would go about it.
Iterate over each string, str
Iterate over each undesired charactered, chr, for each str (Nested Loop)
Use the str.replace(chr, '')
Here is the code.
cleanData = []
for str in string_array:
tmp_str = str #you need to do this because you need to filter every character one by one
for chr in symbolsList:
tmp_str = tmp_str.replace(chr, ' ') #if you want to replace your undesired symbols with a space
cleanData.append(tmp_str)

python remove weird apostrophe and other weird characters not in string.punctuation [duplicate]

This question already has answers here:
Remove punctuation from Unicode formatted strings
(4 answers)
Closed 6 years ago.
This is my string:
mystring = "How’s it going?"
This is what i did:
import string
exclude = set(string.punctuation)
def strip_punctuations(mystring):
for c in string.punctuation:
new_string=''.join(ch for ch in mystring if ch not in exclude)
new_string = chat_string.replace("\xe2\x80\x99","")
new_string = chat_string.replace("\xc2\xa0\xc2\xa0","")
return chat_string
OUTPUT:
If i did not include this line new_string = chat_string.replace("\xe2\x80\x99","") this will be the output:
'How\xe2\x80\x99s it going'
i realized
exclude does not have that weird looking apostrophe in the list:
print set(exclude)
set(['!', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', ';', ':', '=', '<', '?', '>', '#', '[', ']', '\\', '_', '^', '`', '{', '}', '|', '~'])
How do i ensure all such characters are taken out instead of manually replacing them in the future?

If you are working with long texts like news articles or web scraping, then you can either use "goose" or "NLTK" python libraries. These two are not pre-installed. Here are the links to the libraries. goose, NLTK
You can go through the document and learn how to do.
OR
if you don't want to use these libraries, you may want to create your own "exclude" list manually.

import re
toReplace = "how's it going?"
regex = re.compile('[!#%$\"&)\'(+*-/.;:=<?>#\[\]_^`\{\}|~"\\\\"]')
newVal = regex.sub('', toReplace)
print(newVal)
The regex matches all the characters you've set and it replaces them with empty whitespace.

strip punctuation with regex - python

I need to use regex to strip punctuation at the start and end of a word. It seems like regex would be the best option for this. I don't want punctuation removed from words like 'you're', which is why I'm not using .replace().

You don't need regular expression to do this task. Use str.strip with string.punctuation:
>>> import string
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~'
>>> '!Hello.'.strip(string.punctuation)
'Hello'
>>> ' '.join(word.strip(string.punctuation) for word in "Hello, world. I'm a boy, you're a girl.".split())
"Hello world I'm a boy you're a girl"

I think this function will be helpful and concise in removing punctuation:
import re
def remove_punct(text):
new_words = []
for word in text:
w = re.sub(r'[^\w\s]','',word) #remove everything except words and space
w = re.sub(r'_','',w) #how to remove underscore as well
new_words.append(w)
return new_words

If you persist in using Regex, I recommend this solution:
import re
import string
p = re.compile("[" + re.escape(string.punctuation) + "]")
print(p.sub("", "\"hello world!\", he's told me."))
### hello world hes told me
Note also that you can pass your own punctuation marks:
my_punct = ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '.',
'/', ':', ';', '<', '=', '>', '?', '#', '[', '\\', ']', '^', '_',
'`', '{', '|', '}', '~', '»', '«', '“', '”']
punct_pattern = re.compile("[" + re.escape("".join(my_punct)) + "]")
re.sub(punct_pattern, "", "I've been vaccinated against *covid-19*!") # the "-" symbol should remain
### Ive been vaccinated against covid-19

You can remove punctuation from a text file or a particular string file using regular expression as follows -
new_data=[]
with open('/home/rahul/align.txt','r') as f:
f1 = f.read()
f2 = f1.split()
all_words = f2
punctuations = '''!()-[]{};:'"\,<>./?##$%^&*_~'''
# You can add and remove punctuations as per your choice
#removing stop words in hungarian text and english text and
#display the unpunctuated string
# To remove from a string, replace new_data with new_str
# new_str = "My name$## is . rahul -~"
for word in all_words:
if word not in punctuations:
new_data.append(word)
print (new_data)
P.S. - Do the identation properly as per required.
Hope this helps!!