Develop Reference

remove all the special chars from a list [duplicate]

This question already has answers here:
Removing punctuation from a list in python
(2 answers)
Closed last year.
i have a list of strings with some strings being the special characters what would be the approach to exclude them in the resultant list
list = ['ben','kenny',',','=','Sean',100,'tag242']
expected output = ['ben','kenny','Sean',100,'tag242']
please guide me with the approach to achieve the same. Thanks

The string module has a list of punctuation marks that you can use and exclude from your list of words:
import string
punctuations = list(string.punctuation)
input_list = ['ben','kenny',',','=','Sean',100,'tag242']
output = [x for x in input_list if x not in punctuations]
print(output)
Output:
['ben', 'kenny', 'Sean', 100, 'tag242']
This list of punctuation marks includes the following characters:
['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '#', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~']

It can simply be done using the isalnum() string function. isalnum() returns true if the string contains only digits or letters, if a string contains any special character other than that, the function will return false. (no modules needed to be imported for isalnum() it is a default function)
code:
list = ['ben','kenny',',','=','Sean',100,'tag242']
olist = []
for a in list:
if str(a).isalnum():
olist.append(a)
print(olist)
output:
['ben', 'kenny', 'Sean', 100, 'tag242']

my_list = ['ben', 'kenny', ',' ,'=' ,'Sean', 100, 'tag242']
stop_words = [',', '=']
filtered_output = [i for i in my_list if i not in stop_words]
The list with stop words can be expanded if you need to remove other characters.

python remove weird apostrophe and other weird characters not in string.punctuation [duplicate]

This question already has answers here:
Remove punctuation from Unicode formatted strings
(4 answers)
Closed 6 years ago.
This is my string:
mystring = "How’s it going?"
This is what i did:
import string
exclude = set(string.punctuation)
def strip_punctuations(mystring):
for c in string.punctuation:
new_string=''.join(ch for ch in mystring if ch not in exclude)
new_string = chat_string.replace("\xe2\x80\x99","")
new_string = chat_string.replace("\xc2\xa0\xc2\xa0","")
return chat_string
OUTPUT:
If i did not include this line new_string = chat_string.replace("\xe2\x80\x99","") this will be the output:
'How\xe2\x80\x99s it going'
i realized
exclude does not have that weird looking apostrophe in the list:
print set(exclude)
set(['!', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', ';', ':', '=', '<', '?', '>', '#', '[', ']', '\\', '_', '^', '`', '{', '}', '|', '~'])
How do i ensure all such characters are taken out instead of manually replacing them in the future?

If you are working with long texts like news articles or web scraping, then you can either use "goose" or "NLTK" python libraries. These two are not pre-installed. Here are the links to the libraries. goose, NLTK
You can go through the document and learn how to do.
OR
if you don't want to use these libraries, you may want to create your own "exclude" list manually.

import re
toReplace = "how's it going?"
regex = re.compile('[!#%$\"&)\'(+*-/.;:=<?>#\[\]_^`\{\}|~"\\\\"]')
newVal = regex.sub('', toReplace)
print(newVal)
The regex matches all the characters you've set and it replaces them with empty whitespace.

Multiple symbols replace not working

I need to check a string for some symbols and replace them with a whitespace. My code:
string = 'so\bad'
symbols = ['•', '!', '"', '#', '$', '%', '&', '\'', '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '>', '=', '?', '#', '[', ']', '\\', '^', '_', '`', '{', '}', '~', '|', '"', '⌐', '¬', '«', '»', '£', '$', '°', '§', '–', '—']
for symbol in symbols:
string = string.replace(symbol, ' ')
print string
>> sad
Why does it replace a\b with nothing?

This is because \b is ASCII backspace character:
>>> string = 'so\bad'
>>> print string
sad
You can find it and all the other escape characters from Python Reference Manual.
In order to get the behavior you expect escape the backslash character or use raw strings:
# Both result to 'so bad'
string = 'so\\bad'
string = r'so\bad'

The issue you are facing is the use of \ as a escape character.
\b is a special character (backspace)
Use a String literal with prefix r.
With the r, backslashes \ are treated as literal
string = r'so\bad'

You are not replacing anything "\b" is backspace, moving your cursor to the left one step.

Note that even if you omit the symbols list and your for symbol in symbols: code, you will always get the result "sad" when you print string. This is because \b means something as an ascii character, and is being interpreted together.
Check out this stackoverflow answer for a solution on how to work around this issue: How can I print out the string "\b" in Python

Removing punctuation in lists in Python

Creating a Python program that converts the string to a list, uses a loop to remove any punctuation and then converts the list back into a string and prints the sentence without punctuation.
punctuation=['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
str=input("Type in a line of text: ")
alist=[]
alist.extend(str)
print(alist)
#Use loop to remove any punctuation (that appears on the punctuation list) from the list
print(''.join(alist))
This is what I have so far. I tried using something like: alist.remove(punctuation) but I get an error saying something like list.remove(x): x not in list. I didn't read the question properly at first and realized that I needed to do this by using a loop so I added that in as a comment and now I'm stuck. I was, however, successful in converting it from a list back into a string.

import string
punct = set(string.punctuation)
''.join(x for x in 'a man, a plan, a canal' if x not in punct)
Out[7]: 'a man a plan a canal'
Explanation: string.punctuation is pre-defined as:
'!"#$%&\'()*+,-./:;<=>?#[\\]^_`{|}~'
The rest is a straightforward comprehension. A set is used to speed up the filtering step.

I found a easy way to do it:
punctuation = ['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
str = raw_input("Type in a line of text: ")
for i in punctuation:
str = str.replace(i,"")
print str
With this way you will not get any error.

punctuation=['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
result = ""
for character in str:
if(character not in punctuation):
result += character
print result

Here is the answer of how to tokenize the given statements by using python. the python version I used is 3.4.4
Assume that I have text which is saved as one.txt. then I have saved my python program in the directory where my file is (i.e. one.txt). The following is my python program:
with open('one.txt','r')as myFile:
str1=myFile.read()
print(str1)# This is to print the given statements with punctuations(before removal of punctuations)
# The following is the list of punctuations that we need to remove, add any more if I forget
punctuation = ['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
for i in punctuation:
str1 = str1.replace(i," ") #to make empty the place where punctuation is there.
myList=[]
myList.extend(str1.split(" "))
print (str1) #this is to print the given statements without puctions(after Removal of punctuations)
for i in myList:
# print ("____________")
print(i,end='\n')
print ("____________")
==============next I will post for you how to remove stop words============
until that let you comment if it is useful.
Thank you

Have I found a bug in Python's str.endswith()?

According to the Python documentation:
str.endswith(suffix[, start[, end]])
Return True if the string ends with the specified suffix, otherwise return False. suffix can also be a tuple of suffixes to look for. With optional start, test beginning at that position. Withoptional end, stop comparing at that position.
Changed in version 2.5: Accept tuples as suffix.
The following code should return True, but it returns False in Python 2.7.3:
"hello-".endswith(('.', ',', ':', ';', '-' '?', '!'))
It seems str.endswith() ignores anything beyond the forth tuple element:
>>> "hello-".endswith(('.', ',', ':', '-', ';' '?', '!'))
>>> True
>>> "hello;".endswith(('.', ',', ':', '-', ';' '?', '!'))
>>> False
Have I found a bug, or am I missing something?

or am I missing something?
You're missing a comma after the ';' in your tuple:
>>> "hello;".endswith(('.', ',', ':', '-', ';' '?', '!'))
# ^
# comma missing
False
Due to this, ; and ? are concatenated. So, the string ending with ;? will return True for this case:
>>> "hello;?".endswith(('.', ',', ':', '-', ';' '?', '!'))
True
After adding a comma, it would work as expected:
>>> "hello;".endswith(('.', ',', ':', '-', ';', '?', '!'))
True

If you write tuple as
>>> tuple_example = ('.', ',', ':', '-', ';' '?', '!')
then the tuple will become
>>> tuple_example
('.', ',', ':', '-', ';?', '!')
^
# concatenate together
So that is why return False

It has already been pointed out that adjacent string literals are concatenated, but I wanted to add a little additional information and context.
This is a feature that is shared with (and borrowed from) C.
Additionally, this is doesn't act like a concatenation operator like '+', and is treated identically as if they were literally joined together in the source without any additional overhead.
For example:
>>> 'a' 'b' * 2
'abab'
Whether this is useful feature or an annoying design is really a matter of opinion, but it does allow for breaking up string literals among multiple lines by encapsulating the literals within parentheses.
>>> print("I don't want to type this whole string"
"literal all on one line.")
I don't want to type this whole stringliteral all on one line.
That type of usage (along with being used with #defines) is why it was useful in C in the first place and was subsequently brought along in Python.