Develop Reference

Removing all punctuation except white spaces python [duplicate]

It seems like there should be a simpler way than:
import string
s = "string. With. Punctuation?" # Sample string
out = s.translate(string.maketrans("",""), string.punctuation)
Is there?

From an efficiency perspective, you're not going to beat
s.translate(None, string.punctuation)
For higher versions of Python use the following code:
s.translate(str.maketrans('', '', string.punctuation))
It's performing raw string operations in C with a lookup table - there's not much that will beat that but writing your own C code.
If speed isn't a worry, another option though is:
exclude = set(string.punctuation)
s = ''.join(ch for ch in s if ch not in exclude)
This is faster than s.replace with each char, but won't perform as well as non-pure python approaches such as regexes or string.translate, as you can see from the below timings. For this type of problem, doing it at as low a level as possible pays off.
Timing code:
import re, string, timeit
s = "string. With. Punctuation"
exclude = set(string.punctuation)
table = string.maketrans("","")
regex = re.compile('[%s]' % re.escape(string.punctuation))
def test_set(s):
return ''.join(ch for ch in s if ch not in exclude)
def test_re(s): # From Vinko's solution, with fix.
return regex.sub('', s)
def test_trans(s):
return s.translate(table, string.punctuation)
def test_repl(s): # From S.Lott's solution
for c in string.punctuation:
s=s.replace(c,"")
return s
print "sets :",timeit.Timer('f(s)', 'from __main__ import s,test_set as f').timeit(1000000)
print "regex :",timeit.Timer('f(s)', 'from __main__ import s,test_re as f').timeit(1000000)
print "translate :",timeit.Timer('f(s)', 'from __main__ import s,test_trans as f').timeit(1000000)
print "replace :",timeit.Timer('f(s)', 'from __main__ import s,test_repl as f').timeit(1000000)
This gives the following results:
sets : 19.8566138744
regex : 6.86155414581
translate : 2.12455511093
replace : 28.4436721802

Regular expressions are simple enough, if you know them.
import re
s = "string. With. Punctuation?"
s = re.sub(r'[^\w\s]','',s)

For the convenience of usage, I sum up the note of striping punctuation from a string in both Python 2 and Python 3. Please refer to other answers for the detailed description.
Python 2
import string
s = "string. With. Punctuation?"
table = string.maketrans("","")
new_s = s.translate(table, string.punctuation) # Output: string without punctuation
Python 3
import string
s = "string. With. Punctuation?"
table = str.maketrans(dict.fromkeys(string.punctuation)) # OR {key: None for key in string.punctuation}
new_s = s.translate(table) # Output: string without punctuation

myString.translate(None, string.punctuation)

Not necessarily simpler, but a different way, if you are more familiar with the re family.
import re, string
s = "string. With. Punctuation?" # Sample string
out = re.sub('[%s]' % re.escape(string.punctuation), '', s)

string.punctuation is ASCII only! A more correct (but also much slower) way is to use the unicodedata module:
# -*- coding: utf-8 -*-
from unicodedata import category
s = u'String — with - «punctation »...'
s = ''.join(ch for ch in s if category(ch)[0] != 'P')
print 'stripped', s
You can generalize and strip other types of characters as well:
''.join(ch for ch in s if category(ch)[0] not in 'SP')
It will also strip characters like ~*+§$ which may or may not be "punctuation" depending on one's point of view.

I usually use something like this:
>>> s = "string. With. Punctuation?" # Sample string
>>> import string
>>> for c in string.punctuation:
... s= s.replace(c,"")
...
>>> s
'string With Punctuation'

For Python 3 str or Python 2 unicode values, str.translate() only takes a dictionary; codepoints (integers) are looked up in that mapping and anything mapped to None is removed.
To remove (some?) punctuation then, use:
import string
remove_punct_map = dict.fromkeys(map(ord, string.punctuation))
s.translate(remove_punct_map)
The dict.fromkeys() class method makes it trivial to create the mapping, setting all values to None based on the sequence of keys.
To remove all punctuation, not just ASCII punctuation, your table needs to be a little bigger; see J.F. Sebastian's answer (Python 3 version):
import unicodedata
import sys
remove_punct_map = dict.fromkeys(i for i in range(sys.maxunicode)
if unicodedata.category(chr(i)).startswith('P'))

string.punctuation misses loads of punctuation marks that are commonly used in the real world. How about a solution that works for non-ASCII punctuation?
import regex
s = u"string. With. Some・Really Weird、Non？ASCII。 「（Punctuation）」?"
remove = regex.compile(ur'[\p{C}|\p{M}|\p{P}|\p{S}|\p{Z}]+', regex.UNICODE)
remove.sub(u" ", s).strip()
Personally, I believe this is the best way to remove punctuation from a string in Python because:
It removes all Unicode punctuation
It's easily modifiable, e.g. you can remove the \{S} if you want to remove punctuation, but keep symbols like $.
You can get really specific about what you want to keep and what you want to remove, for example \{Pd} will only remove dashes.
This regex also normalizes whitespace. It maps tabs, carriage returns, and other oddities to nice, single spaces.
This uses Unicode character properties, which you can read more about on Wikipedia.

I haven't seen this answer yet. Just use a regex; it removes all characters besides word characters (\w) and number characters (\d), followed by a whitespace character (\s):
import re
s = "string. With. Punctuation?" # Sample string
out = re.sub(ur'[^\w\d\s]+', '', s)

Here's a one-liner for Python 3.5:
import string
"l*ots! o(f. p#u)n[c}t]u[a'ti\"on#$^?/".translate(str.maketrans({a:None for a in string.punctuation}))

This might not be the best solution however this is how I did it.
import string
f = lambda x: ''.join([i for i in x if i not in string.punctuation])

import re
s = "string. With. Punctuation?" # Sample string
out = re.sub(r'[^a-zA-Z0-9\s]', '', s)

Here is a function I wrote. It's not very efficient, but it is simple and you can add or remove any punctuation that you desire:
def stripPunc(wordList):
"""Strips punctuation from list of words"""
puncList = [".",";",":","!","?","/","\\",",","#","#","$","&",")","(","\""]
for punc in puncList:
for word in wordList:
wordList=[word.replace(punc,'') for word in wordList]
return wordList

>>> s = "string. With. Punctuation?"
>>> s = re.sub(r'[^\w\s]','',s)
>>> re.split(r'\s*', s)
['string', 'With', 'Punctuation']

Just as an update, I rewrote the #Brian example in Python 3 and made changes to it to move regex compile step inside of the function. My thought here was to time every single step needed to make the function work. Perhaps you are using distributed computing and can't have regex object shared between your workers and need to have re.compile step at each worker. Also, I was curious to time two different implementations of maketrans for Python 3
table = str.maketrans({key: None for key in string.punctuation})
vs
table = str.maketrans('', '', string.punctuation)
Plus I added another method to use set, where I take advantage of intersection function to reduce number of iterations.
This is the complete code:
import re, string, timeit
s = "string. With. Punctuation"
def test_set(s):
exclude = set(string.punctuation)
return ''.join(ch for ch in s if ch not in exclude)
def test_set2(s):
_punctuation = set(string.punctuation)
for punct in set(s).intersection(_punctuation):
s = s.replace(punct, ' ')
return ' '.join(s.split())
def test_re(s): # From Vinko's solution, with fix.
regex = re.compile('[%s]' % re.escape(string.punctuation))
return regex.sub('', s)
def test_trans(s):
table = str.maketrans({key: None for key in string.punctuation})
return s.translate(table)
def test_trans2(s):
table = str.maketrans('', '', string.punctuation)
return(s.translate(table))
def test_repl(s): # From S.Lott's solution
for c in string.punctuation:
s=s.replace(c,"")
return s
print("sets :",timeit.Timer('f(s)', 'from __main__ import s,test_set as f').timeit(1000000))
print("sets2 :",timeit.Timer('f(s)', 'from __main__ import s,test_set2 as f').timeit(1000000))
print("regex :",timeit.Timer('f(s)', 'from __main__ import s,test_re as f').timeit(1000000))
print("translate :",timeit.Timer('f(s)', 'from __main__ import s,test_trans as f').timeit(1000000))
print("translate2 :",timeit.Timer('f(s)', 'from __main__ import s,test_trans2 as f').timeit(1000000))
print("replace :",timeit.Timer('f(s)', 'from __main__ import s,test_repl as f').timeit(1000000))
This is my results:
sets : 3.1830138750374317
sets2 : 2.189873124472797
regex : 7.142953420989215
translate : 4.243278483860195
translate2 : 2.427158243022859
replace : 4.579746678471565

A one-liner might be helpful in not very strict cases:
''.join([c for c in s if c.isalnum() or c.isspace()])

Here's a solution without regex.
import string
input_text = "!where??and!!or$$then:)"
punctuation_replacer = string.maketrans(string.punctuation, ' '*len(string.punctuation))
print ' '.join(input_text.translate(punctuation_replacer).split()).strip()
Output>> where and or then
Replaces the punctuations with spaces
Replace multiple spaces in between words with a single space
Remove the trailing spaces, if any with
strip()

Why none of you use this?
''.join(filter(str.isalnum, s))
Too slow?

I was looking for a really simple solution. here's what I got:
import re
s = "string. With. Punctuation?"
s = re.sub(r'[\W\s]', ' ', s)
print(s)
'string With Punctuation '

Here's one other easy way to do it using RegEx
import re
punct = re.compile(r'(\w+)')
sentence = 'This ! is : a # sample $ sentence.' # Text with punctuation
tokenized = [m.group() for m in punct.finditer(sentence)]
sentence = ' '.join(tokenized)
print(sentence)
'This is a sample sentence'

# FIRST METHOD
# Storing all punctuations in a variable
punctuation='!?,.:;"\')(_-'
newstring ='' # Creating empty string
word = raw_input("Enter string: ")
for i in word:
if(i not in punctuation):
newstring += i
print ("The string without punctuation is", newstring)
# SECOND METHOD
word = raw_input("Enter string: ")
punctuation = '!?,.:;"\')(_-'
newstring = word.translate(None, punctuation)
print ("The string without punctuation is",newstring)
# Output for both methods
Enter string: hello! welcome -to_python(programming.language)??,
The string without punctuation is: hello welcome topythonprogramminglanguage

with open('one.txt','r')as myFile:
str1=myFile.read()
print(str1)
punctuation = ['(', ')', '?', ':', ';', ',', '.', '!', '/', '"', "'"]
for i in punctuation:
str1 = str1.replace(i," ")
myList=[]
myList.extend(str1.split(" "))
print (str1)
for i in myList:
print(i,end='\n')
print ("____________")

Try that one :)
regex.sub(r'\p{P}','', s)

The question does not have a lot of specifics, so the approach I took is to come up with a solution with the simplest interpretation of the problem: just remove the punctuation.
Note that solutions presented don't account for contracted words (e.g., you're) or hyphenated words (e.g., anal-retentive)...which is debated as to whether they should or shouldn't be treated as punctuations...nor to account for non-English character set or anything like that...because those specifics were not mentioned in the question. Someone argued that space is punctuation, which is technically correct...but to me it makes zero sense in the context of the question at hand.
# using lambda
''.join(filter(lambda c: c not in string.punctuation, s))
# using list comprehension
''.join('' if c in string.punctuation else c for c in s)

Apparently I can't supply edits to the selected answer, so here's an update which works for Python 3. The translate approach is still the most efficient option when doing non-trivial transformations.
Credit for the original heavy lifting to #Brian above. And thanks to #ddejohn for his excellent suggestion for improvement to the original test.
#!/usr/bin/env python3
"""Determination of most efficient way to remove punctuation in Python 3.
Results in Python 3.8.10 on my system using the default arguments:
set : 51.897
regex : 17.901
translate : 2.059
replace : 13.209
"""
import argparse
import re
import string
import timeit
parser = argparse.ArgumentParser()
parser.add_argument("--filename", "-f", default=argparse.__file__)
parser.add_argument("--iterations", "-i", type=int, default=10000)
opts = parser.parse_args()
with open(opts.filename) as fp:
s = fp.read()
exclude = set(string.punctuation)
table = str.maketrans("", "", string.punctuation)
regex = re.compile(f"[{re.escape(string.punctuation)}]")
def test_set(s):
return "".join(ch for ch in s if ch not in exclude)
def test_regex(s): # From Vinko's solution, with fix.
return regex.sub("", s)
def test_translate(s):
return s.translate(table)
def test_replace(s): # From S.Lott's solution
for c in string.punctuation:
s = s.replace(c, "")
return s
opts = dict(globals=globals(), number=opts.iterations)
solutions = "set", "regex", "translate", "replace"
for solution in solutions:
elapsed = timeit.timeit(f"test_{solution}(s)", **opts)
print(f"{solution:<10}: {elapsed:6.3f}")

For serious natural language processing (NLP), you should let a library like SpaCy handle punctuation through tokenization, which you can then manually tweak to your needs.
For example, how do you want to handle hyphens in words? Exceptional cases like abbreviations? Begin and end quotes? URLs? IN NLP it's often useful to separate out a contraction like "let's" into "let" and "'s" for further processing.

Considering unicode. Code checked in python3.
from unicodedata import category
text = 'hi, how are you?'
text_without_punc = ''.join(ch for ch in text if not category(ch).startswith('P'))

You can also do this:
import string
' '.join(word.strip(string.punctuation) for word in 'text'.split())

When you deal with the Unicode strings, I suggest using PyPi regex module because it supports both Unicode property classes (like \p{X} / \P{X}) and POSIX character classes (like [:name:]).
Just install the package by typing pip install regex (or pip3 install regex) in your terminal and hit ENTER.
In case you need to remove punctuation and symbols of any kind (that is, anything other than letters, digits and whitespace) you can use
regex.sub(r'[\p{P}\p{S}]', '', text) # to remove one by one
regex.sub(r'[\p{P}\p{S}]+', '', text) # to remove all consecutive punctuation/symbols with one go
regex.sub(r'[[:punct:]]+', '', text) # Same with a POSIX character class
See a Python demo online:
import regex
text = 'भारत India <><>^$.,,! 002'
new_text = regex.sub(r'[\p{P}\p{S}\s]+', ' ', text).lower().strip()
# OR
# new_text = regex.sub(r'[[:punct:]\s]+', ' ', text).lower().strip()
print(new_text)
# => भारत india 002
Here, I added a whitespace \s pattern to the character class

Regex replacing pairs of dollar signs

I have a string like: The old man $went$ to the $barn$. How would I convert this to The old man ~!went! to the ~!barn!.
If I didn't need to add the ~ in front of the first occurrence, I could simply do text.replace('$', '!') in Python.

Use a capture group so that your replacement string can put the text between the $ back in place.
So the regex would be:
\$([^$]*)\$
And then the replacement string would be:
~!\1!
Regex101 Demo

Yes, regex this. Capture groups will help.
result = re.sub(r'\$(.*?)\$', r'~!\1!', my_str)

Probably regex capturing group is the way to go here, but here a simple way to do it without regex:
>>> s
'The old man $went$ to the $barn$'
>>> r
''
>>> seen = False
>>>
>>> for c in s:
if c=='$':
if seen:
r +='!'
seen = False
else:
r +='~!'
seen=True
else:
r += c
>>> r
'The old man ~!went! to the ~!barn!'

Get string between 2 other strings - Python 2.7.8

So I have a huge string, where some strings occur a lot. I need the text in between.
"I don't need this""This is what I need""I also don't need this."
This happens many times, and I'd like all the strings I need in a list.
There's also a lot of special characters, but no ' so I can use them for strings.
I have tried with the re library, but I can't get it to work.
I tried splitting too
listy = hugestring.split('delim1')
for element in listy:
element = element.split('delim2')
But the second splitting doesn't work.

You could use a regex like this
>>> import re
>>> your_str = "foo This is what I need bar foo This is what I need too bar"
>>> left_delim = "foo "
>>> right_delim = " bar"
>>> pattern = "(?<={})[ \w]*?(?={})".format(left_delim,right_delim)
>>> re.findall(pattern,your_str)
['This is what I need', 'This is what I need too']

This will give you a list of all strings within quotes contained in a string:
import re
in_str = "I don't need this\"This is what I need\"I also don't need this."
out_str = re.findall(r'\"(.+?)\"', in_str)
print out_str
So in the above example, print out_str[0] will give you what you need as there's only the one quote in there.

this is the result of what you say in comment , so whats problem now ?:
>>> n= s.split("I don't need this")
['', "This is what I needI also don't need this."]
>>> [i.split("I also don't need this") for i in n]
[[''], ['This is what I need', '.']]

Create a Reg Exp to search for __word__?

In a program I'm making in python and I want all words formatted like __word__ to stand out. How could I search for words like these using a regex?

Perhaps something like
\b__(\S+)__\b
>>> import re
>>> re.findall(r"\b__(\S+)__\b","Here __is__ a __test__ sentence")
['is', 'test']
>>> re.findall(r"\b__(\S+)__\b","__Here__ is a test __sentence__")
['Here', 'sentence']
>>> re.findall(r"\b__(\S+)__\b","__Here's__ a test __sentence__")
["Here's", 'sentence']
or you can put tags around the word like this
>>> print re.sub(r"\b(__)(\S+)(__)\b",r"<b>\2<\\b>","__Here__ is a test __sentence__")
<b>Here<\b> is a test <b>sentence<\b>
If you need more fine grained control over the legal word characters it's best to be explicit
\b__([a-zA-Z0-9_':])__\b ### count "'" and ":" as part of words
>>> re.findall(r"\b__([a-zA-Z0-9_']+)__\b","__Here's__ a test __sentence:__")
["Here's"]
>>> re.findall(r"\b__([a-zA-Z0-9_':]+)__\b","__Here's__ a test __sentence:__")
["Here's", 'sentence:']

Take a squizz here: http://docs.python.org/library/re.html
That should show you syntax and examples from which you can build a check for word(s) pre- and post-pended with 2 underscores.

The simplest regex for this would be
__.+__
If you want access to the word itself from your code, you should use
__(.+)__

This will give you a list with all such words
>>> import re
>>> m = re.findall("(__\w+__)", "What __word__ you search __for__")
>>> print m
['__word__', '__for__']

\b(__\w+__)\b
\b word boundary
\w+ one or more word characters - [a-zA-Z0-9_]

simple string functions. no regex
>>> mystring="blah __word__ blah __word2__"
>>> for item in mystring.split():
... if item.startswith("__") and item.endswith("__"):
... print item
...
__word__
__word2__

A pythonic way to insert a space before capital letters

I've got a file whose format I'm altering via a python script. I have several camel cased strings in this file where I just want to insert a single space before the capital letter - so "WordWordWord" becomes "Word Word Word".
My limited regex experience just stalled out on me - can someone think of a decent regex to do this, or (better yet) is there a more pythonic way to do this that I'm missing?

You could try:
>>> re.sub(r"(\w)([A-Z])", r"\1 \2", "WordWordWord")
'Word Word Word'

If there are consecutive capitals, then Gregs result could
not be what you look for, since the \w consumes the caracter
in front of the captial letter to be replaced.
>>> re.sub(r"(\w)([A-Z])", r"\1 \2", "WordWordWWWWWWWord")
'Word Word WW WW WW Word'
A look-behind would solve this:
>>> re.sub(r"(?<=\w)([A-Z])", r" \1", "WordWordWWWWWWWord")
'Word Word W W W W W W Word'

Perhaps shorter:
>>> re.sub(r"\B([A-Z])", r" \1", "DoIThinkThisIsABetterAnswer?")

Have a look at my answer on .NET - How can you split a “caps” delimited string into an array?
Edit: Maybe better to include it here.
re.sub(r'([a-z](?=[A-Z])|[A-Z](?=[A-Z][a-z]))', r'\1 ', text)
For example:
"SimpleHTTPServer" => ["Simple", "HTTP", "Server"]

Maybe you would be interested in one-liner implementation without using regexp:
''.join(' ' + char if char.isupper() else char.strip() for char in text).strip()

With regexes you can do this:
re.sub('([A-Z])', r' \1', str)
Of course, that will only work for ASCII characters, if you want to do Unicode it's a whole new can of worms :-)

If you have acronyms, you probably do not want spaces between them. This two-stage regex will keep acronyms intact (and also treat punctuation and other non-uppercase letters as something to add a space on):
re_outer = re.compile(r'([^A-Z ])([A-Z])')
re_inner = re.compile(r'(?<!^)([A-Z])([^A-Z])')
re_outer.sub(r'\1 \2', re_inner.sub(r' \1\2', 'DaveIsAFKRightNow!Cool'))
The output will be: 'Dave Is AFK Right Now! Cool'

I agree that the regex solution is the easiest, but I wouldn't say it's the most pythonic.
How about:
text = 'WordWordWord'
new_text = ''
for i, letter in enumerate(text):
if i and letter.isupper():
new_text += ' '
new_text += letter

I think regexes are the way to go here, but just to give a pure python version without (hopefully) any of the problems ΤΖΩΤΖΙΟΥ has pointed out:
def splitCaps(s):
result = []
for ch, next in window(s+" ", 2):
result.append(ch)
if next.isupper() and not ch.isspace():
result.append(' ')
return ''.join(result)
window() is a utility function I use to operate on a sliding window of items, defined as:
import collections, itertools
def window(it, winsize, step=1):
it=iter(it) # Ensure we have an iterator
l=collections.deque(itertools.islice(it, winsize))
while 1: # Continue till StopIteration gets raised.
yield tuple(l)
for i in range(step):
l.append(it.next())
l.popleft()

To the old thread - wanted to try an option for one of my requirements. Of course the re.sub() is the cool solution, but also got a 1 liner if re module isn't (or shouldn't be) imported.
st = 'ThisIsTextStringToSplitWithSpace'
print(''.join([' '+ s if s.isupper() else s for s in st]).lstrip())