Here's the simplest way to explain this. Here's what I'm using:
re.split('\W', 'foo/bar spam\neggs')
>>> ['foo', 'bar', 'spam', 'eggs']
Here's what I want:
someMethod('\W', 'foo/bar spam\neggs')
>>> ['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']
The reason is that I want to split a string into tokens, manipulate it, then put it back together again.
The docs of re.split mention:
Split string by the occurrences of pattern. If capturing
parentheses are used in pattern, then the text of all groups in the
pattern are also returned as part of the resulting list.
So you just need to wrap your separator with a capturing group:
>>> re.split('(\W)', 'foo/bar spam\neggs')
['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']
If you are splitting on newline, use splitlines(True).
>>> 'line 1\nline 2\nline without newline'.splitlines(True)
['line 1\n', 'line 2\n', 'line without newline']
(Not a general solution, but adding this here in case someone comes here not realizing this method existed.)
another example, split on non alpha-numeric and keep the separators
import re
a = "foo,bar#candy*ice%cream"
re.split('([^a-zA-Z0-9])',a)
output:
['foo', ',', 'bar', '#', 'candy', '*', 'ice', '%', 'cream']
explanation
re.split('([^a-zA-Z0-9])',a)
() <- keep the separators
[] <- match everything in between
^a-zA-Z0-9 <-except alphabets, upper/lower and numbers.
If you have only 1 separator, you can employ list comprehensions:
text = 'foo,bar,baz,qux'
sep = ','
Appending/prepending separator:
result = [x+sep for x in text.split(sep)]
#['foo,', 'bar,', 'baz,', 'qux,']
# to get rid of trailing
result[-1] = result[-1].strip(sep)
#['foo,', 'bar,', 'baz,', 'qux']
result = [sep+x for x in text.split(sep)]
#[',foo', ',bar', ',baz', ',qux']
# to get rid of trailing
result[0] = result[0].strip(sep)
#['foo', ',bar', ',baz', ',qux']
Separator as it's own element:
result = [u for x in text.split(sep) for u in (x, sep)]
#['foo', ',', 'bar', ',', 'baz', ',', 'qux', ',']
results = result[:-1] # to get rid of trailing
Another no-regex solution that works well on Python 3
# Split strings and keep separator
test_strings = ['<Hello>', 'Hi', '<Hi> <Planet>', '<', '']
def split_and_keep(s, sep):
if not s: return [''] # consistent with string.split()
# Find replacement character that is not used in string
# i.e. just use the highest available character plus one
# Note: This fails if ord(max(s)) = 0x10FFFF (ValueError)
p=chr(ord(max(s))+1)
return s.replace(sep, sep+p).split(p)
for s in test_strings:
print(split_and_keep(s, '<'))
# If the unicode limit is reached it will fail explicitly
unicode_max_char = chr(1114111)
ridiculous_string = '<Hello>'+unicode_max_char+'<World>'
print(split_and_keep(ridiculous_string, '<'))
One Lazy and Simple Solution
Assume your regex pattern is split_pattern = r'(!|\?)'
First, you add some same character as the new separator, like '[cut]'
new_string = re.sub(split_pattern, '\\1[cut]', your_string)
Then you split the new separator, new_string.split('[cut]')
You can also split a string with an array of strings instead of a regular expression, like this:
def tokenizeString(aString, separators):
#separators is an array of strings that are being used to split the string.
#sort separators in order of descending length
separators.sort(key=len)
listToReturn = []
i = 0
while i < len(aString):
theSeparator = ""
for current in separators:
if current == aString[i:i+len(current)]:
theSeparator = current
if theSeparator != "":
listToReturn += [theSeparator]
i = i + len(theSeparator)
else:
if listToReturn == []:
listToReturn = [""]
if(listToReturn[-1] in separators):
listToReturn += [""]
listToReturn[-1] += aString[i]
i += 1
return listToReturn
print(tokenizeString(aString = "\"\"\"hi\"\"\" hello + world += (1*2+3/5) '''hi'''", separators = ["'''", '+=', '+', "/", "*", "\\'", '\\"', "-=", "-", " ", '"""', "(", ")"]))
Here is a simple .split solution that works without regex.
This is an answer for Python split() without removing the delimiter, so not exactly what the original post asks but the other question was closed as a duplicate for this one.
def splitkeep(s, delimiter):
split = s.split(delimiter)
return [substr + delimiter for substr in split[:-1]] + [split[-1]]
Random tests:
import random
CHARS = [".", "a", "b", "c"]
assert splitkeep("", "X") == [""] # 0 length test
for delimiter in ('.', '..'):
for _ in range(100000):
length = random.randint(1, 50)
s = "".join(random.choice(CHARS) for _ in range(length))
assert "".join(splitkeep(s, delimiter)) == s
# This keeps all separators in result
##########################################################################
import re
st="%%(c+dd+e+f-1523)%%7"
sh=re.compile('[\+\-//\*\<\>\%\(\)]')
def splitStringFull(sh, st):
ls=sh.split(st)
lo=[]
start=0
for l in ls:
if not l : continue
k=st.find(l)
llen=len(l)
if k> start:
tmp= st[start:k]
lo.append(tmp)
lo.append(l)
start = k + llen
else:
lo.append(l)
start =llen
return lo
#############################
li= splitStringFull(sh , st)
['%%(', 'c', '+', 'dd', '+', 'e', '+', 'f', '-', '1523', ')%%', '7']
replace all seperator: (\W) with seperator + new_seperator: (\W;)
split by the new_seperator: (;)
def split_and_keep(seperator, s):
return re.split(';', re.sub(seperator, lambda match: match.group() + ';', s))
print('\W', 'foo/bar spam\neggs')
If one wants to split string while keeping separators by regex without capturing group:
def finditer_with_separators(regex, s):
matches = []
prev_end = 0
for match in regex.finditer(s):
match_start = match.start()
if (prev_end != 0 or match_start > 0) and match_start != prev_end:
matches.append(s[prev_end:match.start()])
matches.append(match.group())
prev_end = match.end()
if prev_end < len(s):
matches.append(s[prev_end:])
return matches
regex = re.compile(r"[\(\)]")
matches = finditer_with_separators(regex, s)
If one assumes that regex is wrapped up into capturing group:
def split_with_separators(regex, s):
matches = list(filter(None, regex.split(s)))
return matches
regex = re.compile(r"([\(\)])")
matches = split_with_separators(regex, s)
Both ways also will remove empty groups which are useless and annoying in most of the cases.
install wrs "WITHOUT REMOVING SPLITOR" BY DOING
pip install wrs
(developed by Rao Hamza)
import wrs
text = "Now inbox “how to make spam ad” Invest in hard email marketing."
splitor = 'email | spam | inbox'
list = wrs.wr_split(splitor, text)
print(list)
result:
['now ', 'inbox “how to make ', 'spam ad” invest in hard ', 'email marketing.']
I had a similar issue trying to split a file path and struggled to find a simple answer.
This worked for me and didn't involve having to substitute delimiters back into the split text:
my_path = 'folder1/folder2/folder3/file1'
import re
re.findall('[^/]+/|[^/]+', my_path)
returns:
['folder1/', 'folder2/', 'folder3/', 'file1']
I found this generator based approach more satisfying:
def split_keep(string, sep):
"""Usage:
>>> list(split_keep("a.b.c.d", "."))
['a.', 'b.', 'c.', 'd']
"""
start = 0
while True:
end = string.find(sep, start) + 1
if end == 0:
break
yield string[start:end]
start = end
yield string[start:]
It avoids the need to figure out the correct regex, while in theory should be fairly cheap. It doesn't create new string objects and, delegates most of the iteration work to the efficient find method.
... and in Python 3.8 it can be as short as:
def split_keep(string, sep):
start = 0
while (end := string.find(sep, start) + 1) > 0:
yield string[start:end]
start = end
yield string[start:]
May I just leave it here
s = 'foo/bar spam\neggs'
print(s.replace('/', '+++/+++').replace(' ', '+++ +++').replace('\n', '+++\n+++').split('+++'))
['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']
Use re.split and also your regular expression comes from variable and also you have multi separator ,you can use as the following:
# BashSpecialParamList is the special param in bash,
# such as your separator is the bash special param
BashSpecialParamList = ["$*", "$#", "$#", "$?", "$-", "$$", "$!", "$0"]
# aStr is the the string to be splited
aStr = "$a Klkjfd$0 $? $#%$*Sdfdf"
reStr = "|".join([re.escape(sepStr) for sepStr in BashSpecialParamList])
re.split(f'({reStr})', aStr)
# Then You can get the result:
# ['$a Klkjfd', '$0', ' ', '$?', ' ', '$#', '%', '$*', 'Sdfdf']
reference: GNU Bash Special Parameters
Some of those answers posted before, will repeat delimiter, or have some other bugs which I faced in my case. You can use this function, instead:
def split_and_keep_delimiter(input, delimiter):
result = list()
idx = 0
while delimiter in input:
idx = input.index(delimiter);
result.append(input[0:idx+len(delimiter)])
input = input[idx+len(delimiter):]
result.append(input)
return result
In the below code, there is a simple, very efficient and well tested answer to this question. The code has comments explaining everything in it.
I promise it's not as scary as it looks - it's actually only 13 lines of code! The rest are all comments, docs and assertions
def split_including_delimiters(input: str, delimiter: str):
"""
Splits an input string, while including the delimiters in the output
Unlike str.split, we can use an empty string as a delimiter
Unlike str.split, the output will not have any extra empty strings
Conequently, len(''.split(delimiter))== 0 for all delimiters,
whereas len(input.split(delimiter))>0 for all inputs and delimiters
INPUTS:
input: Can be any string
delimiter: Can be any string
EXAMPLES:
>>> split_and_keep_delimiter('Hello World ! ',' ')
ans = ['Hello ', 'World ', ' ', '! ', ' ']
>>> split_and_keep_delimiter("Hello**World**!***", "**")
ans = ['Hello', '**', 'World', '**', '!', '**', '*']
EXAMPLES:
assert split_and_keep_delimiter('-xx-xx-','xx') == ['-', 'xx', '-', 'xx', '-'] # length 5
assert split_and_keep_delimiter('xx-xx-' ,'xx') == ['xx', '-', 'xx', '-'] # length 4
assert split_and_keep_delimiter('-xx-xx' ,'xx') == ['-', 'xx', '-', 'xx'] # length 4
assert split_and_keep_delimiter('xx-xx' ,'xx') == ['xx', '-', 'xx'] # length 3
assert split_and_keep_delimiter('xxxx' ,'xx') == ['xx', 'xx'] # length 2
assert split_and_keep_delimiter('xxx' ,'xx') == ['xx', 'x'] # length 2
assert split_and_keep_delimiter('x' ,'xx') == ['x'] # length 1
assert split_and_keep_delimiter('' ,'xx') == [] # length 0
assert split_and_keep_delimiter('aaa' ,'xx') == ['aaa'] # length 1
assert split_and_keep_delimiter('aa' ,'xx') == ['aa'] # length 1
assert split_and_keep_delimiter('a' ,'xx') == ['a'] # length 1
assert split_and_keep_delimiter('' ,'' ) == [] # length 0
assert split_and_keep_delimiter('a' ,'' ) == ['a'] # length 1
assert split_and_keep_delimiter('aa' ,'' ) == ['a', '', 'a'] # length 3
assert split_and_keep_delimiter('aaa' ,'' ) == ['a', '', 'a', '', 'a'] # length 5
"""
# Input assertions
assert isinstance(input,str), "input must be a string"
assert isinstance(delimiter,str), "delimiter must be a string"
if delimiter:
# These tokens do not include the delimiter, but are computed quickly
tokens = input.split(delimiter)
else:
# Edge case: if the delimiter is the empty string, split between the characters
tokens = list(input)
# The following assertions are always true for any string input and delimiter
# For speed's sake, we disable this assertion
# assert delimiter.join(tokens) == input
output = tokens[:1]
for token in tokens[1:]:
output.append(delimiter)
if token:
output.append(token)
# Don't let the first element be an empty string
if output[:1]==['']:
del output[0]
# The only case where we should have an empty string in the output is if it is our delimiter
# For speed's sake, we disable this assertion
# assert delimiter=='' or '' not in output
# The resulting strings should be combinable back into the original string
# For speed's sake, we disable this assertion
# assert ''.join(output) == input
return output
>>> line = 'hello_toto_is_there'
>>> sep = '_'
>>> [sep + x[1] if x[0] != 0 else x[1] for x in enumerate(line.split(sep))]
['hello', '_toto', '_is', '_there']
I have a string of text that looks like this:
' 19,301 14,856 18,554'
Where is a space.
I'm trying to split it on the white space, but I need to retain all of the white space as an item in the new list. Like this:
[' ', '19,301',' ', '14,856', ' ', '18,554']
I have been using the following code:
re.split(r'( +)(?=[0-9])', item)
and it returns:
['', ' ', '19,301', ' ', '14,856', ' ', '18,554']
Notice that it always adds an empty element to the beginning of my list. It's easy enough to drop it, but I'm really looking to understand what is going on here, so I can get the code to treat things consistently. Thanks.
When using the re.split method, if the capture group is matched at the start of a string, the "result will start with an empty string". The reason for this is so that join method can behave as the inverse of the split method.
It might not make a lot of sense for your case, where the separator matches are of varying sizes, but if you think about the case where the separators were a | character and you wanted to perform a join on them, with the extra empty string it would work:
>> item = '|19,301|14,856|18,554'
>> items = re.split(r'\|', item)
>> print items
['', '19,301', '14,856', '18,554']
>> '|'.join(items)
'|19,301|14,856|18,554'
But without it, the initial pipe would be missing:
>> items = ['19,301', '14,856', '18,554']
>> '|'.join(items)
'19,301|14,856|18,554'
You can do it with re.findall():
>>> s = '\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s\s19,301\s\s\s\s\s\s\s\s\s14,856\s\s\s\s\s\s\s\s18,554'.replace('\\s',' ')
>>> re.findall(r' +|[^ ]+', s)
[' ', '19,301', ' ', '14,856', ' ', '18,554']
You said "space" in the question, so the pattern works with space. For matching runs of any whitespace character you can use:
>>> re.findall(r'\s+|\S+', s)
[' ', '19,301', ' ', '14,856', ' ', '18,554']
The pattern matches one or more whitespace characters or one or more non-whitespace character, for example:
>>> s=' \t\t ab\ncd\tef g '
>>> re.findall(r'\s+|\S+', s)
[' \t\t ', 'ab', '\n', 'cd', '\t', 'ef', ' ', 'g', ' ']
This question already has answers here:
Preserve whitespaces when using split() and join() in python
(3 answers)
Closed 7 years ago.
I want to split strings based on whitespace and punctuation, but the whitespace and punctuation should still be in the result.
For example:
Input: text = "This is a text; this is another text.,."
Output: ['This', ' ', 'is', ' ', 'a', ' ', 'text', '; ', 'this', ' ', 'is', ' ', 'another', ' ', 'text', '.,.']
Here is what I'm currently doing:
def classify(b):
"""
Classify a character.
"""
separators = string.whitespace + string.punctuation
if (b in separators):
return "separator"
else:
return "letter"
def tokenize(text):
"""
Split strings to words, but do not remove white space.
The input must be of type str, not bytes
"""
if (len(text) == 0):
return []
current_word = "" + text[0]
previous_mode = classify(text)
offset = 1
results = []
while offset < len(text):
current_mode = classify(text[offset])
if current_mode == previous_mode:
current_word += text[offset]
else:
results.append(current_word)
current_word = text[offset]
previous_mode = current_mode
offset += 1
results.append(current_word)
return results
It works, but it's so C-style. Is there a better way in Python?
You can use a regular expression:
import re
re.split('([\s.,;()]+)', text)
This splits on arbitrary-width whitespace (including tabs and newlines) plus a selection of punctuation characters, and by grouping the split text you tell re.sub() to include it in the output:
>>> import re
>>> text = "This is a text; this is another text.,."
>>> re.split('([\s.,;()]+)', text)
['This', ' ', 'is', ' ', 'a', ' ', 'text', '; ', 'this', ' ', 'is', ' ', 'another', ' ', 'text', '.,.', '']
If you only wanted to match spaces (and not other whitespace), replace \s with a space:
>>> re.split('([ .,;()]+)', text)
['This', ' ', 'is', ' ', 'a', ' ', 'text', '; ', 'this', ' ', 'is', ' ', 'another', ' ', 'text', '.,.', '']
Note the extra trailing empty string; a split always has a head and a tail, so text starting or ending in a split group will always have an extra empty string at the start or end. This is easily removed.
I have a string:
'Specified, if char, else 10 (default).'
I want to split it into two tuples
words=('Specified', 'if', 'char', 'else', '10', 'default')
separators=(',', ' ', ',', ' ', ' (', ').')
Does anyone have a quick solution of this?
PS: this symbol '-' is a word separator, not part of the word
import re
line = 'Specified, if char, else 10 (default).'
words = re.split(r'\)?[, .]\(?', line)
# words = ['Specified', '', 'if', 'char', '', 'else', '10', 'default', '']
separators = re.findall(r'\)?[, .]\(?', line)
# separators = [',', ' ', ' ', ',', ' ', ' ', ' (', ').']
If you really want tuples pass the results in tuple(), if you do not want words to have the empty entries (from between the commas and spaces), use the following:
words = [x for x in re.split(r'\)?[, .]\(?', line) if x]
or
words = tuple(x for x in re.split(r'\)?[, .]\(?', line) if x)
You can use regex for that.
>>> a='Specified, if char, else 10 (default).'
>>> from re import split
>>> split(",? ?\(?\)?\.?",a)
['Specified', 'if', 'char', 'else', '10', 'default', '']
But in this solution you should write that pattern yourself. If you want to use that tuple, you should convert it contents to regex pattern for that in this solution.
Regex to find all separators (assumed anything that's not alpha numeric
import re
re.findall('[^\w]', string)
I probably would first .split() on spaces into a list, then iterate through the list, using a regex to check for a character after the word boundary.
import re
s = 'Specified, if char, else 10 (default).'
w = s.split()
seperators = []
finalwords = []
for word in words:
match = re.search(r'(\w+)\b(.*)', word)
sep = '' if match is None else match.group(2)
finalwords.append(match.group(1))
seperators.append(sep)
In pass to get both separators and words you could use findall as follows:
import re
line = 'Specified, if char, else 10 (default).'
words = []
seps = []
for w,s in re.findall("(\w*)([), .(]+)", line):
words.append(w)
seps.append(s)
Here's my crack at it:
>>> p = re.compile(r'(\)? *[,.]? *\(?)')
>>> tmp = p.split('Specified, char, else 10 (default).')
>>> words = tmp[::2]
>>> separators = tmp[1::2]
>>> print words
['Specified', 'char', 'else', '10', 'default', '']
>>> print separators
[', ', ', ', ' ', ' (', ').']
The only problem is you can have a '' at the end or the beginning of words if there is a separator at the beginning/end of the sentence without anything before/after it. However, that is easily checked for and eliminated.