I am looking for some thoughts on how I would be able to accomplish these tasks:
Allow the first occurrence of a problem_word, but ban any following uses of it and the rest of the problem words.
No modifications to the original document (.txt file). Only modify for print().
Keep the same structure of the email. If there are line breaks, or tabs, or weird spacings, let them keep their integrity.
Here is the code sample:
import re
# Sample email is "Hello, banned1. This is banned2. What is going on with
# banned 3? Hopefully banned1 is alright."
sample_email = open('email.txt', 'r').read()
# First use of any of these words is allowed; those following are banned
problem_words = ['banned1', 'banned2', 'banned3']
# TODO: Filter negative_words into overused_negative_words
banned_problem_words = []
for w in problem_words:
if sample_email.count(f'\\b{w}s?\\b') > 1:
banned_problem_words.append(w)
pattern = '|'.join(f'\\b{w}s?\\b' for w in banned_problem_words)
def list_check(email, pattern):
return re.sub(pattern, 'REDACTED', email, flags=re.IGNORECASE)
print(list_check(sample_email, pattern))
# Result should be: "Hello, banned1. This is REDACTED. What is going on with
# REDACTED? Hopefully REDACTED is alright."
The repl argument of re.sub can take a function that takes a match object and returns the replacement string. Here is my solution:
import re
sample_email = open('email.txt', 'r').read()
# First use of any of these words is allowed; those following are banned
problem_words = ['banned1', 'banned2', 'banned3']
pattern = '|'.join(f'\\b{w}\\b' for w in problem_words)
occurrences = 0
def redact(match):
global occurrences
occurrences += 1
if occurrences > 1:
return "REDACTED"
return match.group(0)
replaced = re.sub(pattern, redact, sample_email, flags=re.IGNORECASE)
print(replaced)
(As a further note, string.count doesn't support regex, but there is no need to count)
Related
I have a string and rules/mappings for replacement and no-replacements.
E.g.
"This is an example sentence that needs to be processed into a new sentence."
"This is a second example sentence that shows how 'sentence' in 'sentencepiece' should not be replaced."
Replacement rules:
replace_dictionary = {'sentence': 'processed_sentence'}
no_replace_set = {'example sentence'}
Result:
"This is an example sentence that needs to be processed into a new processed_sentence."
"This is a second example sentence that shows how 'processed_sentence' in 'sentencepiece' should not be replaced."
Additional criteria:
Only replace if case is matched, i.e. case matters.
Whole words replacement only, interpunction should be ignored, but kept after replacement.
I was thinking what would the cleanest way to solve this problem in Python 3.x be?
Based on the answer of demongolem.
UPDATE
I am sorry, I missed the fact, that only whole words should be replaced. I updated my code and even generalized it for usage in a function.
def replace_whole(sentence, replace_token, replace_with, dont_replace):
rx = f"[\"\'\.,:; ]({replace_token})[\"\'\.,:; ]"
iter = re.finditer(rx, sentence)
out_sentence = ""
found = []
indices = []
for m in iter:
indices.append(m.start(0))
found.append(m.group())
context_size=len(dont_replace)
for i in range(len(indices)):
context = sentence[indices[i]-context_size:indices[i]+context_size]
if dont_replace in context:
continue
else:
# First replace the word only in the substring found
to_replace = found[i].replace(replace_token, replace_with)
# Then replace the word in the context found, so any special token like "" or . gets taken over and the context does not change
replace_val = context.replace(found[i], to_replace)
# finally replace the context found with the replacing context
out_sentence = sentence.replace(context, replace_val)
return out_sentence
Use regular expressions for finding all occurences and values of your string (as we need to check whether is a whole word or embedded in any kind of word), by using finditer(). You might need to adjust the rx to what your definition of "whole word" is. Then get the context around these values of the size of your no_replace rule. Then check, whether the context contains your no_replace string.
If not, you may replace it, by using replace() for the word only, then replace the occurence of the word in the context, then replace the context in the whole text. That way the replacing process is nearly unique and no weird behaviour should happen.
Using your examples, this leads to:
replace_whole(sen2, "sentence", "processed_sentence", "example sentence")
>>>"This is a second example sentence that shows how 'processed_sentence' in 'sentencepiece' should not be replaced."
and
replace_whole(sen1, "sentence", "processed_sentence", "example sentence")
>>>'This is an example sentence that needs to be processed into a new processed_sentence.'
After some research, this is what I believe to be the best and cleanest solution to my problem. The solution works by calling the match_fun whenever a match has been found, and the match_fun only performs the replacement, if and only if, there is no "no-replace-phrase" overlapping with the current match. Let me know if you need more clarification or if you believe something can be improved.
replace_dict = ... # The code below assumes you already have this
no_replace_dict = ...# The code below assumes you already have this
text = ... # The text on input.
def match_fun(match: re.Match):
str_match: str = match.group()
if str_match not in cls.no_replace_dict:
return cls.replace_dict[str_match]
for no_replace in cls.no_replace_dict[str_match]:
no_replace_matches_iter = re.finditer(r'\b' + no_replace + r'\b', text)
for no_replace_match in no_replace_matches_iter:
if no_replace_match.start() >= match.start() and no_replace_match.start() < match.end():
return str_match
if no_replace_match.end() > match.start() and no_replace_match.end() <= match.end():
return str_match
return cls.replace_dict[str_match]
for replace in cls.replace_dict:
pattern = re.compile(r'\b' + replace + r'\b')
text = pattern.sub(match_fun, text)
I am converting some code to micropython and I got stuck on a particular regular expression.
In python my code is
import re
line = "0-1:24.2.1(180108205500W)(00001.290*m3)"
between_brackets = '\(.*?\)'
brackettext = re.findall(between_brackets, line)
gas_date_str = read_date_time(brackettext[0])
gas_val = read_gas(brackettext[1])
# gas_date_str and gas_val take the string between brackets
# and return a value that can later be used
micropython only implements a limited set of re functions
how do I achieve the same with only the limited functions available?
You could do something along the following lines. Repeatedly use re.search while consuming the string. The implementation here uses a generator function:
import re
def findall(pattern, string):
while True:
match = re.search(pattern, string)
if not match:
break
yield match.group(0)
string = string[match.end():]
>>> list(findall(r'\(.*?\)', "0-1:24.2.1(180108205500W)(00001.290*m3)"))
['(180108205500W)', '(00001.290*m3)']
You can write a method using re.search() that returns a list of all matches:
import re
def find_all(regex, text):
match_list = []
while True:
match = re.search(regex, text)
if match:
match_list.append(match.group(0))
text = text[match.end():]
else:
return match_list
Also, note that your between_brackets regex will not take care of nested brackets:
re.findall('\(.*?\)', "(ac(ssc)xxz)")
>>> ['(ac(ssc)']
for example is the string is "abbacdeffel" and the pattern being "xyyx" replaced with "1234"
so it would result from "abbacdeffel" to "1234cd1234l"
I have tried to think this out but I couldnt come up with anything. At first I thought maybe dictionary could help but still nothing came to mind.
What you're looking to do can be accomplished by using regex, or more commonly known as, Regular Expressions. Regular Expressions in programming enables you to extract what you want and just what you want from a string.
In your case, you want to match the string with the pattern abba so using the following regex:
(\w+)(\w+)\2\1
https://regex101.com/r/hP8lA3/1
You can match two word groups and use backreferences to make sure that the second group comes first, then the first group.
So implementing this in python code looks like this:
First, import the regex module in python
import re
Then, declare your variable
text = "abbacdeffel"
The re.finditer returns an iterable so you can iterate through all the groups
matches = re.finditer(r"(\w)(\w)\2\1", text)
Go through all the matches that the regexp found and replace the pattern with "1234"
for match in matches:
text = text.replace(match.group(0), "1234")
For debugging:
print(text)
Complete Code:
import re
text = "abbacdeffel"
matches = re.finditer(r"(\w)(\w)\2\1", text)
for match in matches:
text = text.replace(match.group(0), "1234")
print(text)
You can learn more about Regular Expressions here: https://regexone.com/references/python
New version of code (there was a bug):
def replace_with_pattern(pattern, line, replace):
from collections import OrderedDict
set_of_chars_in_pattern = set(pattern)
indice_start_pattern = 0
output_line = ""
while indice_start_pattern < len(line):
potential_end_pattern = indice_start_pattern + len(pattern)
subline = line[indice_start_pattern:potential_end_pattern]
print(subline)
set_of_chars_in_subline = set(subline)
if len(set_of_chars_in_subline)!= len(set_of_chars_in_pattern):
output_line += line[indice_start_pattern]
indice_start_pattern +=1
continue
map_of_chars = OrderedDict()
liste_of_chars_in_pattern = []
for char in pattern:
if char not in liste_of_chars_in_pattern:
liste_of_chars_in_pattern.append(char)
print(liste_of_chars_in_pattern)
for subline_char in subline:
if subline_char not in map_of_chars.values():
map_of_chars[liste_of_chars_in_pattern.pop(0)] =subline_char
print(map_of_chars)
wanted_subline = ""
for char_of_pattern in pattern:
wanted_subline += map_of_chars[char_of_pattern]
print("wanted_subline =" + wanted_subline)
if subline == wanted_subline:
output_line += replace
indice_start_pattern += len(pattern)
else:
output_line += line[indice_start_pattern]
indice_start_pattern += 1
return output_line
some test :
test1 = replace_with_pattern("xyyx", "abbacdeffel", "1234")
test2 = replace_with_pattern("abbacdeffel", "abbacdeffel", "1234")
print(test1, test2)
=> 1234cd1234l 1234
Here goes my attempt:
([a-zA-Z])(?!\1)([a-zA-Z])\2\1
Assuming you want to match letters only (if other ranges, change both [a-zA-Z] as appropriate, we have:
([a-zA-Z])
Find the first character, and note it so we can later refer to it with \1.
(?!\1)
Check to see if the next character is not the same as the first, but without advancing the search pointer. This is to prevent aaaa being accepted. If aaaa is OK, just remove this subexpression.
([a-zA-Z])
Find the second character, and note it so we can later refer to it with \2.
\2\1
Now find the second again, then the first again, so we match the full abba pattern.
And finally, to do a replace operation, the full command would be:
import re
re.sub(r'([a-zA-Z])(?!\1)([a-zA-Z])\2\1',
'1234',
'abbacdeffelzzzz')
The r at the start of the regex pattern is to prevent Python processing the backslashes. Without it, you would need to do:
import re
re.sub('([a-zA-Z])(?!\\1)([a-zA-Z])\\2\\1',
'1234',
'abbacdeffelzzzz')
Now, I see the spec has expanded to a user-defined pattern; here is some code that will build that pattern:
import re
def make_re(pattern, charset):
result = ''
seen = []
for c in pattern:
# Is this a letter we've seen before?
if c in seen:
# Yes, so we want to match the captured pattern
result += '\\' + str(seen.index(c)+1)
else:
# No, so match a new character from the charset,
# but first exclude already matched characters
for i in xrange(len(seen)):
result += '(?!\\' + str(i + 1) + ')'
result += '(' + charset + ')'
# Note we have seen this letter
seen.append(c)
return result
print re.sub(make_re('xzzx', '\\d'), 'abba', 'abba1221b99999889')
print re.sub(make_re('xyzxyz', '[a-z]'), '123123', 'abcabc zyxzyyx zyzzyz')
Outputs:
abbaabbab9999abba
123123 zyxzyyx zyzzyz
I struggle to understand the group method in Python's regular expressions library. In this context, I try to do substitutions on a string depending on the matching object.
That is, I want to replace the matched objects (+ and \n in this example) with a particular string in the my_dict dictionary (with rep1 and rep2 respectively).
As seen from this question and answer,
I have tried this:
content = '''
Blah - blah \n blah * blah + blah.
'''
regex = r'[+\-*/]'
for mobj in re.finditer(regex, content):
t = mobj.lastgroup
v = mobj.group(t)
new_content = re.sub(regex, repl_func(mobj), content)
def repl_func(mobj):
my_dict = { '+': 'rep1', '\n': 'rep2'}
try:
match = mobj.group(0)
except AttributeError:
match = ''
else:
return my_dict.get(match, '')
print(new_content)
But I get None for t followed by an IndexError when computing v.
Any explanations and example code would be appreciated.
Despite of Wiktor's truly pythonic answer, there's still the question why the OP's orginal algorithm wouldn't work.
Basically there are 2 issues:
The call of new_content = re.sub(regex, repl_func(mobj), content) will substitute all matches of regex with the replacement value of the very first match.
The correct call has to be new_content = re.sub(regex, repl_func, content).
As documented here, repl_func gets invoked dynamically with the current match object!
repl_func(mobj) does some unnecessary exception handling, which can be simplified:
my_dict = {'\n': '', '+':'rep1', '*':'rep2', '/':'rep3', '-':'rep4'}
def repl_func(mobj):
global my_dict
return my_dict.get(mobj.group(0), '')
This is equivalent to Wiktor's solution - he just got rid of the function definition itself by using a lambda expression.
With this modification, the for mobj in re.finditer(regex, content): loop has become superfluos, as it does the same calculation multiple times.
Just for the sake of completeness here is a working solution using re.finditer(). It builds the result string from the matched slices of content:
my_regx = r'[\n+*/-]'
my_dict = {'\n': '', '+':'rep1' , '*':'rep2', '/':'rep3', '-':'rep4'}
content = "A*B+C-D/E"
res = ""
cbeg = 0
for mobj in re.finditer(my_regx, content):
# get matched string and its slice indexes
mstr = mobj.group(0)
mbeg = mobj.start()
mend = mobj.end()
# replace matched string
mrep = my_dict.get(mstr, '')
# append non-matched part of content plus replacement
res += content[cbeg:mbeg] + mrep
# set new start index of remaining slice
cbeg = mend
# finally add remaining non-matched slice
res += content[cbeg:]
print (res)
The r'[+\-*/]' regex does not match a newline, so your '\n': 'rep2' would not be used. Else, add \n to the regex: r'[\n+*/-]'.
Next, you get None because your regex does not contain any named capturing groups, see re docs:
match.lastgroup
The name of the last matched capturing group, or None if the group didn’t have a name, or if no group was matched at all.
To replace using the match, you do not even need to use re.finditer, use re.sub with a lambda as the replacement:
import re
content = '''
Blah - blah \n blah * blah + blah.
'''
regex = r'[\n+*/-]'
my_dict = { '+': 'rep1', '\n': 'rep2'}
new_content = re.sub(regex, lambda m: my_dict.get(m.group(),""), content)
print(new_content)
# => rep2Blah blah rep2 blah blah rep1 blah.rep2
See the Python demo
The m.group() gets the whole match (the whole match is stored in match.group(0)). If you had a pair of unescaped parentheses in the pattern, it would create a capturing group and you could access the first one with m.group(1), etc.
I have written a really good program that uses text files as word banks for generating sentences from sentence skeletons. An example:
The skeleton
"The noun is good at verbing nouns"
can be made into a sentence by searching a word bank of nouns and verbs to replace "noun" and "verb" in the skeleton. I would like to get a result like
"The dog is good at fetching sticks"
Unfortunately, the handy replace() method was designed for speed, not custom functions in mind. I made methods that accomplish the task of selecting random words from the right banks, but doing something like skeleton = skeleton.replace('noun', getNoun(file.txt)) replaces ALL instances of 'noun' with the single call of getNoun(), instead of calling it for each replacement. So the sentences look like
"The dog is good at fetching dogs"
How might I work around this feature of replace() and make my method get called for each replacement? My minimum length code is below.
import random
def getRandomLine(rsv):
#parameter must be a return-separated value text file whose first line contains the number of lines in the file.
f = open(rsv, 'r') #file handle on read mode
n = int(f.readline()) #number of lines in file
n = random.randint(1, n) #line number chosen to use
s = "" #string to hold data
for x in range (1, n):
s = f.readline()
s = s.replace("\n", "")
return s
def makeSentence(rsv):
#parameter must be a return-separated value text file whose first line contains the number of lines in the file.
pattern = getRandomLine(rsv) #get a random pattern from file
#replace word tags with random words from matching files
pattern = pattern.replace('noun', getRandomLine('noun.txt'))
pattern = pattern.replace('verb', getRandomLine('verb.txt'))
return str(pattern);
def main():
result = makeSentence('pattern.txt');
print(result)
main()
The re module's re.sub function does the job str.replace does, but with far more abilities. In particular, it offers the ability to pass a function for the replacement, rather than a string. The function is called once for each match with a match object as an argument and must return the string that will replace the match:
import re
pattern = re.sub('noun', lambda match: getRandomLine('noun.txt'), pattern)
The benefit here is added flexibility. The downside is that if you don't know regexes, the fact that the replacement interprets 'noun' as a regex may cause surprises. For example,
>>> re.sub('Aw, man...', 'Match found.', 'Aw, manatee.')
'Match found.e.'
If you don't know regexes, you may want to use re.escape to create a regex that will match the raw text you're searching for even if the text contains regex metacharacters:
>>> re.sub(re.escape('Aw, man...'), 'Match found.', 'Aw, manatee.')
'Aw, manatee.'
I don't know if you are asking to edit your code or to write new code, so I wrote new code:
import random
verbs = open('verb.txt').read().split()
nouns = open('noun.txt').read().split()
def makeSentence(sent):
sent = sent.split()
for k in range(0, len(sent)):
if sent[k] == 'noun':
sent[k] = random.choice(nouns)
elif sent[k] == 'nouns':
sent[k] = random.choice(nouns)+'s'
elif sent[k] == 'verbing':
sent[k] = random.choice(verbs)
return ' '.join(sent)
var = raw_input('Enter: ')
print makeSentence(var)
This runs as:
$ python make.py
Enter: the noun is good at verbing nouns
the mouse is good at eating cats