As part of preprocessing my data, I want to be able to replace anything that comes with a slash till the occurrence of space with empty string. For example, \fs24 need to be replaced with empty or \qc23424 with empty. There could be multiple occurrences of tags with slashes which I want to remove. I have created a "tags to be eradicated" list which I aim to consume in a regular expression to clean the extracted text.
Input String: This is a string \fs24 and it contains some texts and tags \qc23424. which I want to remove from my string.
Expected output: This is a string and it contains some texts and tags. which I want to remove from my string.
I am using the regular expression based replace function in Python:
udpated = re.sub(r'/\fs\d+', '')
However, this is not fetching the desired result. Alternately, I have built an eradicate list and replacing that from a loop from top to lower number but this is a performance killer.
Assuming a 'tag' can also occur at the very beginning of your string, and avoid selecting false positives, maybe you could use:
\s?(?<!\S)\\[a-z\d]+
And replace with nothing. See an online demo.
\s? - Optionally match a whitespace character (if a tag is mid-string and therefor preceded by a space);
(?<!\S) - Assert position is not preceded by a non-whitespace character (to allow a position at the start of your input);
\\ - A literal backslash.
[a-z\d]+ - 1+ (Greedy) Characters as per given class.
First, the / doesn't belong in the regular expression at all.
Second, even though you are using a raw string literal, \ itself has special meaning to the regular expression engine, so you still need to escape it. (Without a raw string literal, you would need '\\\\fs\\d+'.) The \ before f is meant to be used literally; the \ before d is part of the character class matching the digits.
Finally, sub takes three arguments: the pattern, the replacement text, and the string on which to perform the replacement.
>>> re.sub(r'\\fs\d+', '', r"This is a string \fs24 and it contains...")
'This is a string and it contains...'
Does that work for you?
re.sub(
r"\\\w+\s*", # a backslash followed by alphanumerics and optional spacing;
'', # replace it with an empty string;
input_string # in your input string
)
>>> re.sub(r"\\\w+\s*", "", r"\fs24 hello there")
'hello there'
>>> re.sub(r"\\\w+\s*", "", "hello there")
'hello there'
>>> re.sub(r"\\\w+\s*", "", r"\fs24hello there")
'there'
>>> re.sub(r"\\\w+\s*", "", r"\fs24hello \qc23424 there")
'there'
'\\' matches '\' and 'w+' matches a word until space
import re
s = r"""This is a string \fs24 and it contains some texts and tags \qc23424. which I want to remove from my string."""
re.sub(r'\\\w+', '', s)
output:
'This is a string and it contains some texts and tags . which I want to remove from my string.'
I tried this and it worked fine for me:
def remover(text, state):
removable = text.split("\\")[1]
removable = removable.split(" ")[0]
removable = "\\" + removable + " "
text = text.replace(removable, "")
state = True if "\\" in text else False
return text, state
text = "hello \\I'm new here \\good luck"
state = True
while state:
text, state = remover(text, state)
print(text)
Related
I want to get the content between single quotes, but only if it contains a certain word (i.e 'sample_2'). It additionally should not match ones with white space.
Input example: (The following should match and return only: ../sample_2/file and sample_2/file)
['asdf', '../sample_2/file', 'sample_2/file', 'example with space', sample_2, sample]
Right now I just have that matched the first 3 items in the list:
'(.\S*?)'
I can't seem to find the right regex that would return those containing the word 'sample_2'
If you want specific words/characters you need to have them in the regular expression and not use the '\S'. The \S is the equivalent to [^\r\n\t\f\v ] or "any non-whitespace character".
import re
teststr = "['asdf', '../sample_2/file', 'sample_2/file', 'sample_2 with spaces','example with space', sample_2, sample]"
matches = re.findall(r"'([^\s']*sample_2[^\s]*?)',", teststr)
# ['../sample_2/file', 'sample_2/file']
Based on your wording, you suggest the desired word can change. In that case, I would recommend using re.compile() to dynamically create a string which then defines the regular expression.
import re
word = 'sample_2'
teststr = "['asdf', '../sample_2/file', 'sample_2/file', ' sample_2 with spaces','example with space', sample_2, sample]"
regex = re.compile("'([^'\\s]*"+word+"[^\\s]*?)',")
matches = regex.findall(teststr)
# ['../sample_2/file', 'sample_2/file']
Also if you haven't heard of this tool yet, check out regex101.com. I always build my regular expressions here to make sure I get them correct. It gives you the references, explanation of what is happening and even lets you test it right there in the browser.
Explanation of regex
regex = r"'([^\s']*sample_2[^\s]*?)',"
Find first apostrophe, start group capture. Capture anything except a whitespace character or the corresponding ending apostrophe. It must see the letters "sample_2" before accepting any non-whitespace character. Stop group capture when you see the closing apostrophe and a comma.
Note: In python, a string " or ' prepositioned with the character 'r' means the text is compiled as a regular expression. Strings with the character 'r' also do not require double-escape '\' characters.
I have a decent familiarity with regex but this is tricky. I need to find instances like this from a SQL case statement:
when col_name = 'this can be a word or sentence'
I can match the above when it's just one word, but when it's more than one word it's not working.
s = """when col_name = 'a sentence of words'"""
x = re.search("when\s(\w+)\s*=\s*\'(\w+)", s)
if x:
print(x.group(1)) # this returns "col_name"
print(x.group(2)) # this returns "a"
I want group(2) to return "a sentence of words" but I'm just getting the first word. That part could either be one word or several. How to do it?
When I add in the second \', then I get no match:
x = re.search("when\s(\w+)\s*=\s*\'(\w+)\'", s)
You may match all characters other than single quotation mark rather than matching letters, digits and connector punctuation ("word" chars) with the Group 2 pattern:
import re
s = """when col_name = 'a sentence of words'"""
x = re.search(r"when\s+(\w+)\s*=\s*'([^']+)", s)
if x:
print(x.group(1)) # this returns "col_name"
print(x.group(2)) # this returns "a sentence of words"
See the Python demo
The [^'] is a negated character class that matches any char but a single quotation mark, see the regex demo.
In case the string can contain escaped single quotes, you may consider replacing [^'] with
If the escape char is ': ([^']*(?:''[^']*)*)
If the escape char is \: ([^\\']*(?:\\.[^'\\]*)*).
Note the use of the raw string literal to define the regex pattern (all backslashes are treated as literal backslashes inside it).
I have a huge string which contains emotions like "\u201d", AS WELL AS "\advance\"
all that I need is to remove back slashed so that:
- \u201d = \u201d
- \united\ = united
(as it breaks the process of uploading it to BigQuery database)
I know it should be somehow this way:
string.replace('\','') But not sure how to keep \u201d emotions.
ADDITIONAL:
Example of Unicode emotions
\ud83d\udc9e
\u201c
\u2744\ufe0f\u2744\ufe0f\u2744\ufe0f
You can split on all '\' and then use a regex to replace your emotions with adding leading '\'
s = '\\advance\\\\united\\ud83d\\udc9e\\u201c\\u2744\\ufe0f\\u2744\\ufe0f\\u2744\\ufe0f'
import re
print(re.sub('(u[a-f0-9]{4})',lambda m: '\\'+m.group(0),''.join(s.split('\\'))))
As your emotions are 'u' and 4 hexa numbers, 'u[a-f0-9]{4}' will match them all, and you just have to add leading backslashes
First of all, you delete every '\' in the string with either ''.join(s.split('\\')) or s.replace('\\')
And then we match every "emotion" with the regex u[a-f0-9]{4} (Which is u with 4 hex letters behind)
And with the regex sub, you replace every match with a leading \\
You could simply add the backslash in front of your string after replacement if your string starts with \u and have at least one digit.
import re
def clean(s):
re1='(\\\\)' # Any Single Character "\"
re2='(u)' # Any Single Character "u"
re3='.*?' # Non-greedy match on filler
re4='(\\d)' # Any Single Digit
rg = re.compile(re1+re2+re3+re4,re.IGNORECASE|re.DOTALL)
m = rg.search(s)
if m:
r = '\\'+s.replace('\\','')
else:
r = s.replace('\\','')
return r
a = '\\u123'
b = '\\united\\'
c = '\\ud83d'
>>> print(a, b, c)
\u123 \united\ \ud83d
>>> print(clean(a), clean(b), clean(c))
\u123 united \ud83d
Of course, you have to split your sting if multiple entries are in the same line:
string = '\\u123 \\united\\ \\ud83d'
clean_string = ' '.join([clean(word) for word in string.split()])
You can use this simple method to replace the last occurence of your character backslash:
Check the code and use this method.
def replace_character(s, old, new):
return (s[::-1].replace(old[::-1],new[::-1], 1))[::-1]
replace_character('\advance\', '\','')
replace_character('\u201d', '\','')
Ooutput:
\advance
\u201d
You can do it as simple as this
text = text.replace(text[-1],'')
Here you just replace the last character with nothing
having a string
string=
""""$deletedFields":["standardizedSkillUrn","standardizedSkill"],"entityUrn":"urn:li:fs_skill:(ACoAAAIv9SQBMzclPm3CZzL1QceTH5W0VrsdxbE,3)","name":"Finance","$type":"voyager.identity.profile.Skill"},{"$deletedFields":["standardizedSkillUrn","standardizedSkill"],"entityUrn":"urn:li:fs_skill:(ACoAAAIv9SQBMzclPm3CZzL1QceTH5W0VrsdxbE,22)","name":"Financial ["standardizedSkillUrn","standardizedSkill"],"entityUrn":"urn:li:fs_skill:(ACoAAAIv9SQBMzclPm3CZzL1QceTH5W0VrsdxbE,34)","name":"Due
Diligence","name":"Strategy""""
What reguar expression can i use to retrieve values after "name": to get Due Dilligence, Financial, and Finance
i have tried
match = re.compile(r'"name"\:(.\w+)')
match.findall(string)
but it returns
['"Finance', '"Financial', '"Due', '"Financial', '"Strategy']
The Due Diligence is split and i want both words as one.
Your whitespace is not detected by regex because /w only searches for non-special characters.
"name"\:(.\w+\s*\w*) accounts for any possible spaces with an extra word (Will not work for three words, but will in your situation)
"name"\:(.\w+\s*\w*"?) accounts for the quotations " at the end of each one but doesn't get Financial.
Example
Edit: Fixed second regex for "Financial
I would use the non-hungry .*? expression with a trailing quote:
import re
string = """$deletedFields":["standardizedSkillUrn","standardizedSkill"],"entityUrn":"urn:li:fs_skill:(ACoAAAIv9SQBMzclPm3CZzL1QceTH5W0VrsdxbE,3)","name":"Finance","$type":"voyager.identity.profile.Skill"},{"$deletedFields":["standardizedSkillUrn","standardizedSkill"],"entityUrn":"urn:li:fs_skill:(ACoAAAIv9SQBMzclPm3CZzL1QceTH5W0VrsdxbE,22)","name":"Financial ["standardizedSkillUrn","standardizedSkill"],"entityUrn":"urn:li:fs_skill:(ACoAAAIv9SQBMzclPm3CZzL1QceTH5W0VrsdxbE,34)","name":"Due Diligence","name":"Strategy"""
# With the leading double quote
match = re.compile(r'"name"\:(".*?)["\[]')
a = match.findall(string)
print a
# Stripping out the leading double quote
match = re.compile(r'"name"\:"(.*?)["\[]')
b = match.findall(string)
print b
And the final output is:
['"Finance', '"Financial ', '"Due Diligence']
['Finance', 'Financial ', 'Due Diligence']
I have a string looks like this:
oldString="this is my {{string-d}}" => "this is my {{(string-d)}}"
oldString2="this is my second{{ new_string-d }}" => "this is my second{{ (new_string-d) }}"
oldString2="this is my second new_string-d " => "this is my second (new_string-d) "
oldString2="this is my second new[123string]-d " => "this is my second (new[123string]-d) "
I want to add brackets whenever I see "-d" right after it and before the word that is attached to it.
I wrote a code that looks for the pattern "-d" in strings and partition the string after finding the pattern to 3 partitions before "-d", after "-d" and "-d" itself then I check the block before "-d" until I find whitespace or "{" and stop and add brackets. my code looks like this:
P.S. I have many files that I read from them and try to modify the string there the example above is just for demonstrating what I'm trying to do.
if ('-d') in oldString:
p = oldString.partition('-d')
v = p[p.index('-d')-1]
beforeString=''
for i in reversed(v):
if i != ' ' or i != '{':
beforeString=i+beforeString
indexNew = v.index(i)
outPutLine = v[:indexNew]+'('+v[indexNew:]
newString = outPutLine + '-d' + ' )'
print newString
the result of running the code will be:
newString = "(this is my {{string-d )"
as you can see that the starting bracket is before "this" instead of before "string" why is this happening? also, I'm not sure if this is the best way to do this kind of find and replace any suggestions would be much appreciated.
>>> import re
>>> oldString = "this is my {{string-d}}"
>>> oldString2 = "this is my second{{ new_string-d }}"
>>> re.sub(r"(\w*-d)", r"(\1)", oldString)
'this is my {{(string-d)}}'
>>> re.sub(r"(\w*-d)", r"(\1)", oldString2)
'this is my second{{ (new_string-d) }}'
Note that this matches "words" assuming that a word is composed of only letters, numbers, and underscores.
Here's a more thorough breakdown of what's happening:
An r before a string literal means the string is a "raw string". It prevents Python from interpreting characters as an escape sequence. For instance, r"\n" is a slash followed by the letter n, rather than being interpreted as a single newline character. I like to use raw strings for my regex patterns, even though it's not always necessary.
the parentheses surrounding \w*-d is a capturing group. It indicates to the regex engine that the contents of the group should be saved for later use.
the sequence \w means "any alphanumeric character or underscore".
* means "zero or more of the preceding item". \w* together means "zero or more alphanumeric characters or underscores".
-d means "a hyphen followed by the letter d.
All together, (\w*-d) means "zero or more alphanumeric characters or underscores, followed by a hyphen and the letter d. Save all of these characters for later use."
The second string describes what the matched data should be replaced with. "\1" means "the contents of the first captured group". The parentheses are just regular parentheses. All together, (\1) in this context means "take the saved content from the captured group, surround it in parentheses, and put it back into the string".
If you want to match more characters than just alphanumeric and underscore, you can replace \w with whatever collection of characters you want to match.
>>> re.sub(r"([\w\.\[\]]*-d)", r"(\1)", "{{startingHere[zero1].my_string-d }}")
'{{(startingHere[zero1].my_string-d) }}'
If you also want to match words ending with "-d()", you can match a parentheses pair with \(\) and mark it as optional using ?.
>>> re.sub(r"([\w\.\[\]]*-d(\(\))?)", r"(\1)", "{{startingHere[zero1].my_string-d() }}")
'{{(startingHere[zero1].my_string-d()) }}'
If you want the bracketing to only take place inside double curly braces, you need something like this:
re.sub(r'({{\s*)([^}]*-d)(\s*}})', r'\1(\2)\3', s)
Breaking that down a bit:
# the target pattern
r'({{\s*)([^}]*-d)(\s*}})'
# ^^^^^^^ capture group 1, opening {{ plus optional space
# ^^^^^^^^^ capture group 2, non-braces plus -d
# ^^^^^^^ capture 3, spaces plus closing }}
The replacement r'\1(\2)\3' just assembles the groups, with
parenthesis around the middle one.
Putting it together:
import re
def quote_string_d(s):
return re.sub(r'({{\s*)([^}]*-d)(\s*}})', r'\1(\2)\3', s)
print(quote_string_d("this is my {{string-d}}"))
print(quote_string_d("this is my second{{ new_string-d }}"))
print(quote_string_d("this should not be quoted other_string-d "))
Output:
this is my {{(string-d)}}
this is my second{{ (new_string-d) }}
this should not be quoted other_string-d
Note the third instance does not get the parentheses, because it's not inside {{ }}.