Develop Reference

Can't use '\1' backreference to capture-group in a function call in re.sub() repr expression

I have a string S = '02143' and a list A = ['a','b','c','d','e']. I want to replace all those digits in 'S' with their corresponding element in list A.
For example, replace 0 with A[0], 2 with A[2] and so on. Final output should be S = 'acbed'.
I tried:
S = re.sub(r'([0-9])', A[int(r'\g<1>')], S)
However this gives an error ValueError: invalid literal for int() with base 10: '\\g<1>'. I guess it is considering backreference '\g<1>' as a string. How can I solve this especially using re.sub and capture-groups, else alternatively?

The reason the re.sub(r'([0-9])',A[int(r'\g<1>')],S) does not work is that \g<1> (which is an unambiguous representation of the first backreference otherwise written as \1) backreference only works when used in the string replacement pattern. If you pass it to another method, it will "see" just \g<1> literal string, since the re module won't have any chance of evaluating it at that time. re engine only evaluates it during a match, but the A[int(r'\g<1>')] part is evaluated before the re engine attempts to find a match.
That is why it is made possible to use callback methods inside re.sub as the replacement argument: you may pass the matched group values to any external methods for advanced manipulation.
See the re documentation:
re.sub(pattern, repl, string, count=0, flags=0)
If repl is a function, it is called for every non-overlapping
occurrence of pattern. The function takes a single match object
argument, and returns the replacement string.
Use
import re
S = '02143'
A = ['a','b','c','d','e']
print(re.sub(r'[0-9]',lambda x: A[int(x.group())],S))
See the Python demo
Note you do not need to capture the whole pattern with parentheses, you can access the whole match with x.group().

Is there a way to use regular expressions in the replacement string in re.sub() in Python?

In Python in the re module there is the following function:
re.sub(pattern, repl, string, count=0, flags=0) – Return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in string by the replacement repl. If the pattern isn’t found, string is returned unchanged.
I've found it can work like this:
print re.sub('[a-z]*\d+','lion','zebra432') # prints 'lion'
I was wondering, is there an easy way to use regular expressions in the replacement string, so that the replacement string contains part of the original regular expression/original string? Specifically, can I do something like this (which doesn't work)?
print re.sub('[a-z]*\d+', 'lion\d+', 'zebra432')
I want that to print 'lion432'. Obviously, it does not. Rather, it prints 'lion\d+'. Is there an easy way to use parts of the matching regular expression in the replacement string?
By the way, this is NOT a special case. Please do NOT assume that the number will always come at the end, the words will always come in the beginning, etc. I want to know a solution to all regexes in general.
Thanks

Place \d+ in a capture group (...) and then use \1 to refer to it:
>>> import re
>>> re.sub('[a-z]*(\d+)', r'lion\1', 'zebra432')
'lion432'
>>>
>>> # You can also refer to more than one capture group
>>> re.sub('([a-z]*)(\d+)', r'\1lion\2', 'zebra432')
'zebralion432'
>>>
From the docs:
Backreferences, such as \6, are replaced with the substring matched
by group 6 in the pattern.
Note that you will also need to use a raw-string so that \1 is not treated as an escape sequence.

re.sub repl function returning \1 does not replace the group

I am trying to write a generic replace function for a regex sub operation in Python (trying in both 2 and 3) Where the user can provide a regex pattern and a replacement for the match. This could be just a simple string replacement to replacing using the groups from the match.
In the end, I get from the user a dictionary in this form:
regex_dict = {pattern:replacement}
When I try to replace all the occurrences of a pattern via this command, the replacement works for replacements for a group number, (such as \1) and I call the following operation:
re.sub(pattern, regex_dict[pattern], text)
This works as expected, but I need to do additional stuff when a match is found. Basically, what I try to achieve is as follows:
replace_function(matchobj):
result = regex_dict[matchobj.re]
##
## Do some other things
##
return result
re.sub(pattern, replace_function, text)
I see that this works for normal replacements, but the re.sub does not use the group information to get the match when the function is used.
I also tried to convert the \1 pattern to \g<1>, hoping that the re.sub would understand it, but to no avail.
Am I missing something vital?
Thanks in advance!
Additional notes: I compile the pattern using strings as in bytes, and the replacements are also in bytes. I have non-Latin characters in my pattern, but I read everything in bytes, including the text where the regex substitution will operate on.
EDIT
Just to clarify, I do not know in advance what kind of replacement the user will provide. It could be some combination of normal strings and groups, or just a string replacement.
SOLUTION
replace_function(matchobj):
repl = regex_dict[matchobj.re]
##
## Do some other things
##
return matchobj.expand(repl)
re.sub(pattern, replace_function, text)

I suspect you're after .expand, if you've got a compiled regex object (for instance), you can provide a string to be taken into consideration for the replacements, eg:
import re
text = 'abc'
# This would be your key in the dict
rx = re.compile('a(\w)c')
# This would be the value for the key (the replacement string, eg: `\1\1\1`)
res = rx.match(text).expand(r'\1\1\1')
# bbb

Using REGEX over split string

I'm trying to split a string into sub string, splitting by the 'AND' term, and after that
clean each sub string from "garbage".
The following code get the error:
AttributeError: 'NoneType' object has no attribute 'group'
import re
def fun(self, str):
for subStr in str.split('AND'):
p = re.compile('[^"()]+')
m = p.match(subStr)
print (m.group())

It means the match is not found, and it returned None.
Note that you might want to use re.search here instead of re.match. re.match matches only at the beginning of the string while re.search can search anywhere in the string.
From the docs:
Python offers two different primitive operations based on regular
expressions: re.match() checks for a match only at the beginning of
the string, while re.search() checks for a match anywhere in the
string (this is what Perl does by default).
If you already know that then you can handle that None using:
if m:
print (m.group())
else:
#do something else

If the code above is what you really want to do, wouldn't it be easier to remove the garbage first using string.translate. Something like:
import string
def clean_and_split(x):
return string.translate(x, None, r'^"()').split("AND")

Making letters uppercase using re.sub in python?

In many programming languages, the following
find foo([a-z]+)bar and replace with GOO\U\1GAR
will result in the entire match being made uppercase. I can't seem to find the equivalent in python; does it exist?

You can pass a function to re.sub() that will allow you to do this, here is an example:
def upper_repl(match):
return 'GOO' + match.group(1).upper() + 'GAR'
And an example of using it:
>>> re.sub(r'foo([a-z]+)bar', upper_repl, 'foobazbar')
'GOOBAZGAR'

Unfortunately this \U\1 syntax could never work in Python because \U in a string literal indicates the beginning of a 32-bit hex escape sequence. For example, "\U0001f4a9" == "💩".
However there are easy alternative to Perl's case conversion escapes available by using a replacement function. In re.sub(pattern, repl, string, count=0, flags=0) the replacement repl is usually a string, but it can also be a callable. If it is a callable, it's passed the Match object and must return a replacement string to be used.
So, for the example given in the question, this is possible:
>>> string = "fooquuxbar"
>>> pattern = "foo([a-z]+)bar"
>>> re.sub(pattern, lambda m: f"GOO{m.group(1).upper()}GAR", string)
'GOOQUUXGAR'
Here is a table of other string methods which might be useful for similar case conversions.
Modifier
Description
Example
Python callable to use
\U
Uppercase
foo BAR --> FOO BAR
str.upper
\L
Lowercase
foo BAR --> foo bar
str.lower or str.casefold
\I
Initial capital
foo BAR --> Foo Bar
str.title
\F
First capital
foo BAR --> Foo bar
str.capitalize

If you already have a replacement string (template), you may not be keen on swapping it out with the verbosity of m.group(1)+...+m.group(2)+...+m.group(3)... Sometimes it's nice to have a tidy little string.
You can use the MatchObject's expand() function to evaluate a template for the match in the same manner as sub(), allowing you to retain as much of your original template as possible. You can use upper on the relevant pieces.
re.sub(r'foo([a-z]+)bar', lambda m: 'GOO' + m.expand(r'\1GAR').upper(), 'foobazbar')
While this would not be particularly useful in the example above, and while it does not aid with complex circumstances, it may be more convenient for longer expressions with a greater number of captured groups, such as a MAC address censoring regex, where you just want to ensure the full replacement is capitalized or not.

You could use some variation of this:
s = 'foohellobar'
def replfunc(m):
return m.groups()[0]+m.groups()[1].upper()+m.groups()[2]
re.sub('(foo)([a-z]+)(bar)',replfunc,s)
gives the output:
'fooHELLObar'

For those coming across this on google...
You can also use re.sub to match repeating patterns. For example, you can convert a string with spaces to camelCase:
def to_camelcase(string):
string = string[0].lower() + string[1:] # lowercase first
return re.sub(
r'[\s]+(?P<first>[a-z])', # match spaces followed by \w
lambda m: m.group('first').upper(), # get following \w and upper()
string)
to_camelcase('String to convert') # --> stringToConvert