Develop Reference

Python regex: how to achieve this complex replacement rule?

I'm working with long strings and I need to replace with '' all the combinations of adjacent full stops . and/or colons :, but only when they are not adjacent to any whitespace. Examples:
a.bcd should give abcd
a..::.:::.:bcde.....:fg should give abcdefg
a.b.c.d.e.f.g.h should give abcdefgh
a .b should give a .b, because . here is adjacent to a whitespace on its left, so it has not to be replaced
a..::.:::.:bcde.. ...:fg should give abcde.. ...:fg for the same reason
Well, here is what I tried (without any success).
Attempt 1:
s1 = r'a.b.c.d.e.f.g.h'
re.sub(re.search(r'[^\s.:]+([.:]+)[^\s.:]+', s1).group(1), r'', s1)
I would expect to get 'abcdefgh' but what I actually get is r''. I understood why: the code
re.search(r'[^\s.:]+([.:]+)[^\s.:]+', s1).group(1)
returns '.' instead of '\.', and thus re.search doesn't understand that it has to replace the single full stop . rather than understanding '.' as the usual regex.
Attempt 2:
s1 = r'a.b.c.d.e.f.g.h'
re.sub(r'([^\s.:]*\S)[.:]+(\S[^\s.:]*)', r'\g<1>\g<2>', s1)
This doesn't work as it returns a.b.c.d.e.f.gh.
Attempt 3:
s1 = r'a.b.c.d.e.f.g.h'
re.sub(r'([^\s.:]*)[.:]+([^\s.:]*)', r'\g<1>\g<2>', s1)
This works on s1, but it doesn't solve my problem because on s2 = r'a .b' it returns a b rather than a .b.
Any suggestion?

There are multiple problems here. Your regex doesn't match what you want to match; but also, your understanding of re.sub and re.search is off.
To find something, re.search lets you find where in a string that something occurs.
To replace that something, use re.sub on the same regular expression instead of re.search, not as well.
And, understand that re.sub(r'thing(moo)other', '', s1) replaces the entire match with the replacement string.
With that out of the way, for your regex, it sounds like you want
r'(?<![\s.:])[.:]+(?![\s.:])' # updated from comments, thanks!
which contains a character class with full stop and colon (notice how no backslash is necessary inside the square brackets -- this is a context where dot and colon do not have any special meaning1), repeated as many times as possible; and lookarounds on both sides to say we cannot match these characters when there is whitespace \s on either side, and also excluding the characters themselves so that there is no way for the regex engine to find a match by applying the + less strictly (it will do its darndest to find a match if there is a way).
Now, the regex only matches the part you want to actually replace, so you can do
>>> import re
>>> s1 = 'name.surname#domain.com'
>>> re.sub(r'(?<![\s.:])[.:]+(?![\s.:])', r'', s1)
'namesurname#domaincom'
though in the broader scheme of things, you also need to know how to preserve some parts of the match. For the purpose of this demonstration, I will use a regular expression which captures into parenthesized groups the text before and after the dot or colon:
>>> re.sub(r'(.*\S)[.:]+(\S.*)', r'\g<1>\g<2>', s1)
'name.surname#domaincom'
See how \g<1> in the replacement string refers back to "whatever the first set of parentheses matched" and similarly \g<2> to the second parenthesized group.
You will also notice that this failed to replace the first full stop, because the .* inside the first set of parentheses matches as much of the string as possible. To avoid this, you need a regex which only matches as little as possible. We already solved that above with the lookarounds, so I will leave you here, though it would be interesting (and yet not too hard) to solve this in a different way.
1 You could even say that the normal regex language (or syntax, or notation, or formalism) is separate from the language (or syntax, or notation, or formalism) inside square brackets!

Python Regex: force greedy match using alternation

I have a regex of the form:
a(bc|de|def)g?
On the string adefg this pattern is matching only up to "ade" and it is clearly quitting on the first match in the alternation group. Removing the ? option from the "g" token allows the pattern to match the entire string. This makes sense since the "?" is non-greedy. [EDIT: I have been corrected, the "?" is greedy, which just seems to add to my confusion. It seemed to me that if the "?" were non-greedy, this was allowing the pattern to quit early when a larger match was available.]
I would like to avoid rearranging the order of the strings in the alternation, and I can solve the problem as is by appending (\b|$) to the pattern, but now I am really curious to know if there are other solutions
For instance, is there any way to make the "?" greedy or to force the alternation not to quit on the first match?

You can't make the | not match its constituents left to right, because matching left to right is its documented behavior. Even if you could make the ? "greedy", it wouldn't work, because the regex matches from beginning to end, so the greediness of the ? couldn't have an effect until after the alternation had already matched.
Greediness doesn't make the regex engine go back to find a "better way" to match; it will match the first way it can. It will only make use of the g? if it has to do so in order for the entire match to succeed, and it won't have to if it can just ignore it and stick with what it matched in the alternation. In other words, once it matches "ade", it can succeed and stop (because it doesn't need to match the "g", since it's optional). It therefore doesn't even consider the other parts of the alternation, since it can find a way to make it work using the first one. A greedy ? doesn't make it go back and retry other things it already matched unless it needs to for the entire match to succeed.
If you are using an alternation where some alternants are substrings of others, you should put them in order so the longest ones come first.
Another possibility is to add a $ to the end of your regex. This will force it to go all the way to the end of the string, so it will backtrack and try the other alternatives, because now "ade" won't be a match (since it doesn't match the $). However, this will only work if you really do want to match the whole string.

You can usually use a negative lookahead, but I don't know the capabilities of Python's regex engine.
a(bc|de(?!f)|def)g?
check here

An obvious way to refactor this expression would be to "unroll" the optional part:
a(bc|de|def)g|a(bc|de|def)
or
(a(bc|de|def))g|\1
to avoid the repetition.

Python regex for int with at least 4 digits

I am just learning regex and I'm a bit confused here. I've got a string from which I want to extract an int with at least 4 digits and at most 7 digits. I tried it as follows:
>>> import re
>>> teststring = 'abcd123efg123456'
>>> re.match(r"[0-9]{4,7}$", teststring)
Where I was expecting 123456, unfortunately this results in nothing at all. Could anybody help me out a little bit here?

#ExplosionPills is correct, but there would still be two problems with your regex.
First, $ matches the end of the string. I'm guessing you'd like to be able to extract an int in the middle of the string as well, e.g. abcd123456efg789 to return 123456. To fix that, you want this:
r"[0-9]{4,7}(?![0-9])"
^^^^^^^^^
The added portion is a negative lookahead assertion, meaning, "...not followed by any more numbers." Let me simplify that by the use of \d though:
r"\d{4,7}(?!\d)"
That's better. Now, the second problem. You have no constraint on the left side of your regex, so given a string like abcd123efg123456789, you'd actually match 3456789. So, you need a negative lookbehind assertion as well:
r"(?<!\d)\d{4,7}(?!\d)"

.match will only match if the string starts with the pattern. Use .search.

You can also use:
re.findall(r"[0-9]{4,7}", teststring)
Which will return a list of all substrings that match your regex, in your case ['123456']
If you're interested in just the first matched substring, then you can write this as:
next(iter(re.findall(r"[0-9]{4,7}", teststring)), None)

Match any characters more than once, but stop at a given character

I am writing a regex that will be used for recognizing commands in a string. I have three possible words the commands could start with and they always end with a semi-colon.
I believe the regex pattern should look something like this:
(command1|command2|command3).+;
The problem, I have found, is that since . matches any character and + tells it to match one or more, it skips right over the first instance of a semi-colon and continues going.
Is there a way to get it to stop at the first instance of a semi-colon it comes across? Is there something other than . that I should be using instead?

The issue you are facing with this: (command1|command2|command3).+; is that the + is greedy, meaning that it will match everything till the last value.
To fix this, you will need to make it non-greedy, and to do that you need to add the ? operator, like so: (command1|command2|command3).+?;
Just as an FYI, the same applies for the * operator. Adding a ? will make it non greedy.

Tell it to find only non-semicolons.
[^;]+

What you are looking for is a non-greedy match.
.+?
The "?" after your greedy + quantifier will make it match as less as possible, instead of as much as possible, which it does by default.
Your regex would be
'(command1|command2|command3).+?;'
See Python RE documentation

Lookahead assertions seem to short-circuit ordering of alternates in regular expressions

I'm working with a (Python-flavored) regular expression to recognize common and idiosyncratic forms and abbreviations of scripture references. Given the following verbose snippet:
>>> cp = re.compile(ur"""
(?:(
# Numbered books
(?:(?:Third|Thir|Thi|III|3rd|Th|3)\ ?
(?:John|Joh|Jhn|Jo|Jn|Jn|J))
# Other books
|Thessalonians|John|Th|Jn)\ ?
# Lookahead for numbers or punctuation
(?=[\d:., ]))
|
# Do the same check, this time at the end of the string.
(
(?:(?:Third|Thir|Thi|III|3rd|Th|3)\ ?
(?:John|Joh|Jhn|Jo|Jn|Jn|J))
|Thessalonians|John|Th|Jn)\.?$
""", re.IGNORECASE | re.VERBOSE)
>>> cp.match("Third John").group()
'Third John'
>>> cp.match("Th Jn").group()
'Th'
>>> cp.match("Th Jn ").group()
'Th Jn'
The intention of this snippet is to match various forms of "Third John", as well as forms of "Thessalonians" and "John" by themselves. In most cases this works fine, but it does not match "Th Jn" (or "Th John"), rather matching "Th" by itself.
I've ordered the appearance of each abbreviation in the expression from longest to shortest expressly to avoid a situation like this, relying on a regular expression's typically greedy behavior. But the positive lookahead assertion seems to be short-circuiting this order, picking the shortest match instead of the greediest match.
Of course, removing the lookahead assertion makes this case work, but breaks a bunch of other tests. How might I go about fixing this?

I've given up after a little try to follow what _sre.so is doing in this case (too complicated!) but a "blind fix" I tried seemed to work -- switch to a negative lookahead assertion for the complementary character set...:
cp = re.compile(ur"""
(?:(
# Numbered books
(?:(?:Third|Thir|Thi|III|3rd|Th|3)\ ?
(?:John|Joh|Jhn|Jo|Jn|Jn|J))
# Other books
|Thessalonians|John|Th|Jn)\ ?
# Lookahead for numbers or punctuation
(?![^\d:., ]))
|
etc. I.e. I changed the original (?=[\d:., ])) positive lookahead into a "double negation" form (negative lookahead for complement) (?![^\d:., ])) and this seems to remove the perturbation. Does this work correctly for you?
I think it's an implementation anomaly in this corner case of _sre.so -- it might be interesting to see what other RE engines do in these two cases, just as a sanity check.

The lookahead isn't really short circuiting anything. The regex is only greedy up to a point. It'll prefer a match in your first big block because it doesn't want to cross that "|" boundary to the second part of the regex and have to check that as well.
Since the whole string doesn't match the first big block (because the lookeahead says it needs to be followed by a particular character rather than end of line) it just matches the "Th" from the "Thessalonians" group and the lookahead sees a space following "Th" in "Th Jn" so it considers this a valid match.
What you'll probably want to do is move the "|Thessalonians|John|Th|Jn)\ ? " group out to another large "|" block. Check your two word books at the beginning of text OR at the end of text OR check for one word books in a third group.
Hope this explanation made sense.

Another alternate solution I discovered while asking the question: switch the order of the blocks, putting the end-of-line check first, then the lookahead assertion last. However, I prefer Alex's double negative solution, and have implemented that.