Develop Reference

Tricky String Normalization in Python

I'm sure this will be easy pickings for more experienced programmers than I, but this problem is bedeviling me and I've made a couple of failed attempts, so I wanted to see what other people might come up with.
I have about a hundred strings that look something like this:
(argument1 OR argument2) | inputlookup my_lookup.csv | `macro1(tag,bunit)` | `macro2(category)` | `macro_3(tag,\"expected\",category)` | `macro4(tag,\"timesync\")`
The goal is to find the arguments to the macro function and replace them with the count of the arguments, so that the final output looks like this:
(argument1 OR argument2) | inputlookup my_lookup.csv | `macro1(2)` | `macro2(1)` | `macro_3(3)` | `macro4(2)`
Python has ways of obtaining the count I need (I was simply counting up the number of commas in a string and adding 1), and Python has plenty of regex-type solutions for inline string replacement, but for the life of me I can't figure out how to combine them.
It seems something like re.sub won't let me identify a substring, count the number of commas in the substring, and then replace the substring with that value (unless I am missing something in the docs).
Can anybody think of a way to do this? Have I missed something obvious?

Solution:
import re
def count_commas(input_str):
c = 0
for s in input_str:
if s == ',':
c += 1
return c
pattern = r'\([A-Za-z0-9,""]+\)'
original_str = '(argument1 OR argument2) | inputlookup my_lookup.csv | `macro1(tag,bunit)` | `macro2(category)` | `macro_3(tag,\"expected\",category)` | `macro4(tag,\"timesync\")`'
matches = re.findall(pattern, original_str)
for match in matches:
comma_count = count_commas(match) + 1
match = match.replace('(', '\(').replace(')', '\)')
original_str = re.sub(r'' + match, '(' + str(comma_count) + ')', original_str)
print (original_str)
Explanation:
pattern : "\([A-Za-z0-9,""]+\)" - backslashes to escape the special characters '(' and ')' in regex, and then I am looking for alphanumeric, comma and quotations (in the square-brackets) which is followed with '+' which means one or more than one repetition of such symbols in the square brackets.
matches : list of all the matches found. Eg - (tag,bunit)
Then, I am looping over all the matches to find the number of commas in the match, followed by replacing the '(' with '\(' and ')' with '\)' so as to escape in regex.
Finally, in the last line of the loop, I am using re.sub to replace the matched string with the comma count in the original string.

python regex splitting at wrong places

I have the following regex and the input string.
pattern = re.compile(r'\s+(?=[^()|^{}|^<>]*(?:\(|\{|\<|$))')
string = "token1 token2 {a | op (b|c) | d}"
print pattern.split(string)
the result is : ["token1","token2","{a | op","(b|c) |d}"]
I want the regex to give the following result : ["token1","token2","{a | op (b|c) | d}"]

string = "token1 token2 {a | op (b|c) | d}"
re.findall(r'\w+|\{.*}',string)
output:
['token1', 'token2', '{a | op (b|c) | d}']

You can simply split by this
\s+(?![^{]*\})
See demo.
https://regex101.com/r/WjQVqZ/1

The raw pattern to use with the split method is r'\s+(?=[^\}]*(?:\{|$))'.
Every time whitespace is encountered, you want to look ahead for a closing curly brace, so you know if the white space is inside of braces - unless an opening curly brace or the end of the string is seen next.

Regexp to match random order words

I have the following pseudo-DSL:
< allow | deny >
< tcp | udp | any >
src < prefix | $ip | #hostgroup | any > [ port number | range | #portgroup | any ]
dst < prefix | $ip | #hostgroup | any > [ port number | range | #portgroup | any ]
[ stateful ]
[ expire YYYYMMDD ] [ log ]
[ # comment ]
The order is fixed, starting from allow up to dst and its port.
That I'm matching with the following, rather dumb, regexp:
m = re.search("^(allow|deny)?\s+(tcp|udp|tcpudp|any)\s+?(src\s\S+)\s*?(port\s+\S+)?\s*?(dst\s\S+)\s?(port\s+\S+)?\s*?(\S+)?\s*?(\S+)?", line)
Pardon me for the n00bness of the questions, but the parts I'm having problems with are:
How can I match stateful, expire <value>, log if all 3 are optional but in case they are present I want to match them in separate groups.
How can I match optional statement port <value> in such a way that the match group will contain only the value, without creating an extra matching group, i.e. without using (port\s+(\S+))?
Thanks!
[edit for more of a problem statement]
To elaborate a bit more, sure I can check whether one of the 3 groups contain either log or stateful, but if I use the same approach, a non-capturing group for expire, aka (?:expire\s(\S+)), I'd need to make an assumption. Unless I can somehow have order-less matching? i.e. match on (stateful|log|(?:expire\s(\S+)))?

How can I match stateful, expire <value>, log if all 3 are optional but in case they are present I want to match them in separate groups.
Use capture groups that have a ? after them so that they will be optional.
Ex. \s*(stateful)?\s*(?:expire (\d{8}))?\s*(log)?
To allow those optional groups to appear in any order in the match string, but still always have them in the same numbered capture group, use a look-ahead (?= ).
Ex. (?=(?:.*(stateful))?)(?=(?:.*expire (\d{8}))?)(?=(?:.*(log))?)
How can I match optional statement port <value> in such a way that the match group will contain only the value, without creating an extra matching group, i.e. without using
(port\s+(\S+))?
Use a non-capturing group (?: ) to put those characters together for the following ? without capturing them. (You probably want to do this for expire above also)
(?:port\s+(\s+))?
Complete Regex

Match every other character in a string

I have a string
k1|v1|k2|v2|k3|v3|k4|v4
and I want to match on every other | so I can change the string to
k1:v1|k2:v2|k3:v3|k4:v4
I know I can match on | by doing a grouping like (|) but I can't figure out how to match only every other pipe.
Thanks.

Match with:
([^|]*)\|([^|]*(\||$))
Replace with $1:$2.
See it in action
General idea:
[^|]* - multiple non-| characters
() defines a group
(\||$) - a | or the end of the string
The entire regex reads as multiple non | characters in the first group, followed by a |, followed by multiple non | characters and a | or end of string in the second group

Write regex for matching all 4 digit numbers between patterns

I am trying to write a regex to find pattern in string. Its gonna have a word 'LAT_LON' then some non word characters and then many 4 digit numbers and after then some alphabet or end of string.
Eg1.
SOME EXAMPLE STRING 12334...
LAT_LON .... 1234 5678 9012 1234
1234 1234
Eg2.
SOME EXAMPLE STRING 1234...
LAT_LON ... 1234 5678 9012 1234
1234 1234 SOMETHING_ELSE
In both the examples I need those 6 4-digit numbers after the pattern 'LAT_LON' and before any other alphabet.
EDIT: I am working in python, although I don't care much about the language. I am fairly new to regex world. So I am just trying some random stuff, nothing very conclusive at all till now.

One way is to capture the numbers then split on whitespace.
LAT_LON[^\da-zA-Z]*(\d{4}(?:\s+\d{4})*)
Then split capture group 1 on whitespace.
LAT_LON [^\da-zA-Z]*
( # (1 start)
\d{4}
(?:
\s+
\d{4}
)*
) # (1 end)
Here is a more verbose formatted version.
( Regex's constructed by RegexFormat 6 )
LAT_LON # Exact 'LAT_LON'
[^\da-zA-Z]* # Optinal chars, 0 to many times
# not digit nor letter (case insensitive)
( # (1 start), Capture all 4 digit numbers
\d{4} # Single 4 digit number
(?: # Cluster group
\s+ # Whitespace(s)
\d{4} # Single 4 digit number
)* # End Cluster, do 0 to many times
) # (1 end)

Let me try it another way, just to have some variation in the answers. I'm going to use awk for the job.
awk '/LAT_LON/,/\n[^0-9]/{printf gensub(/[^0-9 ]/, "", "g", $0) " "}' /path/to/intput/file
With a possible pipe to clean up the output | tr -s ' '.
This code just searches for lines containing LAT_LON, then it will parse each of those lines until a non number is found. On these lines we filter out non spaces or numbers using the gensub.
Note that the regex is fairly simple because we have filtered out all irrelevant parts. A simple non-numerical removal does the job here. See also grep if you want to mess around with regex, in my opinion it's the best way to learn. In particular egrep, which supports an enhanced regex language!