Develop Reference

Regex: Match string (including nested characters) inside 2 characters) while not capturing characters, but non-greedy

I'm done making a markup language, but I'm now making optimized strings for it.
I want there to be nested characters allowed inside the strings, with my RPLY lexer there are 3 objects I need to allow inside of strings (curly braces and backticks), as well as not capturing the characters, such as the regex code I've tried:
(?:`)+([\w\W]+)(?:`)+
But this is greedy, and will only match between the first grave it sees and the last, as well as creating non-captured groups that are unsupported with RPLY.
Is there an alternative to this that's non-greedy but will allow nested characters? (and no non-capturing groups please, I'm using the the RPython distribution of PLY (RPLY) lexer and parser which doesn't support regex groups)
If anybody needs more code, I have 2 Python classes, both for the lexer and parser.
LEXER
from rply import LexerGenerator
class BMLLexer():
def __init__(self):
self.__lexer = LexerGenerator()
def __add_tokens(self):
# Statement definitions
# Note that I need both the OPEN_STATEMENT and CLOSE_STATEMENT to be allowed inside the string.
self.__lexer.add('OPEN_STATEMENT', r'\{')
self.__lexer.add('CLOSE_STATEMENT', r'\}')
# Basic things
# Note that RPLY's parser doesn't allow multiple groups (includes capturing and non-capturing), so the only option would be just to use functions to remove the first and last backtick from the string, or find something in regex that allows me to automatically get rid of the first and last backtick in the string.
self.__lexer.add('STRING', r'(?:`)+([\w\W]+)(?:`)+')
# Ignore spaces
self.__lexer.ignore('\s+')
def build(self):
self.__add_tokens()
return self.__lexer.build()
PARSER
import re
from bmls.language.parser.definitions import BMLDefinitionCapped, BMLDefinitionSingle
from rply import ParserGenerator, Token
class BMLParser():
"""The direct BML parser.
Raises:
SyntaxError: If given invalid syntax, the parser will throw a SyntaxError.
"""
# Init parser
def __init__(self):
"""Initializes the parser.
"""
self.pg = ParserGenerator(
# A list of all token names accepted by the parser.
[
'OPEN_STATEMENT',
'CLOSE_STATEMENT',
'STRING',
],
precedence= [
('left', ['OPEN_STATEMENT']),
('left', ['STRING']),
('right', ['CLOSE_STATEMENT'])
]
)
# Parsing
def parse(self):
"""Parses BML content
Raises:
SyntaxError: If given invalid syntax, the parser will throw a SyntaxError.
Returns:
list: An HTML/XML formatted list of items.
"""
# Multi-expression handling
#self.pg.production('main : expr')
#self.pg.production('main : main expr')
def main(p):
if len(p) == 1:
return p
else:
for x in p[1:]:
p[0].append(x)
return p[0]
# Expression handling
#self.pg.production('expr : STRING OPEN_STATEMENT main CLOSE_STATEMENT')
def definition_capped(p):
name = self.__toSTRING(p[0])
definition1 = self.__toSTRING(p[2])
comp = BMLDefinitionCapped(name, definition1)
return self.__toHTML(comp)
#self.pg.production('expr : OPEN_STATEMENT STRING CLOSE_STATEMENT')
def definition_uncapped(p):
name = self.__toSTRING(p[1])
comp = BMLDefinitionSingle(name)
return self.__toHTML(comp)
# Expression types
# This is where the string is parsed, currently using the function __removeFIrstLast. I wish to replace this with a supported expression.
#self.pg.production('expr : STRING')
def string_expr(p):
if p[0].gettokentype() == 'STRING':
return self.__removeFirstLast(self.__toSTRING(p[0]), '`', '`')
# Error handling
#self.pg.error
def error_handle(token):
raise SyntaxError('Error on Token (\'' + token.gettokentype() + '\' , \'' + token.getstr() + '\')')
# Public utilities
def build(self):
return self.pg.build()
# Private utilities
def __removeFirstLast(self, tok, char, endchar):
if isinstance(tok, str):
if tok.startswith(char) and tok.endswith(endchar):
return re.sub(r'^' + char + r'|' + endchar + r'$', '', tok)
else:
return tok
else:
return tok
def __toHTML(self, tok):
output = ''
if isinstance(tok, BMLDefinitionCapped):
right = ''
try:
for k1 in tok.right:
right += k1
except:
right += tok.right
output += '<' + tok.left + '>' + right + '</' + tok.left.split(' ')[0] + '>'
elif isinstance(tok, BMLDefinitionSingle):
output += '<' + tok.left + '>'
elif isinstance(tok, Token):
output += tok.getstr()
else:
output += tok
return output
def __toSTRING(self, tok):
if isinstance(tok, Token):
return tok.getstr()
else:
return tok
def __toINT(self, tok):
if isinstance(tok, Token):
return int(tok.getstr())
else:
return int(tok)

Python equivalent of Fortran list-directed input

I'd like to be able to read data from an input file in Python, similar to the way that Fortran handles a list-directed read (i.e. read (file, *) char_var, float_var, int_var).
The tricky part is that the way Fortran handles a read statement like this is very "forgiving" as far as the input format is concerned. For example, using the previous statement, this:
"some string" 10.0, 5
would be read the same as:
"some string", 10.0
5
and this:
"other string", 15.0 /
is read the same as:
"other string"
15
/
with the value of int_var retaining the same value as before the read statement. And trickier still this:
"nother string", , 7
will assign the values to char_var and int_var but float_var retains the same value as before the read statement.
Is there an elegant way to implement this?

That is indeed tricky - I found it easier to write a pure-python stated-based tokenizer than think on a regular expression to parse each line (tough it is possible).
I've used the link provided by Vladimir as the spec - the tokenizer have some doctests that pass.
def tokenize(line, separator=',', whitespace="\t\n\x20", quote='"'):
"""
>>> tokenize('"some string" 10.0, 5')
['some string', '10.0', '5']
>>> tokenize(' "other string", 15.0 /')
['other string', '15.0', '/']
>>> tokenize('"nother string", , 7')
['nother string', '', '7']
"""
inside_str = False
token_started = False
token = ""
tokens = []
separated = False
just_added = False
for char in line:
if char in quote:
if not inside_str:
inside_str = True
else:
inside_str = False
tokens.append(token)
token = ""
just_added = True
continue
if char in (whitespace + separator) and not inside_str:
if token:
tokens.append(token)
token = ""
just_added = True
elif char in separator:
if not just_added:
tokens.append("")
just_added = False
continue
token += char
if token:
tokens.append(token)
return tokens
class Character(object):
def __init__(self, length=None):
self.length = length
def __call__(self, text):
if self.length is None:
return text
if len(text) > self.length:
return text[:self.length]
return "{{:{}}}".format(self.length).format(text)
def make_types(types, default_value):
return types, [default_value] * len[types]
def fortran_reader(file, types, default_char="/", default_value=None, **kw):
types, results = make_types(types, default_value)
tokens = []
while True:
tokens = []
while len(tokens) < len(results):
try:
line = next(file)
except StopIteration:
raise StopIteration
tokens += tokenize(line, **kw)
for i, (type_, token) in enumerate(zip(types, tokens)):
if not token or token in default_char:
continue
results[i] = type_(token)
changed_types = yield(results)
if changed_types:
types, results = make_types(changed_types)
I have not teste this thoughtfully - but for the tokenizer -
it is designed to work in a Python forstatement if the same fields are repeated over and over again - or it can be used with Python's iterators send method to change the values to be read on each iteration.
Please test, and e-mail me (address at my profile) some testing file. If there is indeed nothing similar, maybe this deserves some polishing and be published in Pypi.

Since I was not able to find a solution to this problem, I decided to write my own solution.
The main drivers are a reader class, and a tokenizer. The reader gets one line at a time from the file, passes it to the tokenizer, and assigns to the variables it is given, getting the next line as necessary.
class FortranAsciiReader(file):
def read(self, *args):
"""
Read from file into the given objects
"""
num_args = len(args)
num_read = 0
encountered_slash = False
# If line contained '/' or read into all varialbes, we're done
while num_read < num_args and not encountered_slash:
line = self.readline()
if not line:
raise Exception()
values = tokenize(line)
# Assign elements one-by-one into args, skipping empty fields and stopping at a '/'
for val in values:
if val == '/':
encountered_slash = True
break
elif val == '':
num_read += 1
else:
args[num_read].assign(val)
num_read += 1
if num_read == num_args:
break
The tokenizer splits the line into tokens in accordance with the way that Fortran performs list directed reads, where ',' and white space are separators, tokens may be "repeated" via 4*token, and a / terminates input.
My implementation of the tokenizer is a bit long to reproduce here, and I also included classes to transparently provide the functionality of the basic Fortran intrinsic types (i.e. Real, Character, Integer, etc.). The whole project can be found on my github account, currently at https://github.com/bprichar/PyLiDiRe. Thanks jsbueno for inspiration for the tokenizer.

Python Prefix / Infix / Postfix expression evaluation

Little brief about the code:
I have to do a class that will evaluate prefix, postfix or infix expression. It has to determine whether it is pre/post/infix and convert them to postfix, for example prefixTOpostfix() (others are deleted as not needed now) method in the code which converts from '/x7' to 'x7/', the expression is edited in method edit() from 'x7/' to 'x 7 /'. Both methods work fine, tested on multiple examples (not posting here the whole code as it is and assignment, but also they are not needed. The question being asked is just an error I am getting, don't worry I am not asking for solution to my assignment). There is also assign method, because there can be variables, for example 'a = 3', and it can be somewhere in the expression.
Problem: When i run print(v.evaluate('x 7/')) (what is already in postfix), where x = 14, it returns 2 as it should. However when I run print(v.evaluate('/x 7')) (what is in prefix) it returns None. Both expressions look exactly the same after they ran through their methods 'x 7 /' (I had testing prints in the code), Both stacks are the same. First there is '14', then '14 7', and lastly '2'. When I change return(s.pop()) to return(s.top()), both expressions are evaluated fine as '2'. So why return(s.pop()) doesn't work with the second one ? If there are more questions about the code or something isn't clear enough, tell me I will try to explain it differently.
class MyClass:
class Stack:
...
def __init__(self):
self.table = {}
def __repr__(self):
...
def assign(self, variable, exp):
if '+-*/%' in exp: # temporary solution
exp = self.evaluate(exp)
self.table[variable] = exp
def evaluate(self, exp):
if exp[0] in '+-*/%': # Prefix
new_exp = self.prefixTOpostfix(exp)
self.evaluate(new_exp)
elif exp[len(exp)-1] in '+-*/%': # Postfix
s = self.Stack()
exp = self.edit(exp) # from 'x7/' to 'x 7 /'
for item in exp.split():
if item == '+':
s.push(s.pop() + s.pop())
... # other operations
elif prvok == '/':
temp = s.pop()
if temp == 0:
return None
s.push(s.pop() // temp) # it has to be // !
else: # if it is number / variable
if item in self.table:
s.push(int(self.table[item]))
else:
s.push(int(item))
s.printOUT()
return(s.pop())
else: # Infix
...
def prefixTOpostfix(self, exp):
...
def edit(self, exp):
...

if exp[0] in '+-*/%': # Prefix
new_exp = self.prefixTOpostfix(exp)
self.evaluate(new_exp)
You need to return the result of your recursive call.
if exp[0] in '+-*/%': # Prefix
new_exp = self.prefixTOpostfix(exp)
return self.evaluate(new_exp)

Lexical analysis

I am learning lexers in Python. I am using Ply library for lexical analysis on some strings. I have implemented the following lexical analyzer for some of C++ language syntax.
However, I am facing a strange behavior. When I define the COMMENT states function definitions at the end of other function definitions, the code works fine. If I define COMMENT state functions before other definitions, I get errors as soon as // sectoin starts in the input string starts.
WHAT IS THE REASON BEHIND THAT?
import ply.lex as lex
tokens = (
'DLANGLE', # <<
'DRANGLE', # >>
'EQUAL', # =
'STRING', # "144"
'WORD', # 'Welcome' in "Welcome."
'SEMICOLON', # ;
)
t_ignore = ' \t\v\r' # shortcut for whitespace
states = (
('cppcomment', 'exclusive'), # <!--
)
def t_cppcomment(t): # definition here causes errors
r'//'
print 'MyCOm:',t.value
t.lexer.begin('cppcomment');
def t_cppcomment_end(t):
r'\n'
t.lexer.begin('INITIAL');
def t_cppcomment_error(t):
print "Error FOUND"
t.lexer.skip(1)
def t_DLANGLE(t):
r'<<'
print 'MyLAN:',t.value
return t
def t_DRANGLE(t):
r'>>'
return t
def t_SEMICOLON(t):
r';'
print 'MySemi:',t.value
return t;
def t_EQUAL(t):
r'='
return t
def t_STRING(t):
r'"[^"]*"'
t.value = t.value[1:-1] # drop "surrounding quotes"
print 'MyString:',t.value
return t
def t_WORD(t):
r'[^ <>\n]+'
print 'MyWord:',t.value
return t
webpage = "cout<<\"Hello World\"; // this comment"
htmllexer = lex.lex()
htmllexer.input(webpage)
while True:
tok = htmllexer.token()
if not tok: break
print tok
Regards

Just figured it out. As I have defined comment state as exclusive, it won't use the inclusive state modules (if comment modules are defined at the top, otherwise it uses it for some reason). So you will have redefine all the modules for comment state again. Therefore ply provides error() modules for skipping characters for which specific modules are not defined.

its because you have no rules that accept this or comment
and really you dont care about whats in the comment you can easilly do something like
t_cppcomment_ANYTHING = '[^\r\n]'
just below your t_ignore rule

Efficiently match multiple regexes in Python

Lexical analyzers are quite easy to write when you have regexes. Today I wanted to write a simple general analyzer in Python, and came up with:
import re
import sys
class Token(object):
""" A simple Token structure.
Contains the token type, value and position.
"""
def __init__(self, type, val, pos):
self.type = type
self.val = val
self.pos = pos
def __str__(self):
return '%s(%s) at %s' % (self.type, self.val, self.pos)
class LexerError(Exception):
""" Lexer error exception.
pos:
Position in the input line where the error occurred.
"""
def __init__(self, pos):
self.pos = pos
class Lexer(object):
""" A simple regex-based lexer/tokenizer.
See below for an example of usage.
"""
def __init__(self, rules, skip_whitespace=True):
""" Create a lexer.
rules:
A list of rules. Each rule is a `regex, type`
pair, where `regex` is the regular expression used
to recognize the token and `type` is the type
of the token to return when it's recognized.
skip_whitespace:
If True, whitespace (\s+) will be skipped and not
reported by the lexer. Otherwise, you have to
specify your rules for whitespace, or it will be
flagged as an error.
"""
self.rules = []
for regex, type in rules:
self.rules.append((re.compile(regex), type))
self.skip_whitespace = skip_whitespace
self.re_ws_skip = re.compile('\S')
def input(self, buf):
""" Initialize the lexer with a buffer as input.
"""
self.buf = buf
self.pos = 0
def token(self):
""" Return the next token (a Token object) found in the
input buffer. None is returned if the end of the
buffer was reached.
In case of a lexing error (the current chunk of the
buffer matches no rule), a LexerError is raised with
the position of the error.
"""
if self.pos >= len(self.buf):
return None
else:
if self.skip_whitespace:
m = self.re_ws_skip.search(self.buf[self.pos:])
if m:
self.pos += m.start()
else:
return None
for token_regex, token_type in self.rules:
m = token_regex.match(self.buf[self.pos:])
if m:
value = self.buf[self.pos + m.start():self.pos + m.end()]
tok = Token(token_type, value, self.pos)
self.pos += m.end()
return tok
# if we're here, no rule matched
raise LexerError(self.pos)
def tokens(self):
""" Returns an iterator to the tokens found in the buffer.
"""
while 1:
tok = self.token()
if tok is None: break
yield tok
if __name__ == '__main__':
rules = [
('\d+', 'NUMBER'),
('[a-zA-Z_]\w+', 'IDENTIFIER'),
('\+', 'PLUS'),
('\-', 'MINUS'),
('\*', 'MULTIPLY'),
('\/', 'DIVIDE'),
('\(', 'LP'),
('\)', 'RP'),
('=', 'EQUALS'),
]
lx = Lexer(rules, skip_whitespace=True)
lx.input('erw = _abc + 12*(R4-623902) ')
try:
for tok in lx.tokens():
print tok
except LexerError, err:
print 'LexerError at position', err.pos
It works just fine, but I'm a bit worried that it's too inefficient. Are there any regex tricks that will allow me to write it in a more efficient / elegant way ?
Specifically, is there a way to avoid looping over all the regex rules linearly to find one that fits?

I suggest using the re.Scanner class, it's not documented in the standard library, but it's well worth using. Here's an example:
import re
scanner = re.Scanner([
(r"-?[0-9]+\.[0-9]+([eE]-?[0-9]+)?", lambda scanner, token: float(token)),
(r"-?[0-9]+", lambda scanner, token: int(token)),
(r" +", lambda scanner, token: None),
])
>>> scanner.scan("0 -1 4.5 7.8e3")[0]
[0, -1, 4.5, 7800.0]

You can merge all your regexes into one using the "|" operator and let the regex library do the work of discerning between tokens. Some care should be taken to ensure the preference of tokens (for example to avoid matching a keyword as an identifier).

I found this in python document. It's just simple and elegant.
import collections
import re
Token = collections.namedtuple('Token', ['typ', 'value', 'line', 'column'])
def tokenize(s):
keywords = {'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'}
token_specification = [
('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number
('ASSIGN', r':='), # Assignment operator
('END', r';'), # Statement terminator
('ID', r'[A-Za-z]+'), # Identifiers
('OP', r'[+*\/\-]'), # Arithmetic operators
('NEWLINE', r'\n'), # Line endings
('SKIP', r'[ \t]'), # Skip over spaces and tabs
]
tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
get_token = re.compile(tok_regex).match
line = 1
pos = line_start = 0
mo = get_token(s)
while mo is not None:
typ = mo.lastgroup
if typ == 'NEWLINE':
line_start = pos
line += 1
elif typ != 'SKIP':
val = mo.group(typ)
if typ == 'ID' and val in keywords:
typ = val
yield Token(typ, val, line, mo.start()-line_start)
pos = mo.end()
mo = get_token(s, pos)
if pos != len(s):
raise RuntimeError('Unexpected character %r on line %d' %(s[pos], line))
statements = '''
IF quantity THEN
total := total + price * quantity;
tax := price * 0.05;
ENDIF;
'''
for token in tokenize(statements):
print(token)
The trick here is the line:
tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
Here (?P<ID>PATTERN) will mark the matched result with a name specified by ID.

re.match is anchored. You can give it a position argument:
pos = 0
end = len(text)
while pos < end:
match = regexp.match(text, pos)
# do something with your match
pos = match.end()
Have a look for pygments which ships a shitload of lexers for syntax highlighting purposes with different implementations, most based on regular expressions.

It's possible that combining the token regexes will work, but you'd have to benchmark it. Something like:
x = re.compile('(?P<NUMBER>[0-9]+)|(?P<VAR>[a-z]+)')
a = x.match('9999').groupdict() # => {'VAR': None, 'NUMBER': '9999'}
if a:
token = [a for a in a.items() if a[1] != None][0]
The filter is where you'll have to do some benchmarking...
Update: I tested this, and it seems as though if you combine all the tokens as stated and write a function like:
def find_token(lst):
for tok in lst:
if tok[1] != None: return tok
raise Exception
You'll get roughly the same speed (maybe a teensy faster) for this. I believe the speedup must be in the number of calls to match, but the loop for token discrimination is still there, which of course kills it.

This isn't exactly a direct answer to your question, but you might want to look at ANTLR. According to this document the python code generation target should be up to date.
As to your regexes, there are really two ways to go about speeding it up if you're sticking to regexes. The first would be to order your regexes in the order of the probability of finding them in a default text. You could figure adding a simple profiler to the code that collected token counts for each token type and running the lexer on a body of work. The other solution would be to bucket sort your regexes (since your key space, being a character, is relatively small) and then use a array or dictionary to perform the needed regexes after performing a single discrimination on the first character.
However, I think that if you're going to go this route, you should really try something like ANTLR which will be easier to maintain, faster, and less likely to have bugs.