I'm trying to develop a grammar which can parse expression starting with parenthesis and ending parenthesis. There can be any combination of characters inside the parenthesis. I've written the following code, following the Hello World program from pyparsing.
from pyparsing import *
select = Literal("select")
predicate = "(" + Word(printables) + ")"
selection = select + predicate
print (selection.parseString("select (a)"))
But this throws error. I think it may be because printables also consist of ( and ) and it's somehow conflicting with the specified ( and ).
What is the correct way of doing this?
You could use alphas instead of printables.
from pyparsing import *
select = Literal("select")
predicate = "(" + Word(alphas) + ")"
selection = select + predicate
print (selection.parseString("select (a)"))
If using { } as the nested characters
from pyparsing import *
expr = Combine(Suppress('select ') + nestedExpr('{', '}'))
value = "select {a(b(c\somethinsdfsdf##!#$###$##))}"
print( expr.parseString( value ) )
output: [['a(b(c\\somethinsdfsdf##!#$###$##))']]
The problem with ( ) is they are used as the default quoting characters.
Related
I'm trying to parse a file, actually some portions of the file. The file contains information about hardwares in a server and each line starts with a keyword denoting the type of hardware. For example:
pci24 u2480-L0
fcs1 g4045-L1
pci25 h6045-L0
en192 v7024-L3
pci26 h6045-L1
Above example doesnt show a real file but it's simple and quite enough to demonstrate the need. I want only to parse the lines starting with "pci" and skip others. I wrote a grammer for lines starting with "pci":
grammar_pci = Group ( Word( "pci" + nums ) + Word( alphanums + "-" ) )
I've also wrote a grammar for lines not starting with "pci":
grammar_non_pci = Suppress( Regex( r"(?!pci)" ) )
And then build a grammar that sum up above two:
grammar = ( grammar_pci | grammar_non_pci )
Then i read the file and send it to parseString:
with open("foo.txt","r") as f:
data = grammar.parseString(f.read())
print(data)
But no data is written as output. What am i missing? How to parse data skipping the lines not starts with a specific keyword?
Thanks.
You are off to a good start, but you are missing a few steps, mostly having to do with filling in gaps and repetition.
First, look at your expression for grammar_non_pci:
grammar_non_pci = Suppress( Regex( r"(?!pci)" ) )
This correctly detects a line that does not start with "pci", but it doesn't actually parse the line's content.
The easiest way to add this is to add a ".*" to the regex, so that it will parse not only the "not starting with pci" lookahead, but also the rest of the line.
grammar_non_pci = Suppress( Regex( r"(?!pci).*" ) )
Second, your grammar just processes a single instance of an input line.
grammar = ( grammar_pci | grammar_non_pci )
grammar needs to be repetitive
grammar = OneOrMore( grammar_pci | grammar_non_pci, stopOn=StringEnd())
[EDIT: since you are up to pyparsing 3.0.9, this can also be written as follows]
grammar = (grammar_pci | grammar_non_pci)[1, ...: StringEnd()]
Since grammar_non_pci could actually match on an empty string, it could repeat forever at the end of the file - that's why the stopOn argument is needed.
With these changes, your sample text should parse correctly.
But there is one issue that you'll need to clean up, and that is the definition of the "pci"-prefixed word in grammar_pci.
grammar_pci = Group ( Word( "pci" + nums ) + Word( alphanums + "-" ) )
Pyparsing's Word class takes 1 or 2 strings of characters, and uses them as a set of the valid characters for the initial word character and the body word characters. "pci" + nums gives the string "pci0123456789", and will match any word group using any of those characters. So it will match not only "pci00" but also "cip123", "cci123", "p0c0i", or "12345".
To resolve this, use "pci" + Word(nums) wrapped in Combine to represent only word groups that start with "pci":
grammar_pci = Group ( Combine("pci" + Word( nums )) + Word( alphanums + "-" ) )
Since you seem comfortable using Regex items, you could also write this as
grammar_pci = Group ( Regex(r"pci\d+") + Word( alphanums + "-" ) )
These changes should get you moving forward on your parser.
Read each line at a time, and if starts with pci, add it to the list data; otherwise, discard it:
data = []
with open("foo.txt", "r") as f:
for line in f:
if line.startswith('pci'):
data.append(line)
print(data)
If you still need to do further parsing with your Grammar, you can now parse the list data, knowing that each item does indeed start with pci.
Digging deeper into grammars and PEG in special, I wanted to have a DSL with the following syntax:
a OR (b AND c)
I am using parsimonious here with the following grammar:
from parsimonious.grammar import Grammar
grammar = Grammar(
"""
expr = (term operator term)+
term = (lpar term rpar) / (variable operator variable)
operator = and / or
or = _? "OR" _?
and = _? "AND" _?
variable = ~r"[a-z]+"
lpar = "("
rpar = ")"
_ = ~r"\s*"
"""
)
print(grammar.parse('a OR (b AND c)'))
However, this fails for the above text with
parsimonious.exceptions.ParseError: Rule 'variable' didn't match at '(b AND c)' (line 1, column 6).
Why? Haven't I specified term as ( term ) or term ?
Why does it choose the rule for variable instead (which of course fails) ?
The first thing in expr is a term, so that's what the parser looks for.
A term in your grammar is either
( term )
or
variable operator variable
And the input is
a OR (b AND c)
That doesn't start with a ( so the only way it can be a term is if it matches variable operator variable. a is a variable; OR is an operator. So the next thing to match is variable.
Perhaps what you want is:
expr = term (operator term)*
term = (lpar expr rpar) / variable
I am trying to parse the following text using pyparsing.
acp (SOLO1,
"solo-100",
"hi here is the gift"
"Maximum amount of money, goes",
430, 90)
jhk (SOLO2,
"solo-101",
"hi here goes the wind."
"and, they go beyond",
1000, 320)
I have tried the following code but it doesn't work.
flag = Word(alphas+nums+'_'+'-')
enclosed = Forward()
nestedBrackets = nestedExpr('(', ')', content=enclosed)
enclosed << (flag | nestedBrackets)
print list(enclosed.searchString (str1))
The comma(,) within the quotation is producing undesired results.
Well, I might have oversimplified slightly in my comments - here is a more complete
answer.
If you don't really have to deal with nested data items, then a single-level parenthesized
data group in each section will look like this:
LPAR,RPAR = map(Suppress, "()")
ident = Word(alphas, alphanums + "-_")
integer = Word(nums)
# treat consecutive quoted strings as one combined string
quoted_string = OneOrMore(quotedString)
# add parse action to concatenate multiple adjacent quoted strings
quoted_string.setParseAction(lambda t: '"' +
''.join(map(lambda s:s.strip('"\''),t)) +
'"' if len(t)>1 else t[0])
data_item = ident | integer | quoted_string
# section defined with no nesting
section = ident + Group(LPAR + delimitedList(data_item) + RPAR)
I wasn't sure if it was intentional or not when you omitted the comma between
two consecutive quoted strings, so I chose to implement logic like Python's compiler,
in which two quoted strings are treated as just one longer string, that is "AB CD " "EF" is
the same as "AB CD EF". This was done with the definition of quoted_string, and adding
the parse action to quoted_string to concatenate the contents of the 2 or more component
quoted strings.
Finally, we create a parser for the overall group
results = OneOrMore(Group(section)).parseString(source)
results.pprint()
and get from your posted input sample:
[['acp',
['SOLO1',
'"solo-100"',
'"hi here is the giftMaximum amount of money, goes"',
'430',
'90']],
['jhk',
['SOLO2',
'"solo-101"',
'"hi here goes the wind.and, they go beyond"',
'1000',
'320']]]
If you do have nested parenthetical groups, then your section definition can be
as simple as this:
# section defined with nesting
section = ident + nestedExpr()
Although as you have already found, this will retain the separate commas as if they
were significant tokens instead of just data separators.
I am working with pyparsing and found it to be excellent for developing a simple DSL that allows me to extract data fields out of MongoDB and do simple arithmetic operations on them. I am now trying to extend my tools such that I can apply functions of the form Rank[Person:Height] to the fields and potentially include simple expressions as arguments to the function calls. I am struggling hard with getting the parsing syntax to work. Here is what I have so far:
# Define parser
expr = Forward()
integer = Word(nums).setParseAction(EvalConstant)
real = Combine(Word(nums) + "." + Word(nums)).setParseAction(EvalConstant)
# Handle database field references that are coming out of Mongo,
# accounting for the fact that some fields contain whitespace
dbRef = Combine(Word(alphas) + ":" + Word(printables) + \
Optional(" " + Word(alphas) + " " + Word(alphas)))
dbRef.setParseAction(EvalDBref)
# Handle function calls
functionCall = (Keyword("Rank") | Keyword("ZS") | Keyword("Ntile")) + "[" + expr + "]"
functionCall.setParseAction(EvalFunction)
operand = functionCall | dbRef | (real | integer)
signop = oneOf('+ -')
multop = oneOf('* /')
plusop = oneOf('+ -')
# Use parse actions to attach Eval constructors to sub-expressions
expr << operatorPrecedence(operand,
[
(signop, 1, opAssoc.RIGHT, EvalSignOp),
(multop, 2, opAssoc.LEFT, EvalMultOp),
(plusop, 2, opAssoc.LEFT, EvalAddOp),
])
My issue is that when I test a simple expression like Rank[Person:Height] I am getting a parse exception:
ParseException: Expected "]" (at char 19), (line:1, col:20)
If I use a float or arithmetic expression as the argument like Rank[3 + 1.1] the parsing works ok, and if I simplify the dbRef grammar so its just Word(alphas) it also works. Cannot for the life of me figure out whats wrong with my full grammar. I have tried rearranging the order of operands as well as simplifying the functionCall grammar to no avail. Can anyone see what I am doing wrong?
Once I get this working I would want to take a last step and introduce support for variable assignment in expressions ..
EDIT: Upon further testing, if I remove the printables from dbRef grammar, things work ok:
dbRef = Combine(Word(alphas) + OneOrMore(":") + Word(alphanums) + \
Optional("_" + Word(alphas)))
HOWEVER, if I add the character "-" to dbRef (which I need for DB fields like "Class:S-N"), the parser fails again. I think the "-" is being consumed by the signop in my operatorPrecedence?
What appears to happen is that the ] character at the end of your test string (Rank[Person:Height]) gets consumed as part of the dbRef token, because the portion of this token past the initial : is declared as being made of Word(printables) (and this character set, unfortunately includes the square brackets characters)
Then the parser tries to produce a functionCall but is missing the closing ] hence the error message.
A tentative fix is to use a character set that doesn't include the square brackets, maybe something more explicit like:
dbRef = Combine(Word(alphas) + ":" + Word(alphas, alphas+"-_./") + \
Optional(" " + Word(alphas) + " " + Word(alphas)))
Edit:
Upon closer look, the above is loosely correct, but the token hierarchy is wrong (e.g. the parser attempts to produce a functionCall as one operand of an an expr etc.)
Also, my suggested fix will not work because of the ambiguity with the - sign which should be understood as a plain character when within a dbRef and as a plusOp when within an expr. This type of issue is common with parsers and there are ways to deal with this, though I'm not sure exactly how with pyparsing.
Found solution - the issue was that my grammar for dbRef was consuming some of the characters that were part of the function specification. New grammar that works correctly:
dbRef = Combine(Word(alphas) + OneOrMore(":") + Word(alphanums) + \
Optional(oneOf("_ -") + Word(alphas)))
I'm trying to use pyparsing to parse function calls in the form:
f(x, y)
That's easy. But since it's a recursive-descent parser, it should also be easy to parse:
f(g(x), y)
That's what I can't get. Here's a boiled-down example:
from pyparsing import Forward, Word, alphas, alphanums, nums, ZeroOrMore, Literal
lparen = Literal("(")
rparen = Literal(")")
identifier = Word(alphas, alphanums + "_")
integer = Word( nums )
functor = identifier
# allow expression to be used recursively
expression = Forward()
arg = identifier | integer | expression
args = arg + ZeroOrMore("," + arg)
expression << functor + lparen + args + rparen
print expression.parseString("f(x, y)")
print expression.parseString("f(g(x), y)")
And here's the output:
['f', '(', 'x', ',', 'y', ')']
Traceback (most recent call last):
File "tmp.py", line 14, in <module>
print expression.parseString("f(g(x), y)")
File "/usr/local/lib/python2.6/dist-packages/pyparsing-1.5.6-py2.6.egg/pyparsing.py", line 1032, in parseString
raise exc
pyparsing.ParseException: Expected ")" (at char 3), (line:1, col:4)
Why does my parser interpret the functor of the inner expression as a standalone identifier?
Nice catch on figuring out that identifier was masking expression in your definition of arg. Here are some other tips on your parser:
x + ZeroOrMore(',' + x) is a very common pattern in pyparsing parsers, so pyparsing includes a helper method delimitedList which allows you to replace that expression with delimitedList(x). Actually, delimitedList does one other thing - it suppresses the delimiting commas (or other delimiter if given using the optional delim argument), based on the notion that the delimiters are useful at parsing time, but are just clutter tokens when trying to sift through the parsed data afterwards. So you can rewrite args as args = delimitedList(arg), and you will get just the args in a list, no commas to have to "step over".
You can use the Group class to create actual structure in your parsed tokens. This will build your nesting hierarchy for you, without having to walk this list looking for '(' and ')' to tell you when you've gone down a level in the function nesting:
arg = Group(expression) | identifier | integer
expression << functor + Group(lparen + args + rparen)
Since your args are being Grouped for you, you can further suppress the parens, since like the delimiting commas, they do their job during parsing, but with grouping of your tokens, they are no longer necessary:
lparen = Literal("(").suppress()
rparen = Literal(")").suppress()
I assume 'h()' is a valid function call, just no args. You can allow args to be optional using Optional:
expression << functor + Group(lparen + Optional(args) + rparen)
Now you can parse "f(g(x), y, h())".
Welcome to pyparsing!
The definition of arg should be arranged with the item that starts with another at the left, so it is matched preferentially:
arg = expression | identifier | integer
Paul's answer helped a lot. For posterity, the same can be used to define for loops, as follows (simplified pseudo-parser here, to show the structure):
from pyparsing import (
Forward, Group, Keyword, Literal, OneOrMore)
sep = Literal(';')
if_ = Keyword('if')
then_ = Keyword('then')
elif_ = Keyword('elif')
end_ = Keyword('end')
if_block = Forward()
do_block = Forward()
stmt = other | if_block
stmts = OneOrMore(stmt + sep)
case = Group(guard + then_ + stmts)
cases = case + OneOrMore(elif_ + case)
if_block << if_ + cases + end_