Develop Reference

Recursive regular expressions for defining syntax using 'fr' strings

When creating grammar rules for a language I am making, I would like to be able to check syntax and step through it instead of the current method which often will miss syntax errors.
I've started off using regular expressions to define the grammar like so:
add = r"(\+)"
sub = r"(-)"
mul = r"(\*)"
div = r"(\\)"
pow = r"(\^)"
bin_op = fr"({add}|{sub}|{mul}|{div}|{pow})"
open_br = r"(\()"
close_br = r"(\))"
open_sq = r"(\[)"
close_sq = r"(\])"
dot = r"(\.)"
short_id = r"([A-Za-z]\d*)" # i.e. "a1", "b1232", etc.
long_id = r"([A-Za-z0-9]+)" # i.e. "sin2", "tan", etc. for use in assignment
long_id_ref = r"('" + long_id + "')" # i.e. "'sin'", for referencing
#note that "'sin'" is one value while "sin" = "s" * "i" * "n"
id_assign = fr"({short_id}|{long_id})" # for assignment
id_ref = fr"({short_id}|{long_id_ref})" # for reference (apostrophes)
integer = r"(\d+)" # i.e 123
float = fr"(\d+{dot}\d+)" # i.e. 3.4
operand = fr"({integer}|{float}|{id_ref})"
Now the issue here is that definitions may be recursive, for example in expression = fr"{expression}{bin_op}{expression}|({open_br}{expression}{close_br})|({expression}{open_sq}{expression}{close_sq})|..." and as you can see, I have shown some possible expressions that would be recursive. The issue is, of course, that expression is not defined when defining expression therefore an error would be raised.
It seems that (?R) would not work since it would copy everything before it not the whole string. Does Python's regex have a way of dealing with this or do I have to create my own BNF or regex interpreter that supports recursion?
Alternatively would it be feasible to use regular expressions but not use any recursion?
I know that there are 3rd-party applications that can help with this but I'd like to be able to do it all myself without external code.

Python - How to obtain a dictionary(or even a list) of variables in a specific scope; more specific than locals/globals()

So, the title pretty much says it all.
for instance let's look at the below example code:
## How can I obtain a dict/list (like locals()) of all the variables in second and/or third layer scopes via a command
# coming from the first layer?
## Or another example would be how could I obtain the variables "locals() style" from the thirdlayer via a
# command from the second layer?
# essentially can a parent function/class access a list/dict of a child function
# or class??
def firstLayer():
a = 4.7
q = locals()
print(q)
# local vars of 1st layer
def secondlayer():
b = 7
r = locals()
print(r)
# local vars of 2nd layer
def thirdlayer():
c = False
s = locals()
i = globals()
print('c:\n', c, "\nglobals from 3rd layer:\n\t", i)
# local vars of 3rd layer
thirdlayer()
secondlayer()
firstLayer()
sample_var = globals()
print(sample_var)
# returns the list of global variables
to reiterate what I said in the comments in the code, essentially is their any way I can get a list of all the variables local to a 'child' scope? I know functions are shut off, but if their is no way to do this is their any more complicated code that could achieve this and I could integrate it into a function or class if necessary.
EDIT:
To elaborate further; here's the situation i'm in.
def varsfunc():
font1 = "Harlow Solid"
grey = '#454545'
font2 = 'Mistral'
font3 = 'Italic 35px Times New Roman'
pnk = 'pink'
grn = 'green'
return locals()
Essentially, I am creating a module and the user must create some type of function that they list all of they variables they would like to declare to be used to modify a css file. Essentially, I would like to allow the user to not have to type "return locals()". I want to achieve it by having the end-users wrap the above example function in a decorator that will do the equivalent of returning locals() of the exact scope I want. The decorator does not work for me because it is in an outer scope.
TO BE EVEN MORE CLEAR:
I need a decorator/function that wraps another function(i.e. a decorator), that can access and create a list of a child element.
def module_decorator_func_thing():
r = command_that_acts_like_locals()_but_for_child_scopes
def user_var_list():
font1 = 'green'
font2 = 'pink'
# back in "module_decorator_func_thing"'s scope
print(r) # this variable should contain only a dict/list containing the
# the following:
# r = {'font1': 'green', 'font2': 'pink')
currently users need to do this:
def vars_func_container():
font1 = 'green'
font2 = 'pink'
return locals() # <---- I want the user to not have to type this and for
# a function decorator to take care of it instead possibly.
Info for #aguy and others wishing for more info.
The dictionary/list that I am obtaining via your guys' tips will be sent to this function to do the real job of the program.
(If I were to start using lists, i'd need to convert to a dictionary but that's no problem for me to solve.)
The dict of variables is used with this function to "compile/compyle"(Pun on the word 'Python' + 'compile) and is insert in the "variables" parameter. e.g. you execute the function like this.
compyle("My sample title", return_stylesheet_from_func(*insert .css filename),
return_variables_from_function(*insert function containing variables*), "**True/False to turn on compilation**",
"**True/False to turn on annotations/suggestions**")
def compyle(title, style_sheet, variables, boolean=False, boolean2=True):
"""
:param title: The name you wish your .css file to be named.
:param style_sheet: The name of the multi-line string that will compose your .css file
:param variables: The name of the dictionary containing your .pcss variables
:param boolean: A.K.A the "Compiler Parameter" - Turns the compiler on or off
:param boolean2: A.K.A the "Annotation Parameter" - Turns annotations on or off
:return: returns compiled .pcss text as normal .css style text to be utilized with .html
"""
# -----------------------------------
file_name = title + ".css"
replace_num = len(variables.keys())
counter = replace_num
content = style_sheet
# -----------------------------------
# add theme support with namedtuple's formatted to mimic structs in C/C++
# this will be a major feature update as well as a nice way to allow the future prospect of integrating C/C++ into
# the compiler. Info: https://stackoverflow.com/questions/35988/c-like-structures-in-python
for k, v in variables.items():
counter -= 1
content = content.replace(k, v, replace_num)
if counter == 0:
break
else:
pass
looped_content = str(content)
id_content = looped_content.replace("hash_", "#")
output = id_content.replace("dot_", ".")
if boolean is True:
if boolean2 is True:
output = " /* --- Pyle Sheet --- */\n" + output
with open(file_name, 'w') as writ:
writ.write(output)
writ.close()
print('compiled successfully; The file was saved as ' + "\"" + file_name + "\".")
elif boolean2 is False:
pass
else:
logging.warning("An Error Occurred - see module, documentation, or online Q&A for assistance.")
elif boolean is False:
if boolean2 is True:
print('compiled successfully; The file ' + "\"" + file_name + "\"" + "was not saved/created.")
elif boolean2 is False:
pass
else:
logging.warning("An Error Occurred - see module, documentation, or online Q&A for assistance.")
else:
logging.warning('An Error Occurred with the Compile Parameter (See: boolean in pyle_sheets source file) - \ '
'see module, documentation, or online Q&A for assistance.')

I can't see any way to do this without getting pretty deep; what follows is the simplest solution I've come up with.
how it works
Using the ast module, we go through the code of the given function and find all the assignments. These are evaluated in a given namespace and this namespace is returned.
the code
import ast
import functools
import inspect
def returnAssignments(f):
#functools.wraps(f)
def returner():
assignments = dict()
for node in ast.walk(ast.parse(inspect.getsource(f))):
if isinstance(node, ast.Assign):
exec(compile(ast.Module([node]), '<ast>', 'exec'),
globals(),
assignments)
return assignments
return returner
usage
from ra import returnAssignments
#returnAssignments
def foo():
this = 'something'
that = 37
the_other = object()
print(foo())
output
rat#pandion:~/tmp$ python test.py
{'this': 'something', 'that': 37, 'the_other': <object object at 0x10205b130>}

I wonder if such a crude solution as I provide here might be useful to you. Note that I haven't tested it on all cases, so it might be a bit rough. Also, it returns everything as a string, a behavior which you might want to further change.
Define the function:
def get_local_vars_from_function(f):
import inspect
s = inspect.getsourcelines(f)[0]
d = {}
for l in s:
if '=' in l:
var, val = l.split('=')
var = var.strip()
val = val.strip()
d[var] = val
return d
Then to use it:
In[91]: get_local_vars_from_function(user_var_list)
Out[91]: {'font1': "'green'", 'font2': "'pink'"}

Automatically set results name in pyparsing?

Is there any way to get pyparsing to automatically set the resultsName of a grammar element to whatever it's named in my source code? That is, I would write code like
my_number = Word(nums)
and it would automatically execute
my_number.setResultsName('my_number')

You should be able to simply do:
my_number = Word(nums)('my_number')
using the shortcut for .setResultsName. Python, in general, makes it hard to get at the name of the variable in question.
As an alternative, if you had a list of them as a dictionary you could do something like:
for key,val in grammar_dict.items():
grammar_dict[key] = val.setResultsName(key)

Using the inspect module you can do what you want. Define a function srn:
import inspect
def srn(expr):
"""Sets the results name to the variable *name* of `expr`"""
cf = inspect.currentframe()
of = inspect.getouterframes(cf)[1]
fs = inspect.getframeinfo(of[0]).code_context[0].strip()
# name of FIRST parameter
try:
args = fs[fs.find('(') + 1:-1].split(',')
n = args[0]
if n.find('=') != -1:
name = n.split('=')[1].strip()
else:
name = n
expr.resultsName = name
except IndexError:
pass
return expr
Then after my_number = Word(nums) call:
srn(my_number)
and my_number contains "my_number" as resultsName.
Disclaimer 1: This approach works well, but is quite hacky and dives deep into the Python internals, so do it at your own risk.
Disclaimer 2: The idea for this is not mine - I got it somewhere from StackOverflow but I don't now exactly where...

python variable concatination with different equators

How would I be able to make a variable that contains another = sign in it, like this:
newval = dict[key_to_find] = int(change)

The problem with doing this is that in Python the '=' is an assignment operator rather than a symbol.
Python is reading what you wrote as trying to assign to two variables. For example:
foo = "test" and bar = "test" can be written as foo = bar = "test"
Creating a variable name with an '=' it can't be done.
If you are trying to create a new variable that has a string value that comes from two other sources you can use the format that intboolstring suggested, or the format function is pretty handy.
Going along with your example:
newval = "{dict} = {int}".format(dict=dict[key_to_find], int = int(change))
To give a simpler example of how this works:
var_a = 'Hello'
var_b = 'world'
variable = "{a} {b}!".format(a = var_a, b = var_b)
Variable will print:
"Hello world!"

I'm basing the answer off of
variable that contains another = sign in it
I think you are trying to concatenate dict[key_to_find] and int(change) with an equal sign in the middle. This can be done with the following
newval = str(dict[key_to_find]) + " = " + str(int(change))
The reason that I'm leaving in the int cast is because if you had change as 7.5, then you would want it in the string as 7.

Decorator to define function-local statics - fine details of AST-munging

I am trying to produce a better answer to the frequently-asked question "How do I do function-local static variables in Python?" (1, 2, 3, ...) "Better" means completely encapsulated in a decorator, that can be used in any context where a function definition may appear. In particular, it must DTRT when applied to methods and nested functions; it must play nice with other decorators applied to the same function (in any order); it must accept arbitrary initializers for the static variables, and it must not modify the formal parameter list of the decorated function. Basically, if this were to be proposed for inclusion in the standard library, nobody should be able to object on quality-of-implementation grounds.
Ideal surface syntax would be
#static_vars(a=0, b=[])
def test():
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
I would also accept
#static_vars(a=0, b=[])
def test():
static.b.append(static.a)
static.a += 1
sys.stdout.write(repr(static.b) + "\n")
or similar, as long as the namespace for the static variables is not the name of the function! (I intend to use this in functions that may have very long names.)
A slightly more motivated example involves precompiled regular expressions that are only relevant to one function:
#static_vars(encode_re = re.compile(
br'[\x00-\x20\x7F-\xFF]|'
br'%(?!(?:[0-9A-Fa-f]{2}|u[0-9A-Fa-f]{4}))')
def encode_nonascii_and_percents(segment):
segment = segment.encode("utf-8", "surrogateescape")
return encode_re.sub(
lambda m: "%{:02X}".format(ord(m.group(0))).encode("ascii"),
segment).decode("ascii")
Now, I already have a mostly-working implementation. The decorator rewrites each function definition as if it had read like so (using the first example):
def _wrap_test_():
a = 0
b = 1
def test():
nonlocal a, b
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
test = _wrap_test_()
del _wrap_test_
It seems that the only way to accomplish this is to munge the AST. I have code that works for simple cases (see below) but I strongly suspect it is wrong in more complicated cases. For instance, I think it will break if applied to a method definition, and of course it also breaks in any situation where inspect.getsource() fails.
So the question is, first, what should I do to make it work in more cases, and second, is there a better way to define a decorator with the same black-box effects?
Note 1: I only care about Python 3.
Note 2: Please assume that I have read all of the proposed solutions in all of the linked questions and found all of them inadequate.
#! /usr/bin/python3
import ast
import functools
import inspect
import textwrap
def function_skeleton(name, args):
"""Return the AST of a function definition for a function named NAME,
which takes keyword-only args ARGS, and does nothing. Its
.body field is guaranteed to be an empty array.
"""
fn = ast.parse("def foo(*, {}): pass".format(",".join(args)))
# The return value of ast.parse, as used here, is a Module object.
# We want the function definition that should be the Module's
# sole descendant.
assert isinstance(fn, ast.Module)
assert len(fn.body) == 1
assert isinstance(fn.body[0], ast.FunctionDef)
fn = fn.body[0]
# Remove the 'pass' statement.
assert len(fn.body) == 1
assert isinstance(fn.body[0], ast.Pass)
fn.body.clear()
fn.name = name
return fn
class static_vars:
"""Decorator which provides functions with static variables.
Usage:
#static_vars(foo=1, bar=2, ...)
def fun():
foo += 1
return foo + bar
The variables are implemented as upvalues defined by a wrapper
function.
Uses introspection to recompile the decorated function with its
context changed, and therefore may not work in all cases.
"""
def __init__(self, **variables):
self._variables = variables
def __call__(self, func):
if func.__name__ in self._variables:
raise ValueError(
"function name {} may not be the same as a "
"static variable name".format(func.__name__))
fname = inspect.getsourcefile(func)
lines, first_lineno = inspect.getsourcelines(func)
mod = ast.parse(textwrap.dedent("".join(lines)), filename=fname)
# The return value of ast.parse, as used here, is a Module
# object. Save that Module for use later and extract the
# function definition that should be its sole descendant.
assert isinstance(mod, ast.Module)
assert len(mod.body) == 1
assert isinstance(mod.body[0], ast.FunctionDef)
inner_fn = mod.body[0]
mod.body.clear()
# Don't apply decorators twice.
inner_fn.decorator_list.clear()
# Fix up line numbers. (Why the hell doesn't ast.parse take a
# starting-line-number argument?)
ast.increment_lineno(inner_fn, first_lineno - inner_fn.lineno)
# Inject a 'nonlocal' statement declaring the static variables.
svars = sorted(self._variables.keys())
inner_fn.body.insert(0, ast.Nonlocal(svars))
# Synthesize the wrapper function, which will take the static
# variableas as arguments.
outer_fn_name = ("_static_vars_wrapper_" +
inner_fn.name + "_" +
hex(id(self))[2:])
outer_fn = function_skeleton(outer_fn_name, svars)
outer_fn.body.append(inner_fn)
outer_fn.body.append(
ast.Return(value=ast.Name(id=inner_fn.name, ctx=ast.Load())))
mod.body.append(outer_fn)
ast.fix_missing_locations(mod)
# The new function definition must be evaluated in the same context
# as the original one. FIXME: supply locals if appropriate.
context = func.__globals__
exec(compile(mod, filename="<static-vars>", mode="exec"),
context)
# extract the function we just defined
outer_fn = context[outer_fn_name]
del context[outer_fn_name]
# and call it, supplying the static vars' initial values; this
# returns the adjusted inner function
adjusted_fn = outer_fn(**self._variables)
functools.update_wrapper(adjusted_fn, func)
return adjusted_fn
if __name__ == "__main__":
import sys
#static_vars(a=0, b=[])
def test():
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
test()
test()
test()
test()

Isn't this what classes are for?
import sys
class test_class:
a=0
b=[]
def test(self):
test_class.b.append(test_class.a)
test_class.a += 1
sys.stdout.write(repr(test_class.b) + "\n")
t = test_class()
t.test()
t.test()
[0]
[0, 1]
Here is a version of your regexp encoder:
import re
class encode:
encode_re = re.compile(
br'[\x00-\x20\x7F-\xFF]|'
br'%(?!(?:[0-9A-Fa-f]{2}|u[0-9A-Fa-f]{4}))')
def encode_nonascii_and_percents(self, segment):
segment = segment.encode("utf-8", "surrogateescape")
return encode.encode_re.sub(
lambda m: "%{:02X}".format(ord(m.group(0))).encode("ascii"),
segment).decode("ascii")
e = encode()
print(e.encode_nonascii_and_percents('foo bar'))
foo%20bar
There is always the singleton class.
Is there a simple, elegant way to define Singletons in Python?