Develop Reference

Recursive regular expressions for defining syntax using 'fr' strings

When creating grammar rules for a language I am making, I would like to be able to check syntax and step through it instead of the current method which often will miss syntax errors.
I've started off using regular expressions to define the grammar like so:
add = r"(\+)"
sub = r"(-)"
mul = r"(\*)"
div = r"(\\)"
pow = r"(\^)"
bin_op = fr"({add}|{sub}|{mul}|{div}|{pow})"
open_br = r"(\()"
close_br = r"(\))"
open_sq = r"(\[)"
close_sq = r"(\])"
dot = r"(\.)"
short_id = r"([A-Za-z]\d*)" # i.e. "a1", "b1232", etc.
long_id = r"([A-Za-z0-9]+)" # i.e. "sin2", "tan", etc. for use in assignment
long_id_ref = r"('" + long_id + "')" # i.e. "'sin'", for referencing
#note that "'sin'" is one value while "sin" = "s" * "i" * "n"
id_assign = fr"({short_id}|{long_id})" # for assignment
id_ref = fr"({short_id}|{long_id_ref})" # for reference (apostrophes)
integer = r"(\d+)" # i.e 123
float = fr"(\d+{dot}\d+)" # i.e. 3.4
operand = fr"({integer}|{float}|{id_ref})"
Now the issue here is that definitions may be recursive, for example in expression = fr"{expression}{bin_op}{expression}|({open_br}{expression}{close_br})|({expression}{open_sq}{expression}{close_sq})|..." and as you can see, I have shown some possible expressions that would be recursive. The issue is, of course, that expression is not defined when defining expression therefore an error would be raised.
It seems that (?R) would not work since it would copy everything before it not the whole string. Does Python's regex have a way of dealing with this or do I have to create my own BNF or regex interpreter that supports recursion?
Alternatively would it be feasible to use regular expressions but not use any recursion?
I know that there are 3rd-party applications that can help with this but I'd like to be able to do it all myself without external code.

How do I detect variables in a python eval expression

Let's say i get a string to eval
temperature*x
And I have two sets of variables - the easy one:
easy_ns = {'x':3, 'y':4}
And a harder one:
harder = ['temperature', 'sumofall']
Each of which will take significant time to calculate and I don't want to calculate them unless they are part of the user supplied expression
E.g. I don't want to start the detection of "temperature" unless I know it is required
I may have some variables in my namespace that are "inexpensive" but others I would like to postpone calculating as much as possible
How do I get a list of variables from my eval string before it is evaluated
I know I can try: eval() except: and I will get a:
NameError: name 'temperature' is not defined
Is there a pythonic way of extracting the exact variable name?
Is there a nice way to build your namespace for lazy evaluation?
Something like
namespace = {'x':3, 'y':4, 'temperature':lazy_temperature_function}
So that only when my expression is evaluated
res=eval('temperature*x')
is my lazy temperature function called
And yes of course - I absolutely do have to use 'eval' - that is why I have posted these questions
The scenario is that I get an input file with set of keys and values and then the user can supply an expression he wants me to calculate from a combination of those values and some generated variables that I do not want to calculate unless the user includes them in his/her expression

You could, if you really really have to, parse the code using the ast module. The ast.parse helper will give you an AST tree representation of the code:
import ast
code = "temperature*x"
st = ast.parse(code)
for node in ast.walk(st):
if type(node) is ast.Name:
print(node.id)
This will print:
temperature
x
So this only extracts the variable names, like you said. It seems like a first step, but I'm not sure what you are trying to do so maybe a different approach is better.
Edit: If I understand your problem correctly, you want some values to be calculated only if they appear in an expression? I tried something like this:
>>> import ast
>>> code = "temperature*x"
>>> x = 5
>>> def lazy_temperature():
return 78
...
>>> names = [node.id for node in ast.walk(ast.parse(code))
if type(node) is ast.Name]
>>> ns = {name: (globals()['lazy_%s' % name])()
if ('lazy_%s' % name) in globals()
else globals()[name]
for name in names}
>>> ns
{'x': 5, 'temperature': 78}
>>> eval(code, ns)
390
This snippet will load the value out of the current scope, unless there's a function called lazy_<name>. This function will be called in case the <name> part appears in the expression.

You could make it a lambda function, and simply execute it whenever you need to, as such:
a = lambda : 5*temperature
a()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "<stdin>", line 1, in <lambda>
> NameError: global name 'temperature' is not defined
temperature = 100
a()
> 500
In this way, you don't look for execute unless you consciously want to.
However, you could also make sure to only enter the lambda function if temperature exists. You can do that by assigning temperature to None in the beginning of your file, and only enter the lambda if you need to:
temperature = None
if temperature:
a()
# do something else
If you don't want to use the parens `fn() - You could also build a class to do this with a property.
class a(object):
#property
def temp_calc(self):
return self.temp*5
In this way, you can do the following:
temp_obj = a()
temp_obj.temp_calc
This will return an error since you don't have a "temp" attribute. But you can assign it if you need to:
temp_obj.temp = 5
temp_obj.temp_calc
> 25
There are lots of options here, i hope these few help.

How do I get the alias of a function call?

The problem I'm having is I don't know how to get the alias of a function call.
what the code works on is for tracking the text:
"struct(...)"
in the code:
x = struct(...)
when calling:
x()
How the IDE works is 'x = struct(...)' only defines an area in a call-tree which creates an indent in the tree, it doesn't actually do anything in the file data.
When you call 'x()' is when the file data is highlighted with the color of 'color = hash("struct(...)")'.
So I need to get the alias 'x' so I can track the call in the code...
I can't exactly help anyone reproduce this as the code for doing so is rather large...
but I just need ideas as I can't seem to find any decent examples on Google.
I'm looking for cases such as:
the obvious:
x = struct(...)
the not so obvious:
p,x,t = 0, struct(...), True
the highly irrelevant:
p,x,t = (
0,
struct(...),
True )
all resulting in a call to x()
I'm using tokenize to get the call-name of struct(), and I have the entire code stored in 'self.codeeditor.data'...
How do I use "struct" to get "x"??
EDIT:
I could mention that x would be in instance of a dynamically created _SUB_STRUCT class returned by struct().

I don't think tokenize will really work here; you'd be better off working on the level of syntax trees using ast and looking for Assign nodes.
For example:
>>> [n.targets[0].id for n in ast.walk(ast.parse("x = struct()"))
if isinstance(n, ast.Assign)
and isinstance(n.value, ast.Call)
and n.value.func.id == 'struct']
['x']

You should be able to use a dict to get the functionality you desire:
# Relates a label to a function
function_maps = {'a':str, 'b':int, 'c':print}
# Will print 'Hello World!'
function_maps['c']('Hello World!')
# Lets look for the `int` function
for key, value in function_maps.items():
if value == int:
print('The alias for `int` is: {}'.format(key))
Without seeing more of your code, this is the way I would suggest doing it.

Alternatives for 3 lines of java-like code?

Assume you have a function, that sometimes returns a value, and sometimes doesn't, because there really is nothing you could return in this case, not even a default value or something. Now you want to do something with the result, but of course only when there is one.
Example:
result = function_call(params)
if result:
print result
Is there a way to write this in a more pythonic way, maybe even in one line?
Like that:
print function_call(params) or #nothing
(Note that I mean it shouldn't print "nothing" or "None". It should actually just not print at all, if the result is None)

No; in Python, name binding is a statement and so cannot be used as an expression within a statement. Since print is also a statement you're going to require 3 lines; in Python 3 you could write:
result = function_call(params)
print(result) if result else None
This isn't quite true for name binding within a comprehension or generator, where name binding is a syntax item that has statement-like semantics:
[print(result) for result in generator_call(params) if result]
As Kos says, you can abuse this to create a one-element comprehension:
[print(result) for result in (function_call(params), ) if result]
Another syntax item that performs name binding and can similarly be abused is the lambda expression:
(lambda result: print(result) if result else None)(function_call(params))
Note that in both these cases the operation on the return value must be an expression and not a statement.

I think the more Pythonic version is actually closer to your original:
result = function_call(params)
if result is not None:
do_something(result)
Checking for is (not) None seems very idiomatic to me - I've used it several times myself and I've also seen it used elsewhere[citation-needed].

From the answers up to now I would do that:
>>> from __future__ import print_function #if Python2.7
>>> def filtered_print(txt):
... txt and print(txt)
...
>>> filtered_print('hello world')
hello world
>>> filtered_print('None')
None
>>> filtered_print(None)
>>>
If someone else has a better solution in mind, I am still open for alternatives, though!

Python: multiple possible values for function arguments

I've inherited some Python code that looks like this:
name = 'London'
code = '0.1'
notes = 'Capital of England'
ev = model.City(key=key, code=code, name=name or code, notes=notes)
In the spirit of learning, I'd like to know what's going on with the name or code argument. Is this saying 'Use name if it's not null, otherwise use code'?
And what is the technical term for supplying multiple possible arguments like this, so I can read up on it in the Python docs?
Thanks!

Almost. It says use name if it does not evaluate to false. Things that evaluate to false include, but are not limited to:
False
empty sequences ((), [], "")
empty mappings ({})
0
None
Edit Added the link provided by SilentGhost in his comment to the answer.

In python, the or operator returns the first operand, unless it evaluates to false, in which case it returns the second operand. In effect this will use name, with a default fallback of code if name is not specified.

Fire up a Python console:
>>> name = None
>>> code = 0.1
>>> name or code
0.10000000000000001
In case name evaluates to false the expression will evaluate to code. Otherwise name will be used.

Correct, that idiom take the first value that evaluates to True (generally not None). Use with care since valid values (like zero) may inadvertently be forsaken. A safer approach is something like:
if name is not None:
# use name
or
name if name is not None else code

You've it it roughly correct, but 'null' is not precisely what decides. Basically anything that will evaluate to false (0, false, empty string '') will cause the second string to be displayed instead of the first. 'x or y' in this sense is kind of equivalent to:
if x: x
else: y
Some console play:
x = ''
y = 'roar'
x or y
-'roar'
x = 'arf'
x or y
-'arf'
x = False
x or y
-'roar'

In the spirit of learning, I'd like to
know what's going on with the name or
code argument. Is this saying 'Use
name if it's not null, otherwise use
code'?
yes basically but Null in python can mean more than one thing (empty string , none ..)
like in your case:
>>> name = 'London'
>>> code = 0.1
>>> name or code
'London'
>>> name = ''
>>> code = 0.1
>>> name or code
0.1000....
but it weird thew that a function parameter can be integer sometimes and a string other times.
Hope this can help :=)