I have this program:
a = 0
def f(c):
a = c
return a + g(a)
def g(b):
return a + b
f(3) #returns 6
How does it return the value 6? So far I know how binding environments are created when the python interpreter evaluates a variable and a procedure. Also I know the details how a new environment is made when a simple function (in the sense it doesn't call a new function) is called. After the call g(a) inside f, is there a new environment created exclusive for g that is one level below the global environment, or one level below the environment of f?
It's exactly the same as:
a = 0
def f(c):
var = c
return var + g(var)
def g(b):
return a + b
f(3)
And the same as:
def f(c):
var = c
return var + g(var)
def g(b):
return 0 + b
f(3)
And even more simplified:
def f(c):
return c + g(c)
def g(b):
return b
f(3)
When only reading the values of global variables, you can use them inside a function just fine. But if you try to assign to a global variable you need to use the global keyword, or otherwise you shadow the global with the local variable:
x= 123
def foo():
x = 999 # shadows global x
print(x) # 999
print(x) # 123
x = 123
def foo():
global x
x = 999 # changes global x
print(x) # 999
print(x) # 999
Related
The Question
Is there a straightforward algorithm for figuring out if a variable is "used" within a given scope?
In a Python AST, I want to remove all assignments to variables that are not otherwise used anywhere, within a given scope.
Details
Motivating example
In the following code, it is obvious to me (a human), that _hy_anon_var_1 is unused, and therefore the _hy_anon_var_1 = None statements can be removed without changing the result:
# Before
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
_hy_anon_var_1 = None
else:
n = 3 * n + 1
_hy_anon_var_1 = None
yield n
# After
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
else:
n = 3 * n + 1
yield n
Bonus version
Extend this to []-lookups with string literals as keys.
In this example, I would expect _hyx_letXUffffX25['x'] to be eliminated as unused, because _hyx_letXUffffX25 is local to h, so _hyx_letXUffffX25['x'] is essentially the same thing as a local variable. I would then expect _hyx_letXUffffX25 itself to be eliminated once there are no more references to it.
# Before
def h():
_hyx_letXUffffX25 = {}
_hyx_letXUffffX25['x'] = 5
return 3
# After
def h():
return 3
From what I can tell, this is somewhat of an edge case, and I think the basic algorithmic problem is the same.
Definition of "used"
Assume that no dynamic name lookups are used in the code.
A name is used if any of these are true in a given scope:
It is referenced anywhere in an expression. Examples include: an expression in a return statement, an expression on the right-hand side of an assignment statement, a default argument in a function definition, being referenced inside a local function definition, etc.
It is referenced on the left-hand side of an "augmented assignment" statement, i.e. it is an augtarget therein. This might represent "useless work" in a lot of programs, but for the purpose of this task that's OK and distinct from being an entirely unused name.
It is nonlocal or global. These might be useless nonlocals or globals, but because they reach beyond the given scope, it is OK for my purposes to assume that they are "used".
Please let me know in the comments if this seems incorrect, or if you think I am missing something.
Examples of "used" and "unused"
Example 1: unused
Variable i in f is unused:
def f():
i = 0
return 5
Example 2: unused
Variable x in f is unused:
def f():
def g(x):
return x/5
x = 10
return g(100)
The name x does appear in g, but the variable x in g is local to g. It shadows the variable x created in f, but the two x names are not the same variable.
Variation
If g has no parameter x, then x is in fact used:
def f():
x = 10
def g():
return x/5
return g(100)
Example 3: used
Variable i in f is used:
def f():
i = 0
return i
Example 4: used
Variable accum in silly_map and silly_sum is used in both examples:
def silly_map(func, data):
data = iter(data)
accum = []
def _impl():
try:
value = next(data)
except StopIteration:
return accum
else:
accum.append(value)
return _impl()
return _impl()
def silly_any(func, data):
data = iter(data)
accum = False
def _impl():
nonlocal accum, data
try:
value = next(data)
except StopIteration:
return accum
else:
if value:
data = []
accum = True
else:
return _impl()
return _impl()
The solution below works in two parts. First, the syntax tree of the source is traversed and all unused target assignment statements are discovered. Second, the tree is traversed again via a custom ast.NodeTransformer class, which removes these offending assignment statements. The process is repeated until all unused assignment statements are removed. Once this is finished, the final source is written out.
The ast traverser class:
import ast, itertools, collections as cl
class AssgnCheck:
def __init__(self, scopes = None):
self.scopes = scopes or cl.defaultdict(list)
#classmethod
def eq_ast(cls, a1, a2):
#check that two `ast`s are the same
if type(a1) != type(a2):
return False
if isinstance(a1, list):
return all(cls.eq_ast(*i) for i in itertools.zip_longest(a1, a2))
if not isinstance(a1, ast.AST):
return a1 == a2
return all(cls.eq_ast(getattr(a1, i, None), getattr(a2, i, None))
for i in set(a1._fields)|set(a2._fields) if i != 'ctx')
def check_exist(self, t_ast, s_path):
#traverse the scope stack and remove scope assignments that are discovered in the `ast`
s_scopes = []
for _ast in t_ast:
for sid in s_path[::-1]:
s_scopes.extend(found:=[b for _, b in self.scopes[sid] if AssgnCheck.eq_ast(_ast, b) and \
all(not AssgnCheck.eq_ast(j, b) for j in s_scopes)])
self.scopes[sid] = [(a, b) for a, b in self.scopes[sid] if b not in found]
def traverse(self, _ast, s_path = [1]):
#walk the ast object itself
_t_ast = None
if isinstance(_ast, ast.Assign): #if assignment statement, add ast object to current scope
self.traverse(_ast.targets[0], s_path)
self.scopes[s_path[-1]].append((True, _ast.targets[0]))
_ast = _ast.value
if isinstance(_ast, (ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef)):
s_path = [*s_path, (nid:=(1 if not self.scopes else max(self.scopes)+1))]
if isinstance(_ast, (ast.FunctionDef, ast.AsyncFunctionDef)):
self.scopes[nid].extend([(False, ast.Name(i.arg)) for i in _ast.args.args])
_t_ast = [*_ast.args.defaults, *_ast.body]
self.check_exist(_t_ast if _t_ast is not None else [_ast], s_path) #determine if any assignment statement targets have previously defined names
if _t_ast is None:
for _b in _ast._fields:
if isinstance((b:=getattr(_ast, _b)), list):
for i in b:
self.traverse(i, s_path)
elif isinstance(b, ast.AST):
self.traverse(b, s_path)
else:
for _ast in _t_ast:
self.traverse(_ast, s_path)
Putting it all together:
class Visit(ast.NodeTransformer):
def __init__(self, asgn):
super().__init__()
self.asgn = asgn
def visit_Assign(self, node):
#remove assignment nodes marked as unused
if any(node.targets[0] == i for i in self.asgn):
return None
return node
def remove_assgn(f_name):
tree = ast.parse(open(f_name).read())
while True:
r = AssgnCheck()
r.traverse(tree)
if not (k:=[j for b in r.scopes.values() for k, j in b if k]):
break
v = Visit(k)
tree = v.visit(tree)
return ast.unparse(tree)
print(remove_assgn('test_name_assign.py'))
Output Samples
Contents of test_name_assign.py:
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
_hy_anon_var_1 = None
else:
n = 3 * n + 1
_hy_anon_var_1 = None
yield n
Output:
def hailstone_sequence(n: int) -> Iterable[int]:
while n != 1:
if 0 == n % 2:
n //= 2
else:
n = 3 * n + 1
yield n
Contents of test_name_assign.py:
def h():
_hyx_letXUffffX25 = {}
_hyx_letXUffffX25['x'] = 5
return 3
Output:
def h():
return 3
Contents of test_name_assign.py:
def f():
i = 0
return 5
Output:
def f():
return 5
Contents of test_name_assign.py:
def f():
x = 10
def g():
return x/5
return g(100)
Ouptut:
def f():
x = 10
def g():
return x / 5
return g(100)
Consider this example:
def func1():
val = 1
res = [1]
def fun2():
print(res)
print(val)
val = 2
fun2()
print(val)
func1()
It raises the following exception:
UnboundLocalError: local variable 'val' referenced before assignment
List res can be accessed by fun2, but val cannot. I know list is mutable and int is not, but is there a way to make val accessible by fun2 as well? In a class, I could easily achieve that with self.val, but is there a way to do it inside a function?
Use the nonlocal statement to make a variable defined in an enclosing function accesible inside the inner function, like so:
def func1():
val = 1
res = [1]
def fun2():
nonlocal val
print(res)
print(val)
val = 2
fun2()
print(val)
func1()
See also: earlier SO question.
You can do it in the following way:
def func1():
val = 1
res = [1]
def fun2(val=val, res=res):
print(res)
print(val)
val = 2
return val
val = fun2()
print(val)
Output is then
>>> func1()
[1]
1
2
I am trying to implement some functions but I'm getting an error with len_link function:
NameError: global name 'len_link' is not defined
While the other function is working perfectly fine, any clue why this error is occurring in the first place?
class Solution:
# #param A : head node of linked list
# #param B : head node of linked list
def len_link(A):
temp=A.head
count=0
while(temp):
count+=1
temp=temp.next
return count
def longerlist(B,n):
for i in range(n):
B = B.next
return B
def getIntersectionNode(self, A, B):
m = len_link(A)
n = len_link(B)
d = abs(m-n)
if m>n :
A = longerlist(A,n)
elif m<n:
B = longerlist(B,n)
while A!= None and B!= None:
if A.val == B.val:
return A
A = A.next
B = B.next
You need to call Solution.len_link not just len_link. Otherwise Python expects it to be a name at global scope.
Also, since len_link does not take a self parameter, it needs to be decorated with #staticmethod:
#staticmethod
def len_link(A):
Or create a function outside of that class, by this you will able to call it globally or go with classname.functionname()
I need to write a function (say fun1) that has one argument, because it will be used in other function (fun2). The latter requires a function with a single argument. However, I need to pass other parameters to function fun1. How can I do this in Python without using global variables? Or this is the only way?
Addition: If it is important, fun2 is some optimization function from scipy.optimize. Below is an example of passing additional parameter c to function fun1 using global. In the first call, function fun2 takes fun1 as x+1, but in the second call, fun1 is x+2. I would like to make similar, but without using global. Hopefully, the example clarifies the question. (The example is changed).
def fun1(x) :
global c
return x + c
def fun2(f1, x) :
return f1(x)
# main program
global c
x0= 1
c= 1; y= fun2(fun1, x0); print(y) # gives 2
c= 2; y= fun2(fun1, x0); print(y) # gives 3
If I've understood your question correctly, there are quite a number of ways to do what you want and avoid using global variables. Here they are.
Given:
x0 = 1
def fun2(f1, x):
return f1(x)
All of these techniques accomplish your goal:
#### #0 -- function attributes
def fun1(x):
return x + fun1.c
fun1.c = 1; y = fun2(fun1, x0); print(y) # --> 2
fun1.c = 2; y = fun2(fun1, x0); print(y) # --> 3
#### #1 -- closure
def fun1(c):
def wrapper(x):
return x + c
return wrapper
y = fun2(fun1(c=1), x0); print(y) # --> 2
y = fun2(fun1(c=2), x0); print(y) # --> 3
#### #2 -- functools.partial object
from functools import partial
def fun1(x, c):
return x + c
y = fun2(partial(fun1, c=1), x0); print(y) # --> 2
y = fun2(partial(fun1, c=2), x0); print(y) # --> 3
#### #3 -- function object (functor)
class Fun1(object):
def __init__(self, c):
self.c = c
def __call__(self, x):
return x + self.c
y = fun2(Fun1(c=1), x0); print(y) # --> 2
y = fun2(Fun1(c=2), x0); print(y) # --> 3
#### #4 -- function decorator
def fun1(x, c):
return x + c
def decorate(c):
def wrapper(f):
def wrapped(x):
return f(x, c)
return wrapped
return wrapper
y = fun2(decorate(c=1)(fun1), x0); print(y) # --> 2
y = fun2(decorate(c=2)(fun1), x0); print(y) # --> 3
Note that writing c= arguments wasn't always strictly required in the calls -- I just put it in all of the usage examples for consistency and because it makes it clearer how it's being passed.
The fact that that function can be called even without those other parameters suggests, that they are optional and have some default value. So you should use default arguments.
def fun1(foo, bar='baz'):
# do something
This way you can call function fun1('hi') and bar will default to 'baz'. You can also call it fun1('hi', 15).
If they don't have any reasonable default, you can use None as the default value instead.
def fun1(foo, bar=None):
if bar is None:
# `bar` argument was not provided
else:
# it was provided
What you are looking for is a method in a class.
you define a class, with a method fun1 and an instance variable c. it is accessed from anywhere using the . notation:
class A:
def fun1(self, x):
return x + self.c
Let's define fun2, for the example:
def fun2(f, p):
return f(p)
We can now use a.c it like you did with the global varaible c:
>>> a = A() # create an instance and initialize it
>>> # "self.c" is undefined yet
>>>
>>> a.c = 1 # "self.c" will be 1
>>> fun2(a.fun1, 1)
2
>>> a.c = 2 # now "self.c" will be 2
>>> fun2(a.fun1, 1) # same arguments, different result
3
Here you can learn more about classes.
Just add the extra parameters with default values:
def fun1(param1, param2=None, param3=None):
...
Then you can call fun1 from fun2 like this:
def fun2():
something = fun1(42)
And from somewhere else you can call it like this:
fun1(42, param2=60)
You may use the decorators to pass it
the very decorators:
def jwt_or_redirect(fn):
#wraps(fn)
def decorator(*args, **kwargs):
...
return fn(*args, **kwargs)
return decorator
def jwt_refresh(fn):
#wraps(fn)
def decorator(*args, **kwargs):
...
new_kwargs = {'refreshed_jwt': 'xxxxx-xxxxxx'}
new_kwargs.update(kwargs)
return fn(*args, **new_kwargs)
return decorator
and the final function:
#jwt_or_redirect
#jwt_refresh
def home_page(*args, **kwargs):
return kwargs['refreched_jwt']
Is there any way to modify the bound value of one of the variables inside a closure? Look at the example to understand it better.
def foo():
var_a = 2
var_b = 3
def _closure(x):
return var_a + var_b + x
return _closure
localClosure = foo()
# Local closure is now "return 2 + 3 + x"
a = localClosure(1) # 2 + 3 + 1 == 6
# DO SOME MAGIC HERE TO TURN "var_a" of the closure into 0
# ...but what magic? Is this even possible?
# Local closure is now "return 0 + 3 + x"
b = localClosure(1) # 0 + 3 +1 == 4
It is quite possible in python 3 thanks to the magic of nonlocal.
def foo():
var_a = 2
var_b = 3
def _closure(x, magic = None):
nonlocal var_a
if magic is not None:
var_a = magic
return var_a + var_b + x
return _closure
localClosure = foo()
# Local closure is now "return 2 + 3 + x"
a = localClosure(1) # 2 + 3 + 1 == 6
print(a)
# DO SOME MAGIC HERE TO TURN "var_a" of the closure into 0
localClosure(0, 0)
# Local closure is now "return 0 + 3 + x"
b = localClosure(1) # 0 + 3 +1 == 4
print(b)
I don't think there is any way to do that in Python. When the closure is defined, the current state of variables in the enclosing scope is captured and no longer has a directly referenceable name (from outside the closure). If you were to call foo() again, the new closure would have a different set of variables from the enclosing scope.
In your simple example, you might be better off using a class:
class foo:
def __init__(self):
self.var_a = 2
self.var_b = 3
def __call__(self, x):
return self.var_a + self.var_b + x
localClosure = foo()
# Local closure is now "return 2 + 3 + x"
a = localClosure(1) # 2 + 3 + 1 == 6
# DO SOME MAGIC HERE TO TURN "var_a" of the closure into 0
# ...but what magic? Is this even possible?
localClosure.var_a = 0
# Local closure is now "return 0 + 3 + x"
b = localClosure(1) # 0 + 3 +1 == 4
If you do use this technique I would no longer use the name localClosure because it is no longer actually a closure. However, it works the same as one.
I've found an alternate answer answer to Greg's, slightly less verbose because it uses Python 2.1's custom function attributes (which conveniently enough can be accessed from inside their own function).
def foo():
var_b = 3
def _closure(x):
return _closure.var_a + var_b + x
_closure.func_dict['var_a'] = 2
return _closure
localClosure = foo()
# Local closure is now "return 2 + 3 + x"
a = localClosure(1) # 2 + 3 + 1 == 6
# DO SOME MAGIC HERE TO TURN "var_a" of the closure into 0
# ...but what magic? Is this even possible?
# apparently, it is
localClosure.var_a = 0
# Local closure is now "return 0 + 3 + x"
b = localClosure(1) # 0 + 3 +1 == 4
Thought I'd post it for completeness. Cheers anyways.
We've done the following. I think it's simpler than other solutions here.
class State:
pass
def foo():
st = State()
st.var_a = 2
st.var_b = 3
def _closure(x):
return st.var_a + st.var_b + x
def _set_a(a):
st.var_a = a
return _closure, _set_a
localClosure, localSetA = foo()
# Local closure is now "return 2 + 3 + x"
a = localClosure(1) # 2 + 3 + 1 == 6
# DO SOME MAGIC HERE TO TURN "var_a" of the closure into 0
localSetA(0)
# Local closure is now "return 0 + 3 + x"
b = localClosure(1) # 0 + 3 +1 == 4
print a, b
I worked around a similar limitation by using one-item lists instead of a plain variable. It's ugly but it works because modifying a list item doesn't get treated as a binding operation by the interpreter.
For example:
def my_function()
max_value = [0]
def callback (data)
if (data.val > max_value[0]):
max_value[0] = data.val
# more code here
# . . .
results = some_function (callback)
store_max (max_value[0])
Question
Is there any way to modify the bound value of one of the variables inside a closure?
TLDR
Yes, this is possible starting in Python 3.7.0 alpha 1:
localClosure.__closure__[0].cell_contents = 0
Details
In Python, a closure remembers the variables from the scope in which it was defined by using a special __closure__ attribute. The __closure__ attribute is a tuple of cell objects representing the variables from the outer scope, and the values of those variables are stored in the cell_contents attribute of each cell.
Given the code from the question, this can be seen by running the following:
# print the list of cells
print(localClosure.__closure__)
# (<cell at 0x7f941ca27a00: int object at 0x7f941a621950>, <cell at 0x7f941ca27eb0: int object at 0x7f941a621970>)
# print the values in the cells
print(', '.join(str(cell.cell_contents) for cell in localClosure.__closure__))
# 2, 3
# print the value in the first cell (var_a)
print(localClosure.__closure__[0].cell_contents)
# 2
The cell_contents attribute of the cell objects first became writable with bpo-30486 which was first included in Python 3.7.0 alpha 1
Complete working example:
def foo():
var_a = 2
var_b = 3
def _closure(x):
return var_a + var_b + x
return _closure
localClosure = foo()
# Local closure is now "return 2 + 3 + x"
a = localClosure(1) # 2 + 3 + 1 == 6
# DO SOME MAGIC HERE TO TURN "var_a" of the closure into 0
# ...but what magic? Is this even possible?
# the magic
# this changes the value in the cell representing var_a to be 0
localClosure.__closure__[0].cell_contents = 0
# Local closure is now "return 0 + 3 + x"
b = localClosure(1) # 0 + 3 + 1 == 4
slightly different from what was asked, but you could do:
def f():
a = 1
b = 2
def g(x, a=a, b=b):
return a + b + x
return g
h = f()
print(h(0))
print(h(0,2,3))
print(h(0))
and make the closure the default, to be overridden when needed.
Maybe there's a further approach (even if it seems to be some years too late for my proposal :-)
def foo():
def _closure(x):
return _closure.var_a + _closure.var_b + x
_closure.var_a = 2
_closure.var_b = 3
return _closure
localClosure = foo()
# Local closure is now "return 2 + 3 + x"
a = localClosure(1) # 2 + 3 + 1 == 6
print(a)
# DO SOME MAGIC HERE TO TURN "var_a" of the closure into 0
# ...but what magic? Is this even possible?
localClosure.var_a = 0
# Local closure is now "return 0 + 3 + x"
b = localClosure(1) # 0 + 3 +1 == 4
print(b)
From my point of view the class solution proposed is easier to read. But if you try to modiy a free variable inside a decorator this solution might come in handy: In comparison to a class based solution it's easier to work with functools.wraps to preserve the meta data of the decorated function.
Why not make var_a and var_b arguments of the function foo?
def foo(var_a = 2, var_b = 3):
def _closure(x):
return var_a + var_b + x
return _closure
localClosure = foo() # uses default arguments 2, 3
print localClosure(1) # 2 + 3 + 1 = 6
localClosure = foo(0, 3)
print localClosure(1) # 0 + 3 + 1 = 4
def foo():
var_a = 2
var_b = 3
def _closure(x):
return var_a + var_b + x
return _closure
def bar():
var_a = [2]
var_b = [3]
def _closure(x):
return var_a[0] + var_b[0] + x
def _magic(y):
var_a[0] = y
return _closure, _magic
localClosureFoo = foo()
a = localClosureFoo(1)
print a
localClosureBar, localClosureBarMAGIC = bar()
b = localClosureBar(1)
print b
localClosureBarMAGIC(0)
b = localClosureBar(1)
print b