I'm trying to build a counter in python with the property of closure. The code in the following works:
def generate_counter():
CNT = [0]
def add_one():
CNT[0] = CNT[0] + 1
return CNT[0]
return add_one
However when I change the list CNT to a var, it did not work:
def generate_counter1():
x = 0
def add_one():
x = x + 1
return x
return add_one
when I print the closure property of an instance, I found the __closure__ of the second case is none:
>>> ct1 = generate_counter()
>>> ct2 = generate_counter1()
>>> print(ct1.__closure__[0])
<cell at 0xb723765c: list object at 0xb724370c>
>>> print(ct2.__closure__)
None
Just wondering why the index in outer function has to be a list?
Thanks for the answers! Found the docs which clearly explained this problem
https://docs.python.org/3/faq/programming.html#why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value
Python determines the scope of a name by looking at name binding behaviour; assignment is one such behaviour (function parameters, importing, the target in for target ... or while .. as target are other examples). A name you bind to in a function is considered local. See the Naming and Binding section of the reference documentation.
So the name x in your second example is a local variable, because you assigned directly to it:
x = x + 1
In fact, because you never gave that x a local value, you'll get an exception when you try to use that function; the local name is unbound when you try to read it:
>>> generate_counter1()()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in add_one
UnboundLocalError: local variable 'x' referenced before assignment
In your first example no such binding takes place; you are instead altering the contents of CNT, what that name references is not altered.
If you are using Python 3, you can override the decision to make a name local, by using a nonlocal statement:
def generate_counter2():
x = 0
def add_one():
nonlocal x
x = x + 1
return x
return add_one
By making x non-local, Python finds it in the parent context and creates a closure for it again.
>>> def generate_counter2():
... x = 0
... def add_one():
... nonlocal x
... x = x + 1
... return x
... return add_one
...
>>> generate_counter2().__closure__
(<cell at 0x1078c62e8: int object at 0x1072c8070>,)
nonlocal is new in Python 3; in Python 2 you are limited to tricks like using a mutable list object to evade the binding rule. Another trick would be to assign the counter to an attribute of the nested function; again, this avoids binding a name in the current scope:
def generate_counter3():
def add_one():
add_one.x += 1
return add_one.x
add_one.x = 0
return add_one
It doesn't have to be a list, it just has to be an mutable object which you mutate, not reassign.
From the docs:
If a variable is assigned a value anywhere within the function’s body, it’s assumed to be a local unless explicitly declared as global.
Thus, in your second example, x is considered local (to the inner function), and, after your first assignment, you're shadowing the outer 'x'.
On the other hand, in the first example (since you don't assign a value to CNT) it operates on the CNT defined in the outer function.
Related
The last paragraph of https://docs.python.org/3/reference/executionmodel.html#resolution-of-names says
Class definition blocks and arguments to exec() and eval() are special in the context of name resolution. A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution with an exception that unbound local variables are looked up in the global namespace.
What does the last sentence of the quoted text mean? At first I inferred from it that the following code will print 1
a = 1
def foo():
a = 2
def bar():
class Bar:
b = a
print(Bar.b)
bar()
foo()
but I was wrong - a module consisting of the code above, when run, prints 2, i.e. the name a in the class definition, even though it's not bound in the class definition block and not bound in the local block just outside of it, isn't looked up in the global namespace, contrary to what the docs say.
I tried another code snippet described below (using a del statement, which is a construct that binds the variable in it)
a = 1
def foo():
a = 2
def bar():
class Bar:
del a
print(Bar.b)
bar()
foo()
but the del statement raises NameError: name 'a' is not defined.
So, I don't understand, what does that sentence mean?
According to the docs,
if a name is bound in a block, it is a local variable of that block, unless declared as nonlocal or global.
In your first code block, a is not bound to anything in your class Bar definition, so it is not a local variable of that block.
One way of binding a name is by using it on the left-hand side of an assignment statement. Here's an example.
a = 1
def foo():
a = 2
class Bar:
b = a
a = 3
print(Bar.b)
foo()
Result:
1
This demonstrates the principle of "unbound local variables are looked up in the global namespace" -- b = a uses the value of the global a rather than the value of the a local to foo.
In your second example, a is considered to be local to the class Bar block, because "A target occurring in a del statement is also considered bound" for the purpose of determining the scope of a name. But "unbound local variables are looked up in the global namespace" is not relevant, since del does not need to look up the value of the name in order to unbind it.
For good measure, we can experimentally confirm that a del statement signals to the interpreter that a name should be considered local.
a = 1
def foo():
a = 2
class Bar:
print(a)
del a
foo()
Result:
1
Traceback (most recent call last):
File "C:\Users\Kevin\Desktop\test.py", line 7, in <module>
foo()
File "C:\Users\Kevin\Desktop\test.py", line 4, in foo
class Bar:
File "C:\Users\Kevin\Desktop\test.py", line 6, in Bar
del a
NameError: name 'a' is not defined
Here we see that print(a) successfully looks up the value of the local variable a, and then on the next line it crashes because del can't delete an unbound local variable.
A local variable of a scope is any name defined anywhere in this scope. Notably, the variable itself is local, not the value assigned to it - a local variable can exist before or regardless of assignment.
>>> def foo(): # new function scope
... a = 3 # local variable `a`
... b: int # local variable `b`
... c = 3 # local variable `c`
... del c
... print(x)
... x = 3 # local variable `x`
... foo()
UnboundLocalError: local variable 'x' referenced before assignment
An unbound local variable is such a local variable with no value bound to it. In the above example, all of b, c and x are unbound at some point.
Neither of your examples access an unbound local variable. Both lookup the name a, but never assign to it.
In a function block, referencing unbound local variables is an error, namely UnboundLocalError. It does not matter whether that name exists in an enclosing scope as well.
>>> x = 1
>>> def foo():
... b = x # local variable is looked up locally
... x = 2 # make `x` a local variable
... foo()
UnboundLocalError: local variable 'x' referenced before assignment
In a class block, referencing unbound local variables falls back to a lookup in the global scope. This may or may not succeed.
>>> x = 1
>>> class Foo:
... b = x # local variable is looked up locally *or globally*
... x = 2 # make `x` a local variable
... print(Foo.b, Foo.x)
1 2
>>> class Foo:
... b = y # local variable is looked up locally *or globally*
... y = 2 # make `y` a local variable
... print(Foo.b, Foo.y)
NameError: name 'y' is not defined
I think I can make a supplement.
Python pre-computes which frame contains each name before executing the body of the function.
This means such phenomenon:
In [1]: a = 1
In [2]: def test():
...: print(a)
...:
In [3]: test()
1
In [4]: def test():
...: print(a)
...: a = 1
...:
In [5]: test()
---------------------------------------------------------------------------
UnboundLocalError Traceback (most recent call last)
<ipython-input-5-fbd55f77ab7c> in <module>
----> 1 test()
<ipython-input-4-a08051373573> in test()
1 def test():
----> 2 print(a)
3 a = 1
4
UnboundLocalError: local variable 'a' referenced before assignment
In [6]:
Error local variable 'a' referenced before assignment means Python pre-computes that function test frame has a local variable named a, we have to first assign one object to a and then reference it later.
My main() function is:
def main():
...
def _help():
...
1 a += somelist
2 a.append(something)
a=[]
_help()
What's weird is that line 2 works perfectly fine, but line 1 throws an UnboundLocalError: Local variable 'a' referenced before assignment.
Even when I declare a as a global variable at the top of either main or _help, it still doesn't work. Why is this?
Both of these lines are editing the same variable which makes me think either both or neither of them should work. How do I get line 1 to work?
Whenever you use <variable> = <something> in Python, Python automatically assumes it is a local variable, unless specifically told otherwise.
For example:
a = 1
def f():
if False:
a = 0
print(a) # UnboundLocalError
f()
In this case, += works as assignment as well, but .append does not assign to a, but calls a method.
This is fixed by placing a nonlocal a in your function, so it can assign to the a outside of its scope:
def main():
...
def _help():
nonlocal a
a += somelist # Works!
But in this case, you can just do a.extend(somelist).
I have this code:
def r():
i += 1
return i
def f():
return x*a
i = 0
a=2
x=3
print f()
print r()
I get this error for r(), but not for f():
~$ python ~/dev/python/inf1100/test.py
6
Traceback (most recent call last):
File "/home/marius/dev/python/inf1100/test.py", line 18, in <module>
print r()
File "/home/marius/dev/python/inf1100/test.py", line 2, in r
i += 1
UnboundLocalError: local variable 'i' referenced before assignment
Why can f() use variables defined outside of the function, whilst r() cannot?
That's because r reassigns the global variable i. f on the other hand just uses it. Remember that i += 1 is the same as i = i + 1.
Unless you explicitly tell it otherwise, Python treats all variables used within a function as being local. Furthermore, since there is no variable i defined within the local scope of r, it throws the error.
If you want to reassign a global variable within a function, you have to put:
global var
at the top of your function to explicitly declare var to be global.
So, to make r work, it should be rewritten to this:
def r():
global i
i += 1
return i
I'd like to point out that most of the time, this:
x = 1
def f():
global x
x += 1
f()
is bad practice, and you want to use parameters instead:
x = 1
def f(a_number):
return a_number + 1
x = f(x)
Also, here:
def r():
global i
i += 1
return i
return i is redundant, the variable is increased by the calling of the function.
Also this part of the Python FAQ is relevant and useful.
This piece:
def r():
i += 1
return i
not only uses global variables, but also tries to modify them (or more accurately: assign different value to global variable i).
To make it work, you can just declare this variable as global:
def r():
global i
i += 1
return i
In r you are shadowing your global i. Since it is not assigned before you attempt to add to it, you get an error.
A possible solution is to use global i in the r function like so
def r():
global i
i += 1
return i
If you assign into a variable, python will consider the variable local and won't bother looking for a global variable of the same name. Use global as suggested in other answers.
Why lambda function to get the list of values i = 4 .During the call lambda, enclosing scope does not exist. The function f has finished work and returned control (the variable i does not exist).
def f():
L = []
for i in range(5):
L.append(lambda x: i ** x)
return L
L = f()
L[0]
def f1(N):
def f2(X):
return X**N
return f2
f=f1(2)
f (3)
9
g = f1(3)
g(3)
27
f(3)
9
Python uses closures to capture references to the original variable. The lambda objects retain a reference to the i name, through which the value can be accessed. This means that the i variable continues to live on after f completes.
You can introspect this closure in the .__closure__ tuple on the lambda objects; functions have the same attribute:
>>> L[0].__closure__
(<cell at 0x1077f8b78: int object at 0x107465880>,)
>>> L[0].__closure__[0]
<cell at 0x1077f8b78: int object at 0x107465880>
>>> L[0].__closure__[0].cell_contents
4
This is also why all lambdas in your list L refer to the value 4, and not to the numbers 0 through to 4. They all refer to the same closure:
>>> L[0].__closure__[0] is L[1].__closure__[0]
True
The closure refers to the variable, not to the value of that variable at the time the closure was defined. At the end of the loop i was last set to 4, so when looking up i in the lambda closure 4 will be found, for all lambdas in your list.
If you want your lambdas to refer to the value of i during the loop, capture it in a keyword argument:
def f():
L = []
for i in range(5):
L.append(lambda x, i=i: i ** x)
return L
Now i is a local variable to the lambda, not a closure.
Alternatively, create an entirely new scope from which to draw the closure:
def create_lambda(i):
return lambda x: i ** x
def f():
return [create_lambda(i) for i in range(5)]
Now create_lambda() is a new scope with it's own local i for the lambda closure to refer to. The lambdas then each have their own closures:
>>> L[0].__closure__[0] is L[1].__closure__[0]
False
Closures refer to a variable in a specific namespace; each time you call a function a new local namespace is created, so each closure refers to i in create_lambda in a separate namespace from other calls to create_lambda.
I am reading this article about decorator.
At Step 8 , there is a function defined as:
def outer():
x = 1
def inner():
print x # 1
return inner
and if we run it by:
>>> foo = outer()
>>> foo.func_closure # doctest: +ELLIPSIS
it doesn't print x. According to the explanation :
Everything works according to Python’s scoping rules - x is a local
variable in our function outer. When inner prints x at point #1 Python
looks for a local variable to inner and not finding it looks in the
enclosing scope which is the function outer, finding it there.
But what about things from the point of view of variable lifetime? Our
variable x is local to the function outer which means it only exists
while the function outer is running. We aren’t able to call inner till
after the return of outer so according to our model of how Python
works, x shouldn’t exist anymore by the time we call inner and perhaps
a runtime error of some kind should occur.
However, I don't really understand what the second paragraph means.
I understand inner() does get the value of x but why it doesn't print x out?
thanks
UPDATE:
Thanks all for the answers. Now I understand the reason.
the "return inner" is just a pointer to inner() but it doesn't get executed, that is why inner() doesn't print x as it is not called at all
I understand inner() does get the value of x but why it doesn't print
x out?
It doesn't print out anything because you've not called the inner function yet.
>>> def outer():
x = 1
def inner():
print x # 1
return inner
...
>>> func = outer()
>>> func
<function inner at 0xb61e280c>
>>> func()
1
This is called a closure, i.e even though the outer function is not in stack(finished executing) anymore but still the inner function that was returned from it remembers it's state.(i.e value of x)
>>> def outer():
x = 1
y = 2
def inner():
z=3
print x
return inner
...
>>> func = outer()
>>> func.func_code.co_freevars #returns the variables that were used in closure
('x',)
From the source code on how python decides it's a closure or not:
459 if len(code.co_freevars) == 0:
460 closure = NULL
461 else:
462 len(closure) == len(code.co_freevars)
In py3.x you can also modify the value of x using nonlocal statement inside inner function.
>>> def outer():
x = 1
def inner():
nonlocal x
x += 1
print (x)
return inner
...
>>> func = outer()
>>> func()
2
>>> func()
3
>>> func()
4
You are not calling inner. You have called outer, which returns inner, but without calling it. If you want to call inner, do foo() (since you assinged the result of outer() to the name foo).
The paragraph you quoted is sort of tangential to this issue. You say you already understand why inner gets the value of x, which is what that paragraph is explaining. Basically, if a local variable is used in a nested function, and that nested function is returned, the value of the variable is stored along with the returned function, even if the scope where that variable was defined in no longer active. Normally x would be gone after outer finished, because x is just local to outer. But outer returns inner, which still needs access to x. So x gets wrapped up into what's called a closure, so it can still be accessed by inner later on.