This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 8 years ago.
I'm running Python 3.4.2, and I'm confused at the behavior of my code. I'm trying to create a list of callable polynomial functions with increasing degree:
bases = [lambda x: x**i for i in range(3)]
But for some reason it does this:
print([b(5) for b in bases])
# [25, 25, 25]
Why is bases seemingly a list of the last lambda expression, in the list comprehension, repeated?
The problem, which is a classic
"gotcha", is
that the i referenced in the lambda functions is not looked up until the
lambda function is called. At that time, the value of i is the last value it
was bound to when the for-loop ended, i.e. 2.
If you bind i to a default value in the definition of the lambda functions, then each i becomes a local variable, and its default value is evaluated and bound to the function at the time the lambda is defined rather than called.
Thus, when the lambda is called, i is now looked up in the local scope, and its default value is used:
In [177]: bases = [lambda x, i=i: x**i for i in range(3)]
In [178]: print([b(5) for b in bases])
[1, 5, 25]
For reference:
Python scopes and namespaces
As an alternate solution, you could use a partial function:
>>> bases = [(lambda i: lambda x: x**i)(i) for i in range(3)]
>>> print([b(5) for b in bases])
[1, 5, 25]
The only advantage of that construction over the classic solution given by #unutbu is that way, you cannot introduce sneaky bugs by calling your function with the wrong number of arguments:
>>> print([b(5, 8) for b in bases])
# ^^^
# oups
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <listcomp>
TypeError: <lambda>() takes 1 positional argument but 2 were given
As suggested by Adam Smith in a comment bellow, instead of using "nested lambda" you could use functools.partial with the same benefit:
>>> import functools
>>> bases = [functools.partial(lambda i,x: x**i,i) for i in range(3)]
>>> print([b(5) for b in bases])
[1, 5, 25]
>>> print([b(5, 8) for b in bases])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <listcomp>
TypeError: <lambda>() takes 2 positional arguments but 3 were given
a more 'pythonic' approach:
using nested functions:
def polyGen(degree):
def degPolynom(n):
return n**degree
return degPolynom
polynoms = [polyGen(i) for i in range(5)]
[pol(5) for pol in polynoms]
output:
>> [1, 5, 25, 125, 625]
I don't think the "why this happens" aspect of the question has been answered yet.
The reason that names non-local names in a function are not considered constants is so that these non-local names will match the behaviour of global names. That is, changes to a global name after a function is created are observed when the function is called.
eg.
# global context
n = 1
def f():
return n
n = 2
assert f() == 2
# non-local context
def f():
n = 1
def g():
return n
n = 2
assert g() == 2
return g
assert f()() == 2
You can see that in both the global and non-local contexts that if the value of a name is changed, then that change is reflected in future invocations of the function that references the name. If globals and non-locals were treated differently then that would be confusing. Thus, the behaviour is made consistent. If you need the current value of a name to made constant for a new function then the idiomatic way is to delegate the creation of the function to another function. The function is created in the creating-function's scope (where nothing changes), and thus the value of the name will not change.
eg.
def create_constant_getter(constant):
def constant_getter():
return constant
return constant_getter
getters = [create_constant_getter(n) for n in range(5)]
constants = [f() for f in getters]
assert constants == [0, 1, 2, 3, 4]
Finally, as an addendum, functions can modify non-local names (if the name is marked as such) just as they can modify global names. eg.
def f():
n = 0
def increment():
nonlocal n
n += 1
return n
return increment
g = f()
assert g() + 1 == g()
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 months ago.
Improve this question
In Python, I have an expensive function a(x) which is currently evaluated multiple times within another function. At the start of that function, I want to evaluate a(x) once, and reassign the variable name a locally to that value. But I keep getting an Unbound Local Error.
As a MWE:
def a(x):
return x+1
def main(x):
a = a(x)
return a**2
main(3)
#---------------------------------------------------------------------------
#UnboundLocalError Traceback (most recent call last)
#Input In [62], in <cell line: 8>()
# 5 a = a(x)
# 6 return a**2
#----> 8 main(3)
#
#Input In [62], in main(x)
# 4 def main(x):
#----> 5 a = a(x)
# 6 return a**2
#UnboundLocalError: local variable 'a' referenced before assignment
Obviously, a workaround is to use a line like b = a(x) instead.
But why is this happening? And how can I reassign the variable name a?
The error happens because in this line:
a = a(x)
you are redefining a to be a local variable. This occurs for all uses for a within that scope, including the right hand side of the expression. a doesn't start off as a global and then become a local at some point during the function's execution at runtime; if there is an assignment to a anywhere in the function, then a is now a local everywhere in that function.
That means that when that line is actually executed, it's attempting to call the local variable a before anything has been assigned to it.
Here's a simpler demonstration of the same effect, where we aren't changing the type of a, and we aren't referencing it on the same line where we're assigning to it:
>>> a = 42
>>> def foo():
... print(a) # should be 42, right?
... if False:
... a = 42 # should be a no-op, right?
...
>>> foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in foo
UnboundLocalError: local variable 'a' referenced before assignment
In this case, the rebinding of a happens after it's used, and it happens in a block that will never even actually be executed when the function is called -- but it doesn't matter, because at the time the function is defined, the existence of that assignment makes a a local variable.
You can actually see this by inspecting foo itself, without calling it:
>>> foo.__code__.co_varnames
('a',)
Compare with an implementation that doesn't make a into a local:
>>> def foo():
... print(a)
...
>>> foo()
42
>>> foo.__code__.co_varnames
()
The error is occurring because you have already named 'a' as a function that you can't assign as a variable again instead of
a = a(x)
you can name it some other variable like 'b' like this
def main(x):
b = a(x)
return b**2
Try declaring a as global variable:
def a(x):
return x+1
def main(x):
global a # <<<<<<<< only this line need to be added.
a = a(x) # Now a is first called (right side) and then reassigned
return a**2
print(main(3))
Now it works!
Output:
16
If you check what is a now:
print(a)
Output:
4
So global a has changed. Its no more a function and it cannot be called now:
print(a(3))
Output:
TypeError: 'int' object is not callable
alist = []
def show(*args, **kwargs):
alist.append(*args, **kwargs)
print(alist)
>>> show('tiger')
['tiger']
>>> show('tiger','cat')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in show
TypeError: append() takes exactly one argument (2 given)
>>> show('tiger','cat', {'name':'tom'})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in show
TypeError: append() takes exactly one argument (3 given)
Since the method append of alist only accepts one argument, why not detect a syntax error on the line alist.append(*args, **kwargs) in the definition of the method show?
It's not a syntax error because the syntax is perfectly fine and that function may or may not raise an error depending on how you call it.
The way you're calling it:
alist = []
def show(*args, **kwargs):
alist.append(*args, **kwargs)
print(alist)
>>> show('tiger')
['tiger']
>>> show('tiger','cat')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in show
TypeError: append() takes exactly one argument (2 given)
A different way:
alist = []
def show(*args, **kwargs):
alist.append(*args, **kwargs)
print(alist)
>>> show('tiger')
['tiger', 'tiger']
>>> class L: pass
...
>>> alist = L()
>>> alist.append = print
>>> show('tiger','cat')
tiger cat
<__main__.L object at 0x000000A45DBCC048>
Python objects are strongly typed. The names that bind to them are not. Nor are function arguments. Given Python's dynamic nature it would be extremely difficult to statically predict what type a variable at a given source location will be at execution time, so the general rule is that Python doesn't bother trying.
In your specific example, alist is not in the local scope. Therefore it can be modified after your function definition was executed and the changes will be visible to your function, cf. code snippets below.
So, in accord with the general rule: predicting whether or not alist will be a list when you call .append? Near-impossible. In particular, the interpreter cannot predict that this will be an error.
Here is some code just to drive home the point that static type checking is by all practical means impossible in Python. It uses non-local variables as in your example.
funcs = []
for a in [1, "x", [2]]:
def b():
def f():
print(a)
return f
funcs.append(b())
for f in funcs:
f()
Output:
[2] # value of a at definition time (of f): 1
[2] # value of a at definition time (of f): 'x'
[2] # value of a at definition time (of f): [2]
And similarly for non-global non-local variables:
funcs = []
for a in [1, "x", [2]]:
def b(a):
def f():
print(a)
a = a+a
return f
funcs.append(b(a))
for f in funcs:
f()
Output:
2 # value of a at definition time (of f): 1
xx # value of a at definition time (of f): 'x'
[2, 2] # value of a at definition time (of f): [2]
It's not a syntax error because it's resolved at runtime. Syntax errors are caught initially during parsing. Things like unmatched brackets, undefined variable names, missing arguments (this is not a missing argument *args means any number of arguments).
show has no way of knowing what you'll pass it at runtime and since you are expanding your args variable inside show, there could be any number of arguments coming in and it's valid syntax! list.append takes one argument! One tuple, one list, one int, string, custom class etc. etc. What you are passing it is some number elements depending on input. If you remove the * it's all dandy as its one element e.g. alist.append(args).
All this means that your show function is faulty. It is equipped to handle args only when its of length 1. If its 0 you also get a TypeError at the point append is called. If its more than that its broken, but you wont know until you run it with the bad input.
You could loop over the elements in args (and kwargs) and add them one by one.
alist = []
def show(*args, **kwargs):
for a in args:
alist.append(a)
for kv in kwargs.items():
alist.append(kv)
print(alist)
I'm trying to build a counter in python with the property of closure. The code in the following works:
def generate_counter():
CNT = [0]
def add_one():
CNT[0] = CNT[0] + 1
return CNT[0]
return add_one
However when I change the list CNT to a var, it did not work:
def generate_counter1():
x = 0
def add_one():
x = x + 1
return x
return add_one
when I print the closure property of an instance, I found the __closure__ of the second case is none:
>>> ct1 = generate_counter()
>>> ct2 = generate_counter1()
>>> print(ct1.__closure__[0])
<cell at 0xb723765c: list object at 0xb724370c>
>>> print(ct2.__closure__)
None
Just wondering why the index in outer function has to be a list?
Thanks for the answers! Found the docs which clearly explained this problem
https://docs.python.org/3/faq/programming.html#why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value
Python determines the scope of a name by looking at name binding behaviour; assignment is one such behaviour (function parameters, importing, the target in for target ... or while .. as target are other examples). A name you bind to in a function is considered local. See the Naming and Binding section of the reference documentation.
So the name x in your second example is a local variable, because you assigned directly to it:
x = x + 1
In fact, because you never gave that x a local value, you'll get an exception when you try to use that function; the local name is unbound when you try to read it:
>>> generate_counter1()()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in add_one
UnboundLocalError: local variable 'x' referenced before assignment
In your first example no such binding takes place; you are instead altering the contents of CNT, what that name references is not altered.
If you are using Python 3, you can override the decision to make a name local, by using a nonlocal statement:
def generate_counter2():
x = 0
def add_one():
nonlocal x
x = x + 1
return x
return add_one
By making x non-local, Python finds it in the parent context and creates a closure for it again.
>>> def generate_counter2():
... x = 0
... def add_one():
... nonlocal x
... x = x + 1
... return x
... return add_one
...
>>> generate_counter2().__closure__
(<cell at 0x1078c62e8: int object at 0x1072c8070>,)
nonlocal is new in Python 3; in Python 2 you are limited to tricks like using a mutable list object to evade the binding rule. Another trick would be to assign the counter to an attribute of the nested function; again, this avoids binding a name in the current scope:
def generate_counter3():
def add_one():
add_one.x += 1
return add_one.x
add_one.x = 0
return add_one
It doesn't have to be a list, it just has to be an mutable object which you mutate, not reassign.
From the docs:
If a variable is assigned a value anywhere within the function’s body, it’s assumed to be a local unless explicitly declared as global.
Thus, in your second example, x is considered local (to the inner function), and, after your first assignment, you're shadowing the outer 'x'.
On the other hand, in the first example (since you don't assign a value to CNT) it operates on the CNT defined in the outer function.
I saw this below piece of code in a tutorial and wondering how it works.
Generally, the lambda takes a input and returns something but here it does not take anything and still it works.
>>> for i in range(3):
... a.append(lambda:i)
...
>>> a
[<function <lambda> at 0x028930B0>, <function <lambda> at 0x02893030>, <function
<lambda> at 0x028930F0>]
lambda:i defines the constant function that returns i.
Try this:
>>> f = lambda:3
>>> f()
You get the value 3.
But there's something more going on. Try this:
>>> a = 4
>>> g = lambda:a
>>> g()
gives you 4. But after a = 5, g() returns 5. Python functions "remember" the environment in which they're executed. This environment is called a "closure". By modifying the data in the closure (e.g. the variable a in the second example) you can change the behavior of the functions defined in that closure.
In this case a is a list of function objects defined in the loop.
Each of which will return 2.
>>> a[0]()
2
To make these function objects remember i values sequentially you should rewrite the code to
>>> for i in range(3):
... a.append(lambda x=i:x)
...
that will give you
>>> a[0]()
0
>>> a[1]()
1
>>> a[2]()
2
but in this case you get side effect that allows you to not to use remembered value
>>> a[0](42)
42
I'm not sure what you mean by "it works". It appears that it doesn't work at all. In the case you have presented, i is a global variable. It changes every time the loop iterates, so after the loop, i == 2. Now, since each lambda function simply says lambda:i each function call will simply return the most recent value of i. For example:
>>> a = []
>>> for i in range(3):
a.append(lambda:1)
>>> print a[0]()
2
>>> print a[1]()
2
>>> print a[2]()
In other words, this does not likely do what you expect it to do.
lambda defines an anonymous inline function. These functions are limited compared to the full functions you can define with def - they can't do assignments, and they just return a result. However, you can run into interesting issues with them, as defining an ordinary function inside a loop is not common, but lambda functions are often put into loops. This can create closure issues.
The following:
>>> a = []
>>> for i in range(3):
... a.append(lambda:i)
adds three functions (which are first-class objects in Python) to a. These functions return the value of i. However, they use the definition of i as it existed at the end of the loop. Therefore, you can call any of these functions:
>>> a[0]()
2
>>> a[1]()
2
>>> a[2]()
2
and they will each return 2, the last iteration of the range object. If you want each to return a different number, use a default argument:
>>> for i in range(3):
... a.append(lambda i=i:i)
This will forcibly give each function an i as it was at that specific point during execution.
>>> a[0]()
0
>>> a[1]()
1
>>> a[2]()
2
Of course, since we're now able to pass an argument to that function, we can do this:
>>> b[0](5)
5
>>> b[0](range(3))
range(0, 3)
It all depends on what you're planning to do with it.
This is more a curiosity than anything, but I just noticed the following. If I am defining a self-referential lambda, I can do it easily:
>>> f = lambda: f
>>> f() is f
True
But if I am defining a self-referential list, I have to do it in more than one statement:
>>> a = [a]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> a = []
>>> a.append(a)
>>> a[0] is a
True
>>> a
[[...]]
I also noticed that this is not limited to lists but seems like any other expression other than a lambda can not reference the variable left of the assignment. For example, if you have a cyclic linked-list with one node, you can't simply go:
>>> class Node(object):
... def __init__(self, next_node):
... self.next = next_node
...
>>> n = Node(n)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'n' is not defined
Instead, you have to do it in two statements:
>>> n = Node(None)
>>> n.next = n
>>> n is n.next
True
Does anyone know what the philosophy behind this difference is? I understand that a recursive lambda are used much more frequently, and hence supporting self-reference is important for lambdas, but why not allow it for any assignment?
EDIT: The answers below clarify this quite nicely. The reason is that variables in lambdas in Python are evaluated each time the lambda is called, not when it's defined. In this sense they are exactly like functions defined using def. I wrote the following bit of code to experiment with how this works, both with lambdas and def functions in case it might help clarify it for anyone.
>>> f = lambda: f
>>> f() is f
True
>>> g = f
>>> f = "something else"
>>> g()
'something else'
>>> f = "hello"
>>> g()
'hello'
>>> f = g
>>> g() is f
True
>>> def f():
... print(f)
...
>>> f()
<function f at 0x10d125560>
>>> g = f
>>> g()
<function f at 0x10d125560>
>>> f = "test"
>>> g()
test
>>> f = "something else"
>>> g()
something else
The expression inside a lambda is evaluated when the function is called, not when it is defined.
In other words, Python will not evaluate the f inside your lambda until you call it. And, by then, f is already defined in the current scope (it is the lambda itself). Hence, no NameError is raised.
Note that this is not the case for a line like this:
a = [a]
When Python interprets this type of line (known as an assignment statement), it will evaluate the expression on the right of the = immediately. Moreover, a NameError will be raised for any name used on the right that is undefined in the current scope.
Because a lambda is a function, and the function body is not executed until the function is called.
In other words, the other way to do it is this:
def f():
return f
But you're correct that you can't do it in an expression because def is a statement, so it can't be used in an expression.
We can see when we disassemble the lambda function (this is identical output in Python 2.6 and 3.3)
>>> import dis
>>> f = lambda: f
>>> dis.dis(f)
1 0 LOAD_GLOBAL 0 (f)
3 RETURN_VALUE
We demonstrate that we do not need to load f until it is called, whereupon it is already defined globally, and therefore stored, so this works:
>>> f is f()
True
But when we do:
>>> a = [a]
We have an error (if a is previously undefined), and if we disassemble Python's implementation of this.
>>> def foo():
... a = [a]
...
>>> dis.dis(foo)
2 0 LOAD_FAST 0 (a)
3 BUILD_LIST 1
6 STORE_FAST 0 (a)
9 LOAD_CONST 0 (None)
12 RETURN_VALUE
we see that we attempt to load a before we have stored it.
There's no special-casing required to make this happen; it's just how it works.
A lambda expression is not any different from a normal function, really. Meaning, I can do this:
x = 1
def f():
print x + 2
f()
3
x = 2
f()
4
As you can see, inside the function, the value of x does not have a predefined value - it's looked up when we actually run f. This includes the value of the function itself: We don't look up what f represents until we actually run it, and by then it exists.
Doing that as a lambda doesn't work any differently:
del x
f = lambda: x+2
f()
NameError: global name 'x' is not defined
x = 2
f()
4
works similarly. In this case, I went ahead and deleted x so it was no longer in the scope when f was defined, and running f in this case correctly shows that x doesn't exist. But after we define x, then f works again.
This is different in the list case, because we are actually generating an object right now, and so everything on the right side has to be bound, right now. The way python works (as i understand it, and at least in practice this has been useful) is to consider that everything on the right side is deferenced & bound and then processed, and only after that's all complete are the value(s) on the left side bound and assigned.
Since the same value is on the right side and the left, when python tries to bind the name on the right side, it doesn't exist yet.