I'm trying to return from a function a list of functions, each of which uses variables from the outside scope. This isn't working. Here's an example which demonstrates what's happening:
a = []
for i in range(10):
a.append(lambda x: x+i)
a[1](1) # returns 10, where it seems it should return 2
Why is this happening, and how can I get around it in python 2.7 ?
The i refers to the same variable each time, so i is 9 in all of the lambdas because that's the value of i at the end of the loop. Simplest workaround involves a default argument:
lambda x, i=i: x+i
This binds the value of the loop's i to a local variable i at the lambda's definition time.
Another workaround is to define a lambda that defines another lambda, and call the first lambda:
(lambda i: lambda x: x+i)(i)
This behavior makes a little more sense if you consider this:
def outerfunc():
def innerfunc():
return x+i
a = []
for i in range(10):
a.append(innerfunc)
return a
Here, innerfunc is defined once, so it makes intuitive sense that you are only working with a single function object, and you would not expect the loop to create ten different closures. With a lambda it doesn't look like the function is defined only once, it looks like you're defining it fresh each time through the loop, but in fact it is functionally the same as the the long version.
Because i isn't getting evaluated when you define the anonymous function (lambda expression) but when it's called. You can see this by adding del i before a[1](1): you'll get NameError: global name 'i' is not defined on the a[1](1) line.
You need to fix the value of i into the lambda expression every time, like so:
a = [lambda x, i=i: x+i for i in range(10)]
a[1](1) # returns 2
Another, more general solution - also without lambdas:
import operator
from functools import partial
a = []
for i in range(10):
a.append(partial(operator.add, i))
a[1][(1) # returns 2
The key aspect here is functools.partial.
Related
This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Lambda in a loop [duplicate]
(4 answers)
Closed last year.
I am trying to generate a list of lambdas that I will later apply to an object, but when I try to do it via a comprehension or a loop over a list, the reference to the variable is kept, rather than the value itself. Let me illustrate.
Assume your object class is something like this:
class Object:
def function(self, x):
print(x)
So when you create the object and invoke it you get something like this:
o = Object()
o.function(0)
>>> 0
Now, if I manually construct my list of lambdas it would look like this:
lambdas = [
lambda x: x.function(0),
lambda x: x.function(1),
lambda x: x.function(2)
]
Which I can then apply to my previously created object:
for l in lambdas:
l(o)
>>> 0
>>> 1
>>> 2
However, when I generate the lambda list from another list, I only get the reference to the latest element of the list:
lambdas = [lambda x: x.function(i) for i in range(2)]
for l in lambdas:
l(o)
>>> 2
>>> 2
>>> 2
On closer inspection I can see that each lambda has a different memory address, so they are NOT references to the same function.
So I can only assume that the lambda is keeping a reference to i which has a final value of 2 and therefore when invoked, it takes the value.
So my question is if its possible to set the value of the variable inside the lambda before invocation?
Note: The usa case for a list of lambdas is to pass to the agg function of a Pandas groupby on a DataFrame. I am not looking for a solution to the pandas problem, but curious about the general solution.
Generator Option
Just change lambdas to a generator instead of a list, this will cause it redefine i on every call:
lambdas = (lambda x: x.function(i) for i in range(2))
for l in lambdas:
print(l(o))
Full code:
class Object:
def function(self, x):
print(x)
o = Object()
o.function(0) #manual call
lambdas = (lambda x: x.function(i) for i in range(2))
for l in lambdas:
l(o)
Output:
0 #output from manual call
0 #output from generator
1 #output from generator
List Option
If you need a list for things like lambdas[0](o) you can send i to lambda each iteration by using i=i like so:
lambdas = [lambda x, i=i: x.function(i) for i in range(2)]
Example of second option:
class Object:
def function(self, x):
print(x)
o = Object()
lambdas = [lambda x, i=i: x.function(i) for i in range(2)] #notice the cahnge
for i in range(len(lambdas)):
lambdas[i](o) #notice the change
Output:
0
1
What takes place is that in this expression, the "living" (nonlocal) i variable is
used inside each lambda created. And at the end of the for loop, its value is the last value taken - which will be used when the lambdas are actually called.
lambdas = [lambda x: x.function(i) for i in range(2)]
The fix for that is to create an intermediary namespace which will "freeze" the nonlocal variable value at the time the lambda is created. This is usually done with another lambda:
lambdas = [(lambda i: (lambda x: x.function(i)))(i) for i in range(2)]
So, bear with me - in the above expression, for each execution of the for i loop, a new, disposable lambda i is created and called imediatelly with the current value of the i used in the for. Inside it, this value is bound to a local i variable, that is unique to this disposable lambda i (in Python internal workings, it gets its own "cell"). This unique iis then used in the second, permanent, lambda x expression. Whenever that one is called, it will use the i value persisted in the outter lambda i call. The external lambda i then returns the lambda x expression as its result, but its nonlocal i is bound to the value used inside the lambda i, not the one used in the for i.
This is a common problem in Python, but can't be fixed because it is part of how the language works.
There is a shorter, and working, form to "freeze" the i from for i when each lambda i is created, that does not require an outer function scope: when a function is created, the values passed as default for its parameters are stored along with the function. Then, if one stores the current value of i as a default value, it won't change when the variable i itself does:
lambdas = [lambda x, i=i: x.function(i) for i in range(2)]
Here, in the lambda x, i=i: snippet, the value of i in the scope the lambda is created is stored as the default value for the parameter i, which works as a local (in contrast with a nonlocal) variable inside the lambda function itself.
This question already has answers here:
What do lambda function closures capture?
(7 answers)
Closed 6 months ago.
I was surprised that this assertion fails:
x = 42
x = lambda: x
assert x() == 42
It seems that x ends up recursively referring to itself, so that x(), x()(), etc. are all functions.
What is the rule used to parse this, and where is this documented?
By the way (not unexpectedly given the above), the original value of x has no references left after the lambda definition:
class X:
def __del__(self): print('deleting')
x = X()
x = lambda: x # 'deleting' is printed here
The variable x is created by the first assignment, and rebound with the second assignment.
Since the x in the lambda isn't evaluated until the lambda is called, calling it will evaluate to the most recently assigned value.
Note that this is not dynamic scoping - if it were dynamic, the following would print "99", but it prints "<function ...":
x = 42
x = lambda: x
def test(f):
x = 99
print(f())
test(x)
The first assignment is irrelevant; the x in the body of the lambda is bound late:
x = lambda: x # no need for a prior assignment
x = lambda: y # notice: no NameError occurs, *until it is called*
This is the same reason that creating lambdas in a loop is tricky, and is also used to make trees with the standard library defaultdict:
tree = lambda: defaultdict(tree)
t = tree()
t['foo']['bar']['baz'] = 'look ma, no intermediate steps'
A lambda is an anonymous function object. Python completely resolves whatever is on the right side of an equation to a single anonymous object and then resolves whatever is on the left side for assignment.
x = lambda: x
first compiles lambda: x into a function object that returns whatever happens to be in x at the time it is called. It then rebinds x with this function object, deleting whatever object happened to be there before.
Now x is a function that returns whatever is in x... which is a function that returns whatever is in x, etc... So you can write x()()()()()() as many times as you want, and still get that orginal lambda:x function object.
Python functions have a local namespace but only variables assigned in the function reside there. Since x isn't assigned in the lambda, it's resolved in the containing scope - that is, the module level "x". An identical piece of code is
def x():
return x
Contrast this with
def x():
x = 1
return x
Now, the parameter x is a local variable and is unrelated to the global x.
I am currently trying to create a lambda function that will pass a variable to both a function that is a input to a lambdified function and to the lambdified function itself. My Python version is 2.7, and my sympy version is 1.3.
I am able to have the lambdify function (f) work correctly when passed the correct argument (Y). I then attempt to create a lambda function that will pass a variable (z) to a function (controlFunc) which will both then be inputted into my lambdify function (f).
The problem seems to be that the lambda function will use the latest lambdify function each iteration (which it should) AND update all the PREVIOUS lambda functions to use the latest lambdify function as well. I believe this isn't an error in my code, but I could easily be wrong.
I have tried setting the lambdify function to a vasriable, and then making a lambda function from that. I have tried using the whole lambdify function in the lambda function. I even attempted to use list comprehension (I believe this is the right term) to evaluate each lambda of a lambdify in a list.
import sympy as sy
import numpy as np
r,s,t,v,w,x,y = sy.symbols('r,s,t,v,w,x,y')
variables = [[t,v,w,x,y]]
inputs = [[r,s]]
L = [[]]
controlledSim = True
ctrl_input = [[10., 10.]]
def controlFunc(x,controlDict):
return ctrl_input[0]
control = [controlFunc for i in range(10)]
controlDict = []
func = [sy.Matrix([[1.*r*s*t*v*w*x*y],
[2.*r*s*t*v*w*x*y],
[3.*r*s*t*v*w*x*y],
[4.*r*s*t*v*w*x*y],
[5.*r*s*t*v*w*x*y]])]
X = [1.,1.,1.,1.,1.]
Y = [1.,1.,1.,1.,1.,10.,10.]
for j in range(len(L)):
if controlledSim == True:
func[j] = list(func[j])
temp = [[] for i in range(len(func[j]))]
f = [[] for i in range(len(func[j]))]
for i in range(len(func[j])):
f[i] = sy.lambdify([np.append(variables[j],inputs[j])], func[j][i])
temp[i] = lambda z: f[i](np.append(z,control[i](z,controlDict)))
func_lambda = lambda z: np.array([lamb(z) for lamb in temp]).T
I know the output of func_lambda(X) should be an array of [100.,200.,300.,400.,500.].
My current results are an array of [500.,500.,500.,500.,500.].
It is pretty common problem with lambda functions inside loops. Lambda expressions are resolved at call-time (not during looping but after). Consider the following slight correction:
temp[i] = lambda z, i=i: f[i](np.append(z,control[i](z,controlDict)))
i=i is a default argument value so it is resolved at function definition. With this modification func_lambda(X) gives me [100. 200. 300. 400. 500.].
I use python 3. However this way should work in python 2 too. Try it.
The answer in this post details nicely how python inner functions don't use the value of closure variables until the inner function actually executes, finding the variable name in the proper scope.
For example:
funcs = [(lambda: x) for x in range(3)]
Calling any of the generated lambdas returns 2:
>>> funcs[0]()
2
>>> funcs[1]()
2
>>> funcs[2]()
2
Is there a way to force the value for x to be determined when the function is defined instead of when it is executed later? In the above example, my desired output is 0, 1, 2, respectively.
More specifically, my use-case is to provide a way for API users to conveniently turn a custom function into a thread using a decorator. For example:
for idx in range(3):
#thread_this(name=f'thread_{idx}')
def custom():
do_something()
print(f'thread_{idx} complete.')
When the final print statement executes, it picks up whatever the current value of idx is in the global scope. With appropriate sleep statements, all 3 threads will print 'thread_2 complete.'
You can use functools.partial, first problem can be solved with,
funcs = [functools.partial(lambda x: x, x) for x in xrange(3)]
It will give you desired result.
However, I could not understand the second usecase.
I have come across this example from Python hitchhikers guide:
def create_multipliers():
return [lambda x, i=i : i * x for i in range(5)]
The example above is the solution to some issues caused with late binding, where variables used in closures are looked up at the time the inner function is called.
What does the i=i mean and why is it making such difference?
It's actually not just for lambdas; any function that takes default parameters will use the same syntax. For example
def my_range(start, end, increment=1):
ans = []
while start < end:
ans.append(start)
start += increment
return ans
(This is not actually how range works, I just thought it would be a simple example to understand). In this case, you can call my_range(5,10) and you will get [5,6,7,8,9]. But you can also call my_range(5,10,increment=2), which will give you [5, 7, 9].
You can get some surprising results with default arguments. As this excellent post describes, the argument is bound at function definition, not at function invocation as you might expect. That causes some strange behavior, but it actually helps us here. Consider the incorrect code provided in your link:
def create_multipliers():
return [lambda x : i * x for i in range(5)]
for multiplier in create_multipliers():
print multiplier(2)
When you call multiplier(2), what is it actually doing? It's taking your input parameter, 2, and returning i * 2. But what is i? The function doesn't have any variable called i in its own scope, so it checks the surrounding scope. In the surrounding scope, the value of i is just whatever value you left it -- in this case 4. So every function gives you 8.
On the other hand, if you provide a default parameter, the function has a variable called i in its own scope. What's the value of i? Well, you didn't provide one, so it uses its default value, which was bound when the function was defined. And when the function was defined, i had a different value for each of the functions in your list!
It is a bit confusing that they've used the same name for the parameter variable as they did for the iterating variable. I suspect you could get the same result with greater readability with
def create_multipliers():
return [(lambda x, y=i: y*x) for i in range(5)]
In that case, each number in the range, will be assigned to the optional parameters of each lambda function:
def create_multipliers():
return [lambda x, i=i : i * x for i in range(5)]
lambda x, i=0
lambda x, i=1
lambda x, i=2
lambda x, i=3
lambda x, i=4
So, you can call the functions now with one parameter (because they already have the default)
for f in create_multipliers():
print(f(3))
0
3
6
9
12
Or you can call the function and give the parameter you want, that's why is optional
for f in create_multipliers():
print(f(3,2))
6
6
6
6
6
There are examples where optional parameter are needed, such as recursion
For example, square in terms of square:
square = lambda n, m=0: 0 if n==m else n+square(n,m+1)
Look that the optional parameter there is used as accumulator