I have come across this example from Python hitchhikers guide:
def create_multipliers():
return [lambda x, i=i : i * x for i in range(5)]
The example above is the solution to some issues caused with late binding, where variables used in closures are looked up at the time the inner function is called.
What does the i=i mean and why is it making such difference?
It's actually not just for lambdas; any function that takes default parameters will use the same syntax. For example
def my_range(start, end, increment=1):
ans = []
while start < end:
ans.append(start)
start += increment
return ans
(This is not actually how range works, I just thought it would be a simple example to understand). In this case, you can call my_range(5,10) and you will get [5,6,7,8,9]. But you can also call my_range(5,10,increment=2), which will give you [5, 7, 9].
You can get some surprising results with default arguments. As this excellent post describes, the argument is bound at function definition, not at function invocation as you might expect. That causes some strange behavior, but it actually helps us here. Consider the incorrect code provided in your link:
def create_multipliers():
return [lambda x : i * x for i in range(5)]
for multiplier in create_multipliers():
print multiplier(2)
When you call multiplier(2), what is it actually doing? It's taking your input parameter, 2, and returning i * 2. But what is i? The function doesn't have any variable called i in its own scope, so it checks the surrounding scope. In the surrounding scope, the value of i is just whatever value you left it -- in this case 4. So every function gives you 8.
On the other hand, if you provide a default parameter, the function has a variable called i in its own scope. What's the value of i? Well, you didn't provide one, so it uses its default value, which was bound when the function was defined. And when the function was defined, i had a different value for each of the functions in your list!
It is a bit confusing that they've used the same name for the parameter variable as they did for the iterating variable. I suspect you could get the same result with greater readability with
def create_multipliers():
return [(lambda x, y=i: y*x) for i in range(5)]
In that case, each number in the range, will be assigned to the optional parameters of each lambda function:
def create_multipliers():
return [lambda x, i=i : i * x for i in range(5)]
lambda x, i=0
lambda x, i=1
lambda x, i=2
lambda x, i=3
lambda x, i=4
So, you can call the functions now with one parameter (because they already have the default)
for f in create_multipliers():
print(f(3))
0
3
6
9
12
Or you can call the function and give the parameter you want, that's why is optional
for f in create_multipliers():
print(f(3,2))
6
6
6
6
6
There are examples where optional parameter are needed, such as recursion
For example, square in terms of square:
square = lambda n, m=0: 0 if n==m else n+square(n,m+1)
Look that the optional parameter there is used as accumulator
Related
The answer in this post details nicely how python inner functions don't use the value of closure variables until the inner function actually executes, finding the variable name in the proper scope.
For example:
funcs = [(lambda: x) for x in range(3)]
Calling any of the generated lambdas returns 2:
>>> funcs[0]()
2
>>> funcs[1]()
2
>>> funcs[2]()
2
Is there a way to force the value for x to be determined when the function is defined instead of when it is executed later? In the above example, my desired output is 0, 1, 2, respectively.
More specifically, my use-case is to provide a way for API users to conveniently turn a custom function into a thread using a decorator. For example:
for idx in range(3):
#thread_this(name=f'thread_{idx}')
def custom():
do_something()
print(f'thread_{idx} complete.')
When the final print statement executes, it picks up whatever the current value of idx is in the global scope. With appropriate sleep statements, all 3 threads will print 'thread_2 complete.'
You can use functools.partial, first problem can be solved with,
funcs = [functools.partial(lambda x: x, x) for x in xrange(3)]
It will give you desired result.
However, I could not understand the second usecase.
I don't understand how this lambda function knows x is equal to 1?
def one(f = None): return 1 if not f else f(1)
def two(f = None): return 2 if not f else f(2)
def plus(y): return lambda x: x+y
one(plus(two()))
3
I know that the inner function two() returns 2 because f is defaulted to None. Thus y= 2. But how does the lambda function know to look to the outmost function for the x value?
plus returns a (lambda) function. That function is passed to one. Within the scope of one, it is called f.
Then f (which is actually the lambda from returned from plus) is called in f(1).
In other words, the code one(plus(two())) does this:
number2 = two()
lambda_function = plus(number2)
result = one(lambda_function)
If you look at one() it passes "1" into a function which you pass in the arguments (if a function is passed. otherwise 1 is returned). Thus, it evaluates to f(1) (see in the else of one). The function you pass to one() is lambda x: x + 2 (since y=2). Thus, this evaluates to lambda 1: 1 + 2
If you call one(lambda x: 50), it returns 50.
Let's run through the steps first:
one gets a value, f, as plus(two()).
plus gets a value as two, two is gonna be 2 since no f.
Okay, so since one gets the value of that, it gonna condition and see that it shouldn't return 1, so does f(1) the f is the unfinished lambda process, the lambda needs one parameter more to add up, so it got 1, so 2 + 1 is 3.
That is the whole process really.
I have two functions that return a list of functions. The functions take in a number x and add i to it. i is an integer increasing from 0-9.
def test_without_closure():
return [lambda x: x+i for i in range(10)]
def test_with_yield():
for i in range(10):
yield lambda x: x+i
I would expect test_without_closure to return a list of 10 functions that each add 9 to x since i's value is 9.
print sum(t(1) for t in test_without_closure()) # prints 100
I expected that test_with_yield would also have the same behavior, but it correctly creates the 10 functions.
print sum(t(1) for t in test_with_yield()) # print 55
My question is, does yielding form a closure in Python?
Yielding does not create a closure in Python, lambdas create a closure. The reason that you get all 9s in "test_without_closure" isn't that there's no closure. If there weren't, you wouldn't be able to access i at all. The problem is that all closures contain a reference¹ to the same i variable, which will be 9 at the end of the function.
This situation isn't much different in test_with_yield. Why, then, do you get different results? Because yield suspends the run of the function, so it's possible to use the yielded lambdas before the end of the function is reached, i.e. before i is 9. To see what this means, consider the following two examples of using test_with_yield:
[f(0) for f in test_with_yield()]
# Result: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[f(0) for f in list(test_with_yield())]
# Result: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
What's happening here is that the first example yields a lambda (while i is 0), calls it (i is still 0), then advances the function until another lambda is yielded (i is now 1), calls the lambda, and so on. The important thing is that each lambda is called before the control flow returns to test_with_yield (i.e. before the value of i changes).
In the second example, we first create a list. So the first lambda is yielded (i is 0) and put into the list, the second lambda is created (i is now 1) and put into the list ... until the last lambda is yielded (i is now 9) and put into the list. And then we start calling the lambdas. So since i is now 9, all lambdas return 9.
¹ The important bit here is that closures hold references to variables, not copies of the value they held when the closure was created. This way, if you assign to the variable inside a lambda (or inner function, which create closures the same way that lambdas do), this will also change the variable outside of the lambda and if you change the value outside, that change will be visible inside the lambda.
No, yielding has nothing to do with closures.
Here is how to recognize closures in Python: a closure is
a function
in which an unqualified name lookup is performed
no binding of the name exists in the function itself
but a binding of the name exists in the local scope of a function whose definition surrounds the definition of the function in which the name is looked up.
The reason for the difference in behaviour you observe is laziness, rather than anything to do with closures. Compare and contrast the following
def lazy():
return ( lambda x: x+i for i in range(10) )
def immediate():
return [ lambda x: x+i for i in range(10) ]
def also_lazy():
for i in range(10):
yield lambda x:x+i
not_lazy_any_more = list(also_lazy())
print( [ f(10) for f in lazy() ] ) # 10 -> 19
print( [ f(10) for f in immediate() ] ) # all 19
print( [ f(10) for f in also_lazy() ] ) # 10 -> 19
print( [ f(10) for f in not_lazy_any_more ] ) # all 19
Notice that the first and third examples give identical results, as do the second and the fourth. The first and third are lazy, the second and fourth are not.
Note that all four examples provide a bunch of closures over the most recent binding of i, it's just that in the first an third case you evaluate the closures before rebinding i (even before you've created the next closure in the sequence), while in the second and fourth case, you first wait until i has been rebound to 9 (after you've created and collected all the closures you are going to make), and only then evaluate the closures.
Adding to #sepp2k's answer you're seeing these two different behaviours because the lambda functions being created don't know from where they have to get i's value. At the time this function is created all it knows is that it has to either fetch i's value from either local scope, enclosed scope, global scope or builtins.
In this particular case it is a closure variable(enclosed scope). And its value is changing with each iteration.
Check out LEGB in Python.
Now to why second one works as expected but not the first one?
It's because each time you're yielding a lambda function the execution of the generator function stops at that moment and when you're invoking it and it will use the value of i at that moment. But in the first case we have already advanced i's value to 9 before we invoked any of the functions.
To prove it you can fetch current value of i from the __closure__'s cell contents:
>>> for func in test_with_yield():
print "Current value of i is {}".format(func.__closure__[0].cell_contents)
print func(9)
...
Current value of i is 0
Current value of i is 1
Current value of i is 2
Current value of i is 3
Current value of i is 4
Current value of i is 5
Current value of i is 6
...
But instead if you store the functions somewhere and call them later then you will see the same behaviour as the first time:
from itertools import islice
funcs = []
for func in islice(test_with_yield(), 4):
print "Current value of i is {}".format(func.__closure__[0].cell_contents)
funcs.append(func)
print '-' * 20
for func in funcs:
print "Now value of i is {}".format(func.__closure__[0].cell_contents)
Output:
Current value of i is 0
Current value of i is 1
Current value of i is 2
Current value of i is 3
--------------------
Now value of i is 3
Now value of i is 3
Now value of i is 3
Now value of i is 3
Example used by Patrick Haugh in comments also shows the same thing: sum(t(1) for t in list(test_with_yield()))
Correct way:
Assign i as a default value to lambda, default values are calculated when function is created and they won't change(unless it's a mutable object). i is now a local variable to the lambda functions.
>>> def test_without_closure():
return [lambda x, i=i: x+i for i in range(10)]
...
>>> sum(t(1) for t in test_without_closure())
55
Id like to define a list of functions by a list of parameters, to actually compose a potential from a finite number of monopoles and dipoles.
Functions in python list, however, for some reason do not behave as expected (by me).
The following example:
def function(coefficients):
phi = []
i = 0
for y in coefficients:
f = lambda x: y*(x**i)
i += 1
phi.append(f)
print f(2)
return phi
> phi = function([1,2,3])
2
8
24
> k0 = phi[0]
> k1 = phi[1]
> k2 = phi[2]
> print k0(2), k1(2), k2(2)
24 24 24
Always the last function in the list is retrieved independent of which function is picked from the list.
Any ideas?
Thanks in advance.
The lambda function holds a reference to the variable y, because of the closure property. So, all the functions have the reference to the same object and they will get the last value of y during the invocation. To fix this,
f = lambda x,y=y,i=i: y*(x**i)
Now, we have created another parameter y and that has the default value y. It makes the function depend on a local reference now. So, even if the value of y changes in the loop, the lambda function no longer refers to that y.
Note: The same explanation is applicable to the variable i as well.
To make things very clear, I would recommend using different variable names
f = lambda x, z=y, j=i: z * (x ** j)
Note: Your original output was 2 8 24. That was because, when you invoke the function, i got incremented and I don't think that is what you really wanted. The actual output should have been 1 4 12.
I'm trying to return from a function a list of functions, each of which uses variables from the outside scope. This isn't working. Here's an example which demonstrates what's happening:
a = []
for i in range(10):
a.append(lambda x: x+i)
a[1](1) # returns 10, where it seems it should return 2
Why is this happening, and how can I get around it in python 2.7 ?
The i refers to the same variable each time, so i is 9 in all of the lambdas because that's the value of i at the end of the loop. Simplest workaround involves a default argument:
lambda x, i=i: x+i
This binds the value of the loop's i to a local variable i at the lambda's definition time.
Another workaround is to define a lambda that defines another lambda, and call the first lambda:
(lambda i: lambda x: x+i)(i)
This behavior makes a little more sense if you consider this:
def outerfunc():
def innerfunc():
return x+i
a = []
for i in range(10):
a.append(innerfunc)
return a
Here, innerfunc is defined once, so it makes intuitive sense that you are only working with a single function object, and you would not expect the loop to create ten different closures. With a lambda it doesn't look like the function is defined only once, it looks like you're defining it fresh each time through the loop, but in fact it is functionally the same as the the long version.
Because i isn't getting evaluated when you define the anonymous function (lambda expression) but when it's called. You can see this by adding del i before a[1](1): you'll get NameError: global name 'i' is not defined on the a[1](1) line.
You need to fix the value of i into the lambda expression every time, like so:
a = [lambda x, i=i: x+i for i in range(10)]
a[1](1) # returns 2
Another, more general solution - also without lambdas:
import operator
from functools import partial
a = []
for i in range(10):
a.append(partial(operator.add, i))
a[1][(1) # returns 2
The key aspect here is functools.partial.