This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 8 months ago.
In python 2.6:
[x() for x in [lambda: m for m in [1,2,3]]]
results in:
[3, 3, 3]
I would expect the output to be [1, 2, 3]. I get the exact same problem even with a non list comprehension approach. And even after I copy m into a different variable.
What am I missing?
To make the lambdas remember the value of m, you could use an argument with a default value:
[x() for x in [lambda m=m: m for m in [1,2,3]]]
# [1, 2, 3]
This works because default values are set once, at definition time. Each lambda now uses its own default value of m instead of looking for m's value in an outer scope at lambda execution time.
The effect you’re encountering is called closures, when you define a function that references non-local variables, the function retains a reference to the variable, rather than getting its own copy. To illustrate, I’ll expand your code into an equivalent version without comprehensions or lambdas.
inner_list = []
for m in [1, 2, 3]:
def Lambda():
return m
inner_list.append(Lambda)
So, at this point, inner_list has three functions in it, and each function, when called, will return the value of m. But the salient point is that they all see the very same m, even though m is changing, they never look at it until called much later.
outer_list = []
for x in inner_list:
outer_list.append(x())
In particular, since the inner list is constructed completely before the outer list starts getting built, m has already reached its last value of 3, and all three functions see that same value.
Long story short, you don't want to do this. More specifically, what you're encountering is an order of operations problem. You're creating three separate lambda's that all return m, but none of them are called immediately. Then, when you get to the outer list comprehension and they're all called the residual value of m is 3, the last value of the inner list comprehension.
-- For comments --
>>> [lambda: m for m in range(3)]
[<function <lambda> at 0x021EA230>, <function <lambda> at 0x021EA1F0>, <function <lambda> at 0x021EA270>]
Those are three separate lambdas.
And, as further evidence:
>>> [id(m) for m in [lambda: m for m in range(3)]]
[35563248, 35563184, 35563312]
Again, three separate IDs.
Look at the __closure__ of the functions. All 3 point to the same cell object, which keeps a reference to m from the outer scope:
>>> print(*[x.__closure__[0] for x in [lambda: m for m in [1,2,3]]], sep='\n')
<cell at 0x00D17610: int object at 0x1E2139A8>
<cell at 0x00D17610: int object at 0x1E2139A8>
<cell at 0x00D17610: int object at 0x1E2139A8>
If you don't want your functions to take m as a keyword argument, as per unubtu's answer, you could instead use an additional lambda to evaluate m at each iteration:
>>> [x() for x in [(lambda x: lambda: x)(m) for m in [1,2,3]]]
[1, 2, 3]
Personally, I find this a more elegant solution. Lambda returns a function, so if we want to use the function, then we should use it. It's confusing to use the same symbol for the 'anonymous' variable in the lambda and for the generator, so in my example I use a different symbol to make it hopefully more clear.
>>> [ (lambda a:a)(i) for i in range(3)]
[0, 1, 2]
>>>
it's faster too.
>>> timeit.timeit('[(lambda a:a)(i) for i in range(10000)]',number=10000)
9.231263160705566
>>> timeit.timeit('[lambda a=i:a for i in range(10000)]',number=10000)
11.117988109588623
>>>
but not as fast as map:
>>> timeit.timeit('map(lambda a:a, range(10000))',number=10000)
5.746963977813721
(I ran these tests more than once, result was the same, this was done in python 2.7, results are different in python 3: the two list comprehensions are much closer in performance and both a lot slower, map remains much faster. )
#unubtu's answer is correct. I recreated the scenario in Groovy with closures. Perhaps is illustrates what is going on.
This is analogous to [x() for x in [lambda: m for m in [1,2,3]]]
arr = []
x = 0
while (x < 3) {
x++
arr.add({ -> x })
}
arr.collect { f -> f() } == [3, 3, 3]
This is analogous to [x() for x in [lambda m=m: m for m in [1,2,3]]]
arr = []
x = 0
while (x < 3) {
x++
arr.add({_x -> { -> _x }}(x))
}
arr.collect { f -> f() } == [1, 2, 3]
Note that this would not happen if i used [1,2,3].each {x -> ... } instead of a while loop. Groovy while loops and Python list comprehensions both share its closure between iterations.
I noticed that too. I concluded that lambda are created only once. So in fact your inner list comprehension will give 3 indentical functions all related to the last value of m.
Try it and check the id() of the elements.
[Note: this answer is not correct; see the comments]
Related
I understand that:
a=[lambda :k for k in [1,2,3]]
[e() for e in a]
returns [3, 3, 3] because it takes the value of k at runtime, and the last value of k is 3. However, I don't understand why, if I do:
k=50
[e() for e in a]
I still get [3, 3, 3]. Why this? I updated the value of k with 50, why do(es) the e() function(s) still read the old k value?
Although you just said it is clear, that
a=[lambda :k for k in [1,2,3]]
[e() for e in a]
returns [3,3,3], this needs a few words. And for sure this is not clear for everybody. The first list comprehensions creates three functions. They all take no arguments and they all return the variable k. Inside the list comprehension k is not a number. K is a variable. And your three functions return that variable. So you have three identical functions. Normally the k variable would be lost. It scopes only withing the list comprehension. You cannot access it any more outside the list comprehension. But in your special case k still lives. It is still captured in the three functions, because these return it. This is what is called a closure (of k). And as k was for iterating through the list [1,2,3] at the end it still is 3. So all three functions return 3. But that is not so obvious. Python could also set it back to zero again or a random number. There is no such statement that k remains at the last value. So even if your code returns [3, 3, 3], that is not obvious and that is not guaranteed.
As said k is not accessible any more outside the list comprehension. It is in the scope of the list comprehension only. Only with the three a-functions you have ever access to that k. And on the other hand, you cannot tell the a-funcions to use another k. The k from the list comprehension is in their closure. The k=50 has a completely different scope.
If you want the other k in the a-functions, you must define them this way:
a=[lambda k : k_compr for k_kompr in [1,2,3]]
[e() for e in a]
and pass the k to them. But I think you question is just informatory. Nobody would ever use such a list comrehension.
While trying to implement an algorithm, I couldn't get python lists to mutate via a function. After reading up on the issue I was suggested by this StackOverflow answer to use [:] in order to mutate the array passed in the function argumemt.
However, as seen in the following code snippet, the issue still persists when trying to mutate the list l. I am expecting the output to be Before: [1,2,3,4]
After: [69, 69, 69, 69], but instead I get back the original value of l as shown below.
def mutate_list(a, b):
c = [69] * 4
a[:] = c[:2] # changed the elements, but array's still unchanged outside function
b[:] = c[2:]
if __name__ == '__main__':
l = [1, 2, 3, 4]
print("Before: {}" .format(l))
mutate_list(l[:2], l[2:])
print("After: {}" .format(l))
Output:
Before: [1, 2, 3, 4]
After : [1, 2, 3, 4]
Any insights into why this is happening?
The error is that you not pass actually the l but two slices of it. You should change it, for example:
def mutate_list(a):
c = [69] * 4
a[:2] = c[:2]
a[2:] = c[2:]
if __name__ == '__main__':
l = [1, 2, 3, 4]
print("Before: {}" .format(l))
mutate_list(l)
print("After: {}" .format(l))
its all about the scope, mutable concept is applicable on list but not to reference variable.
The variables a,b are local variables, hence the scope of the variable will be always function scope.
The operations which you have performed :
a[:]=c[:2]
b[:]=c[2:]
Note: a and b both are list now so you will get following output in the function:
[69,69],[69,69]
but if you use + operator which is use for adding operations then the out out will be like:
[69,69,69,69]
Now whatever I told you that will be a local scope, if you want that the list should be mutable across the program then you have to specify the scope of the list as global inside function and on that variable you can do changes. in this case you also dont need to pass any arguments:
def mutate_list():
global l # telling python to use this global variable in a local function
c = [69] * 4
l=c # assigning new values to actual list i.e l
Now before output will be [1,2,3,4]
and after will be [69,69,69,69]
As pointed out by others, the issue arose from the fact that the function parameters were slices of the original array and as such, the parameters were being passed by value (instead of being passed by reference).
According to #Selcuk 's suggestion, the correct way of doing such an operation would be to pass the original array along with its indices to the function and then perform any slicing inside the function.
NOTE: This concept comes in handy for (recursive) divide-and-conquer algorithms where subarrays must be mutated and combined to form the solution.
This question already has answers here:
How do I pass a variable by reference?
(39 answers)
Closed 8 months ago.
I am not sure I understand the concept of Python's call by object style of passing function arguments (explained here http://effbot.org/zone/call-by-object.htm). There don't seem to be enough examples to clarify this concept well (or my google-fu is probably weak! :D)
I wrote this little contrived Python program to try to understand this concept
def foo( itnumber, ittuple, itlist, itdict ):
itnumber +=1
print id(itnumber) , itnumber
print id(ittuple) , ittuple
itlist.append(3.4)
print id(itlist) , itlist
itdict['mary'] = 2.3
print id(itdict), itdict
# Initialize a number, a tuple, a list and a dictionary
tnumber = 1
print id( tnumber ), tnumber
ttuple = (1, 2, 3)
print id( ttuple ) , ttuple
tlist = [1, 2, 3]
print id( tlist ) , tlist
tdict = tel = {'jack': 4098, 'sape': 4139}
print '-------'
# Invoke a function and test it
foo(tnumber, ttuple, tlist , tdict)
print '-------'
#Test behaviour after the function call is over
print id(tnumber) , tnumber
print id(ttuple) , ttuple
print id(tlist) , tlist
print id(tdict), tdict
The output of the program is
146739376 1
3075201660 (1, 2, 3)
3075103916 [1, 2, 3]
3075193004 {'sape': 4139, 'jack': 4098}
---------
146739364 2
3075201660 (1, 2, 3)
3075103916 [1, 2, 3, 3.4]
3075193004 {'sape': 4139, 'jack': 4098, 'mary': 2.3}
---------
146739376 1
3075201660 (1, 2, 3)
3075103916 [1, 2, 3, 3.4]
3075193004 {'sape': 4139, 'jack': 4098, 'mary': 2.3}
As you can see , except for the integer that was passed, the object id's (which as I understand refers to memeory location) remain unchanged.
So in the case of the integer, it was (effectively) passed by value and the other data structure were (effectively) passed by reference. I tried changing the list , the number and the dictionary to just test if the data-structures were changed in place. The number was not bu the list and the
dictionary were.
I use the word effectively above, since the 'call-by-object' style of argument passing seems to behave both ways depending on the data-structure passed in the above code
For more complicated data structures, (say numpy arrays etc), is there any quick rule of thumb to
recognize which arguments will be passed by reference and which ones passed by value?
The key difference is that in C-style language, a variable is a box in memory in which you put stuff. In Python, a variable is a name.
Python is neither call-by-reference nor call-by-value. It's something much more sensible! (In fact, I learned Python before I learned the more common languages, so call-by-value and call-by-reference seem very strange to me.)
In Python, there are things and there are names. Lists, integers, strings, and custom objects are all things. x, y, and z are names. Writing
x = []
means "construct a new thing [] and give it the name x". Writing
x = []
foo = lambda x: x.append(None)
foo(x)
means "construct a new thing [] with name x, construct a new function (which is another thing) with name foo, and call foo on the thing with name x". Now foo just appends None to whatever it received, so this reduces to "append None to the the empty list". Writing
x = 0
def foo(x):
x += 1
foo(x)
means "construct a new thing 0 with name x, construct a new function foo, and call foo on x". Inside foo, the assignment just says "rename x to 1 plus what it used to be", but that doesn't change the thing 0.
Others have already posted good answers. One more thing that I think will help:
x = expr
evaluates expr and binds x to the result. On the other hand:
x.operate()
does something to x and hence can change it (resulting in the same underlying object having a different value).
The funny cases come in with things like:
x += expr
which translate into either x = x + expr (rebinding) or x.__iadd__(expr) (modifying), sometimes in very peculiar ways:
>>> x = 1
>>> x += 2
>>> x
3
(so x was rebound, since integers are immutable)
>>> x = ([1], 2)
>>> x
([1], 2)
>>> x[0] += [3]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> x
([1, 3], 2)
Here x[0], which is itself mutable, was mutated in-place; but then Python also attempted to mutate x itself (as with x.__iadd__), which errored-out because tuples are immutable. But by then x[0] was already mutated!
Numbers, strings, and tuples in Python are immutable; using augmented assignment will rebind the name.
Your other types are merely mutated, and remain the same object.
This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 7 years ago.
I've found this example somewhere, and I cannot understand it.
>>> mult = [lambda x:x*i for i in range(4)]
>>> mult
[<function <listcomp>.<lambda> at 0xb6823464>, <function <listcomp>.<lambda> at 0xb682341c>, <function <listcomp>.<lambda> at 0xb68234f4>, <function <listcomp>.<lambda> at 0xb682353c>]
>>> [m(2) for m in mult]
[6, 6, 6, 6]
>>>
As I see, the first line creates a list of lambda functions which I can call afterwards. What I don't understand is the output of [6,6,6,6]. As I can see, the elements of mult are located in different parts of memory, so, these are not several references to the same data block. But why do they produce the same results? Do all these functions reference the same i (which is the last one, which is 3), and why is this happening? And is there a quick way to make such a list of lambdas that would produce [0,2,4,6] instead?
i is not bound in the body of the lambda; its value is looked up once the function is called, so you are creating a list of 4 identical functions. In Python 2, the index of a list comprehension "leaks", since a new scope is not created. When you call each m, each sees the same value of i, which is the last value i was assigned in the comprehension.
The solution is to force i to take a specific value in the body of each function. This can be done by using a "do-nothing" parameter, one whose only purpose is to use its default value.
mult = [lambda x, i=i:x*i for i in range(4)]
Using Python, am finding it difficult to get filter() to work with lambda for cases where more than 1 argument needs to be passed as is the case in the following snippet:
max_validation = lambda x,y,z: x < y < z
sequence1 = [1,4,8]
filter(max_validation, sequence1)
It raises the following error:
TypeError: <lambda>() takes exactly 3 arguments (1 given)
Please suggest as to what am doing wrong here.
It's a little bit difficult to figure out exactly what you're trying to do. I'm going to interpret your question, then provide an answer. If this is not correct, please modify your question or comment on this answer.
Question
I have sequences that are exactly three elements long. Here's one:
sequence1 = [1, 4, 8]
I want to ensure that the first element is less than the second element, which should in turn be less than the third element. I've written the following function to do so:
max_validation = lambda x, y, z: x < y < z
How do I apply this using filter? Using filter(max_validation, sequence1) doesn't work.
Answer
Filter applies your function to each element of the provided iterable, picking it if the function returns True and discarding it if the function returns False.
In your case, filter first looks at the value 1. It tries to pass that into your function. Your function expects three arguments, and only one is provided, so this fails.
You need to make two changes. First, put your three-element sequence into a list or other sequence.
sequences = [[1, 4, 8], [2, 3, 9], [3, 2, 3]]
max_validation = lambda x: x[0] < x[1] < x[2] and len(x) == 3
I've added two other sequences to test. Because sequences is a list of a list, each list gets passed to your test function. Even if you're testing just one sequence, you should use [[1, 4, 8]] so that the entire sequence to test gets passed into your function.
I've also modified max_validation so that it accepts just one argument: the list to test. I've also added and len(x) == 3 to ensure that the sequences are only 3 elements in length
The function passed to filter() only gets a single argument passed to it, which is the current element in the iterable being iterated.. If you need something fancier than that then filter() won't do.
Straight from the docs of Python Filters
Note that filter(function, iterable)
is equivalent to [item for item in
iterable if function(item)] if
function is not None and [item for
item in iterable if item] if function
is None.
So, you can just process single arguments with Python filters. That effectively means you cannot use filters for the your example. You would have to write custom-code for the same.
It's possible to do this using a closure:
>>> def foo(a,b):
... def bar(c):
... return a+b+c
... return bar
...
>>> x = foo(1,2)
>>> x(3)
6
>>> y = foo(100,0)
>>> y(1)
101
>>> x(1)
4
I hope you are aware of?
>>> max([1, 4, 8])
8
filter() takes a single argument. In your case, it will take 1. Then 4. Then 8.
This would work for sequences of any length:
all(x < y for x, y in zip(seq, seq[1:]))
What does there happens?
For sequence 1, 2, 3... you take sequences 1, 2, 3... and 2, 3, 4... and zip them together to sequence (1, 2), (2, 3), ...
Then you check if statement 'x < y' holds for every pair.
And this will work for any associative rule you want to check.
Useful links:
slices in Python
zip in Python docs
all in Python docs
I think all others didn't get the point. the error message is for lambda function not for the filter. You should rather call it this way:
filter(max_validation, *sequence1)
add a star on the list transform it into three arguments, then it will work.
I'm in agreement with both #walkingpendulum and #Wesley, depending on the interpretation of the actual problem statement. So parsing through the ambiguity in the problem statement:
If you're sequentially comparing one item to its previous value in an iterable object, a lambda expression is overkill, just
use a list comprehension:
[1 if i < i+1 else 0 for i in sequence1]
If you're comparing objects, then just compare them -- a lambda expression wouldn't work firstly because you're only passing one argument where the lambda expression you defined you're passing three and lambda is generally applied across an iterable object. To compare objects, there are simple constructs for that:
sequence1 == some_other_sequence
and
x, y, z = 1, 2, 3
x < y < z
And lastly, if you want to apply a lambda expression to an iterable object, map can get you there: (arbitrary lambda function)
map(lambda x: x > 1, sequence1)
Otherwise #walkingpendulum and #Wesley cover the other interpretations
You could change
max_validation = lambda x,y,z: x < y < z
to
max_validation = lambda (x,y,z): x < y < z