This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 7 years ago.
I was reading the python tutorial from Python Documentation Release 2.7.10 and I came across something like this.
Code
def fun1(a,L=[]):
L.append(a)
return L
print fun1(1)
print fun1(2)
print fun1(3)
def fun2(a,L = None):
if L is None:
L=[]
L.append(a)
return L
print fun2(1)
print fun2(2)
print fun2(3)
Output
[1]
[1, 2]
[1, 2, 3]
[1]
[2]
[3]
Process finished with exit code 0
If the L=[] in the first function fun1() is getting called only once , the output of fun1()is fine. But then why L=None is getting called every time in fun2().
L=[] in the function declaration makes Python essentially do this:
this function has a parameter named L
its default argument is [], let's set this particular [] aside and use it anytime there's no parameter passed for L
every time the function is called, create a variable L, and assign it either the passed parameter or the value we set aside earlier
So, the [] part is getting executed once, which creates a list object, which is set aside and kept around, which is why it accumulates changes if you modify it. Exactly the same thing happens with None, however None is not being modified nor is it mutable to begin with, so you're not seeing any weird side effects. None is still only being "executed" once, and that particular None value is set aside just as the list was, it's just that you're not doing anything to the None value itself.
When you define a function the values of the default arguments get evaluated, but the body of the function only get compiled. You can examine the result of the function definition via attributes. Theres a __defaults__ attribute containing the defaults and __code__ attribute containing the body (so these are created when the function is defined).
What's happening in the second example is that None do get evaluated at definition (it evaluates to None duh!), but the code that conditionally assigns [] to L only gets compiled and is run each time (the condition passes).
default arguments are evaluated only once. In fun1 you have the same list which you keep adding to. In fun2 you assign a new [] to the parameter and then append to it. As any assignment, its scope will be limited to the block it occurred in.
Related
I'm at the point in learning Python where I'm dealing with the Mutable Default Argument problem.
# BAD: if `a_list` is not passed in, the default will wrongly retain its contents between successive function calls
def bad_append(new_item, a_list=[]):
a_list.append(new_item)
return a_list
# GOOD: if `a_list` is not passed in, the default will always correctly be []
def good_append(new_item, a_list=None):
if a_list is None:
a_list = []
a_list.append(new_item)
return a_list
I understand that a_list is initialized only when the def statement is first encountered, and that's why subsequent calls of bad_append use the same list object.
What I don't understand is why good_append works any different. It looks like a_list would still be initialized only once; therefore, the if statement would only be true on the first invocation of the function, meaning a_list would only get reset to [] on the first invocation, meaning it would still accumulate all past new_item values and still be buggy.
Why isn't it? What concept am I missing? How does a_list get wiped clean every time good_append runs?
It looks like a_list would still be initialized only once
"initialization" is not something that happens to variables in Python, because variables in Python are just names. "initialization" only happens to objects, and it's done via the class' __init__ method.
When you write a = 0, that is an assignment. That is saying "a shall refer to the object that is described by the expression 0". It is not initialization; a can name anything else of any type at any later time, and that happens as a result of assigning something else to a. Assignment is just assignment. The first one is not special.
When you write def good_append(new_item, a_list=None), that is not "initializing" a_list. It is setting up an internal reference to an object, the result of evaluating None, so that when good_append is called without a second parameter, that object is automatically assigned to a_list.
meaning a_list would only get reset to [] on the first invocation
No, a_list gets set to [] any time that a_list is None to begin with. That is, when either None is passed explicitly, or the argument is omitted.
The problem with [] occurs because the expression [] is only evaluated once in this context. When the function is compiled, [] is evaluated, a specific list object is created - that happens to be empty to start - and that object is used as the default.
How does a_list get wiped clean every time good_append runs?
It doesn't. It doesn't need to be.
You know how the problem is described as being with "mutable default arguments"?
None is not mutable.
The problem occurs when you modify the object that the parameter has as a default.
a_list = [] does not modify whatever object a_list previously referred to. It cannot; arbitrary objects cannot magically transform in-place into empty lists. a_list = [] means "a_list shall stop referring to what it previously referred to, and start referring to []". The previously-referred-to object is unchanged.
When the function is compiled, and one of the arguments has a default value, that value - an object - gets baked into the function (which is also, itself, an object!). When you write code that mutates an object, the object mutates. If the object being referred to happens to be the object baked into the function, it still mutates.
But you cannot mutate None. It is immutable.
You can mutate []. It is a list, and lists are mutable. Appending an item to a list mutates the list.
The default value of a_list (or any other default value, for that matter) is stored in the function's interiors once it has been initialized and thus can be modified in any way:
>>> def f(x=[]): return x
...
>>> f.func_defaults
([],)
>>> f.func_defaults[0] is f()
True
resp. for Python 3:
>>> def f(x=[]): return x
...
>>> f.__defaults__
([],)
>>> f.__defaults__[0] is f()
True
So the value in func_defaults is the same which is as well known inside function (and returned in my example in order to access it from outside.
In other words, what happens when calling f() is an implicit x = f.func_defaults[0]. If that object is modified subsequently, you'll keep that modification.
In contrast, an assignment inside the function gets always a new []. Any modification will last until the last reference to that [] has gone; on the next function call, a new [] is created.
In order words again, it is not true that [] gets the same object on every execution, but it is (in the case of default argument) only executed once and then preserved.
The problem only exists if the default value is mutable, which None is not. What gets stored along with the function object is the default value. When the function is called, the function's context is initialized with the default value.
a_list = []
just assigns a new object to the name a_list in the context of the current function call. It does not modify None in any way.
No, in good_insert a_list is not initalised only once.
Each time the function is called without specifying the a_list argument, the default is used and a new instance of list is used and returned, the new list does not replace the default value.
The python tutorial says that
the default value is evaluated only once.
The evaluated (only once) default value is stored internally (name it x for simplicity).
case []:
When you define the function with a_list defaulted to [], if you don't provide a_list, it is assigned the internal variable x when . Therefore, when you append to a_list, you are actually appending to x (because a_list and x refer to the same variable now). When you call the function again without a_list, the updated x is re-assigned to a_list.
case None:
The value None is evaluated once and stored in x. If you don't provide, a_list, the variable x is assigned to a_list. But you don't append to x of course. You reassign an empty array to a_list. At this point x and a_list are different variables. The same way when you call the function again without a_list, it first gets the value None from x but then a_list gets assigned to an empty array again.
Note that, for the a_list = [] case, if you provide an explicit value for a_list when you call the function, the new argument does not override x because that's evaluated only once.
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 2 years ago.
I am new guy. Here is an example from python document.
def funtest(num, L=[]):
L.append(num)
return L
print(funtest(2))
print(funtest(3))
print(funtest(4))
I thought very function call should execute the same thing, which means L=[] in very function call. so I though the output should like this:
[2]
[3]
[4]
but the real answer is in below. It looks like the funtest() inherited the output from the last function call. And then continually pass it to next function call. I am confusing to it.
[2]
[2, 3]
[2, 3, 4]
When I change the code to below case:
def funtest(L=[]):
L.append(L)
return L
print(funtest([2]))
print(funtest([3]))
print(funtest([4]))
the real output go to this:
[2, [...]]
[3, [...]]
[4, [...]]
The output are quite different with the previous one. I am more confused now. Does anyone repair my mind shot? Thanks a lot
to explain what is happening ,
1st case
so the default values in python functions are instantiated only once , only while defining the function , so the list is only created while defining the function and you keep adding to it so now the result contains all the older values
this is true for all values which are mutable when used as default value for arguments in python
what can you do ?
in general use immutable values , and for the case of list you use None as the default value and in the function you check if list is None and create one empty list inside the function
read more here ( the first gotcha ) : https://docs.python-guide.org/writing/gotchas/
2nd case
now you are actually creating infinite list by appending same list to itself when you are doing L.append(L)
and whenever a list is too large to print , python shows those three dots in its place
read more here What does [...] (an ellipsis) in a list mean in Python?
The problem is that the default value of L is a mutable object (in your case an empty list).
In Python default values are evaluated at function definition time.
Every time you call the function, you get the same default value and if you add any content to the object (the list) you'll get all the previously added content.
The correct way to get a default list (or dictionary, or set) is to create it at run time inside the function.
This is a common mistake in Python with beginners devs.
You would like to read similar questions:
"Least Astonishment" and the Mutable Default Argument
Good uses for mutable function argument default values?
move L to inside of function
def funtest(num):
L=[]
L.append(num)
return L
print(funtest(2))
print(funtest(3))
print(funtest(4))
this re set the L
This question already has answers here:
What is the scope of a defaulted parameter in Python?
(7 answers)
Closed 1 year ago.
I understand that one should not use mutable default parameter value in Python (with some exceptions) because this value is evaluated and stored only once when the function is defined, and not each time the function is later called.
My understanding of that is this (using the example below; please excuse my imprecise language as I'm just a beginner in Python programming, who's stuck at the Function chapter of my textbook because of this):
def f(x = [1, 2, 3]):
x.append(4)
print(x)
f()
f()
1) The function f is defined, and x (local variable in f) assumes the default variable of [1, 2, 3] (even before the function is called)
2) When f() is called, x is still [1, 2, 3] due to no argument passed to it, and x continues having its default value
3) x is modified in place with append, becomes [1, 2, 3, 4], and is printed as such
However, this is where my confusion arises. I'd assume that:
4) When f ends, x is destroyed (in the stack or whatever you'd call it) and is no longer associated with the list object [1, 2, 3, 4]**
5) The list object [1, 2, 3, 4] is reclaimed since there's no variable that refers to it anymore
Therefore,
6) When f() is called the second time, I'd expect Python to output an error since x now no longer has a value associated with it. In other words, how can Python reuse the default value from the last evaluation when it's been reclaimed/destroyed?
Appreciate all your help and explanation!
** this understanding I got from Ned Batchelder's page on variable name assignment (see below)
While it may seems to you that at the end of the execution x, the default value, is disposed, it is not.
In fact, Python has a global namespace with all the names available for you to use (built-in functions, classes and functions you import or define).
The content of this namespace is made of objects. Function are objects too.
As a test, if you try this in a script or in the python command line, you will see what I mean:
def f(x = [1, 2, 3]):
x.append(4)
print(x)
print dir(f)
you will see the object nature of the function f. As an objects, the default values are referenced in an attribute, f.func_defaults, therefore they are always available and if mutable they retain the changes, giving you side effects with you may not want.
EDIT: in python 3 the attribute has been replaced by f.__defaults__
There are two references to the list in your case, one is store in the background of the function as the default value to the argument x.
When the function is called without x, a new reference to the same list is created as the local variable x. Then you append to the list via the second reference. And after the call, the second reference is garbage collected. The first reference still points to the same list, which has one element more now.
Or in short: there is only one list all the time.
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 8 years ago.
Hi I am new to python. Can someone explain how the below two pieces of code give different outputs? Does the function gets defined each time it is called?
def f(a, L=[]):
L.append(a)
return L
def f(a, L=None):
if L is None:
L = []
L.append(a)
return L
on running
print f(1)
print f(2)
print f(3)
I get these outputs respectively
[1]
[1, 2]
[1, 2, 3]
[1]
[2]
[3]
This is a very common 'gotcha' for new python developers. In the first example it looks like a new empty list should be created each time the function is called without the second parameter. It isn't. A single list is created when the function object is created, which is basically when the python script is loaded or you finish entering the function in the interactive shell. This list is then used for every invocation of the function, hence the acculumation you see.
The second is the standard way of working round this, and creates a new list instance each time the function is called without the second parameter.
Under the covers Python is putting any default values it finds in the function definition into a property of the function called defaults. You can see how the same instance is present between calls in the interactive shell:
>>> def f(a,b=[]):
... b.append(a)
>>> f.__defaults__
([],)
>>> f(1)
>>> f.__defaults__
([1],)
>>> f(2)
>>> f.__defaults__
([1,2],)
Default arguments in python are evaluated at the function declaration point - when it's first seen by the interpreter.
In your first snippet, the L argument is "given" a list as the default, and so the values are appended.
In the second snippet, L is always None on entry into the function, so gets re-created each time.
The function default parameters are evaluated when the function is defined.
In your case def f(a, L=[]):creates an empty list that is then passed every time the function is called without the L parameter. So every call to the function operates on the same list, that's why it gives different output each time.
In your second function an empty list is created every time the function is called, so it works as expected.
This question already has answers here:
"Least Astonishment" and the Mutable Default Argument
(33 answers)
Closed 8 years ago.
Refer to this
>>> def foo(counter=[0]):
... counter[0] += 1
... print("Counter is %i." % counter[0]);
...
>>> foo()
Counter is 1.
>>> foo()
Counter is 2.
>>>
Default values are initialized only when the function is first evaluated, not each time it is executed, so you can use a list or any other mutable object to maintain static values.
Question> Why the counter can keep its updated value during different callings? Is it true that counter refers to the same memory used to store the temporary list of the default parameter so that it can refer to the same memory address and keep the updated values during the calls?
The object created as the default argument becomes part of the function and persists until the function is destroyed.
First things first: if you are coding in Python: forget about "memory address" - you will never need one. Yes, there are objects, and they are placed in memory, and if you are referring to the same object, it is in the same "memory address" - but that does not matter - there could even be an implementation where objects don't have a memory address at all (just a place in a data structure, for example).
Then, when Python encounters the function body, as it is defined above, it does create a code object with the contents of the function body, and executes the function definition line - resolving any expressions inlined there and setting the results of those expressions as the default parameters for that function. There is nothing "temporary" about these objects. The expressions (in this case [0]) are evaluated, the resulting objects (in this case a Python list with a single element) are created, and assigned to a reference in the function object (a position in the functions's "func_defaults" attribute - remember that functions themselves are objects in Python.)
Whenever you call that function, if you don't pass a value to the counter parameter, it is assigned to the object recorded in the func_defaults attribute -in this case, a Python list. And it is the same Python list that was created at function parsing time.
What happens is that Python lists themselves are mutable: one can change its contents, add more elements, etc...but they are still the same list.
What the code above does is exactly incrementing the element in the position 0 of the list.
You can access this list at any time in the example above by typing foo.func_defaults[0]
If you want to "reset" the counter, you can just do: foo.func_defaults[0][0]=0, for example.
Of course, it is a side effect of how thigns a reimplemented in Python, and though consistent, and even docuemnted, should not be used in "real code". If you need "static variables", use a class instead of a function like the above:
class Foo(object):
def __init__(self):
self.counter = 0
def __call__(self):
self.counter += 1
print("Counter is %i." % self.counter)
And on the console:
>>> foo = Foo()
>>> foo()
Counter is 1.
>>> foo()
Counter is 2.
>>>
Default arguments are evaluated at the function definition, and not its calls. When the foo function object is created (remember, functions are first class citizens in python) its arguments, including default arguments, are all local names bound to the function object. The function object is created when the def: statement is first encountered.
In this case counter is set to a mutable list in the definition of foo(), so calling it without an argument gives the original mutable list instantiated at definition the name counter. This makes calls to foo() use and modify the original list.
Like all function arguments, counter is a local name defined when the function is declared. The default argument, too, is evaluated once when the function is declared.
If, when calling foo(), no value is passed for counter, that default value, the exact object instance provided at function definition time, is given the name counter. Since this is a mutable object (in this case, a list), any changes made to it remain after the function has completed.
The function contains a reference to the list in its default argument tuple, func_defaults, which prevents it from being destroyed.