This question already has answers here:
How do I pass a variable by reference?
(39 answers)
Closed 8 months ago.
In some languages you can pass a parameter by reference or value by using a special reserved word like ref or val. When you pass a parameter to a Python function it never alters the value of the parameter on leaving the function.The only way to do this is by using the global reserved word (or as i understand it currently).
Example 1:
k = 2
def foo (n):
n = n * n #clarity regarding comment below
square = n
return square
j = foo(k)
print j
print k
would show
>>4
>>2
showing k to be unchanged.
In this example the variable n is never changed
Example 2:
n = 0
def foo():
global n
n = n * n
return n
In this example the variable n is changed.
Is there any way in Python to call a function and tell Python that the parameter is either a value or reference parameter instead of using global?
There are essentially three kinds of 'function calls':
Pass by value
Pass by reference
Pass by object reference
Python is a PASS-BY-OBJECT-REFERENCE programming language.
Firstly, it is important to understand that a variable, and the value of the variable (the object) are two seperate things. The variable 'points to' the object. The variable is not the object. Again:
THE VARIABLE IS NOT THE OBJECT
Example: in the following line of code:
>>> x = []
[] is the empty list, x is a variable that points to the empty list, but x itself is not the empty list.
Consider the variable (x, in the above case) as a box, and 'the value' of the variable ([]) as the object inside the box.
PASS BY OBJECT REFERENCE (Case in python):
Here, "Object references are passed by value."
def append_one(li):
li.append(1)
x = [0]
append_one(x)
print x
Here, the statement x = [0] makes a variable x (box) that points towards the object [0].
On the function being called, a new box li is created. The contents of li are the SAME as the contents of the box x. Both the boxes contain the same object. That is, both the variables point to the same object in memory. Hence, any change to the object pointed at by li will also be reflected by the object pointed at by x.
In conclusion, the output of the above program will be:
[0, 1]
Note:
If the variable li is reassigned in the function, then li will point to a separate object in memory. x however, will continue pointing to the same object in memory it was pointing to earlier.
Example:
def append_one(li):
li = [0, 1]
x = [0]
append_one(x)
print x
The output of the program will be:
[0]
PASS BY REFERENCE:
The box from the calling function is passed on to the called function. Implicitly, the contents of the box (the value of the variable) is passed on to the called function. Hence, any change to the contents of the box in the called function will be reflected in the calling function.
PASS BY VALUE:
A new box is created in the called function, and copies of contents of the box from the calling function is stored into the new boxes.
You can not change an immutable object, like str or tuple, inside a function in Python, but you can do things like:
def foo(y):
y[0] = y[0]**2
x = [5]
foo(x)
print x[0] # prints 25
That is a weird way to go about it, however, unless you need to always square certain elements in an array.
Note that in Python, you can also return more than one value, making some of the use cases for pass by reference less important:
def foo(x, y):
return x**2, y**2
a = 2
b = 3
a, b = foo(a, b) # a == 4; b == 9
When you return values like that, they are being returned as a Tuple which is in turn unpacked.
edit:
Another way to think about this is that, while you can't explicitly pass variables by reference in Python, you can modify the properties of objects that were passed in. In my example (and others) you can modify members of the list that was passed in. You would not, however, be able to reassign the passed in variable entirely. For instance, see the following two pieces of code look like they might do something similar, but end up with different results:
def clear_a(x):
x = []
def clear_b(x):
while x: x.pop()
z = [1,2,3]
clear_a(z) # z will not be changed
clear_b(z) # z will be emptied
OK, I'll take a stab at this. Python passes by object reference, which is different from what you'd normally think of as "by reference" or "by value". Take this example:
def foo(x):
print x
bar = 'some value'
foo(bar)
So you're creating a string object with value 'some value' and "binding" it to a variable named bar. In C, that would be similar to bar being a pointer to 'some value'.
When you call foo(bar), you're not passing in bar itself. You're passing in bar's value: a pointer to 'some value'. At that point, there are two "pointers" to the same string object.
Now compare that to:
def foo(x):
x = 'another value'
print x
bar = 'some value'
foo(bar)
Here's where the difference lies. In the line:
x = 'another value'
you're not actually altering the contents of x. In fact, that's not even possible. Instead, you're creating a new string object with value 'another value'. That assignment operator? It isn't saying "overwrite the thing x is pointing at with the new value". It's saying "update x to point at the new object instead". After that line, there are two string objects: 'some value' (with bar pointing at it) and 'another value' (with x pointing at it).
This isn't clumsy. When you understand how it works, it's a beautifully elegant, efficient system.
Hope the following description sums it up well:
There are two things to consider here - variables and objects.
If you are passing a variable, then it's pass by value, which means the changes made to the variable within the function are local to that function and hence won't be reflected globally. This is more of a 'C' like behavior.
Example:
def changeval( myvar ):
myvar = 20;
print "values inside the function: ", myvar
return
myvar = 10;
changeval( myvar );
print "values outside the function: ", myvar
O/P:
values inside the function: 20
values outside the function: 10
If you are passing the variables packed inside a mutable object, like a list, then the changes made to the object are reflected globally as long as the object is not re-assigned.
Example:
def changelist( mylist ):
mylist2=['a'];
mylist.append(mylist2);
print "values inside the function: ", mylist
return
mylist = [1,2,3];
changelist( mylist );
print "values outside the function: ", mylist
O/P:
values inside the function: [1, 2, 3, ['a']]
values outside the function: [1, 2, 3, ['a']]
Now consider the case where the object is re-assigned. In this case, the object refers to a new memory location which is local to the function in which this happens and hence not reflected globally.
Example:
def changelist( mylist ):
mylist=['a'];
print "values inside the function: ", mylist
return
mylist = [1,2,3];
changelist( mylist );
print "values outside the function: ", mylist
O/P:
values inside the function: ['a']
values outside the function: [1, 2, 3]
Python is neither pass-by-value nor pass-by-reference. It's more of "object references are passed by value" as described here:
Here's why it's not pass-by-value. Because
def append(list):
list.append(1)
list = [0]
reassign(list)
append(list)
returns [0,1] showing that some kind of reference was clearly passed as pass-by-value does not allow a function to alter the parent scope at all.
Looks like pass-by-reference then, hu? Nope.
Here's why it's not pass-by-reference. Because
def reassign(list):
list = [0, 1]
list = [0]
reassign(list)
print list
returns [0] showing that the original reference was destroyed when list was reassigned. pass-by-reference would have returned [0,1].
For more information look here:
If you want your function to not manipulate outside scope, you need to make a copy of the input parameters that creates a new object.
from copy import copy
def append(list):
list2 = copy(list)
list2.append(1)
print list2
list = [0]
append(list)
print list
Technically python do not pass arguments by value: all by reference. But ... since python has two types of objects: immutable and mutable, here is what happens:
Immutable arguments are effectively passed by value: string, integer, tuple are all immutable object types. While they are technically "passed by reference" (like all parameters), since you can't change them in-place inside the function it looks/behaves as if it is passed by value.
Mutable arguments are effectively passed by reference: lists or dictionaries are passed by its pointers. Any in-place change inside the function like (append or del) will affect the original object.
This is how Python is designed: no copies and all are passed by reference. You can explicitly pass a copy.
def sort(array):
# do sort
return array
data = [1, 2, 3]
sort(data[:]) # here you passed a copy
Last point I would like to mention which is a function has its own scope.
def do_any_stuff_to_these_objects(a, b):
a = a * 2
del b['last_name']
number = 1 # immutable
hashmap = {'first_name' : 'john', 'last_name': 'legend'} # mutable
do_any_stuff_to_these_objects(number, hashmap)
print(number) # 1 , oh it should be 2 ! no a is changed inisde the function scope
print(hashmap) # {'first_name': 'john'}
So this is a little bit of a subtle point, because while Python only passes variables by value, every variable in Python is a reference. If you want to be able to change your values with a function call, what you need is a mutable object. For example:
l = [0]
def set_3(x):
x[0] = 3
set_3(l)
print(l[0])
In the above code, the function modifies the contents of a List object (which is mutable), and so the output is 3 instead of 0.
I write this answer only to illustrate what 'by value' means in Python. The above code is bad style, and if you really want to mutate your values you should write a class and call methods within that class, as MPX suggests.
Consider that the variable is a box and the value it points to is the "thing" inside the box:
1. Pass by reference : function shares the same box and thereby the thing inside also.
2. Pass by value : function creates a new box, a replica of the old one, including a copy of whatever thing is inside it. Eg. Java - functions create a copy of the box and the thing inside it which can be: a primitive / a reference to an object. (note that the copied reference in the new box and the original both still point to the same object, here the reference IS the thing inside the box, not the object it is pointing to)
3. Pass by object-reference: the function creates a box, but it encloses the same thing the initial box was enclosing. So in Python:
a) if the thing inside said box is mutable, changes made will reflect back in the original box (eg. lists)
b) if the thing is immutable (like python strings and numeric types), then the box inside the function will hold the same thing UNTIL you try to change its value. Once changed, the thing in the function's box is a totally new thing compared to the original one. Hence id() for that box will now give the identity of the new thing it encloses.
The answer given is
def set_4(x):
y = []
for i in x:
y.append(i)
y[0] = 4
return y
and
l = [0]
def set_3(x):
x[0] = 3
set_3(l)
print(l[0])
which is the best answer so far as it does what it says in the question. However,it does seem a very clumsy way compared to VB or Pascal.Is it the best method we have?
Not only is it clumsy, it involves mutating the original parameter in some way manually eg by changing the original parameter to a list: or copying it to another list rather than just saying: "use this parameter as a value " or "use this one as a reference". Could the simple answer be there is no reserved word for this but these are great work arounds?
class demoClass:
x = 4
y = 3
foo1 = demoClass()
foo1.x = 2
foo2 = demoClass()
foo2.y = 5
def mySquare(myObj):
myObj.x = myObj.x**2
myObj.y = myObj.y**2
print('foo1.x =', foo1.x)
print('foo1.y =', foo1.y)
print('foo2.x =', foo2.x)
print('foo2.y =', foo2.y)
mySquare(foo1)
mySquare(foo2)
print('After square:')
print('foo1.x =', foo1.x)
print('foo1.y =', foo1.y)
print('foo2.x =', foo2.x)
print('foo2.y =', foo2.y)
In Python the passing by reference or by value has to do with what are the actual objects you are passing.So,if you are passing a list for example,then you actually make this pass by reference,since the list is a mutable object.Thus,you are passing a pointer to the function and you can modify the object (list) in the function body.
When you are passing a string,this passing is done by value,so a new string object is being created and when the function terminates it is destroyed.
So it all has to do with mutable and immutable objects.
Python already call by ref..
let's take example:
def foo(var):
print(hex(id(var)))
x = 1 # any value
print(hex(id(x))) # I think the id() give the ref...
foo(x)
OutPut
0x50d43700 #with you might give another hex number deppend on your memory
0x50d43700
Related
A dictionary that I pass as an argument seems to behave like a global variable. This was a surprise to me but it seems that it makes sense in Python (see for instance this topic). However, I'm very surprised that I get a different behavior for a different type of variable. Let assume that I have a script main.py that calls two functions from a module.
import mymodule
an_int = 42
a_dict = {'value':42}
mymodule.print_and_reassign(a_dict, an_int)
mymodule.print_again(a_dict, an_int)
with my modules.py containing the following functions
def print_and_reassign(a_dict, an_int):
# Dictionary
print(f"Dictionary value: {a_dict['value']}")
a_dict['value']=970
# Integer
print(f"Integer value: {an_int}")
an_int=970
def print_again(a_dict, an_int):
# Dictionary
print(f"Dictionary value: {a_dict['value']}")
# Integer
print(f"Integer value: {an_int}")
By running main.py, I get the following results
Dictionnary value: 42
Integer value: 42
Dictionnary value: 970
Integer value: 42
This is very counterintuitive to me. Why does the dictionary changed while the integer remains unchanged?
Yes, that's a quite confusing part of programming in Python (or, rather, programming in general).
What happens is essentially that what you have in the beginning is this:
When calling the function, the variables inside the function's scope are actually something different than the outer ones, even though you named them the same and they initially have the same value:
Then you change them, and this happens:
And after the function returns and its scope is destroyed, you are left with this:
Calling a_dict['value'] = 970 underneath manipulates a_dict by calling its __setitem__ method, therefore a_dict's contents change.
Calling an_int = 970 assigns a new value to a local an_int variable, shadowing the value passed as a parameter.
As #jonsharpe said above,
Dictionaries are mutable, integers aren't. And assigning a new value to the local variable is fundamentally different to calling a method on the value it refers to
To reinforce jon's point, try reassigning the variable a_dict inside the function like so: a_dict = {"value": 970}.
def print_and_reassign(a_dict, an_int):
print(f"Dictionary value before: {a_dict['value']}")
a_dict = {"value": 970}
print(f"Dictionary value after: {a_dict['value']}")
print(f"Integer value before: {an_int}")
an_int = 970
print(f"Integer value after: {an_int}")
def print_again(a_dict, an_int):
# Dictionary
print(f"Dictionary value outside: {a_dict['value']}")
# Integer
print(f"Integer value outside: {an_int}")
Now, when you run your code you see that a_dict is not modified outside the function.
Dictionary value before: 42
Dictionary value after: 970
Integer value before: 42
Integer value after: 970
Dictionary value outside: 42
Integer value outside: 42
When you assign a new value to the name a_dict inside the function, it isn't reflected outside the function. This is what happens when you do an_int = 970. Since integers are immutable, this is also what would happen if you did something like an_int += 100.
When you do a_dict['value'] = 970, you modify the value that is referred to by a_dict. Since dicts are mutable, this modification doesn't need a reassignment so it shows in every variable refers to this value.
The fact that there are sets, dictionaries and lists along with integers, strings and floats is a common reason for confusion.
What is the difference between the first group compared to the second? This mind boggling question comes up again and again in myriads of different flavors. So maybe it needs one more answer next to all the existing ones? I think yes, it needs and this motivates me to provide one below in the hope it will be helpful in resolving the confusion.
It is not the dictionary passed to function which shows 'strange' behavior. To get to the core of the reason for experiencing counter-intuitive behavior I will compare here the behavior of a Python integer vs a Python list.
Below a code snippet which demonstrates that you can change values in the list using for this purpose another identifier/reference. This can't be done with an identifier representing an integer value:
L = [3]
id_L_before = id(L)
L[0] = 4 # Changing a value in the list don't change the list id()
assert id(L) == id_L_before
# ---
P = L # using another identifier/reference to the list object
P[0] = 5 # changing a value in the "other" P list
assert L[0] == 5 # changes "magically" the value in the list L
# ---
# You can't do the same with an identifier/reference to an int value:
n = 3
id_n_before = id(n)
n = 4
assert id(n) != id_n_before # Changing the value changes the id()
m = n
m = 5 # changing the value in the "other" int m
assert n == 4 # DOES NOT change "magically" the value in the int n
The code above along with the explanations given as code comment should be enough to sufficiently answer the question.
If you want to see more of the strange and counter-intuitive aspects of Python just visit The Python's Horror Show . which purpose is
to mess with your head but some people have reported strange new Python knowledge as a secondary effect
Here an excerpt:
d = {1: 'a', True: 'b', 1.0: 'c'}
>>> d
{1: 'c'}
This question already has answers here:
Why can a function modify some arguments as perceived by the caller, but not others?
(13 answers)
Closed 7 months ago.
Once I define a list in main() and passed it as a function parameter, I expected that its content will remain the same once we exit the function. I have the following code snippet:
def test(a):
a.append(1000)
return a
def main():
a = [1,2,3]
print(a, test(a))
main()
Output is clear.
# Output: [1,2,3,1000] [1,2,3,1000]
Even though the initial list contains 3 elements, function test() seems to modify its content. This thing does not happen to variables:
def test(a):
a = a + 10
return a
def main():
a = 5
print(a, test(a))
main()
# Output: 5 15
Why is that? Why does Python modify the initial lists and not variables?
a.append()
mutates the a which was passed by reference
a = a + 10
assigns a value to a new local variable in the function called a
Integers are immutable, but if there were some method that could be called on that a (on the right hand side) to mutate it, it would also be reflected in the caller.
Some objects in python (like lists, dicts and sets) are mutable, whereas others (like tuples, frozensets, strings and numbers) are immutable.
Operations on mutable objects modify the object themselves. Many immutable objects also allow operations, which in this case return a new object. (Internally this is implemented with the dunder "magic methods", which in this case just return a new object. x OPERATOR y actually calls a method on x(and/or y).)
Since in python x = y binds the name x to point to y, if y is mutable and you modify it, x will point to the modified y. But if y is immutable (but allows operations), operations on y will create a new y, to which x no longer points:
x = [] # mutable
y = x # y and x both point to a particular '[]'
x += ["hi"]
y == ["hi"] # True
x = 0
y = x # y and x both point to 0
x += 7 # x now points to 7
y == 7 # False
Exactly the same thing goes on when you parse an argument to a function---you bind a name in the function's scope pointing to that object. What then happens to the object pointed to by the name depends upon the type of object and what you do to it. If it's mutable and you mutate the object pointed to by the name in the function, you will mutate the object itself, and any other names bound to that object ouside the function will see the changed object.
This is the reason that we don't generally do this:
def f(x=[]):
...
Since f is evaluated once, when the module is first loaded, x will point to the same list for the duration of the function. Thus if the function does anything to x, x will be in that state next time it's called (which is rarely what you want).
I thought I had a good handle on how Python passes objects (this article seemed enlightening).
Then I tried something simple, just assigning functions to variables.
class Thingy:
def __init__(self):
self.foo = {"egg": [1], "spam": [2]}
def calc(self):
self.foo["egg"][0] = 3
self.foo["spam"][0] = 4
def egg(self):
return self.foo["egg"][0]
def spam(self):
return self.foo["spam"]
thingy = Thingy()
x = thingy.egg()
y = thingy.spam()
print(x) # prints 1
print(y[0]) # prints 2
print(thingy.foo)
thingy.calc()
print(x) # prints 1 (???)
print(y[0]) # prints 4
print(thingy.foo)
I'm not entirely sure what's going on, especially as the value in the dictionary has been updated. My guess is that when the variable x is assigned, it is actually referring to a function whose return value has been evaluated to "1" already.
Is my understanding correct? I'd appreciate a clear explanation of why Python is deciding to treat .egg() and .spam() differently.
The statements
x = thingy.egg()
y = thingy.spam()
create x as an integer and y as a list. But what you must know is that the line y = thingy.spam() is just a shallow copy.
This is how shallow copy is defined by medium :
Shallow copy is a bit-wise copy of an object. A new object is created that has an exact copy of the values in the original object. If any of the fields of the object are references to other objects, just the reference addresses are copied i.e., only the memory address is copied.
So the variable y contains the address (or reference in more layman's term) to the elements of the list, and changing the list changes it also, unlike x where a new memory location is assigned.
when you run
x = thingy.egg()
thingy.egg() return an int witch has the same value as the one in foo["egg"][0] ans is than assigned to x whereas
y = thingy.spam()
thingy.spam() returns a list containing 4 witch is the same list as foo["spam"]. This has to do with object mutability. In your calculate function the 2 integers foo["egg"][0] and foo["spam"][0] are redefined with new objects. However the lists hey are contained in remaine as the same object. As x refers to the integer it remains the old object so 1 and y alsow stays as the same object however that object is a list and its first element now refers to the new int object witch is 4
I am asking because of the classic problem where somebody creates a list of lambdas:
foo = []
for i in range(3):
foo.append((lambda: i))
for l in foo:
print(l())
and unexpectedly gets only twos as output.
The commonly proposed solution is to make i a named argument like this:
foo = []
for i in range(3):
foo.append((lambda i=i: i))
for l in foo:
print(l())
Which produces the desired output of 0, 1, 2 but now something magical has happened. It sort of did what is expected because Python is pass-by-reference and you didn't want a reference.
Still, just adding a new name to something, shouldn't that just create another reference?
So the question becomes what are the exact rules for when something is not a reference?
Considering that ints are immutable and the following works:
x = 3
y = x
x = 5
print(x, y) // outputs 5 3
probably explains why adding that named parameter works. A local i with the same value was created and captured.
Now why, in the case of our lambdas was the same i referenced? I pass an int to function and it is refenced and if I store it in a variable it is copied. Hm.
Basically I am looking for the most concise and abstract way possible to remember exactly how this works. When is the same value referenced, when do I get a copy. If it has any common names and there are programming languages were it works the same that would be interesting as well.
Here is my current assumption:
Arguments are always passed to functions by reference.
Assigning to a variable of immutable type creates a copy.
I am asking anyway, just to make sure and hopefully get some background.
The issue here is how you think of names.
In your first example, i is a variable that is assigned to every time the loop iterates. When you use lambda to make a function, you make a function that accesses the name i and returns it's value. This means as the name i changes, the value returned by the functions also changes.
The reason the default argument trick works is that the name is evaluated when the function is defined. This means the default value is the value the i name points to at that time, not the name itself.
i is a label. 0, 1 and 2 are the objects. In the first case, the program assigns 0 to i, then makes a function that returns i - it then does this with 1 and 2. When the function is called, it looks up i (which is now 2) and then returns it.
In the second example, you assign 0 to i, then you make a function with a default argument. That default argument is the value that is gotten by evaluating i - that is the object 0. This is repeated for 1 and 2. When the function is called, it assigns that default value to a new variable i, local to the function and unrelated to the outer i.
Python doesn't exactly pass by reference or by value (at least, not the way you'd think of it, coming from a language like C++).
In many other languages (such as C++), variables can be thought of as synonymous with the values they hold.
However, in Python, variables are names that point to the objects in memory.
(This is a good explanation (with pictures!))
Because of this, you can get multiple names attached to one object, which can lead to interesting effects.
Consider these equivalent program snippets:
// C++:
int x;
x = 10; // line A
x = 20; // line B
and
# Python:
x = 10 # line C
x = 20 # line D
After line A, the int 10 is stored in memory, say, at the memory address 0x1111.
After line B, the memory at 0x1111 is overwritten, so 0x1111 now holds the int 20
However, the way this program works in python is quite different:
After line C, x points to some memory, say, 0x2222, and the value stored at 0x2222 is 10
After line D, x points to some different memory, say, 0x3333, and the value stored at 0x3333 is 20
Eventually, the orphaned memory at 0x2222 is garbage collected by Python.
Hopefully this helps you get a grasp of the subtle differences between variables in Python and most other languages.
(I know I didn't directly answer your question about lambdas, but I think this is good background knowledge to have before reading one of the good explanations here, such as #Lattyware's)
See this question for some more background info.
Here's some final background info, in the form of oft-quoted but instructive examples:
print 'Example 1: Expected:'
x = 3
y = x
x = 2
print 'x =', x
print 'y =', y
print 'Example 2: Surprising:'
x = [3]
y = x
x[0] = 2
print 'x =', x
print 'y =', y
print 'Example 3: Same logic as in Example 1:'
x = [3]
y = x
x = [2]
print 'x =', x
print 'y =', y
The output is:
Example 1: Expected:
x = 2
y = 3
Example 2: Surprising:
x = [2]
y = [2]
Example 3: Same logic as in Example 1:
x = [2]
y = [3]
foo = []
for i in range(3):
foo.append((lambda: i))
Here since all the lambda's were created in the same scope so all of them point to the same global variable variable i. so, whatever value i points to will be returned when they are actually called.
foo = []
for i in range(3):
foo.append((lambda z = i: id(z)))
print id(i) #165618436
print(foo[-1]()) #165618436
Here in each loop we assign the value of i to a local variable z, as default arguments are calculated when the function is parsed so the value z simply points to the values stored by i during the iteration.
Arguments are always passed to functions by reference?
In fact the z in foo[-1] still points to the same object as i of the last iteration, so yes values are passed by reference but as integers are immutable so changing i won't affect z of the foo[-1] at all.
In the example below all lambda's point to some mutable object, so modifying items in lis will also affect the functions in foo:
foo = []
lis = ([], [], [])
for i in lis:
foo.append((lambda z = i: z))
lis[0].append("bar")
print foo[0]() #prints ['bar']
i.append("foo") # `i` still points to lis[-1]
print foo[-1]() #prints ['foo']
Assigning to a variable of immutable type creates a copy?
No values are never copied.
>>> x = 1000
>>> y = x # x and y point to the same object, but an immutable object.
>>> x += 1 # so modifying x won't affect y at all, in fact after this step
# x now points to some different object and y still points to
# the same object 1000
>>> x #x now points to an new object, new id()
1001
>>> y #still points to the same object, same id()
1000
>>> x = []
>>> y = x
>>> x.append("foo") #modify an mutable object
>>> x,y #changes can be seen in all references to the object
(['foo'], ['foo'])
The list of lambdas problem arises because the i referred to in both snippets is the same variable.
Two distinct variables with the same name exist only if they exist in two separate scopes. See the following link for when that happens, but basically any new function (including a lambda) or class establishes its own scope, as do modules, and pretty much nothing else does. See: http://docs.python.org/2/reference/executionmodel.html#naming-and-binding
HOWEVER, when reading the value of a variable, if it is not defined in the current local scope, the enclosing local scopes are searched*. Your first example is of exactly this behaviour:
foo = []
for i in range(3):
foo.append((lambda: i))
for l in foo:
print(l())
Each lambda creates no variables at all, so its own local scope is empty. When execution hits the locally undefined i, it is located in the enclosing scope.
In your second example, each lambda creates its own i variable in the parameter list:
foo = []
for i in range(3):
foo.append((lambda i=i: i))
This is in fact equivalent to lambda a=i: a, because the i inside the body is the same as the i on the left hand side of the assignment, and not the i on the right hand side. The consequence is that i is not missing from the local scope, and so the value of the local i is used by each lambda.
Update: Both of your assumptions are incorrect.
Function arguments are passed by value. The value passed is the reference to the object. Pass-by-reference would allow the original variable to be altered.
No implicit copying ever occurs on function call or assignment, of any language-level object. Under the hood, because this is pass-by-value, the references to the parameter objects are copied when the function is called, as is usual in any language which passes references by value.
Update 2: The details of function evaluation are here: http://docs.python.org/2/reference/expressions.html#calls . See the link above for the details regarding name binding.
* No actual linear search occurs in CPython, because the correct variable to use can be determined at compile time.
The answer is that the references created in a closure (where a function is inside a function, and the inner function accesses variables from the outer one) are special. This is an implementation detail, but in CPython the value is a particular kind of object called a cell and it allows the variable's value to be changed without rebinding it to a new object. More info here.
The way variables work in Python is actually rather simple.
All variables contain references to objects.
Reassigning a variable points it to a different object.
All arguments are passed by value when calling functions (though the values being passed are references).
Some types of objects are mutable, which means they can be changed without changing what any of their variable names point to. Only these types can be changed when passed, since this does not require changing any references to the object.
Values are never copied implicitly. Never.
The behaviour really has very little to do with how parameters are passed (which is always the same way; there is no distinction in Python where things are sometimes passed by reference and sometimes passed by value). Rather the problem is to do with how names themselves are found.
lambda: i
creates a function that is of course equivalent to:
def anonymous():
return i
That i is a name, within the scope of anonymous. But it's never bound within that scope (not even as a parameter). So for that to mean anything i must be a name from some outer scope. To find a suitable name i, Python will look at the scope in which anonymous was defined in the source code (and then similarly out from there), until it finds a definition for i.1
So this loop:
foo = []
for i in range(3):
foo.append((lambda: i))
for l in foo:
print(l())
Is almost exactly as if you had written this:
foo = []
for i in range(3):
def anonymous():
return i
foo.append(anonymous)
for l in foo:
print(l())
So that i in return i (or lambda: i) ends up being the same i from the outer scope, which is the loop variable. Not that they are all references to the same object, but that they are all the same name. So it's simply not possible for the functions stored in foo to return different values; they're all returning the object referred to by a single name.
To prove it, watch what happens when I remove the variable i after the loop:
>>> foo = []
>>> for i in range(3):
foo.append((lambda: i))
>>> del i
>>> for l in foo:
print(l())
Traceback (most recent call last):
File "<pyshell#7>", line 2, in <module>
print(l())
File "<pyshell#3>", line 2, in <lambda>
foo.append((lambda: i))
NameError: global name 'i' is not defined
You can see that the problem isn't that each function has a local i bound to the wrong thing, but rather than each function is returning the value of the same global variable, which I've now removed.
OTOH, when your loop looks like this:
foo = []
for i in range(3):
foo.append((lambda i=i: i))
for l in foo:
print(l())
That is quite like this:
foo = []
for i in range(3):
def anonymous(i=i):
return i
foo.append(anonymous)
for l in foo:
print(l())
Now the i in return i is not the same i as in the outer scope; it's a local variable of the function anonymous. A new function is created in each iteration of the loop (stored temporarily in the outer scope variable anonymous, and then permanently in a slot of foo), so each one has it's own local variables.
As each function is created, the default value of its parameter is set to the value of i (in the scope defining the functions). Like any other "read" of a variable, that pulls out whatever object is referenced by the variable at that time, and thereafter has no connection to the variable.2
So each function gets the default value of i as it is in the outer scope at the time it is created, and then when the function is called without an argument that default value becomes the value of the i in that function's local scope. Each function has no non-local references, so is completely unaffected by what happens outside it.
1 This is done at "compile time" (when the Python file is converted to bytecode), with no regard for what the system is like at runtime; it is almost literally looking for an outer def block with i = ... in the source code. So local variables are actually statically resolved! If that lookup chain falls all the way out to the module global scope, then Python assumes that i will be defined in the global scope at the point that the code will be run, and just treats i as a global variable whether or not there is a statically visible binding for i at module scope, hence why you can dynamically create global variables but not local ones.
2 Confusingly, this means that in lambda i=i: i, the three is refer to three completely different "variables" in two different scopes on the one line.
The leftmost i is the "name" holding the value that will be used for the default value of i, which exists independently of any particular call of the function; it's almost exactly "member data" stored in the function object.
The second i is an expression evaluated as the function is created, to get the default value. So the i=i bit acts very like an independent statement the_function.default_i = i, evaluated in the same scope containing the lambda expression.
And finally the third i is actually the local variable inside the function, which only exists within a call to the anonymous function.
This question already has answers here:
Does Python make a copy of objects on assignment?
(5 answers)
How do I pass a variable by reference?
(39 answers)
Why can a function modify some arguments as perceived by the caller, but not others?
(13 answers)
Closed last month.
For a project I'm working on, I'm implementing a linked-list data-structure, which is based on the idea of a pair, which I define as:
class Pair:
def __init__(self, name, prefs, score):
self.name = name
self.score = score
self.preferences = prefs
self.next_pair = 0
self.prev_pair = 0
where self.next_pair and self.prev_pair are pointers to the previous and next links, respectively.
To set up the linked-list, I have an install function that looks like this.
def install(i, pair):
flag = 0
try:
old_pair = pair_array[i]
while old_pair.next_pair != 0:
if old_pair == pair:
#if pair in remainders: remainders.remove(pair)
return 0
if old_pair.score < pair.score:
flag = 1
if old_pair.prev_pair == 0: # we are at the beginning
old_pair.prev_pair = pair
pair.next_pair = old_pair
pair_array[i] = pair
break
else: # we are not at the beginning
pair.prev_pair = old_pair.prev_pair
pair.next_pair = old_pair
old_pair.prev_pair = pair
pair.prev_pair.next_pair = pair
break
else:
old_pair = old_pair.next_pair
if flag==0:
if old_pair == pair:
#if pair in remainders: remainders.remove(pair)
return 0
if old_pair.score < pair.score:
if old_pair.prev_pair==0:
old_pair.prev_pair = pair
pair.next_pair = old_pair
pair_array[i] = pair
else:
pair.prev_pair = old_pair.prev_pair
pair.next_pair = old_pair
old_pair.prev_pair = pair
pair.prev_pair.next_pair = pair
else:
old_pair.next_pair = pair
pair.prev_pair = old_pair
except KeyError:
pair_array[i] = pair
pair.prev_pair = 0
pair.next_pair = 0
Over the course of the program, I am building up a dictionary of these linked-lists, and taking links off of some and adding them in others. Between being pruned and re-installed, the links are stored in an intermediate array.
Over the course of debugging this program, I have come to realize that my understanding of the way Python passes arguments to functions is flawed. Consider this test case I wrote:
def test_install():
p = Pair(20000, [3, 1, 2, 50], 45)
print p.next_pair
print p.prev_pair
parse_and_get(g)
first_run()
rat = len(juggler_array)/len(circuit_array)
pref_size = get_pref_size()
print pref_size
print install(3, p)
print p.next_pair.name
print p.prev_pair
When I run this test, I get the following result.
0
0
10
None
10108
0
What I don't understand is why the second call to p.next_pair produces a different result (10108) than the first call (0). install does not return a Pair object that can overwrite the one passed in (it returns None), and it's not as though I'm passing install a pointer.
My understanding of call-by-value is that the interpreter copies the values passed into a function, leaving the caller's variables unchanged. For example, if I say
def foo(x):
x = x+1
return x
baz = 2
y = foo(baz)
print y
print baz
Then 3 and 2 should be printed, respectively. And indeed, when I test that out in the Python interpreter, that's what happens.
I'd really appreciate it if anyone can point me in the right direction here.
In Python, everything is an object. Simple assignment stores a reference to the assigned object in the assigned-to name. As a result, it is more straightforward to think of Python variables as names that are assigned to objects, rather than objects that are stored in named locations.
For example:
baz = 2
... stores in baz a pointer, or reference, to the integer object 2 which is stored elsewhere. (Since the type int is immutable, Python actually has a pool of small integers and reuses the same 2 object everywhere, but this is an implementation detail that need not concern us much.)
When you call foo(baz), foo()'s local variable x also points to the integer object 2 at first. That is, the foo()-local name x and the global name baz are names for the same object, 2. Then x = x + 1 is executed. This changes x to point to a different object: 3.
It is important to understand: x is not a box that holds 2, and 2 is then incremented to 3. No, x initially points to 2 and that pointer is then changed to point to 3. Naturally, since we did not change what object baz points to, it still points to 2.
Another way to explain it is that in Python, all argument passing is by value, but all values are references to objects.
A counter-intuitive result of this is that if an object is mutable, it can be modified through any reference and all references will "see" the change. For example, consider this:
baz = [1, 2, 3]
def foo(x):
x[0] = x[0] + 1
foo(baz)
print baz
>>> [2, 2, 3]
This seems very different from our first example. But in reality, the argument is passed the same way. foo() receives a pointer to baz under the name x and then performs an operation on it that changes it (in this case, the first element of the list is pointed to a different int object). The difference is that the name x is never pointed to a new object; it is x[0] that is modified to point to a different object. x itself still points to the same object as baz. (In fact, under the hood the assignment to x[0] becomes a method call: x.__setitem__().) Therefore baz "sees" the modification to the list. How could it not?
You don't see this behavior with integers and strings because you can't change integers or strings; they are immutable types, and when you modify them (e.g. x = x + 1) you are not actually modifying them but binding your variable name to a completely different object. If you change baz to a tuple, e.g. baz = (1, 2, 3), you will find that foo() gives you an error because you can`t assign to elements of a tuple; tuples are another immutable type. "Changing" a tuple requires creating a new one, and assignment then points the variable to the new object.
Objects of classes you define are mutable and so your Pair instance can be modified by any function it is passed into -- that is, attributes may be added, deleted, or reassigned to other objects. None of these things will re-bind any of the names pointing to your object, so all the names that currently point to it will "see" the changes.
Python does not copy anything when passing variables to a function. It is neither call-by-value nor call-by-reference, but of those two it is more similar to call-by-reference. You could think of it as "call-by-value, but the value is a reference".
If you pass a mutable object to a function, then modifying that object inside the function will affect the object everywhere it appears. (If you pass an immutable object to a function, like a string or an integer, then by definition you can't modify the object at all.)
The reason this isn't technically pass-by-reference is that you can rebind a name so that the name refers to something else entirely. (For names of immutable objects, this is the only thing you can do to them.) Rebinding a name that exists only inside a function doesn't affect any names that might exist outside the function.
In your first example with the Pair objects, you are modifying an object, so you see the effects outside of the function.
In your second example, you are not modifying any objects, you are just rebinding names to other objects (other integers in this case). baz is a name that points to an integer object (in Python, everything is an object, even integers) with a value of 2. When you pass baz to foo(x), the name x is created locally inside the foo function on the stack, and x is set to the pointer that was passed into the function -- the same pointer as baz. But x and baz are not the same thing, they only contain pointers to the same object. On the x = x+1 line, x is rebound to point to an integer object with a value of 3, and that pointer is what is returned from the function and used to bind the integer object to y.
If you rewrote your first example to explicitly create a new Pair object inside your function based on the information from the Pair object passed into it (whether this is a copy you then modify, or if you make a constructor that modifies the data on construction) then your function would not have the side-effect of modifying the object that was passed in.
Edit: By the way, in Python you shouldn't use 0 as a placeholder to mean "I don't have a value" -- use None. And likewise you shouldn't use 0 to mean False, like you seem to be doing in flag. But all of 0, None and False evaluate to False in boolean expressions, so no matter which of those you use, you can say things like if not flag instead of if flag == 0.
I suggest that you forget about implementing a linked list, and simply use an instance of a Python list. If you need something other than the default Python list, maybe you can use something from a Python module such as collections.
A Python loop to follow the links in a linked list will run at Python interpreter speed, which is to say, slowly. If you simply use the built-in list class, your list operations will happen in Python's C code, and you will gain speed.
If you need something like a list but with fast insertion and fast deletion, can you make a dict work? If there is some sort of ID value (string or integer or whatever) that can be used to impose an ordering on your values, you could just use that as a key value and gain lightning fast insert and delete of values. Then if you need to extract values in order, you can use the dict.keys() method function to get a list of key values and use that.
But if you really need linked lists, I suggest you find code written and debugged by someone else, and adapt it to your needs. Google search for "python linked list recipe" or "python linked list module".
I'm going to throw in a slightly complicating factor:
>>> def foo(x):
... x *= 2
... return x
...
Define a slightly different function using a method I know is supported for numbers, lists, and strings.
First, call it with strings:
>>> baz = "hello"
>>> y = foo(baz)
>>> y
'hellohello'
>>> baz
'hello'
Next, call it with lists:
>>> baz=[1,2,2]
>>> y = foo(baz)
>>> y
[1, 2, 2, 1, 2, 2]
>>> baz
[1, 2, 2, 1, 2, 2]
>>>
With strings, the argument isn't modified. With lists, the argument is modified.
If it were me, I'd avoid modifying arguments within methods.