Related
I am very confused by the behaviour below. Cases 1, 3, and 4 perform as I would expect, but case 2 does not. Why does case 2 allow the function to change the value of the dictionary entry globally, even though the dictionary is never returned by the function? A main reason I am using functions is to isolate everything in the function from the rest of the code, but this does not seem to be possible if I choose to use the same variable names inside of the function. I was under the understanding that anything explicitly defined in a function is local to that function, but this does not seem to be the case if the dictionary is defined and passed as an input to the function.
Case 1
>>> def testfun1(a):
... a=2
...
>>> a=0
>>> testfun1(a)
>>> a
0
Case 2
>>> def testfun2(b):
... b['test']=2
...
>>> b={}
>>> testfun2(b)
>>> b
{'test': 2}
Case 3
>>> def testfun3():
... c=2
...
>>> c=0
>>> testfun3()
>>> c
0
Case 4
(explained by this question: Global dictionaries don't need keyword global to modify them?)
>>> def testfun4():
... d['test']=10
...
>>> d={}
>>> testfun4()
>>> d
{'test': 10}
Python's "parameter evaluation strategy" acts a bit different than the languages you're probably used to. Instead of having explicit call by value and call by reference semantics, python has call by sharing. You are essentially always passing the object itself, and the object's mutability determines whether or not it can be modified. Lists and Dicts are mutable objects. Numbers, Strings, and Tuples are not.
You are passing the dictionary to the function, not a copy. Thus when you modify it, you are also modifying the original copy.
To avoid this, you should first copy the dictionary before calling the function, or from within the function (passing the dictionary to the dict function should do it, i.e. testfun4(dict(d)) and defining the function as def testfun4(d):).
To support what #Casey Kuball said, every object in Python is passed by reference. Each function receives a reference to the actual object you passed. Modifying these objects depends on whether they are mutable data types.
In essence, one can say that mutable objects like dictionaries, sets, and lists are passed by reference. Immutable objects like int, str, tuple are passed by value.
You should also note that there are cases where mutable objects are overwritten in a function thereby losing reference to the actual object passed to the function.
>>> def testfun(b):
... b = b or {} # Creates a new object if b is false
... b['test'] = 2
...
>>> b = {}
>>> testfun(b)
>>> b
{}
When you pass a basic object like an integer or a string to a function, if you change it inside the function nothing occurs to the corresponding object outside the function because when you are leading with a basic object, python passes it by value.
However, if you pass a dictionary or a list to a function they are passed by reference, which means you will have that behaviour: the object outside the function is changed, as you have seen.
edit:
In addition, there is a difference between passing by value or by reference: by value, a "copy" of the object is made in order to be used in the function; by reference, the exactly same object is passed through reference and modifications to it inside the function are visible outside. By definition python passes its immutable objects by value, and its mutable objects by reference.
The global keyword is required only for assignment (and likely del, I've never tried it). Object mutations are perfectly valid.
You have passed a dict object to the function and modified it inside the function, so of course it will be modified after the function return. The object is not copied so you modify the same object that you passed, and this question has nothing to do with naming, similar names, scopes etc. as you passed the object explicitly.
Suppose I have a function like:
def foo():
x = 'hello world'
How do I get the function to return x, in such a way that I can use it as the input for another function or use the variable within the body of a program? I tried using return and then using the x variable in another function, but I get a NameError that way.
For the specific case of communicating information between methods in the same class, it is often best to store the information in self. See Passing variables between methods in Python? for details.
def foo():
x = 'hello world'
return x # return 'hello world' would do, too
foo()
print(x) # NameError - x is not defined outside the function
y = foo()
print(y) # this works
x = foo()
print(x) # this also works, and it's a completely different x than that inside
# foo()
z = bar(x) # of course, now you can use x as you want
z = bar(foo()) # but you don't have to
Effectively, there are two ways: directly and indirectly.
The direct way is to return a value from the function, as you tried, and let the calling code use that value. This is normally what you want. The natural, simple, direct, explicit way to get information back from a function is to return it. Broadly speaking, the purpose of a function is to compute a value, and return signifies "this is the value we computed; we are done here".
Directly using return
The main trick here is that return returns a value, not a variable. So return x does not enable the calling code to use x after calling the function, and does not modify any existing value that x had in the context of the call. (That's presumably why you got a NameError.)
After we use return in the function:
def example():
x = 'hello world'
return x
we need to write the calling code to use the return value:
result = example()
print(result)
The other key point here is that a call to a function is an expression, so we can use it the same way that we use, say, the result of an addition. Just as we may say result = 'hello ' + 'world', we may say result = foo(). After that, result is our own, local name for that string, and we can do whatever we want with it.
We can use the same name, x, if we want. Or we can use a different name. The calling code doesn't have to know anything about how the function is written, or what names it uses for things.1
We can use the value directly to call another function: for example, print(foo()).2 We can return the value directly: simply return 'hello world', without assigning to x. (Again: we are returning a value, not a variable.)
The function can only return once each time it is called. return terminates the function - again, we just determined the result of the calculation, so there is no reason to calculate any further. If we want to return multiple pieces of information, therefore, we will need to come up with a single object (in Python, "value" and "object" are effectively synonyms; this doesn't work out so well for some other languages.)
We can make a tuple right on the return line; or we can use a dictionary, a namedtuple (Python 2.6+), a types.simpleNamespace (Python 3.3+), a dataclass (Python 3.7+), or some other class (perhaps even one we write ourselves) to associate names with the values that are being returned; or we can accumulate values from a loop in a list; etc. etc. The possibilities are endless..
On the other hand, the function returns whether you like it or not (unless an exception is raised). If it reaches the end, it will implicitly return the special value None. You may or may not want to do it explicitly instead.
Indirect methods
Other than returning the result back to the caller directly, we can communicate it by modifying some existing object that the caller knows about. There are many ways to do that, but they're all variations on that same theme.
If you want the code to communicate information back this way, please just let it return None - don't also use the return value for something meaningful. That's how the built-in functionality works.
In order to modify that object, the called function also has to know about it, of course. That means, having a name for the object that can be looked up in a current scope. So, let's go through those in order:
Local scope: Modifying a passed-in argument
If one of our parameters is mutable, we can just mutate it, and rely on the caller to examine the change. This is usually not a great idea, because it can be hard to reason about the code. It looks like:
def called(mutable):
mutable.append('world')
def caller():
my_value = ['hello'] # a list with just that string
called(my_value)
# now it contains both strings
If the value is an instance of our own class, we could also assign to an attribute:
class Test:
def __init__(self, value):
self.value = value
def called(mutable):
mutable.value = 'world'
def caller():
test = Test('hello')
called(test)
# now test.value has changed
Assigning to an attribute does not work for built-in types, including object; and it might not work for some classes that explicitly prevent you from doing it.
Local scope: Modifying self, in a method
We already have an example of this above: setting self.value in the Test.__init__ code. This is a special case of modifying a passed-in argument; but it's part of how classes work in Python, and something we're expected to do. Normally, when we do this, the calling won't actually check for changes to self - it will just use the modified object in the next step of the logic. That's what makes it appropriate to write code this way: we're still presenting an interface, so the caller doesn't have to worry about the details.
class Example:
def __init__(self):
self._words = ['hello']
def add_word(self):
self._words.append('world')
def display(self):
print(*self.words)
x = Example()
x.add_word()
x.display()
In the example, calling add_word gave information back to the top-level code - but instead of looking for it, we just go ahead and call display.3
See also: Passing variables between methods in Python?
Enclosing scope
This is a rare special case when using nested functions. There isn't a lot to say here - it works the same way as with the global scope, just using the nonlocal keyword rather than global.4
Global scope: Modifying a global
Generally speaking, it is a bad idea to change anything in the global scope after setting it up in the first place. It makes code harder to reason about, because anything that uses that global (aside from whatever was responsible for the change) now has a "hidden" source of input.
If you still want to do it, the syntax is straightforward:
words = ['hello']
def add_global_word():
words.append('world')
add_global_word() # `words` is changed
Global scope: Assigning to a new or existing global
This is actually a special case of modifying a global. I don't mean that assignment is a kind of modification (it isn't). I mean that when you assign a global name, Python automatically updates a dict that represents the global namespace. You can get that dict with globals(), and you can modify that dict and it will actually impact what global variables exist. (I.e., the return from globals() is the dictionary itself, not a copy.)5
But please don't. That's even worse of an idea than the previous one. If you really need to get the result from your function by assigning to a global variable, use the global keyword to tell Python that the name should be looked up in the global scope:
words = ['hello']
def replace_global_words():
global words
words = ['hello', 'world']
replace_global_words() # `words` is a new list with both words
Global scope: Assigning to or modifying an attribute of the function itself
This is a rare special case, but now that you've seen the other examples, the theory should be clear. In Python, functions are mutable (i.e. you can set attributes on them); and if we define a function at top level, it's in the global namespace. So this is really just modifying a global:
def set_own_words():
set_own_words.words = ['hello', 'world']
set_own_words()
print(*set_own_words.words)
We shouldn't really use this to send information to the caller. It has all the usual problems with globals, and it's even harder to understand. But it can be useful to set a function's attributes from within the function, in order for the function to remember something in between calls. (It's similar to how methods remember things in between calls by modifying self.) The functools standard library does this, for example in the cache implementation.
Builtin scope
This doesn't work. The builtin namespace doesn't contain any mutable objects, and you can't assign new builtin names (they'll go into the global namespace instead).
Some approaches that don't work in Python
Just calculating something before the function ends
In some other programming languages, there is some kind of hidden variable that automatically picks up the result of the last calculation, every time something is calculated; and if you reach the end of a function without returning anything, it gets returned. That doesn't work in Python. If you reach the end without returning anything, your function returns None.
Assigning to the function's name
In some other programming languages, you are allowed (or expected) to assign to a variable with the same name as the function; and at the end of the function, that value is returned. That still doesn't work in Python. If you reach the end without returning anything, your function still returns None.
def broken():
broken = 1
broken()
print(broken + 1) # causes a `TypeError`
It might seem like you can at least use the value that way, if you use the global keyword:
def subtly_broken():
global subtly_broken
subtly_broken = 1
subtly_broken()
print(subtly_broken + 1) # 2
But this, of course, is just a special case of assigning to a global. And there's a big problem with it - the same name can't refer to two things at once. By doing this, the function replaced its own name. So it will fail next time:
def subtly_broken():
global subtly_broken
subtly_broken = 1
subtly_broken()
subtly_broken() # causes a `TypeError`
Assigning to a parameter
Sometimes people expect to be able to assign to one of the function's parameters, and have it affect a variable that was used for the corresponding argument. However, this does not work:
def broken(words):
words = ['hello', 'world']
data = ['hello']
broken(data) # `data` does not change
Just like how Python returns values, not variables, it also passes values, not variables. words is a local name; by definition the calling code doesn't know anything about that namespace.
One of the working methods that we saw is to modify the passed-in list. That works because if the list itself changes, then it changes - it doesn't matter what name is used for it, or what part of the code uses that name. However, assigning a new list to words does not cause the existing list to change. It just makes words start being a name for a different list.
For more information, see How do I pass a variable by reference?.
1 At least, not for getting the value back. If you want to use keyword arguments, you need to know what the keyword names are. But generally, the point of functions is that they're an abstraction; you only need to know about their interface, and you don't need to think about what they're doing internally.
2 In 2.x, print is a statement rather than a function, so this doesn't make an example of calling another function directly. However, print foo() still works with 2.x's print statement, and so does print(foo()) (in this case, the extra parentheses are just ordinary grouping parentheses). Aside from that, 2.7 (the last 2.x version) has been unsupported since the beginning of 2020 - which was nearly a 5 year extension of the normal schedule. But then, this question was originally asked in 2010.
3Again: if the purpose of a method is to update the object, don't also return a value. Some people like to return self so that you can "chain" method calls; but in Python this is considered poor style. If you want that kind of "fluent" interface, then instead of writing methods that update self, write methods that create a new, modified instance of the class.
4 Except, of course, that if we're modifying a value rather than assigning, we don't need either keyword.
5 There's also a locals() that gives you a dict of local variables. However, this cannot be used to make new local variables - the behaviour is undefined in 2.x, and in 3.x the dict is created on the fly and assigning to it has no effect. Some of Python's optimizations depend on the local variables for a function being known ahead of time.
>>> def foo():
return 'hello world'
>>> x = foo()
>>> x
'hello world'
You can use global statement and then achieve what you want without returning value from
the function. For example you can do something like below:
def foo():
global x
x = "hello world"
foo()
print x
The above code will print "hello world".
But please be warned that usage of "global" is not a good idea at all and it is better to avoid usage that is shown in my example.
Also check this related discussion on about usage of global statement in Python.
ie we have the global declaration, but no local.
"Normally" arguments are local, I think, or they certainly behave that way.
However if an argument is, say, a list and a method is applied which modifies the list, some surprising (to me) results can ensue.
I have 2 questions: what is the proper way to ensure that a variable is truly local?
I wound up using the following, which works, but it can hardly be the proper way of doing it:
def AexclB(a,b):
z = a+[] # yuk
for k in range(0, len(b)):
try: z.remove(b[k])
except: continue
return z
Absent the +[], "a" in the calling scope gets modified, which is not desired.
(The issue here is using a list method,
The supplementary question is, why is there no "local" declaration?
Finally, in trying to pin this down, I made various mickey mouse functions which all behaved as expected except the last one:
def fun4(a):
z = a
z = z.append(["!!"])
return z
a = ["hello"]
print "a=",a
print "fun4(a)=",fun4(a)
print "a=",a
which produced the following on the console:
a= ['hello']
fun4(a)= None
a= ['hello', ['!!']]
...
>>>
The 'None' result was not expected (by me).
Python 2.7 btw in case that matters.
PS: I've tried searching here and elsewhere but not succeeded in finding anything corresponding exactly - there's lots about making variables global, sadly.
It's not that z isn't a local variable in your function. Rather when you have the line z = a, you are making z refer to the same list in memory that a already points to. If you want z to be a copy of a, then you should write z = a[:] or z = list(a).
See this link for some illustrations and a bit more explanation http://henry.precheur.org/python/copy_list
Python will not copy objects unless you explicitly ask it to. Integers and strings are not modifiable, so every operation on them returns a new instance of the type. Lists, dictionaries, and basically every other object in Python are mutable, so operations like list.append happen in-place (and therefore return None).
If you want the variable to be a copy, you must explicitly copy it. In the case of lists, you slice them:
z = a[:]
There is a great answer than will cover most of your question in here which explains mutable and immutable types and how they are kept in memory and how they are referenced. First section of the answer is for you. (Before How do we get around this? header)
In the following line
z = z.append(["!!"])
Lists are mutable objects, so when you call append, it will update referenced object, it will not create a new one and return it. If a method or function do not retun anything, it means it returns None.
Above link also gives an immutable examle so you can see the real difference.
You can not make a mutable object act like it is immutable. But you can create a new one instead of passing the reference when you create a new object from an existing mutable one.
a = [1,2,3]
b = a[:]
For more options you can check here
What you're missing is that all variable assignment in python is by reference (or by pointer, if you like). Passing arguments to a function literally assigns values from the caller to the arguments of the function, by reference. If you dig into the reference, and change something inside it, the caller will see that change.
If you want to ensure that callers will not have their values changed, you can either try to use immutable values more often (tuple, frozenset, str, int, bool, NoneType), or be certain to take copies of your data before mutating it in place.
In summary, scoping isn't involved in your problem here. Mutability is.
Is that clear now?
Still not sure whats the 'correct' way to force the copy, there are
various suggestions here.
It differs by data type, but generally <type>(obj) will do the trick. For example list([1, 2]) and dict({1:2}) both return (shallow!) copies of their argument.
If, however, you have a tree of mutable objects and also you don't know a-priori which level of the tree you might modify, you need the copy module. That said, I've only needed this a handful of times (in 8 years of full-time python), and most of those ended up causing bugs. If you need this, it's a code smell, in my opinion.
The complexity of maintaining copies of mutable objects is the reason why there is a growing trend of using immutable objects by default. In the clojure language, all data types are immutable by default and mutability is treated as a special cases to be minimized.
If you need to work on a list or other object in a truly local context you need to explicitly make a copy or a deep copy of it.
from copy import copy
def fn(x):
y = copy(x)
I am very confused by the behaviour below. Cases 1, 3, and 4 perform as I would expect, but case 2 does not. Why does case 2 allow the function to change the value of the dictionary entry globally, even though the dictionary is never returned by the function? A main reason I am using functions is to isolate everything in the function from the rest of the code, but this does not seem to be possible if I choose to use the same variable names inside of the function. I was under the understanding that anything explicitly defined in a function is local to that function, but this does not seem to be the case if the dictionary is defined and passed as an input to the function.
Case 1
>>> def testfun1(a):
... a=2
...
>>> a=0
>>> testfun1(a)
>>> a
0
Case 2
>>> def testfun2(b):
... b['test']=2
...
>>> b={}
>>> testfun2(b)
>>> b
{'test': 2}
Case 3
>>> def testfun3():
... c=2
...
>>> c=0
>>> testfun3()
>>> c
0
Case 4
(explained by this question: Global dictionaries don't need keyword global to modify them?)
>>> def testfun4():
... d['test']=10
...
>>> d={}
>>> testfun4()
>>> d
{'test': 10}
Python's "parameter evaluation strategy" acts a bit different than the languages you're probably used to. Instead of having explicit call by value and call by reference semantics, python has call by sharing. You are essentially always passing the object itself, and the object's mutability determines whether or not it can be modified. Lists and Dicts are mutable objects. Numbers, Strings, and Tuples are not.
You are passing the dictionary to the function, not a copy. Thus when you modify it, you are also modifying the original copy.
To avoid this, you should first copy the dictionary before calling the function, or from within the function (passing the dictionary to the dict function should do it, i.e. testfun4(dict(d)) and defining the function as def testfun4(d):).
To support what #Casey Kuball said, every object in Python is passed by reference. Each function receives a reference to the actual object you passed. Modifying these objects depends on whether they are mutable data types.
In essence, one can say that mutable objects like dictionaries, sets, and lists are passed by reference. Immutable objects like int, str, tuple are passed by value.
You should also note that there are cases where mutable objects are overwritten in a function thereby losing reference to the actual object passed to the function.
>>> def testfun(b):
... b = b or {} # Creates a new object if b is false
... b['test'] = 2
...
>>> b = {}
>>> testfun(b)
>>> b
{}
When you pass a basic object like an integer or a string to a function, if you change it inside the function nothing occurs to the corresponding object outside the function because when you are leading with a basic object, python passes it by value.
However, if you pass a dictionary or a list to a function they are passed by reference, which means you will have that behaviour: the object outside the function is changed, as you have seen.
edit:
In addition, there is a difference between passing by value or by reference: by value, a "copy" of the object is made in order to be used in the function; by reference, the exactly same object is passed through reference and modifications to it inside the function are visible outside. By definition python passes its immutable objects by value, and its mutable objects by reference.
The global keyword is required only for assignment (and likely del, I've never tried it). Object mutations are perfectly valid.
You have passed a dict object to the function and modified it inside the function, so of course it will be modified after the function return. The object is not copied so you modify the same object that you passed, and this question has nothing to do with naming, similar names, scopes etc. as you passed the object explicitly.
In C++, void somefunction(int) passes a value, while void somefunction(int&) passes a reference. In Java, primitives are passed by value, while objects are passed by reference. How does python make this decision?
Edit: Since everything is passed by reference, why does this:
def foo(num):
num *= 2
a = 4
foo(a)
print(a)
print '4' instead of '8'?
It passes everything by reference. Even when you specify a numeric value, it is a reference against a table containing that value. This is the difference between static and dynamic languages. The type stays with the value, not with the container, and variables are just references towards a "value space" where all values live. You can assume this value space containing all the possible immutable objects (integers, floats, strings) plus all the mutable ones you create (lists, dicts, objects). Of course, their existence is made concrete only when you involve them (that means, if you never use the number 42 in your program, no allocated space exist for the value 42 in the "value space")
It does that because the number it is referring to is a immutable object. 4 is 4 no matter what.
def foo(num): # here, num is referring to the immutable entity 4
num *= 2 # num now refers to the immutable entity 8
a = 4 # a now is pointing to the immutable entity 4
foo(a) # a is still referring to the same entity 4
print(a) # prints what a refers to, still 4
However, if you do this
def foo(l): # here, l refers to the list it receives
l.append(5) # the list is appended with the number 5
a = [] # a now is pointing to a specific mutable list
foo(a) # a is still referring to the same specific mutable list
print(a) # prints what a refers to, the specific mutable list which now contains [5]
There is disagreement on terminology here. In the Java community, they say that everything is passed by value: primitives are passed by value; references are passed by value. (Just search this site for Java and pass by reference if you don't believe this.) Note that "objects" are not values in the language; only references to objects are.
The distinction that they use is that, in Java, when you pass a reference, the original reference variable in the caller's scope can never be changed (i.e. made to point to a different object) by the callee, which should be possible in pass by reference. Only the object pointed to by the reference may be mutated, but that is irrelevant.
Python values work the exact same way as references in Java. If we use the same definition, then we would say that everything in Python is a reference, and everything is passed by value. Of course, some in the Python community use a different definition.
The disagreement on terminology is the source of most of the confusion.
Since you mention C++, the Python code you have would be equivalent to something like this in C++:
void foo(const int *num) {
num = new int(*num * 2);
}
const int *a = new int(4);
foo(a);
print(a);
Note that the argument is a pointer, which is most similar to references in Java and Python.
In response to your edit, it is because integers are immutable in Python. So a is not changed for the same reason it is not changed when running this code:
a = 4
num = a
num *= 2
print(a)
You aren't changing num (and therefore a) in place, you are creating a new number and assigning it to num.
Arguments are actually passed by value. The function is passed the object the variable refers to, not the variable itself. A function cannot rebind a caller's variables. A function cannot change an immutable object, but can change (request changes to) a mutable one.
Everything is passed by reference. Everything is an object, too.
This is not really about the function call semantics but the assignment semantics. In Python assignment is done by rebinding the reference, not by overwriting the original object. This is why the example code prints 4 instead of 8 - it has nothing to do with mutability of objects as such, more that the *= operator is not a mutator but a multiplication followed by an assignment. Here the num *= 2 is essentially rebinding the 'num' name in that function to a new object of value 'num * 2'. The original value you passed in is left unaltered throughout.