Develop Reference

var passed to a function by kwargs is not updated [duplicate]

This question already has answers here:
How do I pass a variable by reference?
(39 answers)
Closed 8 months ago.
In some languages you can pass a parameter by reference or value by using a special reserved word like ref or val. When you pass a parameter to a Python function it never alters the value of the parameter on leaving the function.The only way to do this is by using the global reserved word (or as i understand it currently).
Example 1:
k = 2
def foo (n):
n = n * n #clarity regarding comment below
square = n
return square
j = foo(k)
print j
print k
would show
>>4
>>2
showing k to be unchanged.
In this example the variable n is never changed
Example 2:
n = 0
def foo():
global n
n = n * n
return n
In this example the variable n is changed.
Is there any way in Python to call a function and tell Python that the parameter is either a value or reference parameter instead of using global?

There are essentially three kinds of 'function calls':
Pass by value
Pass by reference
Pass by object reference
Python is a PASS-BY-OBJECT-REFERENCE programming language.
Firstly, it is important to understand that a variable, and the value of the variable (the object) are two seperate things. The variable 'points to' the object. The variable is not the object. Again:
THE VARIABLE IS NOT THE OBJECT
Example: in the following line of code:
>>> x = []
[] is the empty list, x is a variable that points to the empty list, but x itself is not the empty list.
Consider the variable (x, in the above case) as a box, and 'the value' of the variable ([]) as the object inside the box.
PASS BY OBJECT REFERENCE (Case in python):
Here, "Object references are passed by value."
def append_one(li):
li.append(1)
x = [0]
append_one(x)
print x
Here, the statement x = [0] makes a variable x (box) that points towards the object [0].
On the function being called, a new box li is created. The contents of li are the SAME as the contents of the box x. Both the boxes contain the same object. That is, both the variables point to the same object in memory. Hence, any change to the object pointed at by li will also be reflected by the object pointed at by x.
In conclusion, the output of the above program will be:
[0, 1]
Note:
If the variable li is reassigned in the function, then li will point to a separate object in memory. x however, will continue pointing to the same object in memory it was pointing to earlier.
Example:
def append_one(li):
li = [0, 1]
x = [0]
append_one(x)
print x
The output of the program will be:
[0]
PASS BY REFERENCE:
The box from the calling function is passed on to the called function. Implicitly, the contents of the box (the value of the variable) is passed on to the called function. Hence, any change to the contents of the box in the called function will be reflected in the calling function.
PASS BY VALUE:
A new box is created in the called function, and copies of contents of the box from the calling function is stored into the new boxes.

You can not change an immutable object, like str or tuple, inside a function in Python, but you can do things like:
def foo(y):
y[0] = y[0]**2
x = [5]
foo(x)
print x[0] # prints 25
That is a weird way to go about it, however, unless you need to always square certain elements in an array.
Note that in Python, you can also return more than one value, making some of the use cases for pass by reference less important:
def foo(x, y):
return x**2, y**2
a = 2
b = 3
a, b = foo(a, b) # a == 4; b == 9
When you return values like that, they are being returned as a Tuple which is in turn unpacked.
edit:
Another way to think about this is that, while you can't explicitly pass variables by reference in Python, you can modify the properties of objects that were passed in. In my example (and others) you can modify members of the list that was passed in. You would not, however, be able to reassign the passed in variable entirely. For instance, see the following two pieces of code look like they might do something similar, but end up with different results:
def clear_a(x):
x = []
def clear_b(x):
while x: x.pop()
z = [1,2,3]
clear_a(z) # z will not be changed
clear_b(z) # z will be emptied

OK, I'll take a stab at this. Python passes by object reference, which is different from what you'd normally think of as "by reference" or "by value". Take this example:
def foo(x):
print x
bar = 'some value'
foo(bar)
So you're creating a string object with value 'some value' and "binding" it to a variable named bar. In C, that would be similar to bar being a pointer to 'some value'.
When you call foo(bar), you're not passing in bar itself. You're passing in bar's value: a pointer to 'some value'. At that point, there are two "pointers" to the same string object.
Now compare that to:
def foo(x):
x = 'another value'
print x
bar = 'some value'
foo(bar)
Here's where the difference lies. In the line:
x = 'another value'
you're not actually altering the contents of x. In fact, that's not even possible. Instead, you're creating a new string object with value 'another value'. That assignment operator? It isn't saying "overwrite the thing x is pointing at with the new value". It's saying "update x to point at the new object instead". After that line, there are two string objects: 'some value' (with bar pointing at it) and 'another value' (with x pointing at it).
This isn't clumsy. When you understand how it works, it's a beautifully elegant, efficient system.

Hope the following description sums it up well:
There are two things to consider here - variables and objects.
If you are passing a variable, then it's pass by value, which means the changes made to the variable within the function are local to that function and hence won't be reflected globally. This is more of a 'C' like behavior.
Example:
def changeval( myvar ):
myvar = 20;
print "values inside the function: ", myvar
return
myvar = 10;
changeval( myvar );
print "values outside the function: ", myvar
O/P:
values inside the function: 20
values outside the function: 10
If you are passing the variables packed inside a mutable object, like a list, then the changes made to the object are reflected globally as long as the object is not re-assigned.
Example:
def changelist( mylist ):
mylist2=['a'];
mylist.append(mylist2);
print "values inside the function: ", mylist
return
mylist = [1,2,3];
changelist( mylist );
print "values outside the function: ", mylist
O/P:
values inside the function: [1, 2, 3, ['a']]
values outside the function: [1, 2, 3, ['a']]
Now consider the case where the object is re-assigned. In this case, the object refers to a new memory location which is local to the function in which this happens and hence not reflected globally.
Example:
def changelist( mylist ):
mylist=['a'];
print "values inside the function: ", mylist
return
mylist = [1,2,3];
changelist( mylist );
print "values outside the function: ", mylist
O/P:
values inside the function: ['a']
values outside the function: [1, 2, 3]

Python is neither pass-by-value nor pass-by-reference. It's more of "object references are passed by value" as described here:
Here's why it's not pass-by-value. Because
def append(list):
list.append(1)
list = [0]
reassign(list)
append(list)
returns [0,1] showing that some kind of reference was clearly passed as pass-by-value does not allow a function to alter the parent scope at all.
Looks like pass-by-reference then, hu? Nope.
Here's why it's not pass-by-reference. Because
def reassign(list):
list = [0, 1]
list = [0]
reassign(list)
print list
returns [0] showing that the original reference was destroyed when list was reassigned. pass-by-reference would have returned [0,1].
For more information look here:
If you want your function to not manipulate outside scope, you need to make a copy of the input parameters that creates a new object.
from copy import copy
def append(list):
list2 = copy(list)
list2.append(1)
print list2
list = [0]
append(list)
print list

Technically python do not pass arguments by value: all by reference. But ... since python has two types of objects: immutable and mutable, here is what happens:
Immutable arguments are effectively passed by value: string, integer, tuple are all immutable object types. While they are technically "passed by reference" (like all parameters), since you can't change them in-place inside the function it looks/behaves as if it is passed by value.
Mutable arguments are effectively passed by reference: lists or dictionaries are passed by its pointers. Any in-place change inside the function like (append or del) will affect the original object.
This is how Python is designed: no copies and all are passed by reference. You can explicitly pass a copy.
def sort(array):
# do sort
return array
data = [1, 2, 3]
sort(data[:]) # here you passed a copy
Last point I would like to mention which is a function has its own scope.
def do_any_stuff_to_these_objects(a, b):
a = a * 2
del b['last_name']
number = 1 # immutable
hashmap = {'first_name' : 'john', 'last_name': 'legend'} # mutable
do_any_stuff_to_these_objects(number, hashmap)
print(number) # 1 , oh it should be 2 ! no a is changed inisde the function scope
print(hashmap) # {'first_name': 'john'}

So this is a little bit of a subtle point, because while Python only passes variables by value, every variable in Python is a reference. If you want to be able to change your values with a function call, what you need is a mutable object. For example:
l = [0]
def set_3(x):
x[0] = 3
set_3(l)
print(l[0])
In the above code, the function modifies the contents of a List object (which is mutable), and so the output is 3 instead of 0.
I write this answer only to illustrate what 'by value' means in Python. The above code is bad style, and if you really want to mutate your values you should write a class and call methods within that class, as MPX suggests.

Consider that the variable is a box and the value it points to is the "thing" inside the box:
1. Pass by reference : function shares the same box and thereby the thing inside also.
2. Pass by value : function creates a new box, a replica of the old one, including a copy of whatever thing is inside it. Eg. Java - functions create a copy of the box and the thing inside it which can be: a primitive / a reference to an object. (note that the copied reference in the new box and the original both still point to the same object, here the reference IS the thing inside the box, not the object it is pointing to)
3. Pass by object-reference: the function creates a box, but it encloses the same thing the initial box was enclosing. So in Python:
a) if the thing inside said box is mutable, changes made will reflect back in the original box (eg. lists)
b) if the thing is immutable (like python strings and numeric types), then the box inside the function will hold the same thing UNTIL you try to change its value. Once changed, the thing in the function's box is a totally new thing compared to the original one. Hence id() for that box will now give the identity of the new thing it encloses.

The answer given is
def set_4(x):
y = []
for i in x:
y.append(i)
y[0] = 4
return y
and
l = [0]
def set_3(x):
x[0] = 3
set_3(l)
print(l[0])
which is the best answer so far as it does what it says in the question. However,it does seem a very clumsy way compared to VB or Pascal.Is it the best method we have?
Not only is it clumsy, it involves mutating the original parameter in some way manually eg by changing the original parameter to a list: or copying it to another list rather than just saying: "use this parameter as a value " or "use this one as a reference". Could the simple answer be there is no reserved word for this but these are great work arounds?

class demoClass:
x = 4
y = 3
foo1 = demoClass()
foo1.x = 2
foo2 = demoClass()
foo2.y = 5
def mySquare(myObj):
myObj.x = myObj.x**2
myObj.y = myObj.y**2
print('foo1.x =', foo1.x)
print('foo1.y =', foo1.y)
print('foo2.x =', foo2.x)
print('foo2.y =', foo2.y)
mySquare(foo1)
mySquare(foo2)
print('After square:')
print('foo1.x =', foo1.x)
print('foo1.y =', foo1.y)
print('foo2.x =', foo2.x)
print('foo2.y =', foo2.y)

In Python the passing by reference or by value has to do with what are the actual objects you are passing.So,if you are passing a list for example,then you actually make this pass by reference,since the list is a mutable object.Thus,you are passing a pointer to the function and you can modify the object (list) in the function body.
When you are passing a string,this passing is done by value,so a new string object is being created and when the function terminates it is destroyed.
So it all has to do with mutable and immutable objects.

Python already call by ref..
let's take example:
def foo(var):
print(hex(id(var)))
x = 1 # any value
print(hex(id(x))) # I think the id() give the ref...
foo(x)
OutPut
0x50d43700 #with you might give another hex number deppend on your memory
0x50d43700

Sharing data among instances of a mapping function in Python

I'm writing a graph-walking script with this function that applies a mapping function given as an argument to every vertex, but can't figure out how to share the closure since Python doesn't have pointers:
def walk_vertices(graph, function, closure):
for vertex in graph.vertices:
function(vertex, closure)
return closure
This could be used, for example, to sum up the values of integer vertices, and in C-based languages, the closure would be the pointer to the running sum. What is a good way of implementing this in Python? Thanks.

Actually i think about languages as python or java as only having pointers to objects.
while function can not rebind to what object closure 'points' in walk_vertices, if the object closure 'points' to is mutable it can of course change it. In your example you talk about a sum. That would of course be an integer or floating point number. These are immutable in python, so closure would point to an object but you can't mutate it:
x = 5
def something(ref):
# you can't change where x points to from here.
# and because an int is immutable you can't change it.
ref = 10 # rebinds ref, but not x
something(x)
print(x) # still 5
But, if you pass a mutable object you can actually store information. One way to have a very simple mutable object is just to use a list of size 1.
x = [5]
def something(ref):
# you can't change where x points to from here.
ref = 5 # rebinds ref, but not x
something(x)
print(x) # still [5]
def something2(ref):
# ref is a mutable object, so
ref[0] = 10 # ref points to the same list, but contents of list is now 10
something2(x)
print(x) # now [10]
The same construction works with any mutable object. So a dict or a class are usable as well.
class EmptyClass:
pass
x = EmptyClass()
x.data = 5
def something(ref):
ref.data = 10
something(x)
print(x.data) # now prints 10
To sum it up, python always passes something comparable to pointers. But because it has some types that are immutable you can't always use that to pass data back. You have to pass a mutable object.
Also python has no equivalent of taking the pointer of a local variable. So while everything is a pointer to an object you can't get a pointer to an pointer without having an object in between like in the list case (pointer to list of pointer).
What you can do is use 'reflection' to change the value of a local variable via locals()
x = 5
def something(d):
d['x'] = 10
something(locals())
print(x) # now prints 10

It's not clear what you mean by the "closure". If you just want the callback to be able to maintain state from call to call, there are plenty of ways to do that, such as with closures in the functional programming sense:
def walk_vertices(graph, callback):
...
def f(graph):
running_sum = 0
def callback(vertex):
# Python 3 only
nonlocal running_sum
running_sum += vertex.value
walk_vertices(graph, callback)
return running_sum
def python2f(graph):
running_sum = [0]
def callback(vertex):
# awkward hack
running_sum[0] += vertex.value
walk_vertices(graph, callback)
return running_sum[0]
But I think the most natural way would be to use a generator. Instead of walk_vertices walking the graph and applying a callback to each vertices, it yields each vertex. Then the calling code can iterate over the vertices with an ordinary loop, without needing to write an awkward callback:
def walk_vertices(graph):
# Pretend generating vertices is more complicated than this:
for vertex in graph.vertices:
yield vertex
running_sum = 0
for v in walk_vertices(graph):
running_sum += v.value
# or just
vertex_sum = sum(v.value for v in walk_vertices(graph))

python: need a deepcopy equivalent breaking all shared identity

Due to some constrains I need to create a fresh copy of an object alongwith fresh copies of all its attributes and for attributes of its attributes and so on recursively.
Existing deepcopy() is recursive, but when multiple objects within the tree being copied have the same starting identity, they also have the same ending identity (even though their ending identities don't match their starting identities).
For the following case:
class A:
def __init__(self, x):
self.x = x
v = A(1)
o = [v, v]
copy.deepcopy does following:
dc_o = copy.deepcopy(o)
assert dc_o[0] is not o[0] # new identity from the original
assert dc_o[0] is dc_o[1] # but maintains identity within the copied tree
assert dc_o[0] == dc_o[1] # ...as well as value
But, what I need is:
r_dc_o = recursive_deepcopy(o)
assert r_dc_o[0] is not o[0] # new identity from the original
assert r_dc_o[0] is not r_dc_o[1] # also new identity from elsewhere inside copy
assert r_dc_o[0] == r_dc_o[1] # while maintaining the same value
How can I do this?

Fully automating a recursive deepcopy in a way that didn't memoize objects would be extremely dangerous -- it would mean you couldn't have any kind of objects with internal references preserved in a way that would make those references useful after the copy operation (think about objects with a "parent" link, or objects that link to a shared registry or similar resource). That said, if you really wanted to do this (and you shouldn't -- it will break a great many objects passed through the operation), you can accomplish it by constructing a memo dictionary that ignored attempts at adding keys, and passing that as a second argument to deepcopy().
So, here we are:
import copy
class baddict(dict):
def __setitem__(self, k, v):
pass
class A:
def __init__(self, x):
self.x = x
def __eq__(self, other):
self.x == other.x
v = A(1)
o = [v, v]
r_dc_o = copy.deepcopy(o, baddict())
assert r_dc_o[0] is not r_dc_o[1]
assert r_dc_o[0] == r_dc_o[1]
I'd suggest thinking about why you need this behavior, and trying to come up with a better way to accomplish it. Even a baddict implementation that looked at the value and skipped memoizing only if values were instances of a specific class would be safer than what we're doing here.

How to remove duplicates in set for objects?

I have set of objects:
class Test(object):
def __init__(self):
self.i = random.randint(1,10)
res = set()
for i in range(0,1000):
res.add(Test())
print len(res) = 1000
How to remove duplicates from set of objects ?
Thanks for answers, it's work:
class Test(object):
def __init__(self, i):
self.i = i
# self.i = random.randint(1,10)
# self.j = random.randint(1,20)
def __keys(self):
t = ()
for key in self.__dict__:
t = t + (self.__dict__[key],)
return t
def __eq__(self, other):
return isinstance(other, Test) and self.__keys() == other.__keys()
def __hash__(self):
return hash(self.__keys())
res = set()
res.add(Test(2))
...
res.add(Test(8))
result: [2,8,3,4,5,6,7]
but how to save order ? Sets not support order. Can i use list instead set for example ?

Your objects must be hashable (i.e. must have __eq__() and __hash__() defined) for sets to work properly with them:
class Test(object):
def __init__(self):
self.i = random.randint(1, 10)
def __eq__(self, other):
return self.i == other.i
def __hash__(self):
return self.i
An object is hashable if it has a hash value which never changes during its lifetime (it needs a __hash__() method), and can be compared to other objects (it needs an __eq__() or __cmp__() method). Hashable objects which compare equal must have the same hash value.
Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally.
If you have several attributes, hash and compare a tuple of them (thanks, delnan):
class Test(object):
def __init__(self):
self.i = random.randint(1, 10)
self.k = random.randint(1, 10)
self.j = random.randint(1, 10)
def __eq__(self, other):
return (self.i, self.k, self.j) == (other.i, other.k, other.j)
def __hash__(self):
return hash((self.i, self.k, self.j))

Your first question is already answered by Pavel Anossov.
But you have another question:
but how to save order ? Sets not support order. Can i use list instead set for example ?
You can use a list, but there are a few downsides:
You get the wrong interface.
You don't get automatic handling of duplicates. You have to explicitly write if foo not in res: res.append(foo). Obviously, you can wrap this up in a function instead of writing it repeatedly, but it's still extra work.
It's going to be a lot less efficient if the collection can get large. Basically, adding a new element, checking whether an element already exists, etc. are all going to be O(N) instead of O(1).
What you want is something that works like an ordered set. Or, equivalently, like a list that doesn't allow duplicates.
If you do all your adds first, and then all your lookups, and you don't need lookups to be fast, you can get around this by first building a list, then using unique_everseen from the itertools recipes to remove duplicates.
Or you could just keep a set and a list or elements by order (or a list plus a set of elements seen so far). But that can get a bit complicated, so you might want to wrap it up.
Ideally, you want to wrap it up in a type that has exactly the same API as set. Something like an OrderedSet akin to collections.OrderedDict.
Fortunately, if you scroll to the bottom of that docs page, you'll see that exactly what you want already exists; there's a link to an OrderedSet recipe at ActiveState.
So, copy it, paste it into your code, then just change res = set() to res = OrderedSet(), and you're done.

I think you can easily do what you want with a list as you asked in your first post since you defined the eq operator :
l = []
if Test(0) not in l :
l.append(Test(0))
My 2 cts ...

Pavel Anossov's answer is great for allowing your class to be used in a set with the semantics you want. However, if you want to preserve the order of your items, you'll need a bit more. Here's a function that de-duplicates a list, as long as the list items are hashable:
def dedupe(lst):
seen = set()
results = []
for item in lst:
if item not in seen:
seen.add(item)
results.append(item)
return results
A slightly more idiomatic version would be a generator, rather than a function that returns a list. This gets rid of the results variable, using yield rather than appending the unique values to it. I've also renamed the lst parameter to iterable, since it will work just as well on any iterable object (such as another generator).
def dedupe(iterable):
seen = set()
for item in iterable:
if item not in seen:
seen.add(item)
yield item

Python Scoping/Static Misunderstanding

I'm really stuck on why the following code block 1 result in output 1 instead of output 2?
Code block 1:
class FruitContainer:
def __init__(self,arr=[]):
self.array = arr
def addTo(self,something):
self.array.append(something)
def __str__(self):
ret = "["
for item in self.array:
ret = "%s%s," % (ret,item)
return "%s]" % ret
arrayOfFruit = ['apple', 'banana', 'pear']
arrayOfFruitContainers = []
while len(arrayOfFruit) > 0:
tempFruit = arrayOfFruit.pop(0)
tempB = FruitContainer()
tempB.addTo(tempFruit)
arrayOfFruitContainers.append(tempB)
for container in arrayOfFruitContainers:
print container
**Output 1 (actual):**
[apple,banana,pear,]
[apple,banana,pear,]
[apple,banana,pear,]
**Output 2 (desired):**
[apple,]
[banana,]
[pear,]
The goal of this code is to iterate through an array and wrap each in a parent object. This is a reduction of my actual code which adds all apples to a bag of apples and so forth. My guess is that, for some reason, it's either using the same object or acting as if the fruit container uses a static array. I have no idea how to fix this.

You should never use a mutable value (like []) for a default argument to a method. The value is computed once, and then used for every invocation. When you use an empty list as a default value, that same list is used every time the method is invoked without the argument, even as the value is modified by previous function calls.
Do this instead:
def __init__(self,arr=None):
self.array = arr or []

Your code has a default argument to initialize the class. The value of the default argument is evaluated once, at compile time, so every instance is initialized with the same list. Change it like so:
def __init__(self, arr=None):
if arr is None:
self.array = []
else:
self.array = arr
I discussed this more fully here: How to define a class in Python

As Ned says, the problem is you are using a list as a default argument. There is more detail here. The solution is to change __init__ function as below:
def __init__(self,arr=None):
if arr is not None:
self.array = arr
else:
self.array = []

A better solution than passing in None — in this particular instance, rather than in general — is to treat the arr parameter to __init__ as an enumerable set of items to pre-initialize the FruitContainer with, rather than an array to use for internal storage:
class FruitContainer:
def __init__(self, arr=()):
self.array = list(arr)
...
This will allow you to pass in other enumerable types to initialize your container, which more advanced Python users will expect to be able to do:
myFruit = ('apple', 'pear') # Pass a tuple
myFruitContainer = FruitContainer(myFruit)
myOtherFruit = file('fruitFile', 'r') # Pass a file
myOtherFruitContainer = FruitContainer(myOtherFruit)
It will also defuse another potential aliasing bug:
myFruit = ['apple', 'pear']
myFruitContainer1 = FruitContainer(myFruit)
myFruitContainer2 = FruitContainer(myFruit)
myFruitContainer1.addTo('banana')
'banana' in str(myFruitContainer2)
With all other implementations on this page, this will return True, because you have accidentally aliased the internal storage of your containers.
Note: This approach is not always the right answer: "if not None" is better in other cases. Just ask yourself: am I passing in a set of objects, or a mutable container? If the class/function I'm passing my objects in to changes the storage I gave it, would that be (a) surprising or (b) desirable? In this case, I would argue that it is (a); thus, the list(...) call is the best solution. If (b), "if not None" would be the right approach.