I was working on a leetcode question and ran into a problem where I'd have to deepcopy a list. I found a solution that used type() like such:
orignallist=[1,2,3]
deepcopylist=type(orignallist)(orignallist)
Sure enough, it works and deepcopylist is a deepcopy but how on earth is this working? Python's type() documentation doesn't make any mention of this and I also don't understand how parentheses work with the second (orignallist) added in.
First off, it's not a deep copy. You've made a shallow copy, exactly equivalent to what list(orignallist) would produce (it doesn't matter, because all the values contained in your example list are immutable types, specifically int, but if they weren't, the distinction between deep and shallow copies would be important).
Second, all type(orignallist) is doing is extracting the class that the object bound to orignallist is an instance of, in this case, list. It's runtime determined, so if orignallist was actually a set, it would get set, but right here it's getting list. After that, it's nothing special, it's just constructing an instance of whatever orignallist is using orignallist as the argument to the constructor. If you want to see what it's doing, you can do it piecemeal:
>>> orignallist=[1,2,3]
>>> type_of_orignallist = type(orignallist)
>>> type_of_orignallist is list # It's just another alias to list
True
>>> type_of_orignallist(orignallist) # Since it's an alias of list, calling it makes a new list
[1, 2, 3]
In any event, the correct way to deep copy any object in Python is the copy.deepcopy routine:
>>> import copy
>>> lst_of_lst = [[]] # List with mutable element to demonstrate difference between shallow and deep copy
>>> shallow_copy = type(lst_of_lst)(lst_of_lst) # Or lst_of_lst[:], or lst_of_lst.copy(), or list(lst_of_lst)
>>> deep_copy = copy.deepcopy(lst_of_lst)
>>> lst_of_lst[0].append(1)
>>> lst_of_lst is shallow_copy # We copied the outer list structure
False
>>> lst_of_lst
[[1]]
>>> shallow_copy # Oops, shallow, not deep
[[1]]
>>> lst_of_lst[0] is shallow_copy[0] # Because we didn't copy the inner list
>>> deep_copy # Does what it says on the tin
[[]]
>>> lst_of_lst is deep_copy
False
>>> lst_of_lst[0] is deep_copy[0] # Yep, it recursively deepcopied so the inner list differs
False
Related
Could you please help me understand why does deepcopy not work for all the elements in the dictionary from the example below?
import copy
a = [{'id':1, 'list':[1,2,3], 'num':3}, {'id':2,' list':[4,5,6], 'num':65}]
b = {i['id']:copy.deepcopy(i) for i in a}
In [1]: print(id(a) == id(b))
Out[1]: False
In [2]: print(id(a[0]) == id(b[1]))
Out[2]: False
In [3]: print(id(a[0]['list']) == id(b[1]['list']))
Out[3]: False
In [4]: print(id(a[0]['num']) == id(b[1]['num']))
Out[4]: True
In particular, the values associated to the 'num' key are the same while those for the 'list' key seem to have been copied successfully with deepcopy. I'm guessing it has to do with the data type of the value being stored, could someone please point me in the right direction?
Thanks!
This has nothing to do with the dict comprehension but, as you suggested, with the data type:
>>> import copy
>>> x = 1
>>> copy.deepcopy(x) is x
True
>>> x = [1]
>>> copy.deepcopy(x) is x
False
The distinction made by #mengban is correct: you have mutable and immutable objects (that depends on the type of the object). Typical examples of immutable objects are: integers (0, 1, 2...), floats (3.14159), but also strings ("foo") and tuples ((1, 3)). Typical examples of mutable objects are: lists ([1, 2, 3]) or dictionaries ({'a': 1, 'b': 2}).
Basically, the deepcopy of an immutable object returns the object itself: no actual copy is performed (there's a little trick with tuples: I'll explain it later):
>>> x = "foo"
>>> copy.deepcopy(x) is x
True
>>> x = (1, 2)
>>> copy.deepcopy(x) is x
True
And deepcopy of mutable objects creates a new instance of the object having the same elements.
This the right behavior because when you have acquired a deep copy o2 of an object o, the contract is that this is your copy. No operation performed on o should be able to modify o2. If o is immutable, this is guaranteed for free. But if o is mutable, then you need to create a new instance, having the same content (this implies a recursive deep copy).
Now what's the matter with tuples?
>>> o = ([1], [2])
>>> copy.deepcopy(o) is o
False
Even if the tuple itself is immutable, maybe one of its elements could be mutable. If I give you a reference o2 to the value of o (ie o2 = o), you can write o2[0].append(10) and my object o is modified. Hence the deepcopy function looks for mutable objects in the tuple, and decides whether an actual copy is necessary or not.
Bonus: have a look at the deepcopy implementation. The _deepcopy_dispatch maps types to the actual copier:
_deepcopy_dispatch = d = {}
...
d[int] = _deepcopy_atomic
d[float] = _deepcopy_atomic
d[bool] = _deepcopy_atomic
...
d[str] = _deepcopy_atomic
...
d[list] = _deepcopy_list
...
d[tuple] = _deepcopy_tuple
...
d[dict] = _deepcopy_dict
...
While _deepcopy_atomic simply returns the value, _deepcopy_list, _deepcopy_tuple, _deepcopy_dict... perform usually an in-depth copy.
You can check the _deepcopy_tuple function to understand the process. Basically, deep copy every element until an actual copy is made. If a copy was made, create a new tuple of deep copies. Else return the initial tuple.
If you don't want a reference in your dictionary in your new dictionary you can do the following:
new_dictionary = json.loads(json.dumps(old_dictionary))
There is a big difference between mutable and immutable types in python.
In general, variable types in Python include lists, dictionaries, and collections. Immutable types include strings, int, float, and tuples.
Re-assigning a variable of an immutable type is actually re-creating an object of an immutable type and re-pointing the original variable to the newly created object (a new memory address is opened up), if no other variables refer to the original object (That is, the reference count is 0), the original object will be recycled.
This question already has answers here:
What is the difference between shallow copy, deepcopy and normal assignment operation?
(12 answers)
Closed 6 years ago.
Say I have a list a with some values, and I did a b = a[:]. Then modifying the contents of list b won't change list a as per what I've read. So, this means its a deep copy. But python documentation still refers to this as shallow copy. Can someone clear this for me?
To demonstrate what shallow copy means:
a = [ [1,2], [3,4,5] ]
b = a[:] # make a shallow copy
a is b # not the same object, because this is a copy
=> False
a == b # same value, because this is a copy
=> True
a[0] is b[0] # elements are the *same objects*, because this is a *shallow* copy
=> True
Changing the structure of a will not be reflected in b, because this is a copy:
a.pop()
len(a)
=> 1
len(b)
=> 2
To demonstrate the difference from a deep copy: changing an object contained in a (as opposed to a's structure) in-place, is reflected in b, because b references the same objects as a.
a[0][0] = 'XYZ'
b[0]
=> ['XYZ', 2]
From python docs
The difference between shallow and deep copying is only relevant for
compound objects (objects that contain other objects, like lists or
class instances):
A shallow copy constructs a new compound object and then (to the
extent possible) inserts references into it to the objects found in
the original. A deep copy constructs a new compound object and then,
recursively, inserts copies into it of the objects found in the
original.
Shallow copy creates a new object only for the top level/object, then copies by reference all sub-objects. Deep copy creates new object for the top object/level and for all sub-objects too.
Then modifying the contents of list "b" won't change list "a" as per what I've read.
As in, if you take out some of the contents or switch them out for other contents, it won't affect a. Mutations to the objects held in b will be visible through both lists, because both lists hold the same objects.
>>> class Mutable(object):
... def __init__(self, x):
... self.x = x
... def __repr__(self):
... return 'Mutable({})'.format(self.x)
...
>>> a = [Mutable(1), Mutable(2)]
>>> b = a[:]
>>> del b[1] # Doesn't affect a
>>> a
[Mutable(1), Mutable(2)]
>>> b[0].x = 5 # Visible through a
>>> a
[Mutable(5), Mutable(2)]
Then modifying the contents of list "b" won't change list "a" as per what I've read. So, this means its a deep copy.
No, it does not. A shallow copy differs from a deep copy in whether contained values are copied or not.
In your case, the list is copied, but the two resulting lists will contain the same objects. Adding or removing a value in one list won't affect the other list, but changes to a contained object will be reflected.
This question already has answers here:
What is the difference between shallow copy, deepcopy and normal assignment operation?
(12 answers)
Closed 8 years ago.
I was under the impression that deepcopy copied everything recursively down a tree, but I came upon a situation that seemed to go against what I previously believed.
>>> item = "hello"
>>> a["hello"] = item
>>> b = copy.deepcopy(a)
>>> id(a)
31995776
>>> id(b)
32733616 # I expected this
>>> id(a["hello"])
140651836041376
>>> id(b["hello"])
140651836041376 # I did not expect this
The id of a and b are different, which I expected, but the internal item is still the same object. Does deepcopy only copy to a certain depth? Or is this something specific to the way Python stores strings? (I got a similar result with integers as well)
deepcopy only needs to create copies of mutable objects, like lists and dictionaries. Strings and integers are immutable; they can't be changed in-place, so there's no need to explicitly create a copy, and a reference to the same object is inserted instead.
Here is a quick demo, showing the difference between lists (mutable) and tuples (immutable):
>>> import copy
>>> l = [[1, 2], (3, 4)]
>>> l2 = copy.deepcopy(l)
>>> l2[0] is l[0]
False # created new list
>>> l2[1] is l[1]
True # didn't create new tuple
I am trying to understand when are objects copied in Python.
The easy case is explicit copy, for example:
A = [ 1,2,3 ]
B = list(A) # this is a copy of A
Are there scenarios which implicitly copy an object? For example some languages using "pass by value" will copy objects used as functions arguments (I know that this is not the case in Python). Are there such examples of implicit copy in Python?
Hopefully it will better illustrate what is happening in your code:
>>> a = object()
>>> b = object()
>>> A = [a, b]
>>> B = list(A)
>>> A
[<object object at 0x1002b7090>, <object object at 0x1002b70a0>]
>>> B
[<object object at 0x1002b7090>, <object object at 0x1002b70a0>]
>>> A is B
False
What you can see is list() actually creates a new instance of a list based on A. This is why A is B returns false. However the contents of both list are identical
In practical terms, any indirect assignment of an immutable object can be considered a copy. For example:
a = "abc"
b = a
a += "def"
print b
--> "abc"
Technically, there is no copying occurring. When you add to a, you are actually creating a new object and assigning the variable name "a" to it while the variable "b" continues to reference the original string.
As far as functions go, there is no copying that occurs in passed variables; however, the same rules apply for mutable and immutable arguments. If you were to pass an immutable object, any changes would only impact the local "copy".
list() is not doing anything special in your example.
list(iterable):
Return a list whose items are the same and in the same order as
iterable‘s items. iterable may be either a sequence, a container that
supports iteration, or an iterator object. If iterable is already a
list, a copy is made and returned, similar to iterable[:]. For
instance, list('abc') returns ['a', 'b', 'c'] and list( (1, 2, 3) )
returns [1, 2, 3]. If no argument is given, returns a new empty list,
[].
I've searched around and tried a lot of stuff but I can't get this to work. I think the problem is something to do with how Python list names point to the list, rather than being the actual list, but I still can't figure it out. The situation is this (it's a list of dictionaries):
list_original = [dictionary1, dictionary2, dictionary3]
dictionary2_modified = dictionarymodifier(dictionary2) #some function that modifies the chosen element
list_modified = [i for i in list_original] #makes copy of original list
for i,n in enumerate(dictionary_original):
if i==1:
list_modified[1] = dictionary2_modified #replaces element with modified version
return list_original, list_modified
And many similar things, but I always either get two of the original list or two of the new list! I add that I'm using python 2.4 and don't have a choice in that.
Many thanks for any help
Mutable vs. Immutable
You need to know the difference between mutable and immutable elements. Namely, both dictionaries and lists in Python are mutable. Which means that if you modify it in one place, it is also modified in the other place.
In addition, the variable of mutable type (like list or dict) can contain immutable elements (eg. str), as well as the other way around: variable of immutable type (eg. tuple) can contain mutable elements (such as list or dict).
Example for mutability
So, this shows the mutability using the example of list:
>>> a = [1, 2, 3, 4]
>>> b = a
>>> a
[1, 2, 3, 4]
>>> b
[1, 2, 3, 4]
>>> a[2] = 'x'
>>> a
[1, 2, 'x', 4]
>>> b
[1, 2, 'x', 4]
How to obtain a copy of list or dict
To obtain a copy of list, you simply can do this instead:
new_list = old_list[:] # the slicing at the end just takes the whole list
In case of dict this is generally sufficient:
new_dict = old_dict.copy()
Nested lists / dicts
However, although lists / dicts that are flat or contain only mutable elements, can be copied the way I showed, to obtain a copy of more complex mutable data structures you need to do something more...
In such case very helpful may be the copy module with its deepcopy function. Documentation of copy module says more about its purpose:
Assignment statements in Python do not copy objects, they create bindings between a target and an object. For collections that are mutable or contain mutable items, a copy is sometimes needed so one can change one copy without changing the other. This module provides generic shallow and deep copy operations (explained below).
Is your dictionarymodifier actually mutating dictionary2 in place? If so, the way you build your list is irrelevant.
Simply using list_modified = list(list_original) works fine to create a shallow copy of the list, which you can then modify to your heart's content, but only if you don't modify the items in the original list (which you can't if they're immutable built-in things like numbers or strings, so beginners often mistake this for a deep copy).
If you really need to copy the list, you can use copy.deepcopy to do so.
You need to create a copy of the list.
copy=list(original)
original[0] = None