What would be the best way to transfer references of objects from one list to another (move objects from one list to another). For clarity, I need to remove the objects from d[1] after copying
class MyObject:
def __init__(self,v):
self.value = v
d = {1: [MyObject("obj1"),MyObject("obj2")], 2: []}
#which one?
#d[2] = [obj for obj in d[1]]
#d[2] = d[1][:]
#d[2] = d[1].copy()
#clear d[1]
#d[1] = []
for i in range(len(d[1])):
d[2].append(d[1].pop(0))
for o in d[2]:
print (o.value)
Which approach is best depends a bit on the details of the surrounding code. Does any other variable or data structure contain a reference to either of your lists? If not, you can just rebind the references in the dict (which takes O(1) time since no copying happens):
d = {1: [MyObject("obj1"),MyObject("obj2")], 2: []}
d[2] = d[1]
d[1] = []
If other references to the existing lists might exist and you want them to continue referring to the correct values (e.g. an old reference to d[1] should still reference d[1] after the changes), then you want to do a slice assignment followed by a clear (this is O(N)):
d[2][:] = d[1] # copy data
d[1].clear()
I don't think there's a good reason to use any other approach unless you have some other logic to apply (for instance, if you only want to copy some of the values and not others).
Adding a dictionary obscures the use case. Based on your examples, it's not clear if you want a copy of the object or a list referencing the same objects.
Assuming the latter, consider the simplified case. It's really as simple as assigning the list to another variable:
>>> class MyObject(object):
... def __init__(self, v):
... self.value = v
...
>>> x = [MyObject(1), MyObject(2)]
>>> y = x
>>> x[1].value
2
Now, both x and y are a list of the same referenced objects. If I change the object in one list, it will change in the other:
>>> y[1].value = 3
>>> x[1].value
3
In your use case (a dictionary with list values), this is quite simple:
d[2] = d[1]
You can then delete the 1 key if necessary:
del d[1]
Voila!
Related
This question already has answers here:
How do I initialize a dictionary of empty lists in Python?
(7 answers)
Closed 2 years ago.
I came across this behavior that surprised me in Python 2.6 and 3.2:
>>> xs = dict.fromkeys(range(2), [])
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: [1]}
However, dict comprehensions in 3.2 show a more polite demeanor:
>>> xs = {i:[] for i in range(2)}
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: []}
>>>
Why does fromkeys behave like that?
Your Python 2.6 example is equivalent to the following, which may help to clarify:
>>> a = []
>>> xs = dict.fromkeys(range(2), a)
Each entry in the resulting dictionary will have a reference to the same object. The effects of mutating that object will be visible through every dict entry, as you've seen, because it's one object.
>>> xs[0] is a and xs[1] is a
True
Use a dict comprehension, or if you're stuck on Python 2.6 or older and you don't have dictionary comprehensions, you can get the dict comprehension behavior by using dict() with a generator expression:
xs = dict((i, []) for i in range(2))
In the first version, you use the same empty list object as the value for both keys, so if you change one, you change the other, too.
Look at this:
>>> empty = []
>>> d = dict.fromkeys(range(2), empty)
>>> d
{0: [], 1: []}
>>> empty.append(1) # same as d[0].append(1) because d[0] references empty!
>>> d
{0: [1], 1: [1]}
In the second version, a new empty list object is created in every iteration of the dict comprehension, so both are independent from each other.
As to "why" fromkeys() works like that - well, it would be surprising if it didn't work like that. fromkeys(iterable, value) constructs a new dict with keys from iterable that all have the value value. If that value is a mutable object, and you change that object, what else could you reasonably expect to happen?
To answer the actual question being asked: fromkeys behaves like that because there is no other reasonable choice. It is not reasonable (or even possible) to have fromkeys decide whether or not your argument is mutable and make new copies every time. In some cases it doesn't make sense, and in others it's just impossible.
The second argument you pass in is therefore just a reference, and is copied as such. An assignment of [] in Python means "a single reference to a new list", not "make a new list every time I access this variable". The alternative would be to pass in a function that generates new instances, which is the functionality that dict comprehensions supply for you.
Here are some options for creating multiple actual copies of a mutable container:
As you mention in the question, dict comprehensions allow you to execute an arbitrary statement for each element:
d = {k: [] for k in range(2)}
The important thing here is that this is equivalent to putting the assignment k = [] in a for loop. Each iteration creates a new list and assigns it to a value.
Use the form of the dict constructor suggested by #Andrew Clark:
d = dict((k, []) for k in range(2))
This creates a generator which again makes the assignment of a new list to each key-value pair when it is executed.
Use a collections.defaultdict instead of a regular dict:
d = collections.defaultdict(list)
This option is a little different from the others. Instead of creating the new list references up front, defaultdict will call list every time you access a key that's not already there. You can there fore add the keys as lazily as you want, which can be very convenient sometimes:
for k in range(2):
d[k].append(42)
Since you've set up the factory for new elements, this will actually behave exactly as you expected fromkeys to behave in the original question.
Use dict.setdefault when you access potentially new keys. This does something similar to what defaultdict does, but it has the advantage of being more controlled, in the sense that only the access you want to create new keys actually creates them:
d = {}
for k in range(2):
d.setdefault(k, []).append(42)
The disadvantage is that a new empty list object gets created every time you call the function, even if it never gets assigned to a value. This is not a huge problem, but it could add up if you call it frequently and/or your container is not as simple as list.
This question already has answers here:
How do I initialize a dictionary of empty lists in Python?
(7 answers)
Closed 2 years ago.
I came across this behavior that surprised me in Python 2.6 and 3.2:
>>> xs = dict.fromkeys(range(2), [])
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: [1]}
However, dict comprehensions in 3.2 show a more polite demeanor:
>>> xs = {i:[] for i in range(2)}
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: []}
>>>
Why does fromkeys behave like that?
Your Python 2.6 example is equivalent to the following, which may help to clarify:
>>> a = []
>>> xs = dict.fromkeys(range(2), a)
Each entry in the resulting dictionary will have a reference to the same object. The effects of mutating that object will be visible through every dict entry, as you've seen, because it's one object.
>>> xs[0] is a and xs[1] is a
True
Use a dict comprehension, or if you're stuck on Python 2.6 or older and you don't have dictionary comprehensions, you can get the dict comprehension behavior by using dict() with a generator expression:
xs = dict((i, []) for i in range(2))
In the first version, you use the same empty list object as the value for both keys, so if you change one, you change the other, too.
Look at this:
>>> empty = []
>>> d = dict.fromkeys(range(2), empty)
>>> d
{0: [], 1: []}
>>> empty.append(1) # same as d[0].append(1) because d[0] references empty!
>>> d
{0: [1], 1: [1]}
In the second version, a new empty list object is created in every iteration of the dict comprehension, so both are independent from each other.
As to "why" fromkeys() works like that - well, it would be surprising if it didn't work like that. fromkeys(iterable, value) constructs a new dict with keys from iterable that all have the value value. If that value is a mutable object, and you change that object, what else could you reasonably expect to happen?
To answer the actual question being asked: fromkeys behaves like that because there is no other reasonable choice. It is not reasonable (or even possible) to have fromkeys decide whether or not your argument is mutable and make new copies every time. In some cases it doesn't make sense, and in others it's just impossible.
The second argument you pass in is therefore just a reference, and is copied as such. An assignment of [] in Python means "a single reference to a new list", not "make a new list every time I access this variable". The alternative would be to pass in a function that generates new instances, which is the functionality that dict comprehensions supply for you.
Here are some options for creating multiple actual copies of a mutable container:
As you mention in the question, dict comprehensions allow you to execute an arbitrary statement for each element:
d = {k: [] for k in range(2)}
The important thing here is that this is equivalent to putting the assignment k = [] in a for loop. Each iteration creates a new list and assigns it to a value.
Use the form of the dict constructor suggested by #Andrew Clark:
d = dict((k, []) for k in range(2))
This creates a generator which again makes the assignment of a new list to each key-value pair when it is executed.
Use a collections.defaultdict instead of a regular dict:
d = collections.defaultdict(list)
This option is a little different from the others. Instead of creating the new list references up front, defaultdict will call list every time you access a key that's not already there. You can there fore add the keys as lazily as you want, which can be very convenient sometimes:
for k in range(2):
d[k].append(42)
Since you've set up the factory for new elements, this will actually behave exactly as you expected fromkeys to behave in the original question.
Use dict.setdefault when you access potentially new keys. This does something similar to what defaultdict does, but it has the advantage of being more controlled, in the sense that only the access you want to create new keys actually creates them:
d = {}
for k in range(2):
d.setdefault(k, []).append(42)
The disadvantage is that a new empty list object gets created every time you call the function, even if it never gets assigned to a value. This is not a huge problem, but it could add up if you call it frequently and/or your container is not as simple as list.
I create many object then I store in a list. But I want to delete them after some time because I create news one and don't want my memory goes high (in my case, it jumps to 20 gigs of ram if I don't delete it).
Here is a little code to illustrate what I trying to do:
class test:
def __init__(self):
self.a = "Hello World"
def kill(self):
del self
a = test()
b = test()
c = [a,b]
print("1)Before:",a,b)
for i in c:
del i
for i in c:
i.kill()
print("2)After:",a,b)
A and B are my objects. C is a list of these two objects. I'm trying to delete it definitely with a for-loop in C: one time with DEL and other time with a function. It's not seem to work because the print continue to show the objects.
I need this because I create 100 000 objects many times. The first time I create 100k object, the second time another 100k but I don't need to keep the previous 100k. If I don't delete them, the memory usage goes really high, very quickly.
tl;dr;
mylist.clear() # Added in Python 3.3
del mylist[:]
are probably the best ways to do this. The rest of this answer tries to explain why some of your other efforts didn't work.
cpython at least works on reference counting to determine when objects will be deleted. Here you have multiple references to the same objects. a refers to the same object that c[0] references. When you loop over c (for i in c:), at some point i also refers to that same object. the del keyword removes a single reference, so:
for i in c:
del i
creates a reference to an object in c and then deletes that reference -- but the object still has other references (one stored in c for example) so it will persist.
In the same way:
def kill(self):
del self
only deletes a reference to the object in that method. One way to remove all the references from a list is to use slice assignment:
mylist = list(range(10000))
mylist[:] = []
print(mylist)
Apparently you can also delete the slice to remove objects in place:
del mylist[:] #This will implicitly call the `__delslice__` or `__delitem__` method.
This will remove all the references from mylist and also remove the references from anything that refers to mylist. Compared that to simply deleting the list -- e.g.
mylist = list(range(10000))
b = mylist
del mylist
#here we didn't get all the references to the objects we created ...
print(b) #[0, 1, 2, 3, 4, ...]
Finally, more recent python revisions have added a clear method which does the same thing that del mylist[:] does.
mylist = [1, 2, 3]
mylist.clear()
print(mylist)
Here's how you delete every item from a list.
del c[:]
Here's how you delete the first two items from a list.
del c[:2]
Here's how you delete a single item from a list (a in your case), assuming c is a list.
del c[0]
If the goal is to delete the objects a and b themselves (which appears to be the case), forming the list [a, b] is not helpful. Instead, one should keep a list of strings used as the names of those objects. These allow one to delete the objects in a loop, by accessing the globals() dictionary.
c = ['a', 'b']
# create and work with a and b
for i in c:
del globals()[i]
To delete all objects in a list, you can directly write list = []
Here is example:
>>> a = [1, 2, 3]
>>> a
[1, 2, 3]
>>> a = []
>>> a
[]
This question already has answers here:
How do I initialize a dictionary of empty lists in Python?
(7 answers)
Closed 2 years ago.
I came across this behavior that surprised me in Python 2.6 and 3.2:
>>> xs = dict.fromkeys(range(2), [])
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: [1]}
However, dict comprehensions in 3.2 show a more polite demeanor:
>>> xs = {i:[] for i in range(2)}
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: []}
>>>
Why does fromkeys behave like that?
Your Python 2.6 example is equivalent to the following, which may help to clarify:
>>> a = []
>>> xs = dict.fromkeys(range(2), a)
Each entry in the resulting dictionary will have a reference to the same object. The effects of mutating that object will be visible through every dict entry, as you've seen, because it's one object.
>>> xs[0] is a and xs[1] is a
True
Use a dict comprehension, or if you're stuck on Python 2.6 or older and you don't have dictionary comprehensions, you can get the dict comprehension behavior by using dict() with a generator expression:
xs = dict((i, []) for i in range(2))
In the first version, you use the same empty list object as the value for both keys, so if you change one, you change the other, too.
Look at this:
>>> empty = []
>>> d = dict.fromkeys(range(2), empty)
>>> d
{0: [], 1: []}
>>> empty.append(1) # same as d[0].append(1) because d[0] references empty!
>>> d
{0: [1], 1: [1]}
In the second version, a new empty list object is created in every iteration of the dict comprehension, so both are independent from each other.
As to "why" fromkeys() works like that - well, it would be surprising if it didn't work like that. fromkeys(iterable, value) constructs a new dict with keys from iterable that all have the value value. If that value is a mutable object, and you change that object, what else could you reasonably expect to happen?
To answer the actual question being asked: fromkeys behaves like that because there is no other reasonable choice. It is not reasonable (or even possible) to have fromkeys decide whether or not your argument is mutable and make new copies every time. In some cases it doesn't make sense, and in others it's just impossible.
The second argument you pass in is therefore just a reference, and is copied as such. An assignment of [] in Python means "a single reference to a new list", not "make a new list every time I access this variable". The alternative would be to pass in a function that generates new instances, which is the functionality that dict comprehensions supply for you.
Here are some options for creating multiple actual copies of a mutable container:
As you mention in the question, dict comprehensions allow you to execute an arbitrary statement for each element:
d = {k: [] for k in range(2)}
The important thing here is that this is equivalent to putting the assignment k = [] in a for loop. Each iteration creates a new list and assigns it to a value.
Use the form of the dict constructor suggested by #Andrew Clark:
d = dict((k, []) for k in range(2))
This creates a generator which again makes the assignment of a new list to each key-value pair when it is executed.
Use a collections.defaultdict instead of a regular dict:
d = collections.defaultdict(list)
This option is a little different from the others. Instead of creating the new list references up front, defaultdict will call list every time you access a key that's not already there. You can there fore add the keys as lazily as you want, which can be very convenient sometimes:
for k in range(2):
d[k].append(42)
Since you've set up the factory for new elements, this will actually behave exactly as you expected fromkeys to behave in the original question.
Use dict.setdefault when you access potentially new keys. This does something similar to what defaultdict does, but it has the advantage of being more controlled, in the sense that only the access you want to create new keys actually creates them:
d = {}
for k in range(2):
d.setdefault(k, []).append(42)
The disadvantage is that a new empty list object gets created every time you call the function, even if it never gets assigned to a value. This is not a huge problem, but it could add up if you call it frequently and/or your container is not as simple as list.
I occasionally see the list slice syntax used in Python code like this:
newList = oldList[:]
Surely this is just the same as:
newList = oldList
Or am I missing something?
[:] Shallow copies the list, making a copy of the list structure containing references to the original list members. This means that operations on the copy do not affect the structure of the original. However, if you do something to the list members, both lists still refer to them, so the updates will show up if the members are accessed through the original.
A Deep Copy would make copies of all the list members as well.
The code snippet below shows a shallow copy in action.
# ================================================================
# === ShallowCopy.py =============================================
# ================================================================
#
class Foo:
def __init__(self, data):
self._data = data
aa = Foo ('aaa')
bb = Foo ('bbb')
# The initial list has two elements containing 'aaa' and 'bbb'
OldList = [aa,bb]
print OldList[0]._data
# The shallow copy makes a new list pointing to the old elements
NewList = OldList[:]
print NewList[0]._data
# Updating one of the elements through the new list sees the
# change reflected when you access that element through the
# old list.
NewList[0]._data = 'xxx'
print OldList[0]._data
# Updating the new list to point to something new is not reflected
# in the old list.
NewList[0] = Foo ('ccc')
print NewList[0]._data
print OldList[0]._data
Running it in a python shell gives the following transcript. We can see the
list being made with copies of the old objects. One of the objects can have
its state updated by reference through the old list, and the updates can be
seen when the object is accessed through the old list. Finally, changing a
reference in the new list can be seen to not reflect in the old list, as the
new list is now referring to a different object.
>>> # ================================================================
... # === ShallowCopy.py =============================================
... # ================================================================
... #
... class Foo:
... def __init__(self, data):
... self._data = data
...
>>> aa = Foo ('aaa')
>>> bb = Foo ('bbb')
>>>
>>> # The initial list has two elements containing 'aaa' and 'bbb'
... OldList = [aa,bb]
>>> print OldList[0]._data
aaa
>>>
>>> # The shallow copy makes a new list pointing to the old elements
... NewList = OldList[:]
>>> print NewList[0]._data
aaa
>>>
>>> # Updating one of the elements through the new list sees the
... # change reflected when you access that element through the
... # old list.
... NewList[0]._data = 'xxx'
>>> print OldList[0]._data
xxx
>>>
>>> # Updating the new list to point to something new is not reflected
... # in the old list.
... NewList[0] = Foo ('ccc')
>>> print NewList[0]._data
ccc
>>> print OldList[0]._data
xxx
Like NXC said, Python variable names actually point to an object, and not a specific spot in memory.
newList = oldList would create two different variables that point to the same object, therefore, changing oldList would also change newList.
However, when you do newList = oldList[:], it "slices" the list, and creates a new list. The default values for [:] are 0 and the end of the list, so it copies everything. Therefore, it creates a new list with all the data contained in the first one, but both can be altered without changing the other.
As it has already been answered, I'll simply add a simple demonstration:
>>> a = [1, 2, 3, 4]
>>> b = a
>>> c = a[:]
>>> b[2] = 10
>>> c[3] = 20
>>> a
[1, 2, 10, 4]
>>> b
[1, 2, 10, 4]
>>> c
[1, 2, 3, 20]
Never think that 'a = b' in Python means 'copy b to a'. If there are variables on both sides, you can't really know that. Instead, think of it as 'give b the additional name a'.
If b is an immutable object (like a number, tuple or a string), then yes, the effect is that you get a copy. But that's because when you deal with immutables (which maybe should have been called read only, unchangeable or WORM) you always get a copy, by definition.
If b is a mutable, you always have to do something extra to be sure you have a true copy. Always. With lists, it's as simple as a slice: a = b[:].
Mutability is also the reason that this:
def myfunction(mylist=[]):
pass
... doesn't quite do what you think it does.
If you're from a C-background: what's left of the '=' is a pointer, always. All variables are pointers, always. If you put variables in a list: a = [b, c], you've put pointers to the values pointed to by b and c in a list pointed to by a. If you then set a[0] = d, the pointer in position 0 is now pointing to whatever d points to.
See also the copy-module: http://docs.python.org/library/copy.html
Shallow Copy: (copies chunks of memory from one location to another)
a = ['one','two','three']
b = a[:]
b[1] = 2
print id(a), a #Output: 1077248300 ['one', 'two', 'three']
print id(b), b #Output: 1077248908 ['one', 2, 'three']
Deep Copy: (Copies object reference)
a = ['one','two','three']
b = a
b[1] = 2
print id(a), a #Output: 1077248300 ['one', 2, 'three']
print id(b), b #Output: 1077248300 ['one', 2, 'three']