I create many object then I store in a list. But I want to delete them after some time because I create news one and don't want my memory goes high (in my case, it jumps to 20 gigs of ram if I don't delete it).
Here is a little code to illustrate what I trying to do:
class test:
def __init__(self):
self.a = "Hello World"
def kill(self):
del self
a = test()
b = test()
c = [a,b]
print("1)Before:",a,b)
for i in c:
del i
for i in c:
i.kill()
print("2)After:",a,b)
A and B are my objects. C is a list of these two objects. I'm trying to delete it definitely with a for-loop in C: one time with DEL and other time with a function. It's not seem to work because the print continue to show the objects.
I need this because I create 100 000 objects many times. The first time I create 100k object, the second time another 100k but I don't need to keep the previous 100k. If I don't delete them, the memory usage goes really high, very quickly.
tl;dr;
mylist.clear() # Added in Python 3.3
del mylist[:]
are probably the best ways to do this. The rest of this answer tries to explain why some of your other efforts didn't work.
cpython at least works on reference counting to determine when objects will be deleted. Here you have multiple references to the same objects. a refers to the same object that c[0] references. When you loop over c (for i in c:), at some point i also refers to that same object. the del keyword removes a single reference, so:
for i in c:
del i
creates a reference to an object in c and then deletes that reference -- but the object still has other references (one stored in c for example) so it will persist.
In the same way:
def kill(self):
del self
only deletes a reference to the object in that method. One way to remove all the references from a list is to use slice assignment:
mylist = list(range(10000))
mylist[:] = []
print(mylist)
Apparently you can also delete the slice to remove objects in place:
del mylist[:] #This will implicitly call the `__delslice__` or `__delitem__` method.
This will remove all the references from mylist and also remove the references from anything that refers to mylist. Compared that to simply deleting the list -- e.g.
mylist = list(range(10000))
b = mylist
del mylist
#here we didn't get all the references to the objects we created ...
print(b) #[0, 1, 2, 3, 4, ...]
Finally, more recent python revisions have added a clear method which does the same thing that del mylist[:] does.
mylist = [1, 2, 3]
mylist.clear()
print(mylist)
Here's how you delete every item from a list.
del c[:]
Here's how you delete the first two items from a list.
del c[:2]
Here's how you delete a single item from a list (a in your case), assuming c is a list.
del c[0]
If the goal is to delete the objects a and b themselves (which appears to be the case), forming the list [a, b] is not helpful. Instead, one should keep a list of strings used as the names of those objects. These allow one to delete the objects in a loop, by accessing the globals() dictionary.
c = ['a', 'b']
# create and work with a and b
for i in c:
del globals()[i]
To delete all objects in a list, you can directly write list = []
Here is example:
>>> a = [1, 2, 3]
>>> a
[1, 2, 3]
>>> a = []
>>> a
[]
Related
def add_alpha(tuple_in):
list_out = []
for t in tuple_in:
t.append(10)
a = ([1,1], [2,2], [3,3])
b = a
print(id(a), id(b))
add_alpha(a)
print(id(a), id(b))
print(b)
The above example shows that, even though a is an immutable object, we can technically alter its elements. How is that so? Wouldn't it make more sense if a was assigned a new id, so that we are not changing b as well?
There are two separate misconceptions here.
Tuples can have mutable elements
a is immutable with respect to which lists a[0], a[1] and a[3] point to.
In other words
l = a[0]
# do something (that does not reassign a)
print(l is a[0])
will always print True.
However, you can still mutate a[0]. A list cannot know that it is in some tuple and shall now be resistant to mutations.
If you issued
l = [1, 2, 3]
t = (l, 'foo')
you would expect l.append(1) to work fine. (And it does.)
Assignment never copies data
Python names work like labels for objects in memory. Assigning another name creates a new label, nothing more. When you issue
a = Foo() # some mutable object, or with mutable elements
b = a
mutate(a) # mutate a or one of its elements
the change will be seen across all names (a and b), because there only ever was one mutable object in memory. a = b didn't copy anything.
This question already has answers here:
How do I clone a list so that it doesn't change unexpectedly after assignment?
(24 answers)
Closed 2 years ago.
This one is making me absolutely crazy, so any help would be much appreciated. I have a program where I'm iterating through a list in a function. Here's a toy model of the problem:
masterList = ["person","woman","man","camera","television"]
workingList = masterList
def removeItem (workingList):
item = workingList.pop(2)
print("Test 1:",workingList)
removeItem(workingList)
print("Test 2:", workingList)
print("Test 3:", masterList)
As expected, "Test 1" prints out the list with an item removed.
However, "Test 2" also prints out the list with an item removed. I wasn't expecting that, but no matter. That's not the real problem, but I'm sure there's something I don't understand here about variable scopes and variable shadowing.
No the real problem is "Test 3", which as you can see, is printing out the masterList, which shouldn't even be touched by the removeItem function or the pop function within it. And yet, it too is printing out the list with an item removed.
How can this be happening, and how can I prevent it?
Thanks so much!
Cheers,
Ari
Python lists are mutable objects.
m = list([1, 2, 3])
n = m
a = 5
b = a
id(a) == id(b)
# id() return "identity" of the object.
# True, Both a and b are references to the same object.
id(m) == id(n)
# True, Both m and n are references to the same object.
b = b + 2
id(a) == id(b)
# False, a new object on separate location is created, which b will point.
n.pop()
id(m) == id(n)
# True, the object is mutated, and m and n still references to the same object.
Since, python lists are mutable, m and n will still be reference to the same object after mutation. Whereas, for immutable objects like int, a new object will be created and the identifier will refer to the new object.
Gist is, in your scenario, there has been only one object since python lists are mutable.
However, if you need the original list unchanged when the new list is modified, you can use the copy() method.
new_list = original_list.copy()
The ids of new_list and original_list is different.
Learn here about mutability in detail: https://medium.com/#meghamohan/mutable-and-immutable-side-of-python-c2145cf72747.
You have to make a copy of the masterList, otherwise workingList is nothing more than a reference.
If your list does not contain other containers (which yours doesn't), then a shallow copy is sufficient. There are numerous ways to make a shallow copy but slicing is the most optimal.
masterList = ["person","woman","man","camera","television"]
workingList = masterList[:] #copy via slice
a bunch of other ways to make a shallow copy
workingList = masterList * 1
workingList = masterList.copy()
workingList = list(masterList)
workingList = [*masterList]
import copy
workingList = copy.copy(masterList)
If you have a list that does possess something that holds a reference (like other containers, classes, etc), then you need to make a deepcopy.
import copy
a = [[1, 2, 3], ['a', 'b', 'c']]
b = copy.deepcopy(a)
Looks like I figured it out. The two lists are actually the same list unless you use list.copy()
So replacing the top two lines with:
masterList = ["person","woman","man","camera","television"]
workingList = masterList.copy()
makes everything perform as expected! Well, I still can't say I understand how the variable scopes work in their entirety, but at least this solves the major problem.
Let's say we have the following list and we are creating an iterator for it:
lst = [1,2,3]
itr = iter(lst)
Next lets say we are changing our list with completely different values:
lst = ['a', 'b', 'c']
And if I we run the following loop:
for x in itr:
print x
We will get '1,2,3'. But why? As far as I understand, iterator doesn't copy all values from iterating object. At least iterator for list from three elements has the same size as a list of 100000 elements. sys.getsizeof(i) returns 64. How can iterator be so small by size and keep 'old' values of list?
The iterator itself contains a reference to the list. Since lst is rebound instead of mutated, this reference does not change.
>>> lst = [1, 2, 3]
>>> itr = iter(lst)
>>> lst[:] = ['a', 'b', 'c']
>>> for x in itr:
... print x
...
a
b
c
The iterator references a list object not a name. So reassigning the name lst to another object does not affect the iterator in anyway; names are bound to objects, and refer to objects, but the names are not the object themselves.
You can get a snoop of the object the iterator is referencing with gc.get_referents:
>>> import gc
>>> lst = [1,2,3]
>>> itr = iter(lst) # return an iterator for the list
>>> lst = ['a', 'b', 'c'] # Bind name lst to another object
>>> gc.get_referents(itr)[0]
[1, 2, 3]
As you'll notice, the iterator is still referring to the first list object.
The following reference will help you learn more about names and binding in Python:
Execution model - Naming and binding
Welcome to Python's object reference system. The variable names do not really have a deep relationship with the actual object stored in memory.
lst = [1, 2, 3]
itr = iter(lst) # iter object now points to the list pointed to by lst
print(next(itr)) # prints 1
# Both `lst` and `lst1` now refer to the same list
lst1 = lst
# `lst` now points to a new list, while `lst1` still points to the original list.
lst = ['a', 'b', 'c']
print(next(itr)) # prints 2
lst.append(4)
lst1.append(5) # here the list pointed to by `itr` is updated
for i in itr:
print(i) # prints 3, 5
TL;DR: Python variable names are just tags, that refer to some object in space.
When you call iter on the list named lst, the iterator object points to the actual object, and not the name lst.
If you can modify the original object, by calling append, extend, pop, remove, etc, the iterator's output will be affected. But when you assign a new value to lst, a new object is created (if it didn't previously exist), and lst simply starts pointing to that new object.
The garbage collector will delete the original object if no other object is pointing to it (itr is pointing to it in this case, so the original object won't be deleted yet).
http://foobarnbaz.com/2012/07/08/understanding-python-variables/
Extra:
lst1.extend([6, 7, 8])
next(itr) # raises StopIteration
This doesn't have anything to do with object referencing, the iterator just stores internally that it has iterated the complete list.
Context: I needed to randomly erase some precise element of a few lists of numbers, extracting some random indexes and saving them in a set called aleaindex (done, it properly works, thanks to some SO users' help). Now, I'd like to substitute the old lists a, b, etc with the new, eventually shorter ones newa, newb, etc. Here is the function:
def myfunction(N, other_parameters, a, b, c):
...
while (...):
aleaindex.add(random.randint(..., ...))
...
new_a = [v for i, v in enumerate(a) if i not in aleaindex]
while a: a.pop()
a = new_a[:]
...
and so on for the other lists b, c, etc.
Problem: the function seems to correctly modify them within the module (checked by printing) but then, when I print the modified lists outside the module, that is in the "main" file, lists are as they had not modified. Where am I wrong?
This line:
a=new_a[:]
overwrites the variable a with a new object. Outside the function or module, the old object is still pointed at by a (or whatever it was called there). Try:
new_a = [v for i, v in enumerate(a) if i not in aleaindex]
while a:
a.pop()
a[:] = new_a[:]
Explanation
To see this, just try the following.
>>> a = [1,2,3,4]
>>> b = a
>>> print b
[1, 2, 3, 4]
>>> a[:] = [2,3]
>>> print b
[2, 3]
>>> a = [5]
>>> print b
[2, 3]
Example in function!
If the variable is mutable (and a normal list is), this works:
>>> def f(a):
... a[0] = 2
>>> b = [3]
>>> f(b)
>>> print b
[2]
Variables are not passed by value - you can edit a mutable value.
I do not know what you are trying to do but from your snippets you are clearly lost. Your code does not make much sense and there are more more than one problem. Nonetheless, the problem you asked about - why the list is not fully changed? - seems to be related to this loop:
while a: a.pop()
a = new_a[:]
Suppose we call your function this way:
list1 = [1, 2, 3, 4, 5, 6, 7]
myfunction(N, other_parameters, list1, [], [])
What will happen is, when you call the first line, you will get a variable called list1 and it will point to a list:
When you call the function myfunction(), the function, among other things, create a variable called a which will point to the same list pointed by list1:
So far, so good. Then we get at the loop below:
while a:
a.pop()
a = new_a[:]
In the first line of it (a.pop()), you get an item out of the list. Since both variables a and list1 points to the same list, you would see the same result...
...if it were not for the next line of the loop (a = new_a[:]). In this line, you are making the a variable to point to another list:
Now, every operation you execute on a will be in this list, which is in no way related to list1. For example, you can execute a.pop() at the next iteration to get it:
However, it makes no sense at all, because the line a = new_a[:] will replace the list pointed to a again for yet another different list:
So, what is the solution? I don't know. As I have said, it is not possible (to me, at least) to make sense from your code. You have to reflect a bit more about what you are trying to do and explain it to us, with a bit more of context.
There is no function in the code you have posted. I suspect the problem is that you are not returning the new value.
Your code likely does something like:
a = "foo"
def func():
a = "bar" # uh-oh this not the same as the global a
func()
At this point global a is unchanged because the a local to func is not the same variable.
You want to do:
a = "foo"
def func():
return "bar"
a = func()
That code assigns to the a in global scope, changing it.
I have a list that I add items to as the program runs and I have created a copy of this list so I can change/modify/examine the copy WITHOUT changing the original.
This is what I've got so far:
import copy
originallist.append(stuff)
combined=''.join(originallist)
copyoriginal=copy.deepcopy(originallist)
##this didnt work as copyoriginal stayed empty [] while originallist would change.
#i also tried
copyoriginal=list(originallist)
#AND
copyoriginal=originallist[:]
#and with both cases copyoriginal remained at [] while originallist would change!!
Why does it not change?
A copy of the list will not follow changes made to the original list; python's list are completely separate from one another once a copy has been made.
What you are trying to do is not possible without custom classes.
If you want to "keep up with changes" to another list, you may want to consider using a simple reference instead of a copy:
a = [1, 2, 3]
b = a # b refers to a
a[0] = 3
print b # prints [3, 2, 3]
You can always make a copy later when you decide you want to stop tracking changes:
b = b[:]
If you want to names to refer to the same object you can do this:
a = b = []
a.append(1)
b.append(2)
print(b)
# returns [1,2]
That won't let you change one of them without affecting the other, but it will let you see changes in either.