I'm wondering if there is a way to "pop all" items from a list in Python?
It can be done in a few lines of code, but the operation seems so simple I just assume there has to be a better way than making a copy and emptying the original. I've googled quite a bit and searched here, but to no avail.
I realize that popping all items will just return a copy of the original list, but that is exactly why I want to do just that. I don't want to return the list, but rather all items contained therein, while at the same time clearing it.
class ListTest():
def __init__(self):
self._internal_list = range(0, 10)
def pop_all(self):
result, self._internal_list = self._internal_list[:], []
return result
# ... instead of:
# return self._internal_list.pop_all()
t = ListTest()
print "popped: ", t.pop_all()
print "popped: ", t.pop_all()
... which of course returns the expected:
popped: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
popped: []
This is exactly how it's done. The only substantial improvement that I can think of is to empty the list in-place:
def pop_all(l):
r, l[:] = l[:], []
return r
The difference between this and your version is the behavior on a list that is referenced from various places:
>>> a = [1, 2, 3]
>>> b = a
>>> pop_all(a)
[1, 2, 3]
>>> b
[]
In fact, why not just
def pop_all(self):
result, self._internal_list = self._internal_list, []
return result
... if you are reassigning self._internal_list anyway, why not just return the old one instead of copying it?
I had a need for this just now, and decided on using a list comprehension rather than have a subclass or subroutine like this:
popped = [l.pop(0) for item in list(l)]
Related
I am trying out python, basically newbie. What I wanted to do is to save or store all generated list into one list and access that list later. The list is generated from a loop.
Basically my code is:
def doSomething():
list = []
....some parsing procedures....
......
...here needed texts are extracted...
...so I am looping to extract all texts I need...
for a in something:
list.append(a)
After the first execution, list is now populated...
then the program proceeds into the next page which is basically the same structure and then again invoke the doSomething function.
I hope this is now clear..
Assuming the first, second and third loop etc. generated this:
1st loop: [1,2,3]
2nd loop: [4,5,6]
3rd loop: [7,8,9]
I wanted to save these lists into one list and access it later so that:
alllist = [1,2,3,4,5,6,7,8,9]
How can I achieve this?
you might pass the values to another function like:
def newFunction(args, results =[]):
result.append(args)
return result
then call the function that generates list:
doSomething()
newFunction(doSomething())
if we print newFunction(doSomething()), we will see the appended lists from doSomething function
Identation is important in python. Your code does not have proper Indentation. Other programming languages use { and } to group statements whereas Python uses whitespace.
for a in something:
list.append(a)
for b in something2:
list.append(b)
However, I would recommend using something + something2 directly.
This should help:
lst = [1, 2, 3]
lst.extend([4, 5, 6])
lst.extend([7, 8, 9])
# [1, 2, 3, 4, 5, 6, 7, 8, 9]
You can store your first three lists from the for loop in variables, and then you can do a for loop for each list, and append it to a new list in order to get the output that you are seeking.
first_list = [1,2,3]
second_list = [4,5,6]
third_list = [7,8,9]
new_list = []
for number in first_list:
new_list.append(number)
for number in second_list:
new_list.append(number)
for number in third_list:
new_list.append(number)
print(new_list)
The output now in new_list is:
[1, 2, 3, 4, 5, 6, 7, 8, 9]
you may use extend method:
def doSomething(something):
list = []
for a in something:
list.append(a)
# ....this loop for some definite number...
return list
allist = []
allist.extend(doSomething(smt1))
allist.extend(doSomething(smt2))
allist.extend(doSomething(smt3))
What you really need (I think) is what other languages might call a 'static'. There are several solutions to this, including writing a class. I often use a closure for this.
In this example, the first function called, initial sets-up four attributes, a localList (don't call a variable list, it masks the list class), start, and two inner functions. References to those functions are returned (without being called) and they each have localList retained in their context.
The use of nonlocal (Python 3 required) is required to indicate the context of start.
The advantage of this is that the actual mechanism is encapsulated, and we don't have globals. If you had variables called localList, start, inner, and inner_get elsewhere in the program they would not collide.
def initial():
localList = []
start = 1
def inner():
nonlocal start
for a in range(start, start + 3):
localList.append(a)
start += 3
def inner_get():
return localList
return inner, inner_get
# get the function references
doSomething, getIt = initial()
for i in range(3):
doSomething()
# get the list
print(getIt())
# or
alllist = getIt()
Gives:
[1, 2, 3, 4, 5, 6, 7, 8, 9]
Yes its a bit complicated, but its a useful technique to get your head around if you can.
I had figure it out, I have to create a function that store all the lists during the loops:
def funcThatStoreList(args, result =[]):
result.append(args)
Observe the following code:
class permcom:
def __init__(self, INPUT_SET, IS_PERM, REPETITION):
self.end_set = []
self.input_set = INPUT_SET
self.is_perm = IS_PERM
self.repetition = REPETITION
def helpfunc(self, seen, depth, current):
if depth == 0:
self.end_set.append(seen)
else:
for i in range(0, len(self.input_set)):
if(self.repetition):
seen.append(self.input_set[i])
if(self.is_perm):
self.helpfunc(seen, depth - 1, 0)
else:
self.helpfunc(seen, depth - 1, i)
del seen[-1]
# return all permutations with repetition
def rapwr(INPUT_SET, subset_size):
instance = permcom(INPUT_SET, True, True)
A = []
instance.helpfunc(A, subset_size, 0)
return instance.end_set
A = [1,2,3]
B = rapwr(A, 2)
for i in range(0, len(B)):
print B[i]
The output is the following:
[]
[]
[]
[]
[]
[]
[]
[]
[]
However, the intended output is this:
[1, 1]
[1, 2]
[1, 3]
[2, 1]
[2, 2]
[2, 3]
[3, 1]
[3, 2]
[3, 3]
I've spent way too much time looking at this code and, unfortunately, I still cannot figure out exactly what's wrong. There must be something fundamental that I'm not understanding about how member variables work in Python, but I still don't quite understand what's going on here and why the code isn't working. Can somebody explain this?
Short answer
What you need is list slicing [:]. Changing the statement
if depth == 0:
self.end_set.append(seen)
to
if depth == 0:
self.end_set.append(seen[:])
Gives the expected answer
Long answer
Try this sample code in a python interpreter
a = [1,2]
b = []
b.append(a)
a[0] = 3
print b
# output is [[3, 2]]
Now try this code
a = [1,2]
b = []
b.append(a[:])
a[0] = 3
print b
# output is [[1, 2]]
Why did this happen? In the first case when you appended a to the list b, it was not the value of a that was appended, it was a reference/tag to the [1,2] value. You can verify this by printing id(b[0]) and id(a). Both will be the same value. Hence when you modify any value in the a list, the value in the b list also changes.
Same is the case in your code. Since you are doing del seen[-1], the corresponding value in self.end_set is also removed. You can confirm this by printing the value of self.end_set in the depth == 0 block.
To avoid this you append a clone of one list to the other list. This is done by using the splicing syntax [:]. This creates a copy of the list from the start to the end of the list. You can learn more about slicing here.
PS: Try printing the id() of the two lists when you use slicing, the values will be different
Here is what I got
a = [1,2]
b = []
b.append(a)
print id(b[0])
#output is 43337352L
print id(a)
#output is 43337352L
b = []
b.append(a[:])
print id(b[0])
#output is 43337608L
Take a look at this python memory model diagram for a better understanding of the above
Update: some advice
Since B and self.input_set are both lists, prefer using the idiomatic for i in B and for i in self.input_set.
Make sure your function names are understandable. It might help you out someday. Generally if you are made to write a comment for a variable or function name, it is better to name the function/variable with a shortened version of the comment itself. So rapwr can be renamed to return_all_permutations_with repetition. Though the name is large, its easy to understand what it does without looking at the method body now.
This question already has answers here:
How to remove items from a list while iterating?
(25 answers)
Closed 6 years ago.
I am still learning the basics of python, and I have just spent a while reading about how to remove an item from a list in python from within a for loop. Everything I've read suggests complex ways of doing this, and they say that you cannot remove an item from a list while you're iterating over it. However... this seems to work:
class Object():
def __init__(self):
self.y = 0
object_list = [Object(), Object(), Object()]
for thing in object_list:
thing.y += 1
if thing.y > 10:
object_list.remove(thing)
Why is this working when others say it isn't and write complicated workarounds? Is it because you aren't allowed to do it in Python 2 but can in Python 3?
And is this the right way to do this? Will it work as I want it or will it be prone to bugs? Would it be advisable to iterate over the list in reverse order if I plan to remove items?
Sorry if this has been answered before, but it's hard to know which resources refer to what as they all just say "python" in the tag (at least, the ones I've been reading, maybe that's because all the ones I have read are python 2?)
Thanks!
EDIT:
Sorry, there were a couple of copy and paste errors... I've fixed them...
EDIT:
I've been watching another one of Raymond Hettinger's videos... He mentions a way of removing items from a dictionary while iterating over it by using dict.keys(). Something like:
d = {'text': 'moreText', 'other': 'otherText', 'blah': 'moreBlah'}
for k in d.keys():
if k.startswith('o'):
del d[k]
Apparently using the keys makes it safe to remove the item while iterating. Is there an equivalent for lists? If there was I could iterate backwards over the list and remove items safely
Here are some examples
def example1(lst):
for item in lst:
if item < 4:
lst.remove(item)
return lst
def example2(lst):
for item in lst[:]:
if item < 4:
lst.remove(item)
return lst
def example3(lst):
i = 0
while i < len(lst):
if lst[i] < 4:
lst.pop(i)
else:
i += 1
return lst
def example4(lst):
return [item for item in lst if not item < 4]
def example5(lst):
for item in reversed(lst):
if item < 4:
lst.remove(item)
return lst
def example6(lst):
for i, item in reversed(list(enumerate(lst))):
if item < 4:
lst.pop(i)
return lst
def example7(lst):
size = len(lst) - 1
for i, item in enumerate(reversed(lst)):
if item < 4:
lst.pop(size - i)
return lst
def example8(lst):
return list(filter(lambda item: not item < 4, lst))
import itertools
def example9(lst):
return list(itertools.filterfalse(lambda item: item < 4, lst))
# Output
>>> lst = [1, 1, 2, 3, 2, 3, 4, 5, 6, 6]
>>> example1(lst[:])
[1, 3, 3, 4, 5, 6, 6]
>>> example2(lst[:])
[4, 5, 6, 6]
>>> example3(lst[:])
[4, 5, 6, 6]
>>> example4(lst[:])
[4, 5, 6, 6]
>>> example5(lst[:])
[4, 5, 6, 6]
>>> example6(lst[:])
[4, 5, 6, 6]
>>> example7(lst[:])
[4, 5, 6, 6]
>>> example8(lst[:])
[4, 5, 6, 6]
>>> example9(lst[:])
[4, 5, 6, 6]
Example 1
This example involves iterating through the list and removing values from it. The issue with this is that you are modifying the list as you go through it so your list changes during iteration and so some elements get skipped over.
Example 2
Here we are iterating over a shallow copy of the list instead of the list itself. The issue with this is if you have a large list it could be expensive to do so.
Example 3
The following is an example using pop instead of remove, the issue with remove is that it removes the first instance of the value it finds from the list. This will typically be of no issue unless you have objects which are equal. (See example 10)
Example 4
Instead of modifying the list here instead we create a new list using list comprehension allowing only specified values.
Example 5
This is an example of iterating through the list in reverse, the difference is that we use the built-in reversed function to apply a for-loop to in stead of a while loop with a counter.
Example 6
Similar example using pop instead.
Example 7
Better example using pop as we don't have to cast back to a list to use the reversed function.
Example 8
Example using the built-in filter method to remove the specified values.
Example 9
Similar example using the filerfalse method from itertools
class Example(object):
ID = 0
def __init__(self, x):
self._x = x
self._id = str(Example.ID)
Example.ID += 1
def __eq__(self, other):
return self._x == other._x
def __repr__(self):
return 'Example({})'.format(self._id)
def example10():
lst = [Example(5), Example(5)]
print(lst)
lst.remove(lst[1])
return lst
#Output
>>> example10()
[Example(0), Example(1)]
[Example(1)]
Example 10
Here we create two Example objects with the same values and by the equality method they are equal. The ID variable is there to help us differentiate between the two. Now we have specified that we want to remove the 2nd object from the list, however because both are equal the first item is actually removed instead.
Timings
These are pretty rough times and can vary slightly depending on your device. Although these identify which one is faster, this was tested with a list of 10,000 items so if you don't have anything close to that then any choice is fine really.
import timeit
import random
# Code from above is here
def test(func_name):
global test_lst
test_lst = lst[:]
return timeit.timeit("{}(test_lst)".format(func_name),
setup="from __main__ import {}, test_lst".format(func_name), number = 1)
if __name__ == '__main__':
NUM_TRIALS = 1000
lst = list(range(10000))
random.shuffle(lst) # Don't have to but makes it a bit interesting
test_list = lst[:]
for func in ('example2', 'example3', 'example4', 'example5',
'example6', 'example7', 'example8', 'example9'):
trials = []
for _ in range(NUM_TRIALS):
trials.append(test(func))
print(func, sum(trials) / len(trials) * 10000)
#Output
example2 8.487979147454494
example3 20.407155912623292
example4 5.4595031069025035
example5 7.945100572479213
example6 14.43537688078149
example7 9.088818018676008
example8 14.898256300967116
example9 13.865010859443247
It will work. However it's never a good idea to modify an object while you're iterating over it. You'll likely to get unexpected behaviour.
If I did this:
my_list = [1, 2, 3, 4]
for x in my_list:
if x+1 in my_list:
my_list.remove(x+1)
I'd expect my_list = [1] at the end. 1 remove 2, 2 removes 3, and 3 removes 4. If I check though I find my_list=[1,3]. This is because 2 was removed from the list in the first loop, so the second loop used 3 to remove 4, and 3 is still in the list.
Sorry if this is a duplicate question, I searched and couldn't find anything to help.
I'm currently trying to compare two lists. If there are any matching items I will remove them all from one of the lists.
However the results I have are buggy. Here is a rough but accurate representation of the method I'm using:
>>> i = [1,2,3,4,5,6,7,8,9]
>>> a = i
>>> c = a
>>> for b in c:
if b in i:
a.remove(b)
>>> a
[2, 4, 6, 8]
>>> c
[2, 4, 6, 8]
So I realised that the main issue is that as I remove items it shortens the list, so Python then skips over the intermediate item (seriously annoying). As a result I made a third list to act as an intermediate that can be looped over.
What really baffles me is that this list seems to change also even when I haven't directly asked it to!
In python, when you write this:
i = [1,2,3,4,5,6,7,8,9]
You create an Object (in this case, a list) and you assign it to the name i. Your next line, a = i, tells the interpreter that the name a refers to the same Object. If you want them to be separate Object you need to copy the original list. You can do that via the slicing shorthand, i[:], or you can use a = list(i) to be more explicit.
The easiest way to do this is use a set to determine shared items in a and b:
for x in set(a).intersection(b):
a.remove(x)
Your statements a = i and c = a merely make new names that reference the same object. Then as you removed things from a, it's removed from b and i, since they are the same object. You'll want to make copies of the lists instead, like so
a = i[:]
c = a[:]
a = i Doesn't make a copy of a list, it just sets another variable, i to point at your list a. Try something like this:
>>> i = [1, 2, 3, 2, 5, 6]
>>> s = []
>>> for i in t:
if i not in s:
s.append(i)
>>> s
[1, 2, 3, 5, 6]
You can also use set which guarantees no duplicates, but doesn't preserve the order:
list(set(i))
For an assignment we were asked to create a function that would reverse all the elements in an arbitrarily nested list. So inputs to the function should return something like this:
>>> seq = [1,[2,[3]]]
>>> print arb_reverse(seq)
[[[3],2],1]
>>> seq = [9,[[],[0,1,[[],[2,[[],3]]]],[],[[[4],5]]]]
>>> print arb_reverse(seq)
[[[[5,[4]]],[],[[[[3,[]],2],[]],1,0],[]],9]
I came up with a recursive solution which works well:
def arb_reverse(seq):
result = []
for element in reversed(seq):
if not is_list(element):
result.append(element)
else:
result.append(arb_reverse(element))
return result
But for a bit of a personal challenge I wanted to create a solution without the use of recursion. One version of this attempt resulted in some curious behavior which I am not understanding. For clarification, I was NOT expecting this version to work properly but the resulting input mutation does not make sense. Here is the iterative version in question:
def arb_reverse(seq):
elements = list(seq) #so input is not mutated, also tried seq[:] just to be thorough
result = []
while elements:
item = elements.pop()
if isinstance(item, list):
item.reverse() #this operation seems to be the culprit
elements += item
else:
result.append(item)
return result
This returns a flattened semi-reversed list (somewhat expected), but the interesting part is what it does to the input (not expected)...
>>> a = [1, [2, [3]]]
>>> arb_reverse(a)
[2, 3, 1]
>>> a
[1, [[3], 2]]
>>> p = [1, [2, 3, [4, [5, 6]]]]
>>> print arb_reverse(p)
[2, 3, 4, 5, 6, 1]
>>> print p
[1, [[[6, 5], 4], 3, 2]]
I was under the impression that by passing the values contained in the input to a variable using list() or input[:] as i did with elements, that I would avoid mutating the input. However, a few print statements later revealed that the reverse method had a hand in mutating the original list. Why is that?
The list() call is making a new list with shallow-copied lists from the original.
Try this (stolen from here):
from copy import deepcopy
listB = deepcopy(listA)
Try running the following code through this tool http://people.csail.mit.edu/pgbovine/python/tutor.html
o1 = [1, 2, 3]
o2 = [4, 5, 6]
l1 = [o1, o2]
l2 = list(l1)
l2[0].reverse()
print l2
print l1
Specifically look at what happens when l2[0].reverse() is called.
You'll see that when you call list() to create a copy of the list, the lists still reference the same objects.