This question already has answers here:
Strange result when removing item from a list while iterating over it
(8 answers)
Closed 7 years ago.
l = range(100)
for i in l:
print i,
print l.pop(0),
print l.pop(0)
The above python code gives the output quite different from expected. I want to loop over items so that I can skip an item while looping.
Please explain.
Never alter the container you're looping on, because iterators on that container are not going to be informed of your alterations and, as you've noticed, that's quite likely to produce a very different loop and/or an incorrect one. In normal cases, looping on a copy of the container helps, but in your case it's clear that you don't want that, as the container will be empty after 50 legs of the loop and if you then try popping again you'll get an exception.
What's anything BUT clear is, what behavior are you trying to achieve, if any?! Maybe you can express your desires with a while...?
i = 0
while i < len(some_list):
print i,
print some_list.pop(0),
print some_list.pop(0)
I've been bitten before by (someone else's) "clever" code that tries to modify a list while iterating over it. I resolved that I would never do it under any circumstance.
You can use the slice operator mylist[::3] to skip across to every third item in your list.
mylist = [i for i in range(100)]
for i in mylist[::3]:
print(i)
Other points about my example relate to new syntax in python 3.0.
I use a list comprehension to define mylist because it works in Python 3.0 (see below)
print is a function in python 3.0
Python 3.0 range() now behaves like xrange() used to behave, except it works with values of arbitrary size. The latter no longer exists.
The general rule of thumb is that you don't modify a collection/array/list while iterating over it.
Use a secondary list to store the items you want to act upon and execute that logic in a loop after your initial loop.
Use a while loop that checks for the truthfulness of the array:
while array:
value = array.pop(0)
# do some calculation here
And it should do it without any errors or funny behaviour.
Try this. It avoids mutating a thing you're iterating across, which is generally a code smell.
for i in xrange(0, 100, 3):
print i
See xrange.
I guess this is what you want:
l = range(100)
index = 0
for i in l:
print i,
try:
print l.pop(index+1),
print l.pop(index+1)
except IndexError:
pass
index += 1
It is quite handy to code when the number of item to be popped is a run time decision.
But it runs with very a bad efficiency and the code is hard to maintain.
This slice syntax makes a copy of the list and does what you want:
l = range(100)
for i in l[:]:
print i,
print l.pop(0),
print l.pop(0)
Related
I have been getting the memory error but unable to understand the reason behind that. Below is the code. Using the list and appending just two more elements in the list.
mylist = ['phonon', 'communication']
for i in mylist:
mylist.append(i.upper())
print(mylist)
It will be really very helpful if anyone can help me with that.
for i in mylist:
mylist.append(i.upper())
is basically appending to your list while iterating on it. So the iteration never ends.
You want to do:
mylist += [i.upper() for i in mylist]
in that last case, the right-hand expression is computed from a non-modified mylist, then its elements are appended to the existing mylist.
note that the above is equivalent to
mylist.extend([i.upper() for i in mylist])
or
mylist.extend(list(map(str.upper,mylist)))
note that for both snippets above it is necessary to convert to list, or you get the same memory error if the inside is lazily evaluated. The last snippet is probably the fastest because it doesn't use any python loops at all, map, and no lambda
For all 3 snippets the result is:
['phonon', 'communication', 'PHONON', 'COMMUNICATION']
I would add your edited elements to a new list:
uppers = []
for i in mylist:
uppers.append(i.upper())
print(uppers)
Building off Jean's answer:
mylist.extend(i.upper() for i in mylist)
Should give you the desired result.
When I run the following code, it prints. However, I expected only one 1 rather than two.
for i in (1,1):
print(i)
Output
1
1
You are iterating over a tuple which contains two elements with value 1 so it prints 1 twice. Your code is equivalent to:
list = [1, 1]
for item in list:
print(item)
If you want to loop over a range of numbers:
for i in range(1, 2):
print(i)
Or if you want to print unique numbers or values in list or tuple convert it into the set it will automatically remove the duplicates
newList = set(list)
for value in newList:
print(value)
Sets and tuples are different. I suspect you are confusing them. On a set:
for i in {1, 1}:
print(i)
1
On a tuple:
for i in (1, 1):
print(i)
1
1
Think of sets as being like sets in math, and tuples as being more like sequences - you can have redundancies in a sequence, but not in a set.
After reading #KeshavGarg's answer, I suspect you thought that (a,b) in Python would mean stuff in a through b. As you're probably aware by now, this is not the case - you need range to get that. Interestingly (and I admit tangentially), the syntax we're discussing here varies by language. In MATLAB, the range syntax looks a lot more like what I assume you thought the Python range syntax was:
>> for i=1:4
disp(i)
end
There has been some discussion of implementing range literals (a la Matlab) in Python. This introduces a variety of interesting new problems, which you can read about in the documentation linked in the previous sentence.
For loops are always inclusive in Python: they always run over all elements of the iterator (other than exceptions such as break, etc.). What probably has you confused is the range syntax. range(1,1) will create a range object with one element. It is the range function, not the for-loop, that is exclusive, in the sense of not including the stop argument in the range object.
This question already has answers here:
How to test multiple variables for equality against a single value?
(31 answers)
Closed 6 years ago.
import os
os.chdir('G:\\f5_automation')
r = open('G:\\f5_automation\\uat.list.cmd.txt')
#print(r.read().replace('\n', ''))
t = r.read().split('\n')
for i in range(len(t)):
if ('inherited' or 'device-group' or 'partition' or 'template' or 'traffic-group') in t[i]:
t.pop(i)
print(i,t[i])
In the above code, I get an index error at line 9: 'if ('inherited' or 'device-group'...etc.
I really don't understand why. How can my index be out of range if it's the perfect length by using len(t) as my range?
The goal is to pop any indexes from my list that contain any of those substrings. Thank you for any assistance!
This happens because you are editing the list while looping through it,
you first get the length which is 10 for example, then you loop through the thing 10 times. but as soon as you've deleted one thing the list will only be 9 long.
A way around this is to create a new list of things you want to keep and use that one instead.
I've slightly edited your code and done something similar.
t = ['inherited', 'cookies', 'device-group']
interesing_things = []
for i in t:
if i not in ['inherited', 'device-group', 'partition', 'template', 'traffic-group']:
interesing_things.append(i)
print(i)
Let's say len(t) == 5.
We'll process i taking values [0,1,2,3,4]
After we process i = 0, we pop one value from t. len(t) == 4 now. This would mean error if we get to i = 4. However, we're still going to try to go up to 4 because our range is already inited to be up to 4.
Next (i = 1) step ensures an error on i = 3.
Next (i = 2) step ensures an error on i = 2, but that is already processed.
Next (i = 3) step yields an error.
Instead, you should do something like this:
while t:
element = t.pop()
print(element)
On a side note, you should replace that in check with sets:
qualities_we_need = {'inherited', 'device-group', 'partition'} # put all your qualities here
And then in loop:
if qualities_we_need & set(element):
print(element)
If you need indexes you could either use one more variable to keep track of index of value we're currently processing, or use enumerate()
As many people said in the comments, there are several problems with your code.
The or operator sees the values on its left and right as booleans and returns the first one that is True (from left to right). So your parenthesis evaluates to 'inherited' since any non-empty string is True. As a result, even if your for loop was working, you would be popping elements that are equal to 'inherited' only.
The for loop is not working though. That happens because the size of the list you are iterating over is changing as you loop through and you will get an index-out-of-range error if an element of the list is actually equal to 'inherited' and gets popped.
So, take a look at this:
import os
os.chdir('G:\\f5_automation')
r = open('G:\\f5_automation\\uat.list.cmd.txt')
print(r.read().replace('\n', ''))
t = r.read().split('\n')
t_dupl = t[:]
for i, items in enumerate(t_dupl):
if items in ['inherited', 'device-group', 'partition', 'template', 'traffic-group']:
print(i, items)
t.remove(items)
By duplicating the original list, we can use its items as a "pool" of items to pick from and modify the list we are actually interested in.
Finally, know that the pop() method returns the item it removes from the list and this is something you do not need in your example. remove() works just fine for you.
As a side note, you can probably replace your first 5 lines of code with this:
with open('G:\\f5_automation\\uat.list.cmd.txt', 'r') as r:
t = r.readlines()
the advantage of using the with statement is that it automatically handles the closing of the file by itself when the reading is done. Finally, instead of reading the whole file and splitting it on linebreaks, you can just use the built-in readlines() method which does exactly that.
This question already has answers here:
Removing Item From List - during iteration - what's wrong with this idiom? [duplicate]
(9 answers)
Closed 4 years ago.
I want to remove all even numbers in a list. But something confused me...
This is the code.
lst = [4,4,5,5]
for i in lst:
if i % 2 == 0:
print i
lst.remove(i)
print lst
It prints [4, 5, 5] Why not [5, 5]?
It should be like this
for i in lst[:]:
if i % 2 == 0:
print i
lst.remove(i)
print lst
Problem:
You are modifying the list while you iterate over it. Due to which the iteration is stopped before it could complete
Solution:
You could iterate over copy of the list
You could use list comprehension :
lst=[i for i in lst if i%2 != 0]
By using list.remove, you are modifying the list during the iteration. This breaks the iteration giving you unexpected results.
One solution is to create a new list using either filter or a list comprehension:
>>> filter(lambda i: i % 2 != 0, lst)
[5, 5]
>>> [i for i in lst if i % 2 != 0]
[5, 5]
You can assign either expression to lst if needed, but you can't avoid creating a new list object with these methods.
Other answers have already mentioned that you're modifying the list while iterating over it, and offered better ways to do it. Personally I prefer the list comprehension method:
odd_numbers = [item for item in numbers if item % 2 != 0]
For your specified case of a very small list, I would definitely go with that.
However, this does create a new list, which could be a problem if you have a very large list. In the case of integers, large probably means millions at least, but to be precise, it's however large it needs to be to start giving you issues with memory usage. In that case, here are a couple ways to do it.
One way is similar to the intent of the code in your question. You iterate over the list, removing the even numbers as you go. However, to avoid the problems that modifying a list you're iterating over can cause, you iterate over it backwards. There are ways to iterate forward, but this is simpler.
Here's one way using a while loop:
# A one hundred million item list that we don't want to copy
# even just the odd numbers from to put into a new list.
numbers = range(100000000) # list(range(100000000)) in Python 3
index = len(numbers) - 1 # Start on the index of the last item
while index >= 0:
if numbers[index] % 2 == 0:
numbers.pop(index)
index -= 1
Here's another way using a for loop:
# A one hundred million item list that we don't want to copy
# even just the odd numbers from to put into a new list.
numbers = range(100000000) # list(range(100000000)) in Python 3
for index in xrange(len(numbers) - 1, -1, -1): # range(...) in Python 3
if numbers[index] % 2 == 0:
numbers.pop(index)
Notice in both the while loop and for loop versions, I used numbers.pop(index), not numbers.remove(numbers[index]). First of all, .pop() is much more efficient because it provides the index, whereas .remove() would have to search the list for the first occurrence of the value. Second, notice that I said, "first occurrence of the value". That means that unless every item is unique, using .remove() would remove a different item than the one the loop is currently on, which would end up leaving the current item in the list.
There's one more solution I want to mention, for situations where you need to keep the original list, but don't want to use too much more memory to store a copy of the odd numbers. If you only want to iterate over the odd numbers once (or you're so averse to using memory that you'd rather recalculate things when you need to), you can use a generator. Doing so would let you iterate over the odd numbers in the list without needing any additional memory, apart from the inconsequential amount used by the generator mechanism.
A generator expression is defined exactly like a list comprehension, except that it's enclosed in parentheses instead of square brackets:
odd_numbers = (item for item in numbers if item % 2 != 0)
Remember that the generator expression is iterating over the original list, so changing the original list mid-iteration will give you the same problems as modifying a list while iterating over it in a for loop. In fact, the generator expression is itself using a for loop.
As an aside, generator expressions shouldn't be relegated only to very large lists; I use them whenever I don't need to calculate a whole list in one go.
Summary / TLDR:
The "best" way depends exactly what you're doing, but this should cover a lot of situations.
Assuming lists are either "small" or "large":
If your list is small, use the list comprehension (or even the generator expression if you can). If it's large, read on.
If you don't need the original list, use the while loop or for loop methods to remove the even numbers entirely (though using .pop(), not .remove()). If you do need the original list, read on.
If you're only iterating over the odd numbers once, use the generator expression. If you're iterating over them more than once, but you're willing to repeat computation to save memory, use the generator expression.
If you're iterating over the odd numbers too many times to recompute them each time, or you need random access, then use a list comprehension to make a new list with only the odd numbers in them. It's going to use a lot of memory, but them's the breaks.
As a general principle, you should not modify a collection while you are iterating over it. This leads to skipping of some elements, and index error in some cases.
Instead of removing elements from list, it would be easier if you just create another reference with same name. It has lesser time complexity too.
lst = filter(lambda i: i % 2 !=0, lst)
I know that it is not allowed to remove elements while iterating a list, but is it allowed to add elements to a python list while iterating. Here is an example:
for a in myarr:
if somecond(a):
myarr.append(newObj())
I have tried this in my code and it seems to work fine, however I don't know if it's because I am just lucky and that it will break at some point in the future?
EDIT: I prefer not to copy the list since "myarr" is huge, and therefore it would be too slow. Also I need to check the appended objects with "somecond()".
EDIT: At some point "somecond(a)" will be false, so there can not be an infinite loop.
EDIT: Someone asked about the "somecond()" function. Each object in myarr has a size, and each time "somecond(a)" is true and a new object is appended to the list, the new object will have a size smaller than a. "somecond()" has an epsilon for how small objects can be and if they are too small it will return "false"
Why don't you just do it the idiomatic C way? This ought to be bullet-proof, but it won't be fast. I'm pretty sure indexing into a list in Python walks the linked list, so this is a "Shlemiel the Painter" algorithm. But I tend not to worry about optimization until it becomes clear that a particular section of code is really a problem. First make it work; then worry about making it fast, if necessary.
If you want to iterate over all the elements:
i = 0
while i < len(some_list):
more_elements = do_something_with(some_list[i])
some_list.extend(more_elements)
i += 1
If you only want to iterate over the elements that were originally in the list:
i = 0
original_len = len(some_list)
while i < original_len:
more_elements = do_something_with(some_list[i])
some_list.extend(more_elements)
i += 1
well, according to http://docs.python.org/tutorial/controlflow.html
It is not safe to modify the sequence
being iterated over in the loop (this
can only happen for mutable sequence
types, such as lists). If you need to
modify the list you are iterating over
(for example, to duplicate selected
items) you must iterate over a copy.
You could use the islice from itertools to create an iterator over a smaller portion of the list. Then you can append entries to the list without impacting the items you're iterating over:
islice(myarr, 0, len(myarr)-1)
Even better, you don't even have to iterate over all the elements. You can increment a step size.
In short: If you'are absolutely sure all new objects fail somecond() check, then your code works fine, it just wastes some time iterating the newly added objects.
Before giving a proper answer, you have to understand why it considers a bad idea to change list/dict while iterating. When using for statement, Python tries to be clever, and returns a dynamically calculated item each time. Take list as example, python remembers a index, and each time it returns l[index] to you. If you are changing l, the result l[index] can be messy.
NOTE: Here is a stackoverflow question to demonstrate this.
The worst case for adding element while iterating is infinite loop, try(or not if you can read a bug) the following in a python REPL:
import random
l = [0]
for item in l:
l.append(random.randint(1, 1000))
print item
It will print numbers non-stop until memory is used up, or killed by system/user.
Understand the internal reason, let's discuss the solutions. Here are a few:
1. make a copy of origin list
Iterating the origin list, and modify the copied one.
result = l[:]
for item in l:
if somecond(item):
result.append(Obj())
2. control when the loop ends
Instead of handling control to python, you decides how to iterate the list:
length = len(l)
for index in range(length):
if somecond(l[index]):
l.append(Obj())
Before iterating, calculate the list length, and only loop length times.
3. store added objects in a new list
Instead of modifying the origin list, store new object in a new list and concatenate them afterward.
added = [Obj() for item in l if somecond(item)]
l.extend(added)
You can do this.
bonus_rows = []
for a in myarr:
if somecond(a):
bonus_rows.append(newObj())
myarr.extend( bonus_rows )
Access your list elements directly by i. Then you can append to your list:
for i in xrange(len(myarr)):
if somecond(a[i]):
myarr.append(newObj())
make copy of your original list, iterate over it,
see the modified code below
for a in myarr[:]:
if somecond(a):
myarr.append(newObj())
I had a similar problem today. I had a list of items that needed checking; if the objects passed the check, they were added to a result list. If they didn't pass, I changed them a bit and if they might still work (size > 0 after the change), I'd add them on to the back of the list for rechecking.
I went for a solution like
items = [...what I want to check...]
result = []
while items:
recheck_items = []
for item in items:
if check(item):
result.append(item)
else:
item = change(item) # Note that this always lowers the integer size(),
# so no danger of an infinite loop
if item.size() > 0:
recheck_items.append(item)
items = recheck_items # Let the loop restart with these, if any
My list is effectively a queue, should probably have used some sort of queue. But my lists are small (like 10 items) and this works too.
You can use an index and a while loop instead of a for loop if you want the loop to also loop over the elements that is added to the list during the loop:
i = 0
while i < len(myarr):
a = myarr[i];
i = i + 1;
if somecond(a):
myarr.append(newObj())
Expanding S.Lott's answer so that new items are processed as well:
todo = myarr
done = []
while todo:
added = []
for a in todo:
if somecond(a):
added.append(newObj())
done.extend(todo)
todo = added
The final list is in done.
Alternate solution :
reduce(lambda x,newObj : x +[newObj] if somecond else x,myarr,myarr)
Assuming you are adding at the last of this list arr, You can try this method I often use,
arr = [...The list I want to work with]
current_length = len(arr)
i = 0
while i < current_length:
current_element = arr[i]
do_something(arr[i])
# Time to insert
insert_count = 1 # How many Items you are adding add the last
arr.append(item_to_be inserted)
# IMPORTANT!!!! increase the current limit and indexer
i += 1
current_length += insert_count
This is just boilerplate and if you run this, your program will freeze because of infinite loop. DO NOT FORGET TO TERMINATE THE LOOP unless you need so.