What is typically regarded as more Pythonic/better/faster to use, the reverse method or the reversed built-in function?
Both in action:
_list = list(xrange(4))
print _list
rlist = list(reversed(_list))
print rlist
_list.reverse()
print _list
foo.reverse() actually reverses the elements in the container. reversed() doesn't actually reverse anything, it merely returns an object that can be used to iterate over the container's elements in reverse order. If that's what you need, it's often faster than actually reversing the elements.
There seems to be a great difference. I really thought it's the other way round.
Why is rearranging the values in a list faster than creating a new one from an iterator ?
from decorators import bench
_list = range(10 ** 6)
# bench
def foo():
list(reversed(_list))
# bench
def bar():
_list.reverse()
foo()
bar()
print foo.time
print bar.time
0.167278051376
0.0122621059418
Depends on whether you want to reverse the list in-place (i.e. change the list) or not. No other real difference.
Often using reversed leads to nicer code.
Without knowing real stats about performance, _list.reverse() modifies the list itself, whereas reversed(_list) returns an iterator ready to traverse the list in reversed order. That's a big difference itself.
If that's not a problem, object.reverse() seems more readable to me, but maybe you have specific speed requirements. And if reverse() does not belong to 80% of software that's consuming resources, I wouldn't bother (as a general rule of thumb).
_list.reverse() does an in-place reversal and does not return a value
reversed(_list) does not change _list, but returns a reverse iterable
object
_list[::-1] does not change _list, but returns reversed slice
example:
_list = [1,2,3]
ret1 = list(reversed(_list))
ret2 = _list[::-1] #reverse order slice
ret3 = _list.reverse() #no value set in ret3
print('ret1,ret2,ret3,_list:',ret1,ret2,ret3,_list)
_list = [1,2,3]
for x in reversed(_list):
print(x)
output:
ret1,ret2,ret3,_list: [3, 2, 1] [3, 2, 1] None [3, 2, 1]
3
2
1
It is always better to use reversed() if eventually you are going to modify the list in iterator making list immutable and working with immutable data is always better especially when your doing functional programming.
Expanding on #Niklas R answer:
import timeit
print('list.reverse() - real-list', timeit.timeit('_list.reverse()', '_list = list(range(1_000))'))
print('list.reverse() - iterator', timeit.timeit('_list = range(1_000); list(_list).reverse()')) # can't really use .reverse() since you need to cast it first
print('reversed() - real-list', timeit.timeit('list(reversed(_list))', '_list = list(range(1_000))'))
print('reversed() - iterator', timeit.timeit('_list = range(1_000); list(reversed(_list))'))
print('list-comprehension - real-list', timeit.timeit('_list[::-1]', '_list = list(range(1_000))'))
print('list-comprehension - iterator', timeit.timeit('_list = range(1_000); _list[::-1]'))
Results:
list.reverse() - real-list 0.29828099999576807
list.reverse() - iterator 11.078685999964364 # can't really use .reverse() since you need to cast it first
reversed() - real-list 3.7131450000451878
reversed() - iterator 12.048991999938153
list-comprehension - real-list 2.2268580000381917
list-comprehension - iterator 0.4313809999730438
(less is better/faster)
the built-in function reversed(seq) that will return a reverse iterator which is an object representing the stream of data that will return successive items of this steam. Generating this reverse iterator is O(1) in time/space complexity and using it to iterate through the elements of a list will be O(N), O(1) in time/space complexity where N is the length of the list. This is the approach you want to go for if you simply want to iterate on the reversed list without modifying it. This approach gives the best performances in this case.
_list.reverse() reverses the list in-place. The operation is O(N), O(1) in time/space complexity as it will have to go through half of the elements of the list to reverse them and it doesn't store the results in a new list. This approach is good if you do NOT need to keep the original list and you have several passes/operations to do on the elements of the reversed list and if you have to save the processed reversed list.
Last but not least, using slicing [::-1] creates a new object of the list/a copy in reversed order. The operation is O(N), O(N) in space/time complexity as you have to copy all the elements of the list in the new one and this new list will also consume the same amount of space as the original list. This approach is great if you need to keep the original list and have a reversed copy of it stored in a different object for further processing.
To summarize, depending on your use case you will have to use one of those 3 approaches that have slightly different objectives and completely different performances.
Related
What is the worst case time complexity of the following code:
temp_lst = [(1, "one"), (2, "two"), (3, "three")]
if 5 not in [i[0] for i in temp_lst]:
print("5 is not here")
My understanding is that it's O(n^2) because you're both building a list while searching this list, so equivalent would be some sort of for loop inside another for loop.
Assuming n means the length of temp_lst, then this code has a time complexity of O(n).
The list comprehension [i[0] for i in temp_list] is equivalent to the following loop, which is clearly O(n):
result = []
for i in temp_lst:
result.append(i[0])
The resulting list has the same length, n, so the expression 5 not in ..., which is implemented as a linear search, also takes O(n) time.
The list comprehension and the linear search are done one after the other, so we should add, not multiply: O(n) + O(n) = O(n).
This is linear--build the lookup list once and traverse it once. O(2n) reduces to O(n).
Having said that, this appears to be a problematic design that I'd consider to be an antipattern, absent of any further information. If the tuples are indeed sequentially numbered and unique, then this structure makes far more sense as
temp_lst = ["one", "two", "three"]
Now, we can simply say if 5 < len(temp_lst) and we have O(1) lookup time. Not only that, the code is simpler and there is no redundant information. If you need 1-indexing, either add a None to the front of the list or subtract 1 from all your lookups.
If the numbers aren't sequential and would leave holes in the list, then a dict is likely the appropriate structure:
users = {"51232": "bob", "12342": "amy", "17652": "carol"}
Again, we have O(1) lookup time when searching by id, "51232" in users.
Suppose I have a list
myList = [a,b,c,d,e]
And a function
def doSomething(list):
#Does something to the list
And I want to call the function iteratively like this:
doSomething([b,c,d,e])
doSomething([a,c,d,e])
doSomething([a,b,d,e])
doSomething([a,b,c,e])
doSomething([a,b,c,d])
The first thing that comes to mind would be something like this:
for x in range(0,len(myList)):
del myList[x]
doSomething(myList)
But this doesn't really work, because each time I call del it actually deletes the element. I sort of just want to 'hide' the element each time I call the function. Is there a way to do this?
You can use itertools.combinations for this:
import itertools
for sublist in itertools.combinations([a, b, c, d, e], 4):
# 4 is the number of elements in each sublist.
# If you do not know the length of the input list, use len() - 1
doSomething(sublist)
This will make sublist a tuple. If you need it to be a list, you can call list() on it before passing it to doSomething().
If you care about the order in which the doSomething() calls are done, you will want to reverse the order of iteration so that it begins by removing the first element instead of the last element:
for sublist in reversed(list(itertools.combinations([a, b, c, d, e], 4))):
doSomething(sublist)
This is less efficient because all of the sublists must be generated up front instead of one at a time. mgilson in the comments suggests reversing the input list and then reversing each sublist, which should be more efficient but the code may be harder to read.
Normally, looping over indices is a bad idea -- but in this case, it seems that you want to remove elements at a given index (iteratively) so looping over indices actually seems appropriate for once.
You could use list.pop for this purpose, but it turns out that would be an extra O(N) operation for each turn of the loop (once to copy the list, once to remove the i'th element). We can do it differently by removing the element while we're copying...
for i in range(len(lst)):
new_list = [x for j, x in enumerate(lst) if j != i]
doSomething(new_list)
Note however that it isn't guaranteed that this will be faster than the naive approach:
for i in range(len(lst)):
new_list = lst[:] # lst.copy() in python3.x
new_list.pop(i)
doSomething(new_list)
The naive approach has the advantage that the any indexing that needs to be done in .pop is pushed to C code which is genearally faster than doing python comparisons.
I have a list of strings, and calling a function on each string which returns a string. The thing I want is to update the string in the list. How can I do that?
for i in list:
func(i)
The function func() returns a string. i want to update the list with this string. How can it be done?
If you need to update your list in place (not create a new list to replace it), you'll need to get indexes that corresponds to each item you get from your loop. The easiest way to do that is to use the built-in enumerate function:
for index, item in enumerate(lst):
lst[index] = func(item)
You can reconstruct the list with list comprehension like this
list_of_strings = [func(str_obj) for str_obj in list_of_strings]
Or, you can use the builtin map function like this
list_of_strings = map(func, list_of_strings)
Note : If you are using Python 3.x, then you need to convert the map object to a list, explicitly, like this
list_of_strings = list(map(func, list_of_strings))
Note 1: You don't have to worry about the old list and its memory. When you make the variable list_of_strings refer a new list by assigning to it, the reference count of the old list reduces by 1. And when the reference count drops to 0, it will be automatically garbage collected.
First, don't call your lists list (that's the built-in list constructor).
The most Pythonic way of doing what you want is a list comprehension:
lst = [func(i) for i in lst]
or you can create a new list:
lst2 = []
for i in lst:
lst2.append(func(i))
and you can even mutate the list in place
for n, i in enumerate(lst):
lst[n] = func(i)
Note: most programmers will be confused by calling the list item i in the loop above since i is normally used as a loop index counter, I'm just using it here for consistency.
You should get used to the first version though, it's much easier to understand when you come back to the code six months from now.
Later you might also want to use a generator...
g = (func(i) for i in lst)
lst = list(g)
You can use map() to do that.
map(func, list)
I would like to extend a list while looping over it:
for idx in xrange(len(a_list)):
item = a_list[idx]
a_list.extend(fun(item))
(fun is a function that returns a list.)
Question:
Is this already the best way to do it, or is something nicer and more compact possible?
Remarks:
from matplotlib.cbook import flatten
a_list.extend(flatten(fun(item) for item in a_list))
should work but I do not want my code to depend on matplotlib.
for item in a_list:
a_list.extend(fun(item))
would be nice enough for my taste but seems to cause an infinite loop.
Context:
I have have a large number of nodes (in a dict) and some of them are special because they are on the boundary.
'a_list' contains the keys of these special/boundary nodes. Sometimes nodes are added and then every new node that is on the boundary needs to be added to 'a_list'. The new boundary nodes can be determined by the old boundary nodes (expresses here by 'fun') and every boundary node can add several new nodes.
Have you tried list comprehensions? This would work by creating a separate list in memory, then assigning it to your original list once the comprehension is complete. Basically its the same as your second example, but instead of importing a flattening function, it flattens it through stacked list comprehensions. [edit Matthias: changed + to +=]
a_list += [x for lst in [fun(item) for item in a_list] for x in lst]
EDIT: To explain what going on.
So the first thing that will happen is this part in the middle of the above code:
[fun(item) for item in a_list]
This will apply fun to every item in a_list and add it to a new list. Problem is, because fun(item) returns a list, now we have a list of lists. So we run a second (stacked) list comprehension to loop through all the lists in our new list that we just created in the original comprehension:
for lst in [fun(item) for item in a_list]
This will allow us to loop through all the lists in order. So then:
[x for lst in [fun(item) for item in a_list] for x in lst]
This means take every x (that is, every item) in every lst (all the lists we created in our original comprehension) and add it to a new list.
Hope this is clearer. If not, I'm always willing to elaborate further.
Using itertools, it can be written as:
import itertools
a_list += itertools.chain(* itertools.imap(fun, a_list))
or, if you're aiming for code golf:
a_list += sum(map(fun, a_list), [])
Alternatively, just write it out:
new_elements = map(fun, a_list) # itertools.imap in Python 2.x
for ne in new_elements:
a_list.extend(ne)
As you want to extend the list, but loop only over the original list, you can loop over a copy instead of the original:
for item in a_list[:]:
a_list.extend(fun(item))
Using generator
original_list = [1, 2]
original_list.extend((x for x in original_list[:]))
# [1, 2, 1, 2]
What is typically regarded as more Pythonic/better/faster to use, the reverse method or the reversed built-in function?
Both in action:
_list = list(xrange(4))
print _list
rlist = list(reversed(_list))
print rlist
_list.reverse()
print _list
foo.reverse() actually reverses the elements in the container. reversed() doesn't actually reverse anything, it merely returns an object that can be used to iterate over the container's elements in reverse order. If that's what you need, it's often faster than actually reversing the elements.
There seems to be a great difference. I really thought it's the other way round.
Why is rearranging the values in a list faster than creating a new one from an iterator ?
from decorators import bench
_list = range(10 ** 6)
# bench
def foo():
list(reversed(_list))
# bench
def bar():
_list.reverse()
foo()
bar()
print foo.time
print bar.time
0.167278051376
0.0122621059418
Depends on whether you want to reverse the list in-place (i.e. change the list) or not. No other real difference.
Often using reversed leads to nicer code.
Without knowing real stats about performance, _list.reverse() modifies the list itself, whereas reversed(_list) returns an iterator ready to traverse the list in reversed order. That's a big difference itself.
If that's not a problem, object.reverse() seems more readable to me, but maybe you have specific speed requirements. And if reverse() does not belong to 80% of software that's consuming resources, I wouldn't bother (as a general rule of thumb).
_list.reverse() does an in-place reversal and does not return a value
reversed(_list) does not change _list, but returns a reverse iterable
object
_list[::-1] does not change _list, but returns reversed slice
example:
_list = [1,2,3]
ret1 = list(reversed(_list))
ret2 = _list[::-1] #reverse order slice
ret3 = _list.reverse() #no value set in ret3
print('ret1,ret2,ret3,_list:',ret1,ret2,ret3,_list)
_list = [1,2,3]
for x in reversed(_list):
print(x)
output:
ret1,ret2,ret3,_list: [3, 2, 1] [3, 2, 1] None [3, 2, 1]
3
2
1
It is always better to use reversed() if eventually you are going to modify the list in iterator making list immutable and working with immutable data is always better especially when your doing functional programming.
Expanding on #Niklas R answer:
import timeit
print('list.reverse() - real-list', timeit.timeit('_list.reverse()', '_list = list(range(1_000))'))
print('list.reverse() - iterator', timeit.timeit('_list = range(1_000); list(_list).reverse()')) # can't really use .reverse() since you need to cast it first
print('reversed() - real-list', timeit.timeit('list(reversed(_list))', '_list = list(range(1_000))'))
print('reversed() - iterator', timeit.timeit('_list = range(1_000); list(reversed(_list))'))
print('list-comprehension - real-list', timeit.timeit('_list[::-1]', '_list = list(range(1_000))'))
print('list-comprehension - iterator', timeit.timeit('_list = range(1_000); _list[::-1]'))
Results:
list.reverse() - real-list 0.29828099999576807
list.reverse() - iterator 11.078685999964364 # can't really use .reverse() since you need to cast it first
reversed() - real-list 3.7131450000451878
reversed() - iterator 12.048991999938153
list-comprehension - real-list 2.2268580000381917
list-comprehension - iterator 0.4313809999730438
(less is better/faster)
the built-in function reversed(seq) that will return a reverse iterator which is an object representing the stream of data that will return successive items of this steam. Generating this reverse iterator is O(1) in time/space complexity and using it to iterate through the elements of a list will be O(N), O(1) in time/space complexity where N is the length of the list. This is the approach you want to go for if you simply want to iterate on the reversed list without modifying it. This approach gives the best performances in this case.
_list.reverse() reverses the list in-place. The operation is O(N), O(1) in time/space complexity as it will have to go through half of the elements of the list to reverse them and it doesn't store the results in a new list. This approach is good if you do NOT need to keep the original list and you have several passes/operations to do on the elements of the reversed list and if you have to save the processed reversed list.
Last but not least, using slicing [::-1] creates a new object of the list/a copy in reversed order. The operation is O(N), O(N) in space/time complexity as you have to copy all the elements of the list in the new one and this new list will also consume the same amount of space as the original list. This approach is great if you need to keep the original list and have a reversed copy of it stored in a different object for further processing.
To summarize, depending on your use case you will have to use one of those 3 approaches that have slightly different objectives and completely different performances.