Develop Reference

Inefficient code for removing duplicates from a list in Python - interpretation?

I am writing a Python program to remove duplicates from a list. My code is the following:
some_values_list = [2,2,4,7,7,8]
unique_values_list = []
for i in some_values_list:
if i not in unique_values_list:
unique_values_list.append(i)
print(unique_values_list)
This code works fine. However, an alternative solution is given and I am trying to interpret it (as I am still a beginner in Python). Specifically, I do not understand the added value or benefit of creating an empty set - how does that make the code clearer or more efficient? Isn´t it enough to create an empty list as I have done in the first example?
The code for the alternative solution is the following:
a = [10,20,30,20,10,50,60,40,80,50,40]
dup_items = set()
uniq_items = []
for x in a:
if x not in dup_items:
uniq_items.append(x)
dup_items.add(x)
print(dup_items)
This code also throws up an error TypeError: set() missing 1 required positional argument: 'items' (This is from a website for Python exercises with answers key, so it is supposed to be correct.)

Determining if an item is present in a set is generally faster than determining if it is present in a list of the same size. Why? Because for a set (at least, for a hash table, which is how CPython sets are implemented) we don't need to traverse the entire collection of elements to check if a particular value is present (whereas we do for a list). Rather, we usually just need to check at most one element. A more precise way to frame this is to say that containment tests for lists take "linear time" (i.e. time proportional to the size of the list), whereas containment tests in sets take "constant time" (i.e. the runtime does not depend on the size of the set).

Lookup for an element in a list takes O(N) time (you can find an element in logarithmic time, but the list should be sorted, so not your case). So if you use the same list to keep unique elements and lookup newly added ones, your whole algorithm runs in O(N²) time (N elements, O(N) average lookup). set is a hash-set in Python, so lookup in it should take O(1) on average. Thus, if you use an auxiliary set to keep track of unique elements already found, your whole algorithm will only take O(N) time on average, chances are good, one order better.

In most cases sets are faster than lists. One of this cases is when you look for an item using "in" keyword. The reason why sets are faster is that, they implement hashtable.
So, in short, if x not in dup_items in second code snippet works faster than if i not in unique_values_list.
If you want to check the time complexity of different Python data structures and operations, you can check this link
.
I think your code is also inefficient in a way that for each item in list you are searching in larger list. The second snippet looks for the item in smaller set. But that is not correct all the time. For example, if the list is all unique items, then it is the same.
Hope it clarifies.

Questions regarding code in a Data Structures and Algorithms book

I have been refreshing my knowledge of data structures and algorithms using a book. I came across some sample code in the book, including some run time analysis values that I cannot seem to make sense of. I don't know if I am overthinking this or I am missing something extremely simple. Please help me out.
This is the case where they explain the logic behind adding an element at a specific index in a list. I get the logic, it is pretty simple. Move all the elements, starting with the rightmost one, one index to the right to make space for the element at the index. The code for this in the book is given by:
for j in range(self._n, k, −1):
self._A[j] = self._A[j−1]
What I do not get is the range of the loop. Technically, self._n is equivalent to len(list) (an internal state maintained by the list). And if you start at len(list), you are immediately at an IndexOutOfBoundError. Secondly, even if that were not true, the loop replaces n with n-1. Nowhere does it actually move n to n+1 first, so that value is lost. Am I missing something here? I actually tried these conditions out on the Python interpreter and they seem to validate my understanding.
Some of the run time analyses for list operations seems to confuse me. For example:
data.index(value) --> O(k+1)
value in data --> O(k+1)
data1 == data2 (similarly !=, <, <=, >, >=) --> O(k+1)
data[j:k] --> O(k−j+1)
I do not get why the +1 at the end of each running time analysis. Let us consider the data.index(value) operation, which basically returns the first index at which a certain value is found. At the worst case, it should iterate through all n elements of the list, but if not, if the search finds something at index k, then it returns from there. Why the O(k+1) there? The same logic applies to the other cases too, especially the list slicing. When you slice a list, isn't it just O(k-j)? On the contrary, the actual indices are j to k-1.
This understanding should be quite elementary and I really feel silly not being able to understand it. Or I don't know if there are genuine errata in the book and I understand it correctly. Could someone please clarify this for me? Help is much appreciated.

Note (from the comments): the book in question is Data Structures and Algorithms in Python by Goodrich, Tamassia and Goldwasser, and the questions are about pages 202 to 204.
If you actually look at the whole definition of insert from the book, it makes more sense.
def insert(self, k, value):
if self.n == self.capacity:
self.resize(2 * self.capacity)
for j in range(self.n, k, −1):
self.A[j] = self.A[j−1]
self.A[k] = value
self.n += 1
The first line implies that self.n is the number of elements, and corresponds to the index past-the-end, which means that, for a user of the list, accessing it at that index would be erroneous. But this code belongs to the list, and because it has a capacity in addition to a size, it can use self.A[n] if self.n < self.capacity (which is true when the for loop starts).
The loop simply moves the last element (at index n-1) to the next space in memory, which is out of bounds for a user, but not internaly. At the end, n is incremented to reflect the new size, and n-1 becomes the index of that "next space in memory", which now contains the last element.
As for the time complexity of the different operations: well, they are not incorrect. Even though O(n+1) = O(n), you can still write O(n+1) if you want to, and it might be more "precise" in some cases.
For example, it is written that data.index(value) has a complexity of O(k+1), with k the index of the value being search for. Well, if that value is at the very beginning, then k = 0 and the complexity is O(0+1) = O(1). And it's true: if you always search for a value that you know is at the very beginning, even though this operation is pointless, it has a constant time complexity. If you initially wrote O(k) instead, then you would get O(0) for that last operation, which I have never seen used, but would make me think that the operation is instantaneous.
The same thing happens for slicing: they probably wrote O(k−j+1) because if you only take one element, then j = k and the complexity is O(1) instead of O(0).
Note that time complexity isn't usually defined in terms of the actual indices of a particular application of the function, but instead in terms of the total number of elements in the container on which the function is used. You can think of it as the mean complexity for using the function with every possible index, which in the cases of index and slicing, is simply O(n).

For the first case, I think the assumption is that you have a list of fixed maximum length and you are supposed to lose the last datapoint. Also, are you certain that self._n == len(n) and not self._n == len(n)-1?
For the second case, as far as I understand, O(k+1) is the same as O(k), so it doesn't make sense to say O(k+1). But again if we really want to know how someone may count up to k+1... I would guess that guy is counting starting from 0. So to go from 0th to the kth index will take k+1 operations.
This is just an opinion, and an uniformed one, so please take it with a tablespoon of salt. I think that book is no good man.

python 2.7 set and list remove time complexity

Wondering the time complexity of remove of list, and remove of set.
My thought and study result is,
Removal of list is O(n)
Removal of set is O(1).
I just studied some discussion, but never prove it. If anyone could shed some lights, it will be great. Especially how set implements with O(1) removal?
Using Python 2.7.
a = set([1,2,3,4,5])
b = [1,2,3,4,5]
a.remove(3)
b.remove(3)
print a
print b

From the docs:
list.remove(x)
Remove the first item from the list whose value is x.
It is an error if there is no such item.
Without going into the details of the implementation, the item to remove can be anywhere in the list. A linear scan time is necessary in order to find the item before it can be removed. Once you find the index of the item to remove, you need to shift all the elements down by one index. In any case there's index amount of traversal and size - index amount of shifting involved. Therefore the removal time is equivalent to traversing the entire list: O(n).
You can find the source here: https://hg.python.org/cpython/file/tip/Objects/listobject.c#l2197 (also look for list_ass_slice(..)).
However, a set is different. It uses the hash value of the object being stored to locate it in its buckets. On an average, locating of objects using hash value is almost constant time. Note that it might not always be constant time where there's hash collision and a further search is required. But assuming a good hash function, it usually is.
UPDATE: I must thank Stefan Pochmann for pointing out the mistake.

Why l.insert(0, i) is slower than l.append(i) in python?

I tested two different ways to reverse a list in python.
import timeit
value = [i for i in range(100)]
def rev1():
v = []
for i in value:
v.append(i)
v.reverse()
def rev2():
v = []
for i in value:
v.insert(0, i)
print timeit.timeit(rev1)
print timeit.timeit(rev2)
Interestingly, the 2nd method that inserts the value to the first element is pretty much slower than the first one.
20.4851300716
73.5116429329
Why is this? In terms of operation, inserting an element to the head doesn't seem that expensive.

insert is an O(n) operation as it requires all elements at or after the insert position to be shifted up by one. append, on the other hand, is generally O(1) (and O(n) in the worst case, when more space must be allocated). This explains the substantial time difference.
The time complexities of these methods are thoroughly documented here.
I quote:
Internally, a list is represented as an array; the largest costs come from growing beyond the current allocation size (because everything must move), or from inserting or deleting somewhere near the beginning (because everything after that must move).
Now, going back to your code, we can see that rev1() is an O(n) implementation whereas rev2() is in fact O(n2), so it makes sense that rev2() will be much slower.

In Python, lists are implemented as arrays. If you append one element to an array, the reserved space for an array is simply expanded. If you prepend an element, all elements are shifted by 1 and that is very expensive.

you can confirm this by reading about python lists online. Python implements a list as an array, where the size of the array is actually typically larger than the size of your current list. The unused elements are at the end of the array and represent new elements that could be added to the END of the list, not the beginning. Python uses a classical amortized cost approach so that on average, appending to the end of the list takes O(1) time if you do a bunch of appends, although occasionally a single append will cause the array to become full so a new larger array needs to be created, and all the data copied to the new array. On the other hand, if you always insert at the front of the list, then in the underlying array all elements need to be moved over one index to make room for the new element at the beginning of the array. So, to summarize, if you create a list by doing N insertions, then the total running time will be O(N) if you always append new items to the end of the list, and it will be O(N^2) if you always append to the front of the list.

Is a.insert(0,x) an o(n) function? Is a.append an O(1) function? Python

I am trying to move even numbers in an array to the front and odd numbers to the back of the array. The problem asks to do this in a Linear Algorithm and do this In Place.
I came up with this:
def sort(a):
for i in range(0,len(a)-1):
if a[i]%2==0:
a.insert(0,a.pop(i))
return a
The issue is that, someone told me that technically, a.insert is an o(n) function so technically this would be considered a non-linear algorithm (when including the for i in range part of the function). Since the forum that asked this question is a couple months old, I couldn't ask for an explanation.
Basically I believe he said "Technically" because since this inserts it at the front, it does not check another N number of elements in the array, therefore making it run for practical purposes at O(n) and not O(n^2). Is this a correct assessment?
Also, someone on the forum used a.append to modify the above and changed it to look for odd numbers. No one replied so I was wondering, is a.append not an o(n) function since it moves it to the end? Is it o(1)?
Thanks for explanations and clarifications!

insert at the 0th index of a list requires shifting every other element along which makes it an O(N) operation. However, if you use a deque this operation is O(1).
append is an amortized O(1) operation since it simply requires adding the item on to the end of the list and no shifting is done. Sometimes the list needs to grow so it is not always an O(1) operation.

That is correct - insertion at the front of a Python standard list is O(n). Python lists are implemented as arrays, and thus inserting something at the front of the list requires shifting the entire contents over one spot. Appending, on the other hand, does not require any shifting, and thus is amortized O(1).
Note, however, that a.pop(i) is also an O(n) operation, because it requires shifting everything after the popped item over one spot. Thus, simply modifying your code to use append() instead of insert() would still not result in a linear algorithm.
A linear-time algorithm wouldn't use pop() but instead would do something like swap elements around so that the rest of the list doesn't have to be modified. For example, consider this:
def even_to_front(a_list):
next_even = 0
for idx in xrange(len(a_list)):
if not a_list[idx] % 2:
# a_list[idx] is even, so swap it towards the front
a_list[idx], a_list[next_even] = a_list[next_even], a_list[idx]
next_even += 1

Check this table of complexity
Insert - O(n)
Append - O(1) (lists are over allocated)

Here's how it can be done without append/insert or dequeue
def sort(L):
i, j = 0, len(L)-1
while i<j:
# point i to the next odd number from the start
while i<j and not L[i]%2: i+=1
# point j to the next even number from the end
while i<j and L[j]%2: j-=1
L[i],L[j] = L[j],L[i]

Every time you pop element from a list, you have to copy the trailing portion of the list to move it over one index to fill the hole left by the removed element. This is linear in the distance between the popped element and the tail of the list.
Every time you insert an element into a list, you have to copy the trailing portion of the list to move it over one index to create a spot to insert the new element. This is linear in the distance between the position into which you're inserting the element and the tail of the list.
If you use collections.deque, you can append and pop to both the front and the back in O(1) time. However, removing an element from the middle still be linear (and I think you'd have to write it yourself).