Related
So, I know I can get a random list from a population using the random module,
l = [0, 1, 2, 3, 4, 8 ,9]
print(random.sample(l, 3))
# [1, 3, 2]
But, how do I get the list of the unselected ones? Do, I need to remove them manually from the list? Or, is there a method to get them too?
Edit: The list l from example doesn't contain the same items multiple times, but when it does I wouldn't want it removed more than it's selected as sample.
l = [0, 1, 2, 3, 4, 8 ,9]
s1 = set(random.sample(l, 3))
s2 = set(l).difference(s1)
>>> s1
{0, 3, 8}
>>> s2
{1, 2, 4, 9}
Update: same items multiple times
You can shuffle your list first and partition your population after in two:
l = [7, 4, 5, 4, 5, 9, 8, 6, 6, 6, 9, 8, 6, 3, 8]
pop = l[:]
random.shuffle(pop)
pop1, pop2 = pop[:3], pop[3:]
>>> pop1
[8, 4, 9]
>>> pop2
[7, 6, 8, 6, 5, 6, 9, 6, 5, 8, 4, 3]
Because your list can contain multiple same items, you can change to the approach below:
import random
l = [0, 1, 2, 3, 4, 8 ,9]
random.shuffle(l)
selected = l[:3]
unselected = l[3:]
print(selected)
# [4, 0, 1]
print(unselected)
# [8, 2, 3, 9]
If you want to keep track of duplicates, you could count the items of each type and compare the population count to the sample count.
If you don't care about the order of items in the population, you could do it like this:
from collections import Counter
import random
population = [1, 1, 2, 2, 9, 7, 9]
sample = random.sample(population, 3)
pop_count = Counter(population)
samp_count = Counter(sample)
unsampled = [
k
for k in pop_count
for i in range(pop_count[k] - samp_count[k])
]
If you care about the order in the population, you could do something like this:
check = sample.copy()
unsampled = []
for val in population:
if val in check:
check.remove(val)
else:
unsampled.append(val)
Or there's this weird list comprehension (not recommended):
check = sample.copy()
unsampled = [
x
for x in population
if x not in check or check.remove(x)
]
The if clause here uses two tricks:
both parts of the test will be Falseish if x is not in check (list.remove() always returns None), and
remove() will only be called if the first part fails, i.e., if x is in check.
Basically, if (and only if) x is in check, it will bomb through and check the next condition, which will also be False (None), but will have the side effect of removing one copy of x from check.
You can do with:
import random
l = [0, 1, 2, 3, 4, 8 ,9]
rand = random.sample(l, 3)
rest = list(set(l) - set(rand))
print(f"initial list: {l}")
print(f"random list: {rand}")
print (f"rest list: {rest}")
Result:
initial list: [0, 1, 2, 3, 4, 8, 9]
random list: [2, 9, 0]
rest list: [8, 1, 3, 4]
I was attempting to remove all duplicated numbers in a list.
I was trying to understand what is wrong with my code.
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
for x in numbers:
if numbers.count(x) >= 2:
numbers.remove(x)
print(numbers)
The result I got was:
[1, 1, 6, 5, 2, 3]
I guess the idea is to write code yourself without using library functions. Then I would still suggest to use additional set structure to store your previous items and go only once over your array:
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
unique = set()
for x in numbers:
if x not in unique:
unique.add(x)
numbers = list(unique)
print(numbers)
If you want to use your code then the problem is that you modify collection in for each loop, which is a big NO NO in most programming languages. Although Python allows you to do that, the problem and solution are already described in this answer: How to remove items from a list while iterating?:
Note: There is a subtlety when the sequence is being modified by the loop (this can only occur for mutable sequences, i.e. lists). An internal counter is used to keep track of which item is used next, and this is incremented on each iteration. When this counter has reached the length of the sequence the loop terminates. This means that if the suite deletes the current (or a previous) item from the sequence, the next item will be skipped (since it gets the index of the current item which has already been treated). Likewise, if the suite inserts an item in the sequence before the current item, the current item will be treated again the next time through the loop. This can lead to nasty bugs that can be avoided by making a temporary copy using a slice of the whole sequence, e.g.,
for x in a[:]:
if x < 0: a.remove(x)
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
Using a shallow copy of the list:
for x in numbers[:]:
if numbers.count(x) >= 2:
numbers.remove(x)
print(numbers) # [1, 6, 5, 2, 3]
Alternatives:
Preserving the order of the list:
Using dict.fromkeys()
print(list(dict.fromkeys(numbers).keys())) # [1, 6, 5, 2, 3]
Using more_itertools.unique_everseen(iterable, key=None):
from more_itertools import unique_everseen
print(list(unique_everseen(numbers))) # [1, 6, 5, 2, 3]
Using pandas.unique:
import pandas as pd
print(pd.unique(numbers).tolist()) # [1, 6, 5, 2, 3]
Using collections.OrderedDict([items]):
from collections import OrderedDict
print(list(OrderedDict.fromkeys(numbers))) # [1, 6, 5, 2, 3]
Using itertools.groupby(iterable[, key]):
from itertools import groupby
print([k for k,_ in groupby(numbers)]) # [1, 6, 5, 2, 3]
Ignoring the order of the list:
Using numpy.unique:
import numpy as np
print(np.unique(numbers).tolist()) # [1, 2, 3, 5, 6]
Using set():
print(list(set(numbers))) # [1, 2, 3, 5, 6]
Using frozenset([iterable]):
print(list(frozenset(numbers))) # [1, 2, 3, 5, 6]
Why don't you simply use a set:
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
numbers = list(set(numbers))
print(numbers)
Before anything, the first advice I can give is to never edit over an array that you are looping. All kinds of wacky stuff happens. Your code is fine (I recommend reading other answers though, there's an easier way to do this with a set, which pretty much handles the duplication thing for you).
Instead of removing number from the array you are looping, just clone the array you are looping in the actual for loop syntax with slicing.
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
for x in numbers[:]:
if numbers.count(x) >= 2:
numbers.remove(x)
print(numbers)
print("Final")
print(numbers)
The answer there is numbers[:], which gives back a clone of the array. Here's the print output:
[1, 1, 1, 6, 5, 5, 2, 3]
[1, 1, 6, 5, 5, 2, 3]
[1, 6, 5, 5, 2, 3]
[1, 6, 5, 5, 2, 3]
[1, 6, 5, 5, 2, 3]
[1, 6, 5, 2, 3]
[1, 6, 5, 2, 3]
[1, 6, 5, 2, 3]
[1, 6, 5, 2, 3]
Final
[1, 6, 5, 2, 3]
Leaving a placeholder here until I figure out how to explain why in your particular case it's not working, like the actual step by step reason.
Another way to solve this making use of the beautiful language that is Python, is through list comprehension and sets.
Why a set. Because the definition of this data structure is that the elements are unique, so even if you try to put in multiple elements that are the same, they won't appear as repeated in the set. Cool, right?
List comprehension is some syntax sugar for looping in one line, get used to it with Python, you'll either use it a lot, or see it a lot :)
So with list comprehension you will iterate an iterable and return that item. In the code below, x represents each number in numbers, x is returned to be part of the set. Because the set handles duplicates...voila, your code is done.
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
nubmers_a_set = {x for x in numbers }
print(nubmers_a_set)
This seems like homework but here is a possible solution:
import numpy as np
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
filtered = list(np.unique(numbers))
print(filtered)
#[1, 2, 3, 5, 6]
This solution does not preserve the ordering. If you need also the ordering use:
filtered_with_order = list(dict.fromkeys(numbers))
Why don't you use fromkeys?
numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
numbers = list(dict.fromkeys(numbers))
Output: [1,6,5,2,3]
The flow is as follows.
Now the list is [1, 1, 1, 1, 6, 5, 5, 2, 3] and Index is 0.
The x is 1. The numbers.count(1) is 4 and thus the 1 at index 0 is removed.
Now the numbers list becomes [1, 1, 1, 6, 5, 5, 2, 3] but the Index will +1 and becomes 1.
The x is 1. The numbers.count(1) is 3 and thus the 1 and index 1 is removed.
Now the numbers list becomes [1, 1, 6, 5, 5, 2, 3] but the Index will +1 and becomes 2.
The x will be 6.
etc...
So that's why there are two 1's.
Please correct me if I am wrong. Thanks!
A fancy method is to use collections.Counter:
>>> from collections import Counter
>>> numbers = [1, 1, 1, 1, 6, 5, 5, 2, 3]
>>> c = Counter(numbers)
>>> list(c.keys())
[1, 6, 5, 2, 3]
This method have a linear time complexity (O(n)) and uses a really performant library.
You can try:
from more_itertools import unique_everseen
items = [1, 1, 1, 1, 6, 5, 5, 2, 3]
list(unique_everseen(items))
or
from collections import OrderedDict
>>> items = [1, 1, 1, 1, 6, 5, 5, 2, 3]
>>> list(OrderedDict.fromkeys(items))
[1, 2, 0, 3]
more you can find here
How do you remove duplicates from a list whilst preserving order?
This question already has answers here:
How to find most common elements of a list? [duplicate]
(11 answers)
Closed 6 years ago.
I have a list such as this:
lst = [1, 3, 5, 1, 5, 6, 1, 1, 3, 4, 5, 2, 3, 4, 5, 3, 4]
I would like to find all the elements which occur most frequently.
So I would like:
most = [1, 3, 5]
1, 3, and 5 would occur the most, which is 4 times. What's a fast, pythonic way to do this? I tried methods shown here:
How to find most common elements of a list?.
But it only gives me the top 3, I need all elements. Thank you.
With collections.Counter and a list comprehension:
from collections import Counter
lst = [1, 3, 5, 1, 5, 6, 1, 1, 3, 4, 5, 2, 3, 4, 5, 3, 4]
r = [x for x, _ in Counter(lst).most_common(3)]
print(r)
# [1, 3, 5]
You can generalize for values with highest count by using max on the counter values:
c = Counter(lst)
m = max(c.values())
r = [k for k in c if c[k] == m]
print(r)
# [1, 3, 5]
For large iterables, to efficiently iterate through the counter and stop once the required items have been taken, you can use itertools.takewhile with most_common without any parameters:
from itertools import takewhile
c = Counter(lst)
m = max(c.values())
r = [x for x, _ in takewhile(lambda x: x[1]==m, c.most_common())]
print(r)
# [1, 3, 5]
You gain by not having to iterate through all the items in the counter object, although there is some overhead with having to sort the items using most_common; so I'm sure if this absolutely efficient after all. You could do some experiments with timeit.
You can also get the same result with groupby from itertools module and list comprehension in this way:
from itertools import groupby
a = [1, 3, 5, 1, 5, 6, 1, 1, 3, 4, 5, 2, 3, 4, 5, 3, 4]
most_common = 3
final = [k for k,v in groupby(sorted(a), lambda x: x) if len(list(v)) > most_common]
Output:
print(final)
>>> [1, 3, 5]
You can do the following if you like to print all the most frequent,
from collections import Counter
words=[1, 3, 5, 1, 5, 6, 1, 1, 3, 4, 5, 2, 3, 4, 5, 3, 4]
most= [word for word, word_count in Counter(words).most_common()]
print (most)
>>>
[1, 3, 5, 4, 2, 6]
Please note, if you want to limit, you can enter the number inside most_common() function. Ex: ...most_common(3)]. Hope this answers your question.
Have a list arr = [1,3,4,5,2,3,4,2,5,7,3,8,1,9,6,2,1,2,1,3,4,3,4,6,9]
want to remove the duplicate values so that the original list should contains single instances of all elements. Do not want to create a extra list and append the elements from list. Also do not want to use inbuilt "set".
Tried to do that with some code as below:
l = len(arr)
for x in range(l):
for y in range(x+1,l):
if arr[x] == arr[y]:
del arr[y]
Tried the above code and its throwing error
"IndexError: list index out of range"
What I understand is whiling deleting the value the size of the list is changing for which its throwing the error. So I made the below changes. But still its failing with same error:
l = len(arr)
for x in range(l):
for y in range(x+1,l):
if arr[x] == arr[y]:
t = y
del arr[y]
y = t - 1
Can some one help me out on this?
Thanks in Advance.
You are trying to make the code more efficient by caching the length of the list in the local variable l. However, that is not helpful because the list is being trimmed inside the loop, and you are not keeping the cached length variable in sync.
for index in range(len(arr)-1,0,-1):
if arr[index] in arr[:index]:
del arr[index]
By going backwards through the array and looking for earlier occurrences of each element, you can avoid having to worry about the length of the list changing all the time.
This method also preserves the order in which elements occur in the original array. Note the instruction is to only remove duplicates (a.k.a. subsequent occurrences).
For example the list [9,3,4,3,5] should reduce to [9,3,4, 5] as the second occurrence of 3 is considered a duplicate and should be removed.
How about this approach:
>>> set(arr)
set([1, 2, 3, 4, 5, 6, 7, 8, 9]) #Just to compare it with the results below.
>>> arr = [1,3,4,5,2,3,4,2,5,7,3,8,1,9,6,2,1,2,1,3,4,3,4,6,9]
>>> arr.sort()
>>> arr
[1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 6, 6, 7, 8, 9, 9]
>>> for i in arr:
while arr.count(i) > 1:
del arr[i]
>>> arr
[1, 2, 3, 4, 5, 6, 7, 8, 9]
Another approach is to find, after sorting your list, the length of the sublist to delete for each number:
>>> arr = [1,3,4,5,2,3,4,2,5,7,3,8,1,9,6,2,1,2,1,3,4,3,4,6,9]
>>> arr.sort()
>>> arr
[1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 6, 6, 7, 8, 9, 9]
>>> for i,j in enumerate(arr):
del arr[i+1:i+arr.count(j)]
>>> arr
[1, 2, 3, 4, 5, 6, 7, 8, 9]
This question already has answers here:
Repeating elements of a list n times
(14 answers)
Closed 5 months ago.
I want to write a function that reads a list [1,5,3,6,...]
and gives [1,1,5,5,3,3,6,6,...].
Any idea how to do it?
>>> a = range(10)
>>> [val for val in a for _ in (0, 1)]
[0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9]
N.B. _ is traditionally used as a placeholder variable name where you do not want to do anything with the contents of the variable. In this case it is just used to generate two values for every time round the outer loop.
To turn this from a list into a generator replace the square brackets with round brackets.
>>> a = [1, 2, 3]
>>> b = []
>>> for i in a:
b.extend([i, i])
>>> b
[1, 1, 2, 2, 3, 3]
or
>>> [a[i//2] for i in range(len(a)*2)]
[1, 1, 2, 2, 3, 3]
numpy.repeat does what you want:
import numpy as np
yourList = [1,5,3,6]
n = 2
list(np.repeat(yourList, n))
result:
[1, 1, 5, 5, 3, 3, 6, 6]
If you don't mind using numpy arrays you can also omit the list() call in the last line.
If you already have the roundrobin recipe described in the documentation for itertools—and it is quite handy—then you can just use
roundrobin(my_list, my_list)
I would use zip and itertools.chain.
>>> import itertools
>>> l = [1,5,3,6,16]
>>> list(itertools.chain(*zip(l,l)))
[1, 1, 5, 5, 3, 3, 6, 6, 16, 16]
Note: I only used list to consume the generator to make it fit for printing. You probably don't need the list call in your code...
It is possible use list multiplication. Case you need each list member together just use sorted method.
>>> lst = [1,2,3,4]
>>> sorted(lst*2)
[1,1,2,2,3,3,4,4]
With a little slicing...
>>> a = [3, 1, 4, 1, 5]
>>> a[:0] = a[::2] = a[1::2] = a[:]
>>> a
[3, 3, 1, 1, 4, 4, 1, 1, 5, 5]
I would use
import itertools
foo = [1, 5, 3, 6]
new = itertools.chain.from_iterable([item, item] for item in foo)
new will be an iterator that lazily iterates over the duplicated items. If you need the actual list computed, you can do list(new) or use one of the other solutions.
One can use zip and flat the list
a = [3, 1, 4, 1, 5]
sum(zip(a,a), ()) # (3, 3, 1, 1, 4, 4, 1, 1, 5, 5)
The output is a tuple, but conversion to a list is easy.
Regarding flatting a tuple with sum see https://stackoverflow.com/a/952946/11769765 and python: flat zip.
For as much as Guido dislikes the functional operators, they can be pretty darned handy:
>>> from operator import add
>>> a = range(10)
>>> b = reduce(add, [(x,x) for x in a])
For a more general approach you could go with a list comprehension and a factor term.
Example
sample_list = [1,2,3,4,5]
factor = 2
new_list = [entry for entry in sample_list for _ in range(factor)]
Out:
>>> new_list
[1, 1, 2, 2, 3, 3, 4, 4, 5, 5]
Changing the factor variable will change how many entry of each item in the list you will have in the new list.
You could also wrap it up in a function:
def multiply_list_entries(list_, factor = 1):
list_multiplied = [entry for entry in list_ for _ in range(factor)]
return list_multiplied
>>> multiply_list_entries(sample_list, factor = 3)
[1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5]
ls1=[1,2,3]
ls2=[]
for i in ls1:
ls2.append(i)
ls2.append(i)
This code duplicates each elements in ls1
the result ls2 --> [1,1,2,2,3,3]