Develop Reference

Remove rows that contains at least one duplicate value

I have a list of indexes A :
A = np.array([[1, 4, 3],
[1, 2, 5],
[6, 7, 8],
[9, 10, 2],
[11, 3, 12]])
I would like to delete each lines that contains at least one duplicated value (no matter in which column it's located) to obtain an array with no duplicates indexes :
[[1, 4, 3],
[6, 7, 8],
[9, 10, 2],
Is there a quick and convenient way to perform this ?

Given that no sub-list should be in the output if a value in it has already been in any previous sub-list then one possible solution can look like this,
# Assuming that A is a 2-D list
output = []
# Add all values to a set
st = set()
for i in range(len(A)):
for j in range(len(A[i])):
if A[i][j] not in st:
st.add(A[i][j])
for i in range(len(A)):
flag = True
j = 0
while j < len(A[i]):
if A[i][j] not in st:
# set flag to false if value not in set meaning a duplicate is encountered
flag = False
break
else:
# if a value is in the set then remove it from set
st.remove(A[i][j])
j += 1
# append A[i] to output only if the sub-list did not encounter any duplicates
if flag:
output.append(A[i])
The time complexity will be O(n2).
When the input is,
A = [[1, 4, 3],
[1, 2, 5],
[6, 7, 8],
[9, 10, 2],
[11, 3, 12]]
It provides the output,
[[1, 4, 3], [6, 7, 8], [9, 10, 2]]

mask means dont show the element.
mask = 0 means show element
In beginning assume all are unmasked
one by one check if there is intersection with next and if so, mask that
If something is masked, dont use it for masking next elements
Show all unamsked element
mask = np.zeros((A.shape[0],), dtype=bool)
for i in range(A.shape[0]):
if mask[i]: continue
a = set(A[i])
for j in range(i+1, A.shape[0]):
if not a.intersection(A[j]): continue
mask[j] = True
A[~mask]

Python array segmented in ranges

I have the following array [1, 4, 7, 9, 2, 10, 5, 8] and I need to separate the array in 3 different arrays: one for values between 0 and 3, anther for 3 to 6 and anther for 6 and 25.The result must be something like that:
array1 = [1, 2]
array2 = [4, 5]
array3 = [7, 9, 10, 8]
Any idea about how to do it simple?

First, define your "pole" numbers
Second, generate your intervals from those "pole" numbers
Third, define as many lists as there are intervals.
Then for each interval, scan the list and appends items in the relevant list if they belong to the interval
code:
source = [1, 4, 7, 9, 2, 10, 5, 8]
poles = (0,3,6,25)
intervals = [(poles[i],poles[i+1]) for i in range(len(poles)-1)]
# will generate: intervals = [(0,3),(3,6),(6,25)]
output = [list() for _ in range(len(intervals))]
for out,(start,stop) in zip(output,intervals):
for s in source:
if start <= s <stop:
out.append(s)
print(output)
result:
[[1, 2], [4, 5], [7, 9, 10, 8]]
This solution has the advantage of being adaptable to more than 3 lists/intervals by adding more "pole" numbers.
EDIT: There's a nice & fast solution (O(log(N)*N)) if the output lists order don't matter:
first sort the input list
then generate the sliced sub-lists using bisect which returns insertion position of the provided numbers (left & right)
like this:
import bisect
source = sorted([1, 4, 7, 9, 2, 10, 5, 8])
poles = (0,3,6,25)
output = [source[bisect.bisect_left(source,poles[i]):bisect.bisect_right(source,poles[i+1])] for i in range(len(poles)-1)]
print(output)
result:
[[1, 2], [4, 5], [7, 8, 9, 10]]

You can do that in a very simple way using a combination of a for loop and range functions:
lists = ([], [], [])
for element in [1, 4, 7, 9, 2, 10, 5, 8]:
if element in range(0, 3):
lists[0].append(element)
elif element in range(3, 6):
lists[1].append(element)
elif element in range(6, 25):
lists[2].append(element)
array1, array2, array3 = lists

"One-line" solution using set.intersection(*others) and range(start, stop[, step]) functions:
l = [1, 4, 7, 9, 2, 10, 5, 8]
l1, l2, l3 = (list(set(l).intersection(range(3))), list(set(l).intersection(range(3,6))), list(set(l).intersection(range(6,25))))
print(l1)
print(l2)
print(l3)
The output:
[1, 2]
[4, 5]
[8, 9, 10, 7]
https://docs.python.org/3/library/stdtypes.html?highlight=intersection#set.intersection

how to merge two sublists sharing any number in common?(2)

Given that:
list=[[1,2,3],[3,4,5],[5,6],[6,7],[9,10],[10,11]]
I have asked a similar question before, I have tried the code on
how to merge two sublists sharing any number in common?
but I am stuck in my code now.
I want to merge the sublists that share a common number,
e.g. [1,2,3] and [3,4,5] can merge to give [1,2,3,4,5] as they share a common number, 3.
In [[1,2,3],[3,4,5],[5,6]], although [1,2,3] and [3,4,5] share a common number, 3,
[3,4,5] and [5,6] also share a common number, 5, so I want all three of them to merge then gives
[1,2,3,4,5,6].
So for list,
my expected result is:
[[1,2,3,4,5,6,7],[9,10,11]]
I have tried the following code but don't know what is wrong, can anyone help?
s = map(set, list)
for i, si in enumerate(s):
for j, sj in enumerate(s):
if i != j and si & sj:
s[i] = si | sj
s[j] = set()
list=[list(el) for el in s if el]
print list
>>>[[5, 6, 7], [9, 10, 11]]

def merge_containing(input_list):
merged_list = [set(input_list[0])]
i = 0
for sub_list in input_list:
if not set(sub_list).intersection(set(merged_list[i])): # 1
merged_list.append(set()) # 2
i += 1 # 2
merged_list[i].update(set(sub_list)) # 3
return [sorted(sub_list) for sub_list in merged_list] # 4
mylist=[[1,2,3],[3,4,5],[5,6],[6,7],[9,10],[10,11]]
print merge_containing(mylist)
Output:
[[1, 2, 3, 4, 5, 6, 7], [9, 10, 11]]
How does it work:
Check if the sub_list set shares any common member with the current
index set of the merged_list.
If it doesn't, add a new empty set to the merged_list and increment
the index.
Adds the sub_list set to the set at index in the merged_list.
Converts from set to list and return

def merge(list_of_lists):
number_set = set()
for l in list_of_lists:
for item in l:
number_set.add(item)
return sorted(number_set)
if __name__ == '__main__':
list_of_lists = [[1,2,3],[3,4,5],[5,6],[6,7],[9,10],[10,11]]
merged = merge(list_of_lists)
print merged

I'm posting this as a new answer since the OP already accepted my other.
But as pointed out by #Eithos,
the input:
[[3,4], [1,2], [1,3]]
should return
[[1,2,3,4]]
and the input
[[1,2,3],[3,4,5],[5,6],[6,7],[9,10],[10,11],[65,231,140], [13,14,51]]
should return
[[1, 2, 3, 4, 5, 6, 7], [9, 10, 11], [13, 14], [51], [65], [140], [231]]
Here's my attempt:
from itertools import chain
def merge_containing(input_list):
print " input:", input_list
chain_list = sorted(set(chain(*input_list))) # 1
print " chain:",chain_list
return_list = []
new_sub_list = []
for i, num in enumerate(chain_list):
try:
if chain_list[i + 1] == chain_list[i] + 1: # 2
new_sub_list.append(num)
else:
new_sub_list.append(num) # 3
return_list.append(new_sub_list)
new_sub_list = []
except IndexError:
new_sub_list.append(num) # 3
return_list.append(new_sub_list)
print 'result:', return_list
return return_list
mylist = [[3,4], [1,2], [1,3]]
merge_containing(mylist)
print
mylist = [[1,2,3],[3,4,5],[5,6],[6,7],[9,10],[10,11],[65,231,140], [13,14,51]]
merge_containing(mylist)
Output:
input: [[3, 4], [1, 2], [1, 3]]
chain: [1, 2, 3, 4]
result: [[1, 2, 3, 4]]
input: [[1, 2, 3], [3, 4, 5], [5, 6], [6, 7], [9, 10], [10, 11], [65, 231, 140], [13, 14, 51]]
chain: [1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 51, 65, 140, 231]
result: [[1, 2, 3, 4, 5, 6, 7], [9, 10, 11], [13, 14], [51], [65], [140], [231]]
Explanation:
This one is a little hacky them the last one
I use itertool.chain to flat all the lists and them I sort it.
Then I check if the current number is within the range of 1 digit from the next
If it is I store it in the new_sub_list, if not I store in the new_sub_list, then store new_sub_list in the return_list, and empty the new_sub_list.
Note the try/except Index Error, it to avoid comparing the last item of the list with one that doesn't exist,

Well... I couldn't resist answering #f.rodrigues' last answer with one of my own.
I have to be honest though, this final version was heavily influenced by jonrsharpe's solution (the code went through various revisions, each one more efficient until I realized his method was the only way to press the most amount of juice) over here: Using sublists to create new lists where numbers don't repeat
Which made me wonder... why are we answering the same question over and over again (from the very same person)? This question, how to merge two sublists sharing any number in common? and the one with jonrsharpe's solution.
Anyway, this joins lists in the way outlined in his first question, but, like the solutions he already received over there, this one also works just as well for solving this problem.
sequence = [[1, 4, 9], [2, 3, 6], [4, 13, 50], [13, 23, 29], [2, 3, 7]]
def combineSequences(seq):
for index, y in enumerate(seq):
while True:
for x in seq[index + 1:]:
if any(i in x for i in seq[index]):
seq.remove(x)
y.extend(x)
break
else:
index += 1
break
return [sorted(set(l)) for l in seq]
sequence = [[1, 4, 9], [2, 3, 6], [4, 13, 50], [13, 23, 29], [2, 3, 7]]
print combineSequences(sequence)
>>> [[1, 4, 9, 13, 23, 29, 50], [2, 3, 6, 7]]
sequence = [[3, 4], [1, 2], [1, 3]]
print combineSequences(sequence)
>>> [[1, 2, 3, 4]]
This solution operates under a different assumption than the one I made earlier, just to clarify. This simply joins lists that have a common number. If the idea, however, was to only have them separated by intervals of 1, see my other answer.
That's it!

Okay. This solution may be difficult to grasp at first, but it's quite logical and succint, IMO.
The list comprehension basically has two layers. The outer layer will itself create separate lists
for each value of i (outer index) that satisfies the condition that the value it points to in the list is not equal to the value pointed to by the previous index + 1. So every numerical jump greater than one will create a new list within the outer list comprehension.
The math around the second (inner list comprehension) condition is a bit more complicated to explain, but essentially the condition seeks to make sure that the inner list only begins counting from the point where the outer index is at, stopping to where once again there is a numerical jump greater than one.
Assuming an even more complicated input:
listVar=[[1,2,3],[3,4,5],[5,6],[6,7],[9,10],[10,11],[65,231,140], [13,14,51]]
# Flattens the lists into one sorted list with no duplicates.
l = sorted(set([x for b in listVar for x in b]))
lGen = xrange(len(l))
print [
[l[i2] for i2 in lGen if l[i2] + i == l[i] + i2]
for i in lGen if l[i] != l[i-1] + 1
]
>>> [[1, 2, 3, 4, 5, 6, 7], [9, 10, 11], [13, 14], [51], [65], [140], [231]]

How can I merge lists that have common elements within a list of lists?

I have a list like [[1, 2, 4], [2, 5], [0, 3, 7, 8], [12, 3, 6], [18, 14]]. How can I get a list that contains lists of all the lists that contain overlapping elements added together? For the example input, the result should be [[1, 2, 4, 5], [0, 3, 6, 7, 8, 12], [14, 18]].

a = [[1, 2, 4], [2, 5], [0, 3, 7, 8], [12, 3, 6], [18, 14]]
result = []
for s in a:
s = set(s)
for t in result:
if t & s:
t.update(s)
break
else:
result.append(s)
This will go one-by-one through the list and create a set from the current sublist (s). Then it will check in the results, if there is another set t that has a non-empty intersection with it. If that’s the case, the items from s are added to that set t. If there is no t with a non-empty intersection, then s is a new independent result and can be appended to the result list.
A problem like this is also a good example for a fixed-point iteration. In this case, you would look at the list and continue to merge sublists as long as you could still find lists that overlap. You could implement this using itertools.combinations to look at pairs of sublists:
result = [set(x) for x in a] # start with the original list of sets
fixedPoint = False # whether we found a fixed point
while not fixedPoint:
fixedPoint = True
for x, y in combinations(result, 2): # search all pairs …
if x & y: # … for a non-empty intersection
x.update(y)
result.remove(y)
# since we have changed the result, we haven’t found the fixed point
fixedPoint = False
# abort this iteration
break

One way I can think of doing this is through recursion. Start with one item, then loop until you find every number it's connected to. For each of these numbers, you must do the same. Hence the recursion. To make it more efficient, store numbers you've visited in a list and check it at the beginning of each recursive sequence to make sure you don't repeat any explorations.

A two liner:
a_set = [set(x) for x in a]
result = [list(x.union(y)) for i,x in enumerate(a_set) for y in a_set[i:]
if x.intersection(y) and x != y]

I have left the last step for you:
a = [[1, 2, 4], [2, 5], [0, 3, 7, 8], [12, 3, 6], [18, 14]]
result = [[1, 2, 4, 5], [0, 3, 6, 7, 8, 12], [14, 18]]
# each sub list
result2 = []
count = 0
print a
for sub_list in a:
print count
print "sub_list: " + str(sub_list)
a.pop(count)
print "a: " + str(a)
#each int
sub_list_extend_flag = False
for int_in_sub_list in sub_list:
print "int_in_sub_list: " + str(int_in_sub_list)
for other_sub_list in a:
print "current_other_sub_list: " + str(other_sub_list)
if int_in_sub_list in other_sub_list:
sub_list_extend_flag = True
other_sub_list.extend(sub_list)
result2.append(list(set(other_sub_list)))
if not sub_list_extend_flag:
result2.append(sub_list)
count += 1
print result2

Simple answer:
a = [[1, 2, 4], [2, 5], [0, 3, 7, 8], [12, 3, 6], [18, 14]]
for x in a:
for y in x:
print y
its more simple than first one:
box=[]
a = [[1, 2, 4], [2, 5], [0, 3, 7, 8], [12, 3, 6], [18, 14]]
for x in a:
for y in x:
box.append(y)
print box
Result:[1, 2, 4, 2, 5, 0, 3, 7, 8, 12, 3, 6, 18, 14]
And with this, you can compare the numbers:
box=[]
box2=""
a = [[1, 2, 4], [2, 5], [0, 3, 7, 8], [12, 3, 6], [18, 14]]
for x in a:
for y in x:
box.append(y)
print box
for a in box:
box2+=str(a)
print box2
Result: 12425037812361814
Also you can make it more cute:
print " ".join(box2)
Result: 1 2 4 2 5 0 3 7 8 1 2 3 6 1 8 1 4

How to merge multiple arrays in pairs

I have a problem with "pairing" arrays into one (by index). Here is an example:
INPUT:
inputArray = [[0, 1, 2, 3, 4], [2, 3, 5, 7, 8], [9, 6, 1]]
EXPECTED OUTPUT:
outputArray =
[[0,2,9],
[1,3,6],
[2,5,1],
[3,7,chooseRandom()],
[4,8,chooseRandom()]]
Questions:
How to avoid "out of range" "index error" problem
How to write chooseRandom() to choose N neighbour
Answers:
[SOLVED] Solutions provided by #jonrsharpe & #Christian & #Decency works as
expected
Clarification:
By N neighbour I mean:
I'm using python but feel free to share your thoughts in any language.

I think the following will do what you want:
from itertools import izip_longest # 'zip_longest' in Python 3.x
from random import choice
# Step 1
outputArray = list(map(list, izip_longest(*inputArray)))
# Step 2
for index, arr in enumerate(outputArray):
if any(item is None for item in arr):
valid = [item for item in arr if item is not None]
outputArray[index] = [choice(valid) if item is None else item
for item in arr]
This has two steps:
Combine all sub-lists of inputArray to the length of the longest sub-array, filling with None: [[0, 2, 9], [1, 3, 6], [2, 5, 1], [3, 7, None], [4, 8, None]]; and
Work through the outputArray, finding any sub-lists that contain None and replacing the None with a random choice from the other items in the sub-list that aren't None.
Example output:
[[0, 2, 9], [1, 3, 6], [2, 5, 1], [3, 7, 3], [4, 8, 8]]

Here's my approach to the problem, in Python 3.4. I don't really know what you mean by "choose N neighbour" but it should be pretty easy to write that however you'd like in the context below.
inputArray = [[0, 1, 2, 3, 4], [2, 3, 5, 7, 8], [9, 6, 1]]
import itertools
zipped = itertools.zip_longest(*inputArray, fillvalue=None)
outputArray = [list(item) for item in zipped]
# [[0, 2, 9], [1, 3, 6], [2, 5, 1], [3, 7, None], [4, 8, None]]
# Now replace the sentinel None in our sublists
for sublist in outputArray:
for i, element in enumerate(sublist):
if element is None:
sublist[i] = chooseRandom()
print(outputArray)

Not the most pythonic way, but you could try using this code snipped, read the comments in the code below:
import itertools, random
inputArray = [ [0, 1, 2, 3, 4], [2, 3, 5, 7, 8], [9, 6, 1] ]
outputArray = []
max_length = max(len(e) for e in inputArray) # maximum length of the sublists in <inputArray>
i = 0 # to keep the index of sublists of <outputArray>
for j in range(max_length):
outputArray.append([]) # add new sublist
for e in inputArray: # iterate through each element of <inputArray>
try:
outputArray[i].append(e[j]) # try to append the number, if an exception is raised
# then the code in the <except> clause will be executed
except IndexError as e:
outputArray[i].append(random.randint(0, 10)) # add the random number
i += 1 # increase the sublists index on each iteration
print outputArray
# [[0, 2, 9], [1, 3, 6], [2, 5, 1], [3, 7, 3], [4, 8, 7]]
Note:
You may want to change the part
random.randint(0, 10)
to get the "N neighbour".

Let me know whether you like this code:
import random
array = [[0, 1, 2, 3, 4], [2, 3, 5, 7, 8], [9, 6, 1]]
max_len = max([len(l) for l in array])
dictionary = {}
for l in array:
for i in range(0,len(l)):
if dictionary.has_key(i):
dictionary[i].append(l[i])
else:
dictionary[i] = [l[i]]
for i in range(len(l),max_len):
if dictionary.has_key(i):
dictionary[i].append(random.choice(l))
else:
dictionary[i] = [random.choice(l)]
print dictionary.values()