Develop Reference

Fastest way to sample most numbers with minimum difference larger than a value from a Python list

Given a list of 20 float numbers, I want to find a largest subset where any two of the candidates are different from each other larger than a mindiff = 1.. Right now I am using a brute-force method to search from largest to smallest subsets using itertools.combinations. As shown below, the code finds a subset after 4 s for a list of 20 numbers.
from itertools import combinations
import random
from time import time
mindiff = 1.
length = 20
random.seed(99)
lst = [random.uniform(1., 10.) for _ in range(length)]
t0 = time()
n = len(lst)
sample = []
found = False
while not found:
# get all subsets with size n
subsets = list(combinations(lst, n))
# shuffle to ensure randomness
random.shuffle(subsets)
for subset in subsets:
# sort the subset numbers
ss = sorted(subset)
# calculate the differences between every two adjacent numbers
diffs = [j-i for i, j in zip(ss[:-1], ss[1:])]
if min(diffs) > mindiff:
sample = set(subset)
found = True
break
# check subsets with size -1
n -= 1
print(sample)
print(time()-t0)
Output:
{2.3704888087015568, 4.365818049020534, 5.403474619948962, 6.518944556233767, 7.8388969285727015, 9.117993839791751}
4.182451486587524
However, in reality I have a list of 200 numbers, which is infeasible for a brute-froce enumeration. I want a fast algorithm to sample just one random largest subset with a minimum difference larger than 1. Note that I want each sample has randomness and maximum size. Any suggestions?

My previous answer assumed you simply wanted a single optimal solution, not a uniform random sample of all solutions. This answer assumes you want one that samples uniformly from all such optimal solutions.
Construct a directed acyclic graph G where there is one node for each point, and nodes a and b are connected when b - a > mindist. Also add two virtual nodes, s and t, where s -> x for all x and x -> t for all x.
Calculate for each node in G how many paths of length k exist to t. You can do this efficiently in O(n^2 k) time using dynamic programming with a table P[x][k], filling initially P[x][0] = 0 except P[t][0] = 1, and then P[x][k] = sum(P[y][k-1] for y in neighbors(x)).
Keep doing this until you reach the maximum k - you now know the size of the optimal subset.
Uniformly sample a path of length k from s to t using P to weight your choices.
This is done by starting at s. We then look at each neighbor of s and choose one randomly with a weighting dictated by P[s][k]. This gives us our first element of the optimal set.
We then repeatedly perform this step. We are at x, look at the neighbors of x and pick one randomly using weights P[x][k-i] where i is the step we're at.
Use the nodes you sampled in 3 as your random subset.
An implementation of the above in pure Python:
import random
def sample_mindist_subset(xs, mindist):
# Construct directed graph G.
n = len(xs)
s = n; t = n + 1 # Two virtual nodes, source and sink.
neighbors = {
i: [t] + [j for j in range(n) if xs[j] - xs[i] > mindist]
for i in range(n)}
neighbors[s] = [t] + list(range(n))
neighbors[t] = []
# Compute number of paths P[x][k] from x to t of length k.
P = [[0 for _ in range(n+2)] for _ in range(n+2)]
P[t][0] = 1
for k in range(1, n+2):
for x in range(n+2):
P[x][k] = sum(P[y][k-1] for y in neighbors[x])
# Sample maximum length path uniformly at random.
maxk = max(k for k in range(n+2) if P[s][k] > 0)
path = [s]
while path[-1] != t:
candidates = neighbors[path[-1]]
weights = [P[cn][maxk-len(path)] for cn in candidates]
path.append(random.choices(candidates, weights)[0])
return [xs[i] for i in path[1:-1]]
Note that if you want to sample from the same set of numbers many times, you don't have to recompute P every single time and can re-use it.

I probably don't fully understand the question, because right now the solution is quite trivial. EDIT: yes, I misunderstood after all, the OP does not just want an optimal solution, but wishes to randomly sample from the set of optimal solutions. This answer is not incorrect but it also is an answer to a different question than what OP is interested in.
Simply sort the numbers and greedily construct the subset:
def mindist_subset(xs, mindist):
result = []
for x in sorted(xs):
if not result or x - result[-1] > mindist:
result.append(x)
return result
Sketch of proof of correctness.
Suppose we have a solution S given input array A that is of optimal size. If it does not contain min(A) note that we could remove min(S) from S and add min(A) since this would only increase the distance between min(S) and the second smallest number in S. Conclusion: we can without loss of generality assume that min(A) is part of an optimal solution.
Now we can apply this argument recursively. We add min(A) to a solution and remove all elements too close to min(A), giving remaining elements A'. Then we're left with a subproblem where exactly the same argument applies, we can choose min(A') as our next element of the solution, etc.

Find the indices of the two largest and two smallest values in a matrix in python

I am attempting to find the indices of the two smallest and the two largest values in python:
I have
import sklearn
euclidean_matrix=sklearn.metrics.pairwise_distances(H10.T,metric='euclidean')
max_index =np.where(euclidean_matrix==np.max(euclidean_matrix[np.nonzero(euclidean_matrix)]))
min_index=np.where(euclidean_matrix==np.min(euclidean_matrix[np.nonzero(euclidean_matrix)]))
min_index
max_index
I get the following output
(array([158, 272]), array([272, 158]))
(array([ 31, 150]), array([150, 31]))
the above code only returns the indices of the absolute smallest and the absolute largest values of the matrix, I would like to find the indices of the next smallest value and the indices of the next largest value. How can I do this? Ideally I would like to return the indices of the 2 largest values of the matrix and the indices of the two smallest values of the matrix. How can I do this?

I can think of a couple ways of doing this. Some of these depend on how much data you need to search through.
A couple of caveats: You will have to decide what to do when there are 1, 2, 3 elements only Or if all the same value, do you want min, max, etc to be identical? What if there are multiple items in max or min or min2, max2? which should be selected?
run min then remove that element run min on the rest. run max then remove that element and run on the rest (note that this is on the original and not the one with min removed). This is the least efficient method since it requires searching 4 times and copying twice. (Actually 8 times because we find the min/max then find the index.) Something like the in the pseudo code.
PSEUDO CODE:
max_index = np.where(euclidean_matrix==np.max(euclidean_matrix[np.nonzero(euclidean_matrix)]))
tmp_euclidean_matrix = euclidean_matrix #make sure this is a deepcopy
tmp_euclidean_matrix.remove(max_index) #syntax might not be right?
max_index2 = np.where(tmp_euclidean_matrix==np.max(tmp_euclidean_matrix[np.nonzero(tmp_euclidean_matrix)]))
min_index = np.where(euclidean_matrix==np.min(euclidean_matrix[np.nonzero(euclidean_matrix)]))
tmp_euclidean_matrix = euclidean_matrix #make sure this is a deepcopy
tmp_euclidean_matrix.remove(min_index) #syntax might not be right?
min_index2 = np.where(tmp_euclidean_matrix==np.min(tmp_euclidean_matrix[np.nonzero(tmp_euclidean_matrix)]))
Sort the data (if you need it sorted anyway this is a good option) then just grab two smallest and largest. This isn't great unless you needed it sorted anyway because of many copies and comparisons to sort.
PSEUDO CODE:
euclidean_matrix.sort()
min_index = 0
min_index2 = 1
max_index = len(euclidean_matrix) - 1
max_index2 = max_index - 1
Best option would be to roll your own search function to run on the data, this would be most efficient because you would go through the data only once to collect them.
This is just a simple iterative approach, other algorithms may be more efficient. You will want to validate this works though.
PSEUDO CODE:
def minmax2(array):
""" returns (minimum, second minimum, second maximum, maximum)
"""
if len(array) == 0:
raise Exception('Empty List')
elif len(array) == 1:
#special case only 1 element need at least 2 to have different
minimum = 0
minimum2 = 0
maximum2 = 0
maximum = 0
else:
minimum = 0
minimum2 = 1
maximum2 = 1
maximum = 0
for i in range(1, len(array)):
if array[i] <= array[minimum]:
# a new minimum (or tie) will shift the other minimum
minimum2 = minimum
minimum = i
elif array[i] < array[minimum2]:
minimum2 = i
elif array[i] >= array[maximum]:
# a new maximum (or tie) will shift the second maximum
maximum2 = maximum
maximum = i
elif array[i] > array[maximum2]:
maximum2 = i
return (minimum, minimum2, maximum2, maximum)
edit: Added pseudo code

Guidance on removing a nested for loop from function

I'm trying to write the fastest algorithm possible to return the number of "magic triples" (i.e. x, y, z where z is a multiple of y and y is a multiple of x) in a list of 3-2000 integers.
(Note: I believe the list was expected to be sorted and unique but one of the test examples given was [1,1,1] with the expected result of 1 - that is a mistake in the challenge itself though because the definition of a magic triple was explicitly noted as x < y < z, which [1,1,1] isn't. In any case, I was trying to optimise an algorithm for sorted lists of unique integers.)
I haven't been able to work out a solution that doesn't include having three consecutive loops and therefore being O(n^3). I've seen one online that is O(n^2) but I can't get my head around what it's doing, so it doesn't feel right to submit it.
My code is:
def solution(l):
if len(l) < 3:
return 0
elif l == [1,1,1]:
return 1
else:
halfway = int(l[-1]/2)
quarterway = int(halfway/2)
quarterIndex = 0
halfIndex = 0
for i in range(len(l)):
if l[i] >= quarterway:
quarterIndex = i
break
for i in range(len(l)):
if l[i] >= halfway:
halfIndex = i
break
triples = 0
for i in l[:quarterIndex+1]:
for j in l[:halfIndex+1]:
if j != i and j % i == 0:
multiple = 2
while (j * multiple) <= l[-1]:
if j * multiple in l:
triples += 1
multiple += 1
return triples
I've spent quite a lot of time going through examples manually and removing loops through unnecessary sections of the lists but this still completes a list of 2,000 integers in about a second where the O(n^2) solution I found completes the same list in 0.6 seconds - it seems like such a small difference but obviously it means mine takes 60% longer.
Am I missing a really obvious way of removing one of the loops?
Also, I saw mention of making a directed graph and I see the promise in that. I can make the list of first nodes from the original list with a built-in function, so in principle I presume that means I can make the overall graph with two for loops and then return the length of the third node list, but I hit a wall with that too. I just can't seem to make progress without that third loop!!

from array import array
def num_triples(l):
n = len(l)
pairs = set()
lower_counts = array("I", (0 for _ in range(n)))
upper_counts = lower_counts[:]
for i in range(n - 1):
lower = l[i]
for j in range(i + 1, n):
upper = l[j]
if upper % lower == 0:
lower_counts[i] += 1
upper_counts[j] += 1
return sum(nx * nz for nz, nx in zip(lower_counts, upper_counts))
Here, lower_counts[i] is the number of pairs of which the ith number is the y, and z is the other number in the pair (i.e. the number of different z values for this y).
Similarly, upper_counts[i] is the number of pairs of which the ith number is the y, and x is the other number in the pair (i.e. the number of different x values for this y).
So the number of triples in which the ith number is the y value is just the product of those two numbers.
The use of an array here for storing the counts is for scalability of access time. Tests show that up to n=2000 it makes negligible difference in practice, and even up to n=20000 it only made about a 1% difference to the run time (compared to using a list), but it could in principle be the fastest growing term for very large n.

How about using itertools.combinations instead of nested for loops? Combined with list comprehension, it's cleaner and much faster. Let's say l = [your list of integers] and let's assume it's already sorted.
from itertools import combinations
def div(i,j,k): # this function has the logic
return l[k]%l[j]==l[j]%l[i]==0
r = sum([div(i,j,k) for i,j,k in combinations(range(len(l)),3) if i<j<k])

#alaniwi provided a very smart iterative solution.
Here is a recursive solution.
def find_magicals(lst, nplet):
"""Find the number of magical n-plets in a given lst"""
res = 0
for i, base in enumerate(lst):
# find all the multiples of current base
multiples = [num for num in lst[i + 1:] if not num % base]
res += len(multiples) if nplet <= 2 else find_magicals(multiples, nplet - 1)
return res
def solution(lst):
return find_magicals(lst, 3)
The problem can be divided into selecting any number in the original list as the base (i.e x), how many du-plets we can find among the numbers bigger than the base. Since the method to find all du-plets is the same as finding tri-plets, we can solve the problem recursively.
From my testing, this recursive solution is comparable to, if not more performant than, the iterative solution.

This answer was the first suggestion by #alaniwi and is the one I've found to be the fastest (at 0.59 seconds for a 2,000 integer list).
def solution(l):
n = len(l)
lower_counts = dict((val, 0) for val in l)
upper_counts = lower_counts.copy()
for i in range(n - 1):
lower = l[i]
for j in range(i + 1, n):
upper = l[j]
if upper % lower == 0:
lower_counts[lower] += 1
upper_counts[upper] += 1
return sum((lower_counts[y] * upper_counts[y] for y in l))
I think I've managed to get my head around it. What it is essentially doing is comparing each number in the list with every other number to see if the smaller is divisible by the larger and makes two dictionaries:
One with the number of times a number is divisible by a larger
number,
One with the number of times it has a smaller number divisible by
it.
You compare the two dictionaries and multiply the values for each key because the key having a 0 in either essentially means it is not the second number in a triple.
Example:
l = [1,2,3,4,5,6]
lower_counts = {1:5, 2:2, 3:1, 4:0, 5:0, 6:0}
upper_counts = {1:0, 2:1, 3:1, 4:2, 5:1, 6:3}
triple_tuple = ([1,2,4], [1,2,6], [1,3,6])

Comparing elements in a list with its neighbours

I have a list of 29400 values in Python and I'm trying to check if each element of the list is larger than both the 2000 neighbors to its left and the 2000 neighbors to its right. If the element is larger than its 4000 neighbors, I want to retrieve the index of the element. If there aren't 2000 neighbors to the right I just want to compare it with future elements until I reach the end of the list, and vice versa if there aren't 2000 values to the left.
def find_peaks(t):
prev = []
future = []
peak_index = []
for i in range(len(t)-2000): # compare element with previous values
for j in range(1,2001):
if t[i]<t[i+j]:
prev.append(False)
break
if j==2000:
if t[i]<t[i+j]:
prev.append(False)
break
prev.append(True)
for i in range(1999,len(t)-1): # compare element with future values
for j in range(1,2001):
if t[i]<t[i-j]:
future.append(False)
break
if j==2000:
if t[i]<t[i-j]:
future.append(False)
break
future.append(True)
future = future[::-1] # reverse list
for i in range(0,len(prev)-1):
if prev[i] == True:
if prev[i] == True:
peak_index.append(i)
Does anyone know of any better ways to go about this? I was having trouble comparing elements near the end and the beginning of the list- If there aren't 2000 elements left in the list for me to compare with, then the list wraps around to the beginning of the list, which isn't something I want.

You can use some list comprehension, so the actual search becomes a one-liner. I cannot judge about speed and beauty, but it takes just some seconds on my machine.
import random
# create list of random numbers and manually insert two peaks
t = [random.randrange(1, 1000) for r in range(29400)] # found this here: https://stackoverflow.com/questions/16655089/python-random-numbers-into-a-list
t[666] = 2000
t[6666] = 2000
# finds the peak elements
peaks = [index for index, value in enumerate(t) if value == max(t[max(index-2000, 0):min(index+2000, len(t))])]
print peaks # includes 666 and 6666

A naive, iterative version would be to go through each element and do look behind/ahead a certain number of times to see if there are values larger than it, and if not consider it a pick:
def find_peaks(data, span=2000):
peaks = []
for i, value in enumerate(data):
peak = False # start with the assumed non-peak
for j in range(max(0, i - 1), max(0, i - span - 1), -1): # look behind
peak = value > data[j] # check if our value is larger than the selected neighbor
if not peak: # not a peak, break away
break
if peak: # look behind passed, look ahead:
for j in range(i + 1, min(i + span + 1, len(data))):
if value <= data[j]: # look ahead failed, break away
peak = False
break
if peak: # if look ahead passed...
peaks.append(i) # add it to our peaks list
return peaks
This is the most performant way to do it with look ahead/behind as it breaks away immediately when a condition is not met instead of checking every element with every element.
If you want to count in the neighbors that are of the same value when calculating a peak (so your current peak candidate is the same) you can use peak = value >= data[j] in the look behind and if value < data[j] in the look ahead portions.

My solution does not involve any language powerful inbuilt methods. Its just plain logic to find the target peaks. Basically I am iterating over each element and then in inner loop checking if its previous or future neighbors are present and less than the current element. I am initializing a isPeak variable to True for if none of the neighbors in both direction are greater than the current element, which indicates your current element is peak element. After that just getting the index of target element.
def find_peaks(t , neighbour_length=2000):
peak_index = []
for i in range(len(t)): # compare element with previous values
isPeak = True #intialize to true
for j in range(i, neighbour_length + i):
# Check if previous index value is present
if (2*i-j-1 >= 0):
# Check if next neighbour is less or break
if(t[i] <= t[2*i-j-1]):
isPeak = False
break
# Check if Future element is present
if (j+i+1 < len(t)):
#Check if next future neighbour ir less or break
if(t[i] <= t[i+j+1]):
isPeak = False
break
if(isPeak):
peak_index.append(i)
return peak_index
Hope it Helps!

Minimum of 4th Element in NxNx4 list (Python)

Hi I've been reading up on finding the minimum of a multidimensional list, but if I have an N x N x 4 list, how do I get the minimum between every single 4th element? All other examples have been for a small example list using real indices. I suppose I'll be needing to define indices in terms of N....
[[[0,1,2,3],[0,1,2,3],...N],[[0,1,2,3],[0,1,2,3],...N].....N]
And then there's retrieving their indices.
I don't know what to try.
If anyone's interested in the actual piece of code:
relative = [[[[100] for k in range(5)] for j in range(N)] for i in range(N)]
What the following does is fill in the 4th element with times satisfying the mathematical equations. The 0th, 1st, 2nd and 3rd elements of relative have positions and velocities. The 4th spot is for the time taken for the i and jth particles to collide (redundant values such as i-i or j-i are filled with the value 100 (because it's big enough for the min function not to retrieve it). I need the shortest collision time (hence the 4th element comparisons)
def time(relative):
i = 0
t = 0
while i<N:
j = i+1
while j<N and i<N:
rv = relative[i][j][0]*relative[i][j][2]+relative[i][j][1]*relative[i][j][3] #Dot product of r and v
if rv<0:
rsquared = (relative[i][j][0])**2+(relative[i][j][1])**2
vsquared = (relative[i][j][2])**2+(relative[i][j][3])**2
det = (rv)**2-vsquared*(rsquared-diameter**2)
if det<0:
t = 100 #For negative times, assign an arbitrarily large number to make sure min() wont pick it up.
elif det == 0:
t = -rv/vsquared
elif det>0:
t1 = (-rv+sqrt((rv)**2-vsquared*(rsquared-diameter**2)))/(vsquared)
t2 = (-rv-sqrt((rv)**2-vsquared*(rsquared-diameter**2)))/(vsquared)
if t1-t2>0:
t = t2
elif t1-t2<0:
t = t1
elif rv>=0:
t = 100
relative[i][j][4]=t #Put the times inside the relative list for element ij.
j = j+1
i = i+1
return relative
I've tried:
t_fin = min(relative[i in range(0,N-1)][j in range(0,N-1)][4])
Which compiles but always returns 100 even thought I've checked it isnt the smallest element.

If you want the min of 4th element of NxNx4 list,
min([x[3] for lev1 in relative for x in lev1])