Related
I have list/array of integers, call a subarray a peak if it goes up and then goes down. For example:
[5,5,4,5,4]
contains
[4,5,4]
which is a peak.
Also consider
[6,5,4,4,4,4,4,5,6,7,7,7,7,7,6]
which contains
[6,7,7,7,7,7,6]
which is a peak.
The problem
Given an input list, I would like to find all the peaks contained in it of minimal length and report them. In the example above, [5,6,7,7,7,7,7,6] is also a peak but we remove the first element and it remains a peak so we don't report it.
So for input list:
L = [5,5,5,5,4,5,4,5,6,7,8,8,8,8,8,9,9,8]
we would return
[4,5,4] and [8,9,9,8] only.
I am having problems devising a nice algorithm for this. Any help would be hugely appreciated.
Using itertools
Here is a short solution using itertools.groupby to detect peaks. The groups identifying peaks are then unpacked to yield the actual sequence.
from itertools import groupby, islice
l = [1, 2, 1, 2, 2, 0, 0]
fst, mid, nxt = groupby(l), islice(groupby(l), 1, None), islice(groupby(l), 2, None)
peaks = [[f[0], *m[1], n[0]] for f, m, n in zip(fst, mid, nxt) if f[0] < m[0] > n[0]]
print(peaks)
Output
[[1, 2, 1], [1, 2, 2, 0]]
Using a loop (faster)
The above solution is elegant but since three instances of groupby are created, the list is traversed three times.
Here is a solution using a single traversal.
def peaks(lst):
first = 0
last = 1
while last < len(lst) - 1:
if lst[first] < lst[last] == lst[last+1]:
last += 1
elif lst[first] < lst[last] > lst[last+1]:
yield lst[first:last+2]
first = last + 1
last += 2
else:
first = last
last += 1
l = [1, 2, 1, 2, 2, 0, 0]
print(list(peaks(l)))
Output
[[1, 2, 1], [1, 2, 2, 0]]
Notes on benchmark
Upon benchmarking with timeit, I noticed an increase in performance of about 20% for the solution using a loop. For short lists the overhead of groupby could bring that number up to 40%. The benchmark was done on Python 3.6.
I tried to solve the following problem with Python:
But I got stuck at generating a single valid table. I was expecting the program to display a valid matrix, but in order for the program to continue and not print None, I had to assign a 7 for a square that has no possibles. What should be fixed?
My code so far:
from pprint import pprint
import sys
import random
sys.setrecursionlimit(10000)
def getPossiblesForSquare(sqx,sqy,matrix):
'''Gets the possible entries of matrix[sqy][sqx].
Assumes it equals 0.'''
assert matrix[sqy][sqx]==0
# get the row that it is on
rowon=matrix[sqy]
# columns are a little trickier
colon=[matrix[row][sqx] for row in range(5)]
# find the possibilities!
possibles=list(range(1,7))
for item in list(set(rowon+colon)): # remove duplicates
if not (item == 0) and (item in possibles):
del possibles[possibles.index(item)]
random.shuffle(possibles)
return possibles
def getPossiblesForMatrix(matrix):
'''Gets all the possible squares for a matrix.'''
possiblesdict={}
for y in range(6):
for x in range(6):
if matrix[y][x]==0:
possiblesdict[(y,x)]=getPossiblesForSquare(x,y,MATRIX)
return possiblesdict
def flattenList(matrix):
result=[]
for i in matrix:
if not isinstance(i,list):
result+=[i]
else:
result+=flattenList(i)
return result
def getPossibleMatrix(startMatrix, iteration=0, yon=1, prevZeroInd=None):
if 0 not in flattenList(startMatrix):
print('RESULT:\n\n')
return startMatrix
else:
# find&fill in the first blank one
ind=flattenList(startMatrix).index(0)
y=ind//6
x=ind%6
if (x,y)==prevZeroInd:
startMatrix[y][x]=7
else:
possibles=getPossiblesForSquare(x,y,startMatrix)
if len(possibles)==0:
startMatrix[y][x]=7
else:
startMatrix[y][x]=possibles[0]
if iteration <= 6:
return getPossibleMatrix(startMatrix, iteration+1, yon, (x,y)) # <<BUG
else:
if yon!=4:
return getPossibleMatrix(startMatrix, 0, yon+1, (x,y))
MATRIX=[[1,2,3,4,5,6],
[2,0,0,0,0,5],
[3,0,0,0,0,4],
[4,0,0,0,0,3],
[5,0,0,0,0,2],
[6,5,4,3,2,1]]
result=getPossibleMatrix(MATRIX)
pprint(result)
Why your script hangs:
Essentially your script encounters problems here:
for item in list(set(rowon + colon)): # remove duplicates
if not (item == 0) and (item in possibles):
del possibles[possibles.index(item)]
At the third iteration, for the third cell your if condition is evaluated as true for all possible values [1 to 6] (if you output the matrix you will see that the set() you are creating contains all elements), so you always return zero, re-check the values, return zero ad infinitum.
If you're looking to brute-force a solution out of this, you might want to update the sqx and sqy to go to a different cell when possibles is empty.
Another additional small mistake I located was:
# you are checking values 1-5 and not 1-6!
possibles = list(range(1, 6)) # should be range(7) [exclusive range]
Don't forget that range is exclusive, it doesn't include (excludes) the upper limit.
There exist of course, different ways to tackle this problem.
A possible -alternate- solution:
Read this for the general, alternate view of how to solve this. If you do not want to see a possible solution, skip the 'code' part.
The solution matrix (one of the possible ones) has this form (unless I am making a horrible mistake):
MATRIX = [[1, 2, 3, 4, 5, 6],
[2, 3, 6, 1, 4, 5],
[3, 1, 5, 2, 6, 4],
[4, 6, 2, 5, 1, 3],
[5, 4, 1, 6, 3, 2],
[6, 5, 4, 3, 2, 1]]
The Logic is as follows:
You must observe the symmetry present in the matrix. Specifically, every row and every column displays a 'flip and reverse' symmetry. For example, the first and last rows are connected by this equation :
row[0] = REV(flip(row[n]))
Similarly, all additional rows (or columns) have their corresponding counterpart:
row[1] = REV(flip(row[n-1]))
and so on.
So, for n=6 this essentially boils down to finding the (n / 2) -1 (because we already know the first and last row!) and afterwards flipping them (not the finger), reversing them and assigning them to their corresponding rows.
In order to find these values we can observe the matrix as a combination of smaller matrices:
These make the first two (unknown) rows of the matrix:
sub_matrix = [[1, 2, 3, 4, 5, 6],
[2, 0, 0, 0, 0, 5],
[3, 0, 0, 0, 0, 4],
[6, 5, 4, 3, 2, 1]]
and the other two rows can be made by finding the correct values for these two.
Observe the restrictions in hand:
In column [1][1] and [1][m-1] we cannot:
place a 2 or a 5
In columns [1][2] and [1][m-2] we cannot:
place the previous values ([2, 5]) along with ([3, 4]) so, we cannot have a value in [2,3,4,5]
For the inner columns we're left with the set set(1-6) - set(2-5) = [1, 6]
and since we get a normal row and a single inverted and flipped row for this, we can arbitrarily select a value and add it as a column value.
By using another list we can keep track of the values used and fill out the rest of the cells.
Coding this: (Spoilers)
Note: I did not use numpy for this. You can and should though. (Also, Python 2.7)
Also, I did not use recursion for this (you can try to, by finding the same matrix for bigger values of n [I believe it's a good fit for a recursive function].
First, in order to not type this all the time, you can create a n x n matrix as follows: (This isn't much of a spoiler.)
# our matrix must be n x n with n even.
n = 6
# Create n x n matrix.
head = [i for i in xrange(1, n + 1)] # contains values from 1 to n.
zeros = [0 for i in xrange(1, n-1)] # zeros
tail = [i for i in xrange(n, 0, -1)] # contains values from n to 1.
# Add head and zeros until tail.
MATRIX = [([j] + zeros + [(n+1)-j]) if j != 1 else head for j in xrange(1, n)]
# Add tail
MATRIX.append(tail)
Then, create the smaller (n/2 + 1) x n array:
# Split matrix and add last row.
sub_matrix = MATRIX[:(n / 2)] + [tail]
Afterwards, a small function called sub = fill_rows(sub_matrix) comes in and takes care of business:
def fill_rows(mtrx):
# length of the sub array (= 4)
sub_n = len(mtrx)
# From 1 because 0 = head
# Until sub_n -1 because the sub_n - 1 is = tail (remember, range is exclusive)
for current_row in xrange(1, sub_n - 1):
print "Current row: " + str(current_row)
# -- it gets messy here --
# get values of inner columns and filter out the zeros (matrix[row][n / 2] == 0 evaluates to False)
col_vals_1 = [mtrx[row][n / 2] for row in xrange(0, sub_n) if mtrx[row][(n / 2)]]
col_vals_2 = [mtrx[row][(n / 2) - 1] for row in xrange(0, sub_n) if mtrx[row][(n / 2) - 1]]
col_vals = col_vals_1 + col_vals_2
# print "Column Values = " + str(col_vals)
row_vals = [mtrx[current_row][col] for col in xrange(0, n) if mtrx[current_row][col]]
# print "Row Values = " + str(row_vals)
# Find the possible values by getting the difference of the joined set of column + row values
# with the range from (1 - 6).
possible_values = list(set(xrange(1, n + 1)) - set(row_vals + col_vals))
print "Possible acceptable values: " + str(possible_values)
# Add values to matrix (pop to remove them)
# After removing add to the list of row_values in order to check for the other columns.
mtrx[current_row][(n-1)/2] = possible_values.pop()
row_vals.append(mtrx[current_row][(n - 1) / 2])
mtrx[current_row][(n-1)/2 + 1] = possible_values.pop()
row_vals.append(mtrx[current_row][(n-1) / 2 + 1])
# New possible values for remaining columns of the current row.
possible_values = list(set(xrange(1, n + 1)) - set(row_vals))
print "Possible acceptable values: " + str(possible_values)
# Add values to the cells.
mtrx[current_row][(n - 2)] = possible_values.pop()
mtrx[current_row][1] = possible_values.pop()
# return updated sub-matrix.
return mtrx
The only thing left to do now is take those two rows, flip them, reverse them and add the head and tail to them:
print '=' * 30 + " Out " + "=" * 30
# Remove first and last rows.
del sub[0]
sub.pop()
# reverse values in lists
temp_sub = [l[::-1] for l in sub]
# reverse lists in matrix.
temp_sub.reverse()
# Add them and Print.
pprint([head] + sub + temp_sub + [tail])
This outputs what, I hope, is the right matrix:
============================== Out ==============================
[[1, 2, 3, 4, 5, 6],
[2, 3, 6, 1, 4, 5],
[3, 1, 5, 2, 6, 4],
[4, 6, 2, 5, 1, 3],
[5, 4, 1, 6, 3, 2],
[6, 5, 4, 3, 2, 1]]
Additionally
By using this way of solving it the answer to the problem in hand becomes more easy. Viewing the matrix as a combination of these sub-matrices you can tell how many of these combinations might be possible.
As a closing note, it would be a good work-out to modify it a bit in order to allow it to find this array for an arbitrary (but even) number of n.
You have infinite recursion. Your first three iterations are fine: your second row adds one possibility at a time, turning from 200005 into 214305. At this point, you find that there are no possible choices. You overwrite the existing 0 with a new 0, and then fail to backtrack. Each of these is an error: 6 is a possible value, and you have no recovery code.
Here is the alteration I made to track the problem; additions are in double-star containers. When you have a sick program, learn how to ask where it hurts. The print function is an excellent instrument.
def getPossibleMatrix(startMatrix, **iter=0**):
if 0 not in flattenList(startMatrix):
return startMatrix
else:
# find&fill in the first blank one
ind=flattenList(startMatrix).index(0)
y=ind//6
x=ind%6
possibles=getPossiblesForSquare(x,y,startMatrix)
if len(possibles)==0:
startMatrix[y][x]=0
**print ("No possibles; current location remains 0")**
else:
startMatrix[y][x]=possibles[0]
****print ("Added first possible")
print (startMatrix)
if iter <= 6:**
return getPossibleMatrix(startMatrix, **iter+1**) # <<BUG
Given the following problem, what is the most efficient (or reasonably efficient) way to do this in Python:
Problem. Given a list of lists,
L = [list_0, list_1, list_2, list_3, ..., list_n]
where len(list_i) <= 3, let's say, for each list inside of L. How can we split up L into L_1, L_2, L_3, where L_1 has only length 1 lists, L_2 has only length 2 lists, and L_3 has only length 3 lists?
Potential Solutions. Here's the best I could do; I've also included a sample set here as well. It runs in around 8.6 seconds on my PC.
import time
# These 4 lines make a large sample list-of-list to test on.
asc_sample0 = [[i] for i in range(500)]
asc_sample1 = [[i,j] for i in range(500) for j in range(20)]
asc_sample2 = [[i,j,k] for i in range(20) for j in range(10) for k in range(20)]
asc_sample = asc_sample0 + asc_sample1 + asc_sample2
start = time.clock()
cells0 = [i for i in asc if len(i) == 1]
cells1 = [i for i in asc if len(i) == 2]
cells2 = [i for i in asc if len(i) == 3]
print time.clock() - start
I also attempted to "pop" elements off and append to lists cells0, etc., but this took significantly longer. I also attempted to append and then remove that element so I could get through in one loop which worked okay when there were, say, 10^10 lists of size 1, but only a few of size 2 and 3, but, in general, it was not efficient.
I'd mostly appreciate some neat ideas. I know that one of the answers will most likely be "Write this in C", but for now I'd just like to look at Python solutions for this.
An old fashioned solution might work better here:
cells0, cells1, cells2 = [], [], []
for lst in asc_sample:
n = len(lst)
if n == 1:
cells0.append(lst)
elif n == 2:
cells1.append(lst)
else:
cells2.append(lst)
This is definitely one of the best because it runs in parallel. Another thing that you should look at though is the itertools.groupby and the built-in filter method.
result = dict()
for lst in L:
result.setdefault(len(lst), []).append(lst)
print result
Output
{
1: [[0], [1], [2], [3]],
2: [[0, 0], [0, 1], [0, 2]],
3: [[0, 0, 0], [0, 0, 1], [0, 0, 2]]
}
Indexing a list/tuple should be faster than doing key lookups. This is about 30% faster than the version given in the question
cells = [],[],[],[] # first list here isn't used, but it's handy for the second version
for i in asc:
cells[len(i)].append(i)
Slightly faster again by extracting the append methods (On larger lists this is almost twice as fast as the OP)
cells = [],[],[],[]
appends = [x.append for x in cells]
for i in asc:
appends[len(i)](i)
I'm learning Python and I have a problem with this seems to be simple task.
I want to find all possible combination of numbers that sum up to a given number.
for example: 4 -> [1,1,1,1] [1,1,2] [2,2] [1,3]
I pick the solution which generate all possible subsets (2^n) and then yield just those that sum is equal to the number. I have a problem with the condition. Code:
def allSum(number):
#mask = [0] * number
for i in xrange(2**number):
subSet = []
for j in xrange(number):
#if :
subSet.append(j)
if sum(subSet) == number:
yield subSet
for i in allSum(4):
print i
BTW is it a good approach?
Here's some code I saw a few years ago that does the trick:
>>> def partitions(n):
if n:
for subpart in partitions(n-1):
yield [1] + subpart
if subpart and (len(subpart) < 2 or subpart[1] > subpart[0]):
yield [subpart[0] + 1] + subpart[1:]
else:
yield []
>>> print list(partitions(4))
[[1, 1, 1, 1], [1, 1, 2], [2, 2], [1, 3], [4]]
Additional References:
http://mathworld.wolfram.com/Partition.html
http://en.wikipedia.org/wiki/Partition_(number_theory)
http://www.site.uottawa.ca/~ivan/F49-int-part.pdf
Here is an alternate approach which works by taking a list of all 1s and recursively collapsing it by adding subsequent elements, this should be more efficient than generating all possible subsets:
def allSum(number):
def _collapse(lst):
yield lst
while len(lst) > 1:
lst = lst[:-2] + [lst[-2] + lst[-1]]
for prefix in _collapse(lst[:-1]):
if not prefix or prefix[-1] <= lst[-1]:
yield prefix + [lst[-1]]
return list(_collapse([1] * number))
>>> allSum(4)
[[1, 1, 1, 1], [1, 1, 2], [2, 2], [1, 3], [4]]
>>> allSum(5)
[[1, 1, 1, 1, 1], [1, 1, 1, 2], [1, 2, 2], [1, 1, 3], [2, 3], [1, 4], [5]]
You can strip off the last value if you don't want the trivial case. If you will just be looping over the results remove the list call and just return the generator.
This is equivalent to the problem described in this question and can use a similar solution.
To elaborate:
def allSum(number):
for solution in possibilites(range(1, number+1), number):
expanded = []
for value, qty in zip(range(1, number+1), solution):
expanded.extend([value]*qty)
yield expanded
That translates this question into that question and back again.
That solution doesn't work, right? It will never add a number to a subset more than once, so you will never get, for example, [1,1,2]. It will never skip a number, either, so you will never get, for example, [1,3].
So the problem with your solution is twofold: One, you are not actually generating all possible subsets in the range 1..number. Two, The set of all subsets will exclude things that you should be including, because it will not allow a number to appear more than once.
This kind of problem can be generalized as a search problem. Imagine that the numbers you want to try are nodes on a tree, and then you can use depth-first search to find all paths through the tree that represent a solution. It's an infinitely large tree, but luckily, you never need to search all of it.
Can someone please explain algorithm for itertools.permutations routine in Python standard lib 2.6? I don't understand why it works.
Code is:
def permutations(iterable, r=None):
# permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
# permutations(range(3)) --> 012 021 102 120 201 210
pool = tuple(iterable)
n = len(pool)
r = n if r is None else r
if r > n:
return
indices = range(n)
cycles = range(n, n-r, -1)
yield tuple(pool[i] for i in indices[:r])
while n:
for i in reversed(range(r)):
cycles[i] -= 1
if cycles[i] == 0:
indices[i:] = indices[i+1:] + indices[i:i+1]
cycles[i] = n - i
else:
j = cycles[i]
indices[i], indices[-j] = indices[-j], indices[i]
yield tuple(pool[i] for i in indices[:r])
break
else:
return
You need to understand the mathematical theory of permutation cycles, also known as "orbits" (it's important to know both "terms of art" since the mathematical subject, the heart of combinatorics, is quite advanced, and you may need to look up research papers which could use either or both terms).
For a simpler introduction to the theory of permutations, wikipedia can help. Each of the URLs I mentioned offers reasonable bibliography if you get fascinated enough by combinatorics to want to explore it further and gain real understanding (I did, personally -- it's become somewhat of a hobby for me;-).
Once you understand the mathematical theory, the code is still subtle and interesting to "reverse engineer". Clearly, indices is just the current permutation in terms of indices into the pool, given that the items yielded are always given by
yield tuple(pool[i] for i in indices[:r])
So the heart of this fascinating machinery is cycles, which represents the permutation's orbits and causes indices to be updated, mostly by the statements
j = cycles[i]
indices[i], indices[-j] = indices[-j], indices[i]
I.e., if cycles[i] is j, this means that the next update to the indices is to swap the i-th one (from the left) with the j-th one from the right (e.g., if j is 1, then the last element of indices is being swapped -- indices[-1]). And then there's the less frequent "bulk update" when an item of cycles reached 0 during its decrements:
indices[i:] = indices[i+1:] + indices[i:i+1]
cycles[i] = n - i
this puts the ith item of indices at the very end, shifting all following items of indices one to the left, and indicates that the next time we come to this item of cycles we'll be swapping the new ith item of indices (from the left) with the n - ith one (from the right) -- that would be the ith one again, except of course for the fact that there will be a
cycles[i] -= 1
before we next examine it;-).
The hard part would of course be proving that this works -- i.e., that all permutations are exhaustively generated, with no overlap and a correctly "timed" exit. I think that, instead of a proof, it may be easier to look at how the machinery works when fully exposed in simple cases -- commenting out the yield statements and adding print ones (Python 2.*), we have
def permutations(iterable, r=None):
# permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
# permutations(range(3)) --> 012 021 102 120 201 210
pool = tuple(iterable)
n = len(pool)
r = n if r is None else r
if r > n:
return
indices = range(n)
cycles = range(n, n-r, -1)
print 'I', 0, cycles, indices
# yield tuple(pool[i] for i in indices[:r])
print indices[:r]
while n:
for i in reversed(range(r)):
cycles[i] -= 1
if cycles[i] == 0:
print 'B', i, cycles, indices
indices[i:] = indices[i+1:] + indices[i:i+1]
cycles[i] = n - i
print 'A', i, cycles, indices
else:
print 'b', i, cycles, indices
j = cycles[i]
indices[i], indices[-j] = indices[-j], indices[i]
print 'a', i, cycles, indices
# yield tuple(pool[i] for i in indices[:r])
print indices[:r]
break
else:
return
permutations('ABC', 2)
Running this shows:
I 0 [3, 2] [0, 1, 2]
[0, 1]
b 1 [3, 1] [0, 1, 2]
a 1 [3, 1] [0, 2, 1]
[0, 2]
B 1 [3, 0] [0, 2, 1]
A 1 [3, 2] [0, 1, 2]
b 0 [2, 2] [0, 1, 2]
a 0 [2, 2] [1, 0, 2]
[1, 0]
b 1 [2, 1] [1, 0, 2]
a 1 [2, 1] [1, 2, 0]
[1, 2]
B 1 [2, 0] [1, 2, 0]
A 1 [2, 2] [1, 0, 2]
b 0 [1, 2] [1, 0, 2]
a 0 [1, 2] [2, 0, 1]
[2, 0]
b 1 [1, 1] [2, 0, 1]
a 1 [1, 1] [2, 1, 0]
[2, 1]
B 1 [1, 0] [2, 1, 0]
A 1 [1, 2] [2, 0, 1]
B 0 [0, 2] [2, 0, 1]
A 0 [3, 2] [0, 1, 2]
Focus on the cycles: they start as 3, 2 -- then the last one is decremented, so 3, 1 -- the last isn't zero yet so we have a "small" event (one swap in the indices) and break the inner loop. Then we enter it again, this time the decrement of the last gives 3, 0 -- the last is now zero so it's a "big" event -- "mass swap" in the indices (well there's not much of a mass here, but, there might be;-) and the cycles are back to 3, 2. But now we haven't broken off the for loop, so we continue by decrementing the next-to-last (in this case, the first) -- which gives a minor event, one swap in the indices, and we break the inner loop again. Back to the loop, yet again the last one is decremented, this time giving 2, 1 -- minor event, etc. Eventually a whole for loop occurs with only major events, no minor ones -- that's when the cycles start as all ones, so the decrement takes each to zero (major event), no yield occurs on that last cycle.
Since no break ever executed in that cycle, we take the else branch of the for, which returns. Note that the while n may be a bit misleading: it actually acts as a while True -- n never changes, the while loop only exits from that return statement; it could equally well be expressed as if not n: return followed by while True:, because of course when n is 0 (empty "pool") there's nothing more to yield after the first, trivial empty yield. The author just decided to save a couple of lines by collapsing the if not n: check with the while;-).
I suggest you continue by examining a few more concrete cases -- eventually you should perceive the "clockwork" operating. Focus on just cycles at first (maybe edit the print statements accordingly, removing indices from them), since their clockwork-like progress through their orbit is the key to this subtle and deep algorithm; once you grok that, the way indices get properly updated in response to the sequencing of cycles is almost an anticlimax!-)
It is easier to answer with a pattern in results than words(Except you want to know the math part of the theory),
so prints out would be the best way to explain.
The most subtle thing is that,
after looping to the end, it would reset itself to the first turn of the last round, and start the next looping down, or continually reset to first turn of the last even the bigger round, like a clock.
The part of code doing the reset job:
if cycles[i] == 0:
indices[i:] = indices[i+1:] + indices[i:i+1]
cycles[i] = n - i
whole:
In [54]: def permutations(iterable, r=None):
...: # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
...: # permutations(range(3)) --> 012 021 102 120 201 210
...: pool = tuple(iterable)
...: n = len(pool)
...: r = n if r is None else r
...: if r > n:
...: return
...: indices = range(n)
...: cycles = range(n, n-r, -1)
...: yield tuple(pool[i] for i in indices[:r])
...: print(indices, cycles)
...: while n:
...: for i in reversed(range(r)):
...: cycles[i] -= 1
...: if cycles[i] == 0:
...: indices[i:] = indices[i+1:] + indices[i:i+1]
...: cycles[i] = n - i
...: print("reset------------------")
...: print(indices, cycles)
...: print("------------------")
...: else:
...: j = cycles[i]
...: indices[i], indices[-j] = indices[-j], indices[i]
...: print(indices, cycles, i, n-j)
...: yield tuple(pool[i] for i in indices[:r])
...: break
...: else:
...: return
part of the result:
In [54]: list(','.join(i) for i in permutations('ABCDE', 3))
([0, 1, 2, 3, 4], [5, 4, 3])
([0, 1, 3, 2, 4], [5, 4, 2], 2, 3)
([0, 1, 4, 2, 3], [5, 4, 1], 2, 4)
reset------------------
([0, 1, 2, 3, 4], [5, 4, 3])
------------------
([0, 2, 1, 3, 4], [5, 3, 3], 1, 2)
([0, 2, 3, 1, 4], [5, 3, 2], 2, 3)
([0, 2, 4, 1, 3], [5, 3, 1], 2, 4)
reset------------------
([0, 2, 1, 3, 4], [5, 3, 3])
------------------
([0, 3, 1, 2, 4], [5, 2, 3], 1, 3)
([0, 3, 2, 1, 4], [5, 2, 2], 2, 3)
([0, 3, 4, 1, 2], [5, 2, 1], 2, 4)
reset------------------
([0, 3, 1, 2, 4], [5, 2, 3])
------------------
([0, 4, 1, 2, 3], [5, 1, 3], 1, 4)
([0, 4, 2, 1, 3], [5, 1, 2], 2, 3)
([0, 4, 3, 1, 2], [5, 1, 1], 2, 4)
reset------------------
([0, 4, 1, 2, 3], [5, 1, 3])
------------------
reset------------------(bigger reset)
([0, 1, 2, 3, 4], [5, 4, 3])
------------------
([1, 0, 2, 3, 4], [4, 4, 3], 0, 1)
([1, 0, 3, 2, 4], [4, 4, 2], 2, 3)
([1, 0, 4, 2, 3], [4, 4, 1], 2, 4)
reset------------------
([1, 0, 2, 3, 4], [4, 4, 3])
------------------
([1, 2, 0, 3, 4], [4, 3, 3], 1, 2)
([1, 2, 3, 0, 4], [4, 3, 2], 2, 3)
([1, 2, 4, 0, 3], [4, 3, 1], 2, 4)
I recently stumbled upon the very same question during my journey of reimplementing permutation algorithms, and would like to share my understanding of this interesting algorithm.
TL;DR: This algorithm is based on a recursive permutation generation algorithm (backtracking based and utilizes swapping elements), and is transformed (or optimized) into an iteration form. (possibly to improve efficiency and prevent stack overflow)
Basics
Before we start, I have to make sure we use the same notation as the original algorithm.
n refers to the length of iterable
r refers to the length of one output permutation tuple
And share a simple observation (as discussed by Alex):
Whenever the algorithm yield an output, it just takes the first r elements of the indices list.
cycles
First, let’s discuss the variable cycles and build some intuition. With some debugging prints, we can see that cycles act like a countdown (of time or clock, something like 01:00:00 -> 00:59:59 -> 00:59:58):
Every item is initialized to range(n, n-r, -1), resulting in cycles[0]=n, cycles[1]=n-1...cycles[i]=n-i
Usually, only the last element is decreased, and each decrement (given after the decrement cycles[r-1] !=0) yields an output (a permutation tuple). We can intuitively name this case tick.
Whenever an element (assuming that’s cycles[i]) decreases to 0, it triggers a decrease on the element before it (cycles[i-1]). Then the triggering element (cycles[i]) is restored to its initial value (n-i). This behavior is similar to a borrowed minus, or the reset of minutes when the second reaches 0 in a clock countdown. We can intuitively name this branch reset.
To further confirm our intuition, add some print statements to the algorithm, and run it with the parameter iterable="ABCD", r=2. We can see the following changes of the cycles variable. Note that square brackets indicate a “tick” happening, yielding an output, and the curly braces indicates a “reset” happening, which don’t yield output.
[4,3] -> [4,2] -> [4,1] -> {4,0} -> {4,3} ->
[3,3] -> [3,2] -> [3,1] -> {3,0} -> {3,3} ->
[2,3] -> [2,2] -> [2,1] -> {2,0} -> {2,3} ->
[1,3] -> [1,2] -> [1,1] -> {1,0} -> {1,3} -> {0,3} -> {4,3}
Using the initial values and change pattern of cycles, we can come to a possible interpretation of the meaning of cycles: number of the remaining permutations (outputs), at each index. When initialized, cycles[0]=n represents that there is initially n possible choices at index 0, and cycles[1]=n-1 represents that there is initially n-1 possible choices at index 1, all the way down to cycles[r-1]=n-r+1. This interpretation of cycles matches math, as with some simple combinational math calculation we can confirm that is indeed the case. Another supporting evidence is that whenever the algorithm ends, we have P(n,r) ( P(n,r)=n*(n-1)*...*(n-r+1) ) ticks (counting the initial yield before entering while as a tick).
indices
Now we come to the more complex part, the indices list. As this is essentially a recursive algorithm (more precisely backtracking), I would like to start from a sub-problem (i=r-1): When the value from index 0 to index r-2 (inclusive) in indices is fixed, and only the value at index r-1 (in other words, the last element in indices) is changing. Also, I will introduce a concrete example (iterable="ABCDE", r=3), and we will be focusing on how it generates the first 3 outputs: ABC, ABD, ABE.
Following the sub-problem, we split the list of indices into 3 parts, and give them names,
fixed : indices[0:r-2] (inclusive)
changing: indices[r-1] (only one value)
backlog: indices[r:n-1] (the remaining parts beside the first two)
As this is a backtracking algorithm, we need to keep an invariant unmodified before and after the execution. The invariant is
The sublist contains changing and backlog (indices[r-1:n-1]), which is modified during the execution but restored when it ends.
Now we can turn to the interaction between cycles and indices during the mysterious while loop. Some of the operations have been outlined by Alex, and I further elaborate.
In each tick, the element in the changing part is swapped with some element in the backlog part, and the relative order in the backlog part is maintained.
Using the characters to visualize the indices, and curly braces highlights the backlog part:
ABC{DE} -> ABD{CE} -> ABE{CD}
When reset happens, the element in the changing part is moved to the back of backlog, thus restoring the initial layout of the sublist (containing the changing part and the backlog part)
Using the characters to visualize the indices, and curly braces highlights the changing part:
AB{E}CD -> ABCD{E}
During this execution (of i=r-1), only the tick phase can yield outputs, and it yields n-r+1 outputs in total, matching the initial value of cycles[i]. This is also a result of mathematically we can only have n-r+1 permutation choices when the fixed part is fixed.
After cycles[i] is decreased to 0, the reset phase kicks in, resetting cycles[i] to n-r+1 and restoring the invariant sublist. This phase marks the end of this execution, and indicates that all possible permutation choices giving the fixed prefix part have been outputted.
Therefore, we have shown that, in this sub-problem (i=r-1), this algorithm is indeed a valid backtracking algorithm, as it
Outputs all possible values given the precondition (fixed prefix part)
Keeps the invariant unmodified (restored in reset phase)
This proof(?) can also be generalized to other values of i, thus proofing(?) the correctness of this permutation generation algorithm.
Reimplementation
Phew! That’s a long read, and you may want to have some more tinkering (more print) with the algorithm to be fully convinced. In essence, we can simplify the underlying principle of the algorithm as the following pseudo-code:
// precondition: the fixed part (or prefix) is fixed
OUTPUT initial_permutation // also invokes the next level
WHILE remaining_permutation_count > 0
// tick
swap the changing element with an element in backlog
OUTPUT current_permutation // also invokes the next level
// reset
move the changing element behind the backlog
And here is a Python implementation using simple backtracking:
# helpers
def swap(list, i, j):
list[i], list[j] = list[j], list[i]
def move_to_last(list, i):
list[i:] = list[i+1:] + [list[i]]
def print_first_n_element(list, n):
print("".join(list[:n]))
# backtracking dfs
def permutations(list, r, changing_index):
if changing_index == r:
# we've reached the deepest level
print_first_n_element(list, r)
return
# a pseudo `tick`
# process initial permutation
# which is just doing nothing (using the initial value)
permutations(list, r, changing_index + 1)
# note: initial permutaion has been outputed, thus the minus 1
remaining_choices = len(list) - 1 - changing_index
# for (i=1;i<=remaining_choices;i++)
for i in range(1, remaining_choices+1):
# `tick` phases
# make one swap
swap_idx = changing_index + i
swap(list, changing_index, swap_idx)
# finished one move at current level, now go deeper
permutations(list, r, changing_index + 1)
# `reset` phase
move_to_last(list, changing_index)
# wrapper
def permutations_wrapper(list, r):
permutations(list, r, 0)
# main
if __name__ == "__main__":
my_list = ["A", "B", "C", "D"]
permutations_wrapper(my_list, 2)
Now all the remaining step is just to show that the backtracking version is equivalent to the iteration version in itertools source code. It should be pretty easy once you grasp why this algorithm works. Following the great tradition of various CS textbooks, this is left as an exercise to the reader.