Develop Reference

Why is my code to find combinations so slow for big numbers?

My code find all combinations of a list of numbers with a given sum. The code is working well, but when trying big numbers (like 100 or 200), the code is taking way too long.
Any advices on how to make the code much faster ?
def check(target, lst):
def _a(idx, l, r, t):
if t == sum(l): r.append(l)
elif t < sum(l): return
for u in range(idx, len(lst)):
_a(u, l + [lst[u]], r, t)
return r
return len(_a(0, [], [], target))
print(check(200, (1, 2, 5, 10, 20, 50, 100, 200, 500)))

Make the inner function simpler (only give it index and remaining target, and return the number) and then memoize it?
from functools import lru_cache
def check(target, lst):
#lru_cache(None)
def a(idx, t):
if t == 0: return 1
elif t < 0: return 0
return sum(a(u, t - lst[u])
for u in range(idx, len(lst)))
return a(0, target)
print(check(200, (1, 2, 5, 10, 20, 50, 100, 200, 500)))

you could use itertools to iterate through every combination of every possible size, and filter out everything that doesn't sum to 10:
import itertools
numbers = [1, 2, 3, 7, 7, 9, 10]
result = [seq for i in range(len(numbers), 0, -1) for seq in
itertools.combinations(numbers, i) if sum(seq) == 10]
print result
result
[(1, 2, 7), (1, 2, 7), (1, 9), (3, 7), (3, 7), (10,)]
Unfortunately this is something like O(2^N) complexity, so it isn't suitable for input lists larger than, say, 20 elements.

Product of prime factors of a number, less than that number

First of all, I apologise for the title, I did not know how to put my problem in words. Well, here it is:
For an integer a greater than 1, let F be a sorted list of prime factors of a. I need to find all tuples c (filled with whole numbers), such that length of each tuple is equal to the size of F and (F[0] ** c[0]) * (F[1] ** c[1]) * (...) < a. I should add that I write in Python.
Example:
a = 60
F = [2,3,5]
# expected result:
C = {(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 1, 0), (0, 1, 1), (0, 2, 0),
(0, 2, 1), (0, 3, 0), (1, 0, 0), (1, 0, 1), (1, 0, 2), (1, 1, 0), (1, 1, 1),
(1, 2, 0), (1, 3, 0), (2, 0, 0), (2, 0, 1), (2, 1, 0), (2, 2, 0), (3, 0, 0),
(3, 0, 1), (3, 1, 0), (4, 0, 0), (4, 1, 0), (5, 0, 0)}
I generated this result using itertools.product(), specifically:
m = math.floor(round(math.log(a, min(F)), 12))
for i in itertools.product(range(m + 1), repeat=len(F)):
if math.prod([F[j] ** i[j] for j in range(len(F))]) < a: print(i)
I think it works but it's inefficient. For example number 5 appears only in one tuple, but was checked many more times! Is there any way to make it faster? I would use multiple while loops (with break statements) but since I don't know what is the length of F, I don't think that is possible.

You base all your range limits on just min(F). Let's customize each to the log(a, factor) to reduce the cases:
from math import ceil, log, prod
from itertools import product
a = 60
F = [2, 3, 5]
ranges = [range(0, ceil(log(a, factor))) for factor in F]
C = []
for powers in product(*ranges):
if prod(F[i] ** power for i, power in enumerate(powers)) < a:
C.append(powers)
print(C)
By my measure, your code generates 216 test cases to come up with 25 results, but the above code only generates 1/3 of those test cases.

You could iterate over all the "valid" tuples with a generator, like so:
def exponent_tuples(prime_factors, limit):
def next_tuple(t):
n = math.prod(f ** tt for f, tt in zip(prime_factors, t))
for idx, (f, tt) in enumerate(zip(prime_factors, t)):
n *= f
if n < limit:
return (0,) * idx + (tt + 1,) + t[idx + 1 :]
n //= f**(tt+1)
return None
t = (0,) * len(prime_factors)
while t is not None:
yield t
t = next_tuple(t)
for t in exponent_tuples([2, 3, 5], 60):
print(t)
The idea here is to basically increment the tuple entries like digits of a number and have the respective digit roll over to zero and carry the 1 whenever you reach the defined limit.
I'm pretty sure this does exactly what you want, except for maybe the order in which it yields the tuples (can be adjusted by modifying the next_tuple function)
EDIT: Simplified the code a bit

The almost cooked proposition would go like this (shell execution)
>>> max_exponents(42,[2,3,7])
[5, 3, 1]
>>> #pick 2
>>> max_exponents(42//2**2,[3,7])
[2, 1]
>>> #pick 1
>>> max_exponents(42//(2**2*3**1),[7])
[0]
I'm almost done. This will adapt to any number of factors !

Somehow your proposition reduces to this (more readable form ?)
import math as m
import pprint
a = 60
prime_factors = [2,3,5]
exponents =list(map(lambda x:m.floor(m.log(a,x)),prime_factors))
rez = []
for i in range(exponents[0]+1):
for j in range(exponents[1]+1):
for k in range(exponents[2]+1):
if 2**i*3**j*5**k <= a:
rez.append((i,j,k))
pprint.pprint(rez)
and you would like to know wether there's a way to make if faster (with less tests). So we're no more on the implementation side, but more on the conception (algorithm) side ?
For example, once the first exponent c[0] has been chosen, the next ones should be selected amongst the one fitting in a//(2**c[a]) as the other answerer proposed i guess

Python, permutation to permuation-index function

I have some permutations of a list:
>>> import itertools
>>> perms = list(itertools.permutations([0,1,2,3]))
>>> perms
[(0, 1, 2, 3), (0, 1, 3, 2), (0, 2, 1, 3), (0, 2, 3, 1), (0, 3, 1, 2), (0, 3, 2, 1), (1, 0, 2, 3), (1, 0, 3, 2), (1, 2, 0, 3), (1, 2, 3, 0), (1, 3, 0, 2), (1, 3, 2, 0), (2, 0, 1, 3), (2, 0, 3, 1), (2, 1, 0, 3), (2, 1, 3, 0), (2, 3, 0, 1), (2, 3, 1, 0), (3, 0, 1, 2), (3, 0, 2, 1), (3, 1, 0, 2), (3, 1, 2, 0), (3, 2, 0, 1), (3, 2, 1, 0)]
>>> len(perms)
24
What function can I use (without access to the list perm) to get the index of an arbitrary permutation, e.g. (0, 2, 3, 1) -> 3?
(You can assume that permuted elements are always an ascending list of integers, starting at zero.)
Hint: The factorial number system may be involved. https://en.wikipedia.org/wiki/Factorial_number_system

Off the top of my head I came up with the following, didn't test it thoroughly.
from math import factorial
elements = list(range(4))
permutation = (3, 2, 1, 0)
index = 0
nf = factorial(len(elements))
for n in permutation:
nf //= len(elements)
index += elements.index(n) * nf
elements.remove(n)
print(index)
EDIT: replaced nf /= len(elements) with nf //= len(elements)

I suppose this is a challenge, so here is my (recursive) answer:
import math
import itertools
def get_index(l):
# In a real function, there should be more tests to validate that the input is valid, e.g. len(l)>0
# Terminal case
if len(l)==1:
return 0
# Number of possible permutations starting with l[0]
span = math.factorial(len(l)-1)
# Slightly modifying l[1:] to use the function recursively
new_l = [ val if val < l[0] else val-1 for val in l[1:] ]
# Actual solution
return get_index(new_l) + span*l[0]
get_index((0,1,2,3))
# 0
get_index((0,2,3,1))
# 3
get_index((3,2,1,0))
# 23
get_index((4,2,0,1,5,3))
# 529
list(itertools.permutations((0,1,2,3,4,5))).index((4,2,0,1,5,3))
# 529

You need to write your own function. Something like this would work
import math
def perm_loc(P):
N = len(P)
assert set(P) == set(range(N))
def rec(perm):
nums = set(perm)
if not perm:
return 0
else:
sub_res = rec(perm[1:]) # Result for tail of permutation
sub_size = math.factorial(len(nums) - 1) # How many tail permutations exist
sub_index = sorted(nums).index(perm[0]) # Location of first element in permutaiotn
# in the sorted list of number
return sub_index * sub_size + sub_res
return rec(P)
The function that does all the work is rec, with perm_loc just serving as a wrapper around it. Note that this algorithm is based on the nature of the permutation algorithm that itertools.permutation happens to use.
The following code tests the above function. First on your sample, and then on all permutations of range(7):
print perm_loc([0,2,3,1]) # Print the result from the example
import itertools
def test(N):
correct = 0
perms = list(itertools.permutations(range(N)))
for (i, p) in enumerate(perms):
pl = perm_loc(p)
if i == pl:
correct += 1
else:
print ":: Incorrect", p, perms.index(p), perm_loc(N, p)
print ":: Found %d correct results" % correct
test(7) # Test on all permutations of range(7)

from math import factorial
def perm_to_permidx(perm):
# Extract info
n = len(perm)
elements = range(n)
# "Gone"s will be the elements of the given perm
gones = []
# According to each number in perm, we add the repsective offsets
offset = 0
for i, num in enumerate(perm[:-1], start=1):
idx = num - sum(num > gone for gone in gones)
offset += idx * factorial(n - i)
gones.append(num)
return offset
the_perm = (0, 2, 3, 1)
print(perm_to_permidx(the_perm))
# 3
Explanation: All permutations of a given range can be considered as a groups of permutations. So, for example, for the permutations of 0, 1, 2, 3 we first "fix" 0 and permute rest, then fix 1 and permute rest, and so on. Once we fix a number, the rest is again permutations; so we again fix a number at a time from the remaining numbers and permute the rest. This goes on till we are left with one number only. Every level of fixing has a corresponding (n-i)! permutations.
So this code finds the "offsets" for each level of permutation. The offset corresonds to where the given permutation starts when we fix numbers of perm in order. For the given example of (0, 2, 3, 1), we first look at the first number in the given perm which is 0, and figure the offset as 0. Then this goes to gones list (we will see its usage). Then, at the next level of permutation we see 2 as the fixing number. To calculate the offset for this, we need the "order" of this 2 among the remaining three numbers. This is where gones come into play; if an already-fixed and considered number (in this case 0) is less than the current fixer, we subtract 1 to find the new order. Then offset is calculated and accumulated. For the next number 3, the new order is 3 - (1 + 1) = 1 because both previous fixers 0 and 2 are at the "left" of 3.
This goes on till the last number of the given perm since there is no need to look at it; it will have been determined anyway.

Filter generated permutations in python

I want to generate permutations of elements in a list, but only keep a set where each element is on each position only once.
For example [1, 2, 3, 4, 5, 6] could be a user list and I want 3 permutations. A good set would be:
[1,2,3,5,4,6]
[2,1,4,6,5,3]
[3,4,5,1,6,2]
However, one could not add, for example, [1,3,2,6,5,4] to the above, as there are two permutations in which 1 is on the first position twice, also 5 would be on the 5th position twice, however other elements are only present on those positions once.
My code so far is :
# this simply generates a number of permutations specified by number_of_samples
def generate_perms(player_list, number_of_samples):
myset = set()
while len(myset) < number_of_samples:
random.shuffle(player_list)
myset.add(tuple(player_list))
return [list(x) for x in myset]
# And this is my function that takes the stratified samples for permutations.
def generate_stratified_perms(player_list, number_of_samples):
user_idx_dict = {}
i = 0
while(i < number_of_samples):
perm = generate_perms(player_list, 1)
for elem in perm:
if not user_idx_dict[elem]:
user_idx_dict[elem] = [perm.index(elem)]
else:
user_idx_dict[elem] += [perm.index(elem)]
[...]
return total_perms
but I don't know how to finish the second function.
So in short, I want to give my function a number of permutations to generate, and the function should give me that number of permutations, in which no element appears on the same position more than the others (once, if all appear there once, twice, if all appear there twice, etc).

Let's starting by solving the case of generating n or fewer rows first. In that case, your output must be a Latin rectangle or a Latin square. These are easy to generate: start by constructing a Latin square, shuffle the rows, shuffle the columns, and then keep just the first r rows. The following always works for constructing a Latin square to start with:
1 2 3 ... n
2 3 4 ... 1
3 4 5 ... 2
... ... ...
n 1 2 3 ...
Shuffling rows is a lot easier than shuffling columns, so we'll shuffle the rows, then take the transpose, then shuffle the rows again. Here's an implementation in Python:
from random import shuffle
def latin_rectangle(n, r):
square = [
[1 + (i + j) % n for i in range(n)]
for j in range(n)
]
shuffle(square)
square = list(zip(*square)) # transpose
shuffle(square)
return square[:r]
Example:
>>> latin_rectangle(5, 4)
[(2, 4, 3, 5, 1),
(5, 2, 1, 3, 4),
(1, 3, 2, 4, 5),
(3, 5, 4, 1, 2)]
Note that this algorithm can't generate all possible Latin squares; by construction, the rows are cyclic permutations of each other, so you won't get Latin squares in other equivalence classes. I'm assuming that's OK since generating a uniform probability distribution over all possible outputs isn't one of the question requirements.
The upside is that this is guaranteed to work, and consistently in O(n^2) time, because it doesn't use rejection sampling or backtracking.
Now let's solve the case where r > n, i.e. we need more rows. Each column can't have equal frequencies for each number unless r % n == 0, but it's simple enough to guarantee that the frequencies in each column will differ by at most 1. Generate enough Latin squares, put them on top of each other, and then slice r rows from it. For additional randomness, it's safe to shuffle those r rows, but only after taking the slice.
def generate_permutations(n, r):
rows = []
while len(rows) < r:
rows.extend(latin_rectangle(n, n))
rows = rows[:r]
shuffle(rows)
return rows
Example:
>>> generate_permutations(5, 12)
[(4, 3, 5, 2, 1),
(3, 4, 1, 5, 2),
(3, 1, 2, 4, 5),
(5, 3, 4, 1, 2),
(5, 1, 3, 2, 4),
(2, 5, 1, 3, 4),
(1, 5, 2, 4, 3),
(5, 4, 1, 3, 2),
(3, 2, 4, 1, 5),
(2, 1, 3, 5, 4),
(4, 2, 3, 5, 1),
(1, 4, 5, 2, 3)]
This uses the numbers 1 to n because of the formula 1 + (i + j) % n in the first list comprehension. If you want to use something other than the numbers 1 to n, you can take it as a list (e.g. players) and change this part of the list comprehension to players[(i + j) % n], where n = len(players).

If runtime is not that important I would go for the lazy way and generate all possible permutations (itertools can do that for you) and then filter out all permutations which do not meet your requirements.
Here is one way to do it.
import itertools
def permuts (l, n):
all_permuts = list(itertools.permutations(l))
picked = []
for a in all_permuts:
valid = True
for p in picked:
for i in range(len(a)):
if a[i] == p[i]:
valid = False
break
if valid:
picked.append (a)
if len(picked) >= n:
break
print (picked)
permuts ([1,2,3,4,5,6], 3)

Iterating Lists in Python, Ruby, Haskell (or whatever)

Update: I realize that I put the question very badly. Here's a second run.
Consider the following function:
myList = []
optimumList = []
def findOptimumListItems():
n = 5
for i in range (n + 1):
for j in range (n + 1 - i):
myList.append((i, j, n-i-j))
for i in myList:
win = 0.0
draw = 0.0
for j in myList:
score = 0
if (i[0] > j[0]):
score += 1
if (i[0] == j[0]):
score += 0.5
if (i[1] > j[1]):
score += 1
if (i[1] == j[1]):
score += 0.5
if (i[2] > j[2]):
score += 1
if (i[2] == j[2]):
score += 0.5
if (score == 2):
win += 1
if (score == 1.5):
draw += 1
if (win/(len(myList)-win-draw) > 1.0):
optimumList.append(i)
return optimumList
First I make a list. For n = 5 the generated list is:
[(0, 0, 5), (0, 1, 4), (0, 2, 3), (0, 3, 2), (0, 4, 1),
(0, 5, 0), (1, 0, 4), (1, 1, 3), (1, 2, 2), (1, 3, 1),
(1, 4, 0), (2, 0, 3), (2, 1, 2), (2, 2, 1), (2, 3, 0),
(3, 0, 2), (3, 1, 1), (3, 2, 0), (4, 0, 1), (4, 1, 0),
(5, 0, 0)]
Then, the function takes each element of the list and compares it with the list itself. This is how you do it: Say I'm comparing [0, 0, 5] against [3, 1, 1]. 0 loses to 3 (so no points), 0 loses to 1, so no points, 5 wins against 1 (1 point for that). A draw gets 0.5 points, a win gets 1 point. For any item, if wins are more than loses then that item is considered optimum and is added to the optimum list.
For n = 5, the optimum list is:
[(0, 2, 3), (0, 3, 2), (1, 1, 3), (1, 2, 2), (1, 3, 1), (2, 0, 3),
(2, 1, 2), (2, 2, 1), (2, 3, 0), (3, 0, 2), (3, 1, 1), (3, 2, 0)]
My question is: How can I write the above function in a concise way? I'm especially interested in functional algorithms. Python, Ruby, Java, Haskell answers will be appreciated. (Having said that, if you have a neat solution in any language; that's okay.)
Sorry for repeating the same question. I agree that the original question was messy and hard to understand. I hope it's clear now.
Update (upon rampion's comment): Is there an efficient algorithm for this (or this type) problem?

Second Update: Great -- now I understand exactly what you want. This does the same thing as the code in your most recent edit:
def optimize(myList):
score_tup = lambda tup_a, tup_b: sum(1.0 if a > b else 0.5 if a == b else 0 for a, b in zip(tup_a, tup_b))
scores = ((tup_a, [score_tup(tup_a, tup_b) for tup_b in myList]) for tup_a in myList)
scores = ((tup, score.count(2), score.count(1.5)) for tup, score in scores)
return [tup for tup, win, draw in scores if (win * 1.0 / (len(myList) - win - draw)) > 1.0]
a = 5
myList = [(i, j, a-i-j) for i in range(a + 1) for j in range(a + 1 - i)]
print myList
print optimize(myList)
If you want to see previous versions of this answer, check the edits; this was getting too long.

in Haskell:
optimize :: Int -> [(Int,Int,Int)]
optimize n = filter optimal [ (a,b,c) | a <- [0..n], b <- [0..(n-a)], let c = n - a - b ]
where optimal x = (>0) . sum $ map (comp x) xs
comp (a,b,c) (a',b',c') = signum $ vs a a' + vs b b' + vs c c'
vs x x' = case compare x x' of
GT -> 1
EQ -> 0
LT -> -1
Though this is fairly concise, it's not very efficient (we compare (0,3,2) with (0,2,3) and vice versa, when we only need to do that once).

This isn't done yet, but it's a good start, I think.
It's written in Ruby.
>> l = [1,2,3]
>> l.map {|n| l.map{|i| i > n ? 1 : 0.5 }}.flatten.inject(0){|start, n| start + n}
=> 6.0

What is this for? Comparing each item in the list with every other item in the list will take an extremely large time ( O(n^2), I believe), especially as the list grows in size. If you give us some context, we may be able to tell you a better way to do this.
Anyway, here's what I came up with for comparing all of your items:
>>> for i in range(len(myList)):
... for x in range(len(myList)):
... if x != i:
... if myList[i][0] > myList[x][0]:
... score += 1
... if myList[i][0] < myList[x][0]:
... score += .5
...
Untested, as it never finished running, so there may be a mistake.

If I'm doing this correctly the comparison function is non-transformative, and what is returned is the same list you had before the comparison. That being said my functional program for generating the list is
def triple_tuple_generator(a):
"""Given an integer 'a', returns a generator of triple tuples of length 'a(a-1), where the tuple values are over the range 'a-1=i' (i,i-1,a-2*i+1)."""
for i in range(a):
for j in range(a-1):
yield (i,j,a-1-i-j)
This is a generator so consume as you wish. If I was good enough at working with summations I would prove my hunch, but I'm a physicist not a mathematician. ;) Let me know if I got this right.