I'm trying to figure out how to iterate over an arbitrary number of loops where each loop depends on the most recent outer loop. The following code is an example of what I want to do:
def function(z):
n = int(log(z))
tupes = []
for i_1 in range(1, n):
for i_2 in range(1, i_1):
...
...
...
for i_n in range(1, i_{n - 1}):
if i_1*i_2*...*i_n > z:
tupes.append((i_1, i_2,..., i_n))
return tupes
While I'd like this to work for any z > e**2, it's sufficient for it to work for zs up to e**100. I know that if I take the Cartesian product of the appropriate ranges that I'll end up with a superset of the tuples I desire, but I'd like to obtain only the tuples I seek.
If anyone can help me with this, I'd greatly appreciate it. Thanks in advance.
Combinations can be listed in ascending order; in fact, this is the default behavior of itertools.combinations.
The code:
for i1 in range(1,6):
for i2 in range(1,i1):
for i3 in range(1,i2):
print (i3, i2, i1)
# (1, 2, 3)
# (1, 2, 4)
# ...
# (3, 4, 5)
Is equivalent to the code:
from itertools import combinations
for combination in combinations(range(1,6), 3):
print combination
# (1, 2, 3)
# (1, 2, 4)
# ...
# (3, 4, 5)
Using the combinations instead of the Cartesian product culls the sample space down to what you want.
The logic in your question implemented recursively (note that this allows for duplicate tuples):
import functools
def f(n, z, max_depth, factors=(), depth=0):
res = []
if depth == max_depth:
product = functools.reduce(lambda x, y: x*y, factors, 1)
if product > z:
res.append(factors)
else:
for i in range(1, n):
new_factors = factors + (i,)
res.extend(f(i, z, factors=new_factors, depth=depth+1, max_depth=max_depth))
return res
z = np.e ** 10
n = int(np.log(z))
print(f(n, z, max_depth=8))
yields
[(8, 7, 6, 5, 4, 3, 2, 1),
(9, 7, 6, 5, 4, 3, 2, 1),
(9, 8, 6, 5, 4, 3, 2, 1),
(9, 8, 7, 5, 4, 3, 2, 1),
(9, 8, 7, 6, 4, 3, 2, 1),
(9, 8, 7, 6, 5, 3, 2, 1),
(9, 8, 7, 6, 5, 4, 2, 1),
(9, 8, 7, 6, 5, 4, 3, 1),
(9, 8, 7, 6, 5, 4, 3, 2)]
As zondo suggested, you'll need to use a function and recursion to accomplish this task. Something along the lines of the following should work:
def recurse(tuplesList, potentialTupleAsList, rangeEnd, z):
# No range to iterate over, check if tuple sum is large enough
if rangeEnd = 1 and sum(potentialTupleAsList) > z:
tuplesList.append(tuple(potentialTupeAsList))
return
for i in range(1, rangeEnd):
potentialTupleAsList.append(i)
recurse(tuplesList, potentialTupleAsList, rangeEnd - 1, z)
# Need to remove item you used to make room for new value
potentialTupleAsList.pop(-1)
Then you could call it as such to get the results:
l = []
recurse(l, [], int(log(z)), z)
print l
Your innermost loop can (if reached at all) only go over range(1, 1). Since the endpoint is not included, the loop will not iterate over any values. The shortest implementation of your function is thus:
def function(z):
return []
If you are content with tuples of length smaller than n, then I propose the following solution:
import math
def function(z):
def f(tuples, loop_variables, product, end):
if product > z:
tuples.append(loop_variables)
for i in range(end - 1, 0, -1):
f(tuples, loop_variables + (i,), product * i, i)
n = int(math.log(z))
tuples = []
f(tuples, (), 1, n)
return tuples
The time complexity is not good though: With n nested loops over O(n) elements, we are on the order of n**n steps.
Related
With greate respect to the answer of #alias there: (Find minimum sum) I would like to solve similar puzzle. Having 4 agents and 4 type of works. Each agent does work on some price (see initial matrix in the code). I need find the optimal allocation of agents to the particular work. Following code almost copy paste from the mentioned answer:
initial = ( # Row - agent, Column - work
(7, 7, 3, 6),
(4, 9, 5, 4),
(5, 5, 4, 5),
(6, 4, 7, 2)
)
opt = Optimize()
agent = [Int(f"a_{i}") for i, _ in enumerate(initial)]
opt.add(And(*(a != b for a, b in itertools.combinations(agent, 2))))
for w, row in zip(agent, initial):
opt.add(Or(*[w == val for val in row]))
minTotal = Int("minTotal")
opt.add(minTotal == sum(agent))
opt.minimize(minTotal)
print(opt.check())
print(opt.model())
Mathematically correct answer: [a_2 = 4, a_1 = 5, a_3 = 2, a_0 = 3, minTotal = 14] is not working for me, because I need get index of agent instead.
Now, my question - how to rework the code to optimize by indexes instead of values? I've tried to leverage the Array but have no idea how to minimize multiple sums.
You can simply keep track of the indexes and walk each row to pick the corresponding element. Note that the itertools.combinations can be replaced by Distinct. We also add extra checks to make sure the indices are between 1 and 4 to ensure there's no out-of-bounds access:
from z3 import *
initial = ( # Row - agent, Column - work
(7, 7, 3, 6),
(4, 9, 5, 4),
(5, 5, 4, 5),
(6, 4, 7, 2)
)
opt = Optimize()
def choose(i, vs):
if vs:
return If(i == 1, vs[0], choose(i-1, vs[1:]))
else:
return 0
agent = [Int(f"a_{i}") for i, _ in enumerate(initial)]
opt.add(Distinct(*agent))
for a, row in zip(agent, initial):
opt.add(a >= 1)
opt.add(a <= 4)
opt.add(Or(*[choose(a, row) == val for val in row]))
minTotal = Int("minTotal")
opt.add(minTotal == sum(choose(a, row) for a, row in zip (agent, initial)))
opt.minimize(minTotal)
print(opt.check())
print(opt.model())
This prints:
sat
[a_1 = 1, a_0 = 3, a_2 = 2, a_3 = 4, minTotal = 14]
which I believe is what you're looking for.
Note that z3 also supports arrays, which you can use for this problem. However, in SMTLib, arrays are not "bounded" like in programming languages. They're indexed by all elements of their domain type. So, you won't get much of a benefit from doing that, and the above formulation seems to be the most straightforward.
My code find all combinations of a list of numbers with a given sum. The code is working well, but when trying big numbers (like 100 or 200), the code is taking way too long.
Any advices on how to make the code much faster ?
def check(target, lst):
def _a(idx, l, r, t):
if t == sum(l): r.append(l)
elif t < sum(l): return
for u in range(idx, len(lst)):
_a(u, l + [lst[u]], r, t)
return r
return len(_a(0, [], [], target))
print(check(200, (1, 2, 5, 10, 20, 50, 100, 200, 500)))
Make the inner function simpler (only give it index and remaining target, and return the number) and then memoize it?
from functools import lru_cache
def check(target, lst):
#lru_cache(None)
def a(idx, t):
if t == 0: return 1
elif t < 0: return 0
return sum(a(u, t - lst[u])
for u in range(idx, len(lst)))
return a(0, target)
print(check(200, (1, 2, 5, 10, 20, 50, 100, 200, 500)))
you could use itertools to iterate through every combination of every possible size, and filter out everything that doesn't sum to 10:
import itertools
numbers = [1, 2, 3, 7, 7, 9, 10]
result = [seq for i in range(len(numbers), 0, -1) for seq in
itertools.combinations(numbers, i) if sum(seq) == 10]
print result
result
[(1, 2, 7), (1, 2, 7), (1, 9), (3, 7), (3, 7), (10,)]
Unfortunately this is something like O(2^N) complexity, so it isn't suitable for input lists larger than, say, 20 elements.
I want to generate permutations of elements in a list, but only keep a set where each element is on each position only once.
For example [1, 2, 3, 4, 5, 6] could be a user list and I want 3 permutations. A good set would be:
[1,2,3,5,4,6]
[2,1,4,6,5,3]
[3,4,5,1,6,2]
However, one could not add, for example, [1,3,2,6,5,4] to the above, as there are two permutations in which 1 is on the first position twice, also 5 would be on the 5th position twice, however other elements are only present on those positions once.
My code so far is :
# this simply generates a number of permutations specified by number_of_samples
def generate_perms(player_list, number_of_samples):
myset = set()
while len(myset) < number_of_samples:
random.shuffle(player_list)
myset.add(tuple(player_list))
return [list(x) for x in myset]
# And this is my function that takes the stratified samples for permutations.
def generate_stratified_perms(player_list, number_of_samples):
user_idx_dict = {}
i = 0
while(i < number_of_samples):
perm = generate_perms(player_list, 1)
for elem in perm:
if not user_idx_dict[elem]:
user_idx_dict[elem] = [perm.index(elem)]
else:
user_idx_dict[elem] += [perm.index(elem)]
[...]
return total_perms
but I don't know how to finish the second function.
So in short, I want to give my function a number of permutations to generate, and the function should give me that number of permutations, in which no element appears on the same position more than the others (once, if all appear there once, twice, if all appear there twice, etc).
Let's starting by solving the case of generating n or fewer rows first. In that case, your output must be a Latin rectangle or a Latin square. These are easy to generate: start by constructing a Latin square, shuffle the rows, shuffle the columns, and then keep just the first r rows. The following always works for constructing a Latin square to start with:
1 2 3 ... n
2 3 4 ... 1
3 4 5 ... 2
... ... ...
n 1 2 3 ...
Shuffling rows is a lot easier than shuffling columns, so we'll shuffle the rows, then take the transpose, then shuffle the rows again. Here's an implementation in Python:
from random import shuffle
def latin_rectangle(n, r):
square = [
[1 + (i + j) % n for i in range(n)]
for j in range(n)
]
shuffle(square)
square = list(zip(*square)) # transpose
shuffle(square)
return square[:r]
Example:
>>> latin_rectangle(5, 4)
[(2, 4, 3, 5, 1),
(5, 2, 1, 3, 4),
(1, 3, 2, 4, 5),
(3, 5, 4, 1, 2)]
Note that this algorithm can't generate all possible Latin squares; by construction, the rows are cyclic permutations of each other, so you won't get Latin squares in other equivalence classes. I'm assuming that's OK since generating a uniform probability distribution over all possible outputs isn't one of the question requirements.
The upside is that this is guaranteed to work, and consistently in O(n^2) time, because it doesn't use rejection sampling or backtracking.
Now let's solve the case where r > n, i.e. we need more rows. Each column can't have equal frequencies for each number unless r % n == 0, but it's simple enough to guarantee that the frequencies in each column will differ by at most 1. Generate enough Latin squares, put them on top of each other, and then slice r rows from it. For additional randomness, it's safe to shuffle those r rows, but only after taking the slice.
def generate_permutations(n, r):
rows = []
while len(rows) < r:
rows.extend(latin_rectangle(n, n))
rows = rows[:r]
shuffle(rows)
return rows
Example:
>>> generate_permutations(5, 12)
[(4, 3, 5, 2, 1),
(3, 4, 1, 5, 2),
(3, 1, 2, 4, 5),
(5, 3, 4, 1, 2),
(5, 1, 3, 2, 4),
(2, 5, 1, 3, 4),
(1, 5, 2, 4, 3),
(5, 4, 1, 3, 2),
(3, 2, 4, 1, 5),
(2, 1, 3, 5, 4),
(4, 2, 3, 5, 1),
(1, 4, 5, 2, 3)]
This uses the numbers 1 to n because of the formula 1 + (i + j) % n in the first list comprehension. If you want to use something other than the numbers 1 to n, you can take it as a list (e.g. players) and change this part of the list comprehension to players[(i + j) % n], where n = len(players).
If runtime is not that important I would go for the lazy way and generate all possible permutations (itertools can do that for you) and then filter out all permutations which do not meet your requirements.
Here is one way to do it.
import itertools
def permuts (l, n):
all_permuts = list(itertools.permutations(l))
picked = []
for a in all_permuts:
valid = True
for p in picked:
for i in range(len(a)):
if a[i] == p[i]:
valid = False
break
if valid:
picked.append (a)
if len(picked) >= n:
break
print (picked)
permuts ([1,2,3,4,5,6], 3)
Say I have a list of tuples [(0, 1, 2, 3), (4, 5, 6, 7), (3, 2, 1, 0)], I would like to remove all instances where a tuple is reversed e.g. removing (3, 2, 1, 0) from the above list.
My current (rudimentary) method is:
L = list(itertools.permutations(np.arange(x), 4))
for ll in L:
if ll[::-1] in L:
L.remove(ll[::-1])
Where time taken increases exponentially with increasing x. So if x is large this takes ages! How can I speed this up?
Using set comes to mind:
L = set()
for ll in itertools.permutations(np.arange(x), 4):
if ll[::-1] not in L:
L.add(ll)
or even, for slightly better performance:
L = set()
for ll in itertools.permutations(np.arange(x), 4):
if ll not in L:
L.add(ll[::-1])
The need to keep the first looks like it forces you to iterate with a contitional.
a = [(0, 1, 2, 3), (4, 5, 6, 7), (3, 2, 1, 0)]
s = set(); a1 = []
for t in a:
if t not in s:
a1.append(t)
s.add(t[::-1])
Edit: The accepted answer addresses the example code (i.e. the itertools permutations sample). This answers the generalized question for any list (or iterable).
From a previous question I learned something interesting. If Python's itertools.product is fed a series of iterators, these iterators will be converted into tuples before the Cartesian product begins. Related questions look at the source code of itertools.product to conclude that, while no intermediate results are stored in memory, tuple versions of the original iterators are created before the product iteration begins.
Question: Is there a way to create an iterator to a Cartesian product when the (tuple converted) inputs are too large to hold in memory? Trivial example:
import itertools
A = itertools.permutations(xrange(100))
itertools.product(A)
A more practical use case would take in a series of (*iterables[, repeat]) like the original implementation of the function - the above is just an example. It doesn't look like you can use the current implementation of itertools.product, so I welcome in submission in pure python (though you can't beat the C backend of itertools!).
Here's an implementation which calls callables and iterates iterables, which are assumed restartable:
def product(*iterables, **kwargs):
if len(iterables) == 0:
yield ()
else:
iterables = iterables * kwargs.get('repeat', 1)
it = iterables[0]
for item in it() if callable(it) else iter(it):
for items in product(*iterables[1:]):
yield (item, ) + items
Testing:
import itertools
g = product(lambda: itertools.permutations(xrange(100)),
lambda: itertools.permutations(xrange(100)))
print next(g)
print sum(1 for _ in g)
Without "iterator recreation", it may be possible for the first of the factors. But that would save only 1/n space (where n is the number of factors) and add confusion.
So the answer is iterator recreation. A client of the function would have to ensure that the creation of the iterators is pure (no side-effects). Like
def iterProduct(ic):
if not ic:
yield []
return
for i in ic[0]():
for js in iterProduct(ic[1:]):
yield [i] + js
# Test
x3 = lambda: xrange(3)
for i in iterProduct([x3,x3,x3]):
print i
This can't be done with standard Python generators, because some of the iterables must be cycled through multiple times. You have to use some kind of datatype capable of "reiteration." I've created a simple "reiterable" class and a non-recursive product algorithm. product should have more error-checking, but this is at least a first approach. The simple reiterable class...
class PermutationsReiterable(object):
def __init__(self, value):
self.value = value
def __iter__(self):
return itertools.permutations(xrange(self.value))
And product iteslf...
def product(*reiterables, **kwargs):
if not reiterables:
yield ()
return
reiterables *= kwargs.get('repeat', 1)
iterables = [iter(ri) for ri in reiterables]
try:
states = [next(it) for it in iterables]
except StopIteration:
# outer product of zero-length iterable is empty
return
yield tuple(states)
current_index = max_index = len(iterables) - 1
while True:
try:
next_item = next(iterables[current_index])
except StopIteration:
if current_index > 0:
new_iter = iter(reiterables[current_index])
next_item = next(new_iter)
states[current_index] = next_item
iterables[current_index] = new_iter
current_index -= 1
else:
# last iterable has run out; terminate generator
return
else:
states[current_index] = next_item
current_index = max_index
yield tuple(states)
Tested:
>>> pi2 = PermutationsReiterable(2)
>>> list(pi2); list(pi2)
[(0, 1), (1, 0)]
[(0, 1), (1, 0)]
>>> list(product(pi2, repeat=2))
[((0, 1), (0, 1)), ((0, 1), (1, 0)), ((1, 0), (0, 1)), ((1, 0), (1, 0))]
>>> giant_product = product(PermutationsReiterable(100), repeat=5)
>>> len(list(itertools.islice(giant_product, 0, 5)))
5
>>> big_product = product(PermutationsReiterable(10), repeat=2)
>>> list(itertools.islice(big_product, 0, 5))
[((0, 1, 2, 3, 4, 5, 6, 7, 8, 9), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)),
((0, 1, 2, 3, 4, 5, 6, 7, 8, 9), (0, 1, 2, 3, 4, 5, 6, 7, 9, 8)),
((0, 1, 2, 3, 4, 5, 6, 7, 8, 9), (0, 1, 2, 3, 4, 5, 6, 8, 7, 9)),
((0, 1, 2, 3, 4, 5, 6, 7, 8, 9), (0, 1, 2, 3, 4, 5, 6, 8, 9, 7)),
((0, 1, 2, 3, 4, 5, 6, 7, 8, 9), (0, 1, 2, 3, 4, 5, 6, 9, 7, 8))]
I'm sorry to up this topic but after spending hours debugging a program trying to iterate over recursively generated cartesian product of generators. I can tell you that none of the solutions above work if not working with constant numbers as in all the examples above.
Correction :
from itertools import tee
def product(*iterables, **kwargs):
if len(iterables) == 0:
yield ()
else:
iterables = iterables * kwargs.get('repeat', 1)
it = iterables[0]
for item in it() if callable(it) else iter(it):
iterables_tee = list(map(tee, iterables[1:]))
iterables[1:] = [it1 for it1, it2 in iterables_tee]
iterable_copy = [it2 for it1, it2 in iterables_tee]
for items in product(*iterable_copy):
yield (item, ) + items
If your generators contain generators, you need to pass a copy to the recursive call.