Develop Reference

How to find nearest divisor to given value with modulo zero

I'm trying to preprocess a dataset for a neuronal network. Therefore, I need to reshape an array with the shape (2040906, 1) into an array of batches.
I need a batch size around 1440 rows but 2040906 is not dividable (with a remainder of zero) by that number obviously.
I tried to just calculate the modulo of the division and drop as many rows as the remainder so the division will result in a modulo of zero. But dropping rows of my dataset is not what I want to do.
So this is an example snippet to reproduce the problem.
import numpy as np
x = np.ones((2040906, 1))
np.split(x, 1440)
The perfect solution for me would be some kind of function, that returns the nearest divisor for a given value that has a remainder of 0.

Not sure this is the most elegant solution, but you can do the following:
Get all divisor for the number in question
def getDivisors(n, res=None) :
res = res or []
i = 1
while i <= n :
if (n % i==0) :
res.append(i),
i = i + 1
return res
getDivisors(2040906)
Out[4]:
[1,
2,
3,
6,
7,
14,
21,
42,
48593,
97186,
145779,
291558,
340151,
680302,
1020453,
2040906]
Return the closest divisor
def get_closest_split(n, close_to=1440):
all_divisors = getDivisors(n)
for ix, val in enumerate(all_divisors):
if close_to < val:
if ix == 0: return val
if (val-close_to)>(close_to - all_divisors[ix-1]):
return all_divisors[ix-1]
return val
def get_closest_split(n, close_to=1440)
Out[6]: 42
Which in your case, would return 42 as the only divisor closest to 1440. Thus, np.split(x, 42) should work.

Looking for the largest divisor is not a good approach because of two reasons.
The size of array might be prime number.
The divisor may be too large or too small resulting in ineffective learning.
The better idea is to pad dataset with samples randomly selected from the whole dataset to make it divisible by optimal batch size. Here is the simple trick to compute the size of padded array divisible by 1440
(-x.shape[0] % 1440) + x.shape[0]
However, when data is ordered (like time series) then padding cannot be used because there no way to construct representative content of padding data.
The alternative solution would be minimization of truncated data. One can search through a range a available padding to find requires minimal truncation.
def find_best_divisor(size, low, high, step=1):
minimal_truncation, best_divisor = min((size % divisor, divisor)
for divisor in range(low, high, step))
return best_divisor
This approach is nice because it allows to utilize data well and use padding suitable for training.

Another solution for finding either the closest larger divisor, or closest smaller divisor.
import numpy as np
def get_whole_ceil(n,near):
nn = np.divide(n,np.linspace(1,np.ceil(n/near),int(np.ceil(n/near))))
return(nn[nn%1==0][-1])
def get_whole_floor(n,near):
nn = np.divide(n,np.linspace(np.floor(n/near),n,int(n-np.floor(n/near)+1)))
return(nn[nn%1==0][0])
get_whole_ceil(2040906,1440)
Out[1]: 48593.0
get_whole_floor(2040906,1440)
Out[1]: 42.0

Sometimes it is easier to solve to more general problem than to solve the problem at hand. So I look for prime factors and the calculate all possible products between them. In this case, it's also x40 faster. I also took note from #tstanisl to allow you to limit the amount of work done.
You can use divisors() for a sorted list of divisors, then look for the nearest one.
from itertools import chain, combinations
from functools import reduce # Valid in Python 2.6+, required in Python 3
import operator
def prime_factors(n, up_to=None):
"""
Returns prime factors for 'n' up to 'up_to', excluding 1 (unless n == 1)
as a sequence of tuples '(b, e)', 'b' being the factor and 'e' being
the exponent of that factor.
"""
if up_to is None:
up_to = n
for i in range(2, min(n, up_to)):
if n % i == 0:
factors = prime_factors(n//i, up_to=up_to)
if factors:
# we always get the smallest factor last, so if it is
# the same as the current number we're looking at,
# add up the exponent
last_factor, last_exp = factors[-1]
if last_factor == i:
return factors[:-1] + ((i, last_exp+1),)
return factors + ((i,1),)
if up_to is not None and up_to < n:
return tuple()
return ((n,1),)
# thanks to https://docs.python.org/dev/library/itertools.html#itertools-recipes
def powerset(iterable):
"""
Generates the powerset of a given iterable.
>>> list(powerset([1,2,3]))
[(), (1,), (2,), (3,), (1, 2), (1, 3), (2, 3), (1, 2, 3)]
"""
s = list(iterable)
return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))
# thanks to https://stackoverflow.com/questions/595374/whats-the-function-like-sum-but-for-multiplication-product
def prod(t):
return reduce(operator.mul, t, 1)
def divisors(n, up_to=None):
"""
Returns a sorted list of divisors of 'n'. If 'up_to' is specified,
only prime factors up to 'up_to' will be considered when calculating
the list of divisors.
"""
return [1] + sorted([
prod(fs)
for comb in powerset(prime_factors(n, up_to))
if comb
for fs in itertools.product(*(
tuple(b**ei for ei in range(1,e+1))
for b,e in comb))
])
# >>> divisors(2040906)
# [1, 2, 3, 6, 7, 14, 21, 42, 48593, 97186,
# 145779, 291558, 340151, 680302, 1020453, 2040906]
# >>> divisors(2040906, 48592)
# [1, 2, 3, 6, 7, 14, 21, 42]
# >>> %timeit divisors(2040906)
# 100 loops, best of 5: 3.93 ms per loop
# >>> %timeit getDivisors(2040906) # from answer by #calestini
# 10 loops, best of 5: 170 ms per loop

I created a simple code for this, and it works well for me.
def get_closest_divisor(num, divisor):
for i in range(num):
if ( num % divisor > 0):
num = num + 1
return num
Then by running this function
get_closest_divisor(33756, 512)
[Out]: 33792

Find the smallest number that is not in a list of intervals

I have to find the smallest number that is not in a list of intervals. For example, i have a list: [(2, 6), (0, 3), (9, 10)]. I have to find the smallest number by checking all numbers from 0 to 10 until i find the smallest one that is not in any of these ranges.
So, the number can't be between 2 and 6 (including both), 0 and 3, 9 and 10.
I've written a function and i can't get the right answer. I always get all first numbers that the program checks, but i just need the last one.
And i don't understand why it doesn't work with return.
intervals = [(2, 6), (0, 3), (9, 10)]
def smallest_number(intervals):
i = 0
while True:
for first, second in intervals:
if i in range(first, second + 1):
i += 1
print(i)
print(smallest_number(intervals))
I expect the output 7, but i get a loop of 7's or numbers from 1 to 7, depending on where i put print(i).

Here's one approach:
from itertools import chain
intervals = [(2, 6), (0, 3), (9, 10)]
ranges = chain.from_iterable(range(i, j+1) for i, j in l)
min(set(range(10)).difference(ranges))
# 7
Details
The first step creates a set from the ranges contained in the tuples using a generator comprehension and calling range with both elements from each tuple. The result is flattened using itertools.chain:
ranges = chain.from_iterable(range(i, j+1) for i, j in l)
print(list(ranges))
# {0, 1, 2, 3, 4, 5, 6, 9, 10}
Now we only need to look for the min of the difference between this set and a range up to n
diff = set(range(10)).difference(ranges)
# {7, 8}

Your code will print all the numbers that are in the intervals, not those that are not, before it ends in an infinite loop. Try this:
def smallest_number(intervals):
i = 0
while True:
for first, second in intervals:
if i in range(first, second + 1):
i += 1
break # break from inner loop
else:
break # break from outer loop
return i # return number
Also, while if i in range(first, second + 1): is okay (and fast) in Python 3, I'd rather suggest using if first <= i <= second:
Or shorter, using next and any:
def smallest_number(intervals, max_ = 10):
return next((i for i in range(max_ + 1)
if not any(first <= i <= second for first, second in intervals)),
None) # None is default if none is found

I don't have Python here to check, but it seems your function does not return anything, so print(smallest_number(intervals)) prints None.
def smallest_number(intervals):
i = 0
while True:
for first, second in intervals:
if i in range(first, second + 1):
i += 1
else:
return i
Now print(smallest_number(intervals)) should work.

My solution with protection from case when there is no such minimal number.
checking = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
intervals = [(2, 6), (0, 3), (9, 10)]
min_number = None
unique = set()
for left, right in intervals:
unique.update(range(left, right + 1))
subset = set(checking) - unique
if subset:
min_number = min(subset)
print(min_number)

This might be a good choice if you want a list of permitted numbers, not just the smallest.
import itertools
def flatten(list_of_sublists):
return itertools.chain.from_iterable(list_of_sublists)
def excluded_numbers(intervals):
excluded_ranges = [range(lower, upper + 1) for lower, upper in intervals]
excluded_numbers = set(flatten(excluded_ranges))
return excluded_numbers
def permitted_numbers(intervals):
excluded = excluded_numbers(intervals)
max_excluded = max(excluded)
candidates = set(range(max_excluded + 2))
return candidates - excluded
def smallest_number(intervals):
return min(permitted_numbers(intervals))

Sort the intervals by their start (end points if the start points are equal). Then start with the first interval and keep expanding the right bound till the next interval's start point is greater than our current coverage end point.
Take (2,6),(0,3),(9,10)
Sorting: (0,3),(2,6),(9,10)
current coverage = [0,3]
since 3 < 6 and 2<3, we can increase our right point to 6
so coverage becomes [0,6]
but 6 < 9, so 7 is the smallest number which isn't in any of the intervals. Also you need to check if the start point of the very first interval is > 0. In that case the answer will be 0.

What is an efficient way of counting the number of unique multiplicative and additive pairs in a list of integers in Python?

Given a sorted array A = [n, n+1, n+2,... n+k] elements, I am trying to count the unique number of multiplicative and additive pairs such that the condition xy >= x+y is satisfied. Where x and y are indices of the list, and y > x.
Here is my minimum working example using a naive brute force approach:
def minimum_working_example(A):
A.sort()
N = len(A)
mpairs = []
x = 0
while x < N:
for y in range(N):
if x<y and (A[x]*A[y])>=(A[x]+A[y]):
mpairs.append([A[x], A[y]])
else:
continue
x+=1
return len(mpairs)
A = [1,2,3,4,5]
print(minimum_working_example(A))
#Output = 6, Unique pairs that satisfy xy >= x+y: (2, 3), (2, 4), (2, 5), (3, 4), (3, 5), (4, 5)
However this approach has an exponential time complexity for large lists.
What sorting or searching algorithms exist that will allow me to implement a more efficient solution?

This question has a closed-form mathematical solution, but if you'd prefer to implement in a programming langauge, you just need to find all unique pairs of numbers from your list, and count the number that satisfy your requirement. itertools.combinations is your friend here:
import itertools
A = [1,2,3,4,5]
pairs = []
for x, y in itertools.combinations(A, 2):
if x*y >= x + y:
pairs.append((x,y))
Output
[(2, 3), (2, 4), (2, 5), (3, 4), (3, 5), (4, 5)]

Basic algebra ... solve for one variable in terms of the other:
xy >= x + y
xy - y >= x
y(x-1) >= x
Now, if your elements are all positive integers, you get
if x == 1, no solution
if x == 2, y >= 2
else x > 2
y >= x/(x-1)
In this last case, x/(x-1) is a fraction between 1 and 2; again,
y >= 2
Solves the inequality.
This gives you a trivially accessible solution in O(1) time; if you want the pairs themselves, you're constrained by the printing, which is O(n^2) time.

So using the fact that x*y >= x+y if both (mistake in my original comment) x and y are >=2 (see #Prune's answer for details), then you may as well remove 0 and 1 from your list if they appear, because they won't make any suitable pair.
So now assuming all numbers or >=2 and you have k of them (e.g. replace k by k-1 in the following operation if you have n=1), all possible pairs will satisfy your condition. And the number of pairs among k elements is the well known formula k*(k-1)/2 (google it if you don't know about it). The time to compute this number is essentially the same (one multiplication, one division) no matter what value of k you have (unless you start going to crazy big numbers), so complexity is O(1).
This assumes your integers are positive, if not the formula will be slightly more complicated but still possible as a closed form solution.

If you want a more mathematical solution, consider that xy > x+y has no solutions for y=1. Otherwise, you can algebraically work this out to x > y/(y-1). Now if we have two consecutive, positive integers and divide the larger by the smaller, we either get exactly 2 (if y=2) or get some fraction between 1 and 2 exclusive. Note that x has to be greater than this y/(y-1) quotient, but also has to be less than y. If y=2, then the only possible x value in our list of positive integers has to be 1, in which case there are no matches because 1 is not greater than 2/1. So this all simplifies to "For each number y in our list, count all of the values x that are in the range of [2,y)." If you do the math, this should come out to adding 1 + 2 + 3 + ... + k, which is simply k(k+1)/2. Again, we're assuming n and k are positive integers; you can derive a slightly more complicated formula when you take into account cases for n <= 0.
But assuming you DO want to stick with a brute force approach, and not do a little mathematical reasoning to find a different approach: I tried out several variations, and here's a faster solution based on the following.
You said the list is already sorted, so I dropped the sorting function.
Likewise, the "else: continue" isn't necessary, so for simplicity I dropped that.
Instead of looping through all x and y values, then checking if x < y, you can just make your second loop check y values in the range from x+1 to y. BUT...
You can use itertools to generate the unique pairs of all numbers in your list A
If you ultimately really only care about the length of the pairs list and not the number pairs themselves, then you can just count the pairs along the way instead of storing them. Otherwise you can run out of memory at high N values.
I get slightly faster results with the equivalent test of x(y-1)-y>0. More so than with x(y-1)>y too.
So here's what I have:
def example4(A):
mpair_count = 0
for pair in itertools.combinations(A, 2):
if pair[0]*(pair[1]-1) - pair[1] > 0:
mpair_count += 1
return mpair_count
Here's everything timed:
from timeit import default_timer as timer
import itertools
def minimum_working_example(A):
A.sort()
N = len(A)
mpairs = []
x = 0
while x < N:
for y in range(N):
if x<y and (A[x]*A[y])>=(A[x]+A[y]):
mpairs.append([A[x], A[y]])
else:
continue
x+=1
return len(mpairs)
# Cutting down the range
def example2(A):
N = len(A)
mpairs = []
x = 0
while x < N:
for y in range(x+1,N):
if (A[x]*A[y])>=(A[x]+A[y]):
mpairs.append([A[x], A[y]])
x += 1
return len(mpairs)
# Using itertools
def example3(A):
mpair_count = 0
for pair in itertools.combinations(A, 2):
if pair[0]*pair[1] > sum(pair):
mpair_count += 1
return mpair_count
# Using itertools and the different comparison
def example4(A):
mpair_count = 0
for pair in itertools.combinations(A, 2):
if pair[0]*(pair[1]-1) - pair[1] > 0:
mpair_count += 1
return mpair_count
# Same as #4, but slightly different
def example5(A):
mpair_count = 0
for pair in itertools.combinations(A, 2):
if pair[0]*(pair[1]-1) > pair[1]:
mpair_count += 1
return mpair_count
A = range(1,5000)
start = timer()
print(minimum_working_example(A))
end = timer()
print(end - start)
start = timer()
print(example2(A))
end = timer()
print(end - start)
start = timer()
print(example3(A))
end = timer()
print(end - start)
start = timer()
print(example4(A))
end = timer()
print(end - start)
start = timer()
print(example5(A))
end = timer()
print(end - start)
Result:
12487503
8.29403018155
12487503
7.81883932384
12487503
3.39669140954
12487503
2.79594281764
12487503
2.92911447083

Different ways to traverse flight of stairs

I am trying to write a code in python for the following: There are n stairs. I want to display the different ways(not the sum total no. of ways) of reaching stair n from stair 1. The catch here is i can skip not more than m stairs at a time. Please help. Note: m and n will be input by user.
The following code displays the total no of ways. but not what all the different ways are:
# A program to count the number of ways to reach n'th stair
# Recursive function used by countWays
def countWaysUtil(n,m):
if n <= 1:
return n
res = 0
i = 1
while i<=m and i<=n:
res = res + countWaysUtil(n-i, m)
i = i + 1
return res
# Returns number of ways to reach s'th stair
def countWays(s,m):
return countWaysUtil(s+1, m)
# Driver program
s,m = 4,2
print "Number of ways =",countWays(s, m)

This seems to be a sort of generalized Fibonacci sequence, except that instead of f(n) = f(n-1) + f(n-2) you have f(n) = f(n-1) + ... + f(n-m). Your code should work, but the massive recursion will give you very high complexity for larger values of n (on the order of O(m^n) if I'm not mistaken). The key to solving this sort of problem is to memorize the results for lower input values in a list:
def ways_up_stairs(n, m):
ways = [1] + [None] * n
for i in range(1, n+1):
ways[i] = sum(ways[max(0, i-m):i])
return ways[n]
Some example input and output:
print(ways_up_stairs(4,2)) # 5
print(ways_up_stairs(4,3)) # 7
print(ways_up_stairs(4,4)) # 8
If you want the actual step-sequences, not the sums, you can easily adapt the code accordingly, using nested list comprehensions:
def ways_up_stairs(n, m):
ways = [[(0,)]] + [None] * n
for i in range(1, n+1):
ways[i] = [w + (i,) for ws in ways[max(0, i-m):i] for w in ws]
return ways[n]
print(ways_up_stairs(4,2))
# [(0, 2, 4), (0, 1, 2, 4), (0, 1, 3, 4), (0, 2, 3, 4), (0, 1, 2, 3, 4)]
Note that you might have to adapt the code a bit, as it is e.g. not really clear whether "skip up to m steps" means that you can take 1..m or 1..m+1 steps, but if you have the expected result for some input, making those "one-off" adaptations should be easy.

Finding the largest palindrome product of two 3-digit numbers: what is the error in logic?

I thought of solving this problem in the following way: start with two variables with value 999, multiplying one by another in a loop that decrements one or the other until a palindrome is found. The code is this:
def is_palindrome(n):
if str(n) == str(n)[::-1]:
return True
else:
return False
def largest_palindrome_product_of_3_digit():
x = 999
y = 999
for i in reversed(range(x + y + 1)):
if is_palindrome(x * y):
return x * y
if i % 2 == 0:
x -= 1
else:
y -= 1
The result of my method is 698896, while the correct result is 906609. Could you point me where my logic is incorrect?

Here are a couple of hints:
If n=y*x is any number in the range(600000, 700000) (for example) with y<=x, and x<1000, what's the smallest possible value of x?
If n is a palindromic number, both its first and last digit are 6, so what does that imply about the last digits of x & y?
Now generalize and figure out an efficient algorithm. :)
I've never done this problem before, but I just coded a reasonably fast algorithm that's around 2000 times faster than a brute-force search that uses
for x in xrange(2, 1000):
for y in xrange(2, x+1):
n = y*x
#etc
According to timeit.py, the brute-force algorithm takes around 1.29 seconds on my old machine, the algorithm I hinted at above takes around 747 microseconds.
Edit
I've improved my bounds (and modified my algorithm slightly) and brought the time down to 410 µsec. :)
To answer your questions in the comment:
Yes, we can start x at the square root of the beginning of the range, and we can stop y at x (just in case we find a palindromic square).
What I was getting at with my 2nd hint is that for x=10*I+i, y=10*J+j, we don't need to test all 81 combinations of i and j, we only need to test the ones where (i*j)%10 equals the digit we want. So if we know that our palindrome starts and ends with 9 then (i, j) must be in [(1, 9), (3, 3), (7, 7), (9, 1)].
I don't think I should post my actual code here; it's considered bad form on SO to post complete solutions to Project Euler problems. And perhaps some SO people don't even like it when people supply hints. Maybe that's why I got down-voted...

You're missing possible numbers.
You're considering O(x+y) numbers and you need to consider O(x * y) numbers. Your choices are, essentially, to either loop one of them from 999, down to 1, then decrement the other and...
Simple demonstration:
>>> want = set()
>>> for x in [1, 2, 3, 4, 5]:
... for y in [1, 2, 3, 4, 5]:
... want.add(x * y)
...
>>> got = set()
>>> x = 5
>>> y = 5
>>> for i in reversed(range(x + y + 1)):
... got.add(x * y)
... if i % 2:
... x -= 1
... else:
... y -= 1
...
>>> want == got
False
Alternatively, you do know the top of the range (999 * 999) and you can generate all palindromic numbers in that range, from the highest to the lowest. From there, doing a prime factorization and checking if there's a split of the factors that multiply to two numbers in the range [100,999] is trivial.