numpy large integer failed - python

I recently work on some project Euler problems
Smallest multiple
Problem 5
2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder.
What is the smallest positive number that is evenly divisible by all of the numbers from 1 to 20?
I wrote my code it works great
def factor_finder(n, j=2):
factor_list = []
if n == 2:
return [2]
elif n == 3:
return [3]
else:
while n >= j * 2:
while n % j == 0:
n = int(n / j)
factor_list.append(j)
j += 1
if n > 1:
factor_list.append(n)
return factor_list
def smallest_multiples(n):
from functools import reduce
factor_list = []
final_list = []
for i in range(2, n + 1):
factor_list += factor_finder(i)
# print(factor_list)
for i in set(factor_list):
l1 = []
l2 = []
for j in factor_list:
if j == i:
l1.append(j)
else:
if len(l1) > len(l2):
l2 = l1
l1 = []
else:
l1 = []
# print(l2)
final_list += l2
# print(final_list)
return (
np.array(final_list).cumprod()[-1],
reduce((lambda x, y: x * y), final_list),
)
The result is:
%time
smallest_multiples(1000)
CPU times: user 5 µs, sys: 0 ns, total: 5 µs
Wall time: 32.4 µs
(-4008056434385126912,
7128865274665093053166384155714272920668358861885893040452001991154324087581111499476444151913871586911717817019575256512980264067621009251465871004305131072686268143200196609974862745937188343705015434452523739745298963145674982128236956232823794011068809262317708861979540791247754558049326475737829923352751796735248042463638051137034331214781746850878453485678021888075373249921995672056932029099390891687487672697950931603520000)
My question is why numpy.cumprod() failed to get the right number. I thought numpy is the very number tool. Can Somebody give me some idea?

The problem is that the number reached a size that meant that it was no longer representable by ints in Python. If you look here, you'll see that ints max out in size around 19 digits (i.e. 2^63 from 63 bits + sign bit) and then go into overflow. Numpy is based in C which uses fixed-precision for much faster computations with the trade off that it is limited by the 64bit integer and will overflow. Some functions in numpy even guard against this by converting to floats to do calculations which can hold even more digits.
If you tell numpy to use "object" as your datatype, there is a significant time penalty but it'll let you use the arbitrary-precision that you're used to in Python. For your code, it would look like:
return (
np.cumprod(final_list, dtype="object")[-1],
reduce((lambda x, y: x * y), final_list),
)
More about overflow in numpy.

Numerical analysis is not number theory. Correctness is not the only goal but has to be weighed against efficiency. Arbitrary precision numbers (like large integers) are slow, so numpy defaults to using fixed length integers. These just overflow when they become too large. You can instruct numpy to use arbitrary precision integers, but you will lose much of its speed:
np.arange(1, 100).prod() # fast but wrong
# 0
np.arange(1, 100, dtype=object).prod() # slow but correct
# 933262154439441526816992388562667004907159682643816214685929638952175999932299156089414639761565182862536979208272237582511852109168640000000000000000000000

Related

Code for factoring doesn't work with large numbers?

I have a large 512 bit number n and I need to rewrite n-1 as m*2k
Here is the code I wrote:
# write (n-1) = m*2^k (where m is odd)
k = 0 # number of times we were able to divide by 2
total = (n-1)
while total % 2 == 0:
total /= 2
k += 1
m = int(total)
assert (n-1) == (2**k) * m # this does not hold true for large values of n for some reason
The problem is that it doesn't work for large (515 bit) values of n such as:
8711599454063889217821738854601954834373650047096243407624954758041578156381215983765719390767527065267731131102484447503200895621045535585981917487924709
For the above value of n, my code found k = 460 and m =2926172291557515
When I evaluate 2926172291557515 * 2**460 in python I get:
8711599454063889889401923055669626316647070894345982715097720460936366477064539266279767451213791729696559357170292404522606916263895951485640687369584640
Which does not equal n-1. Does anyone know why this could be happening? I assume it's related to having such large numbers (this code works fine for lower numbers that I test it with.
The problem arises because you are using /= which is float division. Replace it with //=, or integer division, and your code will work.
#Primusa is correct. In Python3.x / (or /=) returns a float value, so loses a lot of precision.
>>> n = 8711599454063889217821738854601954834373650047096243407624954758041578156381215983765719390767527065267731131102484447503200895621045535585981917487924709
>>> total = (n-1)
>>> total / 2
4.355799727031945e+153
>>> total // 2
4355799727031944608910869427300977417186825023548121703812477379020789078190607991882859695383763532633865565551242223751600447810522767792990958743962354

Python 3 - float(X) * i = int(Z)

I have a very large number, both before and after the decimal, but for this I'll just call it 4.58.
I want to know the number, Y, that will yield me an integer if multiplied by X and not any sort of float number.
Here is my code:
from decimal import *
setcontext(ExtendedContext)
getcontext().prec = 300
x=Decimal('4.58')
while True:
i=1
a=Decimal(i*x)
if float(a).is_integer():
print(i*x)
break
else:
i=+1
However, this method is incredibly slow and inefficient. I was wondering how could I implement continued fractions or some other method to make it predict the value of Y?
Edit
The decimal module stores float numbers more accurately (As strings), so 0.5 won't become 0.499999999.
Edit 2
I've got X (4.58).
I want to know what number will multiply by X to make an integer; as efficiently as possible.
Edit 3
Okay, maybe not my best question yet.
Here's my dilemma.
I've got a number spat out from a trivial programme I made. That number is a decimal number, 1.5.
All I want to do is find what integer will multiply by my decimal to yield another integer.
For 1.5, the best answer will be 2. (1.5*2=3) (float*int=int)
My while-loop above will do that, eventually, but I just wanted to know whether there was a better way to do this, such as continued fractions; and if there was, how could I implement it.
Edit 4
Here's my code thanks to user6794072. It's lengthy but functional.
from gmpy2 import mpz, isqrt
from fractions import Fraction
import operator
import functools
from decimal import *
setcontext(ExtendedContext)
getcontext().prec = 300
def factors(n):
n = mpz(n)
result = set()
result |= {mpz(1), n}
def all_multiples(result, n, factor):
z = n
f = mpz(factor)
while z % f == 0:
result |= {f, z // f}
f += factor
return result
result = all_multiples(result, n, 2)
result = all_multiples(result, n, 3)
for i in range(1, isqrt(n) + 1, 6):
i1 = i + 1
i2 = i + 5
if not n % i1:
result |= {mpz(i1), n // i1}
if not n % i2:
result |= {mpz(i2), n // i2}
return result
j=Decimal('4.58')
a=(Fraction(j).numerator)
b=(Fraction(j).denominator)
y=(factors(a))
x=(factors(b))
q=([item for item in x if item not in y])
w=([item for item in y if item not in x]) q.extend(w)
p=(functools.reduce(operator.mul, q, 1)) ans=(p*j)
print(ans)
If I understand your question correctly, you want to find the smallest integer (i) that can be multiplied to a non-integer number (n) so that:
i*n is an integer
I would do this by finding the factors of the numerator and denominator for n. In your example, if n = 4.58, then you can extract 458 for the numerator and 100 for the denominator.
The multiples of 458 are 2 and 229
The multiples of 100 are 2, 2, 5, 5
You can cross off one instance of 2 for the numerator and denominator. Then your solution is just multiplying the remaining factors in the denominator: in this case, 2*5*5 or 50.
Well think of what if you wanted to reach z = 1 and then use the fact that z == z * 1 to scale the answer. For any float x != 0.0, y = 1/x will yield z = 1, so for arbitrary integer z, just use y = z/x.
I'm not a Python programmer, but what about round function?

Prime number generation using Fibonacci possible?

I'm generating prime numbers from Fibonacci as follows (using Python, with mpmath and sympy for arbitrary precision):
from mpmath import *
def GCD(a,b):
while a:
a, b = fmod(b, a), a
return b
def generate(x):
mp.dps = round(x, int(log10(x))*-1)
if x == GCD(x, fibonacci(x-1)):
return True
if x == GCD(x, fibonacci(x+1)):
return True
return False
for x in range(1000, 2000)
if generate(x)
print(x)
It's a rather small algorithm but seemingly generates all primes (except for 5 somehow, but that's another question). I say seemingly because a very little percentage (0.5% under 1000 and 0.16% under 10K, getting less and less) isn't prime. For instance under 1000: 323, 377 and 442 are also generated. These numbers are not prime.
Is there something off in my script? I try to account for precision by relating the .dps setting to the number being calculated. Can it really be that Fibonacci and prime numbers are seemingly so related, but then when it's get detailed they aren't? :)
For this type of problem, you may want to look at the gmpy2 library. gmpy2 provides access to the GMP multiple-precision library which includes gcd() and fib() functions which calculate the greatest common divisor and the n-th fibonacci numbers quickly, and only using integer arithmetic.
Here is your program re-written to use gmpy2.
import gmpy2
def generate(x):
if x == gmpy2.gcd(x, gmpy2.fib(x-1)):
return True
if x == gmpy2.gcd(x, gmpy2.fib(x+1)):
return True
return False
for x in range(7, 2000):
if generate(x):
print(x)
You shouldn't be using any floating-point operations. You can calculate the GCD just using the builtin % (modulo) operator.
Update
As others have commented, you are checking for Fibonacci pseudoprimes. The actual test is slightly different than your code. Let's call the number being tested n. If n is divisible by 5, then the test passes if n evenly divides fib(n). If n divided by 5 leaves a remainder of either 1 or 4, then the test passes if n evenly divides fib(n-1). If n divided by 5 leaves a remainder of either 2 or 3, then the test passes if n evenly divides fib(n+1). Your code doesn't properly distinguish between the three cases.
If n evenly divides another number, say x, it leaves a remainder of 0. This is equivalent to x % n being 0. Calculating all the digits of the n-th Fibonacci number is not required. The test just cares about the remainder. Instead of calculating the Fibonacci number to full precision, you can calculate the remainder at each step. The following code calculates just the remainder of the Fibonacci numbers. It is based on the code given by #pts in Python mpmath not arbitrary precision?
def gcd(a,b):
while b:
a, b = b, a % b
return a
def fib_mod(n, m):
if n < 0:
raise ValueError
def fib_rec(n):
if n == 0:
return 0, 1
else:
a, b = fib_rec(n >> 1)
c = a * ((b << 1) - a)
d = b * b + a * a
if n & 1:
return d % m, (c + d) % m
else:
return c % m, d % m
return fib_rec(n)[0]
def is_fib_prp(n):
if n % 5 == 0:
return not fib_mod(n, n)
elif n % 5 == 1 or n % 5 == 4:
return not fib_mod(n-1, n)
else:
return not fib_mod(n+1, n)
It's written in pure Python and is very quick.
The sequence of numbers commonly known as the Fibonacci numbers is just a special case of a general Lucas sequence L(n) = p*L(n-1) - q*L(n-2). The usual Fibonacci numbers are generated by (p,q) = (1,-1). gmpy2.is_fibonacci_prp() accepts arbitrary values for p,q. gmpy2.is_fibonacci(1,-1,n) should match the results of the is_fib_pr(n) given above.
Disclaimer: I maintain gmpy2.
This isn't really a Python problem; it's a math/algorithm problem. You may want to ask it on the Math StackExchange instead.
Also, there is no need for any non-integer arithmetic whatsoever: you're computing floor(log10(x)) which can be done easily with purely integer math. Using arbitrary-precision math will greatly slow this algorithm down and may introduce some odd numerical errors too.
Here's a simple floor_log10(x) implementation:
from __future__ import division # if using Python 2.x
def floor_log10(x):
res = 0
if x < 1:
raise ValueError
while x >= 1:
x //= 10
res += 1
return res

Fast way to place bits for puzzle

There is a puzzle which I am writing code to solve that goes as follows.
Consider a binary vector of length n that is initially all zeros. You choose a bit of the vector and set it to 1. Now a process starts that sets the bit that is the greatest distance from any 1 bit to $1$ (or an arbitrary choice of furthest bit if there is more than one). This happens repeatedly with the rule that no two 1 bits can be next to each other. It terminates when there is no more space to place a 1 bit. The goal is to place the initial 1 bit so that as many bits as possible are set to 1 on termination.
Say n = 2. Then wherever we set the bit we end up with exactly one bit set.
For n = 3, if we set the first bit we get 101 in the end. But if we set the middle bit we get 010 which is not optimal.
For n = 4, whichever bit we set we end up with two set.
For n = 5, setting the first gives us 10101 with three bits set in the end.
For n = 7, we need to set the third bit to get 1010101 it seems.
I have written code to find the optimal value but it does not scale well to large n. My code starts to get slow around n = 1000 but I would like to solve the problem for n around 1 million.
#!/usr/bin/python
from __future__ import division
from math import *
def findloc(v):
count = 0
maxcount = 0
id = -1
for i in xrange(n):
if (v[i] == 0):
count += 1
if (v[i] == 1):
if (count > maxcount):
maxcount = count
id = i
count = 0
#Deal with vector ending in 0s
if (2*count >= maxcount and count >= v.index(1) and count >1):
return n-1
#Deal with vector starting in 0s
if (2*v.index(1) >= maxcount and v.index(1) > 1):
return 0
if (maxcount <=2):
return -1
return id-int(ceil(maxcount/2))
def addbits(v):
id = findloc(v)
if (id == -1):
return v
v[id] = 1
return addbits(v)
#Set vector length
n=21
max = 0
for i in xrange(n):
v = [0]*n
v[i] = 1
v = addbits(v)
score = sum([1 for j in xrange(n) if v[j] ==1])
# print i, sum([1 for j in xrange(n) if v[j] ==1]), v
if (score > max):
max = score
print max
Latest answer (O(log n) complexity)
If we believe the conjecture by templatetypedef and Aleksi Torhamo (update: proof at the end of this post), there is a closed form solution count(n) calculable in O(log n) (or O(1) if we assume logarithm and bit shifting is O(1)):
Python:
from math import log
def count(n): # The count, using position k conjectured by templatetypedef
k = p(n-1)+1
count_left = k/2
count_right = f(n-k+1)
return count_left + count_right
def f(n): # The f function calculated using Aleksi Torhamo conjecture
return max(p(n-1)/2 + 1, n-p(n-1))
def p(n): # The largest power of 2 not exceeding n
return 1 << int(log(n,2)) if n > 0 else 0
C++:
int log(int n){ // Integer logarithm, by counting the number of leading 0
return 31-__builtin_clz(n);
}
int p(int n){ // The largest power of 2 not exceeding n
if(n==0) return 0;
return 1<<log(n);
}
int f(int n){ // The f function calculated using Aleksi Torhamo conjecture
int val0 = p(n-1);
int val1 = val0/2+1;
int val2 = n-val0;
return val1>val2 ? val1 : val2;
}
int count(int n){ // The count, using position k conjectured by templatetypedef
int k = p(n-1)+1;
int count_left = k/2;
int count_right = f(n-k+1);
return count_left + count_right;
}
This code can calculate the result for n=100,000,000 (and even n=1e24 in Python!) correctly in no time1.
I have tested the codes with various values for n (using my O(n) solution as the standard, see Old Answer section below), and they still seem correct.
This code relies on the two conjectures by templatetypedef and Aleksi Torhamo2. Anyone wants to proof those? =D (Update 2: PROVEN)
1By no time, I meant almost instantly
2The conjecture by Aleksi Torhamo on f function has been empirically proven for n<=100,000,000
Old answer (O(n) complexity)
I can return the count of n=1,000,000 (the result is 475712) in 1.358s (in my iMac) using Python 2.7. Update: It's 0.198s for n=10,000,000 in C++. =)
Here is my idea, which achieves O(n) time complexity.
The Algorithm
Definition of f(n)
Define f(n) as the number of bits that will be set on bitvector of length n, assuming that the first and last bit are set (except for n=2, where only the first or last bit is set). So we know some values of f(n) as follows:
f(1) = 1
f(2) = 1
f(3) = 2
f(4) = 2
f(5) = 3
Note that this is different from the value that we are looking for, since the initial bit might not be at the first or last, as calculated by f(n). For example, we have f(7)=3 instead of 4.
Note that this can be calculated rather efficiently (amortized O(n) to calculate all values of f up to n) using the recurrence relation:
f(2n) = f(n)+f(n+1)-1
f(2n+1) = 2*f(n+1)-1
for n>=5, since the next bit set following the rule will be the middle bit, except for n=1,2,3,4. Then we can split the bitvector into two parts, each independent of each other, and so we can calculate the number of bits set using f( floor(n/2) ) + f( ceil(n/2) ) - 1, as illustrated below:
n=11 n=13
10000100001 1000001000001
<----> <----->
f(6)<----> f(7) <----->
f(6) f(7)
n=12 n=14
100001000001 10000010000001
<----> <----->
f(6)<-----> f(7) <------>
f(7) f(8)
we have the -1 in the formula to exclude the double count of the middle bit.
Now we are ready to count the solution of original problem.
Definition of g(n,i)
Define g(n,i) as the number of bits that will be set on bitvector of length n, following the rules in the problem, where the initial bit is at the i-th bit (1-based). Note that by symmetry the initial bit can be anywhere from the first bit up to the ceil(n/2)-th bit. And for those cases, note that the first bit will be set before any bit in between the first and the initial, and so is the case for the last bit. Therefore the number of bit set in the first partition and the second partition is f(i) and f(n+1-i) respectively.
So the value of g(n,i) can be calculated as:
g(n,i) = f(i) + f(n+1-i) - 1
following the idea when calculating f(n).
Now, to calculate the final result is trivial.
Definition of g(n)
Define g(n) as the count being looked for in the original problem. We can then take the maximum of all possible i, the position of initial bit:
g(n) = maxi=1..ceil(n/2)(f(i) + f(n+1-i) - 1)
Python code:
import time
mem_f = [0,1,1,2,2]
mem_f.extend([-1]*(10**7)) # This will take around 40MB of memory
def f(n):
global mem_f
if mem_f[n]>-1:
return mem_f[n]
if n%2==1:
mem_f[n] = 2*f((n+1)/2)-1
return mem_f[n]
else:
half = n/2
mem_f[n] = f(half)+f(half+1)-1
return mem_f[n]
def g(n):
return max(f(i)+f(n+1-i)-1 for i in range(1,(n+1)/2 + 1))
def main():
while True:
n = input('Enter n (1 <= n <= 10,000,000; 0 to stop): ')
if n==0: break
start_time = time.time()
print 'g(%d) = %d, in %.3fs' % (n, g(n), time.time()-start_time)
if __name__=='__main__':
main()
Complexity Analysis
Now, the interesting thing is, what is the complexity of calculating g(n) with the method described above?
We should first note that we iterate over n/2 values of i, the position of initial bit. And in each iteration we call f(i) and f(n+1-i). Naive analysis will lead to O(n * O(f(n))), but actually we used memoization on f, so it's much faster than that, since each value of f(i) is calculated only once, at most. So the complexity is actually added by the time required to calculate all values of f(n), which would be O(n + f(n)) instead.
So what's the complexity of initializing f(n)?
We can assume that we precompute every value of f(n) first before calculating g(n). Note that due to the recurrence relation and the memoization, generating the whole values of f(n) takes O(n) time. And the next call to f(n) will take O(1) time.
So, the overall complexity is O(n+n) = O(n), as evidenced by this running time in my iMac for n=1,000,000 and n=10,000,000:
> python max_vec_bit.py
Enter n (1 <= n <= 10,000,000; 0 to stop): 1000000
g(1000000) = 475712, in 1.358s
Enter n (1 <= n <= 10,000,000; 0 to stop): 0
>
> <restarted the program to remove the effect of memoization>
>
> python max_vec_bit.py
Enter n (1 <= n <= 10,000,000; 0 to stop): 10000000
g(10000000) = 4757120, in 13.484s
Enter n (1 <= n <= 10,000,000; 0 to stop): 6745231
g(6745231) = 3145729, in 3.072s
Enter n (1 <= n <= 10,000,000; 0 to stop): 0
And as a by-product of memoization, the calculation of lesser value of n will be much faster after the first call to large n, as you can also see in the sample run. And with language better suited for number crunching such as C++, you might get significantly faster running time
I hope this helps. =)
The code using C++, for performance improvement
The result in C++ is about 68x faster (measured by clock()):
> ./a.out
Enter n (1 <= n <= 10,000,000; 0 to stop): 1000000
g(1000000) = 475712, in 0.020s
Enter n (1 <= n <= 10,000,000; 0 to stop): 0
>
> <restarted the program to remove the effect of memoization>
>
> ./a.out
Enter n (1 <= n <= 10,000,000; 0 to stop): 10000000
g(10000000) = 4757120, in 0.198s
Enter n (1 <= n <= 10,000,000; 0 to stop): 6745231
g(6745231) = 3145729, in 0.047s
Enter n (1 <= n <= 10,000,000; 0 to stop): 0
Code in C++:
#include <cstdio>
#include <cstring>
#include <ctime>
int mem_f[10000001];
int f(int n){
if(mem_f[n]>-1)
return mem_f[n];
if(n%2==1){
mem_f[n] = 2*f((n+1)/2)-1;
return mem_f[n];
} else {
int half = n/2;
mem_f[n] = f(half)+f(half+1)-1;
return mem_f[n];
}
}
int g(int n){
int result = 0;
for(int i=1; i<=(n+1)/2; i++){
int cnt = f(i)+f(n+1-i)-1;
result = (cnt > result ? cnt : result);
}
return result;
}
int main(){
memset(mem_f,-1,sizeof(mem_f));
mem_f[0] = 0;
mem_f[1] = mem_f[2] = 1;
mem_f[3] = mem_f[4] = 2;
clock_t start, end;
while(true){
int n;
printf("Enter n (1 <= n <= 10,000,000; 0 to stop): ");
scanf("%d",&n);
if(n==0) break;
start = clock();
int result = g(n);
end = clock();
printf("g(%d) = %d, in %.3fs\n",n,result,((double)(end-start))/CLOCKS_PER_SEC);
}
}
Proof
note that for the sake of keeping this answer (which is already very long) simple, I've skipped some steps in the proof
Conjecture of Aleksi Torhamo on the value of f
For `n>=1`, prove that:
f(2n+k) = 2n-1+1 for k=1,2,…,2n-1 ...(1)
f(2n+k) = k for k=2n-1+1,…,2n ...(2)
given f(0)=f(1)=f(2)=1
The result above can be easily proven using induction on the recurrence relation, by considering the four cases:
Case 1: (1) for even k
Case 2: (1) for odd k
Case 3: (2) for even k
Case 4: (2) for odd k
Suppose we have the four cases proven for n. Now consider n+1.
Case 1:
f(2n+1+2i) = f(2n+i) + f(2n+i+1) - 1, for i=1,…,2n-1
= 2n-1+1 + 2n-1+1 - 1
= 2n+1
Case 2:
f(2n+1+2i+1) = 2*f(2n+i+1) - 1, for i=0,…,2n-1-1
= 2*(2n-1+1) - 1
= 2n+1
Case 3:
f(2n+1+2i) = f(2n+i) + f(2n+i+1) - 1, for i=2n-1+1,…,2n
= i + (i+1) - 1
= 2i
Case 4:
f(2n+1+2i+1) = 2*f(2n+i+1) - 1, for i=2n-1+1,…,2n-1
= 2*(i+1) - 1
= 2i+1
So by induction the conjecture is proven.
Conjecture of templatetypedef on the best position
For n>=1 and k=1,…,2n, prove that g(2n+k) = g(2n+k, 2n+1)
That is, prove that placing the first bit on the 2n+1-th position gives maximum number of bits set.
The proof:
First, we have
g(2n+k,2n+1) = f(2n+1) + f(k-1) - 1
Next, by the formula of f, we have the following equalities:
f(2n+1-i) = f(2n+1), for i=-2n-1,…,-1
f(2n+1-i) = f(2n+1)-i, for i=1,…,2n-2-1
f(2n+1-i) = f(2n+1)-2n-2, for i=2n-2,…,2n-1
and also the following inequality:
f(k-1+i) <= f(k-1), for i=-2n-1,…,-1
f(k-1+i) <= f(k-1)+i , for i=1,…,2n-2-1
f(k-1+i) <= f(k-1)+2n-2, for i=2n-2,…,2n-1
and so we have:
f(2n+1-i)+f(k-1+i) <= f(2n+1)+f(k-1), for i=-2n-1,…,2n-1
Now, note that we have:
g(2n+k) = maxi=1..ceil(2n-1+1-k/2)(f(i) + f(2n+k+1-i) - 1)
<= f(2n+1) + f(k-1) - 1
= g(2n+k,2n+1)
And so the conjecture is proven.
So in a break with my normal tradition of not posting algorithms I don't have a proof for, I think I should mention that there's an algorithm that appears to be correct for numbers up to 50,000+ and runs in O(log n) time. This is due to Sophia Westwood, who I worked on this problem with for about three hours today. All credit for this is due to her. Empirically it seems to work beautifully, and it's much, much faster than the O(n) solutions.
One observation about the structure of this problem is that if n is sufficiently large (n ≥ 5), then if you put a 1 anywhere, the problem splits into two subproblems, one to the left of the 1 and one to the right. Although the 1s might be placed in the different halves at different times, the eventual placement is the same as if you solved each half separately and combined them back together.
The next observation is this: suppose you have an array of size 2k + 1 for some k. In that case, suppose that you put a 1 on either side of the array. Then:
The next 1 is placed on the other side of the array.
The next 1 is placed in the middle.
You now have two smaller subproblems of size 2k-1 + 1.
The important part about this is that the resulting bit pattern is an alternating series of 1s and 0s. For example:
For 5 = 4 + 1, we get 10101
For 9 = 8 + 1, we get 101010101
For 17 = 16 + 1, we get 10101010101010101
The reason this matters is the following: suppose you have n total elements in the array and let k be the largest possible value for which 2k + 1 ≤ n. If you place the 1 at position 2k + 1, then the left part of the array up to that position will end up getting tiled with alternating 1s and 0s, which puts a lot of 1s into the array.
What's not obvious is that placing the 1 bit there, for all numbers up to 50,000, appears to yield an optimal solution! I've written a Python script that checks this (using a recurrence relation similar to the one #justhalf) and it seems to work well. The reason that this fact is so useful is that it's really easy to compute this index. In particular, if 2k + 1 ≤ n, then 2k ≤ n - 1, so k ≤ lg (n - 1). Choosing the value ⌊lg (n - 1) ⌋ as your choice of k then lets you compute the bit index by computing 2k + 1. This value of k can be computed in O(log n) time and the exponentiation can be done in O(log n) time as well, so the total runtime is Θ(log n).
The only issue is that I haven't formally proven that this works. All I know is that it's right for the first 50,000 values we've tried. :-)
Hope this helps!
I'll attach what I have. Same as yours, alas, time is basically O(n**3). But at least it avoids recursion (etc), so won't blow up when you get near a million ;-) Note that this returns the best vector found, not the count; e.g.,
>>> solve(23)
[6, 0, 11, 0, 1, 0, 0, 10, 0, 5, 0, 9, 0, 3, 0, 0, 8, 0, 4, 0, 7, 0, 2]
So it also shows the order in which the 1 bits were chosen. The easiest way to get the count is to pass the result to max().
>>> max(solve(23))
11
Or change the function to return maxsofar instead of best.
If you want to run numbers on the order of a million, you'll need something radically different. You can't even afford quadratic time for that (let alone this approach's cubic time). Unlikely to get such a huge O() improvement from fancier data structures - I expect it would require deeper insight into the mathematics of the problem.
def solve(n):
maxsofar, best = 1, [1] + [0] * (n-1)
# by symmetry, no use trying starting points in last half
# (would be a mirror image).
for i in xrange((n + 1)//2):
v = [0] * n
v[i] = count = 1
# d21[i] = distance to closest 1 from index i
d21 = range(i, 0, -1) + range(n-i)
while 1:
d, j = max((d, j) for j, d in enumerate(d21))
if d >= 2:
count += 1
v[j] = count
d21[j] = 0
k = 1
while j-k >= 0 and d21[j-k] > k:
d21[j-k] = k
k += 1
k = 1
while j+k < n and d21[j+k] > k:
d21[j+k] = k
k += 1
else:
if count > maxsofar:
maxsofar = count
best = v[:]
break
return best

Compute sum with huge intermediate values

I would like to compute
for values of n up to 1000000 as accurately as possible. Here is some sample code.
from __future__ import division
from scipy.misc import comb
def M(n):
return sum(comb(n,k,exact=True)*(1/n)*(1-k/n)**(2*n-k)*(k/n)**(k-1) for k in xrange(1,n+1))
for i in xrange(1,1000000,100):
print i,M(i)
The first problem is that I get OverflowError: long int too large to convert to float when n = 1101. This is because comb(n,k,exact=True) is too large to be converted to a float. The end result is however always a number around 0.159 .
I asked a related question at How to compute sum with large intermediate values however this question is different for three main reasons.
The formula I want to compute is different which causes different problems.
The solution proposed before to use exact=True does not help here as can be seen in the example I gave. Coding up my own implementation of comb is also not going to work as I still need to perform the floating point division.
I need to compute the answer for much bigger values than before which causes new problems. I suspect it can't be done without coding up the sum in some clever way.
A solution that doesn't crash is to use
from fractions import Fraction
def M2(n):
return sum(comb(n,k,exact=True)*Fraction(1,n)*(1-Fraction(k,n))**(2*n-k)*Fraction(k,n)**(k-1) for k in xrange(1,n+1))
for i in xrange(1,1000000,100):
print i, M2(i)*1.0
Unfortunately it is now so slow that I don't get an answer for n=1101 in a reasonable amount of time.
So the second problem is how to make it fast enough to complete for large n.
You can compute each summand in with a logarithm transformation that replaces multiplication, division, and exponentiation with addition, subtraction, and multiplication, respectively.
def summand(n,k):
lk=log(k)
ln=log(n)
a=(lk-ln)*(k-1)
b=(log(n-k)-ln)*(2*n-k)
c=-ln
d=sum(log(x) for x in xrange(n-k+1,n+1))-sum(log(x) for x in xrange(1,k+1))
return exp(a+b+c+d)
def M(n):
return sum(summand(n,k) for k in xrange(1,n))
Note that when k=n the summand will be zero so I do not compute it since the logarithm will be undefined.
You can use gmpy2. It has arbitrary precision floating point arithmetic with large exponent bounds.
from __future__ import division
from gmpy2 import comb,mpfr,fsum
def M(n):
return fsum(comb(n,k)*(mpfr(1)/n)*(mpfr(1)-mpfr(k)/n)**(mpfr(2)*n-k)*(mpfr(k)/n)**(k-1) for k in xrange(1,n+1))
for i in xrange(1,1000000,100):
print i,M(i)
Here is an excerpt of the output:
2001 0.15857490038127975
2101 0.15857582611615381
2201 0.15857666768820194
2301 0.15857743607577454
2401 0.15857814042739268
2501 0.15857878842787806
2601 0.15857938657957615
Disclaimer: I maintain gmpy2.
A rather brutal method is to compute all the factors and then mutliply in such a way that the result stays around 1.0 (Python 3.x):
def M(n):
return sum(summand(n, k) for k in range(1, n + 1))
def f1(n, k):
for i in range(k - 1):
yield k
for i in range(k):
yield n - i
def f2(n, k):
for i in range(k - 1):
yield 1 / n
for i in range(2 * n - k):
yield 1 - k / n
yield 1 / n
for i in range(2, k + 1):
yield 1 / i
def summand(n, k):
result = 1.0
factors1 = f1(n, k)
factors2 = f2(n, k)
while True:
empty1 = False
for factor in factors1:
result *= factor
if result > 1:
break
else:
empty1 = True
for factor in factors2:
result *= factor
if result < 1:
break
else:
if empty1:
break
return result
For M(1101) I get 0.15855899364641846, but it takes a few seconds. M(2000) takes about 14 seconds and yields 0.15857489065619598.
(I'm sure it can be optimised.)

Categories

Resources