x = 2**1000000
n = 2**100000000
(x**2-2)%n is too slow. I found pow() but I can't use it because I can't subtract 2. (pow(x, 2)-2)%n and (x*x-2)%n are also slow. When I tested (x*x-2) it was fast but when I added the modulo operator it was slow. Is there a way to compute (x**2-2)%n faster?
Are you running this in the interpreter? I did some testing and the main slowdown seemed to come from the interpreter trying to display the result.
If you assign the expression to a variable, the interpreter won't try to display the result, and it will be very quick:
x = 2**1000000
n = 2**100000000
result = (x**2-2)%n
Addendum:
I was also originally thinking along the same lines as MikeW's answer, and if you wanted every part of the code to be fast, you could take advantage of Python's internal base 2 representation of integers and use bitwise left shifts:
x = 1 << 1000000
n = 1 << 100000000
This comes with the caveat that this only works because x and n are powers of 2, and you have to be more careful to avoid making an off-by-one error. This answer is a good explanation of how bitshifts basically work, but Python is bit different than other languages like C, C++, or Java because Python integers are unlimited precision, so you can never left shift a bit completely away like you could in other languages.
Some module rules :
1) (a+b)mod(n) = amod(n)+bmod(N)
2) (a.b)mod(n) = amod(n).bmod(n)
So you can transform your equation into :
(x**2-2)%n ==> (x.x - 2)%n ==> (x%n).(x%n) - (2%n)
If n is always greater than 2, (2%n) is 2 itself.
solving (x%n) :
If x and n are always in 2**value ; if x > n then (x%n)= 0 is the answer and if x < n (x%n)=x
So the answer is either 0-(2%n) or x**2-(2%n)
If x is always a power of 2, and n is always a power of 2, then you can you can compute it easily and quickly using bit operations on a byte array, which you can then reconstitute into a "number".
If 2^N is (binary) 1 followed by N zeroes, then (2^N)^2 is (binary) 1 followed by 2N zeros.
2^3 squared is b'1000000'
If you have a number 2^K (binary 1 followed by K zeroes), then 2^K - 2 will be K-1 1s (ones) followed by a zero.
eg 2^4 is 16 = b'10000', 2^4 - 2 is b'1110'
If you require "% 2^M" then in binary, you just select the last (lower) M bits, and disregard the rest .
9999 is b'10011100001111'
9999 % 2^8 is b'00001111'
'
Hence combining the parts, if x=2^A and n=2^B, then
(x^2 - 2 ) % n
will be: (last B bits of) (binary) (2*A - 1 '1's followed by a '0')
If you want to compute (x ** y - z) % n
it will be equivalent to ((x ** y) % n - z) % n
Python pow function includes as optional parameter a modulo, as it is very often used and can be computed in an optimized way. So you should use:
(pow(x, y, n) - z) % n
OP says in comment : it's slow because I assign x to the answer and I repeat the process.
I try this :
x = 2**(1000*1000)
n = 2**(100*1000*1000)
import time
t0=time.time()
for i in range(6):
x = (x*x-2)%n
t1=time.time()
print(i,t1-t0)
t0=t1
print(x<n)
"""
0 0.0
1 0.4962291717529297
2 0.5937404632568359
3 1.9043104648590088
4 5.708504915237427
5 16.74528479576111
True
"""
It shows that in this problem, it's just slow because x grows, doubling the number of digit at each loop :
In [5]: %timeit u=x%n
149 ns ± 6.42 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
the %n takes absolutely no time if x<n.
Related
Let a,b,c be the first digits of a number (e.g. 523 has a=5, b=2, c=3). I am trying to check if abc == sqrt(a^b^c) for many values of a,b,c. (Note: abc = 523 stands for the number itself.)
I have tried this with Python, but for a>7 it already took a significant amount of time to check just one digit combination. I have tried rewriting the equality as multiple logs, like log_c[log_b[log_a[ (abc)^2 ]]] == 1, however, I encountered Math Domain Errors.
Is there a fast / better way to check this equality (preferably in Python)?
Note: Three digits are an example for StackOverflow. The goal is to test much higher powers with seven to ten digits (or more).
Here is the very basic piece of code I have used so far:
for a in range(1,10):
for b in range(1,10):
for c in range(1,10):
N = a*10**2 + b*10 + c
X = a**(b**c)
if N == X:
print a,b,c
The problem is that you are uselessly calculating very large integers, which can take much time as Python has unlimited size for them.
You should limit the values of c you test.
If your largest possible number is 1000, you want a**b**c < 1000**2, so b**c < log(1000**2, a) = 2*log(1000, a)), so c < log(2*log(1000, a), b)
Note that you should exclude a = 1, as any power of it is 1, and b = 1, as b^c would then be 1, and the whole expression is just a.
To test if the square root of a^b^c is abc, it's better to test if a^b^c is equal to the square of abc, in order to avoid using floats.
So, the code, that (as expected) doesn't find any solution under 1000, but runs very fast:
from math import log
for a in range(2,10):
for b in range(2,10):
for c in range(1,int(log(2*log(1000, a), b))):
N2 = (a*100 + b*10 + c)**2
X = a**(b**c)
if N2 == X:
print(a,b,c)
You are looking for numbers whose square root is equal to a three-digit integer. That means your X has to have at most 6 digits, or more precisely log10(X) < 6. Once your a gets larger, the potential solutions you're generating are much larger than that, so we can eliminate large swathes of them without needing to check them (or needing to calculate a ** b ** c, which can get very large: 9 ** 9 ** 9 has 369_693_100 DIGITS!).
log10(X) < 6 gives us log10(a ** b ** c) < 6 which is the same as b ** c * log10(a) < 6. Bringing it to the other side: log10(a) < 6 / b ** c, and then a < 10 ** (6 / b ** c). That means I know I don't need to check for any a that exceeds that. Correcting for an off-by-one error gives the solution:
for b in range(1, 10):
for c in range(1, 10):
t = b ** c
for a in range(1, 1 + min(9, int(10 ** (6 / t)))):
N = a * 100 + b * 10 + c
X = a ** t
if N * N == X:
print(a, b, c)
Running this shows that there aren't any valid solutions to your equation, sadly!
a**(b**c) will grow quite fast and most of the time it will far exceed three digit number. Most of the calculations you are doing will be useless. To optimize your solution do the following:
Iterate over all 3 digit numbers
For each of these numbers square it and is a power of the first digit of the number
For those that are, check if this power is in turn a power of the second digit
And last check if this power is the third digit
fib1 = 1
fib2 = 2
i = 0
sum = 0
while i < 3999998:
fibn = fib1 + fib2
fib1 = fib2
fib2 = fibn
i += 1
if fibn % 2 == 0:
sum = sum + fibn
print(sum + 2)
The challenge is to add even Fibonacci numbers under 4000000. It works for small limits say 10 numbers. But goes on forever when set for 4000000.
Code is in Python
Yes, there are inefficiencies in your code, but the biggest one is that you're mistaken about what you're computing.
At each iteration i increases by one, and you are checking at each step whether i < 3999998. You are effectively finding the first 4 million fibonacci numbers.
You should change your loop condition to while fib2 < 3999998.
A couple of other minor optimisations. Leverage python's swapping syntax x, y = y, x and its sum function. Computing the sum once over a list is slightly faster then summing them up successively over a loop.
a, b = 1, 2
fib = []
while b < 3999998:
a, b = b, a + b
if b % 2 == 0:
fib.append(b)
sum(fib) + 2
This runs in 100000 loops, best of 3: 7.51 µs per loop, a whopping 3 microseconds faster than your current code (once you fix it, that is).
You are computing the first 4 million fibonacci numbers. It's going to take a while. It took me almost 5 minutes to compute the result, which was about 817 KB of digits, after I replaced fibn % 2 == 0 with fibn & 1 == 0 - an optimization that makes a big difference on such large numbers.
In other words, your code will eventually finish - it will just take a long time.
Update: your version finished after 42 minutes.
I'm doing some probability calculation.
In one of my task, I need to multiply the combination number of choose 8000 samples from 10000 items with 0.8**8000.
The combination number is a long long-number, and with the help of numpy, I get the result of 0.8**8000 as 5.2468172239242176864e-776.
But when I try to multiply these two numbers, I got [9] 34845 segmentation fault ipython -i.
How can I do such multiplication then?
PS: This is a piece of my code
import numpy
d2 = numpy.float128(0.8) ** 8000
d1 = 165555575235503558460892983752748337696863078099010763950122624527927836980322780662408249953188062227721112100054260160204180655980717428736444016909193193353770953722788106404786520413339850951599929567643032803416164290936680088121145665954509987077953596641237451927908536624592636591471456488142060812180933761408708169972797751139799352908109763166895772281109195968567911923343187466596002627570139321755043803267091330804414889831229832744256038117150720178689066894068507531026417815624234453195871008113238128934831837842040515600131726096039123279876153916504647241693083829553081901075278042326502699324012014817969085443550523855284341221708045253558716789811929298590803855947461554713178815399150688529048306222786951038548880400191620565711291586700534540755526276938422405001345270278335726581375322976014611332999126216550500951669985289322635729053541565465940744524663726205818866513444952048185208697438054246674199211750006230637806394882672053335493831407089830994135058867370833787098758113596190447219426121568324685764151601296948654893782399960327514764114467176417125060133454019708700782282480571935020898204763471121684913190735908414301826140125010936910161942130277906874552721346626800201093026689035996876035329180150478191582393837824731994055511844267891121846403164857127885959745644323971338513739214928092232132691519007718752719466750891748327404893783451436251805894736392433617289459646429204124129760273396235033220480921175386059331059354409267348067375581516003852060360378571075522650956157791058846993826792047806030332676423336065499519953076910418838626376480202828151673161942289092221049283902410699951912366163469099917310239336454637062482599733606299329923589714875696509548029668358723465427602758225427644633549944802010973352599970041918971524450218727345622721744933664742499521140235707102217164259438766026322532351208348119475549696983427008567651685921355966036780080415723688044325099562693124488758728102729947753752228785786200998322978801432511608341549234067324280214361346940194251357867820535466891356019219904248859277399657389914429390105240751239760865282709465029549690591863591028864648910033430400L
print d1 * d2
When multiplying an extremely large number by an extremely small number, working with floats can introduce huge inaccuracies. In your case, the magnitude of the numbers is causing overflow errors, so you have bigger problems than just inaccuracies!
Whenever you find yourself in this situation, it can be useful to first check if it is possible to stay in the integer domain, and "massage" the numbers a little first. In your case, it is possible and I'll explain how below.
One operand of the multiplication, the extremely large number, is 8000 samples from 10000 items. Use the closed form equation for the number of combinations, where your sample size n is 10000 and the subset size r is 8000. Exclam (!) here is factorial, which you can find in math.factorial in python.
C(n,r) = n! / r! (n - r)!
The other operand 0.8 ** 8000 is the extremely small number, which by index laws is equal to:
8**8000 / 10**8000
So when we multiply these two numbers together, the answer we want is:
10000! * 8**8000
--------------------------
8000! * 2000! * 10**8000
Let's call this number x and then take logarithms of both sides. Working in the log domain will transform multiplications into additions, and divisions into subtractions, making things more manageable.
from math import log, factorial
numerator = log(factorial(10000)) + 8000*log(8)
denominator = log(factorial(8000)) + log(factorial(2000)) + 8000*log(10)
log_x = numerator - denominator
Now these numbers are of a magnitude that is usable in python.
You will find that log_x is equal to approximately 3214. You now only need to observe that exp(log_x) == x to find your answer. It is a very large, but finite, number.
Arbitrary-precision integers aren't really the way to go for this problem, since you're destroying any precision you had by calling log, so I'll just let scipy.special.gammaln speak for itself (but see my edit below):
from math import log, factorial
from scipy.special import gammaln
def comp_integral(n, r, p, q):
numerator = log(factorial(n)) + r*log(8)
denominator = log(factorial(r)) + log(factorial(n-r)) + r*log(q)
return numerator - denominator
def comp_gamma(n, r, p, q):
comb = gammaln(n+1) - gammaln(n-r+1) - gammaln(r+1)
expon = r*(log(p) - log(q))
return comb+expon
In [220]: comp_integral(10000, 8000, 8, 10)
Out[220]: 3214.267963130871
In [221]: comp_gamma(10000, 8000, 8, 10)
Out[221]: 3214.2679631308811
In [222]: %timeit comp_integral(10000, 8000, 8, 10)
10 loops, best of 3: 80.3 ms per loop
In [223]: %timeit comp_gamma(10000, 8000, 8, 10)
100000 loops, best of 3: 11.4 µs per loop
Note that the outputs are identical up to 14 digits, but the gammaln version is almost 8000 times faster. If you're going to do this a lot, this will count.
EDIT: What gammaln does is to compute the natural log of the gamma function. The gamma function can be thought of as a generalization of factorial, in that factorial(n) == gamma(n+1). So comb(n,r) == gamma(n+1)/(gamma(n-r+1)*gamma(r+1)). Then taking logs turns it into the form above.
Gamma also has values for fractional inputs and for negative numbers. That doesn't really matter here though.
I maintain the gmpy2 library and it can do this very easily.
>>> import gmpy2
>>> gmpy2.comb(10000,8000) * gmpy2.mpfr('0.8')**8000
mpfr('8.6863984366232171e+1395')
Building off of wim's great answer, you can also store this number as a Fraction by building a list of prime factors, doing any cancellations and multiplying everything together.
I've included a rather naive implementation for this problem. It returns a fraction in less than a minute as is but if you implement slightly smarter factorization you can surely make it even faster.
from collections import Counter
from fractions import Fraction
import gmpy2 as gmpy
def get_factors(n):
factors = Counter()
factor = 1
while n != 1:
factor = int(gmpy.next_prime(factor))
while not n % factor:
n //= factor
factors[factor] += 1
return factors
factors = Counter()
# multiply by 10000!
for i in range(10000):
factors += get_factors(i+1)
# multiply by 8^8000
factors[2] += 3*8000
#divide by 2000!
for i in range(2000):
factors -= get_factors(i+1)
#divide by 8000!
for i in range(8000):
factors -= get_factors(i+1)
# divide by 10^8000
factors[2] -= 8000
factors[5] -= 8000
# build Fraction
numer = 1
denom = 1
for f,c in factors.items():
if c>0:
numer *= f**c
elif c<0:
denom *= f**-c
frac = Fraction(numer, denom)
Looks like it's around 8.686*10^1395
Help finding the big-Oh notation for the following code:
i = n
while i > 0:
k = 2 + 2
i = i // 2
I think its n because n is assigned and then looped. Is this right?
A simple way to think of this (which can be used as a general approach) is the following:
the initial value of i is n
the code will stop looping once i reaches 0 (consequently, the last iteration will execute when i is 1)
having executed an arbitrary number of iterations (call that number c), the value of i is
((n / 2) / 2) / ... ) = n / (2 ^ c)
^ divide by 2 c times
So, at the end of the loop, we want (n / (2 ^ c)) = 1. Solving for c gives us c = logn.
So the big-oh complexity is O(logn). (That is, assuming n is an integer and not a float).
The loop repeats the number of times that you can divide n by two. That is O(log n).
For any N, let f(N) be the last five
digits before the trailing zeroes in
N!. For example,
9! = 362880 so f(9)=36288
10! = 3628800 so f(10)=36288
20! = 2432902008176640000 so f(20)=17664
Find f(1,000,000,000,000)
I've successfully tackled this question for the given examples, my function can correctly find f(9), f(10), etc. However it struggles with larger numbers, especially the number the problem asks for - f(10^12).
My current optimizations are as follows: I remove trailing zeros from the multiplier and the sum, and shorten the sum to 5 digits after each multiplication. The code in python is as follows:
def SFTR (n):
sum, a = 1, 2
while a < n+1:
mul = int(re.sub("0+$","",str(a)))
sum *= mul
sum = int(re.sub("0+$","",str(sum))[-5:])
a += 1
return sum
Can anyone tell me why this function is scaling so largely, and why its taking so long. Also, if anyone could hint me in the correct direction to optimize my algorithm. (a name of the general topic will suffice) Thank you.
Update:
I have made some changes for optimization and it is significantly faster, but it is still not fast enough for f(10^12). Can anyone tell me whats making my code slow or how to make it faster?
def SFTR (n):
sum, a = 1, 2
while a < n+1:
mul = a
while(mul % 10 == 0): mul = mul/10
mul = mul % 100000
sum *= mul
while(sum % 10 == 0): sum = sum/10
sum = sum % 100000
a += 1
return sum
mul can get very big. Is that necessary? If I asked you to compute the last 5 non-zero digits of 1278348572934847283948561278387487189900038 * 38758
by hand, exactly how many digits of the first number do you actually need to know?
Building strings frequently is expensive. I'd rather use the modulo operator when truncating to the last five digits.
python -m timeit 'x = str(111111111111111111111111111111111)[-5:]'
1000000 loops, best of 3: 1.09 usec per loop
python -m timeit 'x = 111111111111111111111111111111111 % 100000'
1000000 loops, best of 3: 0.277 usec per loop
The same applies to stripping the trailing zeros. There should be a more efficient way to do this, and you probably don't have to do it in every single step.
I didn't check your algorithm for correctness, though, it's just a hint for optimization.
In fact, you might even note that there are only a restricted set of possible trailing non-zero digits. If I recall correctly, there are only a few thousand possible trailing non-zero digit combinations, when you look only at the last 5 digits. For example, is it possible for the final non-zero digit ever to be odd? (Ignore the special cases of 0! and 1! here.)