While trying to run this code:
l = 1000000
w = [1, 1]
for i in range(2, l):
w.append(w[-1] + w[-2])
computer hangs on and Blue screen of death appears. The only info which I get is about MEMORY MANAGEMENT. Problem occurs in version 2.7 of Python and 3.4 as well.
Code works good for l = 100000.
Can someone explain me exactly why? I am using Windows 10 64-bit, Python 2.7.8 64-bit from Active Python.
EDIT:
Here is R code which works well:
len <- 1000000
fibvals <- numeric(len)
fibvals[1] <- 1
fibvals[2] <- 1
for (i in 3:len) {
fibvals[i] <- fibvals[i-1]+fibvals[i-2]
}
The numbers you're producing are huger than you might realize. For example, here's the size in memory of the last one:
>>> a, b = 1, 1
>>> for i in xrange(2, 1000000):
... a, b = b, a+b
...
>>> sys.getsizeof(b)
92592
That's 92 kilobytes for one integer. All of them put together would be somewhere in the vicinity of 46-ish gigabytes, and you only have 16 gigabytes.
Your R code used 64-bit floating-point numbers, which promptly overflow to infinity at around the 1476th number.
The fibonacci numbers are HUGE. In R and other languages, integers overflow, so not that much memory is required. But in Python, integers simply don't overflow. The 1000000th fibonacci number would require terabytes of space. Once your OS uses up all the physical RAM, it'll switch over to hard disk swap. When it runs out of that, you get a kernel fault.
In python list take too much space in memory. try to use tuple
Example code:
l = 1000000
w = (1,1)
for i in xrange(2,l):
w = w + (w[-1] + w[-2],)
execution of program take time, that depend on your number of cpu's and main memory.
Related
A trainee of mine made this short python script for a problem from Project Euler.
We are using Python 3.9.4
Problem: The series, 11 + 22 + 33 + ... + 1010 = 10405071317. Find the last ten digits of the series, 11 + 22 + 33 + ... + 10001000.
It isn't the best solution but in theory it should work for small values(I know it wont work for bigger values to inefficient and even double is too small.
Here is her Code:
import math
import numpy as np
x = np.arange(1,11)
a=[]
for y in x:
z = y**y
a.append(z)
b=sum(a)
print(a)
Output:
[1, 4, 27, 256, 3125, 46656, 823543, 16777216, 387420489, 1410065408]
The script isn't finished yet obviously but you can see that every power 11, 22, 33 are correct up to 1010 which does not return the correct value.
Do you see any reason for this problem it seem quite odd to me? I could not figure it out.
Yes I know this isn't the best solution and in the end the actually solution will be different but it would still be nice to solve this mystery.
You're probably on some kind of 32-bit systems where numpy defaults to 32-bit integers. This caused the result to be "truncated" to 32 bits. You can verify the result with the following expression:
(10 ** 10) % (2 ** 32)
Use Python built-in int and range unless you need the fancy stuff numpy provides. It's an arbitrary-precision integer implementation and should work for all kinds of integer calculation workload.
The simple solution is to not use numpy unnecessarily. The built-in range works very well:
import math
a=[]
for y in range(1, 11):
z = y**y
a.append(z)
b=sum(a)
print(a)
print(b)
In your case, numpy was using 32 bit integers, which overflowed when they reached their maximum value.
sum=0
n=int(input())
x = list(map(int, input().rstrip().split()))
for i in range(n):
str1=str(2**x[i])
if(len(str1)>2):
str1[len(str1)-2 :]
else:
str1
sum+=int(str1)
print(sum%100)
Input
3
1 2 3
Output
14
Constraints
1<=n<=10^7
0<=x<=10^18
This code is working fine for small values,like n=4 & x=8,7,6,4 and output is=64.but not for the given constraints.
Well... since you do
2**x[i]
which results in an integer value, that will most definately fail when x get's large enough.
You can do the math:
Assuming your computer has 8GB of main memory, that would be 8*8*1024^3=68719476736 bits of memory. So the single largest value that would span your entire(not possible to get this large, but I just want to highlight what the physical limit is here) memory, would be 2**68719476736 which lies somewhere between 2^(10^10) to 2^(10^11)
So as it stands, I think your code cannot work for numbers this large.
I want one Giga byte data string.
using this code
length = 0x20000000
payload = ''.join(random.choice(string.printable) for _ in range(length))
but python excepted and print this error "MemoryError"
full error message :
payload = ''.join(random.choice(string.printable) for _ in range(length))
MemoryError
i found this excepted case in stack overflow.
using "import sys, sys.setrecursionlimit(10**6)" will fix it!
so, i adding this code.
but not solved!!
i can't "import resource". because, can't not install..
I think the best option for you is to use a bytearray:
barray = bytearray()
length = 0x20000000
for _ in range(length):
barray.append(random.choice(string.printable))
This only consumed about 0.5 gigs on my machine.
Note, increasing the recursion limit won't help you here, indeed you aren't using recursion at all. You are just making something that is very large. Just the array of pointers underlying the list that gets created by ''.join will require about 0x20000000 * 8 * 1e-9 == 4.294967296 gigabytes, and that doesn't count the strings in the list themselves, each of which requires a full python object, which is another 40 or so bytes per object, so you see, you were just running out of memory. So taking into account your individual string objects:
>>> 0x20000000 * (48) * 1e-9
25.769803776000003
So you would need over 20 gigs! Doable on some modern laptops, but 8 gigs sure isn't enough.
I need to compute extremely large numbers using Python.
It could be as large as
factorial(n x 10,000,0000)*factorial(n x 10,000,0000)
so I think on a usual 32-bit computer it overflows...
Is there a workaround?
This is not an issue under Python.
>>> 2**100
1267650600228229401496703205376L
Python automatically converts plain integers into long integers and does not overflow.
http://docs.python.org/2/library/stdtypes.html#numeric-types-int-float-long-complex
You want to use bignums. Python has them natively.
Stirling's approximation to n! is (n/e)^n * sqrt(2 * pi * n). This has roughly n * log(n/e) digits.
You're computing (n * 1e8)!^2. This will have roughly 2 * n * 1e8 * log(n * 1e8 / e) digits, which is 2e8 * n * (17 + log(n)). Assuming n is relatively small and log(n) is negligible, this is 3.4e9 * n digits.
You can store around 2.4 digits in a byte, so your result is going to use (1.4 * n) gigabytes of RAM, assuming Python can store the number perfectly efficiently.
A 32 machine can address at most 4GB of RAM, so n = 1 or 2 is theoretically possible to compute, but for n = 3, the machine can't even hold the result in RAM. As you compute your result, Python's going to have to hold in RAM multiple bignums (for example, temporary variables), so the actual memory usage is going to be higher than the calculations above suggest.
In practice, I'd expect on a 64 bit machine with plenty of RAM you can never get to a result for this calculation in Python -- the machine will spend all its time garbage collecting.
As far as I know, integers in Python (3) can be arbitrarily large.
Is there anyway to improve performance of "str(bigint)" and "print bigint" in python ? Printing big integer values takes a lot of time. I tried to use the following recursive technique :
def p(x,n):
if n < 10:
sys.stdout.write(str(x))
return
n >>= 1
l = 10**n
k = x/l
p(k,n)
p(x-k*l,n)
n = number of digits,
x = bigint
But the method fails for certain cases where x in a sub call has leading zeros. Is there any alternative to it or any faster method. ( Please do not suggest using any external module or library ).
Conversion from a Python integer to a string has a running of O(n^2) where n is the length of the number. For sufficiently large numbers, it will be slow. For a 1,000,001 digit number, str() takes approximately 24 seconds on my computer.
If you are really needing to convert very large numbers to a string, your recursive algorithm is a good approach.
The following version of your recursive code should work:
def p(x,n=0):
if n == 0:
n = int(x.bit_length() * 0.3)
if n < 100:
return str(x)
n >>= 1
l = 10**n
a,b = divmod(x, l)
upper = p(a,n)
lower = p(b,n).rjust(n, "0")
return upper + lower
It automatically estimates the number of digits in the output. It is about 4x faster for a 1,000,001 digit number.
If you need to go faster, you'll probably need to use an external library.
For interactive applications, the built-in print and str functions run in the blink of an eye.
>>> print(2435**356)
392312129667763499898262143039114894750417507355276682533585134425186395679473824899297157270033375504856169200419790241076407862555973647354250524748912846623242257527142883035360865888685267386832304026227703002862158054991819517588882346178140891206845776401970463656725623839442836540489638768126315244542314683938913576544051925370624663114138982037489687849052948878188837292688265616405774377520006375994949701519494522395226583576242344239113115827276205685762765108568669292303049637000429363186413856005994770187918867698591851295816517558832718248949393330804685089066399603091911285844172167548214009780037628890526044957760742395926235582458565322761344968885262239207421474370777496310304525709023682281880997037864251638836009263968398622163509788100571164918283951366862838187930843528788482813390723672536414889756154950781741921331767254375186751657589782510334001427152820459605953449036021467737998917512341953008677012880972708316862112445813219301272179609511447382276509319506771439679815804130595523836440825373857906867090741932138749478241373687043584739886123984717258259445661838205364797315487681003613101753488707273055848670365977127506840194115511621930636465549468994140625
>>> str(2435**356)
'392312129667763499898262143039114894750417507355276682533585134425186395679473824899297157270033375504856169200419790241076407862555973647354250524748912846623242257527142883035360865888685267386832304026227703002862158054991819517588882346178140891206845776401970463656725623839442836540489638768126315244542314683938913576544051925370624663114138982037489687849052948878188837292688265616405774377520006375994949701519494522395226583576242344239113115827276205685762765108568669292303049637000429363186413856005994770187918867698591851295816517558832718248949393330804685089066399603091911285844172167548214009780037628890526044957760742395926235582458565322761344968885262239207421474370777496310304525709023682281880997037864251638836009263968398622163509788100571164918283951366862838187930843528788482813390723672536414889756154950781741921331767254375186751657589782510334001427152820459605953449036021467737998917512341953008677012880972708316862112445813219301272179609511447382276509319506771439679815804130595523836440825373857906867090741932138749478241373687043584739886123984717258259445661838205364797315487681003613101753488707273055848670365977127506840194115511621930636465549468994140625'
If however you are printing big integers to (standard output, say) so that they can be read (from standard input) by another process, and you are finding the binary-to-decimal operations impacting the overall performance, you can look at Is there a faster way to convert an arbitrary large integer to a big endian sequence of bytes? (although the accepted answer suggests numpy, which is an external library, though there are other suggestions).