I am trying to do some calculations with python, where I ran out of memory. Therefore, I want to read/write a file in order to free memory. I need a something like a very big list object, so I thought writing a line for each object in the file and read/write to that lines instead of to memory. Line ordering is important for me since I will use line numbers as index. So I was wondering how I can replace lines in python, without moving around other lines (Actually, it is fine to move lines, as long as they return back to where I expect them to be).
Edit
I am trying to help a friend, which is worse than or equal to me in python. This code supposed to find biggest prime number, that divides given non-prime number. This code works for numbers until the numbers like 1 million, but after dead, my memory gets exhausted while trying to make numbers list.
# a comes from a user input
primes_upper_limit = (a+1) / 2
counter = 3L
numbers = list()
while counter <= primes_upper_limit:
numbers.append(counter)
counter += 2L
counter=3
i=0
half = (primes_upper_limit + 1) / 2 - 1
root = primes_upper_limit ** 0.5
while counter < root:
if numbers[i]:
j = int((counter*counter - 3) / 2)
numbers[j] = 0
while j < half:
numbers[j] = 0
j += counter
i += 1
counter = 2*i + 3
primes = [2] + [num for num in numbers if num]
for numb in reversed(primes):
if a % numb == 0:
print numb
break
Another Edit
What about wrinting different files for each index? for example a billion of files with long integer filenames, and just a number inside of the file?
You want to find the largest prime divisor of a. (Project Euler Question 3)
Your current choice of algorithm and implementation do this by:
Generate a list numbers of all candidate primes in range (3 <= n <= sqrt(a), or (a+1)/2 as you currently do)
Sieve the numbers list to get a list of primes {p} <= sqrt(a)
Trial Division: test the divisibility of a by each p. Store all prime divisors {q} of a.
Print all divisors {q}; we only want the largest.
My comments on this algorithm are below. Sieving and trial division are seriously not scalable algorithms, as Owen and I comment. For large a (billion, or trillion) you really should use NumPy. Anyway some comments on implementing this algorithm:
Did you know you only need to test up to √a, int(math.sqrt(a)), not (a+1)/2 as you do?
There is no need to build a huge list of candidates numbers, then sieve it for primeness - the numbers list is not scalable. Just construct the list primes directly. You can use while/for-loops and xrange(3,sqrt(a)+2,2) (which gives you an iterator). As you mention xrange() overflows at 2**31L, but combined with the sqrt observation, you can still successfully factor up to 2**62
In general this is inferior to getting the prime decomposition of a, i.e. every time you find a prime divisor p | a, you only need to continue to sieve the remaining factor a/p or a/p² or a/p³ or whatever). Except for the rare case of very large primes (or pseudoprimes), this will greatly reduce the magnitude of the numbers you are working with.
Also, you only ever need to generate the list of primes {p} once; thereafter store it and do lookups, not regenerate it.
So I would separate out generate_primes(a) from find_largest_prime_divisor(a). Decomposition helps greatly.
Here is my rewrite of your code, but performance still falls off in the billions (a > 10**11 +1) due to keeping the sieved list. We can use collections.deque instead of list for primes, to get a faster O(1) append() operation, but that's a minor optimization.
# Prime Factorization by trial division
from math import ceil,sqrt
from collections import deque
# Global list of primes (strictly we should use a class variable not a global)
#primes = deque()
primes = []
def is_prime(n):
"""Test whether n is divisible by any prime known so far"""
global primes
for p in primes:
if n%p == 0:
return False # n was divisible by p
return True # either n is prime, or divisible by some p larger than our list
def generate_primes(a):
"""Generate sieved list of primes (up to sqrt(a)) as we go"""
global primes
primes_upper_limit = int(sqrt(a))
# We get huge speedup by using xrange() instead of range(), so we have to seed the list with 2
primes.append(2)
print "Generating sieved list of primes up to", primes_upper_limit, "...",
# Consider prime candidates 2,3,5,7... in increasing increments of 2
#for number in [2] + range(3,primes_upper_limit+2,2):
for number in xrange(3,primes_upper_limit+2,2):
if is_prime(number): # use global 'primes'
#print "Found new prime", number
primes.append(number) # Found a new prime larger than our list
print "done"
def find_largest_prime_factor(x, debug=False):
"""Find all prime factors of x, and return the largest."""
global primes
# First we need the list of all primes <= sqrt(x)
generate_primes(x)
to_factor = x # running value of the remaining quantity we need to factor
largest_prime_factor = None
for p in primes:
if debug: print "Testing divisibility by", p
if to_factor%p != 0:
continue
if debug: print "...yes it is"
largest_prime_factor = p
# Divide out all factors of p in x (may have multiplicity)
while to_factor%p == 0:
to_factor /= p
# Stop when all factors have been found
if to_factor==1:
break
else:
print "Tested all primes up to sqrt(a), remaining factor must be a single prime > sqrt(a) :", to_factor
print "\nLargest prime factor of x is", largest_prime_factor
return largest_prime_factor
If I'm understanding you correctly, this is not an easy task. They way I interpreted it, you want to keep a file handle open, and use the file as a place to store character data.
Say you had a file like,
a
b
c
and you wanted to replace 'b' with 'bb'. That's going to be a pain, because the file actually looks like a\nb\nc -- you can't just overwrite the b, you need another byte.
My advice would be to try and find a way to make your algorithm work without using a file for extra storage. If you got a stack overflow, chances are you didn't really run out of memory, you overran the call stack, which is much smaller.
You could try reworking your algorithm to not be recursive. Sometimes you can use a list to substitute for the call stack -- but there are many things you could do and I don't think I could give much general advice not seeing your algorithm.
edit
Ah I see what you mean... when the list
while counter <= primes_upper_limit:
numbers.append(counter)
counter += 2L
grows really big, you could run out of memory. So I guess you're basically doing a sieve, and that's why you have the big list numbers? It makes sense. If you want to keep doing it this way, you could try a numpy bool array, because it will use substantially less memory per cell:
import numpy as np
numbers = np.repeat(True, a/2)
Or (and maybe this is not appealing) you could go with an entirely different approach that doesn't use a big list, such as factoring the number entirely and picking the biggest factor.
Something like:
factors = [ ]
tail = a
while tail > 1:
j = 2
while 1:
if tail % j == 0:
factors.append(j)
tail = tail / j
print('%s %s' % (factors, tail))
break
else:
j += 1
ie say you were factoring 20: tail starts out as 20, then you find 2 tail becomes 10, then it becomes 5.
This is not terrible efficient and will become way too slow for a large (billions) prime number, but it's ok for numbers with small factors.
I mean your sieve is good too, until you start running out of memory ;). You could give numpy a shot.
pytables is excellent for working with and storing huge amounts of data. But first start with implementing the comments in smci's answer to minimize the amount of numbers you need to store.
For a number with only twelve digits, as in Project Euler #3, no fancy integer factorization method is needed, and there is no need to store intermediate results on disk. Use this algorithm to find the factors of n:
Set f = 2.
If n = 1, stop.
If f * f > n, print n and stop.
Divide n by f, keeping both the quotient q and the remainder r.
If r = 0, print q, divide n by q, and go to Step 2.
Otherwise, increase f by 1 and go to Step 3.
This just does trial division by every integer until it reaches the square root, which indicates that the remaining cofactor is prime. Each factor is printed as it is found.
Related
I am fairly new to Python and I have been trying to find a fast way to find primes till a given number.
When I use the Prime of Eratosthenes sieve using the following code:
#Finding primes till 40000.
import time
start = time.time()
def prime_eratosthenes(n):
list = []
prime_list = []
for i in range(2, n+1):
if i not in list:
prime_list.append(i)
for j in range(i*i, n+1, i):
list.append(j)
return prime_list
lists = prime_eratosthenes(40000)
print lists
end = time.time()
runtime = end - start
print "runtime =",runtime
Along with the list containing the primes, I get a line like the one below as output:
runtime = 20.4290001392
Depending upon the RAM being used etc, I usually consistently get a value within an range of +-0.5.
However when I try to find the primes till 40000 using a brute force method as in the following code:
import time
start = time.time()
prime_lists = []
for i in range(1,40000+1):
for j in range(2,i):
if i%j==0:
break
else:
prime_lists.append(i)
print prime_lists
end = time.time()
runtime = end - start
print "runtime =",runtime
This time, along with the the list of primes, I get a smaller value for runtime:
runtime = 16.0729999542
The value only varies within a range of +-0.5.
Clearly, the sieve is slower than the brute force method.
I also observed that the difference between the runtimes in the two cases only increases with an increase in the value 'n' till which primes are to be found.
Can anyone give a logical explanation for the above mentioned behavior? I expected the sieve to function more efficiently than the brute force method but it seems to work vice-versa here.
While appending to a list is not the best way to implement this algorithm (the original algorithm uses fixed size arrays), it is amortized constant time. I think a bigger issue is if i not in list which is linear time. The best change you can make for larger inputs is having the outer for loop only check up to sqrt(n), which saves a lot of computation.
A better approach is to keep a boolean array which keeps track of striking off numbers, like what is seen in the Wikipedia article for the Sieve. This way, skipping numbers is constant time since it's an array access.
For example:
def sieve(n):
nums = [0] * n
for i in range(2, int(n**0.5)+1):
if nums[i] == 0:
for j in range(i*i, n, i):
nums[j] = 1
return [i for i in range(2, n) if nums[i] == 0]
So to answer your question, your two for loops make the algorithm do potentially O(n^2) work, while being smart about the outer for loop makes the new algorithm take up to O(n sqrt(n)) time (in practice, for reasonably-sized n, the runtime is closer to O(n))
I want to find the largest prime number within range(old_number + 1 , 2*old_number)
This is my code so far:
def get_nearest_prime(self, old_number):
for num in range(old_number + 1, 2 * old_number) :
for i in range(2,num):
if num % i == 0:
break
return num
when I call the get_nearest_prime(13)
the correct output should be 23, while my result was 25.
Anyone can help me to solve this problem? Help will be appreciated!
There are lots of changes you could make, but which ones you should make depend on what you want to accomplish. The biggest problem with your code as it stands is that you're successfully identifying primes with the break and then not doing anything with that information. Here's a minimal change that does roughly the same thing.
def get_nearest_prime(old_number):
largest_prime = 0
for num in range(old_number + 1, 2 * old_number) :
for i in range(2,num):
if num % i == 0:
break
else:
largest_prime = num
return largest_prime
We're using the largest_prime local variable to keep track of all the primes you find (since you iterate through them in increasing order). The else clause is triggered any time you exit the inner for loop "normally" (i.e., without hitting the break clause). In other words, any time you've found a prime.
Here's a slightly faster solution.
import numpy as np
def seive(n):
mask = np.ones(n+1)
mask[:2] = 0
for i in range(2, int(n**.5)+1):
if not mask[i]:
continue
mask[i*i::i] = 0
return np.argwhere(mask)
def get_nearest_prime(old_number):
try:
n = np.max(seive(2*old_number-1))
if n < old_number+1:
return None
return n
except ValueError:
return None
It does roughly the same thing, but it uses an algorithm called the "Sieve of Eratosthenes" to speed up the finding of primes (as opposed to the "trial division" you're using). It isn't the fastest Sieve in the world, but it's reasonably understandable without too many tweaks.
In either case, if you're calling this a bunch of times you'll probably want to keep track of all the primes you've found since computing them is expensive. Caching is easy and flexible in Python, and there are dozens of ways to make that happen if you do need the speed boost.
Note that I'm not positive the range you've specified always contains a prime. It very well might, and if it does you can get away with a lot shorter code. Something like the following.
def get_nearest_prime(old_number):
return np.max(seive(2*old_number-1))
I don't completely agree with the name you've chosen since the largest prime in that interval is usually not the closest prime to old_number, but I think this is what you're looking for anyway.
You can use a sublist to check if the number is prime, if all(i % n for n in range(2, i)) means that number is prime due to the fact that all values returned from modulo were True, not 0. From there you can append those values to a list called primes and then take the max of that list.
List comprehension:
num = 13
l = [*range(num, (2*num)+1)]
print(max([i for i in l if all([i % n for n in range(2,i)])]))
Expanded:
num = 13
l = [*range(num, (2*num)+1)]
primes = []
for i in l:
if all([i % n for n in range(2, i)]):
primes.append(i)
print(max(primes))
23
Search for the nearest prime number from above using the seive function
def get_nearest_prime(old_number):
return old_number+min(seive(2*old_number-1)-old_number, key=lambda a:a<0)
I'm trying to create a python program to check if a given number "n" is prime or not. I've first created a program that lists the divisors of n:
import math
def factors(n):
i = 2
factlist = []
while i <= n:
if n% i == 0:
factlist.append(i)
i = i + 1
return factlist
factors(100)
Next, I'm trying to use the "for i in" function to say that if p1 (the list of factors of n) only includes n itself, then print TRUE, and if else, print FALSE. This seems really easy, but I cannot get it to work. This is what I've done so far:
def isPrime(n):
p1 = factors(n)
for i in p1:
if factors(n) == int(i):
return True
return False
Any help is appreciated! This is a hw assignment, so it's required to use the list of factors in our prime test. Thanks in advance!
p1 will only have n if if has length 1. It may be worthwhile to add if len(p1)==1 as the conditional instead.
What I did (I actually managed to get it in 1 line) was use [number for number in range (2, num // 2) if num % number == 0]. This gets all the numbers that are factors of num. Then you can check if the length of that list is > 0. (Wouldn't be prime).
I usually do something like this:
primes = [2,3]
def is_prime(num):
if num>primes[-1]:
for i in range(primes[-1]+2, num+1,2):
if all(i%p for p in primes):
primes.append(i)
return num in primes
This method generates a global list of prime numbers as it goes. It uses that list to find other prime numbers. The advantage of this method is fewer calculations the program has to make, especially when there are many repeated calls to the is_prime function for possible prime numbers that are less than the largest number you have previously sent to it.
You can adapt some of the concepts in this function for your HW.
I'm quite new to programming and I've heard this is a good place to get started. I'm working with python through the Project Euler Questions and I'm stuck on question 7. I have a way of completing it but it is extremely inefficient and the main question I'm asking is if there is a way to check for all factors of an integer in a way that doesn't mean typing them all in.
So far this is what I have:
counter=0
prime_counter=0
for x in range(1,10000):
if x%2 and x%3 and x%4 and x%5 and x%6 and x%7 and x%8 and x%9 and x%10 and x%11 and x%12 and x%13 and x%14 and x%15 and x%16 and x%17 and x%18 and x%19 and x%20 and x%21 and x%22 and x%23 and x%24 and x%25 and x%26 and x%27 and x%28 and x%29 and x%30 and x%31 and x%32 and x%33 and x%34 and x%35 and x%36 and x%37 and x%38 and x%39 and x%40 and x%41 and x%42 and x%43 and x%44 and x%45 and x%46 and x%47 and x%48 and x%49 and x%50 !=0:
counter+=1
prime_counter+=x
if counter==10001:
break
print(counter)
print(prime_counter)
You can see my issue here and feel free to laugh but I'm quite new and I was wondering if I could get some help.
I completely forgot, the whole purpose of the code is to figure out the 10,001st prime number
It looks like you want to mark a number as non-prime if it has a modulus of 0 with a factor. So instead of hardcoding all the factors, you could use some function to generate all these numbers... like range.
The other question is "what should the range of factors for testing be, given a number?". It turns out that you need to only test for the numbers in [2, sqrt(n)]. Also, note that we only need to check if any prime number is a factor
Putting these two pieces together:
primes = [2] # list of all the prime numbers
n = 3
while len(primes) < 10001: # until we find the required number of prime numbers
F = n**0.5
prime = True
for f in (p for p in primes if p<=F):
if not n%f: # found a factor
prime = False
break
if prime:
primes.append(n)
n += 2 # test only the odd numbers
print(primes)
I wrote this Python code to print all the prime numbers until 1000:
primes = []
for prime in xrange(1, 1000):
for b in xrange(2, prime):
if (prime % b == 0):
break
else:
primes.append(prime)
print primes
However, if a number is a prime, it is appended a lot of times before moving to the next number. I tried using continue instead of break but that does not work.
I also added some more code onto that (which works) to simply output the array into a text file. It is so large that I cannot even paste it into here.
How can I append each prime number to the list only once instead of many times?
There's a feature in Python you can use quite effectively here. Just change your indentation like this:
primes = []
for prime in xrange(1, 1000):
for b in xrange(2, prime):
if (prime % b == 0):
break
else:
primes.append(prime)
print primes
That is, de-indent the else clause. This way it is not the else of the if but of the for loop. A for loop can have an else branch which gets executed after the loop has performed all scheduled iterations. If there is a break called, it gets not executed.
You can use that here without having to change a lot of your original code.
Of course, in your original version, each loop iteration in which a tested number did not turn out to be a non-prime, that tested number got appended (which was not what you wanted).
Also note that 1 is not a prime number (at least since Gauss in 1801, some say even since Euclid (600BC) first wrote about prime numbers, although there were heretics since into the 19th century), so your outer loop should start at 2 instead.
But note what Daniel wrote in his answer; you do not have to step through all the way to prime because after reaching the square root of prime, you can bail out. I always compared b * b to prime, but maybe computing sqrt(prime) once even is faster.
And of course, you can be faster by ignoring all even numbers completely. Except 2 none are prime ;-)
You need to only add the number if it reaches the end of the loop without it finding a divisor.
from math import sqrt
prime = True
for b in xrange(2, sqrt(prime)):
if prime % b == 0:
prime = False
break
if prime:
primes.append(prime)
Note I've added an optimization: you only need to test up the square root of a number, because anything after that can't possibly divide.