Develop Reference

How to speed up search for abundant numbers?

Is there a way which this code could be improved so that it would run faster? Currently, this task takes between 11 and 12 seconds to run on my virtual environment
def divisors(n):
return sum([x for x in range(1, (round(n/2))) if n % x == 0])
def abundant_numbers():
return [x for x in range(1, 28123) if x < divisors(x)]
result = abundant_numbers()

Whenever you look for speeding up, you should first check whether the algorithm itself should change. And in this case it should.
Instead of looking for divisors given a number, look for numbers that divide by a divisor. For the latter you can use a sieve-like approach. That leads to this algorithm:
def abundant_numbers(n):
# All numbers are strict multiples of 1, except 0 and 1
divsums = [1] * n
for div in range(2, n//2 + 1): # Corrected end-of-range
for i in range(2*div, n, div):
divsums[i] += div # Sum up divisors for number i
divsums[0] = 0 # Make sure that 0 is not counted
return [i for i, divsum in enumerate(divsums) if divsum > i]
result = abundant_numbers(28123)
This runs quite fast, many factors faster than the translation of your algorithm to numpy.
Note that you had a bug in your code. round(n/2) as the range-end can miss a divisor. It should be n//2+1.

Fastest way to sample n random primes greater than p in Python?

I want to create a dateset of n primes greater than p ~ 2⁵⁰. I want these primes to not be consecutive but have some space in between so the difference between iᵗʰ and (i+1)ᵗʰ prime is not just a few bits.
I am using Sympy's randprime(low, hi) in a loop,
p = [start]
for i in range(n):
curr = int(randprime(start, 2 * start + 1))
p.append(curr)
start = curr
This gets significantly slow for n=10,000. Is there a better (faster) way of accomplishing the prime sampling that I want?

Precompute (or just download) a list of primes once and then sample the list during the runtime.
You can generate the list for 30x smaller upper bound than 2^50 (~10^15) with this algorithm: https://primes.utm.edu/nthprime/algorithm.php
I don't know how to get further with reasonable hardware setup.

When you use the last value (curr) to determine the next range, you are increasing the range exponentially. On average the random prime should fall about midway of the range and the range goes from X to 2X. This will move the range forward by a factor of roughly 1.5X at each iteration. With 10,000 iterations your range will increase by a factor of up to 1.5^10000 (2^5850) which will soon make it very hard for even sympy to produce primes.
If your objective is merely to have a sufficient number of differing bits between the ith and (i+1)th prime, you could stay in the same order of magnitude and filter on a minimal number of distinct bits with the previous prime (instead of increasing the magnitude of the random range).
for example:
def oneBits(N): return N%2 + oneBits(N//2) if N else 0
minDiff = 25 # minimum number of differing bits from ith to (i+1)th
minValue = 2**50
maxValue = minValue*2-1
primes = [randprime(minValue,maxValue)]
count = 10000
for _ in range(count-1):
while True:
p = randprime(minValue,maxValue)
if oneBits(primes[-1]^p)>=minDiff: break
primes.append(p)
Note that I don't have sympy so I tested this a little differently using purely random numbers for which I get the next prime:
Memory efficient primes generator:
def genPrimes(toN):
skips = dict()
maxSkip = int(toN**0.5)
if toN>=2: yield 2
for p in range(3,toN+1,2):
if p not in skips:
yield p
if p <= maxSkip: skips[p*p] = 2*p
else:
stride = skips.pop(p)
multiple = p + stride
while multiple in skips: multiple += stride
skips[multiple] = stride
Function to get next prime from N:
primes = list(genPrimes(2**26)) # for 2^50 max base prime is √(2^51) ~ 2^26
def nextPrime(N):
sieve = [1]*N.bit_length()*20
maxPrime = int((N+len(sieve))**0.5)
for p in primes:
if p>maxPrime: break
offset = (p-N%p)%p
if offset>len(sieve): continue
sieve[offset::p] = [0]*len(range(offset,len(sieve),p))
for p,isPrime in enumerate(sieve,N):
if isPrime: return p
return nextPrime(N+len(sieve))
Minimally spaced 50-bit primes (generator):
import random
def randomPrimes(count,bits=50):
minVal = 2**bits
maxVal = minVal*2-1
minDiff = bits//2
prevPrime = 0
def oneBits(N): return N%2 + oneBits(N//2) if N else 0
for _ in range(count):
while True:
n = random.randint(minVal,maxVal)
if prevPrime and oneBits(prevPrime^n)<minDiff: continue
n = nextPrime(n)
if not prevPrime: break
if oneBits(prevPrime^n)>=minDiff: break
yield n
output:
for p in randomPrimes(10000):
print(p,f"{p:b}")
2071968049418461 111010111000111000110100111100100100101000011011101
1399795190350597 100111110010001101100110111000101000100001100000101
1530818178259709 101011100000100010101100001101110110001101011111101
1140103670657957 100000011001110101100010010010010111111001110100101
1911908333932859 110110010101101111011011001000101100101000100111011
1236033889571977 100011001000010101010010000111010110001010010001001
1752989992684999 110001110100101010111001001110011110000110111000111
1849449158362859 110100100100001000001110000000111010100011011101011
1349704431776567 100110010111000110010001101001101010011001100110111
1142235147712271 100000011101101101101011000001110101011011100001111
...
This steadily takes roughly 0.5 second per iteration (no matter how many iterations are done). I believe sympy should be much faster than my home made fucntions.

How to better implement recurrence relations in code?

I've been struggling with this problem for a couple days now, I'm still new to Python and more math intensive coding so any help would be appreciated, just point me in the right direction :)
So the question was something like:
You have a movie pass which is valid for N days. You can use it in any way you want except for 3 consecutive days or more.
So basically, you can either use your pass on a given day or choose to not, meaning 2 raised to N total possibilities. The valid ways for claiming the pass are then 2 raised to N - invalidCases
You have to find the number of valid cases % (10^9+7)
I found a recurrence relation for the invalid cases which looked like
invalidCases(at_N) = 2^(n-4) + 2*invalidCases(at_N-1) - invalidCases(at_n-4)
So my first impulse was to simply use recursion:
def invalidCases(n):
if(n<3):
return 0;
elif(n==3):
return 1;
else:
return 2**(n-4)+ 2*invalidCases(n-1)- invalidCases(n-4)
Very inefficient, but my equation seemed correct.
My next attempt, I tried memoization, but I kept running into an error at N=1006.
So I changed the recursion limit.
My present attempt(with memoization and increased recursion limit)
import sys
sys.setrecursionlimit(10**6)
T=int(input());
#2**(n-4) + 2*ans(n-1)-ans(n-4)
memo={0:0,1:0,2:0,3:1,4:3,5:8,6:20,7:47} #
def ans(n):
#complete this function
if n not in memo:
memo[n]=(2**(n-4) + 2*ans(n-1)-ans(n-4));
return memo[n];
modulo = 10**9 + 7;
print((2**n-ans(n))%modulo);
Finally, my problem.
I need this code to work for n = 999999.
How do I bring its worst case down to minimum?
Any pointers or tips would be great.

Here's a complete solution which is based on the observation that a valid solution for three or more days must begin with one of the following:
0
10
110
where 1 indicates the pass is used on that day, and 0 indicates it is not.
There are valid(n-1) possibilities for the first form, valid(n-2) possibilities for the second form, and valid(n-3) possibilities for the third form.
The recurrence is then:
valid(n) = valid(n-1) + valid(n-2) + valid(n-3)
The basis cases are valid(0) = 1, valid(1) = 2, and valid(2) = 4. It is important to note that valid(0) is 1, not zero. That's because there is exactly one solution when n=0, namely, the empty sequence. This is not only mathematically correct, but is also needed for the recurrence to work correctly.
The code does three things to make it run fast:
It uses a cache to cache results (memoization), as you had done.
It doesn't store the full results, but instead first applies the modulus, greatly reducing the value ranges.
It preloads the cache, starting with 0 and going up to the desired value. This reduces the maximum recursion depth to one.
Here's the code:
cache = {}
modulus = 10**9 + 7
def valid(n):
if n in cache:
return cache[n]
if n == 0:
v = 1
elif n == 1:
v = 2
elif n == 2:
v = 4
else:
v = valid(n-1) + valid(n-2) + valid(n-3)
v %= modulus
cache[n] = v
return v
def main():
# Preload the cache
for n in range(1000000):
valid(n)
print(valid(999999))
main()
Here's the output:
746580045
It runs in under 2 seconds on my system.
Update: Here's a minimal iterative solution, inspired by the approach used by MFisherKDX. The seed values were constructed in a way that eliminates the need for special-casing (the intial v2 is valid(0)):
modulus = 10**9 + 7
def valid(n):
v0, v1, v2 = 0, 1, 1
for i in range(n):
v0, v1, v2 = v1, v2, (v0 + v1 + v2) % modulus
return v2
print(valid(999999))
This solution is probably as fast as you can get. It discards the intermediate results after using them, which is fine if you're only calling the function once.

Here's my answer. Bottom up solution. Compare with Tom's answer which is top down and equally valid. At each day j it keeps track of the number of possibilities that use the pass on day j as well as the number of possibilities that use the pass on both j and j-1.
def ans(n):
total = 1
tcd = 0 #total used at current day
tcpd = 0 #total used at current and previous day
m = 1000000007
for j in range(0, n):
next_tot = 2*total - tcpd
next_tcd = total - tcpd
next_tcpd = tcd - tcpd
total = next_tot % m
tcd = next_tcd % m
tcpd = next_tcpd % m
return total
print(ans(999999))
Result is 746580045 and takes 400ms on my system.

Python 2 lists of positive integers finding prime number

Given 2 lists of positive integers, find how many ways you can select a number from each of the lists such that their sum is a prime number.
My code is tooo slow As i have both list1 and list 2 containing 50000 numbers each. So any way to make it faster so it solves it in minutes instead of days?? :)
# 2 is the only even prime number
if n == 2: return True
# all other even numbers are not primes
if not n & 1: return False
# range starts with 3 and only needs to go
# up the squareroot of n for all odd numbers
for x in range(3, int(n**0.5)+1, 2):
if n % x == 0: return False
return True
for i2 in l2:
for i1 in l1:
if isprime(i1 + i2):
n = n + 1 # increasing number of ways
s = "{0:02d}: {1:d}".format(n, i1 + i2)
print(s) # printing out

Sketch:
Following #Steve's advice, first figure out all the primes <= max(l1) + max(l2). Let's call that list primes. Note: primes doesn't really need to be a list; you could instead generate primes up the max one at a time.
Swap your lists (if necessary) so that l2 is the longest list. Then turn that into a set: l2 = set(l2).
Sort l1 (l1.sort()).
Then:
for p in primes:
for i in l1:
diff = p - i
if diff < 0:
# assuming there are no negative numbers in l2;
# since l1 is sorted, all diffs at and beyond this
# point will be negative
break
if diff in l2:
# print whatever you like
# at this point, p is a prime, and is the
# sum of diff (from l2) and i (from l1)
Alas, if l2 is, for example:
l2 = [2, 3, 100000000000000000000000000000000000000000000000000]
this is impractical. It relies on that, as in your example, max(max(l1), max(l2)) is "reasonably small".
Fleshed out
Hmm! You said in a comment that the numbers in the lists are up to 5 digits long. So they're less than 100,000. And you said at the start that the list have 50,000 elements each. So they each contain about half of all possible integers under 100,000, and you're going to have a very large number of sums that are primes. That's all important if you want to micro-optimize ;-)
Anyway, since the maximum possible sum is less than 200,000, any way of sieving will be fast enough - it will be a trivial part of the runtime. Here's the rest of the code:
def primesum(xs, ys):
if len(xs) > len(ys):
xs, ys = ys, xs
# Now xs is the shorter list.
xs = sorted(xs) # don't mutate the input list
sum_limit = xs[-1] + max(ys) # largest possible sum
ys = set(ys) # make lookups fast
count = 0
for p in gen_primes_through(sum_limit):
for x in xs:
diff = p - x
if diff < 0:
# Since xs is sorted, all diffs at and
# beyond this point are negative too.
# Since ys contains no negative integers,
# no point continuing with this p.
break
if diff in ys:
#print("%s + %s = prime %s" % (x, diff, p))
count += 1
return count
I'm not going to supply my gen_primes_through(), because it's irrelevant. Pick one from the other answers, or write your own.
Here's a convenient way to supply test cases:
from random import sample
xs = sample(range(100000), 50000)
ys = sample(range(100000), 50000)
print(primesum(xs, ys))
Note: I'm using Python 3. If you're using Python 2, use xrange() instead of range().
Across two runs, they each took about 3.5 minutes. That's what you asked for at the start ("minutes instead of days"). Python 2 would probably be faster. The counts returned were:
219,334,097
and
219,457,533
The total number of possible sums is, of course, 50000**2 == 2,500,000,000.
About timing
All the methods discussed here, including your original one, take time proportional to the product of two lists' lengths. All the fiddling is to reduce the constant factor. Here's a huge improvement over your original:
def primesum2(xs, ys):
sum_limit = max(xs) + max(ys) # largest possible sum
count = 0
primes = set(gen_primes_through(sum_limit))
for i in xs:
for j in ys:
if i+j in primes:
# print("%s + %s = prime %s" % (i, j, i+j))
count += 1
return count
Perhaps you'll understand that one better. Why is it a huge improvement? Because it replaces your expensive isprime(n) function with a blazing fast set lookup. It still takes time proportional to len(xs) * len(ys), but the "constant of proportionality" is slashed by replacing a very expensive inner-loop operation with a very cheap operation.
And, in fact, primesum2() is faster than my primesum() in many cases too. What makes primesum() faster in your specific case is that there are only around 18,000 primes less than 200,000. So iterating over the primes (as primesum() does) goes a lot faster than iterating over a list with 50,000 elements.
A "fast" general-purpose function for this problem would need to pick different methods depending on the inputs.

You should use the Sieve of Eratosthenes to calculate prime numbers.
You are also calculating the prime numbers for each possible combination of sums. Instead, consider finding the maximum value you can achieve with the sum from the lists. Generate a list of all the prime numbers up to that maximum value.
Whilst you are adding up the numbers, you can see if the number appears in your prime number list or not.

I would find the highest number in each range. The range of primes is the sum of the highest numbers.
Here is code to sieve out primes:
def eras(n):
last = n + 1
sieve = [0, 0] + list(range(2, last))
sqn = int(round(n ** 0.5))
it = (i for i in xrange(2, sqn + 1) if sieve[i])
for i in it:
sieve[i * i:last:i] = [0] * (n // i - i + 1)
return filter(None, sieve)
It takes around 3 seconds to find the primes up to 10 000 000. Then I would use the same n ^ 2 algorithm you are using for generating sums. I think there is an n logn algorithm but I can't come up with it.
It would look something like this:
from collections import defaultdict
possible = defaultdict(int)
for x in range1:
for y in range2:
possible[x + y] += 1
def eras(n):
last = n + 1
sieve = [0, 0] + list(range(2, last))
sqn = int(round(n ** 0.5))
it = (i for i in xrange(2, sqn + 1) if sieve[i])
for i in it:
sieve[i * i:last:i] = [0] * (n // i - i + 1)
return filter(None, sieve)
n = max(possible.keys())
primes = eras(n)
possible_primes = set(possible.keys()).intersection(set(primes))
for p in possible_primes:
print "{0}: {1} possible ways".format(p, possible[p])

Subset sum Problem

recently I became interested in the subset-sum problem which is finding a zero-sum subset in a superset. I found some solutions on SO, in addition, I came across a particular solution which uses the dynamic programming approach. I translated his solution in python based on his qualitative descriptions. I'm trying to optimize this for larger lists which eats up a lot of my memory. Can someone recommend optimizations or other techniques to solve this particular problem? Here's my attempt in python:
import random
from time import time
from itertools import product
time0 = time()
# create a zero matrix of size a (row), b(col)
def create_zero_matrix(a,b):
return [[0]*b for x in xrange(a)]
# generate a list of size num with random integers with an upper and lower bound
def random_ints(num, lower=-1000, upper=1000):
return [random.randrange(lower,upper+1) for i in range(num)]
# split a list up into N and P where N be the sum of the negative values and P the sum of the positive values.
# 0 does not count because of additive identity
def split_sum(A):
N_list = []
P_list = []
for x in A:
if x < 0:
N_list.append(x)
elif x > 0:
P_list.append(x)
return [sum(N_list), sum(P_list)]
# since the column indexes are in the range from 0 to P - N
# we would like to retrieve them based on the index in the range N to P
# n := row, m := col
def get_element(table, n, m, N):
if n < 0:
return 0
try:
return table[n][m - N]
except:
return 0
# same definition as above
def set_element(table, n, m, N, value):
table[n][m - N] = value
# input array
#A = [1, -3, 2, 4]
A = random_ints(200)
[N, P] = split_sum(A)
# create a zero matrix of size m (row) by n (col)
#
# m := the number of elements in A
# n := P - N + 1 (by definition N <= s <= P)
#
# each element in the matrix will be a value of either 0 (false) or 1 (true)
m = len(A)
n = P - N + 1;
table = create_zero_matrix(m, n)
# set first element in index (0, A[0]) to be true
# Definition: Q(1,s) := (x1 == s). Note that index starts at 0 instead of 1.
set_element(table, 0, A[0], N, 1)
# iterate through each table element
#for i in xrange(1, m): #row
# for s in xrange(N, P + 1): #col
for i, s in product(xrange(1, m), xrange(N, P + 1)):
if get_element(table, i - 1, s, N) or A[i] == s or get_element(table, i - 1, s - A[i], N):
#set_element(table, i, s, N, 1)
table[i][s - N] = 1
# find zero-sum subset solution
s = 0
solution = []
for i in reversed(xrange(0, m)):
if get_element(table, i - 1, s, N) == 0 and get_element(table, i, s, N) == 1:
s = s - A[i]
solution.append(A[i])
print "Solution: ",solution
time1 = time()
print "Time execution: ", time1 - time0

I'm not quite sure if your solution is exact or a PTA (poly-time approximation).
But, as someone pointed out, this problem is indeed NP-Complete.
Meaning, every known (exact) algorithm has an exponential time behavior on the size of the input.
Meaning, if you can process 1 operation in .01 nanosecond then, for a list of 59 elements it'll take:
2^59 ops --> 2^59 seconds --> 2^26 years --> 1 year
-------------- ---------------
10.000.000.000 3600 x 24 x 365
You can find heuristics, which give you just a CHANCE of finding an exact solution in polynomial time.
On the other side, if you restrict the problem (to another) using bounds for the values of the numbers in the set, then the problem complexity reduces to polynomial time. But even then the memory space consumed will be a polynomial of VERY High Order.
The memory consumed will be much larger than the few gigabytes you have in memory.
And even much larger than the few tera-bytes on your hard drive.
( That's for small values of the bound for the value of the elements in the set )
May be this is the case of your Dynamic programing algorithm.
It seemed to me that you were using a bound of 1000 when building your initialization matrix.
You can try a smaller bound. That is... if your input is consistently consist of small values.
Good Luck!

Someone on Hacker News came up with the following solution to the problem, which I quite liked. It just happens to be in python :):
def subset_summing_to_zero (activities):
subsets = {0: []}
for (activity, cost) in activities.iteritems():
old_subsets = subsets
subsets = {}
for (prev_sum, subset) in old_subsets.iteritems():
subsets[prev_sum] = subset
new_sum = prev_sum + cost
new_subset = subset + [activity]
if 0 == new_sum:
new_subset.sort()
return new_subset
else:
subsets[new_sum] = new_subset
return []
I spent a few minutes with it and it worked very well.

An interesting article on optimizing python code is available here. Basically the main result is that you should inline your frequent loops, so in your case this would mean instead of calling get_element twice per loop, put the actual code of that function inside the loop in order to avoid the function call overhead.
Hope that helps! Cheers

, 1st eye catch
def split_sum(A):
N_list = 0
P_list = 0
for x in A:
if x < 0:
N_list+=x
elif x > 0:
P_list+=x
return [N_list, P_list]
Some advices:
Try to use 1D list and use bitarray to reduce memory footprint at minimum (http://pypi.python.org/pypi/bitarray) so you will just change get / set functon. This should reduce your memory footprint by at lest 64 (integer in list is pointer to integer whit type so it can be factor 3*32)
Avoid using try - catch, but figure out proper ranges at beginning, you might found out that you will gain huge speed.

The following code works for Python 3.3+ , I have used the itertools module in Python that has some great methods to use.
from itertools import chain, combinations
def powerset(iterable):
s = list(iterable)
return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))
nums = input("Enter the Elements").strip().split()
inputSum = int(input("Enter the Sum You want"))
for i, combo in enumerate(powerset(nums), 1):
sum = 0
for num in combo:
sum += int(num)
if sum == inputSum:
print(combo)
The Input Output is as Follows:
Enter the Elements 1 2 3 4
Enter the Sum You want 5
('1', '4')
('2', '3')

Just change the values in your set w and correspondingly make an array x as big as the len of w then pass the last value in the subsetsum function as the sum for which u want subsets and you wl bw done (if u want to check by giving your own values).
def subsetsum(cs,k,r,x,w,d):
x[k]=1
if(cs+w[k]==d):
for i in range(0,k+1):
if x[i]==1:
print (w[i],end=" ")
print()
elif cs+w[k]+w[k+1]<=d :
subsetsum(cs+w[k],k+1,r-w[k],x,w,d)
if((cs +r-w[k]>=d) and (cs+w[k]<=d)) :
x[k]=0
subsetsum(cs,k+1,r-w[k],x,w,d)
#driver for the above code
w=[2,3,4,5,0]
x=[0,0,0,0,0]
subsetsum(0,0,sum(w),x,w,7)