Develop Reference

Precision error in `scipy.stats.binom` method

I am using scipy.stats.binom to work with the binomial distribution. Given n and p, the probability function is
A sum over k ranging over 0 to n should (and indeed does) give 1. Fixing a point x_0, we can add the probabilities in both directions and the two sums ought to add to 1. However the code below yields two different answers when x_0 is close to n.
from scipy.stats import binom
n = 9
p = 0.006985
b = binom(n=n, p=p)
x_0 = 8
# Method 1
cprob = 0
for k in range(x_0, n+1):
cprob += b.pmf(k)
print('cumulative prob with method 1:', cprob)
# Method 2
cprob = 1
for k in range(0, x_0):
cprob -= b.pmf(k)
print('cumulative prob with method 2:', cprob)
I expect the outputs from both methods to agree. For x_0 < 7 it agrees but for x_0 >= 8 as above I get
>> cumulative prob with method 1: 5.0683768775504006e-17
>> cumulative prob with method 2: 1.635963929799698e-16
The precision error in the two methods propagates through my code (later) and gives vastly different answers. Any help is appreciated.

Roundoff errors of the order of the machine epsilon are expected and are inevitable. That these propagate and later blow up means that your problem is very poorly conditioned. You'd need to rethink the algorithm or an implementation, depending on where the bad conditioning comes from.
In your specific example you can get by using either np.sum (which tries to be careful with roundoff), or even math.fsum from the standard library.

Power Function Python

This function is calculates the value of a^b and returns it.My question is if m=log(b)
the best case scenario is that it does m+1 interactions
but what is the worst case? and how many times it enters the while loop?
def power(a,b):
result=1
while b>0: # b is nonzero
if b % 2 == 1:
result=result*a
a=a*a
b = b//2
return result

As #EliSadoff stated in a comment, you need an initial value of result in your function. Insert the line
result = 1
just after the def line. The code then works, and this is a standard way to implicitly use the binary representation of b to quickly get the exponentiation. (The loop invariant is that the value of result * a ** b remains constant, which shows the validity of this algorithm.)
The worst case is where your if b % 2 line is executed every time through the while loop. This will happen whenever b is one less than a power of 2, so every digit in bs binary representation is one. The while loop condition while b>0 is still checked only m+1 times, but each loop now has a little more to do.
There are several ways to speed up your code. Use while b rather than while b>0 and if b & 1 rather than if b % 2 = 1. Use result *= a rather than result = result*a and a *= a rather than a = a*a and b >>= 1 rather than b = b // 2. These are fairly minor improvements, of course. The only way to speed up the loop further is to use non-structured code, which I believe isn't possible in Python. (There is one more modification of a than is necessary but there is no good way to prevent that without a jump into a loop.) There are some variations on this code, such as an inner loop to keep modifying a and b as long as b is even, but that is not always faster.
The final code is then
def power(a, b):
"""Return a ** b, assuming b is a nonnegative integer"""
result = 1
while b:
if b & 1:
result *= a
a *= a
b >>= 1
return result
I cleaned up your code a little to better fit PEP8 (Python style standards). Note that there is no error checking in your code, especially to ensure that b is a nonnegative integer. I believe my code gets an infinite loop if b is a negative integer while yours returns a false result. So please do that error check! Also note that your code says power(0, 0) == 1 which is pretty standard for such a function but still takes some people by surprise.

if statements or while loop conditions?

So for an assignment, I'm supposed to write a function that evaluates two other functions that I previously defined in the program. How, the intricacies of this next function I must define is confusing me how I should set it up. Basically, I have determined that I have to write a while loop, but I can't decide whether to have three different conditions or if statements to evaluate this function. The thing is, the loop must end after one of the three conditions is met; I just don't know how to lay it out currently.
These are the three conditions (where only one must be met in order for the loop/function to terminate):
the absolute value of the polynomial at the current estimate is less than epsilon, in
which case the method is successful.
the absolute value of the derivative of the polynomial at the current estimate is less
than epsilon (this could lead to division by 0), in which case the method failed, or
the number of revisions of the estimate has exceeded timeout (the estimate is not
converging on a solution). This case is also a failure.
This is currently what I have as my code (and it hasn't been running obviously):
def newtonsMethod(poly, x_, epislon, timeout):
"""Calculating root of polynomial using newton's Method."""
estimate = 0
epislon = 1e-20
while (abs(evaluatePoly(poly, x_)) < epislon and \
abs(evaluatePoly(findDerivative(poly), x_)) and estimate > timeout):
estimate = x_ - (evaluatePoly(poly) / evaluatePoly(findDerivative(poly)))
x_ = estimate
print(x_)
How should I go about this? The function name is a requirement of the assignment so it cannot be changed. Also, I am a complete beginner at this stuff (just started last month) and I'm only basically required to have knowledge of data structures, loops, and if statements (and functions). So please keep your responses as simple/dumby proof as possible.
This is all the code I have before that pertains to this question:
def findDerivative(poly):
"""Find the derivative of the polynomial and return it as a list of
coefficents"""
count = 0
polyList = []
for nn in poly:
polyList += [nn * count]
count += 1
return polyList[1:]
def evaluatePoly(poly, x_):
"""Evaluates the polynomial at x = x_ and returns the result as a floating-
point number using Horner's rule"""
#http://mathworld.wolfram.com/HornersRule.html
count = 0
polyEval = []
for nn in poly:
xVal = x_**count
polyEval += [xVal * nn]
count += 1
polySum = sum(polyEval)
return float(polySum)
Edit: 10/23/2016
I should have mentioned this before or explained the assignment in more detail, but estimate is a parameter for the function because it is given as a list later on in the assignment. I have understood that timeout represents that the loop can run through so many times before it "breaks" or ends. The function is to return a two-part tuple where the first variable of the tuple is the refined x_ value and the second variable is a boolean value saying whether the method was successful or not. Using advice from both #Blckknght and #David Bowling , I was able to refine my code to the following:
def newtonsMethod(poly, x_, epsilon, timeout):
"""Calculating root of polynomial using newton's Method."""
deriv_poly = findDerivative(poly)
deriv = evaluatePoly(deriv_poly, x_)
value = evaluatePoly(poly, x_)
count = 1
while True:
x_ -= value / deriv
if abs(value) < epsilon:
boolean = abs(value) < epsilon
xTuple = (x_, boolean)
return xTuple
if abs(deriv) < epsilon or count > timeout:
boolean = abs(deriv) < epsilon or count > timeout
xTuple = (x_, boolean)
return xTuple
count += 1
However, I am still experiencing crashing problems when I run the code. I realized belatedly that I cannot have both boolean variables to equal to True (for my following code, I need boolean to equal True or False) but in order for the loop to stop, one of the if statements must equal True.
I apologize for all the confusion and specifications this code requires; I hope this explanation helps understanding what I need.

The advice given by others to break up your complicated conditional into simpler statements is very good. The logic in your original conditional was faulty, but could be adjusted to work. Yet there were other problems in your function.
There is no need to assign epsilon a value before the loop, because this value is supplied as a function parameter. estimate should be functioning as a counter, but then confusingly gets assigned the value to be used for x_ in the next iteration. I have renamed estimate as iterations in my code below.
You should have used:
abs(evaluatePoly(poly, x_)) > epsilon
since you want the loop to continue so long as the value of the polynomial evaluated at x_ is larger than epsilon. And, to avoid difficulties in the statement that updates the value of x_, you want to continue looping only if:
abs(evaluatePoly(findDerivative(poly), x_)) > epsilon
There was a missing conditional here in your original code. Finally, you want to continue looping if the number of iterations has not yet reached the timeout value. You only want to loop if all of the above conditions are met, so your conditional expression should be:
while abs(evaluatePoly(poly, x_)) > epsilon and \
abs(evaluatePoly(findDerivative(poly), x_)) > epsilon and \
iterations < timeout:
I changed your next assignment statement in two ways. First, the assignment is now made to x_ instead of the old, confusing estimate variable. Second, you were missing the second argument in both of the evaluatePoly() functions. I added a line to increment the iterations counter, and moved the print() out of the loop so that you only see the final result.
Here is the modified code:
def newtonsMethod(poly, x_, epsilon, timeout):
"""Calculating root of polynomial using newton's Method."""
iterations = 0
while abs(evaluatePoly(poly, x_)) > epsilon and \
abs(evaluatePoly(findDerivative(poly), x_)) > epsilon and \
iterations < timeout:
x_ = x_ - (evaluatePoly(poly, x_) / evaluatePoly(findDerivative(poly), x_))
iterations += 1
print(x_)
Here is a sample run using the findDerivative() and evaluatePoly() functions that you provided:
>>> xpr = [0, 0, 5]
>>> newtonsMethod(xpr, -1, 1e-20, 1000)
-2.91038304567e-11
>>> evaluatePoly(xpr, -2.91038304567e-11)
4.235164736261692e-21
Update:
I have looked at the new function definition that you posted, and found the problem. The fundamental problem in your code is that value and deriv are evaluated only once, outside the loop. You need to move these two statements inside of the loop so that they can be updated after each iteration. Also, I would move the assigment to x_ to the end of the loop so that your first guess gets tested before it is updated. You can simplify the bodies of your if statements a bit also. As it is, when you escape the function due to an unacceptably small value for deriv or a timeout, your function reports True. I assume that you want it to report False. Here is how I modified your code:
def newtonsMethod(poly, x_, epsilon, timeout):
"""Calculating root of polynomial using newton's Method."""
deriv_poly = findDerivative(poly)
count = 1
while True:
deriv = evaluatePoly(deriv_poly, x_)
value = evaluatePoly(poly, x_)
if abs(value) < epsilon:
boolean = True
return (x_, boolean)
if abs(deriv) < epsilon or count > timeout:
boolean = False
return (x_, boolean)
x_ -= value / deriv
count += 1
Here is a sample run:
>>> xpr = [0, 0, 5]
>>> newtonsMethod(xpr, -1, 1e-20, 1000)
(-2.9103830456733704e-11, True)
>>> evaluatePoly(xpr, -2.9103830456733704e-11)
4.235164736271502e-21
I am guessing that the slight difference between these results and the earlier results is due to rounding errors and the differing procedures for calculation.

As Barmar commented, using a simpler while condition (possible just while True) is almost always better than a very complicated condition that can't even fit on one line. You'd put the actual logic for ending the loop in one or more if statements inside the loop body:
while True: # no condition here
# do stuff
if failure_condition1():
break # stop looping
if success_condition():
return result() # another way to stop looping is to return a value
if failure_condition2():
raise SomeException() # you could also raise an exception on failure
# do other stuff
Any combination of return, raise or break could be correct for your function, it depends exactly what you want to do when each condition is met (and what you want the API to be).
One advantage to this style of loop is that you can check the conditions wherever it makes sense in your algorithm. The test doesn't have to be at the start of the loop (which often requires you to initialize your variables with made up values that will pass the condition the first time through).
Here's how I'd apply this kind of control flow in your specific problem:
def newtonsMethod(poly, x_, timeout):
"""Calculating root of polynomial using newton's Method."""
estimate = 0
epsilon = 1e-20
deriv_poly = findDerivative(poly)
while True:
value = evaluatePoly(poly, x_)
if abs(value) < epsilon:
return x_
deriv = evaluatePoly(deriv_poly, x_)
if abs(deriv) < epsilon or estimate > timeout:
raise ValueError("Not converging")
#print(x_, value, deriv)
x_ -= value / deriv
estimate += 1
Note that we only need to call evaluatePoly twice per pass, since we save the results rather than recomputing them.
I also fixed a few errors in your code. I use estimate as a counter, rather than an alias for x_ and no longer have epsilon as a parameter that is overwritten in the function body (you also had it misspelled as epislon).

Python - How to improve efficiency of complex recursive function?

In this video by Mathologer on, amongst other things, infinite sums there are 3 different infinite sums shown at 9:25, when the video freezes suddenly and an elephant diety pops up, challenging the viewer to find "the probable values" of the expressions. I wrote the following script to approximate the last of the three (i.e. 1 + 3.../2...) with increasing precision:
from decimal import Decimal as D, getcontext # for accurate results
def main(c): # faster code when functions defined locally (I think)
def run1(c):
c += 1
if c <= DEPTH:
return D(1) + run3(c)/run2(c)
else:
return D(1)
def run2(c):
c += 1
if c <= DEPTH:
return D(2) + run2(c)/run1(c)
else:
return D(2)
def run3(c):
c += 1
if c <= DEPTH:
return D(3) + run1(c)/run3(c)
else:
return D(3)
return run1(c)
getcontext().prec = 10 # too much precision isn't currently necessary
for x in range(1, 31):
DEPTH = x
print(x, main(0))
Now this is working totally fine for 1 <= x <= 20ish, but it starts taking an eternity for each result after that. I do realize that this is due to the exponentially increasing number of function calls being made at each DEPTH level. It is also clear that I won't be able to calculate the series comfortably up to an arbitrary point. However, the point at which the program slows down is too early for me to clearly identify the limit the series it is converging to (it might be 1.75, but I need more DEPTH to be certain).
My question is: How do I get as much out of my script as possible (performance-wise)?
I have tried:
1. finding the mathematical solution to this problem. (No matching results)
2. finding ways to optimize recursive functions in general. According to multiple sources (e.g. this), Python doesn't optimize tail recursion by default, so I tried switching to an iterative style, but I ran out of ideas on how to accomplish this almost instantly...
Any help is appreciated!
NOTE: I know that I could go about this mathematically instead of "brute-forcing" the limit, but I want to get my program running well, now that I've started...

You can store the results of the run1, run2 and run3 functions in arrays to prevent them from being recalculated every time, since in your example, main(1) calls run1(1), which calls run3(2) and run2(2), which in turn call run1(3), run2(3), run1(3) (again) and run3(3), and so on.
You can see that run1(3) is being called evaluated twice, and this only gets worse as the number increases; if we count the number of times each function is called, those are the results:
run1 run2 run3
1 1 0 0
2 0 1 1
3 1 2 1
4 3 2 3
5 5 6 5
6 11 10 11
7 21 22 21
8 43 42 43
9 85 86 85
...
20 160,000 each (approx.)
...
30 160 million each (approx.)
This is actually a variant of a Pascal triangle, and you could probably figure out the results mathematically; but since here you asked for a non mathematical optimization, just notice how the number of calls increases exponentially; it doubles at each iteration. This is even worse since each call will generate thousands of subsequent calls with higher values, which is what you want to avoid.
Therefore what you want to do is store the value of each call, so that the function does not need to be called a thousand times (and itself make thousands more calls) to always get the same result. This is called memoization.
Here is an example solution in pseudo code:
before calling main, declare the arrays val1, val2, val3, all of size DEPTH, and fill them with -1
function run1(c) # same thing for run2 and run3
c += 1
if c <= DEPTH
local3 = val3(c) # read run3(c)
if local3 is -1 # if run3(c) hasn't been computed yet
local3 = run3(c) # we compute it
val3(c) = local3 # and store it into the array
local2 = val2(c) # same with run2(c)
if local2 is -1
local2 = run2(c)
val2(c) = local2
return D(1) + local3/local2 # we use the value we got from the array or from the computation
else
return D(1)
Here I use -1 since your functions seem to only generate positive numbers, and -1 is an easy placeholder for the empty cells. In other cases you might have to use an object as Cabu below me did. I however think this would be slower due to the cost of retrieving properties in an object versus reading an array, but I might be wrong about that. Either way, your code should be much, much faster with it is now, with a cost of O(n) instead of O(2^n).
This would technically allow your code to run forever at a constant speed, but the recursion will actually cause an early stack overflow. You might still be able to get to a depth of several thousands before that happens though.
Edit: As ShadowRanger added in the comments, you can keep your original code and simply add #lru_cache(maxsize=n) before each of your run1, run2 and run3 functions, where n is one of the first powers of two above DEPTH (for example, 32 if depth is 25). This might require an import directive to work.

With some memoization, You could get up to the stack overflow:
from decimal import Decimal as D, getcontext # for accurate results
def main(c): # faster code when functions defined locally (I think)
mrun1 = {} # store partial results of run1, run2 and run3
# This have not been done in the as parameter of the
# run function to be able to reset them easily
def run1(c):
if c in mrun1: # if partial result already computed, return it
return mrun1[c]
c += 1
if c <= DEPTH:
v = D(1) + run3(c) / run2(c)
else:
v = D(1)
mrun1[c] = v # else store it and return the value
return v
def run2(c):
if c in mrun2:
return mrun2[c]
c += 1
if c <= DEPTH:
v = D(2) + run2(c) / run1(c)
else:
v = D(2)
mrun2[c] = v
return v
def run3(c):
if c in mrun3:
return mrun3[c]
c += 1
if c <= DEPTH:
v = D(3) + run1(c) / run3(c)
else:
v = D(3)
mrun3[c] = v
return v
return run1(c)
getcontext().prec = 150 # too much precision isn't currently necessary
for x in range(1, 997):
DEPTH = x
print(x, main(0))
Python will stack overflow if you go over 997.

Python 3 Recursion - Maximum Depth Exceeded

I'm writing a basic recursion function that takes an integer, n, and returns the sum of the first n reciprocals. Inputting 2 should result in 1.5, and inputting 0 should return 0.
sum_to(2) = (1 + 1/2) = 1.5
Here's what I have:
def sum_to(n):
if n>0:
return sum_to(1+1/n) # Not sure if this is correct
return 0
But all I'm getting is a Maximum recursion depth exceeded. I know I could lists to solve this, but recursion is really intriguing and I want to find a solution using it.

From the question, it seems that you want to compute
\sum_{i=1}^{n} 1/i
If the input is 0, then returns 0.
Remember that in a recursive function, you need to define a base case and an inductive clause.
In your question, the inductive clause is:
1.0 / i + sum_of_reciprocals(i-1)
while the base case can be set to 0 when i=0.
Given this formulation, the function with an example looks like:
def sum_to(n):
if n > 0:
return sum_to(n-1) + 1.0 / n
else:
return 0
if __name__ == '__main__':
print(sum_to(3))
and the result is 1.83333333333.
For more information about recursive functions, you can start from the related wikipedia page.

Trace through it to see if you ever reach your end condition:
(PS: at the time I wrote this the question's code was return 1 + sum_to(1/n))
sum_to(2):
n is 2, which is > 0
call sum_to(1/2)
n is .5, which is > 0
call sum_to(1/.5)
n is 2, which is > 0
call sum_to(1/2)
...
Your function never returns. The result would be infinite if you had infinite time and space to finish the calculation, since there is no end to the algorithm.
For your example, just write it this way:
def sum_to(n):
return 1 + 1/n
What is the expected result when n > 2? That will determine how to approach organizing working code.