Develop Reference

Floating-point errors in cube root of exact cubic input

I found myself needing to compute the "integer cube root", meaning the cube root of an integer, rounded down to the nearest integer. In Python, we could use the NumPy floating-point cbrt() function:
import numpy as np
def icbrt(x):
return int(np.cbrt(x))
Though this works most of the time, it fails at certain input x, with the result being one less than expected. For example, icbrt(15**3) == 14, which comes about because np.cbrt(15**3) == 14.999999999999998. The following finds the first 100,000 such failures:
print([x for x in range(100_000) if (icbrt(x) + 1)**3 == x])
# [3375, 19683, 27000, 50653] == [15**3, 27**3, 30**3, 37**3]
Question: What is special about 15, 27, 30, 37, ..., making cbrt() return ever so slightly below the exact result? I can find no obvious underlying pattern for these numbers.
A few observations:
The story is the same if we switch from NumPy's cbrt() to that of Python's math module, or if we switch from Python to C (not surprising, as I believe that both numpy.cbrt() and math.cbrt() delegate to cbrt() from the C math library in the end).
Replacing cbrt(x) with x**(1/3) (pow(x, 1./3.) in C) leads to many more cases of failure. Let us stick to cbrt().
For the square root, a similar problem does not arise, meaning that
import numpy as np
def isqrt(x):
return int(np.sqrt(x))
returns the correct result for all x (tested up to 100,000,000). Test code:
print([x for x in range(100_000) if (y := np.sqrt(x))**2 != x and (y + 1)**2 <= x])
Extra
As the above icbrt() only seems to fail on cubic input, we can correct for the occasional mistakes by adding a fixup, like so:
import numpy as np
def icbrt(x):
y = int(np.cbrt(x))
if (y + 1)**3 == x:
y += 1
return y
A different solution is to stick to exact integer computation, implementing icbrt() without the use of floating-point numbers. This is discussed e.g. in this SO question. An extra benefit of such approaches is that they are (or can be) faster than using the floating-point cbrt().
To be clear, my question is not about how to write a better icbrt(), but about why cbrt() fails at some specific inputs.

This problem is caused by a bad implementation of cbrt. It is not caused by floating-point arithmetic because floating-point arithmetic is not a barrier to computing the cube root well enough to return an exactly correct result when the exactly correct result is representable in the floating-point format.
For example, if one were to use integer arithmetic to compute nine-fifths of 80, we would expect a correct result of 144. If a routine to compute nine-fifths of a number were implemented as int NineFifths(int x) { return 9/5*x; }, we would blame that routine for being implemented incorrectly, not blame integer arithmetic for not handling fractions. Similarly, if a routine uses floating-point arithmetic to calculate an incorrect result when a correct result is representable, we blame the routine, not floating-point arithmetic.
Some mathematical functions are difficult to calculate, and we accept some amount of error in them. In fact, for some of the routines in the math library, humans have not yet figured out how to calculate them with correct rounding in a known-bounded execution time. So we accept that not every math routine is correctly rounded.
Howver, when the mathematical value of a function is exactly representable in a floating-point format, the correct result can be obtained by faithful rounding rather than correct rounding. So this is a desirable goal for math library functions.
Correctly rounded means the computed result equals the number you would obtain by rounding the exact mathematical result to the nearest representable value.1 Faithfully rounded means the computed result is less than one ULP from the exact mathematical result. An ULP is the unit of least precision, the distance between two adjacent representable numbers.
Correctly rounding a function can be difficult because, in general, a function can be arbitrarily close to a rounding decision point. For round-to-nearest, this is midway between two adjacent representable numbers. Consider two adjacent representable numbers a and b. Their midpoint is m = (a+b)/2. If the mathematical value of some function f(x) is just below m, it should be rounded to a. If it is just above, it should be rounded to b. As we implement f in software, we might compute it with some very small error e. When we compute f(x), if our computed result lies in [m-e, m+e], and we only know the error bound is e, then we cannot tell whether f(x) is below m or above m. And because, in general, a function f(x) can be arbitrarily close to m, this is always a problem: No matter how accurately we compute f, no matter how small we make the error bound e, there is a possibility that our computed value will lie very close to a midpoint m, closer than e, and therefore our computation will not tell us whether to round down or to round up.
For some specific functions and floating-point formats, studies have been made and proofs have been written about how close the functions approach such rounding decision points, and so certain functions like sine and cosine can be implemented with correct rounding with known bounds on the compute time. Other functions have eluded proof so far.
In contrast, faithful rounding is easier to implement. If we compute a function with an error bound less than ½ ULP, then we can always return a faithfully rounded result, one that is within one ULP of the exact mathematical result. Once we have computed some result y, we round that to the nearest representable value2 and return that. Starting with y having error less than ½ ULP, the rounding may add up to ½ ULP more error, so the total error is less than one ULP, which is faithfully rounded.
A benefit of faithful rounding is that a faithfully rounded implementation of a function always produces the exact result when the exact result is representable. This is because the next nearest result is one ULP away, but faithful rounding always has an error less than one ULP. Thus, a faithfully rounded cbrt function returns exact results when they are representable.
What is special about 15, 27, 30, 37, ..., making cbrt() return ever so slightly below the exact result? I can find no obvious underlying pattern for these numbers.
The bad cbrt implementation might compute the cube root by reducing the argument to a value in [1, 8) or similar interval and then applying a precomputed polynomial approximation. Each addition and multiplication in that polynomial may introduce a rounding error as the result of each operation is rounded to the nearest representable value in floating-point format. Additionally, the polynomial has inherent error. Rounding errors behave somewhat like a random process, sometimes rounding up, sometimes down. As they accumulate over several calculations, they may happen to round in different directions and cancel, or they may round in the same direction ad reinforce. If the errors happen to cancel by the end of the calculations, you get an exact result from cbrt. Otherwise, you may get an incorrect result from cbrt.
Footnotes
1 In general, there is a choice of rounding rules. The default and most common is round-to-nearest, ties-to-even. Others include round-upward, round-downward, and round-toward-zero. This answer focuses on round-to-nearest.
2 Inside a mathematical function, numbers may be computed using extended precision, so we may have computed results that are not representable in the destination floating-point format; they will have more precision.

What are the odds of a repeat in numpy.random.rand(n) (assuming perfect randomness)?

For the moment, put aside any issues relating to pseudorandom number generators and assume that numpy.random.rand perfectly samples from the discrete distribution of floating point numbers over [0, 1). What are the odds getting at least two exactly identical floating point numbers in the result of:
numpy.random.rand(n)
for any given value of n?
Mathematically, I think this is equivalent to first asking how many IEEE 754 singles or doubles there are in the interval [0, 1). Then I guess the next step would be to solve the equivalent birthday problem? I'm not really sure. Anyone have some insight?

The computation performed by numpy.random.rand for each element generates a number 0.<53 random bits>, for a total of 2^53 equally likely outputs. (Of course, the memory representation isn't a fixed-point 0.stuff; it's still floating point.) This computation is incapable of producing most binary64 floating-point numbers between 0 and 1; for example, it cannot produce 1/2^60. You can see the code in numpy/random/mtrand/randomkit.c:
double
rk_double(rk_state *state)
{
/* shifts : 67108864 = 0x4000000, 9007199254740992 = 0x20000000000000 */
long a = rk_random(state) >> 5, b = rk_random(state) >> 6;
return (a * 67108864.0 + b) / 9007199254740992.0;
}
(Note that rk_random produces 32-bit outputs, regardless of the size of long.)
Assuming a perfect source of randomness, the probability of repeats in numpy.random.rand(n) is 1-(1-0/k)(1-1/k)(1-2/k)...(1-(n-1)/k), where k=2^53. It's probably best to use an approximation instead of calculating this directly for large values of n. (The approximation may even be more accurate, depending on how the approximation error compares to the rounding error accumulated in a direct computation.)

I think you are correct, this is like the birthday problem.
But you need to decide on the number of possible options. You do this by deciding the precision of your floating point numbers.
For example, if you decide to have a precision of 2 numbers after the dot, then there are 100 options(including zero and excluding 1).
And if you have n numbers then the probability of not having a collision is:
or when given R possible numbers and N data points, the probability of no collision is:
And of collision is 1 - P.
This is because the probability of getting any given number is 1/R. And at any point, the probability of a data point not colliding with prior data points is (R-i)/R for i being the index of the data point. But to get the probability of no data points colliding with each other, we need to multiply all the probabilities of data points not colliding with those prior to them. Applying some algebraic operations, we get the equation above.

python value becomes zero, how to prevent

I have a numerical problem while doing likelihood ratio tests in python. I'll not go into too much detail about what the statistics mean, my problems comes down to calculating this:
LR = LR_H0 / LR_h1
where LR is the number of interest and LR_H0 and LR_H1 are numbers that can be VERY close to zero. This leads to a few numerical issues; if LR_H1 is too small then python will recognise this as a division by zero.
ZeroDivisionError: float division by zero
Also, although this is not the main issue, if LR_H1 is small enough to allow the division then the fraction LR_H0 / LR_h1 might become too big (I'm assuming that python also has an upper limit value of what a float can be).
Any tips on what the best way is to circumvent this problem? I'm considering doing something like:
def small_enough( num ):
if num == 0.0:
return *other small number*
else:
return num
But this is not ideal because it would approximate the LR value and I would like to guarantee some precision.

Work with logarithms. Take the log of all your likelihoods, and add or subtract logarithms instead of multiplying and dividing. You'll be able to work with much greater ranges of values without losing precision.

Float precision breakdown in python/numpy when adding numbers

I have some problems due to really low numbers used with numpy. It took me several weeks to trace back my constant problems with numerical integration to the fact, that when I add up floats in a function the float64 precision gets lost. Performing the mathematically identic calculation with a product instead of a sum leads to values that are alright.
Here is a code sample and a plot of the results:
from matplotlib.pyplot import *
from numpy import vectorize, arange
import math
def func_product(x):
return math.exp(-x)/(1+math.exp(x))
def func_sum(x):
return math.exp(-x)-1/(1+math.exp(x))
#mathematically, both functions are the same
vecfunc_sum = vectorize(func_sum)
vecfunc_product = vectorize(func_product)
x = arange(0.,300.,1.)
y_sum = vecfunc_sum(x)
y_product = vecfunc_product(x)
plot(x,y_sum, 'k.-', label='sum')
plot(x,y_product,'r--',label='product')
yscale('symlog', linthreshy=1E-256)
legend(loc='lower right')
show()
As you can see, the summed values that are quite low are scattered around zero or are exactly zero while the multiplicated values are fine...
Please, could someone help/explain? Thanks a lot!

Floating point precision is pretty sensitive to addition/subtraction due to roundoff error. Eventually, 1+exp(x) gets so big that adding 1 to exp(x) gives the same thing as exp(x). In double precision that's somewhere around exp(x) == 1e16:
>>> (1e16 + 1) == (1e16)
True
>>> (1e15 + 1) == (1e15)
False
Note that math.log(1e16) is approximately 37 -- Which is roughly where things go crazy on your plot.
You can have the same problem, but on different scales:
>>> (1e-16 + 1.) == (1.)
True
>>> (1e-15 + 1.) == (1.)
False
For a vast majority of the points in your regime, your func_product is actually calculating:
exp(-x)/exp(x) == exp(-2*x)
Which is why your graph has a nice slope of -2.
Taking it to the other extreme, you're other version is calculating (at least approximately):
exp(-x) - 1./exp(x)
which is approximately
exp(-x) - exp(-x)

This is an example of catastrophic cancellation.
Let's look at the first point where the calculation goes awry, when x = 36.0
In [42]: np.exp(-x)
Out[42]: 2.3195228302435691e-16
In [43]: - 1/(1+np.exp(x))
Out[43]: -2.3195228302435691e-16
In [44]: np.exp(-x) - 1/(1+np.exp(x))
Out[44]: 0.0
The calculation using func_product does not subtract nearly equal numbers, so it avoids the catastrophic cancellation.
By the way, if you change math.exp to np.exp, you can get rid of np.vectorize (which is slow):
def func_product(x):
return np.exp(-x)/(1+np.exp(x))
def func_sum(x):
return np.exp(-x)-1/(1+np.exp(x))
y_sum = func_sum_sum(x)
y_product = func_product_product(x)

The problem is that your func_sum is numerically unstable because it involves a subtraction between two very close values.
In the calculation of func_sum(200), for example, math.exp(-200) and 1/(1+math.exp(200)) have the same value, because adding 1 to math.exp(200) has no effect, since it is outside the precision of 64-bit floating point:
math.exp(200).hex()
0x1.73f60ea79f5b9p+288
(math.exp(200) + 1).hex()
0x1.73f60ea79f5b9p+288
(1/(math.exp(200) + 1)).hex()
0x1.6061812054cfap-289
math.exp(-200).hex()
0x1.6061812054cfap-289
This explains why func_sum(200) gives zero, but what about the points that lie off the x axis? These are also caused by floating point imprecision; it occasionally happens that math.exp(-x) is not equal to 1/math.exp(x); ideally, math.exp(x) is the closest floating-point value to e^x, and 1/math.exp(x) is the closest floating-point value to the reciprocal of the floating-point number calculated by math.exp(x), not necessarily to e^-x. Indeed, math.exp(-100) and 1/(1+math.exp(100)) are very close and in fact only differ in the last unit:
math.exp(-100).hex()
0x1.a8c1f14e2af5dp-145
(1/math.exp(100)).hex()
0x1.a8c1f14e2af5cp-145
(1/(1+math.exp(100))).hex()
0x1.a8c1f14e2af5cp-145
func_sum(100).hex()
0x1.0000000000000p-197
So what you have actually calculated is the difference, if any, between math.exp(-x) and 1/math.exp(x). You can trace the line of the function math.pow(2, -52) * math.exp(-x) to see that it passes through the positive values of func_sum (recall that 52 is the size of the significand in 64-bit floating point).

Python:Which way gives better precision

Is there any difference in precision between one time assignment:
res=n/k
and multiple assignment in for cycle:
for i in range(n):
res+=1/k
?

Floating-point division a/b is not mathematical division a ÷ b, except in very rare* circumstances.
Generally, floating point division a/b is a ÷ b + ε.
This is true for two reasons.
Float numbers (except in rare cases) are an approximation of the decimal number.
a is a + εa.
b is b + εb.
Float numbers uses a base 2 encoding of the digits to the right of the decimal place. When you write 3.1, this is expanded to a base-2 approximation that differs from the real value by a small amount.
Real decimal numbers have the same problem, by the way. Write down the decimal expansion of 1/3. Oops. You have to stop writing decimal places at some point. Binary floating point numbers have the same problem.
Division has a fixed number of binary places, meaning the answer is truncated. If there's a repeating binary pattern, it gets chopped. In rare cases, this doesn't matter. In general, you've introduced error by doing division.
Therefore, when you do something like repeatedly add 1/k values you're computing
1 ÷ k + ε
And adding those up. Your result (if you had the right range) would be
n × (1 ÷ k + ε) = n ÷ k + n × ε
You've multiplied the small error, ε, by n. Making it a big error. (Except in rare cases.)
This is bad. Very bad. All floating point division introduces an error. Your job as a programmer is to do the algebra to avoid or defer division to prevent this. Good software design means good algebra to prevent errors being introduced by the division operator.
[* The rare cases. In rare cases, the small error happens to be zero. The rare cases occur when your floating point values are small whole numbers or fractions that are sums of powers of two 1/2, 1/4, 1/8, etc. In the rare case that you have a benign number with a benign fractional part, the error will be zero.]

Sure, they are different, because of how floating point division works.
>>> res = 0
>>> for x in xrange(5000): res += 0.1
...
>>> res == 5000 * 0.1
False
There's a good explanation in the python official tutorial.

Well if k divides n then definitely the first one is more precise :-) To be serious, if the division is floating point and n > 1 then the first one will be more precise anyway though they will probably give different results, as nosklo said.
BTW, in Python 2.6 the division is integer by default so you'll have very different results. 1/k will always give 0 unless k <= 1.

Floating point arithmetic has representation and roundoff errors. For the types of data floating point numbers are intended to represent, real numbers of reasonable size, these errors are generally acceptable.
If you want to calculate the quotient of two numbers, the right way is simply to say result = n / k (beware if these are both integers and you have not said from __future__ import division, this is not what you may expect). The second way is silly, error-prone, and ugly.
There is some discussion of floating point inexactness in the Python tutorial: http://docs.python.org/tutorial/floatingpoint.html

Even if we charitably assume a floating-point division, there's very definitely a difference in precision; the for loop is executed n - 1 times!
assert (n-1) / k != n / k
Also depends on what res is initialised to in the second case :-)

Certainly there is a difference if you use floating point numbers, unless the Python interpreter/compiler you are using is capable of optimizing away the loop (Maybe Jython or IronPython might be able to? C compilers are pretty good at this).
If you actually want these two approaches to be the same precision though, and you are using integers for your numerator and denominator, you can use the python fractions package
from fractions import Fraction
n,k = 999,1000
res = Fraction(0,1)
for i in range(0,n):
res += Fraction(1,k)
print float(res)