From the specs of IEEE754, a float coded on 64 bits has 11bits for the exponent and 52 bits for the mantissa. Hence, the smaller number that could be coded as a float should be 2**(-2**10). The wikipedia page which I believe is true, give a more exact value of 2**-1022 whose decimal value is approximately 2.2250738585072014e-308.
But, with Python, I can use as float numbers such as 2**-1052, etc. The actual limit on my computer is 2**-1074. From this page of the official documentation, Python usually conforms to IEEE754.
At the same time, the maximal value is 2**1023, which is the given value by the IEEE754 standard.
Why is it so?
Does anyone has an explanation?
What is the actual encoding of a float in Python?
Why does the range for the exponent, which is equal to 1074+1023+1, ie 2098, is not a power of 2?
First, the Python documentation does not specify that IEEE-754 is used. The choice of floating-point implementation is up to each Python implementation. IEEE-754 is very popular but not universal.
In the IEEE-754 basic 64-bit binary floating-point format (binary64), there is a one-bit sign field s, an eleven-bit exponent field e, and a 52-bit primary significand field f (for “fraction”).
The sign bit is 0 for positive and 1 for negative.
If e is all one bits (1111111111, or 2047 in decimal), the object represents an infinity (if f is zero) or a NaN (if f is not zero). (“NaN” stands for “Not a Number”).
If e is neither all zero bits nor all one bits, then it represents an exponent E = e − 1023, and the f field is used to form a number F that is 1.f (that is, the binary numeral “1.” followed by the 52 bits of f). (Equivalently, if we regard f as a binary numeral, we can say F = 1 + f • 2−52.) The number represented is (−1)s • 2E • F. These are called normal numbers.
If e is all zero bits, then it represents an exponent E = 1 − 1023, and the f field is used to form a number F that is 0.f (that is, the binary numeral “0.” followed by the 52 bits of f). (Equivalently, we can say F = 0 + f • 2−52.) Again, the number represented is (−1)s • 2E • F. These are called subnormal numbers.
Note that e = 1 and e = 0 represent the same exponent E, 1 − 1023 = −1022, but change the first bit of F from 1 to 0. F is the significand (the fraction part) of the floating-point number. (People sometimes refer to f as the significand1, but that is incorrect. It is only the field that provides most of the encoding of the mathematical significand. As we see above, the exponent field also contributes to forming the actual significand, F.)
The smallest positive number occurs when s is zero, e is zero, and f is 1 (0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001). Then E = −1022 and F = 0 + 1 • 2−52, so the number represented is (−1)0 • 2−1022 • 2−52 = 2−1074.
The largest finite number occurs when s is zero, e is 2046 (11111111110), and f is all ones (1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111). E = 2046 − 1023 = 1023. Note that f, interpreted as an integer, is 252−1, so F = 1 + *(252−1) • 2−52 = 1 + 1 − 2−52 = 2 - 2−52. So the value represented is (−1)0 • 21023 • (2 - 2−52) = 21024 − 2971.
Footnote
1 The significand is sometimes referred to as the “mantissa,” but that is an old term for the fraction portion of a logarithm. It is not entirely appropriate for floating-point numbers, as mantissas are logarithmic while significands are linear.
binary64 encodes most values with a biased exponent and a 53 bit significand.
With biased exponent as 1 or more**, the value is: 1.the_52_bit_encoded_"mantissa" * 2exponent - bias.
With biased exponent as 0, the value is: 0.the_52_bit_encoded_"mantissa" * 21 - bias.
OP is in the right idea for normal values. But there exist sub-normal values that have various amounts of leading zeros in 0.the_52_bit_encoded_"mantissa". The smallest non-zero is then 0.(51_zeros)1 * 21 - bias or 2-1074.
** When the biased exponent has a maximal value, the number is special: either an infinity or not-a-number.
Related
If I run the following code in python 3
1.0 + 1e-16
I get
1.0
as a result. Im wondering why? Assuming the numbers are stored as IEE754 double precision (64 bit) then I have the following representation of these numbers:
Number 1:
sign: 0
exponent: 01111111111 --> 1023 as decimal number
significand: 0...0
Number 1e-16:
sign: 0
exponent: 01111001101 --> 973 as decimal number
significand: 0010000000111010111110011110111001110101011000010110
In order to add both numbers I have to increase the exponent of the the small number by 50 to fit the exponent of the representation of number 1.0. If I do that the mantissa becomes:
0.0...0100
After adding both mantissas my result should be:
sign: 0
exponent: 01111111111
significand: 0000000000000000000000000000000000000000000000000100
and this is not euqal to 1.0. Can someone explain or give me a hint what I'm overlooking?
It seems your conversion of 1e-16 to binary is not correct: float.hex(1e-16) produces 0x1.cd2b297d889bcp-54, meaning the (unbiased) exponent is -54 in decimal, not -50. Then the leading 1 drops off the end of the significand and you end up with zero.
I am manipulating binary values with python. Here is my statement:
I have a float value, for example: 127.136.
First, I split this value in order to differenciate the integer and the fractional values, then I transform them into bytes:
velocity = 127.136
split_num = str(velocity).split('.')
int_part = int(split_num[0]) # = 127
decimal_part = int(split_num[2]) # = 136
int_part_b = "{0:b}".format(int_part)
decimal_part_b = "{0:b}".format(decimal_part)
I have to send the final value as a 2 bits word.
The total space is 32 bits. 127 is the integer part and 136 is the fractional part. The integer part takes 16 bits and the decimal part takes 16 bits.
So I want to initialize 3 binary word of 32 bits like that:
final_value = 00000000000000000000000000000000
integer_value = 00000000000000000000000000000000
fractional_value = 00000000000000000000000000000000
But then how can I add the value of int_part_b from the 16th bits of integer_value so for example if int_part_b = 10011 I would like to have: integer_value = 00000000000100110000000000000000 and for the fractional one if decimal_part_b = 11110 I would like to have: fractional_value = 00000000000000000000000000011110
And at the end, I sum this two bits values to obtain:
final_value = 00000000000100110000000000011110
Does anyone has any idea about how to do that?
For this encoding to make any sense, the value in the fractional part should be a faction of 2^16 (or any denominator <= 65536 you pick as a convention).
You cannot obtain the fractional part of the float value by string manipulation because that makes you lose the base10 denominator. (e.g. 0.36 and 0.036 are two different fractions but your code will produce 36 in both cases).
Note that there are a lot of different ways to encode numbers in binary (BCD for example) so you probably should be specific on that in your question
Binary encoding:
Here is one way to do it with a denominator of 2^16:
velocity = 127.136
denom = 2**16
intPart,fracPart = divmod(round(velocity*denom),denom)
encoded = (intPart<<16) | fracPart
print(f"{encoded:032b}")
# 00000000011111110010001011010001
Or, more straightforwardly (but only if denom is 2**16):
encoded = round(velocity*2**16)
print(f"{encoded:032b}")
# 00000000011111110010001011010001
Here, the value in the right part (fractional) is 8913, representing a fraction of 8913/65536 (0.1360015...) which is as close to 0.136 as a binary fraction over 16 bits can get.
Decoding the value:
int(encoded[:16],2) + int(encoded[16:],2)/denom
# 127.13600158691406
Decimal encoding:
If you want to represent the fractional part in base 10, you'll have to decide on a base10 denominator that fits within 16 bits (e.g. 10000) and compute your fractional part accordingly.
velocity = 127.136
denom = 10000
intPart,fracPart = divmod(round(velocity*denom),denom)
encoded = (intPart<<16) | fracPart
print(f"{encoded:032b}")
# 00000000011111110000010101010000
Now the fractional part is 1360 representing a fraction of 1360/10000 (0.136) which doesn't lose precision compared to the base 10 original (as long as there are no more than 4 decimal digits)
Decoding the value:
int(encoded[:16],2) + int(encoded[16:],2)/denom
# 127.136
You don't really need 3 differend words, just use the binary shift operator and sum the integer and decimal parts like this:
final_value = (int_part << 16) + decimal_part
print("{:032b}".format(final_value))
So here you shift your integer part 16 bits to the left and then you add your decimal part to it so they will occupy there respective 16-bit words of the 32-bit word.
This is more a curiosity than a technical problem.
I'm trying to better understand how floating point numbers are handled in Python. In particular, I'm curious about the number returned by sys.float_info.epsilon = 2.220446049250313e-16.
I can see, looking up on the documentation on Double-precision floating-point, that this number can also be written as 1/pow(2, 52). So far, so good.
I decided to write a small python script (see below. Disclaimer: this code is ugly and can burn your eyes) which start from eps = 0.1 and makes the comparison 1.0 == 1.0 + eps. If False, it means eps is big enough to make a difference. Then I try to find a smaller number by subtracting 1 from the last digit and adding the digit 1 to the right of the last and looking for False again by incrementing the last digit.
I am pretty confident that the code is ok because at certain point (32 decimal places) I get eps = 0.00000000000000011102230246251567 = 1.1102230246251567e-16 which is very close to 1/pow(2, 53) = 1.1102230246251565e-16 (last digit differs by 2).
I thought the code would no produce sensible numbers after that. However, the script kept working, always zeroing in a more accurate decimal number until it reached 107 decimal places. Beyond that, the code did not find a False to the test. I got very intrigued with that result and could not wrap my head around it.
Does this 107 decimal places float number have any meaning? If positive, what is it particular about it?
If not, what is python doing past the 32 decimal places eps? Surely there is some algorithm python is cranking to get to the 107 long float.
The script.
total = 520 # hard-coded after try-and-error max number of iterations.
dig = [1]
n = 1
for t in range(total):
eps = '0.'+''.join(str(x) for x in dig)
if(1.0 == 1.0 + float(eps)):
if dig[-1] == 9:
print(eps, n)
n += 1
dig.append(1)
else:
dig[-1] += 1
else:
print(eps, n)
n += 1
dig[-1] -= 1
dig.append(1)
The output (part of it). Values are the eps and the number of decimal places
0.1 1
0.01 2
(...)
0.000000000000001 15
0.0000000000000002 16
0.00000000000000012 17
0.000000000000000112 18
0.0000000000000001111 19
0.00000000000000011103 20
(...)
0.0000000000000001110223024625157 31
0.00000000000000011102230246251567 32
0.000000000000000111022302462515667 33
(...)
0.000000000000000111022302462515666368314810887391490808258832543534838643850548578484449535608291625976563 105
0.0000000000000001110223024625156663683148108873914908082588325435348386438505485784844495356082916259765626 106
0.00000000000000011102230246251566636831481088739149080825883254353483864385054857848444953560829162597656251 107
I ran this code in Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32.
Your test involves a double rounding and is finding the number 2−53+2−105.
Many Python implementations use the IEEE-754 binary64 format. (This is not required by the Python documentation.) In this format, the significand (fraction portion) of a floating-point number has 53 bits. (52 are encoded in a primary significand field. 1 is encoded via the exponent field.) For numbers in the interval [1, 2), the significand is scaled (by the exponent portion of the floating-point representation) so that its leading bit corresponds to a value of 1 (20). This means is trailing bit corresponds to a value of 2−52.
Thus, the difference between 1 and the next number representable in this format is 2−52—that is the smallest change that can be made in the number, by increasing the low bit.
Now, suppose x contains 1. If we add 2−52 to it, we will of course get 1+2−52, since that result is representable. What happens if we add something slightly smaller, say ¾•2−52? In this case, the real-number result, 1+¾•2−52, is not representable. It must be rounded to a representable number. The common default rounding method is to round to the nearest representable number. In this case, that is 1+2−52.
Thus, adding to 1 some numbers smaller than 2−52 still produces 1+2−52. What is the smallest number we can add to 1 and get this result?
In case of ties, where the real-number result is exactly halfway between two representable numbers, the common default rounding method uses the one with the even low bit. So, with a choice between 1 (trailing bit 0) and 1+2−52 (trailing bit 1), it chooses 1. That means if we add ½•2−52 to 1, it will produce 1.
If we add any number greater than ½•2−52 to 1, there will be no tie; the real-number result will be nearer to 1+2−52, and that will be the result.
The next question is what is the smallest number greater than ½•2−52 (2−53) that we can add to 1? If the number has to be in the IEEE-754 binary64 format, it is limited by its significand. With the leading bit scaled to represent 2−53, the trailing bit represents 2−53−52 = 2−105.
Therefore, 2−53+2−105 is the smallest binary64 value we can add to 1 to get 1+2−52.
As your program tests values, it works with a decimal numeral. That decimal numeral is converted to the floating-point format and then added to 1. So it is finding the smallest number in the floating-point format that produces a sum greater than 1, and that is the number described above, 2−53+2−105. Its value in decimal is 1.110223024625156663683148108873914908082588325435348386438505485784844495356082916259765625•10−16.
Why can Python represent negative zero in float, but cannot represent negative zero in int?
More specifically:
a = 0.0
print(a)
# 0.0
b = -a
print(b)
# -0.0
BUT:
a = 0
print(a)
# 0
b = -a
print(b)
# 0
(I am aware of the discussion here negative zero in python on negative float zero, but the ints are not really discussed there).
Historically, there were integer formats that could represent both −0 and +0. Both sign-and-magnitude and one’s complement can represent −0 and +0. These proved to be less useful than two’s complement, which won favor and is ubiquitous today.
Two’s complement has some numerical properties that make it a little nicer to implement in hardware, and having two zeros caused some nuisance for programmers. (I heard of bugs such as an account balance being −0 instead of +0 resulting in a person being sent a bill when they should not have been.)
Floating-point uses sign-and-magnitude, so it can represent both −0 and +0. Due to the nature of floating-point, the arithmetic properties of two’s complement would not aid a floating-point implementation as much, and having two zeros allows a programmer to use a little extra information in some circumstances.
So the choices for integer and floating-point formats are motivated by utility, not mathematical necessity.
A Look At Integer Arithmetic
Let’s consider implementing some integer arithmetic in computer hardware using four bits for study. Essentially the first thing we would do is implement unsigned binary arithmetic, so we design some logic gates to make adders and other arithmetic units. So the inputs 0101 and 0011 to the adder produce output 1000.
Next, we want to handle negative numbers. In writing, we handle negative numbers by putting a sign on front, so our first thought might be to do the same thing with bits: Use a bit in front to indicate negative. Now we have a sign-and-magnitude representation. 0001 represents +1, and 1001 represents −1. 0010 represents +2, and 1010 represents −2. 0111 represents +7 , and 1111 represents −7. And, of course, 0000 represents +0, and 1000 represents −0. That is an idea, and then we must implement it. We have already got an adder, and, if we feed it 0010 (2) and 0011 (3), it correctly outputs 0101 (5). But, if we feed it 0011 (3) and 1001 (−1), it outputs 1100 (−4). So we have to modify it. Well, that is not too bad, we have a subtraction unit for unsigned binary, so we can look at the first bit, and, if we are adding a negative number, we subtract instead of adding. That works for some operations; for 0011 and 1001, observing the leading 1 on the second operand and feeding 011 and 001 to the subtraction unit would produce 010 (2), which is correct. But, if we have 0010 and 1011, feeding 010 and 011 to the subtraction unit might produce some error indication (it was originally designed for unsigned binary) or it might “wrap” and produce 111 (because such wrapping, along with a “borrow out” bit in the output, makes the subtraction unit work as part of a design for subtracting wider numbers). Either way, that is wrong for our signed numbers; we want the output of 0010 (2) plus 1011 (−3) to be 1001 (−1). So we have to design new arithmetic units that handle this. Perhaps, when adding numbers of mixed signs, they figure out which one is larger in magnitude, subtract the smaller from the larger, and then apply the sign bit of the larger. In any case, we have a fair amount of work to do just to design the addition and subtraction units.
Another suggestion is, to make a number negative, invert every bit. This is called one’s complement. It is easy to understand and fits the notion of negation—just negate everything. Let’s consider how it affects our arithmetic units. For the combinations of +3 or −3 with +2 or −2, we would want these results: 0011 (3) + 0010 (2) = 0101 (5), 0011 (3) + 1101 (−2) = 0001 (1), 1100 (−3) + 0010 (2) = 1110 (−1), and 1100 (−3) + 1101 (−2) = 1010 (−5). Upon examination, there is a simple way to adapt our binary adder to make this work: Do the addition on all four bits as if they were unsigned binary, and, if there is a carry out of the leading bit, add it back to the low bit. In unsigned binary 0011 + 0010 = 0101 with no carry, so the final output is 0101. 0011 + 1101 = 0000 with a carry, so the final result is 0001. 1100 + 0010 = 1110 with no carry, so the final result is 1110. 1100 + 1101 = 1001 with a carry, so the final result is 1010.
This is nice; our one’s complement adder is simpler than the sign-and-magnitude adder. It does not need to compare magnitudes and does not need to do a subtraction to handle negative numbers. We can make it cheaper and make more profit.
Then somebody comes up with the idea of two’s complement. Instead of inverting every bit, we will conceptually subtract the number from 2n, where n is the number of bits. So 10000 − 0001 = 1111 represents −1, and 1110 is −2, 1101 is −3, and so on. What does this do to our adder?
In unsigned binary, 0010 (2) + 1101 (13) = 1111 (15). In two’s complement, 0010 (2) + 1101 (−3) = 1111 (−1). The bits are the same! This actually works for all two’s complement numbers; adding the bit patterns for unsigned numbers produces the same results we want for adding two’s complement numbers. We can use the exact same logic gates for unsigned binary and two’s complement. That is brilliant, give that employee a raise. That is what modern hardware does; the same arithmetic units are used for adding or subtracting two’s complement numbers as are used for adding or subtracting unsigned numbers.
This is a large part of why two’s complement won out for representing negative integers. It results in simpler, easier, cheaper, faster, and more efficient computers.
(There is a difference between unsigned addition and two’s complement addition: How overflow is detected. In unsigned addition, an overflow occurs if there is a carry out of the high bit. In two’s complement addition, an overflow occurs if there is a carry out of the highest of the magnitude bits, hence a carry into the sign. Adder units commonly handle this by reporting both indications, in one form or another. That information, if desired, is tested in later instructions; it does not affect the addition itself.)
Why do some numbers lose accuracy when stored as floating point numbers?
For example, the decimal number 9.2 can be expressed exactly as a ratio of two decimal integers (92/10), both of which can be expressed exactly in binary (0b1011100/0b1010). However, the same ratio stored as a floating point number is never exactly equal to 9.2:
32-bit "single precision" float: 9.19999980926513671875
64-bit "double precision" float: 9.199999999999999289457264239899814128875732421875
How can such an apparently simple number be "too big" to express in 64 bits of memory?
In most programming languages, floating point numbers are represented a lot like scientific notation: with an exponent and a mantissa (also called the significand). A very simple number, say 9.2, is actually this fraction:
5179139571476070 * 2 -49
Where the exponent is -49 and the mantissa is 5179139571476070. The reason it is impossible to represent some decimal numbers this way is that both the exponent and the mantissa must be integers. In other words, all floats must be an integer multiplied by an integer power of 2.
9.2 may be simply 92/10, but 10 cannot be expressed as 2n if n is limited to integer values.
Seeing the Data
First, a few functions to see the components that make a 32- and 64-bit float. Gloss over these if you only care about the output (example in Python):
def float_to_bin_parts(number, bits=64):
if bits == 32: # single precision
int_pack = 'I'
float_pack = 'f'
exponent_bits = 8
mantissa_bits = 23
exponent_bias = 127
elif bits == 64: # double precision. all python floats are this
int_pack = 'Q'
float_pack = 'd'
exponent_bits = 11
mantissa_bits = 52
exponent_bias = 1023
else:
raise ValueError, 'bits argument must be 32 or 64'
bin_iter = iter(bin(struct.unpack(int_pack, struct.pack(float_pack, number))[0])[2:].rjust(bits, '0'))
return [''.join(islice(bin_iter, x)) for x in (1, exponent_bits, mantissa_bits)]
There's a lot of complexity behind that function, and it'd be quite the tangent to explain, but if you're interested, the important resource for our purposes is the struct module.
Python's float is a 64-bit, double-precision number. In other languages such as C, C++, Java and C#, double-precision has a separate type double, which is often implemented as 64 bits.
When we call that function with our example, 9.2, here's what we get:
>>> float_to_bin_parts(9.2)
['0', '10000000010', '0010011001100110011001100110011001100110011001100110']
Interpreting the Data
You'll see I've split the return value into three components. These components are:
Sign
Exponent
Mantissa (also called Significand, or Fraction)
Sign
The sign is stored in the first component as a single bit. It's easy to explain: 0 means the float is a positive number; 1 means it's negative. Because 9.2 is positive, our sign value is 0.
Exponent
The exponent is stored in the middle component as 11 bits. In our case, 0b10000000010. In decimal, that represents the value 1026. A quirk of this component is that you must subtract a number equal to 2(# of bits) - 1 - 1 to get the true exponent; in our case, that means subtracting 0b1111111111 (decimal number 1023) to get the true exponent, 0b00000000011 (decimal number 3).
Mantissa
The mantissa is stored in the third component as 52 bits. However, there's a quirk to this component as well. To understand this quirk, consider a number in scientific notation, like this:
6.0221413x1023
The mantissa would be the 6.0221413. Recall that the mantissa in scientific notation always begins with a single non-zero digit. The same holds true for binary, except that binary only has two digits: 0 and 1. So the binary mantissa always starts with 1! When a float is stored, the 1 at the front of the binary mantissa is omitted to save space; we have to place it back at the front of our third element to get the true mantissa:
1.0010011001100110011001100110011001100110011001100110
This involves more than just a simple addition, because the bits stored in our third component actually represent the fractional part of the mantissa, to the right of the radix point.
When dealing with decimal numbers, we "move the decimal point" by multiplying or dividing by powers of 10. In binary, we can do the same thing by multiplying or dividing by powers of 2. Since our third element has 52 bits, we divide it by 252 to move it 52 places to the right:
0.0010011001100110011001100110011001100110011001100110
In decimal notation, that's the same as dividing 675539944105574 by 4503599627370496 to get 0.1499999999999999. (This is one example of a ratio that can be expressed exactly in binary, but only approximately in decimal; for more detail, see: 675539944105574 / 4503599627370496.)
Now that we've transformed the third component into a fractional number, adding 1 gives the true mantissa.
Recapping the Components
Sign (first component): 0 for positive, 1 for negative
Exponent (middle component): Subtract 2(# of bits) - 1 - 1 to get the true exponent
Mantissa (last component): Divide by 2(# of bits) and add 1 to get the true mantissa
Calculating the Number
Putting all three parts together, we're given this binary number:
1.0010011001100110011001100110011001100110011001100110 x 1011
Which we can then convert from binary to decimal:
1.1499999999999999 x 23 (inexact!)
And multiply to reveal the final representation of the number we started with (9.2) after being stored as a floating point value:
9.1999999999999993
Representing as a Fraction
9.2
Now that we've built the number, it's possible to reconstruct it into a simple fraction:
1.0010011001100110011001100110011001100110011001100110 x 1011
Shift mantissa to a whole number:
10010011001100110011001100110011001100110011001100110 x 1011-110100
Convert to decimal:
5179139571476070 x 23-52
Subtract the exponent:
5179139571476070 x 2-49
Turn negative exponent into division:
5179139571476070 / 249
Multiply exponent:
5179139571476070 / 562949953421312
Which equals:
9.1999999999999993
9.5
>>> float_to_bin_parts(9.5)
['0', '10000000010', '0011000000000000000000000000000000000000000000000000']
Already you can see the mantissa is only 4 digits followed by a whole lot of zeroes. But let's go through the paces.
Assemble the binary scientific notation:
1.0011 x 1011
Shift the decimal point:
10011 x 1011-100
Subtract the exponent:
10011 x 10-1
Binary to decimal:
19 x 2-1
Negative exponent to division:
19 / 21
Multiply exponent:
19 / 2
Equals:
9.5
Further reading
The Floating-Point Guide: What Every Programmer Should Know About Floating-Point Arithmetic, or, Why don’t my numbers add up? (floating-point-gui.de)
What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg 1991)
IEEE Double-precision floating-point format (Wikipedia)
Floating Point Arithmetic: Issues and Limitations (docs.python.org)
Floating Point Binary
This isn't a full answer (mhlester already covered a lot of good ground I won't duplicate), but I would like to stress how much the representation of a number depends on the base you are working in.
Consider the fraction 2/3
In good-ol' base 10, we typically write it out as something like
0.666...
0.666
0.667
When we look at those representations, we tend to associate each of them with the fraction 2/3, even though only the first representation is mathematically equal to the fraction. The second and third representations/approximations have an error on the order of 0.001, which is actually much worse than the error between 9.2 and 9.1999999999999993. In fact, the second representation isn't even rounded correctly! Nevertheless, we don't have a problem with 0.666 as an approximation of the number 2/3, so we shouldn't really have a problem with how 9.2 is approximated in most programs. (Yes, in some programs it matters.)
Number bases
So here's where number bases are crucial. If we were trying to represent 2/3 in base 3, then
(2/3)10 = 0.23
In other words, we have an exact, finite representation for the same number by switching bases! The take-away is that even though you can convert any number to any base, all rational numbers have exact finite representations in some bases but not in others.
To drive this point home, let's look at 1/2. It might surprise you that even though this perfectly simple number has an exact representation in base 10 and 2, it requires a repeating representation in base 3.
(1/2)10 = 0.510 = 0.12 = 0.1111...3
Why are floating point numbers inaccurate?
Because often-times, they are approximating rationals that cannot be represented finitely in base 2 (the digits repeat), and in general they are approximating real (possibly irrational) numbers which may not be representable in finitely many digits in any base.
While all of the other answers are good there is still one thing missing:
It is impossible to represent irrational numbers (e.g. π, sqrt(2), log(3), etc.) precisely!
And that actually is why they are called irrational. No amount of bit storage in the world would be enough to hold even one of them. Only symbolic arithmetic is able to preserve their precision.
Although if you would limit your math needs to rational numbers only the problem of precision becomes manageable. You would need to store a pair of (possibly very big) integers a and b to hold the number represented by the fraction a/b. All your arithmetic would have to be done on fractions just like in highschool math (e.g. a/b * c/d = ac/bd).
But of course you would still run into the same kind of trouble when pi, sqrt, log, sin, etc. are involved.
TL;DR
For hardware accelerated arithmetic only a limited amount of rational numbers can be represented. Every not-representable number is approximated. Some numbers (i.e. irrational) can never be represented no matter the system.
There are infinitely many real numbers (so many that you can't enumerate them), and there are infinitely many rational numbers (it is possible to enumerate them).
The floating-point representation is a finite one (like anything in a computer) so unavoidably many many many numbers are impossible to represent. In particular, 64 bits only allow you to distinguish among only 18,446,744,073,709,551,616 different values (which is nothing compared to infinity). With the standard convention, 9.2 is not one of them. Those that can are of the form m.2^e for some integers m and e.
You might come up with a different numeration system, 10 based for instance, where 9.2 would have an exact representation. But other numbers, say 1/3, would still be impossible to represent.
Also note that double-precision floating-points numbers are extremely accurate. They can represent any number in a very wide range with as much as 15 exact digits. For daily life computations, 4 or 5 digits are more than enough. You will never really need those 15, unless you want to count every millisecond of your lifetime.
Why can we not represent 9.2 in binary floating point?
Floating point numbers are (simplifying slightly) a positional numbering system with a restricted number of digits and a movable radix point.
A fraction can only be expressed exactly using a finite number of digits in a positional numbering system if the prime factors of the denominator (when the fraction is expressed in it's lowest terms) are factors of the base.
The prime factors of 10 are 5 and 2, so in base 10 we can represent any fraction of the form a/(2b5c).
On the other hand the only prime factor of 2 is 2, so in base 2 we can only represent fractions of the form a/(2b)
Why do computers use this representation?
Because it's a simple format to work with and it is sufficiently accurate for most purposes. Basically the same reason scientists use "scientific notation" and round their results to a reasonable number of digits at each step.
It would certainly be possible to define a fraction format, with (for example) a 32-bit numerator and a 32-bit denominator. It would be able to represent numbers that IEEE double precision floating point could not, but equally there would be many numbers that can be represented in double precision floating point that could not be represented in such a fixed-size fraction format.
However the big problem is that such a format is a pain to do calculations on. For two reasons.
If you want to have exactly one representation of each number then after each calculation you need to reduce the fraction to it's lowest terms. That means that for every operation you basically need to do a greatest common divisor calculation.
If after your calculation you end up with an unrepresentable result because the numerator or denominator you need to find the closest representable result. This is non-trivil.
Some Languages do offer fraction types, but usually they do it in combination with arbitary precision, this avoids needing to worry about approximating fractions but it creates it's own problem, when a number passes through a large number of calculation steps the size of the denominator and hence the storage needed for the fraction can explode.
Some languages also offer decimal floating point types, these are mainly used in scenarios where it is imporant that the results the computer gets match pre-existing rounding rules that were written with humans in mind (chiefly financial calculations). These are slightly more difficult to work with than binary floating point, but the biggest problem is that most computers don't offer hardware support for them.