What we don't know about floating-point numbers
Marcin Polkowski
Translated from Polish. Originally published on morony.pl on 12 Oct 2016.
For the last 4 years I've been teaching lab classes at the Faculty of Physics for several programming courses in Python and C++. A few times I've been asked about floating-point numbers. In class we cover the binary representation of integers, and that's when the question about real numbers comes up. Until now I've answered very briefly that "it's complicated" – very clever, but not without its flaws. With integers, things are simple. In a given number of bits we can store a given range of integers. The problem with floating-point numbers is that there are infinitely many real numbers in any interval, and we only have 32 or 64 bits. So how does it actually work? You can of course read everything about the format itself on Wikipedia – I want to do an experiment. First up: float numbers, i.e. 32-bit ones (1 bit for the sign, 8 bits for the exponent and 23 for the mantissa). 32 bits gives some 4,294,967,296 combinations – few enough that I can check them all. As a test, I check how many numbers fall into 4 intervals: 0-1, 1-2, 1000-1001 and 1000000-1000001:
int main()
{
float f;
uint i;
std::vector<int> range = {0, 1, 1000, 1000000};
std::vector<int> count = {0, 0, 0, 0};
for(i = 0; i<pow(2,32)-1; i++)
{
memcpy(&f, &i, sizeof(f));
for(int a = 0; a < range.size(); a++)
{
if (f>=range[a] && f < range[a]+1)
count[a]++;
}
}
for(int a = 0; a < range.size(); a++)
{
std::cout << range[a] << "-" << range[a]+1 << ":\t" << count[a] << std::endl;
}
}
We get the result in just under 2 minutes. The counts are very interesting:
A few relationships are immediately visible. In the range from 1 to 2 there are exactly 2^23 numbers - as if these were all the "mantissas" for a single exponent. In the range from 0 to 1 there are exactly 2^23*(2^7-1)-1 - that is, all the mantissas for half of the exponents (minus one). The larger the number, the lower the precision - we have 2^31 at our disposal, and we need to be able to represent both very small numbers and gigantic ones.
With single-precision numbers things were simple - I managed to check all the combinations in under 2 minutes. With double precision we have 64 bits, so checking all of them would take 16,000 years. Not worth the time. What we can do instead is play with code like this:
int main()
{
union
{
uint64_t input;
uint64_t output;
} data;
uint64_t x = 5;
data.input = x;
std::bitset<sizeof(uint64_t) * CHAR_BIT> bits(data.output);
for(uint64_t a = 0; a<=2048; a++)
{
for(uint64_t b = 0; b<=pow(2,52)-1; b+=pow(2,52)-1)
{
x = b+(a<<52);
data.input = x;
bits = data.output;
double f;
memcpy(&f, &x, sizeof(f));
std::cout << std::setw(4)<< a<< '\t'<< std::setw(16)<< b << '\t'<< std::setw(12) << f << '\t' << bits << std::endl;
}
std::cout << std::endl;
}
}
The code above prints the range of numbers for each of the 2^11 exponents. The mantissa has 52 bits (~4.5*10^15), so from 1 to 2 we have exactly 2^52 numbers, the same number from 2 to 4, from 4 to 8, and so on:

In the example above, the first column is the exponent, the second is the mantissa, the third is the value of the number, and the fourth is its binary representation.
I don't want to go into the details of the format itself here. What I care about is showing the peculiarities of storing numbers this way. You can modify and run the code above, and you'll learn more about floating-point numbers. On top of that, the examples above should nicely illustrate the colossal difference between single and double precision.
We have to remember that when operating on very large numbers, where floating-point precision is low, we can introduce numerical errors - especially when we combine huge and tiny numbers.
If you learned something new, give it a thumbs up!