Floating and Fixed Point

Floating and Fixed Point are some of those things that I did not get a good explanation when I was being taught about computers. Which is a problem, because they are very integral to how modern computers operate.

Except it hasn't been like that forever. Because when I was a little egg, computers treated floating point processors (FPUs) kinda like they treated GPUs in the late 1990s/early 2000s. It was something extra that you could add to the system, not a core part of it.

Which makes math in old computers something wild.

Let's Begin With Natural Numbers

In general computers are good at representing integers. That's a lie.

Computers can have registers that can be used to represent a range of integer numbers. Because of historical reasons they tend to come in sizes that are multiples of 256, although this is kind of not they way things had to be. Many old computers operated on 127 number ranges.

How do these numbers fit into registers? In binary.

Two possible values for each one of eight bits gives use 2⁸=256 values in an 8-bit register. Of course modern computers tend to use much bigger registers, generally 64-bit (yes, that's what the bittage of computers means).

What About Negative Numbers?

If you want to represent the sign in a number you need to put it somewhere and a register only has those eight (or more) bits to work with, meaning that some of those positive numbers will become non-representable.

There have been a bunch of ways in which negative numbers have been represented in registers but the standard has become two's complement:

Might look a bit weird but it has a huge advantage: with this system, the same circuit that adds or subtracts a number from a positive one works to add or subtract a number from a negative one.

The thing is that we're still representing only 256 possible numbers but this time we have chosen to use the range from −128 to 127. That's an unavoidable fact: we can only represent as many numbers as possible 0 and 1 combinations in the register.

Fixed Point, or Naive Decimal Parts

This is the simplest way of representing a range of… well, we're not even calling them real numbers at this point, but numbers with a decimal part. You just take the integer table from the last section and divide the numbers of the right by a power of two. That means that the last N bits in the byte are representing the decimal part:

So, by assigning some of the bits to the decimal part we can represent a finer collection of numbers, but at the cost of having less total range to work with. It also increases a bit the cost of some operations, because adding or subtracting together fixed-point numbers uses the same circuit as adding or subtracting together integers but multiplication and division requires you to do some bit shifting of the result if you want it to have the decimal point in the same place as the inputs.

Floating Point, or Significant Digits

In engineering there's this concept of significant digits, which basically could be summed as "in every operation and scale, there are parts of the input that are relevant and parts that can be considered noise"

In general terms, and for decimal numbers, many people consider six significant digits to be good enough for most purposes, which means that if your number is one-and-a-bit you need to get at least five decimal digits after the point, but if your number is a-hundred-and-forty-two-and-a-you only need three decimal digits after the point.

Which is why there's the exponential notation for numbers, where we just write those significant digits, and then add en "exponent" which just means "move the period by N spaces in one direction":

The thing is that this format can be used to actually represent binary numbers which are a lot more useful than what fixed point does:

This particular distribution of a four-bit base and four-bit exponent gives us as much maximum range as the plain integers, but also allows us to represents numbers as small as the most precise fixed point.

But Wait, There Is a Problem…

Yes. More than one, in fact.

For starters, the total amount of numbers that you can represent are smaller because some numbers can be represented in more than one way:

This is unavoidable but it is kind of compensated by the actual numbers you can represent being generally more useful, since in a practical context many numbers are not going to be representable.

Then we have the circuitry that you need to make operations with these numbers. It it not trivial. It is in fact much more complex than the one required to do math with integers or fixed-point numbers. Which is the reason that once upon a time FPUs were treated like we treat GPUs now.

And math operations on floating point numbers are prone to rounding errors that can sound very weird to humans. There was an issue once upon a time where a pretty popular family of floating point algorithms resulted in people trying to add 2+2 and getting 3.9 as a result.

But it's been a long time since then, and these days floating point is way beyond the very simple base+exponent format described here. Have a look at the IEEE 754 standard, if you want to know more.

So How Do These Representations Compare?

Here, have a table: