1. The Decimal to Binary Conversion Trap

1.1 Repeating Binary Fractions

In base 10, a fraction can only be represented with a terminating decimal if its denominator's prime factors divide the base ($10 = 2 \times 5$). Consequently, fractions like $1/2$, $1/5$, or $3/10$ terminate exactly, while fractions like $1/3$ repeat infinitely ($0.3333\dots$).

In binary (base 2), a fraction only terminates if its denominator is a power of 2. While $0.5$ (which is $1/2^1$) and $0.25$ ($1/2^2$) terminate exactly in binary, a simple decimal fraction like $0.1$ ($1/10$) does not. The binary fraction for $0.1$ repeats infinitely:

Because a computer must store this number in a finite number of bits, it must truncate this infinite sequence. This truncation introduces a small rounding error, which is the root cause of the floating-point glitch.

1.2 Terminating Decimal Fractions vs Repeating Binary Fractions

Decimal fractions represent divisions of powers of 10. Binary systems store fractions using sums of negative powers of 2 ($2^{-1}, 2^{-2}, 2^{-3}, \dots$). Truncating these sums results in approximation errors.

Common Misconception — Using double precision resolves all rounding issues: A common developer misconception is that switching from single-precision (float) to double-precision (double) eliminated rounding errors. While double precision increases the number of mantissa bits from 23 to 52, it merely delays the error: $0.1$ is still a repeating fraction in binary, and rounding errors still accumulate in long loops.

2. The IEEE 754 Floating-Point Standard

2.1 The Three-Segment Bit Layout

To represent wide ranges of numbers, the IEEE 754 standard uses scientific notation in binary, representing a real number using three distinct bit segments:

• **Sign ($s$)**: A single bit indicating positive (0) or negative (1).
• **Exponent ($e$)**: A set of bits indicating the scale of the number. The exponent is biased to allow representing both large and small scales without storing sign bits.
• **Mantissa ($m$)**: A set of bits representing the significant digits. The mantissa is normalized so that the first digit is always 1 (e.g. $1.xxxx_2$), allowing us to omit the leading 1 from storage.

Mermaid Diagram: The binary layouts of sign, exponent, and mantissa fields in IEEE 754 floating-point standards.

3. Mathematical Formulation of Single and Double Precision

3.1 Binary Scientific Notation

For any normal floating-point number, its value is computed using the following equation:

Where $s$ is the sign bit, $m_i$ are the fractional mantissa bits, $e$ is the integer exponent value, and $p$ is the precision (24 for single-precision, 53 for double-precision). The exponent bias is $127$ for single-precision and $1023$ for double-precision, translating unsigned integers to negative scales.

4. Precision Limits and Round-off Errors

4.1 Big vs Small Addition Cascades

Because the mantissa has a fixed number of bits, adding a very small number to a very large number can result in the small number being ignored. For double-precision numbers, the precision limit is approximately 15-17 decimal digits.

If you calculate $10^{16} + 1$, the exponent difference requires shifting the mantissa of $1$ to the right by 53 bits to align exponents. Since the double-precision mantissa only stores 52 bits, the $1$ is shifted out of range, yielding $10^{16}$ as the result. This causes coordinate drift in physics engines and financial systems.

4.2 Exponent Alignment Alignment and Loss of Significance Mathematics

To add two floating-point numbers, the processor must first rewrite them so they share a common exponent. Let $x_{\text{big}} = 1.0 \times 2^{53}$ and $x_{\text{small}} = 1.0 \times 2^0$. In double-precision representation, $1.0 \times 2^{53}$ matches the maximum limit where the gap between consecutive representable numbers (ULP) is exactly $1.0$:

To add $x_{\text{small}}$ to $x_{\text{big}}$, the ALU aligns exponents by shifting $x_{\text{small}}$ to the right by 53 bits. Let $m_{\text{small}}$ be the mantissa of the smaller number. The shifted mantissa becomes:

Since the double-precision mantissa only stores 52 bits after the implicit leading 1, the shifted bits fall entirely beyond the storage range. Under round-to-nearest-even rules, the fractional term is truncated to zero:

This leads to **Loss of Significance**. If you sum a large list containing one large number and thousands of tiny numbers, the tiny numbers will be wiped out if added sequentially. Developers resolve this using **Kahan Summation**, which maintains a separate running compensation variable $c$ to accumulate the lost bits across steps:

By subtracting the estimated error value from the next input, the Kahan algorithm preserves small details, keeping the cumulative error at $O(1)$ instead of $O(N)$ growth. Below is a comparison table of precision characteristics:

Pitfall — Sorting arrays before parallel sums: While sorting arrays from smallest to largest minimizes absorptive addition errors during sequential loops, it fails during parallel reduction (e.g. GPU tree reductions) where summation order is non-linear. Use compensated sum blocks in parallel pipelines.

5. Special Values: Infinity, NaN, and Subnormal Numbers

5.1 Reserved Exponent Bit Patterns

The IEEE 754 standard reserves specific exponent bit patterns to represent special mathematical states:

• **Infinity ($\pm\infty$)**: Set when the exponent bits are all 1s and the mantissa is all 0s. Occurs during division by zero.
• **Not a Number (NaN)**: Set when the exponent bits are all 1s and the mantissa is non-zero. Occurs during undefined operations like $0/0$ or $\sqrt{-1}$.
• **Subnormal Numbers**: Set when the exponent bits are all 0s and the mantissa is non-zero. These numbers drop the implicit leading 1, allowing representation of tiny numbers close to zero at the expense of precision.

6. Floating-Point Comparison Guidelines

6.1 The Danger of Strict Equality

Because of rounding errors, comparing floating-point numbers using strict equality `==` is dangerous. Even if two calculations should be mathematically equal, small differences in operations can cause them to differ in their final bits.

Instead of strict equality, developers use **Epsilon Comparisons**. We check if the absolute difference between two numbers is less than a small threshold value $\epsilon$ (epsilon):

The choice of $\epsilon$ depends on the scale of the numbers. For large numbers, absolute epsilon fails, and relative epsilon scaling is required.

6.2 Relative Epsilon Formulations and Machine Precision (ULPs)

An absolute epsilon threshold (like $\epsilon = 0.00001$) works well for numbers near $1.0$. However, if we compare large numbers (like $1,000,000.1$ and $1,000,000.2$), their absolute difference ($0.1$) is much larger than $\epsilon$, causing the check to fail even though they are as close as single-precision floats allow. Conversely, for tiny numbers (like $10^{-20}$), an absolute epsilon is too large, treating unrelated values as equal.

To solve this, we scale the threshold relative to the magnitude of the values being compared. The standard **Relative Epsilon** comparison equation is defined as:

This ensures that the error scale adapts to the size of the numbers. In systems programming, we often measure differences in terms of **Unit in the Last Place (ULP)**, which represents the spacing between two consecutive representable floating-point numbers at a given magnitude. The distance between $1.0$ and the next larger float is called the **Machine Epsilon** ($\epsilon_{\text{mach}}$):

Comparing numbers by casting them to integers and checking their ULP difference (integer subtraction of their binary representations) is the most robust scale-independent comparison method used in graphics engines. Below is a comparison table of comparison strategies:

Pitfall — Comparing near zero using relative epsilon: If either $a$ or $b$ is exactly zero, the relative epsilon denominator collapses, and the comparison degrades to absolute epsilon checks. Always combine absolute and relative checks when comparing values close to zero.

7. Advanced: The Mechanics of Floating-Point Adders

7.1 ALU Shift and Rounding Steps

When a CPU ALU adds two floating-point numbers, it must first align their exponents by shifting the mantissa of the smaller number to the right. It then adds the mantissas, normalizes the result, and rounds it to the nearest representable float.

7.2 Floating-Point Adder Pipeline and Guard-Round-Sticky Bit Calculations

Let $x_1 = (-1)^{s_1} \cdot m_1 \cdot 2^{e_1}$ and $x_2 = (-1)^{s_2} \cdot m_2 \cdot 2^{e_2}$ be the two floating-point inputs. To add them, the arithmetic logic unit (ALU) executes a 5-stage hardware pipeline:

**Stage 1: Exponent Alignment.** The ALU finds the exponent difference $\Delta e = e_1 - e_2$ (assuming $e_1 \ge e_2$). To align the values, it sets the output exponent to $e_1$ and shifts the mantissa $m_2$ of the smaller number to the right by $\Delta e$ bits:

**Stage 2: Guard, Round, and Sticky Bits.** Shifting $m_2$ to the right can push bits off the end. To avoid losing precision, the hardware maintains three temporary extra bits to the right of the mantissa:

• **Guard Bit ($G$)**: The first bit shifted beyond the mantissa limit.
• **Round Bit ($R$)**: The second bit shifted beyond the mantissa limit.
• **Sticky Bit ($S$)**: The logical OR of all subsequent bits shifted out. If any 1 is shifted beyond the first two positions, $S$ is set to 1 and remains locked (sticky), securing rounding cues.

**Stage 3: Mantissa Addition.** The ALU adds or subtracts the aligned mantissas based on the sign bits:

**Stage 4: Normalization.** If $m_3 \ge 2$, the result has overflowed the normalized format. The ALU shifts $m_3$ right by 1 bit and increments the exponent $e_1$ by 1. If $m_3 < 1$, the result has underflowed. The ALU shifts $m_3$ left and decrements the exponent until the leading bit is 1.

**Stage 5: Rounding.** Finally, the ALU rounds the mantissa back to 23 bits (single) or 52 bits (double) using the G, R, and S bits. Under the default **Round-to-Nearest-Even** mode, the rounding decision follows these rules:

Where LSB is the least significant bit of the truncated mantissa. This ensures that half-way values are rounded evenly, preventing rounding bias. Below is a comparison table of ALU hardware states:

Pitfall — Denormalized hardware execution latency: When an ALU processes subnormal (denormalized) numbers, the hardware shift logic must bypass the standard fast-path normalized pipelines. On older processors, this triggers microcode traps, making floating-point operations on subnormal numbers up to 100 times slower.

To minimize bias, IEEE 754 uses **Round-to-Nearest-Even** (also known as Banker's Rounding) as the default rounding mode. If the result lies exactly halfway between two representable values, the ALU rounds to the value with a 0 in its least significant bit.

8. Advanced: Data Type Specification Comparison

8.1 Precision Specifications

The table below compares the bit allocations, exponents, and precision limits of different floating-point formats:

Pitfall — Subtraction cancellation errors: Subtracting two nearly identical floating-point numbers causes catastrophic cancellation. The leading matching bits cancel out, leaving the rounded error bits as the most significant digits, multiplying the relative error.

9. JavaScript IEEE 754 Float Decoder from scratch

9.1 Binary Inspection Code

The following JavaScript code extracts the sign, exponent, and mantissa bits of a 32-bit float using ArrayBuffer casting:

10. Interactive: Floating-Point Bit Inspector

Click "Inspect 0.1" to view the binary bits of a 32-bit float. Click "Apply Truncation" to see how finite mantissa bits cause rounding errors:

Precision Drift under Cumulative Addition

The chart below displays the absolute error growth as the decimal 0.1 is added repeatedly in a loop, showing the drift compared to exact arithmetic:

12. Frequently Asked Questions