Floating-point numbers for ML: fp32 vs bf16
Every number in a neural network is stored as sign, exponent and mantissa. This page lets you build those bits yourself, multiply two numbers the way hardware does, see how the formats differ on the number line, and find out why mixed-precision training adds up its results in fp32. Everything runs in your browser; the arithmetic is exact and was cross-checked against PyTorch.
The one-line model: value = (−1)sign × (1 + mantissa/2M) × 2exponent − 127. The exponent decides how big, the mantissa decides how precisely, the sign decides which side of zero. fp32 has M = 23 mantissa bits, bf16 has M = 7 (same 8 exponent bits, so the same range).
1. Convert a number to bits
Type a number (or click any bit below to flip it) and choose a format. Rounding is round-to-nearest, ties-to-even — what hardware does, not truncation.
2. Multiply two numbers, step by step
Hardware multiplies two floats with three independent pieces of work: XOR the signs, add the exponents, and multiply the significands as integers — then round once.
3. The number line: how dense is each format?
Every tick is a number the format can represent. Zoom in on a value and watch bf16, TF32 and fp32 diverge. Where a lane is too dense to draw, it is shaded and the tick count is shown.
Hover (or touch) a lane to read the value of the nearest tick.
4. Why the running sum stays in fp32
A matrix multiply is a long sum of products. If the accumulator itself is bf16, then once it grows, small additions fall below half a tick and vanish. Tensor cores multiply in bf16 but keep the running total in fp32.
Same effect inside a dot product (random bf16 inputs in [0,1))
5. What it buys you: measured on a laptop GPU
One 8192×8192 matrix multiply in a loop on an NVIDIA RTX 2000 Ada Laptop GPU (8 GB), PyTorch 2.10, a single run. Laptop GPUs throttle, so treat these as indicative, not as a benchmark.
| data type | significand bits (incl. implicit 1) | measured TFLOP/s | vs fp32 |
|---|---|---|---|
| fp32 (CUDA cores) | 24 | 4.4 | 1.0× |
| TF32 (tensor cores) | 11 | 10.8 | 2.5× |
| bf16 (tensor cores, fp32 accumulate) | 8 | 25.5 | 5.8× |
Smaller significands mean far smaller multiplier circuits (roughly 24×24 vs 8×8 partial-product bits), and half the bytes to move. In a full training step the gain is smaller, because LayerNorm, softmax, the loss and the optimizer are bound by memory traffic and mostly stay fp32.
Take-aways
- Sign is free: it never causes error.
- Exponent adds exactly when you multiply; it only hurts when the result leaves the range (overflow to ∞, underflow to 0). bf16 keeps fp32's range, fp16 does not.
- Mantissa is the only place rounding happens. Each dropped bit doubles the tick size: bf16 ticks are 216 = 65,536× coarser than fp32's.
- The product of two bf16 numbers has at most 16 significant bits, so it fits exactly in fp32's 24. Multiply in bf16, accumulate in fp32, and the only rounding left is the one on the inputs.
Source: floats-core.js (exact encoder/decoder, tested against PyTorch for fp32, bf16 and fp16 including subnormals and overflow) and floats-ui.js.