Eight-bit floating point is now the ordinary way to serve a model, and the transition happened with remarkably little argument. Hardware support arrived, the accuracy cost turned out to be small for most workloads, and the memory saving is immediate and obvious.
Worth being precise about what you gain. Halving weight size frees memory for cache, which usually buys more concurrency than it buys speed. On hardware with native support the arithmetic is genuinely faster; without it, quantisation is a memory optimisation wearing a performance costume.
Worth being equally precise about what it costs. Sparse models appear to be more sensitive than dense ones — per-expert outlier distributions squeeze badly — and the degradation is not uniform across tasks. It tends to show up on exactly the work that requires precision about small details rather than fluency, which is also the work least likely to be covered by a general benchmark.
The advice is dull: measure on your own task, sequentially, before and after. General benchmarks will tell you the average case, and the average case is not what breaks.
Leave a Reply