Category: Systems

Throughput numbers for inference deployments are quoted constantly and compared almost never, because the quoted figure is usually missing the two pieces of context that determine it.

The first is concurrency. A single stream and thirty-two concurrent streams produce wildly different aggregate figures on identical hardware, and continuous batching means the aggregate climbs long after per-stream latency has started to degrade. A number without a concurrency level attached is not wrong so much as meaningless.

The second is generation length. Short completions are dominated by prefill and scheduling; long ones by decode. A benchmark built on twelve-token answers is measuring the scheduler.

There is a third, less obvious one: determinism. Batch composition changes numerics, and at temperature zero a borderline input can produce different outputs depending on what else was in flight. For generation that is a curiosity. For classification it means an accuracy figure measured under load is partly measuring the scheduler too.

None of this is an argument against publishing numbers. It is an argument for publishing them with the axes labelled.

Eight-bit floating point is now the ordinary way to serve a model, and the transition happened with remarkably little argument. Hardware support arrived, the accuracy cost turned out to be small for most workloads, and the memory saving is immediate and obvious.

Worth being precise about what you gain. Halving weight size frees memory for cache, which usually buys more concurrency than it buys speed. On hardware with native support the arithmetic is genuinely faster; without it, quantisation is a memory optimisation wearing a performance costume.

Worth being equally precise about what it costs. Sparse models appear to be more sensitive than dense ones — per-expert outlier distributions squeeze badly — and the degradation is not uniform across tasks. It tends to show up on exactly the work that requires precision about small details rather than fluency, which is also the work least likely to be covered by a general benchmark.

The advice is dull: measure on your own task, sequentially, before and after. General benchmarks will tell you the average case, and the average case is not what breaks.