Throughput numbers for inference deployments are quoted constantly and compared almost never, because the quoted figure is usually missing the two pieces of context that determine it.
The first is concurrency. A single stream and thirty-two concurrent streams produce wildly different aggregate figures on identical hardware, and continuous batching means the aggregate climbs long after per-stream latency has started to degrade. A number without a concurrency level attached is not wrong so much as meaningless.
The second is generation length. Short completions are dominated by prefill and scheduling; long ones by decode. A benchmark built on twelve-token answers is measuring the scheduler.
There is a third, less obvious one: determinism. Batch composition changes numerics, and at temperature zero a borderline input can produce different outputs depending on what else was in flight. For generation that is a curiosity. For classification it means an accuracy figure measured under load is partly measuring the scheduler too.
None of this is an argument against publishing numbers. It is an argument for publishing them with the axes labelled.
Leave a Reply