The Inference Desk

Author: M. Reyes

  • The quiet cost of context you did not choose

    The quiet cost of context you did not choose

    A pattern worth naming: systems that accumulate context automatically tend to accumulate it faster than anyone budgeted for, and the failure is gradual rather than sharp.

    The mechanics are simple. Cache grows per token and per concurrent request. Automatic history, retrieved documents and tool output all add tokens nobody explicitly asked for. Throughput degrades smoothly, so there is no obvious moment where something breaks — just a system that is slower this month than last, for reasons no single change accounts for.

    The instrumentation to catch this is not complicated: tokens per request, split into what the user supplied and what the system added, tracked over time. Almost nobody has it, because at the point the system is built the ratio is obviously fine.

    It is the same failure mode as unbounded log retention, and it has the same fix. Decide what the ceiling is before you need one, and make the thing that exceeds it visible.

  • Edge inference toolkits are doing the unglamorous work

    Edge inference toolkits are doing the unglamorous work

    Toolkit releases do not generate much discussion, which is a shame, because the constraints at the edge are more interesting than the ones in the datacentre. Fixed memory, no thermal headroom, hardware that will not be replaced for years, and a model that has to keep working when the network does not.

    The recurring theme in recent releases is breadth rather than peak performance: more architectures supported, more generative pipelines, better memory behaviour on modest devices. That is the right priority. Peak throughput on a flagship part is a benchmark; running acceptably on the hardware already deployed is a product.

    The awkward part remains the same as always. Every optimisation that makes a model fit is a change to its behaviour, and the further you push, the less the published evaluation numbers apply to what you are actually shipping. Edge deployment is where the gap between the model you tested and the model you deployed becomes widest, and where the fewest people are measuring it.

  • Self-hosting a small model is now a weekend project

    Self-hosting a small model is now a weekend project

    The gap between reading about local inference and actually running it has closed considerably. A 4B-class model, quantised to eight bits, fits comfortably on a single consumer card with room left for a substantial cache, and the serving stacks have become boring in the good sense.

    The remaining friction is not the model. It is everything around it: container runtimes that need to be told about the accelerator, drivers that must match the toolkit, orchestration layers with their own opinions about devices. Most of an afternoon goes to plumbing and roughly ten minutes to the model itself.

    A few things that reliably cost people time. Device visibility inside nested containers rarely works the way the documentation implies. Scheduler-level device allocation and process-level device selection are different mechanisms and interact badly. And reasoning-tuned models will happily spend an entire token budget thinking, returning nothing, unless the mode is configured deliberately — which looks exactly like a broken endpoint.

    None of this is hard. It is simply undocumented in aggregate, which is a different problem and arguably a worse one.

  • Tokens per second is a bad benchmark on its own

    Tokens per second is a bad benchmark on its own

    Throughput numbers for inference deployments are quoted constantly and compared almost never, because the quoted figure is usually missing the two pieces of context that determine it.

    The first is concurrency. A single stream and thirty-two concurrent streams produce wildly different aggregate figures on identical hardware, and continuous batching means the aggregate climbs long after per-stream latency has started to degrade. A number without a concurrency level attached is not wrong so much as meaningless.

    The second is generation length. Short completions are dominated by prefill and scheduling; long ones by decode. A benchmark built on twelve-token answers is measuring the scheduler.

    There is a third, less obvious one: determinism. Batch composition changes numerics, and at temperature zero a borderline input can produce different outputs depending on what else was in flight. For generation that is a curiosity. For classification it means an accuracy figure measured under load is partly measuring the scheduler too.

    None of this is an argument against publishing numbers. It is an argument for publishing them with the axes labelled.

  • FP8 became the default and mostly nobody noticed

    FP8 became the default and mostly nobody noticed

    Eight-bit floating point is now the ordinary way to serve a model, and the transition happened with remarkably little argument. Hardware support arrived, the accuracy cost turned out to be small for most workloads, and the memory saving is immediate and obvious.

    Worth being precise about what you gain. Halving weight size frees memory for cache, which usually buys more concurrency than it buys speed. On hardware with native support the arithmetic is genuinely faster; without it, quantisation is a memory optimisation wearing a performance costume.

    Worth being equally precise about what it costs. Sparse models appear to be more sensitive than dense ones — per-expert outlier distributions squeeze badly — and the degradation is not uniform across tasks. It tends to show up on exactly the work that requires precision about small details rather than fluency, which is also the work least likely to be covered by a general benchmark.

    The advice is dull: measure on your own task, sequentially, before and after. General benchmarks will tell you the average case, and the average case is not what breaks.

  • Inference silicon stops being a one-horse race

    Inference silicon stops being a one-horse race

    For most of the past few years the interesting question about inference hardware was how many of one vendor’s cards you could get. That is finally changing, and the change is more architectural than commercial.

    AMD’s MI355X has shown competitive throughput on large sparse models, which matters mainly because it establishes that the software stack is no longer the blocker it was. Intel’s Crescent Island is aimed squarely at inference rather than training. There are custom parts in flight from the labs themselves.

    The more interesting entrant is compute-in-memory. Moving data between memory and compute dominates the energy budget of inference; architectures that do arithmetic where the weights already sit sidestep that rather than optimising it. Whether the approach survives contact with real models is unproven, but the physics argument is sound and the funding has arrived.

    What none of this changes yet is the tooling. A model that runs well on one vendor’s stack still takes real work to run well on another’s, and the portable paths tend to cost you the optimisations that made the hardware worth buying. Competition on silicon arrives well before competition on software, and the second one is what actually lowers prices.