Category: Practice

A pattern worth naming: systems that accumulate context automatically tend to accumulate it faster than anyone budgeted for, and the failure is gradual rather than sharp.

The mechanics are simple. Cache grows per token and per concurrent request. Automatic history, retrieved documents and tool output all add tokens nobody explicitly asked for. Throughput degrades smoothly, so there is no obvious moment where something breaks — just a system that is slower this month than last, for reasons no single change accounts for.

The instrumentation to catch this is not complicated: tokens per request, split into what the user supplied and what the system added, tracked over time. Almost nobody has it, because at the point the system is built the ratio is obviously fine.

It is the same failure mode as unbounded log retention, and it has the same fix. Decide what the ceiling is before you need one, and make the thing that exceeds it visible.

The gap between reading about local inference and actually running it has closed considerably. A 4B-class model, quantised to eight bits, fits comfortably on a single consumer card with room left for a substantial cache, and the serving stacks have become boring in the good sense.

The remaining friction is not the model. It is everything around it: container runtimes that need to be told about the accelerator, drivers that must match the toolkit, orchestration layers with their own opinions about devices. Most of an afternoon goes to plumbing and roughly ten minutes to the model itself.

A few things that reliably cost people time. Device visibility inside nested containers rarely works the way the documentation implies. Scheduler-level device allocation and process-level device selection are different mechanisms and interact badly. And reasoning-tuned models will happily spend an entire token budget thinking, returning nothing, unless the mode is configured deliberately — which looks exactly like a broken endpoint.

None of this is hard. It is simply undocumented in aggregate, which is a different problem and arguably a worse one.