The gap between reading about local inference and actually running it has closed considerably. A 4B-class model, quantised to eight bits, fits comfortably on a single consumer card with room left for a substantial cache, and the serving stacks have become boring in the good sense.
The remaining friction is not the model. It is everything around it: container runtimes that need to be told about the accelerator, drivers that must match the toolkit, orchestration layers with their own opinions about devices. Most of an afternoon goes to plumbing and roughly ten minutes to the model itself.
A few things that reliably cost people time. Device visibility inside nested containers rarely works the way the documentation implies. Scheduler-level device allocation and process-level device selection are different mechanisms and interact badly. And reasoning-tuned models will happily spend an entire token budget thinking, returning nothing, unless the mode is configured deliberately — which looks exactly like a broken endpoint.
None of this is hard. It is simply undocumented in aggregate, which is a different problem and arguably a worse one.
Leave a Reply