A pattern worth naming: systems that accumulate context automatically tend to accumulate it faster than anyone budgeted for, and the failure is gradual rather than sharp.
The mechanics are simple. Cache grows per token and per concurrent request. Automatic history, retrieved documents and tool output all add tokens nobody explicitly asked for. Throughput degrades smoothly, so there is no obvious moment where something breaks — just a system that is slower this month than last, for reasons no single change accounts for.
The instrumentation to catch this is not complicated: tokens per request, split into what the user supplied and what the system added, tracked over time. Almost nobody has it, because at the point the system is built the ratio is obviously fine.
It is the same failure mode as unbounded log retention, and it has the same fix. Decide what the ceiling is before you need one, and make the thing that exceeds it visible.
Leave a Reply