Several models now advertise context windows around a million tokens. The number is real in the sense that the model will accept the input. Whether it does anything useful with the middle of it is a separate question, and one the marketing tends not to address.
Two practical notes for anyone building on this.
First, the key-value cache is what actually costs you. Weights are a fixed expense; cache grows with context and concurrency together. A long window that you genuinely fill, across several concurrent requests, will exhaust memory long before the weights do. Quantising the cache helps roughly as much as quantising the weights, and it is less commonly done.
Second, retrieval has not gone away. Filling a million-token window with everything you have is usually worse than filling ten thousand tokens with the right thing, and it is always slower. The long window is most useful when the material genuinely is one document — a codebase, a deposition, a year of logs — rather than as a substitute for deciding what matters.
The window is a capability, not a strategy. Treating it as the latter is the most common way to spend a great deal of money on worse answers.
Leave a Reply