For most of the past few years the interesting question about inference hardware was how many of one vendor’s cards you could get. That is finally changing, and the change is more architectural than commercial.
AMD’s MI355X has shown competitive throughput on large sparse models, which matters mainly because it establishes that the software stack is no longer the blocker it was. Intel’s Crescent Island is aimed squarely at inference rather than training. There are custom parts in flight from the labs themselves.
The more interesting entrant is compute-in-memory. Moving data between memory and compute dominates the energy budget of inference; architectures that do arithmetic where the weights already sit sidestep that rather than optimising it. Whether the approach survives contact with real models is unproven, but the physics argument is sound and the funding has arrived.
What none of this changes yet is the tooling. A model that runs well on one vendor’s stack still takes real work to run well on another’s, and the portable paths tend to cost you the optimisations that made the hardware worth buying. Competition on silicon arrives well before competition on software, and the second one is what actually lowers prices.
Leave a Reply