Computation and Data Movement for Inference

SemiAnalysis, Tanj Bennett, September 21, 2026. Paid post; this file is the fetched free/readable portion. Filed by the 2026-09-22 stock-market daily routine. Not a new AI-infrastructure chain — cluster already overweight.

Summary

Mixture-of-experts serving changes which tensors are active per token. The load-bearing hardware claim for this project's HBM thesis: accelerator HBM is the hot working tier and is too expensive and supply-constrained to be a good default for passive context storage. Inactive KV blobs should leave HBM between turns for cheaper network-attached DRAM (including node-CPU memory) and SSD. Bandwidth, network placement, and orchestration matter more than maximizing HBM capacity on every accelerator. That is mix color on hbm-cowos-as-binding-bottleneck and on the Engrams DRAM/SSD offload already ingested Sep 18 — not less HBM demand.

Article (readable portion)

Mixture of Experts, now widely used in frontier models, has changed both the structure of serving and the economics of useful inference. It did more than increase parameter count. It changed which tensors are active for each token, what must remain close together, which transfers need strong local bandwidth, which can tolerate a weaker network link, and how memory movement, storage, and scheduling contribute to useful throughput.

Inference runs inside a cluster coordinated by an orchestration layer such as NVIDIA Dynamo, Mooncake, or a custom scheduler. A conversation is a sequence of turns. The server keeps a context (KV cache) which is the distillation of the session. There can be thousands of turns per hour when the user launches an agent on a long-running task.

Four operating regimes: prefill, midfill, decode attention, decode experts. Prefill reaches high arithmetic intensity. Decode attention and decode expert work are generally low arithmetic intensity. Treating all four as the same workload gives away the structural advantages MoE models offer.

Prefill workers create context from a substantial block of new tokens; their output is new KV state for every model layer, stored independently of the worker that created it. Midfill workers extend previously processed context. Decode workers generate one or a few tokens per pass. Agentic work using tools creates a repeated sequence of context extension and generation, not merely one prefill followed by one decode.

Reusable context at data-center scale becomes a shared object-store of immutable blobs. Newly idle blobs can first move into shared network-attached DRAM, including node-CPU memory and dedicated memory appliances. As that tier fills, a classifier can discard blobs that are not likely for reuse, or promote longer-lived state to SSD.

Accelerator HBM is the "hot" working tier. It is too expensive and supply-constrained to be a good default for passive context storage. The active prefix and suffix should enter HBM shortly before use and leave promptly after the worker has finished with them. CPU DRAM is a useful staging and assembly tier.

Modeled Kimi K3 frontiers on B200 / B300 / GB200 (SemiAnalysis Inference Simulator; projections, not measured benchmarks): most frontier configurations remain below approximately 80 GB per GPU even after allowing for practical KV-transfer buffers. "This supports the earlier conclusion that bandwidth can be more valuable than maximizing HBM capacity on every accelerator, provided the broader storage and orchestration system moves idle state promptly."

Decode is the largest energy workload in the modeled 1,000-token output. GB200 NVL72 extends scale-up bandwidth across the rack so expert all-to-all can stay inside the scale-up domain.

What this is not

• Not a Micron Q4 print.
• Not a reason to re-rate MU/TSM on mix. Engrams already covered DRAM/SSD offload as mix color, not less HBM demand.
• Not a tenth AI-infrastructure name.

Primary URL

• https://newsletter.semianalysis.com/p/computation-and-data-movement-for