2026 08 03 Feed Semianalysis Kimi K3 THE Manos THE Mythos

Summary

SemiAnalysis's Kimi K3 primer is mostly an architecture explainer (linear attention → DeltaNet → Gated DeltaNet → Kimi Delta Attention), but its inference-performance section is the in-scope part and it argues one tight causal chain: agentic workloads are prefill-heavy and KV-cache-hungry, so HBM capacity per node — not FLOPs — gates serving throughput. K3 does not fit on a single B200 node; on B300 it does, but after the weights the GPU HBM holds only 3.25M tokens of KV cache, and throughput stops scaling above concurrency 8 as the cache thrashes and hit rates fall from a theoretical 95% to under 10%.

It also dates the open-weight price floor: as of 30 July 2026 every OpenRouter provider serving K3 sits at $3 / M input, $15 / M output, with K3 described as "the open frontier model".

Article

(Free portion. The architecture primer is summarized; the inference-economics section — the in-scope material — is reproduced verbatim. Bracketed markers show where the source embedded figures.)

Framing

Kimi K3 took the world by storm at its announcement, sweeping leaderboards and establishing itself as the open frontier model. While the community is eager to understand how Kimi K3 works, many have been surprised by the unconventional techniques driving its performance. This article serves as a primer to understanding the core techniques of the Kimi K3 model architecture.

The bulk of the free portion derives Kimi Delta Attention (KDA), the linear-attention layer in K3's hybrid attention mechanism, tracing it from linear attention through DeltaNet and Gated DeltaNet. The load-bearing systems point in that derivation: linear attention "compresses all past key and value vectors into one hidden state S", whereas "softmax attention requires accessing all past key and value vectors" — i.e. the memory-vs-recall tradeoff that the serving numbers below cash out.

Inference performance (verbatim)

We are actively tracking Kimi K3's inference performance on InferenceX.

As of 30th July, all providers on OpenRouter have a floor of $3 per million tokens input and $15 per million tokens output. Both Nvidia and AMD had Day 0 recipes on vLLM, boasting DRAM offload and DSpark speculative decoding.

[figure — Source: OpenRouter]

On InferenceX, we benchmark Kimi K3 serving performance directly on recorded internal claude code traces. We replay an hour of these traces as they reach a steady state. There is a median of 142k input tokens and a median of 444 output tokens per turn with a median of 65 turns per session. The short output tokens per turn is typical for workloads on agentic harnesses, where the agent calls tools frequently, even edits are tool uses.

This benchmark is a big step up from our previous 8k1k/1k1k benchmark, as it truly reflects real-world agentic use cases. From a systems perspective, it is also realistic and closest to production systems. It can reflect KV cache behavior, including prefix cache and KV offloading to DRAM.

[figure — Source: InferenceX]

For Kimi K3, Day 0 bringup was easier than DSv4 due to better documentation and preparation ahead of weights release. Appropriate images and a speculative decoder model were released at the same time as the weights.

For Nvidia, bringup was simple. But due to the models' sheer size, it doesn't fit on a single B200 node. We had to use PP to get it working. DSpark also didn't work with PP.

[figure — Source: InferenceX]

For B300, the model fits on 1 node and serves well. After accounting for the weights, GPU HBM can only hold 3.25M tok. In the graph below, throughput goes up as batch sizes increase until concurrency increases above 8. This roughly correlates to the 3.25M tok KV cache budget, and cache starts to thrash, resulting in hit rates falling to < 10% when theoretical hit rate is 95%.

[Free portion ends here — "Read more" paywall stub.]