2026 08 24 Feed Semianalysis Agentx Inferencexv3 Does Cuda Moat

Summary

Load-bearing causal claim: on realistic multi-turn agentic coding traffic (not 8k1k kernel benches), NVIDIA's CUDA/vLLM/TRT-LLM stack still holds the inference moat against AMD MI355X for the models that matter, and the gap is software not silicon. After 2026-08-21 vLLM optimizations, B200 surpassed MI355X on DeepSeek V4 Pro perf/$; Qwen3.5 397B SGLang is a "strong hold" with >20× at 90 tok/s/user and "zero competition from AMD"; GLM 5.3 at 150 tok/s/user NVIDIA is up to 5× cost-efficient — "even if the competitor chip hardware was sold for free" (datacenter power still paid) NVIDIA wins. AMD ATOM beats GB300 on some TCO slices but "most AI labs … do not want to use ATOM in production." B300's 50% extra HBM vs B200 shows up as KV-cache hit rate (91% vs 73% under AgentX load), not raw FLOPs. Rubin due later this month.

This is independent written corroboration for cuda-moat-erosion-at-inference / cuda-moat-erosion-to-nvda-rerate: the erosion thesis is not printing on agentic production traffic yet. Also attaches to hbm-cowos-as-binding-bottleneck (B300 HBM capacity → KV hit rate) and inference-demand-to-wafer-scale-advantage (interactivity vs TCO is now the scoring function). Not a new chain — a measured update to the moat.

Article

Since the Claude Code inflection point in November 2025, long-context, multi-turn agentic workloads have grown rapidly. They now dominate traffic for production inferencing. In April 2026, OpenAI’s Enterprise agentic spending overtook ChatGPT spending.

Agentic workflows have decisively taken the baton. Today, we announce AgentX 1.0 - the world’s first fully open source, multi-turn agentic coding inference benchmark at 1 million context, released under Apache 2.0. Our full dashboard is available here.

Source: SemiAnalysis In the past most measured performance based on fixed sequence length prefill and decode workloads, but this is an inaccurate way to measure workloads. Reality is multi-turn, long context, high prefill reuse, with sub agent bursts, KVCache offload, and numerous tool calls. As such we aimed to build the correct way for the industry to measure AI hardware and software performance.

We have spent more than $3M building this dataset. Today, we open source everything. InferenceXv3 implements AgentX, a new realistic scenario in addition to the existing “fixed sequence length” scenarios (8k1k, 1k1k, 1k8k). It improves the benchmark scenarios by using agentic coding traffic instead of the previous single-turn traffic of 8k input and 1k output tokens.

The full matrix runs on ~2MW of continuously operated compute across over 1000 chips spanning a wide range of SKUs, featuring the MI355X, GB300 NVL72, GB200 NVL72, B300, B200, MI325, MI300X, H200, and RTX Pro Servers. Rubin arrives later this month, and TPUs and Mi455X UALoE72 arrive later this year. Please drop a star if you found our free open source work valuable.

It is great to see amazing performance from both NVIDIA and AMD on agentic workloads. NVIDIA does very good on a lot of frontier models while AMD also does well on some frontier models for specific comparsions.

Source: SemiAnalysis GitHub The most valuable thing AgentX produced in its first months was not the initial results. It was the massive industry impact the benchmark is already having. Over 70+ upstream PRs for optimizing real world production agentic workloads across vLLM, SGLang, TensorRT-LLM, ATOM, AITER, Dynamo, LMCache, and Mooncake, uses AgentX as the north star benchmark proxy. Most of these optimization improvements are transferable to production traffic. We deep dive into each of these optimizations later in the article.

Source: SemiAnalysis Open source is a core principle for InferenceX and thus, we open more of the stack than most people who use that word. That includes an open frontend, a public database served through an easily consumable REST API that multiple tier 1 AI lab’s capacity planning teams already consume , public GitHub Actions CI provenance, logs , and accuracy validation on every single point. Crucially, our benchmark configs mainly track recipes.vllm.ai and SGLang cookbook on upstream images such that we are measuring the performance actual customers are experiencing instead of measuring benchmax’ed images.

In three to four weeks, we will release an AgentX update article. It will cover further optimizations to agentic workloads, plus updated performance results from AMD and Nvidia. It is important to understand that the profile of agentic workloads is updating fast. InferenceX will continue to move swiftly to benchmark the relevant workloads.

InferenceX is 100% committed to being open-source - this would not be possible without the contributions and support from our OSS partners. We would like to thank the following people that have made massive contributions to the AgentX 1.0 release:

Inferact/vLLM : Roger Wang, Yifan Qiao, Simon Mo, Jeff Ma, and many others

RedHat/llm-d : Michael Goin, Robert Shaw, Tyler Michael Smith

RadixArk/SGLang : Baizhou Zhang, Yuwei An, Mingyi Lu, and many others

LMCache/TensorMesh : Samuel Shen

Weka : Callan Fox, ValB

MoonCake Maintainers: Teng Ma, Xu Wenjie, Ke Yang

AMD : Thomas Wang, HaiShaw, Andy Luo, Seungrok Jung, Chun Fang, Parth Panchal, Bill He, Theresa Shan, Hongxia, Fangzhou, Gilbert Lei, Yanfei Wang, Duyi Wang, Peng Sun, Lingpeng Jin, Simon Danielsson, Xiaohu Guo, Haichen Zhang, Chang Liu, Doug Lehr, Poovaiah Palangappa, and many others in the AMD Shanghai Development Centre

Nvidia : Xin Li, Anthony Casagrande, Kedar Potdar, Ankur Singh, Ishani Dhanani, Nick Comly, Nvidia Shanghai TensorRT-LLM team, and many others

Anthropic staff, for promptly fixing multiple bugs that made implementing AgentX possible

GitHub: Austen Stone for helping with reliability of GitHub Actions that AgentX uses

And many others

In addition, we are thankful to all who support our open source InferenceX initiative, including Meta, Microsoft, Oracle, OpenAI, MiniMax, Moonshot Kimi, Alibaba Qwen, and Zhipu GLM.

Source: InferenceX A Brief Overview of Agentic Workloads

At a high level, an agentic workload is characterized by four elements:

Multi-turn: a session includes many user / assistant interactions (tens or hundreds) compared to a handful in a chatbot scenario. Multi-turn, long context, high prefill reuse, with sub agent bursts and numerous tool calls.

Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly.

High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached typically tends towards 1.

Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KVCache patterns.

Source: DeepSeek, SemiAnalysis Considering the characteristics above, benchmarking these workloads is fundamentally different from the existing fixed sequence length benchmarks. Namely, agentic inference is inherently a systems problem . Because of extremely high prefix reuse, KV tensors must be efficiently transferred across nodes/ranks (NIXL, MORI-IO, Mooncake). Additionally, different conversations should be routed to different nodes/ranks depending on where the appropriate prefix resides in order to maximize cache hit rate (LLM-d, Dynamo, vLLM/SGLang router). Long context conversations stress the HBM capacity for KV cache and necessitate offloading KV tensors to different tiers of memory (DRAM, SSD), a process that needs to be carried out efficiently (Mooncake Store, LMCache, vLLM Simple Offloading, SGLang HiCache).

This is in contrast to fixed sequence length, single turn workloads where prefix reuse is not relevant and inference performance is largely reflective of baseline chip/kernel performance. This is not to say that the plethora of fixed sequence length data on InferenceX is not important. In fact, stripping away the complexities of agentic serving shows clearly how low-level inference performance optimizations are progressing. It also provides an important baseline for AgentX results.

In an attempt to make the AgentX workloads as realistic as possible, we collected an initial corpus of 393 internal SemiAnalysis anonymous Claude Code traces to replay. To anonymize the content while keeping the original prefix reuse pattern, we use a method similar to the Qwen-Bailian dataset , one of the earliest corpora of production traces. We then use AIPerf to reconstruct the traces according to the original schedule of requests at varying levels of concurrent clients. We worked with Anthropic to ship two Claude Code features to make the AgentX dataset possible. We thank the Anthropic staff for their help.

https://github.com/anthropics/claude-code/issues/49207

https://github.com/anthropics/claude-code/issues/66761

This brief introduction to agentic workloads should give the reader enough context to understand the results in the next section. In a later section, we will provide a deeper technical dive into the methodology, replay harness, and dataset.

Agentic Coding Inference Performance

When looking at inference coding performance, OpenAI, Anthropic, xAI, and other frontier labs focus on three things. They look at performance per dollar versus interactivity (TPOT), TTFT (time to first token), and overall end-to-end task completion. Performance per megawatt is also important, considering that terrestrial datacenter power is a critical constraint (money is a social construct and it appears the labs have an unlimited supply, but power is physically hard to come by in this day and age). Our datacenter model has estimates of quarter by quarter build up of power demand and supply.

In this section, we highlight some of the overall agentic performance themes across frontier models. We strongly encourage the reader to use this as a guide to investigate the results for themselves . All of the data is open source and the community has the opportunity to draw their own conclusions on the current state of real world inference performance.

DeepSeek V4 Pro 0813

DeepSeek V4 Pro 0813 is an ultra popular frontier open weight model from China. It has ~1.6 Trillion parameters with 49 Billion active parameters.

As of August 21, the following graph shows the best performance per SKU for all submissions, normalized by total cost of ownership (TCO).

The ISL/OSL distribution of all requests among all DeepSeek v4 runs was as follows: ISL p50=88k, p90=272k, p95=404k, p99=675k and OSL p50=413, p90=2.2k, p95=3.7k, p99=8.6k.

Source: InferenceX In general, it is important to consider both tokens per second per user (TPS - also known as interactivity) and TTFT, since these often come at the expense of one another . For instance, in the graph above, some SKUs achieve very high throughput at decent interactivity, however TTFT is severely degraded. What is an “acceptable” p90 TTFT varies heavily depending on the application. For most production systems serving agentic workloads, you can expect p90 TTFT to be anywhere from 200-5,000ms. Anything over 5-10s is pushing the boundary of what can be considered “online inference.” There are still practical applications for the ultra-high throughput sector of the curve, where latency does not matter and peak system utilization is desirable (batch processing, very long running agents, etc).

In terms of single node performance, MI355X open-source performance (vLLM) trails behind vendor specific ATOM (AMDs equivalent of TensorRT LLM). We think it’s great that AMD is pushing the frontier quickly with ATOM, however we encourage them to make a higher priority of upstreaming these improvements into vLLM.

Source: InferenceX AMD’s distributed inference (DI) team has made great progress on 8k1k scenarios over the past six months. The team still has some way to go before DI is a viable solution for realistic workloads. In terms of throughput per GPU vs. interactivity, we observe the 1xDEP8+1xDEP8 disagg config is only able to realize slight performance gains in the high throughput scenarios, while actually performing worse in low latency scenarios.

To make matters worse, any increase in throughput at the middle-to-high interactivity configs is overshadowed by the significant spike in p90 TTFT. One of the reasons for this is the use of SGLang’s --enable-prefill-delayer argument above concurrency 64, which postpones prefill admission so DP ranks can form fuller batches (for up to 30 forward passes). Additionally, these points also increase chunked prefill size from 8,192 to 65,536.

Source: InferenceX On e2e latency, ATOM MI355X beats B200 vLLM (it does not beat B300 or B200 SGLang though). The issue with this is that most AI labs in China or the west do not want to use ATOM in production besides 1 small advertising business unit at Alibaba Corp due to tons of missing features. The main Qwen LLM org at baba does not use ATOM in production.

Source: InferenceX Before August 21, 2026, AMD’s MI355X strong SGLang development team was matching B200 vLLM on performance per dollar on end to end (e2e) performance.

Source: InferenceX However, B300 vLLM and B200 SGLang still beat AMD’s MI355X.

Source: InferenceX After August 21, 2026, due to optimizations in vLLM from Inferact and Nvidia, the performance per dollar of Nvidia’s B200 has surpassed that of the MI355X. This is a close race and we are excited to see the performance optimizations over the next couple weeks. We will be publishing an AgentX update article very soon.

AMD has listed their DeepSeekv4 vLLM optimization and includes lots of exciting things that they can do to improve their performance .

Source: SemiAnalysis Now turning to Nvidia. Their most competitive solutions are GB300 Dynamo TRTLLM and GB200 Dynamo vLLM. Both configs rely on PD disagg to achieve high throughput at reasonable interactivity. Additionally, GB300 configs employ wide-EP (DEP32) decode instances in order to achieve higher throughput at the middle of the frontier.

Note that the 2xDEP8+1xDEP12 GB200 point is significantly closer to the 3xDEP8+1xDEP16 GB300 point in terms of TPS compared to TTFT. Again, TTFT is, in general, more sensitive to the “spikiness” of the workload. Since the GB300 point achieves much higher overall concurrency, it incurs more subagent traffic and hence more cold prefills. We can see this in the TTFT chart for the point:

Source: InferenceX

Source: InferenceX When normalized by TCO, B300 vLLM versus B200 vLLM aggregated performance is quite similar. The main difference being that B300 can “squeeze” out extra throughput, given its 50% increase in HBM capacity over B200.

Source: InferenceX We can further visualize this difference using our server metric visualizations, which are new to AgentX.

Under the load of 384 concurrent agentic traces, B300 vLLM DEP8 w/ 3TB DRAM via vLLM simple offloading achieved a 91% HBM cache hit rate with an additional 1.36% DRAM cache hit rate. This is because the HBM KV cache working set size is approximately 43M tokens with this configuration, and the load barely exceeds this number of tokens in flight at any given time.

Source: InferenceX With B200 concurrency 196 (all other parameters stay the same), we see only 73% HBM cache hit rate and rely more heavily on DRAM with an offload cache hit rate of nearly 20%. We observe that the HBM KV cache working set size is 22M tokens, roughly half that of B300.

DRAM KV offloading is typically implemented as a write-through cache, meaning every prefix written to the HBM cache is also written to the DRAM cache. Therefore, it is most effective when the amount of DRAM available for offloading is significantly bigger (a multiple of 1.5-3) than HBM KV cache capacity.

Source: InferenceX H200 SGLang FP8 is able to serve DeepSeek v4 at low concurrency, and is even competitive with B200/MI355X SGLang from a perf/$ standpoint. However, it cannot compete with the newer SKUs in high throughput scenarios due to lack of HBM.

Source: InferenceX Furthermore, the reliance on DRAM KV offloading at higher concurrencies leads to unreasonable latency as the number of users scales.

Source: InferenceX Overall, MI355X performs decently well compared to its main competitors B200 and B300. Performance is most comparable at the lower throughput / lower latency parts of the curve, where only tensor parallelism and more rudimentary kernels are deployed. AMD needs to work on optimizing DEP kernels on MI355X to be more competitive in the high throughput scenarios, especially given the 1.5x HBM over B200.

Source: InferenceX Kimi K3 2.8 Trillion Parameters

Kimi K3 is another frontier open weight model from China that has 2.8 Trillion total parameters. This is in the same range in terms of number of parameters vs Claude’s Mythos/Fable5 model architecture. We use this as an open weights proxy model architecture. The Kimi K3 model is so big that it does not even fit on a single B200 server and requires using wide EP/wide TP or pipeline parallelism in order to fit all of the weights. On vLLM, speculation decoding/DSpark did not compose at all with pipeline parallelism until very recently, so B200 performance on Kimi K3 was horrible and was getting mogged by MI355X since B200 was unable to use speculative decoding with pipeline parallelism.

MI355X vLLM worked out of the box on day 0 for short context single turn workloads, but for long context multi turn workloads, MI355X AITER and Triton kernels suffered a massive panic attack on the first week and upstream vLLM was completely unusable for MI355X on realistic workloads.

Source: SemiAnalysis InferenceX Hopper struggles to serve the AgentX workload for Kimi K3 since Kimi is a massive model and because vLLM maintainers/NVIDIA have not been focusing on optimizing Hopper for Kimi K3. Hopper (SM90) requires custom tuned kernels for K3 along with TP32/EP32 tuned shapes for serving at high interactivity.

Source: SemiAnalysis InferenceX We think it is great that AMD is quickly pushing K3 performance forward with ATOM. However, we encourage AMD to further prioritize upstreaming these improvements into vLLM. ATOM is currently AMD's best-performing engine, but vLLM remains the more relevant comparison for customers using an upstream open-source serving stack.

Source: SemiAnalysis InferenceX On part of the curve between 40 to 60 second e2e latency, MI355X ATOM beats even GB300 NVL72 vLLM on performance per dollar.

Source: SemiAnalysis InferenceX MiniMax M3

Nvidia absolutely destroys all competitors on MiniMax M3 432B. AMD software performance is horrible on MiniMax especially at high context length due to AMD engineering leadership incentivizing tuning only for short context single turn workloads and ignoring long context multi turn workloads.

Source: SemiAnalysis InferenceX B300 TRT-LLM TP2 owns the M3 crown. There is a lack of DP-attention points as it is non-optimal on M3 since KV cache locality becomes a routing constraint. This is further explained later on. For GB200 at concurrency 40, TP4/EP4/DPA gets 0.60x the throughput of plain TP4 at a >3x p90 TTFT. At concurrency 32 it hits 28.8% of cache vs 96.0% theoretical. Each DP rank owns a private quarter of the pool; a 300k-token session re-landing on the wrong rank recomputes everything. No decode config with EP appears on the M3 frontier, likely as the concurrency is not high enough to balance the loads on all experts.

B200/B300 also completely beat their rack-scale counterparts for MiniMax M3 on TCO-normalized throughput. On AgentX, the rack-scale advantage isn’t as pronounced as the Dynamo router can become the bottleneck because its work scales with the number and length of live prefixes. Optimizations on this and the several fixes which moved throughput by double-digit percentages are discussed later on in the article. Also, there are no well tuned kernels for wideEP, wide DCP, nor wide TP. And since GB200/300 have higher TCO so without wide ep/wide DCP, it shows up as worse perf per TCO.

With that being said, we also expect further optimizations from Nvidia on their rack scale solutions for this SKU. We will make sure to highlight these in our follow up article.

No submission currently runs context parallelism, despite P90 ISL of 317k. With 4 KV heads, DCP caps at 2 even at TP8, and the MSA indexer needs its own context-parallel handling (a vLLM PR is opened), see the Context Parallelism section for more discussion on this topic.

Source: SemiAnalysis InferenceX All of Nvidia’s Pareto optimal points include KV offload above concurrency 20, but for AMD none of the Pareto optimal points use KV offload to DRAM. AMD also uses KV offload less than Nvidia on the other models. The reason for this is that GPU-to-CPU transfers for CPU KVCache offloading are highly inefficient on AMD vLLM. The hipMemcpyBatchAsync API was missing until ROCm 7.14. Without hipMemcpyBatchAsync, vLLM’s native Simple CPUOffloading requires doing serialized Memcpy from CPU to GPU instead of batching them into larger message sizes.

It is also worth mentioning that vLLM performance is very comparable to TRT-LLM in terms of throughput versus p90 interactivity. Additionally, vLLM performs better in terms of throughput versus p90 TTFT.

Source: SemiAnalysis InferenceX Qwen3.5 397B

Qwen3.5 397B uses GatedDeltaNet instead of vanilla attention for every couple of layers. GatedDeltaNet was invented at MIT/Nvidia Research and has a theoretically constant state storage requirement instead of vanilla attention’s linear storage requirements. This means that it has lower storage requirements compared to an equivalent dense attention model. Unlike end to end model training research like the Nemotron disaster, Nvidia Research is great at fundamental research like GDN and LatentMoE which is used on frontier models.

Note that this model’s native max context length is 262k tokens, so we use the truncated dataset . This simulates a workload on a smaller model where the max context length would be frequently reached with many compactions, how users would actually use this model.

Qwen3.5 397B is a strong hold for NVIDIA on SGLang versus SGLang, with over 20x better performance at 90 tok/s/user. There is currently zero competition from AMD for Qwen3.5 SGLang.

Source: SemiAnalysis InferenceX Again, we observe Nvidia over optimizing for interactivity at the cost of TTFT, especially in the case of TRT-LLM. In the graph above, all of the Nvidia SGLang submissions have much lower p90 TTFT when compared to TRT-LLM.

Compared to H100, on Qwen3.5, B300 FP4 has 12x better performance per dollar.

Source: SemiAnalysis InferenceX GLM 5.3

GLM 5.3 builds on top of GLM5.2 744B with additional post training. This is a frontier level model.

In terms of OSS SGLang performance, this is another model where Nvidia again beats AMD on realistic agentic inference performance. At 150 tok/s/user p90 interactivity, Nvidia has up to 5x better cost efficiency, With the current state of AMD software, at 150 tok/s/user, Nvidia’s performance advantage is so great that even if the competitor chip hardware was sold for free (but with providers still of course paying for datacenter hosting and power and other operating costs), cost per token would still be cheaper when using Nvidia.

We look forward to AMD’s performance optimizations in the upcoming AgentX update article in a couple of weeks, which will also include some other very exciting results.

Source: SemiAnalysis InferenceX When looking at ATOM, AMD has better performance per dollar than GB300 NVL72 SGLang and even TRTLLM for some parts of the range of p90 E2E Normalized Interactivity. Great work to the AMD team on these results. Again, we look forward to AMD porting over these optimizations to SGLang. We also are looking forward to NVIDIA quickly optimizing GB300 NVL72 in the coming weeks.

Source: SemiAnalysis InferenceX We take a moment to introduce an experimental metric which we call E2E Normalized Interactivity . At a high level, this metric is supposed to evaluate how fast a user experiences responsiveness when considering both TTFT as well as TPS. It is defined by OSL/E2EL. Substituting the fact that E2EL equals TTFT plus OSL times TPOT (in reality only OSL - 1 tokens are decoded), we get the following equation.

This is effectively interactivity (the 1/TPOT portion) plus an additional penalty proportional to TTFT.

Source: SemiAnalysis Please note that this metric is experimental and is not perfect. For instance, it heavily penalizes high TTFT and doesn’t capture all the nuances of certain optimizations such as PD disaggregation. All submissions for AgentX v1.0 optimize for both regular interactivity and TTFT separately. We will continue working on new north star metrics that reflect all nuances of modern agentic inference.

AgentX Industry Impact - Optimizations for Agentic Workloads

[Clipping truncated after the GLM 5.3 / E2E-normalized-interactivity results. Remainder of the RSS item is systems methodology — KV routing, Mooncake, Dynamo, context-parallelism — not required to cite the NVDA-vs-AMD agentic-inference result. Full post: https://newsletter.semianalysis.com/p/agentx-inferencexv3-does-cuda-moat]