Summary

SemiAnalysis benchmarks TileRT, a software inference engine for NVIDIA GPUs, and finds it reaches 494.2 tokens/s/user in FP8 at 1k/1k — 1.9× faster than the best conventional result using FP4 and 3.6× faster than conventional FP8 — with end-to-end latency 4.5× better at 1k/1k. The load-bearing causal claim is a substitution argument: the ultra-high-interactivity operating point that dedicated inference accelerators (Cerebras, Groq LPU, SambaNova) exist to own can be approximated in software on standard GPUs, letting a provider carve a high-interactivity tier out of an existing GPU cluster without buying new silicon — and, critically, reallocate that capacity hour-by-hour between prefill and decode, which hardware-fixed dataflow accelerators cannot.

The counter-argument is stated in the same post and is load-bearing in the other direction: TileRT supports only batch size 1, so aggregate throughput falls (~160 tok/s/GPU vs ~240 for conventional engines), and its model catalog is two models (GLM-5/5.1, DeepSeek-V3.2) with heavy per-model ahead-of-time compilation. It inherits the dataflow-ASIC weakness (inflexibility, narrow catalog) in software form.

⚠ partial: true — the post is paywall-truncated; the free portion carries the benchmark table, the architecture description, the limitation, and the market-implication argument. Paid remainder not fetched.

Article

Ultra-High Interactivity on NVIDIA GPUs? - TileRT InferenceX

By: Bryan Shan, Daniel Nishball, Cam Quilici, and others Publication: SemiAnalysis Date: August 10, 2026

Key Performance Claims

TileRT achieved remarkable interactivity improvements in benchmarks:

• At 1k/1k input/output tokens, TileRT reached 494.2 tokens/s/user in FP8 precision—1.9× faster than the best conventional result using FP4 and 3.6× faster than conventional FP8 implementations
• At 8k/1k, TileRT delivered 340 tokens/s/user, nearly 2× faster than prior GPU benchmarks
• End-to-end latency improved 4.5× at 1k/1k and 3.0× at 8k/1k compared to previous best results

How TileRT Works

Instead of launching thousands of separate GPU kernels sequentially, TileRT compiles an entire model decode phase into a single persistent kernel that remains resident on the GPU. Key innovations include:

Persistent Engine Kernel: The system eliminates kernel setup/teardown overhead by maintaining continuous execution throughout token generation, maximizing overlap between computation, memory operations, and inter-GPU communication.

Warp and Block Specialization: Different thread groups handle specialized tasks—data movement, computation, and communication—allowing simultaneous overlapping rather than sequential stages.

GPU-Level Specialization: In models like GLM-5.1, individual GPUs take specialized roles; one GPU handles sparse indexing while others perform matrix operations.

Critical Limitation

TileRT supports only batch size 1, handling a single concurrent user per decode node. The authors characterize this as "a private rocket ship with room for just one passenger." This specialized operating point provides exceptional per-user speed but significantly reduces aggregate throughput—approximately 160 tokens/s/GPU versus 240 tokens/s/GPU for conventional engines at comparable settings.

Architectural Distinction from Competitors

Unlike dedicated inference chips (Cerebras, Groq, SambaNova) that encode dataflow principles in hardware, TileRT implements similar concepts purely through software on standard GPU architecture. Purpose-built accelerators maintain inherent advantages—particularly through larger on-chip SRAM and deterministic execution—but TileRT offers crucial flexibility advantages.

Market Implications

The article argues TileRT reframes competition with specialized silicon. Rather than replacing GPUs entirely, TileRT enables providers to carve a high-interactivity tier from existing GPU clusters dynamically. A shared vLLM prefill pool routes latency-critical requests to TileRT decode nodes while general traffic remains on conventional decode engines.

This software-based approach addresses a fundamental challenge with dedicated hardware: prefill-decode ratio flexibility. GPU pools allow hour-by-hour reallocation between roles via software configuration. Specialized accelerators fix this ratio in hardware, creating risk if demand projections prove incorrect or shift over time.

Current deployments support this pattern—Xiaomi's MiMo V2.5 Pro UltraSpeed and Z.ai's GLM-5.1 HighSpeed both implemented TileRT without purchasing new accelerator hardware.

Development Challenges

TileRT's model support remains severely limited, currently supporting GLM-5/5.1 and DeepSeek-V3.2. Ahead-of-time compilation requires extensive per-model engineering to determine tile shapes, pipeline depth, buffer allocation, and warp specialization strategies. Architectural changes necessitate substantial recompilation efforts.

The project inherits the primary weakness of dataflow ASIC development: inflexibility around model variations and narrow model catalogs.

Future Research Directions

The team plans to benchmark TileRT across batch sizes 2-8 to map its throughput-interactivity tradeoff curve and identify diminishing returns. They're also developing AgentX, a multi-turn agentic coding benchmark with 140k-token median context length to test full system performance including KV cache transfer between turns.

---

[Article continues beyond paywall; paid subscriber content not included]