KV-cache & inference memory tiering
Prefix/KV caching as an economic lever
KV-cache management, prompt caching and cache offload to CXL/host memory (LMCache, Mooncake, prefix caching in vLLM/SGLang) are now first-order to inference economics for long-context and agentic workloads. Cache reuse and disaggregated prefill/decode are where much of 2025-2026 inference cost reduction actually comes from.
Tech
prefix caching, KV offload
Impl
LMCache, Mooncake
How it fits the stack
KV-cache & inference memory tiering with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
KV-cache & inference memory tiering in the AI stack. KV-cache & inference memory tiering with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 5 entities, 4 relationships
- KV-cache & inference memory tiering —depends on→ High-Bandwidth Memory (HBM)
- KV-cache & inference memory tiering —uses→ SGLang
- KV-cache & inference memory tiering —uses→ vLLM
- NVIDIA TensorRT-LLM / Dynamo —uses→ KV-cache & inference memory tiering
Depends on ↑ · 3