There's a problem with RAG that many people don't realize: every single query hits the vector database. Even if that document hasn't been updated in three months.

Expensive, slow, and unnecessary.

Cache-Augmented Generation (CAG) works by keeping static information directly in the model's KV memory. KV cache is the collection of key and value vectors the model builds internally as it reads each token—you can think of it as the model's understanding of its context. Both OpenAI and Anthropic's APIs support prompt caching, which you can start using today.

A more practical approach is layered RAG + CAG:

- Static data (policies, documents) → cached in KV memory, computed once

- Dynamic data (recent updates, real-time documents) → retrieved on-demand for each query

Faster inference, lower cost, and no more redundant work.

Video: RAG (left) goes through embedding and vector retrieval for every query; RAG + CAG (right) caches already retrieved context in KV memory for direct reuse in subsequent requests.

## How Big Is This Waste?

This problem isn't unique to RAG. Akshay 🚀 shared numbers in his companion article *Your KV Caching Is Broken*. A Stanford study found that around 62% of content sent in every AI agent call is duplicated: the same system prompts, the same tool definitions, the same documents, resent over and over again.

Token prices dropped 80% between 2023 and 2026: a GPT-4 level model went from $30 per million tokens to $0.40. But agent workloads consume 5 to 30 times more tokens per task than regular chat. Total token volume has outpaced price cuts.

Uber rolled out Claude Code company-wide to its entire engineering organization, and burned through its entire 2026 AI budget in four months. Gartner predicts that 40% of AI agent projects will be canceled by 2027 due to cost overruns.

The problem isn't that tokens aren't cheap enough. It's that most tokens shouldn't exist at all.

## Why Is KV Cache So Expensive?

Every time the model processes a piece of text, it calculates a pair of Key and Value vectors for each token, recording the relationship between that token and every other token in the context. These vectors together make up the KV cache.

Computational cost grows with the square of context length: double the context, quadruple the cost. A simple 4K token chat is cheap; a 128K token agent session (with tools, documents, and history all included) is a completely different story.

A single MI300X GPU generates around 15TB of KV cache per day, most of which is discarded once the request finishes.

The KV cache for a system prompt is identical every time it's sent. The KV cache for an uploaded document is identical every time it's referenced. But the model recomputes the same understanding from scratch every single time.

It's like having to re-read a textbook from page one every time someone asks a follow-up question about chapter seven.

## Prefix Caching Has a Hard Ceiling

The industry has noticed this waste, so prompt caching was introduced: if two consecutive requests share the same prefix tokens, the provider saves the KV cache from the first request and reuses it directly for the second.

Anthropic's implementation cuts the cost of cached input tokens by 90%, and achieves a 60-85% hit rate for stable workloads. For teams with stable system prompts and tool definitions, this is the highest-leverage optimization available today.

But prefix matching is byte-level exact. Change one character in the cached region, and it's a complete cache miss.

Three common scenarios hit this wall immediately:

- **Multi-document RAG**: You cache document A and document B separately, then a new query needs both. B's cached state becomes invalid, because it was computed without knowing A existed.

- **Changed document order**: The same three documents appear in a different order, and every permutation is a miss.

- **Growing conversation history**: Every new round of conversation changes the full context after the prefix.

Alibaba Cloud production data confirms this limitation: 10% of KV cache blocks serve 77% of cache hits. Most cached content never gets reused.

## The Second Problem: Caching Slows Down Inference

There's another issue that affects all KV cache tools: caching libraries run inside the inference engine's process. Cache operations (storing, loading, moving KV tensors) and inference computation share resources and can't run at the same time. When the engine is busy managing cache, inference stops; when inference is running, cache operations wait.

It's like a chef having to run to the pantry themselves for every dish they make. Cooking and fetching ingredients can't happen at the same time, so the whole kitchen slows down.

Google's TurboQuant compresses cache down to 3 bits per value with zero accuracy loss. But when it runs inside the inference engine process, it causes a 20%+ slowdown in inference. The compression itself is fine, but cramming it into the same process as inference cancels out the benefit.

Cache management and inference service are fundamentally two different types of workloads: one is I/O bound (moving large tensors between GPU, CPU, and storage), the other is compute bound (matrix multiplication on GPU). Stuffing them into the same process is like running a database and a web server in the same thread. When load picks up, they fight each other for resources.

## LMCache: Move Caching Out of the Inference Engine

LMCache is an open source project that runs cache management in an independent process, communicating with the inference engine via shared GPU memory. The engine only sends it a tiny message: "I need these block IDs. All the heavy lifting of moving KV tensors happens in LMCache's own process, and the inference engine never even notices.

It delivers three concrete benefits:

- **No resource contention**: Cache I/O doesn't block inference, and inference doesn't block cache I/O. The 20% throughput loss from in-process optimization disappears.

- **Zero-copy cross-GPU sharing**: Traditional methods require multiple memory copies to share cache between GPUs. LMCache lets multiple GPUs read and write to the same memory region directly.

- **Parallel multi-level loading**: Cached data can be distributed across GPU memory, CPU memory, local SSD, and remote storage. Traditional approaches check each level sequentially and get stuck on the slowest one; LMCache checks all levels at once and streams in parallel from wherever it gets a hit.

The performance difference is substantial. In tests with an H200 GPU + Qwen3-235B + 50 concurrent users, compared to in-process caching, time to first token (TTFT) is 14x faster, decoding is 4x faster, and startup time drops from over 3 minutes to around 30 seconds.

It also makes financial sense. A cached prompt only needs to be reused 2-3 times per week (around 1% hit rate) to break even. For a 1000-node deployment with 10% hit rate, it saves around $29 million over three years.

## CacheBlend: Solve the Prefix Dependency Problem

The LMCache architecture solves the performance side of the problem. The prefix issue still remains: if you cache A and B separately and a query needs both, B's cached state is invalid because it was computed in isolation without knowing A existed. When you stitch the cached states together, the model can't build a joint understanding of them, and cross-document connections are never computed.

CacheBlend (from the LMCache team, winner of Best Paper at EuroSys 2025) observed that in modern transformers, most tokens only primarily attend to their own local context, and only a small fraction of tokens have strong connections to tokens across document boundaries.

CacheBlend only identifies and recomputes that small number of tokens, and reuses the independently cached KV states for everything else. Multi-document queries (the most common type in RAG applications) run 2 to 4 times faster with no loss in quality, and order doesn't matter anymore.

For teams building RAG, multi-document question answering, or multi-context agents, every document in your knowledge base becomes a reusable cached asset, no matter what position it appears in or what other documents it sits next to.

## Production Ready

LMCache isn't just a research prototype. It comes with Prometheus/OpenTelemetry integration (to monitor hit rate and I/O performance), a Kubernetes operator, and CLI debugging tools.

Its fault tolerance design is worth highlighting. If the inference engine crashes, cached data is preserved in CPU and storage layers, so recovery doesn't require a cold start. If LMCache itself crashes, the inference engine falls back to uncached mode and keeps running, then automatically reconnects once the cache process recovers. Neither type of failure takes down the whole system.

LMCache integrates with vLLM, SGLang, and TensorRT-LLM, supports both NVIDIA and AMD GPUs, and is released under the Apache 2.0 open source license.

Repository: https://github.com/LMCache/LMCache

## Caveats and Diverging Opinions

There are differing opinions in the comment section. Some developers point out that this advantage is only significant in high-repeat-traffic scenarios (customer service bots, FAQ systems), and the overhead of cache management may not be worth it for low-frequency queries. Others question that after stuffing static knowledge into the context window, inference cost may not be cheaper than querying a vector database.

These questions are reasonable. The key to CAG is selective caching: only cache static, high-value knowledge that rarely changes. Stuffing everything into the cache will hit the context window limit. Separating cold data (suitable for caching) from hot data (suitable for retrieval) keeps the system reliable.

Content is primarily sourced from Akshay 🚀's thread and article *Your KV Caching Is Broken*, as well as the official LMCache repository. Akshay cites an article by Avi Chawla on infrastructure issues for KV cache compression in his thread, which is also worth reading.