IMPORTANT: To view this page as Markdown, append `.md` to the URL. For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL.

KV cache offloading

KV cache offloading is the process of moving attention key/value data from GPU memory to lower-cost storage like CPU memory or disk. It frees up GPU resources while preserving the ability to resume inference without recomputation. This helps scale LLM workloads efficiently by balancing performance and memory usage.

Why does KV cache become a bottleneck in LLM inference?​

LLMs rely heavily on the KV cache to speed up inference. The cache stores attention keys and values for every token in the input sequence, allowing the model to reuse them in future steps instead of recalculating them. Although this saves a significant amount of compute resources and delivers faster inference, it comes with a steep memory cost.

As context windows increase, the KV cache size grows linearly with sequence length. This can quickly exhaust available GPU memory, especially in long-context scenarios. Since GPU memory is limited, the KV cache often becomes a bottleneck for running applications that require extended context.

In fact, not all KV cache data needs to stay in GPU memory at all times. In many real-world applications, users may not interact with the LLM continuously. For example, a user might pause while typing or leave and return hours later. In such cases, their KV cache remains in GPU memory, even though it’s not actively being used. Similarly, when multiple users/agents access the same conversation, document, or session at different times, the same KV cache might sit idle on the GPU between interactions (and you don't want to waste GPU resources just for recalculation of the same content).

This results in inefficient memory usage, as valuable GPU memory is tied up by inactive sessions instead of being used to serve new requests. Over time, this limits how many concurrent users the system can support and reduces overall throughput.

To solve these problems, KV cache offloading moves inactive or less frequently accessed cache data from GPU memory to lower-cost, higher-capacity storage such as CPU RAM, local SSDs, or remote object storage. When a user resumes interaction or another user accesses the same content, the cache can be reloaded into GPU memory on demand. This avoids costly recomputation while freeing up GPU resources for active workloads.

How to calculate the KV cache size​

When offloading the KV cache, it’s useful to understand how much memory it actually consumes.

In transformer-based LLMs, each attention layer needs to store two vectors (a key and a value) for every token in the input sequence. Each layer contains multiple attention heads, and all heads typically have the same dimension.

To estimate how much memory the KV cache consumes, use the following calculator:

KV Cache Memory Calculator
Calculate the memory required for KV cache in transformer-based LLMs
Model Parameters
Batch Size (B)
Sequence Length (S)
Common values: 1024, 2048, 4096, 8192
Number of Layers (L)
Attention Heads (H)
Head Dimension (D)
Bit Precision (Q)
KV Cache Size: 0.5 GB
This is the estimated memory required for storing the key and value vectors for all tokens in the sequence.
Formula:
KV Cache Size (GB) = 2 × B × S × L × H × D × (Q / 8) / (1024^3)
= 2 × 1 × 1024 × 32 × 32 × 128 × (16 / 8) / (1024^3) = 0.50 GB
Note:
The KV cache size grows linearly with sequence length. For long conversations or documents, this can become a significant memory bottleneck.

When should you offload the KV cache for LLMs?​

KV cache offloading is especially useful when:

  • You’re deploying LLMs with long context windows, which can cause the KV cache to quickly exceed GPU memory.
  • Multiple users or agents need to interact with the same underlying content or context across sessions. For example, developers working in an IDE with LLM integration often interact with the same code snippet repeatedly.
  • Your deployment is memory-constrained or you need to optimize for infrastructure cost.
  • You’re scaling inference across many distributed workers where GPU resources are limited.
  • Your workloads include intermittent or idle user sessions, where keeping the KV cache in GPU memory would be wasteful.

Benefits of KV cache offloading​

Offloading the KV cache offers several important advantages for scaling and optimizing LLM inference:

  • Better resource utilization. By moving inactive or shared KV data out of GPU memory, you can free up space for new requests. This allows the same GPU to serve more concurrent users or longer input sequences without hitting memory limits.
  • Lower compute costs. GPU memory is expensive and limited. Offloading allows workloads to take advantage of cheaper storage (e.g., CPU RAM or disk), reducing the need to over-provision high-end GPUs just to manage cache.
  • Reduced latency: Offloading allows the model to skip redundant KV computations during inference, especially for overlapping context in multi-turn interactions. This significantly reduces TTFT and overall latency. NVIDIA reports that KV cache offloading can deliver up to 14× faster TTFT for large input sequences compared to recalculating the KV cache from scratch.

Trade-offs in KV cache offloading​

While KV cache offloading can significantly improve memory efficiency and throughput, the speed of the offloading target is critical. If the storage tier (e.g., CPU RAM or disk) is too slow, the overhead of transferring KV data back to the GPU may negate the benefits, especially in latency-sensitive applications.

Make sure the cost of transferring data is lower than recomputing the cache from scratch. This is often the case in long, multi-turn conversations, where reusing previous context is crucial and recomputation would be expensive.

There is also a quality trade-off when the system uses selective KV offloading. During decoding, the runtime may need to decide which keys and values should return to the GPU. If it misses important context tokens, the model can produce worse answers. This risk is high in context-intensive workloads such as multi-document QA, legal review, and codebase reasoning, where many details from the prompt may matter.

This paper highlights the problem: some KV offloading methods perform well on common long-context benchmarks but degrade on tasks that require retrieving many facts from the prompt. The practical lesson is that long context length and context intensity are different things. Before enabling selective KV offloading in production, compare it with a full-attention baseline on tasks that match your workload. Track answer quality alongside TTFT, TPOT, throughput, GPU memory usage, and host-to-device transfer.

How to offload the KV cache​

You almost certainly don't need to build the implementation yourself. Every major inference framework now supports KV cache offloading as a built-in feature, and a second layer of dedicated KV cache systems exists for the cases the framework can't cover on its own.

Built-in offloading in your inference framework​

vLLM, SGLang, and MAX all support offloading to host memory and disk with no extra infrastructure. The mechanism is similar: they extend the existing prefix cache so that completed blocks are demoted to a larger, slower tier, then promoted back to the GPU when a later request hits them. Transfers can run asynchronously on DMA (Direct Memory Access) copy engines, so they overlap with model execution rather than stalling it.

Here are some examples:

max serve --model meta-llama/Llama-3.1-8B-Instruct \
  --kv-connector-config '{"type": "rust_tiered"}'

MAX calls this layer a KV connector. The rust_tiered connector tiers evicted blocks across your host memory and local disk. Override the budgets explicitly when you want to:

max serve --model meta-llama/Llama-3.1-8B-Instruct \
  --kv-connector-config '{
    "type": "rust_tiered",
    "host_offload_max_gb": 128,
    "disk_offload_dir": "/mnt/kv_cache",
    "disk_offload_max_gb": 512
  }'

Dedicated KV cache systems​

In addition to framework-native offloading implementations, you can add a dedicated KV cache layer for extended features:

  • LMCache is a vendor-neutral KV cache management layer designed to optimize LLM inference by reducing TTFT and increasing throughput, especially for long-context workloads. It supports persistent, tiered KV cache offloading to various storage backends including CPU RAM, local disk, Redis, Valkey, Mooncake, and InfiniStore.
  • Mooncake Store is a distributed KV cache storage engine designed specifically for LLM inference. It enables inference engines to store, retrieve, and transfer KV caches across GPUs, nodes, and instances, supporting prefill-decode disaggregation and improving cache reuse. It is widely integrated with systems such as vLLM, SGLang, and LMCache.
  • NVIDIA Dynamo KVBM (KV Block Manager) is a unified memory layer to handle memory allocation, management, and remote sharing of KV blocks for inference tasks across heterogeneous and distributed environments.

Whichever tool you choose, benchmark it with your workloads and measure cache hit rate per tier alongside TTFT and throughput. A correctly configured host tier should show a high hit rate and improved TTFT, while a disk tier that hits rarely is mostly paying transfer cost for nothing.

Additional resources​

Edit this page