Decoding at a million tokens when the KV cache lives in host memory: what actually bounds the step

One-line summary. A measured account of a full decode step on an A100 with the KV cache and its index in pinned host memory, from 32k to one million tokens, across six index designs: which component bounds the step, how to time it honestly, and the one design change that made it 1.67x faster.

What problem are you addressing?

Agent sessions now run to hundreds of thousands of tokens, and a server holds many of them at once. Their KV caches stop fitting beside the weights in GPU memory and move to host RAM, which is what vLLM’s CPU offload, LMCache and similar layers do. At that point a decode step is bounded by what crosses PCIe, and the largest stream is not the model weights or the selected KV rows but the index scan that ranks all n keys to pick the top-k. For Qwen3-8B the 136-bit-per-token index that Double Sparsity, Loki and SparQ read is 160 MB per step at 32k and 5.1 GB at one million tokens, while the selected rows stay near 200 MB.

I measured this regime end to end and then changed the index. Fathom stores the 4-bit K cache channel-major as bit planes and lets each query read only the planes it needs, decided per channel by a water-filling rule. At one million tokens the step is 1.67x faster in GPU time than with the 136-bit scans and 2.5x faster than with a block-landmark index. The same design is not faster once the index is back in HBM, and the talk spends time on why, because that negative result decides whether it belongs in a given stack.

What made it difficult: GPU time and wall-clock disagree by 7 to 25 percent in a Python harness and the gap differs by method; a gather of contiguous runs from pinned memory hits the link rate at 4 KB granularity while one copy call per run gets 0.4 GB/s; overlapping transfer with compute bought nothing because both gathers already ran at link rate; and a bit-extraction kernel does two to four times the integer work per byte of a nibble scan, which is invisible over PCIe and decisive in HBM.

Who is the intended audience?

Infrastructure and platform engineers who run or build LLM serving with long contexts or KV offload, and SREs who have to explain a latency number. Familiarity with attention and the GPU memory hierarchy is enough; no research background needed.

What will the audience learn?

Instrumentation. How to instrument an offloaded decode step (CUDA graphs, profiler traces, fixed-size run lists, bit-exact checks before timing) and which number to report.

Costs. A per-component cost table for six index designs at 256k to 1M tokens, reproducible from a public repository.

A rule. When a read-fewer-bits index pays off (index in host memory) and when it does not (index in HBM), with the arithmetic-per-byte argument that makes it portable across GPUs.

Adoption. What such a store costs in memory (68 B per token per KV head) and what an LMCache or vLLM connector would need to adopt it.

Status of the work

Experimental, measured on rented A100 pods. Published as arXiv:2609.17652 (September 2026); code, every result file and the run chains are public at https://github.com/vivekkalyanarangan30/fathom. Not yet integrated into a production serving stack; the talk is explicit about that boundary.

Outline (40 minutes)

Part 1, 5 min. The regime: sizes at 32k and 1M for an 8B model, what fits where, what an offload engine actually moves per step.

Part 2, 10 min. Measuring a decode step honestly: GPU time vs wall-clock, the issue-cost trap, sync-free accounting.

Part 3, 10 min. Bytes are time over PCIe: the trace breakdown at 1M, six designs at the same link rate, where the 1.67x comes from.

Part 4, 5 min. What did not help: pipelining, launch-config sweeps, alternative top-k, and the HBM control where nothing is faster.

Part 5, 5 min. Fitting it into a serving stack. Then Q&A, 5 min.

Speaker

Vivek Kalyanarangan is Senior Technical Architect, AI at IDfy, where a 20-person team operates more than 40 production ML models across biometric authentication, document recognition and OCR, fraud detection and large-scale NLP. He has 13+ years across analytics, big data and deep learning. He is the author of Quantization and Fast Inference (Manning, 2026) and the freeCodeCamp course LLMs from Scratch, and of the Fathom paper this talk is drawn from.

Paper: https://arxiv.org/abs/2609.17652
Code and measurements: https://github.com/vivekkalyanarangan30/fathom