Inside a bit-plane KV index: superblock layout, per-query water-filling, and why the kernel is slower in HBM

Format. 45-minute internals talk with a live walk through the code (kernels/bitplane.py in the public repository).

What the talk is about

An internals talk on one data structure. Fathom stores a language model’s 4-bit key cache channel-major as bit planes: bit p of 64 tokens’ codes is one 64-bit word, and reading the first t planes of a channel is exactly that channel’s t-bit quantizer with the same block scale, so a prefix read is exact and needs no second copy of the data. On top of that layout each query runs a reverse water-filling rule that decides, per channel, how many planes to read for a bit budget; the water line is found by 30 bisection steps and the whole plan runs as one Triton kernel per query.

I will go through the layout decisions and their consequences. The naive plane order puts one channel’s four planes in one 32-byte sector, which makes a 1-plane read cost the same DRAM traffic as a 4-plane read; a superblock order over 1024 tokens fixes that and makes every (channel, plane) unit 128 contiguous bytes, which is also what lets a gather from pinned host memory run at PCIe link rate at 4 KB granularity. The Triton scan kernel extracts one bit per shift-and-mask and applies G multiply-adds per extracted bit, so in HBM it reaches a fraction of the bandwidth a nibble scan reaches and is 1.4x slower despite reading 38 percent fewer bytes. I will show the integer-operations-per-byte arithmetic that says this is a property of the operation and not of the A100. Over PCIe the arithmetic is idle and the byte ratio becomes the time ratio, which is where the design wins: 1.67x faster decode steps at one million tokens with the index in host memory.

I will also cover the parts that did not work: a tensor-core dot variant that was slower and inexact, two-stage and reduced-precision top-k, launch-config sweeps, and overlapping transfer with compute on a second stream.

Outline

Part 1, 10 min. The store: codes, block scales, planes, and the exactness property of a prefix read.

Part 2, 8 min. Memory order: sectors, superblocks, and the gather that hits link rate.

Part 3, 8 min. The plan: per-query water-filling in a Triton kernel, the bisection, what the plans look like across layers.

Part 4, 10 min. The scan kernel: bit-extraction cost, achieved bandwidth in HBM against a nibble scan, the ops-per-byte model.

Part 5, 5 min. Failed variants and the honest scoreboard. Then Q&A, 4 min.

Audience

Engineers who like memory layouts and kernels. Knowing what attention does is enough; the talk builds the rest.

Status and links

Published as arXiv:2609.17652 (September 2026). Code, every measurement and the run chains: https://github.com/vivekkalyanarangan30/fathom

Speaker

Vivek Kalyanarangan is Senior Technical Architect, AI at IDfy, where a 20-person team operates more than 40 production ML models across biometric authentication, document recognition and OCR, fraud detection and large-scale NLP. He has 13+ years across analytics, big data and deep learning. He is the author of Quantization and Fast Inference (Manning, 2026) and the freeCodeCamp course LLMs from Scratch, and of the Fathom paper this talk is drawn from.