The Fifth Elephant 2026 Winter Edition

Submission type: Talk/session proposal - 30-40 mins.

The session

A language model decoding at 100k tokens scores every key in its cache against the current query to decide which few to attend to. I wanted to know how much of that score a query really needs to see. So I measured it: for seven models and context lengths up to 128k, per query and per channel, how much of the score variance each of the 128 key channels carries, and how the error of the attention output falls as a key is read at 1, 2, 3 or 4 bits per channel. The data has a clear shape. For a given query, score variance concentrates in a few dozen channels, and which channels differ from query to query. Quantization error falls by four with every bit, so the first bit of a heavy channel is worth more than the fourth bit of a light one, and a water-filling rule over the measured variances says how many bits to read where. Reading 46 to 74 bits per key that way reaches the accuracy that fixed-depth methods reach at 136. Models with query-key normalisation behave differently from models without it, and that difference shows up as a choice of basis.

The second half of the talk is about what the measurements did not show, and how I found out. Calibrating the channel statistics on Wikipedia text or on the coding-agent transcripts the model actually decodes changes error by under 10 percent, so the domain of the calibration set barely matters. The measure I built to see the effect downstream, agreement of a coding agent’s next step with exact top-k decoding over 60 real sessions of 80k to 100k tokens, is flat at a 0.5 percent attention budget and separates methods only at 2 percent, which says the budget, not the scan, decides what the agent writes at small k. And the whole thing stops paying once the index moves back into GPU memory. Every number is regenerated from public JSON files by two scripts, and I will show the failures and the unexplained bits alongside the findings.

Outline (35 minutes plus questions)

Part 1, 5 min. The question: how many bits of a key does a query need, and why the answer is a data question before it is an engineering one.

Part 2, 10 min. The measurements: per-channel score variance across seven models and context lengths; error versus bits per channel; the four-per-bit law and the water-filling rule it implies; the worked example on six keys.

Part 3, 8 min. What changes across models: query-key normalisation and the basis choice; what does not change: calibration domain.

Part 4, 8 min. Seeing the effect downstream: synthetic retrieval tasks versus real coding-agent sessions, why a metric with a standard error of 0.05 sees nothing at a 0.5 percent budget, and what it sees at 2 percent.

Part 5, 4 min. Where the finding pays and where it does not, and the open questions.

Takeaways

First, a way to think about how much information a query needs from each key, in bits, with measurements you can rerun.

Second, two negative results about where the accuracy of sparse attention does not come from: calibration domain and, at small budgets, the scan itself.

Third, a worked case of an evaluation metric that fails to see an effect, and what it took to notice.

Audiences

Data practitioners and ML researchers who work with large models and want a measurement-first treatment of an efficiency question, and anyone interested in how an evaluation metric can fail to see an effect.

Bio

Vivek Kalyanarangan is Senior Technical Architect, AI at IDfy, where a 20-person team operates more than 40 production ML models across biometric authentication, document recognition and OCR, fraud detection and large-scale NLP. He has 13+ years across analytics, big data and deep learning. He is the author of Quantization and Fast Inference (Manning, 2026) and the freeCodeCamp course LLMs from Scratch, and of the paper this session is drawn from: Fathom, arXiv:2609.17652 (September 2026), code and all measurements at https://github.com/vivekkalyanarangan30/fathom.

Draft slides

Not yet prepared; the paper (https://arxiv.org/abs/2609.17652) has every figure the session would use. Slides on request before the editorial call.

What I do not know yet

Step agreement with exact top-k decoding has a standard error of about 0.05 over 40 sessions and does not separate methods at a 0.5 percent budget. I want a better downstream measure of whether the sparse read changed what the agent did, and collaborators who can run end-to-end task success with test execution.

Match-making tags

I can help with: technique, experience, critique. I need help with: technique, collaborator, critique. I would like to meet: researchers, practitioners, dataset owners.

Topic tags

evaluation, large language models, efficiency