Rootconf 2026 Annual Conference

Session title

Why LLM citations break across multiple retrievers & llm hops: the one architectural rule to fix attribution and unlock precise groundedness evals

One-line summary

Multiple retrievers each number chunks S1, S2, S3. After a later LLM merge, those numbers no longer mean the same document — and if you let the model type URLs instead, you cannot eval at chunk grain. We built a request-scoped gid registry so the model only emits handles, and we grade each claim against the chunk it actually cited. The mechanism is simple; skipping it is how grounded-looking answers ship ungrounded.

What problem are you addressing?

Citation guides assume one retriever and one LLM call. Our production path is not that: several retrievers fan out on the same question, an LLM consolidates, another LLM writes the customer-facing document, then we render PDF.

Prior assumption: every chunk has URLs, so generate with [S1] and map back.

Approach 1 — local IDs. Each retriever labels chunks S1..Sn (SIds). After generation, map S1 to a URL. That mapping is wrong as soon as two retrievers reuse the same local index for different text.

Possible workaround one might think of :
Why not using different Prefixing IDs per retriever
i.e :
Retriever 1 uses chunk ids of form SIds i.e. S1, S2, S3..
Retriever 2 uses chunk ids of form HIds i.e. H1, H2, H3..
Retriver 3 uses chunk ids of form Dids i,e D1, D2, D3..
....
Retriever N

looks like a fix right?
this works until a source/retriever you don’t own shows up and ignores the convention. In distributed systems, it is highly impossible for one team to own all the retrivers. So every team that owns and exposes a retriever must follow our above convention - else plugging in that retriever breaks our contract for citation generation.

Approach 2 — let the LLM type URLs.
It would work, but with the following issues.
Problems:
Invalid or drifted links. Worse for eval: one URL is many chunks, so “did this URL support the claim?” is the wrong question. The excerpt came from a chunk from this pool while only one or a subset of those chuks were used by llm in presenting the fact/exceprt.

Approach 3 — what we shipped. Ingest chunks from all retrievers on this request, de-dupe, assign a global chunk id (gid) plus gid→URLs, rewrite local S# to gid, ask the LLM for gids only, expand URLs after generation. Evals compare claim text to gid chunk text.

Evals

A footnote can be a legal retrieved pointer and still cite the wrong evidence. We separate the evaluation into a three-stage diagnostic funnel so a failure pinpoints the exact breakdown:

1. Identity (Deterministic — Pointer Legality): Is the cited handle in the retrieved pool for this request?
   • Failure Mode: Invented or Corrupted Pointer (downstream LLM forged an ID like [S99] or mangled formatting).
2. Support (LLM Judge — Evidence Existence): Does the cited chunk actually contain the factual assertion made in the claim?
   • Failure Mode:
     • Misattribution: The model found the fact in Chunk A, but mistakenly stamped Chunk B’s index on it during consolidation.
     • Fabrication / Hallucination: The model invented a claim from whole cloth and attached an innocent valid handle to mimic grounding.
3. Faithfulness (LLM Judge — Factual Mutation & Drift): Given that the cited chunk does contain the relevant evidence, did the model represent it accurately?
   • Failure Mode: Factual Mutation / Extrapolative Hallucination (e.g., source says “revenue grew 10%”, but the model writes “revenue grew 25%”, or flips conditionality like “tested on Linux” into “tested on Linux and Windows”).
   • Rule: We skip this check when Support fails (N/A)—a model cannot be faithful to evidence it never cited.

Example from Production Run:

On an on-demand section evaluation of a generated customer report, our metrics exposed this exact split:

• Identity: 26/26 (100%) — All 26 emitted citations mapped to valid retrieved chunks.
• Support: 15/21 (71%) — Out of 21 distinct claims, 6 failed: the cited chunks contained zero evidence for the generated assertion (including claiming a product capability the cited chunk never states).
• Faithfulness: 13/15 (87%) — Evaluated strictly on the 15 supported claims; 2 suffered factual mutation or number drift.

The Operational Takeaway

A standard source-list or URL-matching dashboard would have marked this document 100% grounded based on valid links alone.

Our pass-through registry turns citation into three distinct operational guarantees:

• A Guardrail: The model emits opaque handles, making invented source URLs impossible by construction.
• Governance: Citation identity survives multi-retriever merges and downstream LLM hops without losing chunk binding.
• A Diagnostic Check: It distinguishes pointer corruption (Identity) from misattribution & fabrication/hallucination (Support) and factual mutation (Faithfulness).

Who is the intended audience?

Platform / MLOps / applied-AI engineers shipping RAG or agents with more than one retrieval source or more than one generation step. Engineering leads who have to answer whether generated text can go in front of a customer.

Level

Intermediate

List one or two practical takeaways.

1. Don’t namespace retriever IDs and don’t let the model type URLs. Assign a request-scoped gid, have the LLM emit only that handle, expand URLs after generation. Simple, and we treat it as mandatory once you have more than one retriever or more than one LLM hop.
2. Persist the chunk text. Grade identity / support / faithfulness at chunk index so a complete source list cannot hide “real link, wrong paragraph.”

What will you share?

• The three approaches, and why 1 and 2 die.
• How gids flow: retrieve → normalize → consolidate → render.
• The three evals: definitions and why are they a must.
• Trade-offs: build vs buy, chunk-boundary duplicates, dropping unknown handles, evals on-demand not on the hot path.
• Live walkthrough: generate → carry → judge on one section. Not a product pitch.

What is your experience with this problem?

• Production system
• Internal engineering project
• Hard-earned engineering lesson

What approaches failed, disappointed, or created unexpected problems?

• Local S# across retrievers: same label, different chunks, wrong URL after merge.
• Convention on the retriever: not scalable when you don’t own the next one.
• Model-typed URLs: hallucinated/broken links; URL→chunk for eval is many-to-one, so support looks better than it is.
• URLs in the prompt: summary/merged nodes can carry many URLs per chunk; that’s token cost and worse reasoning for a mapping the backend already has.

What will you do differently today?

Treat the LLM as untrusted for citations. It only emits opaque handles against a registry built from what was retrieved on this request. URLs are a render step. Identity holds by construction; support and faithfulness are still judged, because a valid gid can still point at the wrong evidence.

What trade-offs did you consider?

• Build vs buy: we looked for a product, library, or SDK that (1) works on a self-hosted model with no vendor citation API, (2) keeps citation identity across multiple unowned retrievers and later LLM hops, and (3) evals at chunk grain, not URL grain. Hosted citation APIs assume one model call and one retrieved set. Attribution libraries score a single-shot answer. Nothing we found did pass-through identity plus claim-vs-chunk eval on this shape, so we built a small registry rather than wrap an incomplete buy.
• Dedup vs exact evidence: identity includes retriever-local id + content, so identical boilerplate in two docs stays two gids. We would rather attribute the document the model actually saw than merge by text.
• Chunk boundaries: different retrievers chunk different spans; we don’t regex-merge them. Some duplicate context in the consolidator is better than a brittle aligner.
• Unknown handle: registry drops it. The sentence stays, uncited. We preferred an obvious hole over a forged footnote.
• The “Eval (Latency vs. Cost)”": Running multi-judge LLM evals (Identity, Support, Faithfulness) synchronously across dozens of sub-queries adds unacceptable user latency. We scoped quick evals to section-level UI triggers, while extending the future scope to include full document citation eval to with help of asynchronous background queue architecture.

How can this help other practitioners?

• Steal the pattern: gid pass-through, model never types URLs — simple, and a must if citations have to survive more than one hop.
• Don’t score grounding at rendered-URL grain.
• Use identity vs support vs faithfulness to split “invalid pointer” from “misattribution/ hallucination” from “misread the chunk.(unfaithful)”

Current state

Production experience

Slides

https://docs.google.com/presentation/d/1CJXW6bZM4YEzhS6Sns6L3OaWUldmh_nZW8oICgU3vjA/edit?usp=drivesdk

Tags

#governance #observability #platformengineering #mlops #casestudy #failurestory #demo

Bio

I am Sai Khalandar working at Nutanix as Software Developer in the SaaS Organization.