The Fifth Elephant 2026 Annual Conference

Abstract / Session Description:
At Razorpay, our lakehouse platform ingests over 6 billion events daily and powers a reporting platform that generates close to a million reports every month. As scale grew, full-refresh denormalization became unsustainable: joins across 10-30 entities consumed heavy compute, report freshness lagged by up to 48 hours, and highly mutating datasets made derived tables expensive to keep correct.

In this talk, we’ll share how we moved from full refreshes to an incremental serving model for reporting. Instead of rebuilding tables, we identify impacted records, use secondary indexes to locate affected entities, and traverse relationships across facts and dimensions using graphs to propagate changes.

We’ll cover the hard cases this framework handles, including dimension-side updates, backdated foreign-key changes, and out-of-order events. We’ll also discuss how Apache Iceberg features such as table layout optimization, metadata pruning, merge tuning, and compaction strategies helped make this practical on our lakehouse platform.

This reduced denormalization compute cost and generation time by over 85%, while significantly improving freshness and scalability.

Who Is This For?
This session is for data platform engineers, lakehouse architects, analytics infrastructure teams, and backend engineers who build or operate reporting, CDC, denormalization, or derived-data systems over large, highly mutating datasets.
It will be especially useful for teams that have outgrown full-refresh pipelines, are struggling with freshness and recomputation cost, or are trying to serve reporting workloads directly from a lakehouse architecture.

Key Takeaways
Why full-refresh denormalization breaks down at reporting scale.
How to identify and process only impacted records instead of rebuilding derived tables.
How secondary indexes and entity-relationship traversal can support incremental propagation.
How to handle dimension-side updates, backdated foreign-key changes, and out-of-order events.
What storage and execution tradeoffs matter when serving reporting workloads from a lakehouse.
How Razorpay reduced denormalization compute cost and generation time by over 85%.

Draft slides - Link

Bio:
Amit Prabhu,
Staff Software Engineer, Razorpay
Linkedin