Rootconf
Rootconf

Rootconf topical edition on Databases

Call for Proposals

All submissions

Nimalan Mahadevan

@nimalan1626

When optimizations fail at scale - think outside the box

Submitted May 9, 2026

Session type - select the format for your session: 15-minute talk – focused engineering experience

Summary

At e6data, our query optimizer and operators are specialized for low-latency dashboard queries that scan terabytes of data. In this talk, we’ll look at two scenarios where our existing optimizations fell short, and how we came up with out-of-the-box ideas to target specific query patterns.

In scenario one, Top-N per group queries can be optimized with a custom window operator that maintains Top-K internally without materializing the window. However, on workloads with high cardinality, this approach begins to degrade performance. We’ll discuss how we rewrote the query to achieve the expected performance.

In scenario two, we had the optimal plan but ended up with a large scan feeding a low-cardinality group-by. With a custom operator, we pushed the grouping inside the scan, reducing the data passed out of it.

Takeaways

Performance is never a one-dimensional problem, and even battle-tested optimizations leave room for improvement. Special cases often admit simple solutions that outperform general-purpose optimization algorithms. This talk also offers practical insights into building custom operators and optimization passes and the challenges of getting them to work in production

Target audience

This talk is aimed at database engineers, query optimizer authors, and engineers building or operating analytical query engines. Familiarity with relational query execution (scans, joins, group-by, window functions) and basic optimizer concepts will be helpful, but the scenarios and techniques are presented at a level accessible to anyone who has worked with a columnar engine

Speaker Bio

Nimalan is a Senior engineer in e6data working on the core engine with close to a decade of experience in performance engineering and solving challenging problems.

Slides

https://docs.google.com/presentation/d/1duTGX9nkQyN359JEN30jMFr4e-CVNEmgJ2VN2qKW-Ko

All submissions

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Rootconf
Rootconf
We care about site reliability, cloud costs, security and data privacy