The Fifth Elephant 2025 Annual Conference CfP

Talk Description

There is an explosion of techniques and methods to harness the intelligence of generative AI into practical use cases, such as converting natural languages to SQL queries for analysts. In this talk, I attempt to separate out the chaff from the wheat by systematically applying these techniques to a text-to-SQL harness and quantifying techniques worth applying in production use-cases for real world data.

This talk will start from a baseline LLM implementation with 40% accuracy against BIRD-bench and walk through the evolution of this harness to get it to 65%, through an exploration of in-context learning techniques such as M-schema, correction steps, prompting changes and more to give a feel of what systematically improving a product looks like.

I will present our findings from adapting this harness against our hand-curated IPL dataset, and show the process of improving user queries iteratively. Here, I will focus on creating an evaluation set, which enables this iterative improvement along with techniques to improve domain knowledge.

Finally, I will end with a demo of our open source project PixieQL, which aims to take all the learnings from this exploration and enable data analysts to iteratively build queries and add domain understanding to the system.

Takeaways:

• How to iterate for increasing accuracy in LLM based applications.
• Designing different evaluation datasets and when to use them.
• Real data > Generated data

Audience

• Engineers who want to increase accuracy and robustness for their LLM applications
• Data analysts looking for an easier workflow

Bio

A backend engineer and an AI explorer, Sezal has worked in large scale logistics, created data pipelines, orchestration engines and set up feature stores, MLOps. Currently, she is obsessed with AI Evals.

An IIT grad, she went on to work in robotics at Carnegie Mellon, and led a healthcare startup tech team before joining nilenso.

Reference Links:

Crickit (ask questions about IPL): http://crickit.nilenso.com/

PixieQL: https://github.com/nilenso/pixieQL

Blog: https://blog.nilenso.com/blog/2025/04/30/exploring-text-to-sql/

Talk content outline: https://www.notion.so/nilenso-software/5thEL-Talk-Notes-Text-to-SQL-beyond-POC-2110f0425dae808aac5ae337e1d1704f?source=copy_link