The Fifth Elephant 2026 Winter Edition

Submission type: Talk/session proposal - 30-40 mins

An AI agent can write SQL, draw charts, and tell you why sales fell. But how do you know it was right? In a real company, we often don’t know the true cause ourselves. So I started building one where I would know. Chowk is a fictional Indian e-commerce marketplace with sellers, payments, festive sales, experiments, and messy data. I designed it with two views: what really happened, and what an analyst could have known at the time. The idea was to use this known truth to test AI analysts. But the first things to fail were my own tests. I had rotated product categories through different sales to make the business calendar realistic. It looked fine, until a check found zero valid comparisons between sales in the same category under similar conditions. Questions like “Did this sale do better than the last one?” had no fair answer. I also found that a year of historical data I’d added for seller analysis lacked the facts needed to answer those questions. Both designs looked reasonable. Neither could support the analyses I wanted to test.

In this talk, I’ll walk through these mistakes and investigate a question from Chowk: why did checkout conversion fall after a sale? I’ll show what the analyst could see, what actually happened, and why a plausible answer can still be wrong. I’ll also test a strong AI agent on a small set of Chowk questions, including cases it gets right and cases where the evidence isn’t enough. Sometimes the best answer is “we can’t tell yet.” I’ll close with how these lessons are shaping Neri, an open-source tool I’m building to help analysts work with AI without giving up their judgment.

Takeaways

1. With analytical benchmarks, test what the analyst could know, not just what the simulator knows. Before judging an AI’s answer, look at the evidence available at the time. If the data doesn’t support a definite answer, a good test should reward honest uncertainty, not confident guessing.

2. Test the test before testing the AI. A dataset can look realistic while the questions built on it are impossible to answer. Check that the necessary data and comparisons exist, and that wrong answers actually receive lower scores.

Which audiences will benefit most from your session?
Analysts and data scientists who use AI for analysis, people building or evaluating AI data agents, and anyone creating synthetic data for testing or teaching. The examples come from everyday business problems. No ML background is needed.

Bio
Kruthika Kumar has spent 14 years working in data and analytics, most recently as Director of Analytics at Razorpay. His work has covered product/business analytics, experimentation, data platforms, and AI-assisted analytics. He is now working on multiple projects, including building Neri, an open-source Analyst Control Centre that helps analysts work with AI while keeping control of their methods and decisions, and Chowk, the fictional marketplace used to test it.

Links
Not yet ready.

What I don’t know yet / what I want help with

1. How do I know Chowk’s problems are realistic? I designed both the company and the mistakes, so I could end up testing my own assumptions. I’ve tried to address this with independent reviews, checks written in code, and cases where nothing is wrong. But I’d like to compare Chowk’s problems with real analytical failures people have encountered at work. I’d welcome examples and criticism.

2. Where does a difficult question become an unfair one? In real business analysis, the available data often cannot settle a question. I want to test whether an AI agent recognizes that limit. But how do we reward honest uncertainty without encouraging agents to avoid difficult questions? I don’t have a satisfactory answer yet.

Matchmaking tags
I can help with: ideas, experience, technique, critique
I need help with: experience, dataset, critique
I’d like to meet: practitioners, researchers, domain experts, tool builders

Topic/domain tags
Synthetic data, evaluation, AI-assisted analytics