Speak at The Fifth Elephant 2026 Annual Conference

Description

Agents are often discussed as if the model is the whole system. In practice, production
agents behave more like running programs: they observe, decide, call tools, read outputs,
recover from errors, and continue. If that loop is the program, the trace is its stack trace.
This talk is a concrete engineering story about using turn-level traces to improve agent
quality. I will show how we run the same real-work scenarios across the same agent with
different tool surfaces, then inspect what actually happened: tool calls, command errors,
retries, latency, token usage, final answers, and judged quality. The goal is not to rank
models. The goal is to understand where the agent loop breaks.
The surprising lesson is that many failures are not bad reasoning. They are bad interfaces:
unclear tool names, brittle command contracts, noisy outputs, missing context, and errors
the agent cannot recover from. Once traces make those failures visible, they become
product work: better help examples, more tolerant read commands, compact output modes,
clearer repair messages, narrower skills, and context surfaces agents can actually use.
I will close with the engineering loop this enables: trace the agent, classify the friction,
redesign the tool, skill, and context contract, rerun the benchmark, and measure whether
retries, errors, latency, cost, and weak answers actually went down. Agent quality becomes
an engineering loop, not a vibe check.

Takeaway

● A practical way to think about agents as running programs whose traces can be debugged.
● How turn-level benchmarks reveal failures in tool contracts, context design, skills, and output
shapes.
● How to separate answer quality from loop quality: an agent can eventually answer correctly
and still expose serious system friction.

● A reusable quality loop for production agents: observe, classify, redesign, rerun, compare.

Who should attend?
Engineers, AI platform teams, data/platform engineers, product engineers, and technical
leaders building production agents, MCP-style tools, internal copilots, coding agents, or
agent evaluation systems. The talk assumes familiarity with LLM-based tools and agents,
but not with any specific framework.

Bio

Prateek Mandloi works at the intersection of AI agents, enterprise work systems, developer
tooling, and evaluation infrastructure. His recent work focuses on making agent systems
measurable and improvable: building CLI and tool surfaces that agents can use reliably,
designing skills and context layers around real workflows, and constructing benchmark
harnesses that reveal where agents lose time, retry, fail, or produce weak evidence. He
previously hosted an accepted Birds of Feather session at The Fifth Elephant 2024 on
enterprise-ready data lifecycle for AI analytics.

{Add the link to draft slides - PDF/PPT - with comments access}

{Add the link to 2-min elevator pitch video}