Description

Many teams now have traces, logs, evaluations, user feedback, and dashboards for their AI agents. The harder question is what those signals should prove, and what teams should change after seeing them.

This Birds of a Feather session is a practitioner discussion on agent quality loops. When an agent gets stuck, retries a tool, burns tokens, calls the wrong interface, produces weak evidence, or returns a plausible but incomplete answer, how do we diagnose the failure? Is it a model problem, a prompt problem, a tool-contract problem, a context problem, an orchestration problem, or an evaluation problem?

We will focus on the engineering loop around agents: observe the turns, classify the failure, decide the owner, change the system, and measure again. Participants will compare what they instrument today, which trace signals are actually useful, and where current agent observability still fails to explain behavior.

This is not a talk, panel, or product demo. The talk submission shares one grounded system story. This BoF asks the room to build a shared map: what should agent quality loops contain, what teams are actually building, and what patterns should become common practice for production agent systems?

Discussion Outline

• What should an agent quality loop prove beyond “the agent called tools”?
• Which failures are easiest or hardest to diagnose from traces?
• How do teams separate model failures from tool, context, schema, skill, or orchestration failures?
• Which signals actually change product decisions: retries, error rates, latency gaps, evidence quality, cost, answer quality, or user feedback?
• What should a reusable agent quality loop look like in 2026?
• Where should loop engineering live: platform engineering, product engineering, ML infrastructure, observability, or evaluation teams?

Takeaways

• A shared vocabulary for diagnosing agent failures from traces and evaluations.
• Practical patterns and lessons from teams building production AI agents.
• A clearer understanding of what belongs in prompts, tools, skills, context, evaluations, orchestration, and product surfaces.
• A starting point for treating agent quality loops as a production engineering discipline.

Who Should Attend?

Practitioners building or operating AI agents, including:

• AI engineers
• Platform engineers
• Data engineers
• Product engineers
• Developer Experience (DevEx) teams
• Observability engineers
• Technical leaders responsible for agent reliability, evaluations, tools, MCP servers, workflow agents, or copilots

Bio

Prateek Mandloi works on agentic AI systems, enterprise work graphs, CLI and tooling interfaces, and benchmark-driven quality loops for AI agents. His current focus is on how agents use tools in real workflows: what they call, where they retry, how they recover, what evidence they use, and how those traces can drive better tool, skill, and context design.

He previously hosted an accepted Birds of a Feather session at The Fifth Elephant 2024, bringing practitioners together around enterprise data lifecycle and AI analytics.