Speak at The Fifth Elephant 2026 Annual Conference

Track: Track 2 – Building & Implementing AI Tools & Agents in Production
Format: Birds of a Feather (BoF) Session

Description

Public benchmarks have helped the AI community measure progress in areas like coding, tool use, retrieval, and reasoning. But many production AI agents operate in domains where no public benchmark exists, such as customer support, incident triage, IT operations, data quality checks, procurement, and finance operations.

These workflows are messy, private, multi-step, and often require tool calls, policy checks, human handoffs, and judgment under ambiguity. Unlike coding agents, where benchmarks such as SWE-bench provide a shared reference point, teams building domain-specific agents often have to create their own evaluation datasets, metrics, rubrics, and judging systems from scratch.

In this BoF session, we will explore how teams can evaluate such agents in a practical and trustworthy way. We will discuss how to define the right unit of evaluation, how to build datasets from production traces or synthetic scenarios, how to evaluate agent trajectories beyond final answers, and whether LLM-as-a-judge can be made reliable enough for production decision-making.

The goal is to have a deep discussion on how we can build AI benchmarks, datasets, and evaluation frameworks for niche domains. In this session, we invite participants to share their learnings and ideas on what “good” evaluation looks like when an AI agent is not just answering a question, but planning, using tools, following policies, taking actions, and deciding how to collaborate with humans.

Discussion Outline

• How do we define evaluation units for agentic workflows: single-turn interactions, full tasks, tool-call traces, or end-to-end resolution?
• How do we build evaluation datasets for private and domain-specific workflows where no public benchmark exists?
• What are the best practices for measuring how an AI agent completes a task?
• How should LLM judges be designed to make them reliable for production use cases?

Takeaways

• A shared understanding of the challenges different domains face in benchmarking and evaluating agentic systems at scale.
• Best practices, learnings, and experiences for building custom evaluation frameworks and benchmarks without relying on existing public benchmarks.
• An understanding of where LLM judges are useful, where they fail, and how to calibrate them against human review.
• Ideas on which aspects of agent evaluation can be standardized or open-sourced across domains.

Who Should Attend?

• ML engineers, data scientists, evaluation engineers, and applied AI engineers building production LLM or agentic systems, or working on measuring AI quality.
• Platform and infrastructure engineers working on AI observability, testing, reliability, and deployment.
• Engineering leaders and product managers trying to justify the cost, reliability, and business value of AI agents.
• Open-source contributors building agent frameworks, evaluation harnesses, workflow automation tools, or developer agents.
• Teams working on customer support automation, incident response, DevOps agents, internal service desks, compliance workflows, data operations, or any domain-specific copilot or agentic systems.