The Fifth Elephant 2026 Annual Conference
Built for humans. Now rebuilding for agents.
AN
Abhijith Neerkaje
AI evals workshop
Submitted Jun 10, 2026
I am submitting for:
Track 2 - Building & implementing AI tools & agents in production
Type of session:
Hands-on workshop - 2-4 hours
Overview
- Why do Agents make mistakes - 3 Gulfs [Comprehension, Specification and Generalization]. (10 min)
- Challenges of evaluating agent responses . Why is it different from standard software testing on ML system testing (10 min)
- Component wise evaluation of agents (What is equivalent of module level testing in Agents) (30 min)
- How to generate synthetic data to evaluate your agents - Hands on activity (20 min)
- How to come up with metrics to evaluate an agent that generates linkedin posts automatically - Error analysis - Group Activity hands on (50 min)
- How to deal with subjectivity among reviewers? (15 min)
- LLM as a judge to evaluate Agents at scale (30 min)
- Wrap up - 15 min
Bio
Abhijith Neerkaje is co-founder Beyond Vectors - https://www.linkedin.com/in/abhijithneerkaje/
Slides
https://drive.google.com/file/d/1TQprCm36GIq3JOD9d-5SJ9XQxTZCH8Vq/view?usp=sharing
Video
https://drive.google.com/file/d/1teOGQr42z7O_GT4BawF5gPF_ua5tHLgl/view?usp=sharing
Comments
Get your hybrid access ticket
Hosted by
Jumpstart better data engineering and AI futures
Supported by
Platinum Sponsor
Atlassian unleashes the potential of every team. Our agile & DevOps, IT service management and work management software helps teams organize, discuss, and compl
Platinum Sponsor
Sahaj is an artisanal technology services company crafting purpose-built AI and data-led solutions for businesses.
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}