📘 Workshop overview
In this workshop we will cover the following points:
- Why AI agents fail—and how to measure them
- Design metrics that matter for AI agents
- Generate synthetic test data at scale
- Perform error analysis and improve prompts
- Build production-ready evaluation pipelines
🎯 Target audience
Techinical product managers
- AI Engineers
- Data Scientists
- Data Analysts
- Data Engineers
- Software developers
Background knowledge prerequisites
Familiarity with python coding
✅ Learning outcomes
Systematic agentic evaluations
🛠 Software installation requirements
Access to google colab
Workshop duration and content plan
This is a two-hour workshop, covering:
- Why do Agents make mistakes - 3 Gulfs [Comprehension, Specification and Generalization]. (10 min)
- Challenges of evaluating agent responses . Why is it different from standard software testing on ML system testing (10 min)
- Component wise evaluation of agents (What is equivalent of module level testing in Agents) (30 min)
- How to generate synthetic data to evaluate your agents - Hands on activity (20 min)
- How to come up with metrics to evaluate an agent that generates linkedin posts automatically - Error analysis - Group Activity hands on (50 min)
- How to deal with subjectivity among reviewers? (15 min)
- LLM as a judge to evaluate Agents at scale (30 min)
- Wrap up - 15 min
About the instructor
Abhijith Neerkaje is a data science and AI leader with over 20 years of experience building AI products across retail, semiconductor manufacturing, and energy. He is the Co-founder of Beyond Vectors AI Pvt Ltd, where he helps mid-career professionals and engineering leaders build expertise in Generative AI and Agentic AI.
Previously, Abhijith led the Data Science & Analytics function at Falabella India, driving AI-powered search, recommendations, pricing, and seller intelligence for one of Latin America’s largest e-commerce platforms. Earlier, at Target, he built machine learning solutions for pricing, merchandising, and supply chain optimization.
Abhijith holds engineering degrees from PES University (formerly PESIT) and the Indian Institute of Science (IISc), Bangalore, and an MS in Engineering and Management from the Massachusetts Institute of Technology (MIT). He was an MIT Tata Fellow and a winner of the MIT Clean Energy Prize (Renewable Energy Track, 2013).
How to attend this workshop
This workshop is open to:
🎟️ Fifth Elephant community members — https://hasgeek.com/fifthelephant#memberships
🎟️ Ticket holders for The Fifth Elephant annual conference — https://hasgeek.com/fifthelephant/enterprise-ai-in-production-meetup#tickets
This workshop is open to 30 participants (in-person) & hybrid access for remote attendees. Seats for in-person participants will be available on first-come-first-served basis. 🎟️
Need more info?
☎️ Call: (91) 7676332020
📧 Email: info@hasgeek.com