After the Prototype - agents, applied AI, and models put to work

Format: 30-minute talk (no demo)

The problem: Teams ship LLM, RAG and agent systems on eval scores that often measure the wrong thing. Judge models drift and disagree, generic metrics miss faithfulness and safety failures, and eval suites stop reflecting production once agents start calling tools.

Why it matters: As applied AI moves past the prototype, eval scores become release gates. A misleading score means shipping regressions with false confidence.

Key takeaways: where common metrics mislead; how to calibrate and cross-check LLM-as-judge with small local models (e.g. toxic-bert) and custom scorers; testing RAG pipelines and agents end-to-end over HTTP; a practical checklist for evals that track real-world quality, safety and faithfulness. Drawn from hands-on evaluation work with LLMValidate.ai.

Who it’s for: ML and AI engineers, QA and test leads, and product owners putting RAG and agentic systems into production.

Speaker bio: Dr Srinivas Padmanabhuni is CTO and Co-founder of testAIng.com (AiEnsured), an ISO/IEC 42001 certified enterprise AI testing company in Bengaluru. Past President of ACM India and author of an upcoming book on Agentic AI. LinkedIn: https://in.linkedin.com/in/spadmanabhuni