After the Prototype - agents, applied AI, and models put to work

What happens when the underlying API that your agentic system relies on changes unexpectedly? This talk is a production case study of how I handled an upstream API change in two weeks for a system already running in production, without regressions.

Abstract

For conventional platform systems, an API version change can often be handled by upgrading to the new version. But what happens if an agentic system is involved? The impact goes beyond the API version change itself. It can also affect how the agent interprets and responds to user queries.
In my case, an upstream API change transformed the structure of the valid data from a flat list into a hierarchy. This required changes to both the platform and the behaviour of an agent already running in production. The challenge was to handle the change within two weeks while preserving existing behaviour.

The agent had to interpret requests at different levels of the hierarchy while remaining generic enough to handle queries we had not explicitly anticipated. Addressing this required changes to the platform, prompt tuning, post-processing validation, and a revised evaluation strategy.
This talk explores the trade-offs involved in balancing generic as well as specific queries, how prompt tuning and post-processing validation worked together, and the role of testing and evaluation when making changes to a live agentic system.

Key takeaways for attendees

• Handling upstream changes to data in an agentic system
• Combining LLM reasoning with deterministic validation
• Evaluating production agents

Target Audience

• Software engineers building LLM-powered applications.
• AI engineers working on agentic systems in production.
• Platform engineers maintain systems that depend on external APIs and structured data.

Prerequisites

Basic understanding of LLM-powered applications and API integrations.
Experience with production agentic systems is helpful but not required.
Preferred format
30 minutes talk

Speaker bio

Mahita D is a Solution Consultant at Sahaj Software. She loves building and breaking agentic systems and experimenting with what’s new in AI-assisted software development. She often shares her learnings through blogs, talks, and workshops.

Previous Workshops

https://hasgeek.com/fifthelephant/from-amnesia-to-photographic-memory-workshop/
https://hasgeek.com/fifthelephant/crafting-multi-agent-systems-blr-edition/
https://hasgeek.com/fifthelephant/crafting-multi-agent-systems-pune-edition/
https://sahaj.ai/events/building-multi-agentic-ai-systems-with-langgraph-hands-on-workshop/

Previous Talks

https://sahaj.ai/events/under-the-hood-of-agentic-ai/
https://youtu.be/zWWWi5tfkn4?si=SXbpzCFN62GDcAxa
https://sahaj.ai/events/intelligent-engineering-reimagining-how-software-is-built-operated-and-evolved-with-ai/
https://sahaj.ai/events/ai-search-and-kubernetes-operators-explained/