Platform Engineering for Production Agents

A hands-on workshop for platform, systems, DevOps, SRE, and backend engineers who need to run AI agents safely, reliably, and cost-effectively in production.

AI agents are no longer just local coding assistants or experimental chatbots. Once agents call tools, access data, run workflows, retry failed steps, and serve users, they become production workloads. They need the same platform discipline as any other distributed system: identity, secrets, isolation, observability, deployment, rollback, cost controls, and operational recovery.

This workshop focuses on the runtime and platform responsibilities that come with supporting agent-based applications in production.

What is this workshop about?

This workshop will help engineers understand what changes when AI agents move from development environments into hosted, production systems.

Participants will learn how an agent request flows across model calls, tool calls, retries, responses, logs, traces, and failure states. The workshop will cover the operational controls needed to make agents observable, bounded, secure, debuggable, and recoverable.

The workshop is not about prompt engineering or chatbot design. It is about the production environment around agents: how to run them, monitor them, constrain them, and recover them when something goes wrong.

What will participants work on?

Participants will work with a mock production incident-management agent built with pi.dev. The agent will run against a simple Go application connected to PgBouncer, PostgreSQL, and Redis. The system emits metrics and traces, and includes routes that fail in predictable ways.

Through this setup, participants will examine how agent workloads behave in production-like conditions, including:

• Liveness and readiness failures: is the agent working, and can it continue accepting requests?
• Observability gaps: what is the agent doing when it appears stuck?
• Cost failures: how can runaway token or tool usage be detected and controlled?
• Safety failures: how can agents be prevented from taking unsafe actions?
• Correctness failures: how do we evaluate whether an agent’s completed work was actually right?

Participants will be introduced to the sample system, the pi.dev incident-management agent, and a sequence of failures that help frame the platform engineering controls required for production agents.

Who is the target audience?

This workshop is for:

• Platform engineers supporting application teams that are building or deploying AI workloads
• Systems engineers working on infrastructure, runtime environments, internal platforms, or developer platforms
• DevOps and SRE practitioners responsible for reliability, deployment, observability, security, and cost controls
• Backend engineers building agents who want to understand the production platform their agents will need
• Engineers experimenting with AI agents who understand containerisation basics and want to move beyond local demos

Background knowledge requirements

You do not need prior experience building AI agents.

You should be comfortable with basic containerization concepts and familiar with how production applications are usually deployed, monitored, and operated.

This workshop is not designed for non-technical participants, business users, or people looking only for an introduction to prompting or AI tools.

Workshop content and plan

The workshop will be hands-on and will cover the following areas:

1. What makes agents different as production workloads

• Difference between coding assistants, local agents, and hosted agent services
• What models, harnesses, and agents are
• Anatomy of an agent request
• Model calls, tool calls, retries, memory, state, and responses
• Why agent workloads are harder to reason about than standard request-response services

2. Runtime behaviour and failure modes

• Tracing an agent request across model and tool boundaries
• Latency, timeout, retry, and partial failure patterns
• Liveness, readiness, observability, cost, safety, and correctness failures
• Model failures, tool failures, permission failures, and workflow failures
• Choosing safe recovery actions when agent execution goes wrong

3. Observability for agents

• Logs, metrics, and traces for agent systems
• Tracking token usage, latency, errors, tool calls, and cost
• Debugging agent behaviour using telemetry
• Separating application bugs, model issues, and platform failures

4. Access, isolation, and safety boundaries

• Bounding what an agent can access
• Tool permissions and scoped credentials
• Secrets management for agent workloads
• Sandboxing, isolation, rate limits, time limits, and spend limits

5. Platform responsibilities

• What belongs in agent code vs what should be provided by the platform
• Minimum production environment for hosted agents
• Deployment, rollback, configuration, and environment management
• Operational checklists for supporting agent-based applications

6. Hands-on exercises

Participants will work with a small agent workload and practice:

• Running it in a containerised environment
• Tracing a request through model calls and tool calls
• Observing latency, failures, token usage, and cost
• Applying runtime limits and access boundaries
• Diagnosing failures and selecting recovery actions
• Evaluating correctness and performance over time

Workshop agenda

• 14:00-14:25: Introduction, what is an agent, and why agent workloads differ from traditional workloads
• 14:25-15:00: Defining the system under test and the test agent; explaining the pi.dev agent and its tools
• 15:00-15:30: Failure modes in agents, and how these differ from failures in traditional applications
• 15:30-16:00: Networking and coffee break
• 16:00-16:20: Observability, liveness, and readiness for agents as production applications
• 16:20-17:00: Infrastructure, cost control, and security boundaries
• 17:00-17:15: Evaluating correctness and performance over time
• 17:15-17:30: A unified layer for addressing agent failure modes
• 17:30-18:00: Open Q&A and wrap-up

Key takeaways

By the end of the workshop, you will be able to:

• Explain how production agents differ from local coding tools and simple AI demos
• Trace one agent request across model calls, tool calls, retries, and responses
• Use metrics and traces to identify latency, errors, token usage, and cost
• Bound an agent’s access to tools, data, time, credentials, and spending
• Decide which controls belong in agent code and which should be platform defaults
• Define the minimum production environment for identity, secrets, isolation, telemetry, deployment, and rollback
• Diagnose model, tool, and agent failures and choose a safe recovery action
• Evaluate correctness and performance for an agent workload over time
• Support application teams that are moving agent workloads into production

Software installation requirements

Participants will need:

• A laptop that can comfortably run 2-3 Docker containers
• Linux or macOS preferred
• Docker installed before the workshop
• An API key for any model provider supported by pi.dev
• A code editor
• Basic familiarity with command-line usage

Participants may use any LLM API provider that is compatible with the workshop exercises.

Low-cost or free options include:

• OpenCode: create an API key on OpenCode Zen and use the free big-pickle model. This model may be rate limited.
• OpenRouter: create an account, generate an API key, and use one of the free models listed on OpenRouter, such as nvidia/nemotron-3.5-lightning:free.

Detailed setup instructions will be shared with registered participants before the workshop. Registered participants may also receive a test version of the workshop code before the session so they can check their setup.

Workshop code

Participants will work through code from: https://github.com/ace03uec/platform_engineering_production_agents

Workshop Presentation : https://drive.google.com/file/d/1khDCpBL_bVF39cKaOFAOfPD2rNnp2Lrl/view?usp=sharing

About the instructor

Giridharan Manivannan is a DevOps and Platform Engineering consultant with over 12 years of experience. He has worked with infrastructure teams at Razorpay and Hasura, helping teams build, operate, and scale production systems.

He writes and shares thoughts on DevOps, platform engineering, AI infrastructure, Chelsea FC, and anime at https://x.com/_dforce. He can be reached at hello@giridharan.dev for work.

How to attend this workshop

This workshop is open to Rootconf members:

https://hasgeek.com/rootconf#memberships

The workshop is open to 30 in-person participants, with hybrid access available for remote attendees. Seats for in-person participants will be available on a first-come, first-served basis.

Contact information

For inquiries about the workshop, contact +91-7676332020 or write to info@hasgeek.com.