Rootconf 2026 Annual Conference

One-line summary

How we turned one coding bot into a multi-tenant agent platform — keyless sandboxes, scoped credentials, per-run tracing, an open-weight model gateway, a shared knowledge layer, budgets and continuous evals — so untrusted, long-running agents get broad context with narrow privilege, at low cost.

What problem are you addressing?

Platforms assume workloads are human-reviewed code, requests are short and stateless, and identity belongs to a service. Agents break all three:

• Untrusted code: each run executes code generated at runtime, inside an agent runtime we didn’t write. We control only what surrounds it: process, file system, network, credentials, model endpoint.
• Long, stateful work: runs last minutes to hours and take irreversible actions (PRs, messages) while pods are deployed and evicted under them.
• Identity is a tuple: every action belongs to a (tenant, agent, user, run), and what a run may reach changes per run.

Today the platform runs 60+ agents built by teams across Razorpay and external customers: ~50,000 tasks and ~1,000 merged PRs a month, 700+ weekly users, 16 external organisations. We’re building it to take that to 1,000 agents. Four tensions shaped it:

1. Context vs. privilege. A useful agent reads across ~2000 repositories, warehouses and live production systems; a safe agent holds almost nothing. How do you give broad context without broad credentials, or any vendor key?
2. Density vs. isolation. Agents can’t each get a microVM and still be fast and cheap. Warm shared workers move the isolation boundary from the run to the tenant. What do you accept on the wrong side of that line?
3. Model capability vs. context quality. Our platform runs 100% on open-weight models. Frontier models make up for vague context by exploring; open-weight models can’t: smaller context windows, weaker tool use. Making them carry production work meant giving them precise, current, compact context about our systems — fast-changing code, metric definitions, ownership, and live state that can’t be indexed.
4. Many authors vs. one platform team. Most agents are written by teams we’ve never met. How do you keep agents self-serve and still answer “what can this agent reach, and what did it do?”

All of this runs at a payments company, so the governance bar is high.

Intended audience

Platform Engineering, SRE / DevOps, Infrastructure, Engineering leaders building internal AI platforms.

Level

Intermediate

Practical takeaways

1. The system around the agent, not the model, makes agents productive. A model alone demos well and stalls in production: it doesn’t know your systems, can’t touch them safely, and nobody can see what it did. Context, scoped tool access and a trace of every run are what made our agents part of daily work. Example: Our recent analytics agent, answers plain-English data questions across 209 domains and 5,800+ metric definitions with no repo access and no credentials — 1000+ queries, rated 4.4/5.
2. Build a platform for agents, not agents. Every agent inherits context, safety, model access and observability instead of rebuilding them, which is how 60+ teams’ agents run on one platform. For leaders building in-house, this choice decides how fast teams ship agents, whether security can say yes, and whether you can switch models without a rewrite.

What will you share?

• Architecture: control plane vs. per-tenant data planes, one execution contract over five agent runtimes, short-lived scoped credentials, a three-layer knowledge service.
• Production experience: ~50K tasks/month, 100% on open-weight models, ~2B tokens/day through our own model gateway across 32 models, multi-hour runs.
• Trade-offs: warm pools vs. per-run microVMs, deterministic search vs. embeddings, open-weight vs. frontier models.
• Real failures, and before/after at each turning point.

What is your experience with this problem?

• Production system
• Real incident or failure
• Hard-earned engineering lesson

What approaches failed, disappointed, or created unexpected problems?

• One identity for every agent. All agents ran as a single service account. It collected every permission any agent needed, and nothing it created or changed could be traced back to the person who asked. We moved to short-lived, per-run tokens that carry the requester’s identity.
• Cost without guardrails. A single task burned ₹1,000. We added per-run budgets and continuous evals, and used those evals to move all our agents onto open-weight models without losing quality.
• Knowledge that couldn’t keep up. Any model will paper over stale or missing context by exploring — more searches, more file reads, more retries. It usually gets there, but every extra step is tokens and time. Since our goal was to cut cost per run, we split knowledge by how fast it goes stale — git for durable knowledge, fresh code searched by agents, live state queried and never indexed — and route each question to the right sources first. Where no certified answer exists, the agent generates one and a human certifies it, so the catalogue grows from use.
• Runs that vanished. In one incident the platform killed 185 sessions with no error and no trace. That made run-level accountability a requirement: one trace per run, wide events, and a terminal state for every run.

What will you do differently today?

• Gateway-only model access and requester-scoped tokens from day one.
• Budgets and evals before the first expensive run, not after.
• Choose the isolation unit up front, not while onboarding the first external tenant.
• One trace and a guaranteed terminal state for every run.

What trade-offs did you consider?

• Warm pools vs. per-run sandboxes: strong isolation between tenants, weaker between runs of one tenant, in exchange for start latency and cost.
• Agentic search vs. embeddings: exact, fresh and explainable, at the cost of fuzzy recall.
• Open-weight vs. frontier models: we went 100% open-weight — cheaper and fully under our control, but weaker on hard tasks. We closed the gap with context and evals, not model size.

How can this help other practitioners?

• Treat agents as untrusted, long-running, stateful workloads, and know which platform primitives stop holding.
• Broad read context, narrow write privilege, run-level accountability.
• Make open-weight models viable by improving context and gating with evals.
• For leaders: invest in the platform before adding more agents or bigger models.

Current state

Production experience.

Tags

#agents #platformengineering #security #governance #developerplatforms #observability #inference #scalability #casestudy