Unavailable

This livestream is restricted

Already a member? Login with your membership email address

Rootconf 2026 Annual Conference

Platforms for AI, and AI for Platforms

Tickets

Loading…

This video is for members only

Nikhilesh Chamarthi

Nikhilesh Chamarthi

@Nikki371

What breaks on the way to 1,000 agents: long-running jobs, narrow privilege, many tenants

Submitted Oct 9, 2026

One-line summary
A typical production agent runs for a long time, with someone’s authority, next to other companies’ agents. Each part of that sentence broke something on Slash, Razorpay’s agent platform, and this talk is what broke, what we chose, and what you can copy.

What problem are you addressing?
Platforms assume requests are short and stateless, code is written and reviewed by people, and every tenant trusts the platform the same way. Agents break all three:

  • Long-running jobs. One request runs for minutes or hours, needs a whole repo on disk, and its memory jumps when it runs a build or opens a browser. Kill the pod and you kill someone’s work.
  • Narrow privilege. Every agent holds a shell and is steered by prompts we didn’t write. A list of allowed tools can’t fence a shell.
  • Many tenants. Teams across Razorpay and other companies build and run their own agents on one platform. Two runs side by side may belong to two companies that must never see each other.

Slash today: 200K+ agent runs a month, eight months after launch. Teams build agents from skills, plugins and workflows, and about half of them are automations that start on an alert or a keyword with nobody asking.

Intended audience
Platform engineering, SRE / DevOps, infrastructure and security engineers, and engineering leaders building internal AI platforms.

Level
Intermediate

Practical takeaways

Runs for a long time

  • Shrink context and cost: load tools on demand, compact early, and pick models by your own evals. Open-weight models made our cost per task about 6× lower.
  • Schedule by priority and workload shape: priority work gets its own pool; quick and long-running tasks go to different pools.
  • Make every run resumable: checkpoint every turn, so a deploy, a scale-down and a node move are all the same resume.

Holds someone’s authority

  • Enforce below the agent: a sandbox per session, with a kernel of its own.
  • Keep keys out of the run: a run holds references and short-lived grants, never keys.
  • Sign who is calling: every tool call carries who it is for, through which agent, for which tenant.

Runs next to other companies

  • Tenants are data, the cell is shared: a tenant is a row and a policy, not infrastructure.
  • The session is the wall.
  • Every shared layer has to know the tenant.

What will you share?

Long-running jobs

  • Context and cost: a long run re-reads its toolbox every turn. Tool search and code mode took input from about 33K tokens a turn to about 4K. Early compaction and a knowledge system cut it further, and open-weight models by default brought cost per task down about 6×.
  • Checkpoint and resume: every turn ends with a checkpoint of files and conversation in object storage; restore takes about 1–3 seconds on any warm pod.
  • Autoscaling: queue size hides work in progress, so scale-down counts in-flight work, not queue depth or CPU. The next ceiling is pods per node.
  • Scheduling: a router sends priority work to its own pool, quick tasks to a shared pool with several runs per pod, and long-running tasks to a pool with one per pod. Packing runs by expected memory footprint keeps pods busy, so you need fewer of them.

Narrow privilege

  • The attack surface: what container-level isolation let through, found in security testing before launch.
  • A sandbox per session with its own guest kernel behind a hypervisor: private files copied in, network denied by default, an allow-list set outside the sandbox, and fail-closed start.
  • Credentials by absence: the sandbox reaches three places outside the pod (a model and tool gateway, an egress proxy, a DNS resolver) and everything else is blocked.
  • Identity: a short signed passport per run, and a tool gateway that allows a call only when the person’s access, the agent’s ceiling and the run’s scope all agree.

Many tenants

  • The approach we considered, a data plane per tenant, and why it wouldn’t scale: idle warm pods per tenant, cloud account limits, and new infrastructure for every tenant.
  • The re-architecture: tenant is data, the cell is shared, the session is the wall, and what customers allowlist lives outside the cell.
  • The seven shared layers a request passes, from the integrations service to the model gateway, and how each one knows the tenant.
  • Open-source options worth evaluating for each piece.

What is your experience with this problem?

  • Production system
  • Real incident or failure
  • Hard-earned engineering lesson

What approaches failed, disappointed, or created unexpected problems?

  • One queue for everyone. A person asking a quick question waited about 27 minutes behind long-running jobs. Priorities only reordered the line; separate pools by priority and shape took that wait to about zero.
  • Heavy jobs sharing a pod. One long-running agent ran its pod out of memory and took every neighbouring run with it.
  • An autoscaler that couldn’t see work. Messages being worked on are hidden from the queue, so the autoscaler saw an empty queue and cut runs mid-flight.
  • Every tool schema in every turn. Long runs paid for the full toolbox on every turn.
  • Container-level isolation. In security testing, one ordinary-looking prompt had the agent copy cloud metadata into a repo. The agent did nothing wrong; it had a shell. We moved enforcement below the agent.
  • A data plane per tenant. Simple to reason about, but idle cost, account limits and onboarding grow with every tenant.

What will you do differently today?

  • Checkpoint every turn from day one, so deploys and scaling never cost anyone’s work.
  • Scale long jobs on waiting plus in-flight work, and schedule by priority and shape.
  • Enforce below the agent, keep keys out of the run, and sign who is calling.
  • Make the tenant data and the session the wall before the first external tenant.

What trade-offs did you consider?

  • Frontier models vs. open-weight models: quality per task measured with our own evals, against cost.
  • One pool vs. pools by priority and shape: simpler operations against queueing quick work behind long work.
  • Several runs per pod vs. one per pod: density against the blast radius of one heavy job.
  • Container isolation vs. a kernel per session: density and start-up time against how hard the wall is.
  • A data plane per tenant vs. one shared cell: the simplest boundary to reason about against idle cost, account limits and onboarding.

How can this help other practitioners?

Treat agents as long-running, untrusted, multi-tenant jobs, and know which platform primitives stop holding: queue depth, CPU scaling, tool allow-lists, a shared kernel, infrastructure per tenant. Copy the patterns: checkpoint and resume, priority and shape pools, a sandbox per session, credentials by absence, signed identity, and tenancy as data.

Current state
In production at Razorpay since March 2026, serving internal teams and external companies.

Tags
#agents #platformengineering #security #multitenancy #scalability #kubernetes #developerplatforms #casestudy

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

One year entry to Rootconf's programmes and activities

Hosted by

We care about site reliability, cloud costs, security and data privacy

Supported by

Platinum Sponsor

Nutanix is a global leader in cloud software, offering organizations a single platform for running apps and data across clouds.

Silver sponsor

Engineers at Roku Bengaluru, building the tech behind the #1 TV streaming platform in the US and Mexico. Come say hello at our meetups.

Bronze Sponsor

Internet infrastructure APIs for IP geolocation and more

Community Sponsor

Real-time Observability & Governance layer for AI agents