Proposal template

Title

100+ clusters used by 100+ teams, and a platform that’s starting to run itself

Summary

Roku runs most of its cloud on RMCP, a multi-cloud Kubernetes platform with 100+ clusters on AWS and GCP. In 10 minutes we’ll show what that scale forced us to build, how we give apps written by AI agents a safe place to run, and what changed when an AI agent joined our platform team.

What problem are you addressing? What real engineering challenge, question, or experience is this session about?

At Roku, 100+ internal teams ship to 37 production Kubernetes clusters (100+ including dev and staging) across AWS and GCP. Most of those teams are not Kubernetes experts, and they shouldn’t need to be. Two problems come with that:

• Running a fleet, not a cluster. Anything done by hand, one cluster at a time, breaks somewhere past a few dozen clusters. Upgrades, namespaces, access and capacity all have to be automated and reviewable.
• AI changes who writes software and how fast. Anyone at Roku can now build an internal tool in an afternoon with an AI agent, and nobody reviews that code the usual way. At the same time, AI agents are opening about half of the merge requests in our own platform repos.

The talk covers three answers: a paved road from one config file to production, VibeGrid (a zero-trust cluster for AI-written apps), and Optimus (an AI agent that does real work on our team, with humans approving what it proposes).

Who is the intended audience?

Platform Platform engineering, SRE, DevOps and infrastructure engineers, plus engineering leaders deciding how far to let AI agents into their platforms.

Level:

Intermediate

List one or two practical takeaways.

• Give app teams one small config file that describes what the app needs, and let the platform generate the Kubernetes manifests, security checks and canary rollout. That is how 100+ teams ship without writing Helm charts.
• Treat every AI-written app, and every AI agent, as an untrusted tenant. Design from the threat (“this app is compromised”) rather than the tool, and put three independent controls in its way.

What will you share?

• Architecture and design decisions: the paved road (rmcp.yaml → build → generated manifests → canary across regions) and the fleet tooling behind it (one cluster registry, spot-first autoscaling).
• Git as the single source of truth: every cluster and every namespace is a reviewed file in git, applied by CI. Because the whole fleet is described in one place, we can build more on top of it, such as the service catalog in Backstage.
• Production experience: VibeGrid has 70+ apps from 20+ teams in the six months since the first one landed, and RMCP repos see 3.5× more merge requests a month than a year ago, about half opened by AI agents.
• Operational challenges: what to automate first when patching, reviews and releases can’t keep up with AI-generated change.

What is your experience with this problem?

• Production system
• Internal engineering project
• Hard-earned engineering lesson

What approaches failed, disappointed, or created unexpected problems?

• Asking app teams to own Helm charts and raw manifests. Most teams aren’t Kubernetes experts, so onboarding was slow and every team solved the same problems differently.
• Treating AI-written apps like any other internal app. The usual code review doesn’t happen, so the usual trust doesn’t hold. That is what led to a separate, locked-down cluster.

What will you do differently today?

• Start from the threat model and the paved road on day one, not after the fleet has grown. Put guardrails that an agent can’t remove in place before letting AI agents touch the platform.

What trade-offs did you consider?

• Why one config file instead of Helm charts? Helm gives teams full control, but it also hands them all of Kubernetes. A small rmcp.yaml says what the app needs and the platform generates the rest, so teams give up some flexibility in exchange for speed, consistency and built-in security and observability.
• Why a separate cluster for AI-written apps instead of namespaces on the shared clusters? A separate cluster with deny-by-default networking costs more to run, but a compromised app has no network path to production at all.
• Why keep humans approving the agent’s changes? Optimus could merge its own changes, but we chose speed with human approval over full autonomy.
• Build vs. buy: RMCP is built on open-source Kubernetes, Istio and managed cloud services. We built the glue (paved road, registry, namespace pipeline) because that is where Roku’s needs differ.

How can this help other practitioners?

• A design approach: model AI-written apps and AI agents as untrusted tenants.
• A pattern to adopt: one small config file per app, with the platform generating everything else.
• A set of operational practices: a release train for cluster upgrades, git as the single source of truth for clusters and namespaces, and patching as a routine release.

Current state

• Production experience

Tags

#platformengineering #kubernetes #sre #developerplatforms #multicloud #aiagents #security #casestudy