BEGIN:VCALENDAR VERSION:2.0 PRODID:-//HasGeek//NONSGML Funnel//EN DESCRIPTION:Platforms for AI\, and AI for Platforms X-WR-CALDESC:Platforms for AI\, and AI for Platforms NAME:Rootconf 2026 Annual Conference X-WR-CALNAME:Rootconf 2026 Annual Conference REFRESH-INTERVAL;VALUE=DURATION:PT12H SUMMARY:Rootconf 2026 Annual Conference TIMEZONE-ID:Asia/Kolkata X-PUBLISHED-TTL:PT12H X-WR-TIMEZONE:Asia/Kolkata BEGIN:VEVENT SUMMARY:Check-in and tea DTSTART:20261120T040000Z DTEND:20261120T043000Z DTSTAMP:20261109T024109Z UID:session/MPES9RnHxjynuu76SokFEj@hasgeek.com SEQUENCE:3 CREATED:20261104T072342Z LAST-MODIFIED:20261104T072540Z LOCATION:TERI\, Bangalore and online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Check-in and tea in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Introduction to the theme DTSTART:20261120T043000Z DTEND:20261120T044000Z DTSTAMP:20261109T024109Z UID:session/R1DgjEYnMw4YKRWdm2Z1bW@hasgeek.com SEQUENCE:2 CREATED:20261104T072440Z GEO:12.9634971;77.6380856 LAST-MODIFIED:20261104T072542Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Introduction to the theme in Auditorium (Ground Floor) in 5 mi nutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:We burned 48 GPU hours finding out how few bits a model needs DTSTART:20261120T044000Z DTEND:20261120T051500Z DTSTAMP:20261109T024109Z UID:session/KmjMA4cRoRDgqrL1ysxSUz@hasgeek.com SEQUENCE:4 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T072456Z DESCRIPTION:**One-line summary.** I spent a year answering one question\, how few bits a language model actually needs\, and the infrastructure I an swered it on failed in five different ways that had nothing to do with mac hine learning. This talk is both halves: the research\, the day a containe r spent training on a tenth of its corpus while every dashboard stayed gre en\, and the question underneath both\, which is who owns a decision that crosses four teams.\n\n## Why any of this exists\n\nI build an open source numeric format and an accelerator to run it on. Before anyone commits sil icon\, somebody has to answer one question: how few bits does a model actu ally need?\n\nQuantisation is the answer to it. Storing every number in a model with fewer digits\, so it uses less memory\, less power and generate s tokens faster. It is rounding. The only interesting question is what the model forgets.\n\nFour people have to agree before it happens. **Cost**\, which GPU tier you are willing to pay for. **Platform**\, RunPod or Modal or the cloud you already have. **Quality**\, how far you compress before it stops being smart. **Hardware**\, Nvidia or the faster new silicon with nobody to ask when it breaks. In most organisations you can settle two of those\, maybe three. Never four. That is the thread that runs through the whole talk.\n\nFrom roughly 878 archived training runs the answer is eigh t bits. Qwen3 8B scores 90.2 on the Berkeley Function Calling Leaderboard after post-training quantisation to eight bit fixed point\, with no retrai ning and no calibration set\, against 91.3 untouched. Drop to six bits and a 4B model falls to 63. At 0.6B it stops being a model at all.\n\n## Answ ering it meant a lot of training runs\, on other people's machines\n\n878 runs across four platforms in a year: a GPU under my desk\, a shared lab m achine\, RunPod and Modal. None of it is a production service. All of it h as the operational character of one\, because a job that dies at hour twen ty is expensive and a job that succeeds while computing the wrong thing is worse.\n\nThe run that cost the most was Llama 3.2 1B\, from scratch\, qu antisation aware\, on one H100\, with a corpus that arrives in shards rath er than all at once. It finished its first leg with no errors.\n\n## The d ebug journey\n\nBy hour six the loss was climbing. That is normal early on \, so I left it. By hour twenty four it still had not come down. Train los s 0.95\, validation loss 5.6.\n\nThen I guessed\, in order.\n\n**Guess one : the eval split is noisy.** Killed by the fact that noise does not climb in a straight line for eighteen hours. This guess cost the most\, because it let the run keep going.\n\n**Guess two: it is overfitting.** Killed by the fact that a 1B model does not overfit a corpus that size in a single d ay.\n\n**Guess three\, which should have been first: count the tokens it a ctually saw.** Steps\, times sequence length\, times batch and accumulatio n. Four numbers already sitting in the log that I had never multiplied. Ni nety six million tokens. Ten percent of the corpus\, thirty times over. Th at is not pretraining\, it is memorisation\, which is exactly why it looke d brilliant on training data and useless on anything real.\n\nNone of this was automated. I read container logs by hand and argued with a chat windo w. There was no dashboard\, no trace and no alert\, because nothing was wa tching for a job that succeeds while computing the wrong thing.\n\nThe cau se was a cache that nobody calls a cache: a container only sees what anoth er container wrote to a shared volume after it explicitly asks again. The data preparation job was still unpacking shards\, both trainers looked onc e\, and neither ever looked back. The whole fix is three lines and a guard clause: reload the volume\, refuse to start on a partial corpus\, and cha in the legs so a six day run survives a platform that kills every containe r at twenty four hours.\n\nForty eight H100 hours\, two containers\, one f ull day each.\n\n## Five failures\, and none of them are exotic\n\nThe sam e run and its neighbours also produced a foreground launch whose remote jo bs died with my laptop's DNS\, a twenty four hour function cap discovered while planning a six day run\, two workspaces holding volumes with the sam e name and different contents\, and a zero sized persistent volume that le t ten hours of work expire as though it never existed.\n\nAnd the quiet on e: results written to a remote archive and never folded back\, so a docume nt in the project said an experiment had never been run while a complete h undred and five cell grid sat in the archive. Recovering it cost one downl oad. Re-running it would have cost a GPU day.\n\nEvery one of these is som ebody's postmortem this quarter. They are not rookie mistakes\, they are t he default behaviour of the platforms\, and they sort into three shapes: * *shared state**\, who guarantees the input is complete before a consumer r eads it. **Lifetime**\, what keeps the remote work alive and for how long. **Ownership**\, whose job it was to notice.\n\n## What I do differently n ow\n\nFour checks\, every run\, no exceptions. A smoke test: one tiny pass end to end on a CPU before anything paid starts. Hash the dataset and ref use to start unless the hash matches. Checkpoint every couple of hours and push a copy to the Hub. And list the GPU tiers before choosing one\, beca use the same mistake on a cheaper tier is a cheaper mistake.\n\nTwo minute s of smoke test decides the shape of six days of training.\n\n## How would you even know the compressed model still works\n\nThis is where the secon d half lands\, and it is the part I got most wrong.\n\nI started with Lang fuse and LangGraph\, an LLM reading the logs as a judge. It never once fla gged a missing tool call\, because a judge grades the text it was given an d silence reads as a perfectly reasonable answer. I now run an open ensemb le through OpenRouter and Promptfoo\, three cheap models from different fa milies so they do not share a blind spot\, with explicit metrics pulled ou t of the logs rather than out of an opinion.\n\nThe failure that made the point: a quantised model scored a perfect 100 on one benchmark category\, because that category scores a model on correctly declining to call a func tion and the model had stopped calling functions at all. It answered 1.2 p ercent of prompts. Averaged across categories it would have been published as mediocre rather than dead.\n\n**Every capability metric needs a livene ss metric beside it.** For tool calling that is the call rate. For your sy stem it is whatever separates correctly did nothing from did nothing.\n\n# # The part I cannot answer\n\nPrecision changes the served model numerical ly without changing its version number. It is a cost decision\, a platform decision\, a quality decision and a hardware decision at once\, and in mo st teams nobody owns all four\, so it either never happens or it happens w ith no validation story. Same shape as the result that sat in an archive n obody owned.\n\nSo I will finish on three questions for the room. Who sign s off when the model changes numerically but not by version? What evidence does your team require? And has anyone ever actually rejected a change on it?\n\n## Who this is for\n\nPlatform engineering\, SRE\, infrastructure\ , MLOps\, and engineering leaders who approve model changes. Anyone whose team runs training or batch jobs on somebody else's hardware. No quantisat ion background needed\; the format is explained in one slide and the failu res are platform failures.\n\n**Level:** intermediate.\n\n## Takeaways\n\n 1. A producer and a consumer sharing a network volume is a distributed sys tems problem\, not a filesystem operation. Make the consumer assert on com pleteness rather than trust what it can see.\n2. A cheap screen beats care ful review. A CPU smoke test and a throughput probe before every paid run has caught more of my expensive mistakes than reading the code ever did.\n 3. Every capability metric needs a liveness metric beside it\, or a system that has stopped working will score well.\n\n## Slides\nhttps://docs.goog le.com/presentation/d/1X_vufDjh4jDsYAdmQI_ij0dz7ftki4c7wHrKuHiM_WQ/edit?us p=sharing\n\n## Format\n\nTwenty to twenty three minutes\, with five minut es kept for questions. Roughly a third of the deck is diagrams: the platfo rm model\, the training loop\, a timeline of the twenty four hours\, the t oken arithmetic\, the race condition\, and the fix.\n\n**Current state:** open source. Every number above traces to a script and an archived result file at github.com/aloshdenny/superfloat\, and the figure scripts run on a laptop with no GPU.\n\n**Tags:** #mlops #infrastructure #failurestory #gp u #serverless #inference #quantization #evaluation #observability #casestu dy\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T071907Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/we-burned-48-gpu-hours-find ing-out-how-few-bits-a-model-needs-KmjMA4cRoRDgqrL1ysxSUz BEGIN:VALARM ACTION:display DESCRIPTION:We burned 48 GPU hours finding out how few bits a model needs in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Private AI operationalized: how hard can it be? DTSTART:20261120T052000Z DTEND:20261120T055500Z DTSTAMP:20261109T024109Z UID:session/6vtufkTudjKJLhMoygzTSx@hasgeek.com SEQUENCE:5 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T072526Z DESCRIPTION:# Updated Submission\n## Title\n\nPrivate AI Operationalized: How Hard Could It Be?\n\n## One-line summary\n\nWhat actually happens when you point real AI agents at models running on hardware you own — and wh at it takes to make them useful\, observable\, and contained.\n\n## What p roblem are you addressing?\n\nA lot of private AI discussions either focus on using hosted APIs or jump straight to datacenter-scale infrastructure. This talk is about the space in between: running AI on hardware you direc tly own and control.\n\nEvery agent today claims "local model support". Ho w hard could it be?\n\nIt depends. Agents are built and tuned for frontier APIs. When you point them at a local model\, the assumptions they make ab out APIs\, context\, and model capability start to break — and most of t hat breakage is invisible from the agent's own interface.\n\nGetting a mod el to run is the easy part. Operating private AI means knowing what fits y our hardware\, what your agents are really doing\, how to make small model s useful\, and how to keep agents contained.\n\n## Who is the intended aud ience?\n\n**UNCHANGED**\n\n## Level\n\n**UNCHANGED**\n\n## Practical takea ways\n\n- A short checklist for deciding whether a private AI agent setup will work on your hardware before you invest in it.\n- A way to see what y our agents actually send to your models — and why it is not what the age nt shows you.\n- Practical techniques to make small local models useful fo r focused tasks.\n- A threat-model-first approach to containing agents\, w ith open-source tooling to reproduce it.\n\n## What will you share?\n\n- * *A quick recap of the foundation (Part 1):** what private AI means\, hardw are choices\, model formats\, quantization\, context\, and KV cache — ju st enough to size a model for your machine.\n- **The software stack:** the controller\, inference runtimes\, agents\, and observability tools I use daily.\n- **Three agents\, three stories:** case studies of running Claude Code\, OpenCode\, and Hermes Agent against local inference — what worke d\, what broke\, and the workarounds.\n- **Looking under the hood:** compa ring what an agent shows\, what its logs record\, and what actually goes o ver the wire.\n- **Making small models work:** how model size\, context\, and agent design interact\, and a measured before/after of tuning a purpos e-built agent on a small model.\n- **Advice for agent builders** who want their tools to work well with local models.\n- **Security and containment: ** the threat landscape across the private AI stack\, and the containment environment I built for my own agents.\n- **A public release** of configs and optimization tips for running open-weight models across NVIDIA\, Intel \, and Apple hardware.\n\n## What is your experience with this problem?\n\ n**UNCHANGED**\n\n## What approaches failed\, disappointed\, or created un expected problems?\n\n- Agents that "support local models" broke in ways t heir own interfaces never showed\; only inspecting the traffic revealed wh y.\n\n## What will you do differently today?\n\n**UNCHANGED**\n\n## What t rade-offs did you consider?\n\n**UNCHANGED**\n\n## How can this help other practitioners?\n\nThe session should help practitioners avoid buying hard ware that cannot run what they need\, picking agents they cannot inspect\, and assuming that a bigger model is the fix when the real problem lies el sewhere.\n\nAttendees should leave knowing which questions to ask before b uilding a private AI agent setup\, how to verify what their agents are doi ng\, and how to keep those agents contained.\n\nThe goal is to help people build small\, self-owned AI systems that are useful enough and boring eno ugh to depend on.\n\n## Current state\n\nIn progress\, with open-source co mponents in daily use. The linked projects are active works in progress an d change regularly because I use them as daily drivers.\n\nThis talk is Pa rt 2 of "Building\, Owning\, and Securing Private AI Infrastructure" (OWAS P 25th Anniversary Virtual Conference\, 22 September 2026): https://anants hri.info/slides/2026-owasp-virtual-building-owning-private-ai-final/\n\nPa rt 1 covered the foundation for a security-focused audience: defining priv ate AI\, hardware\, quantization\, the software stack\, and threat modelin g. Part 2 is about operating it: agents\, observability\, context\, and co ntainment.\n\n## Tags (suggested additions)\n\n#observability #claudecode #opencode #hermesagent #contextengineering #containment\n\n\n# Initial Sub mission\n## One-line summary\nThis session documents the journey of buidin g and operating AI / LLM systems on hardware you directly control be it ho me or office. The key focus is on model choices\, trust\, power\, isolatio n\, maintainability and real-world usefulness.\n\n## What problem are you addressing?\n\nA lot of private AI discussions either focus on using hoste d APIs or jump straight to datacenter-scale infrastructure.\n\nThis talk i s about the space in between: running AI on hardware you directly own and control.\n\nThat might be a single workstation with a GPU\, an Apple syste m with unified memory\, a small collection of machines connected over Tail scale\, or an isolated box running an agent that should never send its con text outside your network.\n\nThe constraints are very different at this s cale.\n\nYou care about things like whether the machine remains usable whi le inference is running\, whether a model can stay loaded all day\, how mu ch heat it produces\, what happens to power consumption\, whether a smalle r model is intelligent enough for the task\, and whether an upgrade is wor th the disruption.\n\nAI/LLM APIs are getting cheaper per token\, while th e number of tokens consumed per task is often increasing. People sometimes move toward self-owned AI primarily as a cost-saving measure. I would arg ue against treating cost as the main reason.\n\nSelf-owned AI is often mor e expensive\, slower\, and harder to maintain than calling a good API.\n\n The value is control: control over models\, data\, network boundaries\, ru ntimes\, tools\, upgrades\, agents\, and eventually the complete stack.\n\ n## Who is the intended audience?\n\nPlatform Engineering\, SRE\, DevOps\, Infrastructure\, Developers\, Security Engineers\, AI/ML practitioners\, and technically curious people experimenting with self-owned AI on hardwar e they directly control.\n\nThe focus is not datacenter-scale AI infrastru cture. It is the smaller end of the spectrum: a workstation\, home lab\, r esearch setup\, small office environment\, or a few systems connected toge ther to provide private inference and agent capabilities.\n\n## Level\n\nI ntermediate.\n\nSome parts will go deeper into inference runtimes\, quanti zation\, model behaviour\, hardware optimization\, networking\, supply-cha in risks\, and agent isolation.\n\nThe talk assumes that attendees are com fortable with infrastructure concepts\, but they do not need prior experie nce operating local LLMs.\n\n## Practical takeaways\n\n1. A practical fram ework for selecting models based on how useful they are on the hardware yo u actually own\, considering intelligence\, heat\, power\, memory pressure \, responsiveness\, tool use\, and maintainability rather than simply mode l size or benchmark numbers.\n\n2. An approach for building self-owned AI where models\, runtimes\, networking\, routing\, prompts\, trust\, airgapp ing\, tools\, and agent workflows are treated as parts of an infrastructur e stack that must be tested\, constrained\, pinned\, and maintained.\n\n## What will you share?\n\nI will share architecture\, experiments\, failure modes\, open-source tooling\, and lessons from running private AI across different hardware and software stacks at workstation\, home-lab\, and sma ll-office scale.\n\nThe talk will cover:\n\n- Why cost alone is usually a poor reason to move from hosted APIs to self-owned AI\n- What control and ownership provide in return for accepting additional cost and operational complexity\n- Why “can this model run?” is different from “should th is model run here?”\n- Choosing models based on workload rather than lea derboard position\n- Why intelligence does not map cleanly to model size\n - Choosing between model sizes\, quantization levels\, dense and MoE archi tectures\n- Power\, heat\, responsiveness\, fan noise\, and system degrada tion as real operational constraints\n- Why highest tokens/sec or prefill performance is not always the right optimization target\n- Experiments opt imizing llama.cpp for inexpensive Intel Arc Pro B70 hardware\, and what th at taught me about useful performance versus benchmark performance\n- Prom pt and context optimization experiments with Hermes and smaller models bef ore assuming that a larger model is required\n- Whether newer models are a ctually better for the task you care about\n- Building your own regression tests and golden tasks for model upgrades\n- Establishing trust through m odel provenance\, version pinning\, behavioural testing\, and controlled t ool use\n- Running airgapped or highly restricted agents\n- Using Tailscal e and similar approaches to connect selected systems while keeping other s ervices isolated\n- Deciding what an agent should be able to see\, what it should connect to\, and what should remain outside its trust boundary\n- Deciding what belongs in the model and what should come from retrieval\, t ools\, or APIs\n- Fine-tuning or adapting smaller quantized models for nar row use cases\n- Using\, customizing and owning llama.cpp and llama-swap a s parts of the inference stack\n- Modifying open-source runtimes when the default behaviour does not match the hardware or workload. Deciding if its specific to you or needs to be upstreamed.\n- Experiences comparing codin g-agent approaches such as Claude Code and OpenCode\, including where mode l quality helped and where the surrounding harness mattered more\n- Using AIDC as an example of moving control into the harness\, including context management\, references\, execution\, review\, and lifecycle management\n- Supply-chain concerns around models\, runtimes\, Python packages\, contai ners\, plugins\, agent skills\, and other dependencies entering a supposed ly private environment\n- Where local models fail\, where APIs remain bett er\, and where a hybrid architecture is the more sensible answer\n\nThe in dividual projects and tools are examples. The aim is to extract operationa l patterns that remain useful even when models and frameworks change.\n\n# # What is your experience with this problem?\n\n- Internal engineering pro ject\n- Open-source project\n- Experiment or prototype\n- Research / inves tigation\n- Hard-earned engineering lesson\n\nOver the last few years I ha ve been running public and private AI models side by side across NVIDIA\, Intel\, and Apple hardware.\n\nMost of this work happens at the scale this talk is aimed at: individual workstations\, home-lab systems\, research m achines\, and small office infrastructure rather than datacenter deploymen ts.\n\nMy work has included:\n\n- Maintaining a customized [llama-swap](ht tps://github.com/anantshri/llama-swap) setup for my own model routing and lifecycle requirements\n- Contributing performance and hardware-specific c hanges to [llama.cpp](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Ap r+author%3Aanantshri)\, including optimization work around [Intel Arc Pro B70](https://blog.anantshri.info/experiments-with-intel-arc-pro-b70-on-deb ian-13/)\n- Experimenting with model sizes\, quantization\, speculative de coding\, context sizes\, prompt optimization\, tool calling\, and differen t inference backends\n- Comparing larger and smaller models on the same wo rkloads rather than assuming parameter count determines usefulness\n- Expe rimenting with Hermes Agent and smaller models to understand how much capa bility can come from better prompts\, tools\, and context\n- Comparing cod ing-agent environments such as Claude Code and OpenCode to understand wher e capability comes from the model and where it comes from the harness\n- R unning local\, isolated\, and selectively connected agents\n- Using Tailsc ale to connect selected AI systems without exposing the complete environme nt\n- Building private AI infrastructure for coding\, security research\, and automation\n- Maintaining security and engineering tools where AI is u sed as part of development\, analysis\, and review\n- Building [AIDC](http s://github.com/cyfinoid/aidc) as an agent-oriented development environment with explicit context\, references\, execution\, and review workflows\n- Operating both local models and hosted APIs depending on the workload\n\nT his has been less about building one polished “AI platform” and more a bout continuously changing the stack until it behaves the way I need and o wning the resultant stack. \n\n## What approaches failed\, disappointed\, or created unexpected problems?\n\nSeveral things turned out to be less us eful than they first appeared.\n\n- Chasing raw tokens/sec often improved benchmark numbers without improving the actual experience of using the sys tem.\n- A model can technically fit in memory and still be the wrong model for the machine. Large models sometimes made the rest of the system unple asant to use because of heat\, memory pressure\, loading time\, fan noise\ , or sustained power draw.\n- Model size also turned out to be an unreliab le shorthand for intelligence. For some workloads\, a smaller model with b etter prompts\, cleaner context\, and better tools was more useful than a larger general model.\n- Newer models were not always better. Some improve d reasoning while regressing in tool calling\, instruction following\, str uctured output\, or behaviour under quantization.\n- Quantizing more aggre ssively sometimes saved enough memory to make a model fit while damaging e xactly the capability I cared about.\n- Long context introduced memory and KV-cache costs that were easy to ignore while looking only at model size. \n- Coding-agent experiments also showed that changing the model does not automatically fix the workflow. Comparing systems such as Claude Code and OpenCode made it increasingly clear that context management\, execution co ntrols\, tool design\, review loops\, and the surrounding harness can matt er as much as the underlying model.\n- Airgapping turned out to be more co mplicated than disconnecting a machine from the internet. Models\, runtime s\, packages\, documentation\, tools\, updates\, and agent capabilities al l need controlled ways to enter the environment.\n- This creates another p roblem: supply chain. A system can be completely private while running and still depend on externally produced models\, packages\, containers\, plug ins\, agent skills\, and binaries.\n- Even deciding what to expose became an architecture problem. Giving an agent access to everything is easy but creates a very large trust boundary. Giving it access to nothing produces a very private but mostly useless system.\nThe biggest lesson has been tha t self-owned AI is much more of an infrastructure problem than an applicat ion problem.\n\n## What will you do differently today?\n\n- I now optimize around the workload rather than the model.\n- I prefer a model that is go od enough\, predictable\, and comfortable on the machine instead of consta ntly replacing it with whatever is newest.\n- Before moving to a larger mo del\, I look at whether the problem can be solved by improving prompts\, r educing unnecessary context\, providing better tools\, or improving the ag ent harness.\n- I test model upgrades against my own workloads before adop ting them.\n- I separate model intelligence from tool-provided knowledge. If accurate information can be retrieved from a deterministic source or AP I\, I prefer teaching the model when and how to use that source instead of trying to embed all knowledge into the model.\n- I increasingly treat the harness as its own engineering problem. A model is only one component in an agent system.\n- I treat model files\, prompt templates\, inference run times\, routing configuration\, tool definitions\, and agent behaviour as versioned parts of the stack.\n- For networking\, I explicitly decide whic h systems should communicate. Tools such as Tailscale make it possible to connect selected nodes without exposing inference endpoints or agents to t he public internet.\n- For agents\, I prefer restricted environments with explicit tools\, limited network access\, and clear separation between rea soning and execution.\n\n## What trade-offs did you consider?\n\nThe large st trade-off is local versus hosted.\n\n- Hosted APIs are often faster\, e asier to operate\, and provide access to more capable frontier models.\n- Self-owned AI provides control over data\, model versions\, routing\, netw ork boundaries\, tools\, upgrades\, and lifecycle.\n- I do not treat this as an either-or decision. I use both. The goal is to decide what belongs w here.\n\nOther trade-offs include:\n\n- Larger model versus smaller specia lized model\n- Model size versus actual task intelligence\n- Better model versus better prompt and context\n- Model capability versus harness capabi lity\n- Q4 versus Q8 versus higher precision\n- Always-loaded model versus load-on-demand\n- Dense versus MoE\n- Raw throughput versus heat and powe r\n- Peak benchmark performance versus stable daily operation\n- Private m odel versus public API fallback\n- Strict airgap versus controlled egress\ n- Broad agent access versus narrowly scoped tools\n- Convenience of conne ctivity versus size of the trust boundary\n- Building custom tooling versu s adapting existing open-source systems\n- Pulling the latest model or run time versus keeping a known-good version\n- Convenience of third-party com ponents versus the supply-chain risk they introduce\n\nThe main thing I op timize for is usefulness over time on hardware I actually own\, not peak p erformance during a benchmark.\n\n## How can this help other practitioners ?\n\nThe session should help practitioners avoid unnecessary hardware purc hases\, benchmark chasing\, and architectures where every problem is solve d by loading a larger model.\n\nThe main patterns I want attendees to take away are:\n\n- Treat self-owned AI as infrastructure\n- Choose models bas ed on the job they need to do\n- Measure behaviour on your own hardware an d workloads\n- Do not use parameter count as a substitute for testing inte lligence\n- Improve prompts\, context\, tools\, and harness design before assuming that the answer is a larger model\n- Test model upgrades like sof tware upgrades\n- Pin what works\n- Understand your power and thermal limi ts\n- Use smaller models when they are good enough\n- Train or adapt behav iour for narrow use cases instead of trying to put all knowledge into the model\n- Use tools for facts and deterministic work\n- Treat the agent har ness as part of the platform\n- Explicitly decide what agents can see\, ac cess\, and communicate with\n- Treat models\, runtimes\, containers\, plug ins\, and agent tools as part of your software supply chain\n- Keep agents constrained\n- Use hosted APIs where they genuinely make more sense\n- Ow n only as much of the stack as you are willing to maintain\n\nThe goal is to help people build small\, self-owned AI systems that are useful enough and boring enough to depend on.\n\n## Current state\n\nIn progress\, with open-source components and daily use across personal\, research\, home-lab \, workstation\, and small-office infrastructure. The linked projects are active works in progress and change regularly because I use them as daily drivers.\n\nAlso\, [I did a run of a tangential topic for security focused audience](https://anantshri.info/timeline/owasp-25th-anniversary-private- ai-infrastructure/). That talk was focused on giving overview and moving t owards security of AI. he proposed talk is a much deeper insight into what I have been building\, operating\, and measuring\, including the results and trade-offs.\n## Tags\n\n#privateai #selfhostedai #inference #agents #h omelab #mlops #platformengineering #devops #sre #security #llama #quantiza tion #airgapped #opensource #power #hardware #supplychain #tailscale #codi ngagents #casestudy #workinprogress GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072225Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/private-ai-operationalized- how-hard-could-it-be-6vtufkTudjKJLhMoygzTSx BEGIN:VALARM ACTION:display DESCRIPTION:Private AI operationalized: how hard can it be? in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Morning break DTSTART:20261120T055500Z DTEND:20261120T063000Z DTSTAMP:20261109T024109Z UID:session/4vkYYVDHD1SmCcxtHHFyvx@hasgeek.com SEQUENCE:1 CREATED:20261104T072719Z GEO:12.9634971;77.6380856 LAST-MODIFIED:20261104T072727Z LOCATION:Lawns - The Energy Resources Institute (TERI)\nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Morning break in Lawns in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Decoding at a million tokens when the KV cache lives in host memor y: what actually bounds the step DTSTART:20261120T063000Z DTEND:20261120T070500Z DTSTAMP:20261109T024109Z UID:session/PwKgVkEMYu81M6K9zqkHHz@hasgeek.com SEQUENCE:3 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261105T071522Z DESCRIPTION:# Decoding at a million tokens when the KV cache lives in host memory: what actually bounds the step\n\n**One-line summary.** A measured account of a full decode step on an A100 with the KV cache and its index in pinned host memory\, from 32k to one million tokens\, across six index designs: which component bounds the step\, how to time it honestly\, and t he one design change that made it 1.67x faster.\n\n## What problem are you addressing?\n\nAgent sessions now run to hundreds of thousands of tokens\ , and a server holds many of them at once. Their KV caches stop fitting be side the weights in GPU memory and move to host RAM\, which is what vLLM's CPU offload\, LMCache and similar layers do. At that point a decode step is bounded by what crosses PCIe\, and the largest stream is not the model weights or the selected KV rows but the index scan that ranks all n keys t o pick the top-k. For Qwen3-8B the 136-bit-per-token index that Double Spa rsity\, Loki and SparQ read is 160 MB per step at 32k and 5.1 GB at one mi llion tokens\, while the selected rows stay near 200 MB.\n\nI measured thi s regime end to end and then changed the index. Fathom stores the 4-bit K cache channel-major as bit planes and lets each query read only the planes it needs\, decided per channel by a water-filling rule. At one million to kens the step is 1.67x faster in GPU time than with the 136-bit scans and 2.5x faster than with a block-landmark index. The same design is not faste r once the index is back in HBM\, and the talk spends time on why\, becaus e that negative result decides whether it belongs in a given stack.\n\nWha t made it difficult: GPU time and wall-clock disagree by 7 to 25 percent i n a Python harness and the gap differs by method\; a gather of contiguous runs from pinned memory hits the link rate at 4 KB granularity while one c opy call per run gets 0.4 GB/s\; overlapping transfer with compute bought nothing because both gathers already ran at link rate\; and a bit-extracti on kernel does two to four times the integer work per byte of a nibble sca n\, which is invisible over PCIe and decisive in HBM.\n\n## Who is the int ended audience?\n\nInfrastructure and platform engineers who run or build LLM serving with long contexts or KV offload\, and SREs who have to explai n a latency number. Familiarity with attention and the GPU memory hierarch y is enough\; no research background needed.\n\n## What will the audience learn?\n\n**Instrumentation.** How to instrument an offloaded decode step (CUDA graphs\, profiler traces\, fixed-size run lists\, bit-exact checks b efore timing) and which number to report.\n\n**Costs.** A per-component co st table for six index designs at 256k to 1M tokens\, reproducible from a public repository.\n\n**A rule.** When a read-fewer-bits index pays off (i ndex in host memory) and when it does not (index in HBM)\, with the arithm etic-per-byte argument that makes it portable across GPUs.\n\n**Adoption.* * What such a store costs in memory (68 B per token per KV head) and what an LMCache or vLLM connector would need to adopt it.\n\n## Status of the w ork\n\nExperimental\, measured on rented A100 pods. Published as arXiv:260 9.17652 (September 2026)\; code\, every result file and the run chains are public at https://github.com/vivekkalyanarangan30/fathom. Not yet integra ted into a production serving stack\; the talk is explicit about that boun dary.\n\n## Outline (40 minutes)\n\n**Part 1\, 5 min.** The regime: sizes at 32k and 1M for an 8B model\, what fits where\, what an offload engine a ctually moves per step.\n\n**Part 2\, 10 min.** Measuring a decode step ho nestly: GPU time vs wall-clock\, the issue-cost trap\, sync-free accountin g.\n\n**Part 3\, 10 min.** Bytes are time over PCIe: the trace breakdown a t 1M\, six designs at the same link rate\, where the 1.67x comes from.\n\n **Part 4\, 5 min.** What did not help: pipelining\, launch-config sweeps\, alternative top-k\, and the HBM control where nothing is faster.\n\n**Par t 5\, 5 min.** Fitting it into a serving stack. Then Q&A\, 5 min.\n\n## Sp eaker\n\nVivek Kalyanarangan is Senior Technical Architect\, AI at IDfy\, where a 20-person team operates more than 40 production ML models across b iometric authentication\, document recognition and OCR\, fraud detection a nd large-scale NLP. He has 13+ years across analytics\, big data and deep learning. He is the author of Quantization and Fast Inference (Manning\, 2 026) and the freeCodeCamp course LLMs from Scratch\, and of the Fathom pap er this talk is drawn from.\n\nPaper: https://arxiv.org/abs/2609.17652\nCo de and measurements: https://github.com/vivekkalyanarangan30/fathom\n\n\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072021Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/decoding-at-a-million-token s-when-the-kv-cache-lives-in-host-memory-what-actually-bounds-the-step-PwK gVkEMYu81M6K9zqkHHz BEGIN:VALARM ACTION:display DESCRIPTION:Decoding at a million tokens when the KV cache lives in host m emory: what actually bounds the step in Auditorium (Ground Floor) in 5 min utes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:When ML serving becomes a distributed systems problem DTSTART:20261120T071000Z DTEND:20261120T074500Z DTSTAMP:20261109T024109Z UID:session/KTgFsJHYUpSnmz6qMBLG9n@hasgeek.com SEQUENCE:8 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073407Z DESCRIPTION:**Deploying an ML model is easy. Operating hundreds of them re liably is not.**\n\nAs ML serving moves into production\, the difficult pr oblems quickly move beyond inference itself:\n\n* How do you manage models consistently from creation to retirement?\n* How do you safely activate\, scale\, and roll out new versions?\n* What happens when replicas disappea r or become unhealthy?\n* How do you recover when infrastructure drifts fr om the desired state?\n* How do you keep Kubernetes complexity away from m odel owners?\n\nAt InMobi\, we faced these challenges while evolving our M L serving infrastructure into a Kubernetes-native control plane for **mode l lifecycle management**. Instead of asking model owners to manage individ ual Kubernetes resources\, we introduced a higher-level **ModelDeployment* * abstraction that captures the desired state of a model while the platfor m continuously reconciles that intent with the underlying infrastructure.\ n\nThis talk walks through that evolution—from deployment automation to a production control plane—and the distributed-systems problems that eme rged as the platform grew.\n\nWe'll follow a model through its journey:\n\ n1. Creating a model deployment\n2. Translating the declaration into Kuber netes resources\n3. Activating replicas and validating readiness\n4. Handl ing failures and recovery\n5. Scaling with changing traffic\n6. Rolling ou t new model versions\n7. Retiring deployments safely\n\nWe'll then explore the challenges that arise across a large fleet:\n\n* Designing idempotent reconciliation\n* Handling partial failures\, retries\, and infrastructur e drift\n* Coordinating scaling\, availability\, and rollouts\n* Isolating models and versions to limit blast radius\n* Keeping infrastructure conce rns behind the platform API\n\nWe'll also look at two aspects that are eas y to underestimate.\n\nFirst\, **the CRD becomes a platform API**. Once te ams depend on `ModelDeployment`\, its fields and semantics become a long-l ived contract. We'll discuss how to evolve that API as the platform and it s infrastructure change while preserving backward compatibility and user i ntent.\n\nSecond\, **readiness is more than a Kubernetes condition**. A mo del may need to load its artifacts\, initialize its runtime\, complete war mup\, pass health checks\, and become reachable through the serving path b efore it is safe to receive production traffic.\n\nFinally\, we'll share t he production lessons behind the platform: where abstractions helped\, whe re they introduced new failure modes\, how responsibilities were divided b etween the control plane and Kubernetes\, and why reliable automation is u ltimately about **designing for failure and recovery**.\n\n## Key Takeaway s\n### From Kubernetes Resources to Platform APIs\n\nLearn how to design a higher-level model deployment abstraction that captures user intent witho ut exposing every underlying infrastructure primitive.\n\n### Managing the Deployment Lifecycle\n\nSee how a control plane takes a deployment from c reation and activation through scaling\, version changes\, retirement\, an d recovery—without exposing the underlying Kubernetes resources.\n\n### Reconciliation as the Reliability Mechanism\n\nUnderstand how continuous r econciliation keeps desired and actual state aligned while handling retrie s\, partial failures\, unhealthy workloads\, interrupted operations\, and infrastructure drift.\n\n### Coordinating Scaling\, Availability\, and Rol louts\n\nExplore how autoscaling\, readiness\, disruption budgets\, replic a availability\, and version transitions interact—and why coordinating t hem is important for safe activation\, scaling\, and rollout.\n\n### Servi ng Isolation\n\nUnderstand how models and versions can be isolated so that failures\, scaling changes\, configuration updates\, and rollouts have a limited blast radius\, while still allowing the platform to coordinate dep endencies when required.\n\n### Evolving a Production CRD\n\nLearn why a C RD becomes a long-lived platform contract and how API evolution\, versioni ng\, and backward compatibility become important as the platform grows.\n\ n### Designing for Failure\n\nTake away practical lessons for building con trol planes where failure\, recovery\, retries\, and reconciliation are no rmal parts of the operating model rather than operational afterthoughts.\n \n## Audience\n\nThis talk is intended for:\n\n* Platform Engineers\n* MLO ps Engineers\n* SREs\n* Infrastructure Engineers\n* Kubernetes Engineers\n * Software Engineers building distributed systems\n\nIt will be particular ly relevant to teams building:\n\n* ML serving platforms\n* Internal devel oper platforms\n* Kubernetes operators and controllers\n* Model deployment infrastructure\n* Control planes for large fleets of production workloads \n\n#mlops #kubernetes #platformengineering #modelserving #distributedsyst ems #controlplanes #scalability #sre\n\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072202Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/when-ml-serving-becomes-a-d istributed-systems-problem-KTgFsJHYUpSnmz6qMBLG9n BEGIN:VALARM ACTION:display DESCRIPTION:When ML serving becomes a distributed systems problem in Audit orium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Lunch break DTSTART:20261120T074500Z DTEND:20261120T084500Z DTSTAMP:20261109T024109Z UID:session/Sk1W2e42n5TF1mxuziJBzp@hasgeek.com SEQUENCE:2 CREATED:20261104T073019Z GEO:12.9634971;77.6380856 LAST-MODIFIED:20261104T073029Z LOCATION:Dining area (1st floor) - The Energy Resources Institute (TERI)\n Bengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Lunch break in Dining area (1st floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Sponsored talk: [100+ clusters used by 100+ teams\, and a platform that’s starting to run itself] DTSTART:20261120T084500Z DTEND:20261120T090500Z DTSTAMP:20261109T024109Z UID:session/VcqVgmCsDGAWWx8mdbQTCW@hasgeek.com SEQUENCE:4 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073224Z DESCRIPTION:# Proposal template\n\n### Title\n100+ clusters used by 100+ t eams\, and a platform that's starting to run itself\n\n### Summary\nRoku r uns most of its cloud on RMCP\, a multi-cloud Kubernetes platform with 100 + clusters on AWS and GCP. In 10 minutes we'll show what that scale forced us to build\, how we give apps written by AI agents a safe place to run\, and what changed when an AI agent joined our platform team.\n\n### What p roblem are you addressing? What real engineering challenge\, question\, or experience is this session about?\n\nAt Roku\, 100+ internal teams ship t o 37 production Kubernetes clusters (100+ including dev and staging) acros s AWS and GCP. Most of those teams are not Kubernetes experts\, and they s houldn't need to be. Two problems come with that:\n\n- Running a fleet\, n ot a cluster. Anything done by hand\, one cluster at a time\, breaks somew here past a few dozen clusters. Upgrades\, namespaces\, access and capacit y all have to be automated and reviewable.\n- AI changes who writes softwa re and how fast. Anyone at Roku can now build an internal tool in an after noon with an AI agent\, and nobody reviews that code the usual way. At the same time\, AI agents are opening about half of the merge requests in our own platform repos.\n\nThe talk covers three answers: a paved road from o ne config file to production\, VibeGrid (a zero-trust cluster for AI-writt en apps)\, and Optimus (an AI agent that does real work on our team\, with humans approving what it proposes).\n\n\n\n### Who is the intended audien ce? \nPlatform Platform engineering\, SRE\, DevOps and infrastructure engi neers\, plus engineering leaders deciding how far to let AI agents into th eir platforms.\n\n### Level: \nIntermediate\n\n### List one or two practic al takeaways.\n - Give app teams one small config file that describes what the app needs\, and let the platform generate the Kubernetes manifests\, security checks and canary rollout. That is how 100+ teams ship without wr iting Helm charts.\n- Treat every AI-written app\, and every AI agent\, as an untrusted tenant. Design from the threat ("this app is compromised") r ather than the tool\, and put three independent controls in its way.\n\n## # What will you share?\n\n- Architecture and design decisions: the paved r oad (rmcp.yaml → build → generated manifests → canary across regions ) and the fleet tooling behind it (one cluster registry\, spot-first autos caling).\n- Git as the single source of truth: every cluster and every nam espace is a reviewed file in git\, applied by CI. Because the whole fleet is described in one place\, we can build more on top of it\, such as the s ervice catalog in Backstage.\n- Production experience: VibeGrid has 70+ ap ps from 20+ teams in the six months since the first one landed\, and RMCP repos see 3.5× more merge requests a month than a year ago\, about half o pened by AI agents.\n- Operational challenges: what to automate first when patching\, reviews and releases can't keep up with AI-generated change.\n \n### What is your experience with this problem?\n- Production system\n- I nternal engineering project\n- Hard-earned engineering lesson\n\n### What approaches failed\, disappointed\, or created unexpected problems?\n- Aski ng app teams to own Helm charts and raw manifests. Most teams aren't Kuber netes experts\, so onboarding was slow and every team solved the same prob lems differently.\n- Treating AI-written apps like any other internal app. The usual code review doesn't happen\, so the usual trust doesn't hold. T hat is what led to a separate\, locked-down cluster.\n\n\n\n### What will you do differently today?\n- Start from the threat model and the paved roa d on day one\, not after the fleet has grown. Put guardrails that an agent can't remove in place before letting AI agents touch the platform.\n\n### What trade-offs did you consider?\n- Why one config file instead of Helm charts? Helm gives teams full control\, but it also hands them all of Kube rnetes. A small rmcp.yaml says what the app needs and the platform generat es the rest\, so teams give up some flexibility in exchange for speed\, co nsistency and built-in security and observability.\n- Why a separate clust er for AI-written apps instead of namespaces on the shared clusters? A sep arate cluster with deny-by-default networking costs more to run\, but a co mpromised app has no network path to production at all.\n- Why keep humans approving the agent's changes? Optimus could merge its own changes\, but we chose speed with human approval over full autonomy.\n- Build vs. buy: R MCP is built on open-source Kubernetes\, Istio and managed cloud services. We built the glue (paved road\, registry\, namespace pipeline) because th at is where Roku's needs differ.\n\n\n### How can this help other practiti oners?\n- A design approach: model AI-written apps and AI agents as untrus ted tenants.\n- A pattern to adopt: one small config file per app\, with t he platform generating everything else.\n- A set of operational practices: a release train for cluster upgrades\, git as the single source of truth for clusters and namespaces\, and patching as a routine release.\n \n\n## # Current state\n- Production experience\n \n\n### Tags\n#platformenginee ring #kubernetes #sre #developerplatforms #multicloud #aiagents #security #casestudy\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072315Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/100-clusters-used-by-100-te ams-and-a-platform-thats-starting-to-run-itself-VcqVgmCsDGAWWx8mdbQTCW BEGIN:VALARM ACTION:display DESCRIPTION:Sponsored talk: [100+ clusters used by 100+ teams\, and a plat form that’s starting to run itself] in Auditorium (Ground Floor) in 5 mi nutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:What breaks on the way to 1\,000 agents: narrow privilege\, long-r unning jobs\, many tenants. DTSTART:20261120T091000Z DTEND:20261120T094500Z DTSTAMP:20261109T024109Z UID:session/BLPcfVWUukXJNrMdvsH8eL@hasgeek.com SEQUENCE:3 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261105T071647Z DESCRIPTION:**One-line summary**\nHow we took Slash\, Razorpay's agent pla tform\, from one shared worker pool to isolated tenants\, sandboxed runs a nd workload-shaped compute\, told as eight things that broke at scale and the decision each one forced.\n\n**What problem are you addressing?**\nPla tforms assume workloads are human-reviewed code\, requests are short and s tateless\, and every tenant trusts the platform the same way. Agents break all three:\n\n- Untrusted code. Each run executes code generated at runti me\, with a shell\, inside an agent harness we didn't write. A list of all owed tools can't fence a shell.\n- Long\, stateful work. Runs take ~10 min utes at the median and hours at the tail\, while pods are scaled\, deploye d and evicted under them.\n- Many tenants. Teams across Razorpay and exter nal companies build their own agents on one platform. Keys\, files and net work access must never cross between them.\n\nSlash today: 200+ agents\, ~ 10K tasks a day\, 98%+ success. About 3 in 10 tasks are started by a perso n\; bots\, schedules and CI start the rest.\n\nThe talk follows two questi ons:\n1. How do you keep tenants\, and runs inside a tenant\, apart\, when every agent has a shell?\n2. How do you find compute for long jobs whose demand swings ~20x through the day\, when CPU barely moves?\n\n**Intended audience**\nPlatform engineering\, SRE / DevOps\, infrastructure and secur ity engineers\, and engineering leaders building internal AI platforms.\n\ n**Level**\nIntermediate\n\n**Practical takeaways**\n- Enforce below the a gent. Prompts and tool lists are not boundaries\; the process\, file syste m\, network and credentials are. A run should hold references and short-li ved grants\, never keys.\n- Make every default refuse. In a multi-tenant s ystem a fallback is a tenant decision nobody made\; routers and data acces s should fail closed.\n- For long jobs on a queue\, load is waiting plus i n-flight work. Not visible queue depth\, and not CPU.\n- Every clock on a run must nest: workflow ceiling < run deadline ≤ claim lease ≤ queue v isibility\, with pod drain inside it. Check it at boot\, or runs lose work or execute twice.\n\n**What will you share?**\n- Tenancy architecture: on e shared control plane that holds no secrets\, and one isolated data plane per tenant (namespace\, queue\, warm pool\, workload role). Tenants are p re-built and declared like pods\, so onboarding is four values\, not ~15 –20 PRs.\n- Run containment: a sandbox in our agent gateway SDK that wra ps any coding harness. It enforces:\n - a kernel file allow-list over a c opied workspace\n - one network exit through a policy proxy with rules pe r calling program\n - blocked escape syscalls and a non-root user\n - mo del keys added outside the sandbox\n\n If it can't enforce\, the run does n't start.\n- Prompt-injection defence: checks at four points in every tur n (prompt\, tool call\, tool result\, reply)\, with walls underneath that hold when a check misses. We'll walk through how each finding from our pre -launch security review is now closed.\n- Compute for long runs:\n - queu es and warm pools split by workload shape\n - a per-turn setup budget\n - scale-down that counts in-flight work\n - a separate pool for fan-out w orkloads\n- Before/after numbers for each fix\, and the alternatives we re jected.\n\n**What is your experience with this problem?**\n- Production sy stem\n- Real incident or failure\n- Hard-earned engineering lesson\n\n**Wh at approaches failed\, disappointed\, or created unexpected problems?**\n- A tool list as the privilege boundary. In a pre-launch security review\, one injected prompt had the agent copy cloud metadata into a repo\, and ot hers read system files. Security held the launch. We moved enforcement bel ow the agent: credentials by absence\, a kernel-enforced sandbox\, one net work exit.\n- Fallbacks during a migration. While two generations of our d ispatcher ran side by side\, every convenience default (route by name\, a default sandbox\, a default account) silently picked a tenant. The router and the database guard now refuse instead of guessing.\n- One queue for ev eryone. People asking quick questions waited ~27 minutes for ~9 minutes of work\, stuck behind long CI jobs. Priority classes only reordered the lin e. Separate queues and pools by workload shape took that wait to ~0.\n- A warm pod is not a warm run. A trivial "hi" took 30–45 s while the model needed ~6 s\; the rest was sequential setup and tool health checks. Treati ng setup as a per-turn budget cut it by about a quarter.\n- An autoscaler that couldn't see work. Messages being worked on are hidden from the queue for hours\, so the autoscaler saw an empty queue and cut ~100 runs mid-fl ight. Scale-down now counts in-flight work.\n- CPU-based scaling. Agents m ostly wait on models and tools\, so CPU sat at ~15–25%\, and most pods w ere parked above demand.\n- Fan-out in a shared pod. One code review spawn ed ~6 agents within a second\, ran its pod out of memory and took the neig hbouring runs with it. Caps limit counts\, not blast radius\; reviews now get their own pool.\n- Clocks out of order. Three-hour runs lost their wor k at the deadline. Runs now release and resume from a checkpoint\, and ~95 % of them finish.\n\n**What will you do differently today?**\n- Choose the isolation unit (a data plane per tenant) before the first external tenant .\n- Keep keys out of runs from day one: model access only via a gateway\, credentials only via brokers.\n- Put containment below the harness: one p olicy for every harness\, failing closed.\n- Write the timing inequality f or every clock on a run\, and check it when a worker boots.\n- Scale long jobs on waiting plus in-flight work.\n\n**What trade-offs did you consider ?**\n- Warm shared pods vs. per-run microVMs: warm pods are fast and cheap \, with isolation at the tenant\; per-run containment comes from a kernel sandbox\, not a VM.\n- Copying the workspace into the sandbox vs. mounting it: we pay copy time on large repos so there is no path back to the host. \n- Pod network policy vs. a per-sandbox proxy: pod rules see one address for several runs and can't name hosts or programs.\n- Shared pre-warmed to ol connections vs. per-run setup: pooling is faster but carries stale per- user credentials\, so we cut setup time instead.\n- Bigger pods vs. a sepa rate pool for fan-out work: bigger pods hit node limits\, and the next wid er fan-out hits the same wall.\n- An off-the-shelf event-driven autoscaler vs. extending our own: the off-the-shelf one clashed with autoscalers oth er teams already run in shared clusters.\n\n**How can this help other prac titioners?**\n- Treat agents as untrusted\, long-running\, multi-tenant jo bs\, and know which platform primitives stop holding: tool allow-lists\, p od network policy\, queue depth\, CPU scaling\, default routes.\n- Copy th e patterns:\n - credentials by absence\n - fail-closed chokepoints\n - kernel containment below the harness\n - in-flight-aware scaling\n - nes ted run clocks\n- For leaders: decide the isolation unit and the key-handl ing model before the first external tenant\; both are expensive to retrofi t.\n\n**Current state**\nIn production at Razorpay since March 2026\, serv ing internal teams and external companies.\n\n**Tags**\n\n#agents #platfor mengineering #security #governance #developerplatforms #observability #inf erence #scalability #casestudy GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072409Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/what-breaks-on-the-way-to-1 -000-agents-narrow-privilege-long-running-jobs-many-tenants-BLPcfVWUukXJNr MdvsH8eL BEGIN:VALARM ACTION:display DESCRIPTION:What breaks on the way to 1\,000 agents: narrow privilege\, lo ng-running jobs\, many tenants. in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:When agents go rogue: building a zero-trust agent mesh DTSTART:20261120T095000Z DTEND:20261120T102500Z DTSTAMP:20261109T024109Z UID:session/Y8WcYkVHEjN95dvoGdZF43@hasgeek.com SEQUENCE:7 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073128Z DESCRIPTION:\n# ***When Agents Go Rogue: Building a Zero-Trust Agent Mesh* **\n## One line summary\nAI agents behave like distributed systems but lac k consistent runtime network controls\, creating a security gap in which p rompt injection can lead to data exfiltration. This session explores build ing an Agent Mesh around a reusable policy check for caller identity\, des tination\, and data\, enabling runtimes\, sidecars\, or existing proxies t o intercept\, observe\, and govern outbound agent traffic without modifyin g the underlying application code.\n\n### What problem are you addressing? \nOnce autonomous execution begins\, AI agents create unpredictable runtim e data paths that IAM does not fully govern. An agent might ingest malicio us instructions from untrusted content\, chain otherwise safe actions into a dangerous sequence\, or silently exfiltrate sensitive data to an unauth orized endpoint.\n\nInference platforms already decide which model to call \, which provider to use\, and how many tokens to spend. A separate securi ty question remains: which caller may reach which tool\, peer agent\, mode l\, or external service\, and what data may cross that hop?\n\nThe core en gineering challenge is containing this behavior at the network layer. Plat form teams need a unified way to intercept and govern agent egress without hardcoding security checks into every individual agent\, tool\, or framew ork.\n\nThe Agent Mesh provides a shared policy decision point for caller identity\, destination\, and data. A trusted runtime\, sidecar\, or existi ng proxy on the request path can invoke this check and enforce its decisio n.\n\n\n#### Who is the intended audience?\nPlatform engineers\, security engineers\, SRE and DevOps professionals\, MLOps engineers\, and developer s building agentic or MCP-style workflows.\n\n#### Level\nIntermediate to advanced.\n\n#### List one or two practical takeaways.\n\n1. How to treat autonomous AI agents as untrusted workloads and authorize their outbound c alls using trusted caller identity\, destination policy\, data rules\, and hop-level auditing.\n \n2. A reference architecture for implementing a Zero-Trust Agent Mesh around one reusable policy check per outbound hop\, enforced by a sidecar\, runtime\, or existing proxy.\n\n#### What will you share?\nI will share:\n\n1. The threat model for indirect prompt injectio n and agent-driven data exfiltration.\n2. YAML-based policies for approved destinations\, sensitive-data indicators\, and rate limits.\n3. Policy de cisions\, audit events\, and execution tracing.\n4. A live before-and-afte r demonstration:\nIn the demo\, an agent will process untrusted content\, access simulated sensitive data\, and attempt to send it to an external en dpoint. I will first run the workflow without enforcement.I will then enab le Agent Mesh and show the request being intercepted\, denied\, and traced without changing the agent’s code or prompts.\n\n\n#### What is your ex perience with this problem?\nExperiment or prototype\nResearch and investi gation\n\nAgent Mesh is an open-source\, experimental implementation devel oped to explore runtime policy enforcement and observability for autonomou s AI workloads.\n\n\n#### What approaches failed\, disappointed\, or creat ed unexpected problems?\nRelying solely on application-level IAM and promp t engineering did not provide sufficient security guarantees. IAM was insu fficient because it primarily governed access to resources but did not con trol the unpredictable flow of data during agent execution.\n\nModifying e very agent or tool integration to implement custom egress checks also crea ted operational overhead that was difficult to scale and maintain across d ifferent agent frameworks. Application-level checks could also be applied inconsistently or bypassed when autonomous workflows invoked tools through unexpected execution paths.\n\n#### What will you do differently today?\n Instead of trusting an agent’s internal reasoning or relying only on app lication-level controls\, we treat the agent as an untrusted workload.\n\n By moving enforcement to an infrastructure layer\, Agent Mesh can intercep t outbound traffic independently of the agent’s prompts or application c ode and apply zero-trust principles during runtime. This approach compleme nts\, rather than replaces\, IAM\, least-privilege access\, secure tool de sign\, and application-level authorization.\n\n\n#### What trade-offs did you consider?\nNetwork layer versus application layer: We chose infrastruc ture-level enforcement to provide consistent controls across different age nt frameworks\, knowing that the network layer has less semantic and appli cation-level context.\nPerformance versus security: Evaluating API and too l requests against destination and sensitive-data policies introduces late ncy. We had to balance this performance overhead against the need for runt ime inspection and enforcement.\nAvailability versus containment: Fail-clo sed enforcement provides stronger containment but may disrupt legitimate w orkflows when policy infrastructure is unavailable. Fail-open enforcement improves availability but increases security risk.\n\n#### How can this he lp other practitioners?\n1. ***A design approach:*** Applying service-mesh and zero-trust patterns to autonomous AI workloads.\n2. ***A way to evalu ate competing approaches:*** Understanding what network-level controls can and cannot protect against in agentic systems.\n3. ***A set of operationa l practices:*** Implementing destination policies\, workload identity\, au dit logging\, and distributed tracing for agent egress.\n\n#### Current st ate:\nExperimental/prototype\n\nPPT Link https://drive.google.com/file/d/1 JDLiLAoO7qPPZPHE0p5XVvOlGOJzTWVB/view?usp=sharing\n\n\n{Add your bio - who you are\; where you work}\nKanika Singhal is a Senior MTS in Flow Network Security at Nutanix with over 15 years of experience building distributed systems\, networking\, and security infrastructure. She has spent much of her career designing and implementing production-grade systems and contin ues to work hands-on with code and system design. Prior to Nutanix\, she w orked as a developer on SD‑WAN technologies at VMware.\n\n#agents #secur ity #zerotrust #platformengineering #observability #governance GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072438Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/when-agents-go-rogue-buildi ng-a-zero-trust-agent-mesh-Y8WcYkVHEjN95dvoGdZF43 BEGIN:VALARM ACTION:display DESCRIPTION:When agents go rogue: building a zero-trust agent mesh in Audi torium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Evening break DTSTART:20261120T102500Z DTEND:20261120T105000Z DTSTAMP:20261109T024109Z UID:session/CbA4jKb6txqbB4Q2M4gGf4@hasgeek.com SEQUENCE:2 CREATED:20261104T073256Z GEO:12.9634971;77.6380856 LAST-MODIFIED:20261104T073305Z LOCATION:Lawns - The Energy Resources Institute (TERI)\nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Evening break in Lawns in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Stop shipping AI on vibes: AI evals for agents DTSTART:20261120T105000Z DTEND:20261120T112500Z DTSTAMP:20261109T024109Z UID:session/AihyGwXUZBmF17ojMXCSMK@hasgeek.com SEQUENCE:10 CATEGORIES:To show a demo,A talk - 30-40 mins CREATED:20261104T072930Z DESCRIPTION:## About this session\n\nEvery team building with LLMs has liv ed the same story. The demo works beautifully\, everyone's impressed\, it ships. Then a real user asks a slightly different question and the answer comes back confidently wrong.\n\nSomeone tweaks the prompt to fix it and q uietly breaks three other things that nobody notices for weeks. The honest truth is that most AI features today are shipped on *vibes*: we skim a fe w outputs\, they look fine\, and we hope. That works right up until your s ingle prompt grows into a retrieval (RAG) pipeline\, and that pipeline gro ws into an *agent* that plans\, calls tools\, and makes decisions on its o wn. At that point "looks fine" stops being an answer.\n\n**AI evals** are how you replace hope with evidence\, a repeatable way to ask **"is my AI d oing the right thing?"** and get a number you can trust *before* your user s find out for you. This session tells the story of why that matters\, and how we built a platform to make evaluation the default\, not an afterthou ght.\n\n## First — what should an agentic eval actually check?\n\nHere's the catch with agents: the answer is only half the story. A plain prompt has one input and one output\, so you grade the output and you're done. An agent takes a *journey* — it interprets a goal\, makes a plan\, calls t ools\, reads the results\, and decides its next move. That means it can la nd on a right-looking answer through a completely broken path (it got luck y\, and it'll break tomorrow)\, or do everything correctly and still fail at the final step. So before you can trust an agent\, you have to grade th e *whole trajectory*\, not just the last reply.\n\nIn practice that boils down to a handful of plain-English questions you should be able to ask of *any* agentic system:\n\n- **Did it understand the goal?** - or confidentl y solve the wrong problem.\n- **Was the plan sensible?** - reasonable step s\, no aimless looping or backtracking.\n- **Did it call the right tools\, the right way?** - correct tool\, correct arguments\, nothing skipped or invented.\n- **Is each step actually correct?** - not just the final answe r\, but the reasoning and intermediate results along the way.\n- **Is the answer grounded?** - backed by what it really retrieved\, not made up.\n- **Did it stay safe?** - no destructive actions\, policy violations\, or le aked data.\n- **Did it finish the job?** - and at a reasonable cost\, late ncy\, and number of steps.\n\nOnce you can ask those questions consistentl y\, evaluation stops being a vibe check and becomes a checklist — and th at's exactly what we set out to make easy.\n\nWe have created an AI evals framework or platform that helps you evaluating your AI agents. I'll walk through it the way a team would actually adopt it. It starts with a small **SDK** you drop into your app: a couple of lines capture a full **trace** of everything the AI did\, every prompt\, model call\, tool\, and retriev al step\, so you finally get *observability* into the black box\; the same SDK manages your prompts centrally\, so you can **version\, update\, and roll back prompts without redeploying code**. On top of that sit three kin ds of evaluation. **Prompt evals** score a single prompt against known exa mples. **Pipeline evals** check a RAG system end to end\, did it fetch the right context\, and is the answer actually grounded in it? **Agentic eval s** judge multi-step agents\, did it choose the right tool\, follow a sens ible plan\, and avoid unsafe actions? The part people find most useful: yo u **design your own scoring**\, mixing simple rules with an *LLM-as-a-judg e* using your own rubric\, and run it two ways\, **offline** against a cur ated test set before you ship\, and **online** as live evals that continuo usly score real production traffic. When something does go wrong\, you can **chat with the live trace** to do root-cause analysis in plain English i nstead of spelunking through logs. **Safety gates** and confidence thresho lds tie it together\, so a change only goes live when the evidence says it 's safe.\n\n## Key takeaways\n\n1. **A mental model for evaluating *any* A I or agentic system.** Separate the three layers : prompt → pipeline → agent — decide what "good" means at each\, and combine deterministic ch ecks with LLM-as-a-judge scoring. You'll also learn why *offline* evals (b efore shipping) and *online* live evals (on real traffic) answer different questions\, and why you need both.\n2. **A concrete blueprint for product ion confidence.** How tracing\, prompt management\, custom scoring\, safet y gates\, and root-cause analysis fit into one workflow\, so you can go fr om "it looked fine in the demo" to shipping AI changes you can actually de fend with data.\n\n## Who should attend\n\n- **AI/ML engineers and data sc ientists** building LLM\, RAG\, or agentic features who want to measure qu ality\, not guess at it.\n- **Platform / backend engineers** responsible f or getting AI into production and keeping it reliable.\n- **Engineering le ads\, PMs\, and QA** who need to trust AI quality and de-risk every releas e.\n- **Anyone exploring agentic workflows** who wants a clear vocabulary for "how do we *know* it works?"\n\n\n## Bio\nI am Heet Gala\, working at Nutanix as a Software developer in the SaaS Oranization.\nlinkedin - https ://www.linkedin.com/in/heet-gala-030634191/\n\n## Draft slides\nhttps://do cs.google.com/presentation/d/1pP-T3K4TErpfThuqRz_q92YqzBxRwwsI8-2HRjtOS88/ edit?usp=sharing\n\n## Draft video\n{Add the link to 2-min elevator pitch video} GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072524Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/stop-shipping-ai-on-vibes-a i-evals-for-agents-AihyGwXUZBmF17ojMXCSMK BEGIN:VALARM ACTION:display DESCRIPTION:Stop shipping AI on vibes: AI evals for agents in Auditorium ( Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Check-in\; tea DTSTART:20261121T043000Z DTEND:20261121T044500Z DTSTAMP:20261109T024109Z UID:session/1CoZjySas6ZHxHUsV5p6V@hasgeek.com SEQUENCE:3 CREATED:20261104T073451Z LAST-MODIFIED:20261104T073505Z LOCATION:TERI\, Bangalore and online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Check-in\; tea in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Day 1 recap\; day 2 programme DTSTART:20261121T044500Z DTEND:20261121T045500Z DTSTAMP:20261109T024109Z UID:session/TXMyvVWAo3pqU454dyhDMN@hasgeek.com SEQUENCE:1 CREATED:20261104T073532Z GEO:12.9634971;77.6380856 LAST-MODIFIED:20261104T073534Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Day 1 recap\; day 2 programme in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Benchmarking AI SRE agents before they touch production DTSTART:20261121T050000Z DTEND:20261121T052500Z DTSTAMP:20261109T024109Z UID:session/T8mjtkJ6Nrz3Mq8DCUEdXa@hasgeek.com SEQUENCE:3 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073548Z DESCRIPTION:# Summary\nOver the past few months we've been evaluating AI S RE agents with several of our customers: OpenSRE\, HolmesGPT\, K8sGPT and a handful of others. We started with the things any SRE would check before trusting one of these on a real incident. That only took us so far\, so w e built a benchmark on top of the open-source SREGym framework\, where the agent's diagnosis is graded against what actually broke. Out of that work came a plug-and-play framework for evaluating AI SRE agents\, which we'll share.\nEverything ran on a production-grade setup\, so the results show how these agents behave under real conditions\, including where they fall short. You'll leave with numbers\, both qualitative and quantitative\, on what to expect from these agents in a real production environment. \n\n# W hat problem are you addressing?\nTeams adopting AI SRE agents need answers to questions that get harder as the agent gets closer to on-call. We set out to answer questions like does it find the right root cause\, does it s ay so when it can't see something\, what does an investigation cost\, what access does it need\, and does it know anything about our systems\, build ing security guardrails at the agent level through the Agent gateway. \n\n What problems we faced during the experiments: \n- AI agents sometimes did not failed loudly and were difficult to debug. A broken integration can s ometimes produces a confident root cause\, and an investigation that is go ing wrong looks the same as one that is going right until we look at it fr om production lense.\n- Continously hydrating the knowledge base so that a gent does not need to cold start everytime it starts a new investigation. \n\nTo compare approaches fairly\, we built one reference setup to showcas e our learnings:\n\n- iximiuz Labs challenges for the first\, typed-prompt experiments as a preliminary analysis.\n- A kind cluster with kube-promet heus-stack and demo microservices under load\, where real alertmanager ale rts flood start the investigations.\n- SREGym for the later ones\, which f or the fault injection and anlysis into a live system and has an LLM judge grading the diagnosis.\n- How different model effects the diagnosis.\n- T ekenomics of the incidence resolution.\n\n# Who is the intended audience?\ n- Platform Engineering\, SRE\, and DevOps engineers who run or are about to run AI SRE agents in production.\n- Engineering leaders deciding whethe r to bring AI into the SRE path\, whether for incident responses\, and ass essing quality RCA backed by evidences. \n \n# Level\nIntermediate. Assum es basics of Agentic loops\, MCPs\, LLM tool callings\n\n# Practical takea ways\nA comparison of different AI SRE tools\, design considerations when building your own AI SRE agents\, cost control patterns\, how to evaluate these AI SRE agents for production quantitatively.\n\n# What will you shar e?\n- How the open-source AI SRE agents compare with each other\, includin g HolmesGPT\, OpenSRE and K8sGPT: their architecture\, strengths\, weaknes ses and limitations.\n- The design trade-offs between general-purpose codi ng agents like Claude Code or Codex (with MCPs) and purpose-built AI SRE a gents for solving production problems\, along with our qualitative and qua ntitative results from the evaluation framework.\n- A demo of HolmesGPT an d K8sGPT running on the SREGym evaluation framework\, and a walkthrough of the tokenomics: where the tokens go and what each investigation costs.\n- How SREGym evaluates agents\, how the LLM-as-a-judge scoring works\, and the benchmark numbers we got.\n\n# Open-source tooling\nThe open source re po has locally runnable kubernetes and docker setups accompanied by demo s creenshots which can be used during the talk.\nhttps://github.com/one2nc/a i_sre\n\n# What is your experience with this problem?\n- Research / invest igation\n- Hard earned engineering lesson\n\n# What approaches failed\, di sappointed\, or created unexpected problems?\n\n- **Broken integrations pr oduced confident answers.**\n * During some of the evaluations\, even whe n the tool integrations were broken the agent still produced a confident l ooking result.\n * A stronger model did not help\, because the missing pi ece was context.\n- **Our own tests made the agent look better than it was .**\n * When we picked the faults using our local prometheus\, grafana st ack on a kind cluster\, HolmesGPT got 21 of 21 root causes right.\n * SRE Gym's decoys and noise brought that to 8 to 10 of 21. The agent latched on to the first anomaly and spent the rest of the run defending it.\n- **Guar ding the API calls to upstream systems during investigation**\n * How to secure/rate limit upstream APIs for e.g. Kubernetes control plane if ag ents get stuck in a loop and keep making expensive API calls to fetch the information. \n\n# How can this help other practitioners?\nA way to run an d evaluate AI SRE agents for production scale. \n\n# Tags\n#observability #agents #Tokenomics #sre #platformengineering #AI SRE #finops\n\n\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072605Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/benchmarking-ai-sre-agents- before-they-touch-production-T8mjtkJ6Nrz3Mq8DCUEdXa BEGIN:VALARM ACTION:display DESCRIPTION:Benchmarking AI SRE agents before they touch production in Aud itorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:How PhonePe’s in-house code review platform (Arnold) helped cont rol the code flood DTSTART:20261121T053000Z DTEND:20261121T061000Z DTSTAMP:20261109T024109Z UID:session/88836Ee3Cx8jumhgFFShxj@hasgeek.com SEQUENCE:2 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073604Z DESCRIPTION:## Learnings from building an in-house Code Review platform\n* *One-line summary - in 1–2 sentences\, describe the engineering problem\ , experience\, or question your session explores.**\nWe discuss how we des igned and scaled PhonePe’s custom agentic code review platform. Moving a way from fragile\, chat-based wrappers we engineered an async\, distribute d pipeline that orchestrates micro-agents for effective code review\, with out removing the Human-in-the-loop. We also talk about some key design dec isions while building the tool.\n\n**What problem are you addressing? What real engineering challenge\, question\, or experience is this session abo ut?**\nAs we navigate this era of frontier LLMs assisting all engineering functions in writing code\, organizations expect this to unlock true speed . Enable engineers with AI tools and harnesses \\- we will naturally obser ve a massive uptick in code generation volume.\n\nBut now that everyone is a 10x engineer\, why aren’t we shipping 10x products at 10x speed?\n\nT he ground reality however is that \\- while organizations have sped up the code generation part of the lifecycle\, the bottleneck has simply shifted down the SDLC pipeline to human code review. A classic realization of Amd ahl’s Law.\n\n> *“Amdahl's Law is a principle in computer science that calculates the theoretical maximum speedup of a system when only a portio n of that system is improved. It dictates that the overall performance gai n is ultimately limited by the sequential (un-improvable) fraction of the task\, showing the law of diminishing returns”*\n\nThis is not to sugges t that we should haphazardly integrate AI and assume that total automation of the code review process will speed things up\; maintaining a human-in- the-loop remains essential. We must now empower engineers with the right t ooling that is effective at improving this speed.\n\n**Who is the intended audience?**\nPlatform Engineering/Infrastructure/Developers\n\n**Level:** intermediate/advanced\n\n**List one or two practical takeaways.**\n- What factors constitute effective code reviews\, and are there ways to perfect this art.\n- Can any org take on building such a platform\n- Consider sel f hosting Arnold in your orgs. \n\n\n**What will you share?**\n- Why we we nt down this path\n- How this platform was built\n- Live demo\n- Open-sour ce tooling\n- Before-and-after experience\n\n\n**What is your experience w ith this problem?**\n- Production system\n- Internal engineering project\n - Open-source project\n\n**What approaches failed\, disappointed\, or crea ted unexpected problems?**\n- Justifying an investment of this stature was a hard problem.\n- Assuming that a simple AI loop was enough to get the b est result was incorrect.\n- Using Gitlab runners to do the the actual rev iew execution\, was a challenge.\n\n**What will you do differently today?* *\n- We discuss a few architectural decisions while building the platform\ , that could have shortened the delivery time\n\n**What trade-offs did you consider?**\nAs a heavily audited payments company\, PhonePe faced a clas sic build-vs-buy dilemma. Existing market solutions lacked affordable\, ai r-gapped setups and the flexibility our teams needed. We chose to build a custom platform that empowers diverse engineering teams while meeting stri ct company-wide standards. The talk goes deeper on this aspect\n\n**How ca n this help other practitioners?**\n- Design decisions when building in-ho use platforms\n- Applying similar principles of scaling a consumer product \, how we improved adoption.\n\n**Current state:**\n- Production experienc e\n- In progress\n- Open source\n\n#codereview #developerplatforms #design #platformengineering #gitlab #ai #agentic GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072641Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/how-phonepes-in-house-code- review-platform-arnold-helped-control-the-code-flood-88836Ee3Cx8jumhgFFShx j BEGIN:VALARM ACTION:display DESCRIPTION:How PhonePe’s in-house code review platform (Arnold) helped control the code flood in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Morning break DTSTART:20261121T061000Z DTEND:20261121T063000Z DTSTAMP:20261109T024109Z UID:session/mS5atgBNSeY7yfQNxmawz@hasgeek.com SEQUENCE:1 CREATED:20261104T073616Z GEO:12.9634971;77.6380856 LAST-MODIFIED:20261104T073619Z LOCATION:Lawns - The Energy Resources Institute (TERI)\nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Morning break in Lawns in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Grounding the agent: how to let LLMs reliably operate infrastructu re DTSTART:20261121T063000Z DTEND:20261121T070500Z DTSTAMP:20261109T024109Z UID:session/FEjvDcfEsb3kgCWAWfyVo6@hasgeek.com SEQUENCE:4 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073639Z DESCRIPTION:### Session title\nGrounding the Agent: How to Let LLMs Reliab ly Operate Infrastructure\n\n### One-line summary\nWe built an MCP server that sits between AI agents (Cursor\, Claude Desktop\, custom agents) and Nutanix Prism Central's v4 REST API. It serves pruned schemas\, code sampl es and live state on demand\, so the agent can only call operations that e xist in the v4 spec\, and writes stay off until an operator turns them on. \n\n### What problem are you addressing?\nLLMs are probabilistic. Infrastr ucture APIs are not. Connecting the two breaks in predictable ways:\n\n- * *Stale training data.** Ask a model for a Nutanix automation script and it writes against the older v3 API it memorised. System prompts saying "use v4" are a suggestion\, not a constraint.\n- **Specs that don't fit.** The v4 API is 1\,000+ operations across 19 namespaces. Request bodies are deep ly nested `$ref` graphs\; a single operation can expand to tens of thousan ds of tokens. Pasting the OpenAPI spec into context costs latency and mone y\, and often exceeds the window outright.\n- **No live state.** RAG over docs can tell you what `listClusters` means. It cannot tell you the live C PU headroom on cluster X during an incident.\n- **Blast radius.** The obvi ous fix\, giving the agent an HTTP tool\, gives it un-sandboxed access to a production datacenter\, with a model that confidently invents parameters .\n\nThe engineering question: how do you expose a large enterprise API to an IDE agent or an SRE agent in real time\, without handing it the keys t o everything?\n\n### Who is the intended audience?\nPlatform Engineering / SRE / DevOps / Infrastructure / Developers building agent tooling / Techn ical architects\n\n### Level\nIntermediate/Advanced\n\n### List one or two practical takeaways\n1. A pattern for putting an MCP server between an LL M and a REST gateway as a schema-pruning\, policy-enforcing adapter: a sma ll fixed tool set (23 tools\, not 1\,000) with discovery\, lazy schema exp ansion and server-side safety.\n2. Concrete techniques for API determinism and context budgeting: ranking operations from keyword tokens\, resolving `$ref` only on demand\, and making the spec itself the allowlist\, so a h allucinated endpoint fails before it reaches the datacenter.\n\n### What w ill you share?\n- **Architecture and design decisions:** why 23 tools (4 d iscovery + 19 per-namespace executors) instead of one tool per operation\; why dynamic tool calls beat a static vector index for live infrastructure \; where the server's job ends and the agent's job begins.\n- **Implementa tion details (from the repo):** `listOperations` ranks operations with fie ld-weighted token scoring (`_score_operation` in `src/generators/tool_gene rator.py`)\; `getOperationSchema` expands `$ref` lazily with a cycle-safe resolver (`src/generators/schema_resolver.py`)\; `getCodeSample` and `getO perationPermissions` give the agent a verified example and the required ro les before it executes anything\; `src/tools/runtime_dispatcher.py` valida tes every call against the loaded v4 operation set.\n- **Safety mechanics: ** `READ_ONLY_MODE` enforced at the dispatcher rather than in the prompt\; 401/403 bodies sanitised so credential and realm details never reach the model\; Prism Central's own RBAC bounding what the configured credential c an do.\n- **Production challenges:** token bloat from schemas (stripping ` readOnly` fields and `$UNKNOWN`/`$REDACTED` enum noise)\; PC errors that w ere reaching the model as successful results\; and how we think about conc urrency for an I/O-bound server.\n- **Live demo:** Cursor or Claude Deskto p connected to a live Prism Central through the server: discover an operat ion from a plain-English request\, pull its pruned schema and sample\, que ry live state\, and refactor a legacy v3-style script into a v4-compliant one.\n\n### What is your experience with this problem?\nInternal engineeri ng project / Open-source project / Hard-earned engineering lesson\n\n### W hat approaches failed\, disappointed\, or created unexpected problems?\n1. **Registering every operation as a tool.** 1\,000+ tool definitions swamp the context and degrade tool selection. Abandoned for the fixed 23-tool m odel with ranked discovery.\n2. **Expanding schemas eagerly.** One operati on's `$ref` graph could dominate the whole prompt. Lazy resolution plus pr uning fixed it.\n3. **Trusting the prompt to enforce v4.** Instructions he lp\, but a model can still reach for remembered v3 shapes. What worked was removing the option: the executor only accepts operation IDs from the v4 artifacts.\n4. **Returning Prism Central errors as normal results.** 4xx/5 xx responses came back as `ok=true`\, so the agent kept going after a fail ed call. This is a known gap we are closing\, and a good example of how an agent can only recover from failures it can see.\n5. **A new HTTP client per call.** Fine for a demo\; connection churn under concurrent agents. Mo ving to a pooled async client.\n\n### What will you do differently today?\ n1. Make error fidelity part of the tool contract from day one: a failed i nfrastructure call must look like a failure to the model.\n2. Put safety i n the server (read-only mode\, and a per-operation allow/deny policy which is on our GA backlog)\, never only in the prompt.\n3. Keep the default sc ope to read/query and code generation\, with write operations an explicit operator decision.\n4. Resolve everything lazily (schemas today\, the oper ation index next) so startup and per-call cost stay flat as the API grows. \n\n### What trade-offs did you consider?\n- **Open standard vs. custom wr appers:** MCP gives native compatibility with Cursor\, Claude Desktop and custom agents. We gave up proprietary\, deeper integrations with any one c lient.\n- **Live API vs. pre-indexed RAG:** live calls cost network latenc y but are the only way to answer operational questions about current state . RAG stays useful for docs\, not for state.\n- **Lexical ranking vs. embe ddings:** token scoring has no model or vector-store dependency and is deb uggable. We gave up synonym matching and plan to add an alias map first\; embeddings only if golden-query tests show lexical ranking is not enough.\ n- **Implementation language and runtime:** the workload is almost entirel y I/O wait on Prism Central\, so the choice of language matters less than connection reuse and async I/O. We optimised for iteration speed and kept the option to revisit this as scale demands.\n- **Single service credentia l vs. per-user passthrough:** simpler to operate and audit\, but the agent acts with the credential's role\, not the end user's. Operators have to s cope that credential deliberately.\n\n### How can this help other practiti oners?\n- A reusable design for making an on-prem or cloud platform API us able by agents without flooding the context.\n- A blueprint for bounding a n LLM integration with existing IAM: scope the credential\, enforce read-o nly in the server\, sanitise auth errors.\n- Operational practices for con text budgeting on a large API surface\, and a way to decide when ranking o r embeddings are actually warranted.\n\n### Current state\nIn progress. Wo rking open-source server in technical preview state moving toward GA.\n\n- Repository: https://github.com/nutanix/ntnx-api-mcp-server/tree/main\n- D eep-dive tech blog: https://www.nutanix.com/blog/mcp-server-for-nutanix-ap i-is-now-available-in-technology-preview#\n- Slides: https://docs.google.c om/presentation/d/1LlakV-JATwQy9cB2nyrfOShlO1-xaeSPi-_Iq3RBa5s/edit?usp=sh aring\n### Tags\n#platformengineering #agents #infrastructure #sre #develo perplatforms #automation #security #demo #mcp #workinprogress\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261105T072758Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/grounding-the-agent-how-to- let-llms-reliably-operate-infrastructure-FEjvDcfEsb3kgCWAWfyVo6 BEGIN:VALARM ACTION:display DESCRIPTION:Grounding the agent: how to let LLMs reliably operate infrastr ucture in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Agents are easy\, memory Is hard DTSTART:20261121T071000Z DTEND:20261121T074500Z DTSTAMP:20261109T024109Z UID:session/2qcxh1s9HSJ88ePnSGkwwt@hasgeek.com SEQUENCE:3 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261106T103214Z DESCRIPTION:# Agents Are Easy\, Memory Is Hard: What Our SRE Agent Is Allo wed to Remember\, and How We Check It\nSession title\nAgents Are Easy\, Me mory Is Hard\n\nOne-line summary\nOur SRE agent answers about 95% of the s upport requests it gets\, but when someone corrected it in one thread\, th e next thread never knew. This talk covers how we gave it a memory we can check and undo\, and why the easier versions didn't work.\n\nWhat problem are you addressing?\n\n1. We run an agent in our platform team's support c hannel. People ask it for Grafana or Jenkins access\, DNS changes\, or hel p with things that are broken right now\, like a stuck rollout\, a failing build or an alert that keeps firing. It looks at the live systems with re ad-only access and replies with what's wrong\, how to fix it and the comma nd to run. The one thing it does on its own is run a small access pipeline . It also picks up PagerDuty incidents. Workers scale with the size of the queue\, so up to five requests run at the same time.\n2. It answers about 95% of what comes in\, so getting a reply isn't the issue. Getting the ri ght one is. When someone corrected the agent\, the correction stayed in th at thread. A teammate once told it to check every cloud account before say ing an IP isn't ours\, and the next question started from scratch.\n3. We tried the two obvious fixes first. Adding lessons to the prompt means a co de change and a deploy each time\, and every request then carries every le sson. Pulling in similar old answers brought back stale text that didn't s ay it was old\, and once the agent quoted a different thread as if it were the current one. Neither approach had a way to take a wrong entry back ou t.\n\nWho is the intended audience?\nSRE\, platform engineering and DevOps people\, and anyone building or running LLM agents.\n\nLevel\nIntermediat e\n\nList one or two practical takeaways\n1. Store how to investigate a ty pe of request\, not the answers. A checked playbook helps with requests yo u haven't seen. A cached answer only helps if the same question comes back \, and it goes stale.\n2. Don't trust whatever proposes a memory. Re-run e ach candidate lesson from scratch before you store it\, and build the way to remove it at the same time as the way to add it.\n\nWhat will you share ?\n\n- How the agent works. The four ways to reach it (a mention\, a chann el message\, a thread reply\, a PagerDuty incident)\, the request flow\, a nd the autoscaling: at least 2 workers\, at most 5\, scaled on queue depth . Several workers can run at once safely because replies are idempotent an d the queue's visibility timeout is longer than our time limit.\n- How the feedback loop works. Once a thread goes quiet\, we draft lessons from it. Each one is checked with a fresh run\, and only the ones that pass are sa ved. Similar questions get them as a hint. A thumbs-down or a contradictio n removes a lesson\, and only one service is allowed to write.\n- What wen t wrong\, with numbers. Corrections that got merged into one lesson\, a th orough run that passed a wrong lesson\, old lessons hiding new ones\, abou t one in six playbooks being near-duplicates\, and a vector index that fou nd about 1 of 190 real matches.\n- One lesson from start to finish\, from the thread it came from to the answer it later shaped\, with the architect ure diagrams.\n\nWhat is your experience with this problem?\n\nProduction system\, real incidents and failures\, internal engineering project\, hard -earned lessons.\nIt's in daily use. Saved playbooks match roughly three i n ten live questions\, and we've traced wording in real answers back to th e exact lesson behind it.\n\nWhat approaches failed\, disappointed\, or cr eated unexpected problems?\n\n- Putting every lesson in the system prompt. \n- Retrieving raw old threads as memory.\n- One merged lesson per correct ion. One test covered one of four points\, so the whole lesson failed.\n- Letting the classifier see existing lessons. It proposed fewer new ones.\n - Scaling workers on CPU. CPU only goes up while a run is active\, not whi le a request is waiting.\n- An approximate vector index on a small table.\ n- Using the prompt to limit what the agent can write. It once told a user it had saved something to memory when it had no way to.\n\nWhat will you do differently today?\n\nI'd decide what a lesson is\, who can write it an d how it gets removed before saving anything. I'd set the similarity thres hold from real duplicate pairs instead of guessing. And I'd record from da y one which lesson shaped each answer\, so a bad rating can point at it.\n \nWhat trade-offs did you consider?\n\n- Playbooks instead of cached answe rs: slower on exact repeats\, better on new requests.\n- A fresh re-run fo r every candidate: it costs a full investigation\, but it doesn't depend o n whatever proposed the lesson.\n- Draft pull requests for missing tools\, reviewed by a person: slower\, but the agent can't give itself permission s.\n- A cap of five workers: lower peak throughput\, but cost and load on the systems it queries stay bounded.\n- Exact search instead of an approxi mate index until the table reaches thousands of rows.\n\nHow can this help other practitioners?\n\nIt's a worked example of agent memory: what a les son is\, who can write it\, how it's checked and how it's removed. It also has a warning about measuring. A high response rate says nothing about ac curacy\, so measure answer quality separately.\n\nCurrent state\nProductio n experience\n\nTags\n#agents #sre #observability #memory #platformenginee ring #devops #kubernetes #casestudy #failurestory #governance GEO:12.9634971;77.6380856 LAST-MODIFIED:20261107T083530Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/agents-are-easy-memory-is-h ard-2qcxh1s9HSJ88ePnSGkwwt BEGIN:VALARM ACTION:display DESCRIPTION:Agents are easy\, memory Is hard in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Lunch break DTSTART:20261121T074500Z DTEND:20261121T084500Z DTSTAMP:20261109T024109Z UID:session/HE8VphPU2tPifennkc3bxd@hasgeek.com SEQUENCE:1 CREATED:20261104T073717Z GEO:12.9634971;77.6380856 LAST-MODIFIED:20261104T073720Z LOCATION:Dining area (1st floor) - The Energy Resources Institute (TERI)\n Bengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Lunch break in Dining area (1st floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Upgrade broke my query! Break free with AI - detect\, debug\, fix with AI-assisted self-service platform DTSTART:20261121T084500Z DTEND:20261121T091000Z DTSTAMP:20261109T024109Z UID:session/VKdD1xiRHihpvSN3L9Ej9N@hasgeek.com SEQUENCE:7 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073659Z DESCRIPTION:## Session title\n>**Upgrade Broke My Query… Break Free with AI**\n Detect\, Debug\, Fix with AI-Assisted Self-Service Platform\n\n## One-line summary\nIn this session\, we will explore why and how data platf orm teams can build a self-service performance analysis platform – autom ating upgrade testing\, regression detection\, and debugging to reduce pla tform-team intervention and help application teams identify and resolve pe rformance issues independently.\n\n## What problem are you addressing?\n\n As a small central data platform team\, our core mandate is to ensure the databases we support run seamlessly across all application teams. Fulfilli ng this exposed a major bottleneck during ClickHouse version upgrades: to identify and debug query regressions\, we had to simulate each team's work load in our lab and deeply understand their unique business logic. Recogni zing this hands-on approach was completely unscalable\, we set out to buil d a self-serve platform that empowers application teams to run their own w orkloads and identify regressions independently of the platform team.\n\nW e named this fully self-serve platform **Velocity**. Velocity expanded wel l beyond its original scope\, not only does it run workloads against multi ple ClickHouse versions to pinpoint degraded queries\, but it also provide s a sandbox for application teams to proactively test their own query opti mizations independently without relying on our central team.\n\nYet\, iden tifying a degraded query is only half the job. To truly solve the problem\ , teams needed the root cause analysis (RCA) and specific remedial steps t o fix performance drops which again needed platform team involvement. To m aintain our goal of a fully self-serve ecosystem\, the platform evolved on ce more. We built **Sherlock** directly on top of Velocity – an AI-assis ted query regression investigation system that automatically generates the RCA and provides the exact steps needed to get query performance back on track.\n\n\n\nClickHouse evolves rapidly\, upgrades can make a set of quer ies unexpectedly slower without causing any functional failures. These reg ressions are often discovered late in the release cycle\, when fixing them becomes expensive.\nOur initial approach relied heavily on a central plat form team to run tests and investigate regressions\, which did not scale a s more application teams and workloads were added.\nThis led to a key ques tion: Can performance testing and regression investigation become a self-s ervice capability for application teams?\n\n**Velocity was our answer.**\n \nThe platform provides a common execution environment where application t eams can run their workloads against ClickHouse versions\, compare results \, identify regressions\, and investigate the reasons behind them.\nWe the n faced a second problem: detecting a regression is only half the job. An application team needs to know why the query regressed and what they can d o about it.\nThat led to Sherlock — an AI-assisted query regression inve stigation system built on top of Velocity.\n\n## Who is the intended audie nce?\nPlatform Engineering / Infrastructure / Developers / Data & Backend Engineers / Engineering leaders\n\nThe talk is particularly relevant for t eams that operate shared data platforms and need to support multiple appli cation teams with different workloads.\n\n## Level\n**intermediate**\nThe talk assumes familiarity with databases\, performance testing\, and basic platform concepts\, but does not require deep ClickHouse internals or AI/L LM expertise.\n\n## Practical takeaways\n- A framework for going beyond re gression detection: combine workload execution\, query telemetry\, databas e-version changes\, best practices\, settings\, and AI-assisted investigat ion to turn “this query got slower” into “this is likely why\, and h ere is what you can try.”\n\n## What will you share?\nCode or implementa tion details\n\n## What is your experience with this problem?\nReal incide nt or failure\n\n## What will you do differently today?\nWe initially buil t Velocity and Sherlock specifically for ClickHouse performance regression analysis. As we developed the platform\, we realized the same approach co uld be generalized beyond ClickHouse to other databases\, streaming system s\, and data services.\nToday\, we would design the platform with this bro ader scope from the start—keeping the execution\, regression detection\, context collection\, and AI-assisted analysis extensible across different data systems.\n\n## What trade-offs did you consider?\nWe balanced centra lized control vs. self-service and chose self-service for better scalabili ty\, accepting the added platform complexity.\nWe also chose diagnosis ove r simple benchmarking—building an RCA AI agent to explain regressions ra ther than just report numbers.\nFor AI analysis\, we chose specialized age nts over one general-purpose agent to provide more focused and actionable recommendations.\n\n## How can this help other practitioners?\nA new way o f thinking about the problem\n\n## Current state\n**Production experience* *\n\n**#clickhouse #databases #performanceengineering #platformengineering #developerplatforms #selfservice #kubernetes #ai #agents #llm #observabil ity #devops #infrastructure #scalability #regressiontesting #benchmarking #casestudy #workinprogress**\n\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261106T103240Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/upgrade-broke-my-query-brea k-free-with-ai-detect-debug-fix-with-ai-assisted-self-service-platform-VKd D1xiRHihpvSN3L9Ej9N BEGIN:VALARM ACTION:display DESCRIPTION:Upgrade broke my query! Break free with AI - detect\, debug\, fix with AI-assisted self-service platform in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Surviving the query-pocalypse! DTSTART:20261121T091500Z DTEND:20261121T095000Z DTSTAMP:20261109T024109Z UID:session/ANUFPAWUKGvH2TzkNWT6mK@hasgeek.com SEQUENCE:4 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073242Z DESCRIPTION:# Proposal template\n\nWe have a new user persona - AI Agents. What does it mean for platform availability\, reliability & performance?\ n\nWe have been building systems for human as a user for a long time\, whi ch led to emergence of patterns to measure availability\, reliability\, SL Is/SLOs/SLAs in a certain manner. For example\, a user seeing a page loadi ng for more than 30s judges the application to be too slow and may churn. Nowadays\, more and more consumption of any application is moving to AI ag ents consuming the application via CLI/MCP and humans reading the final su mmary agents write. Combine that with agents relentless-ness of getting th e task done\, and forgetful-ness of what they did in last session\, introd uces a set of new challenges in application design:\n\n- An expensive API is not distinguishable from a fast API for agents.\n- Agents exercise vari ous query shapes\, they will keep trying what gets their task done.\n- Age nts can learn to use inefficient query shapes (memory layers)\, even if ap plication improves\, older query shapes could continue arriving.\n- Agents don't report bugs\, just workaround issues. \n\nIn a way\, AI agents mak es Hyrum's law become applicable for your application at much sooner rate. This talk covers insights of how agent queries differ from human queries taking examples from a real world Observability platform and what we have done in the platform from availability & reliability point of view. I beli eve this is a relevant topic for many applications whose primary user is b ecoming AI agents.\n\nA rough set of items I have in mind are:\n1. What to monitor? - APIs returning empty results (agents got params wrong?)\, wide r queries (.*\, wide time ranges)\, errors (obviously!)\n2. Error messages - give hints in error messages so that Agents can correct themselves. On that note\, give performance hints as well!\n3. Consider fast path and slo w path based on the source of the query - is a human waiting for the query or it is an AI agent? E.g. we route queries to lambda for higher parallel ization and performance\, do we use lambdas for AI agent originated querie s.\n\n## Intended Audience\nPlatform Engineers\, Application Engineers\, S RE/DevOps\n\n## Level\nBeginner (easy to understand talk)\n\n## Takeaways\ n1. Design applications considering AI agents as the user.\n2. Monitor hum an usage v/s AI usage differently.\n\n{What will you share?\n\n- Architect ure or design decisions\n- Production experience\n- Failure modes and debu gging\n\n{What is your experience with this problem?\n- Production system\ n- Hard-earned engineering lesson\n\n{What approaches failed\, disappointe d\, or created unexpected problems?}\n\n{What will you do differently toda y?}\n\n{What trade-offs did you consider? For example\, \n- Why one archit ecture over another?\n- Why build vs. buy?\n- What did you optimize for?\n - What did you knowingly give up?\n- What alternatives did you consider?}\ n\n{How can this help other practitioners?\n- A design approach\n- A debug ging technique\n- A pattern to adopt\n- A mistake to avoid\n- A way to eva luate competing approaches\n- A set of operational practices\n- A new way of thinking about the problem}\n\n{Current state - choose one:\n- Producti on experience\n- In progress\n- Experimental/prototype\n- Open source\n- R etrospective/lessons from a previous system}\n\n{Add tags to help us under stand your proposal. Tags may describe domains\, techniques\, systems\, or formats\, for example: #observability #inference #agents #security #gover nance #finops #sre #devops #platformengineering #mlops #databases #scalabi lity #kubernetes #developerplatforms #failurestory #demo #workshop #casest udy #workinprogress} GEO:12.9634971;77.6380856 LAST-MODIFIED:20261106T103226Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/surviving-the-query-pocalyp se-ANUFPAWUKGvH2TzkNWT6mK BEGIN:VALARM ACTION:display DESCRIPTION:Surviving the query-pocalypse! in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:3 weeks to 3 hours: building an AI co-pilot for Kubernetes upgrade preparation DTSTART:20261121T095500Z DTEND:20261121T103000Z DTSTAMP:20261109T024109Z UID:session/6QWPVeXF86WUm9V2RC1XsA@hasgeek.com SEQUENCE:6 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073752Z DESCRIPTION:# 3 Weeks to 3 Hours: Building an AI Co-Pilot for Kubernetes U pgrade Preparation at Booking.com\n\n## One-line summary\n\nEvery Kubernet es version upgrade costs our platform team 3–4 weeks of manual preparati on — reading changelogs\, chasing compatibility matrices\, and diffing c lient-go commits across 50+ repositories and 1000+ services. We built an A I-assisted intelligence system at Booking.com that does the analysis in ho urs\, validated it against a real upgrade\, and it caught everything the t eam found manually — plus issues they'd missed.\n\n---\n\n## What proble m are you addressing?\n\nKubernetes upgrades are an invisible tax on every platform team. At Booking.com\, we operate one of the largest Kubernetes platforms in the travel industry. Before a single cluster is touched\, eng ineers spend weeks doing something that looks suspiciously like reading co mprehension homework — at massive scale.\n\n**The numbers tell the story :**\n\n- **3–4 weeks** of senior engineering time consumed per Kubernete s version upgrade — just for preparation\n- **Hundreds of upstream chang es** per Kubernetes release to manually triage\n- **50+ internal repositor ies** to cross-reference against every change\n- **Multiple custom control lers and operators** with client-go dependencies to analyze\n- **Dozens of ecosystem dependencies** (Karpenter\, Argo CD\, CoreDNS\, VictoriaMetrics \, cert-manager\, etcd\, NVIDIA GPU components\, and more) — each with t heir own compatibility matrix scattered across separate upstream repos\n- **~3 Kubernetes releases per year**\, meaning this 3–4 week tax repeats multiple times annually\n- **Effort scales worse than linearly** — every new controller\, dependency\, or cluster added to the platform makes the next upgrade harder\n\nWhat makes this genuinely hard:\n\n- **Client-go pr ovides no upgrade guide.** Engineers literally diff commit histories acros s releases to find what changed in the specific subset of packages they us e. At Booking.com's scale\, this alone can take days.\n- **Compatibility i nformation has no single source of truth.** It's scattered across dozens o f upstream repos\, release notes\, GitHub issues\, and documentation sites .\n- **The process depends on institutional knowledge** that lives in peop le's heads\, not in systems. When those people are unavailable\, upgrades stall.\n\n**We've felt the consequences of missing things.** At Booking.co m\, a cert-manager incompatibility broke customer SRM workloads. An etcd c hange killed Flink jobs in production. An AMI change broke Terraform provi sioning across clusters. Every one of these was discoverable in advance — if anyone had the bandwidth to look.\n\nWe built **Kubernetes Upgrade Intelligence**: a three-track AI-assisted system that reads upstream chang es\, understands our platform's specific usage patterns across all 50+ rep ositories\, identifies what actually matters to Booking.com\, and generate s remediation — while keeping engineers in control of every decision.\n\ nWe validated it against a real Kubernetes 1.33 upgrade at Booking.com. ** It found everything the team found manually\, plus additional issues they' d missed.** It has been used for 1.34 and 1.35 upgrade cycles successfully .\n\n---\n\n## Intended audience\n\nPlatform Engineering / SRE / Infrastru cture / DevOps\n\n---\n\n## Level\n\nIntermediate\n\n---\n\n## Practical t akeaways\n\n1. **A reusable architecture pattern** for combining determini stic code analysis (AST parsing\, dependency graphs\, Sourcegraph queries across 50+ repos) with LLM reasoning — so you get AI that's grounded in facts\, not hallucinations. Tested at Booking.com scale.\n\n2. **A concret e playbook** for building AI-assisted upgrade preparation for your own Kub ernetes platform — including what to feed the model\, what to keep deter ministic\, where humans must stay in the loop\, and how to measure if the AI is actually helping.\n\n---\n\n## What will you share?\n\n- **Architect ure and design decisions** — how the three-track system works at Booking .com\n- **Production experience** — real results from the Kubernetes 1.3 3 upgrade cycle\n- **Before-and-after comparison** — manual vs. AI-assis ted upgrade prep (weeks vs. hours)\n- **Failure modes and debugging** — what the AI gets wrong and how we catch it\n- **Trade-offs and alternative s** — why we made the design choices we did\n- **Live demo** — the cha ngelog intelligence pipeline analyzing real Kubernetes changes against a r eal codebase\n\n---\n\n## Experience with this problem\n\n- **Production s ystem** at Booking.com — one of the world's largest online travel platfo rms\n- **Real incidents and failures** — customer-facing breakages from missed upgrade risks\n- **Internal engineering project** — approved thro ugh Booking.com's Gen AI intake process\n- **Hard-earned engineering lesso n** — built from years of painful manual upgrade cycles (versions 1.25 t hrough 1.31\, each with its own preparation document)\n\n---\n\n## What ap proaches failed\, disappointed\, or created unexpected problems?\n\n**Naiv e LLM approach:** Feeding full Kubernetes changelogs to an LLM produced pl ausible-sounding but unreliable results. The model would confidently flag irrelevant changes and miss critical ones buried in dense release notes. A t Booking.com's scale — where a false positive wastes senior engineering time across multiple teams and a false negative can cause production inci dents — this was unacceptable. Pure LLM inference without grounding in o ur actual codebase was essentially a hallucination generator with a profes sional tone.\n\n**Fully deterministic approach:** Static analysis\, regex- based changelog parsing\, and version constraint solvers were precise but brittle. They caught explicit API removals but missed behavioral changes\, subtle deprecation paths\, and the kind of cross-cutting impact that requ ires understanding context across 50+ repos\, not just matching strings.\n \n**Brute-force client-go diffing:** Early attempts at automated diffing p roduced thousands of changes with no signal about which ones mattered to o ur actual usage patterns. Engineers ended up doing more triage work than t he manual process required.\n\n---\n\n## What will you do differently toda y?\n\nWe now treat **deterministic analysis and AI reasoning as complement ary stages**\, not competing approaches.\n\nDeterministic tooling establis hes facts first — which APIs we use across all repositories\, which mani fests reference deprecated resources\, which dependency versions we run\, which client-go packages are actually imported. AI then interprets those f acts in the context of upstream changes to assess impact\, prioritize risk s\, and draft remediation.\n\nThis **"facts first\, reasoning second"** ar chitecture — validated at Booking.com against real Kubernetes upgrades — dramatically reduced false positives and eliminated the most dangerous category of hallucinations.\n\n---\n\n## Trade-offs considered\n\n| Decis ion | What we chose | What we gave up | Why |\n|---|---|---|---|\n| **Anal ysis approach** | Hybrid: deterministic fact-finding feeds AI reasoning | Pure AI (faster to build) or pure static analysis (more precise) | At Book ing.com's scale\, we needed both coverage and accuracy |\n| **Automation l evel** | Human-in-the-loop at every decision point | Fully autonomous upgr ades | Platform upgrades are safety-critical — the cost of a wrong autom ated decision far exceeds a human review step |\n| **AI input sources** | Official changelogs\, EKS release notes\, upstream Git history only | Broa der web knowledge\, blog posts\, Stack Overflow | We'd rather miss an edge case than act on a hallucination in production |\n| **System architecture ** | Three specialized tracks (changelog\, client-go\, dependencies) | Sin gle monolithic pipeline | Each track has fundamentally different source ma terial\, analysis patterns\, and failure modes |\n| **Optimization target* * | Signal-to-noise ratio and accuracy over speed | Maximum automation | E ngineers trust the system because its findings are consistently actionable |\n\n---\n\n## How can this help other practitioners?\n\n- **A design pat tern to adopt:** The "deterministic grounding + AI reasoning" architecture applies to any domain where AI needs to make recommendations about comple x technical systems — not just Kubernetes upgrades\n- **A mistake to avo id:** Don't point an LLM at changelogs and trust the output. We tried it a t Booking.com scale. It doesn't work. Ground everything in your actual cod ebase first.\n- **A way to evaluate competing approaches:** Framework for deciding what to keep deterministic vs. what to delegate to AI in safety-c ritical engineering workflows\n- **Operational metrics that matter:** How to measure AI-assisted engineering quality (signal-to-noise ratio\, false positive rate\, inference accuracy\, escaped issues) so you know if it's a ctually helping or just generating confident noise\n- **A new way of think ing about the problem:** Upgrade preparation isn't a task to automate end- to-end — it's an intelligence problem. The right framing changes what yo u build.\n\n---\n\n## Current state\n\n**Production experience** — valid ated against Kubernetes 1.33\, 1.34\, 1.35 upgrade cycle at Booking.com. T he changelog intelligence track found all manually-identified upgrade cons iderations plus additional findings. Client-go intelligence has been succe ssfully applied to real code changes. Dependency compatibility intelligenc e flagged the right upgrade ordering for dependencies. \n\n---\nSlide deck : https://docs.google.com/presentation/d/1mKqHNrWfJlcXqcetoasnHLBwQ0qw4d I9NaSZpreBE98/edit?usp=sharing \n\n## Tags\n\n`#kubernetes` `#platformengi neering` `#ai` `#agents` `#sre` `#upgrades` `#casestudy` `#devops` `#scala bility` `#infrastructure` `#booking`\n GEO:12.9634971;77.6380856 LAST-MODIFIED:20261106T103244Z LOCATION:Auditorium (Ground Floor) - The Energy Resources Institute (TERI) \nBengaluru\nIN ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/3-weeks-to-3-hours-building -an-ai-co-pilot-for-kubernetes-upgrade-preparation-6QWPVeXF86WUm9V2RC1XsA BEGIN:VALARM ACTION:display DESCRIPTION:3 weeks to 3 hours: building an AI co-pilot for Kubernetes upg rade preparation in Auditorium (Ground Floor) in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:TraceQL Internals: Building a Query Language for Distributed Trace s DTSTART:20261212T053000Z DTEND:20261212T061500Z DTSTAMP:20261109T024109Z UID:session/3PsbiRj91NUHoo4BRgQMCV@hasgeek.com SEQUENCE:1 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073839Z DESCRIPTION:*Title:*\n\nTraceQL Internals: Building a Query Language for D istributed Traces\n\n*One-line summary:*\n\nHow Grafana Tempo stores and q ueries petabytes of trace data on object storage\, and why and how we buil t TraceQL\, a query language designed for traces.\n\n*What problem are you addressing? What real engineering challenge\, question\, or experience is this session about?*\n\nTraces are large\, deeply nested\, and have high- cardinality attributes on span so indexing them in a database gets expensi ve fast at large scale. \n\nTempo started out as a daabase that only suppo rted trace lookup by ID\, backed by object storage with the goals to makin g tracing accessable by making it cheaper to store traces.\n\nOur User kep t asking to support search and aggregation across traces so we buildt a qu ery layer on top of object storage without incresing the cost. \nTraceQL i s designed to express trace-specific questions like span relationships\, s tructure that SQL and PromQL don't support natively.\n\n\n*Who is the inte nded audience? Platform Engineering/SRE/DevOps/Infrastructure/Developers/E ngineering leaders/Other*\n\nPlatform Engineering\, SRE\, Infrastructure\, and Developers building observability or data systems\n\n*Level: beginner /intermediate/advanced*\n\nIntermediate to advanced\n\n*List one or two pr actical takeaways.*\n\n1. How columnar storage on object stores can be a g ood way to handle high-volume telemetry data.\n2. How TraceQL's design fol lows from trace-specific use cases and OpenTelemetry's data model helped u s along the way.\n\n*What will you share?*\n\nArchitecture and design deci sions\, implementation details\, production experience\, trade-offs and al ternatives\, open-source tooling.\n\nTempo's storage layout with Apache Pa rquet\, query planning and sharding across object storage\, TraceQL langua ge design and TraceQL metrics are all open source at: https://github.com/g rafana/tempo\n\n*What is your experience with this problem?*\n\nProduction system\, open-source project\, internal engineering project\, hard-earned engineering lesson\n\n*What approaches failed\, disappointed\, or created unexpected problems?*\n\nMany\, including how it was not support to suppo rt seach on the old block format\, and how we started with a Apache Parque t based format\, Nested Data Model in Apache Parquet and then how we evolv ed it across multiple iterations\, and we at version 5 of the Apache Parqu et based format.\n\n*What will you do differently today?*\n\nThe ecosystem has new file based coloumer storage formats that has showed up in last 2- 3 years\, so if I was starting today\, I would also evalauate those and se e if if something like https://vortex.dev/ is a better fit over Apache Par quet.\n\n*What trade-offs did you consider?*\n\nObject storage and columna r files vs an index database (cost vs query latency).\nA purpose-built lan guage vs extending SQL or PromQL.\nWhat we optimized for (storage cost\, s cale) and what we knowingly gave up (some query latency\, indexing\, etc). \n\n*How can this help other practitioners?*\n\nA design approach for stor ing and querying high-volume telemetry cheaply\, and a way to evaluate ind ex vs columnar-on-object-storage trade-offs.\nIt will also help folks thin k about designing a purpose built system from the ground up\, and what dec isions and tradeoffs are made along the way.\n\n*Current state*\n\nProduct ion experience (and open source)\n\n#observability #databases #scalability LAST-MODIFIED:20261104T073841Z LOCATION:TERI\, Bangalore and online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/traceql-internals-building- a-query-language-for-distributed-traces-3PsbiRj91NUHoo4BRgQMCV BEGIN:VALARM ACTION:display DESCRIPTION:TraceQL Internals: Building a Query Language for Distributed T races in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:From AI Agents to AI Platform: Lessons From Building an In-House M ulti-Agent Platform DTSTART:20261212T062500Z DTEND:20261212T071000Z DTSTAMP:20261109T024109Z UID:session/HUAwqJjuKBZyrnbaUfoq47@hasgeek.com SEQUENCE:0 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073900Z DESCRIPTION:# From AI Agents to AI Platform: Lessons From Building an In-H ouse Multi-Agent Platform\n\n## One-line Summary\n\nBuilding AI agents is relatively easy\; operating them reliably in production is not. This talk shares how we evolved from standalone AI agents to a production-grade mult i-agent platform that emphasizes deterministic execution\, observability\, evaluations\, cost control\, and continuous improvement.\n\n---\n\n## Wha t problem are you addressing? What real engineering challenge\, question\, or experience is this session about?\n\nAs AI agents became integral to o ur production workflows\, we quickly discovered that the agents themselves were only a small part of the engineering challenge. The real complexity lay in orchestrating long-running workflows\, managing failures\, maintain ing state\, controlling costs\, evaluating quality\, and ensuring predicta ble behavior across hundreds of executions.\n\nWe evaluated several popula r agent frameworks and SDKs\, but found that our production requirements - deterministic execution\, strict SLAs\, resumability\, execution transpar ency\, operational observability\, and platform-level governance - require d capabilities beyond what generic frameworks could provide. This led us t o build an in-house multi-agent platform that provides orchestration\, sta ndardized execution semantics\, guardrails\, model abstraction\, and share d infrastructure for AI workloads. The platform is intentionally orchestra ted rather than fully autonomous because predictability matters more than autonomy in production.\n\nMore recently\, we extended the platform with a feedback loop that continuously evaluates agent performance\, generates i mproved skill definitions\, validates them through A/B testing\, and safel y promotes better versions - moving from simply running agents to enabling continuous improvement of agent capabilities.\n\nThis session focuses on the engineering lessons learned while building and operating this platform \, rather than on prompt engineering or LLM fundamentals.\n\n---\n\n## Who is the intended audience?\n\n- Platform Engineers\n- Infrastructure Engin eers\n- DevOps Engineers\n- Site Reliability Engineers (SREs)\n- Backend E ngineers\n- AI Platform Engineers\n- Engineering Managers and Tech Leads r esponsible for production AI systems\n\n---\n\n## Level\n\n**Intermediate to Advanced**\n\n---\n\n## Practical Takeaways\n\n- Understand the anatomy of a production-grade multi-agent platform.\n- Learn how to design a prod uction-ready multi-agent platform that prioritizes determinism\, reliabili ty\, scalability\, and operational simplicity over autonomous agent behavi or.\n- Learn how evaluations\, observability\, cost controls\, and self-im proving feedback loops can be integrated into an AI platform to continuous ly improve agent performance while maintaining production safety.\n\n---\n \n## What will you share?\n\nThis talk will cover:\n\n- The evolution from standalone AI agents to a reusable internal AI platform.\n- Why we chose to build our own multi-agent platform instead of adopting existing agent f rameworks.\n- The anatomy of a production-grade multi-agent platform\, inc luding orchestration\, execution\, state management\, guardrails\, tool in tegration\, and model abstraction.\n- Design decisions around deterministi c execution\, checkpointing\, retries\, failure recovery\, queues\, DAGs\, and long-running workflows.\n- Building evaluation pipelines to measure a gent quality instead of relying on intuition.\n- Cost optimization techniq ues for operating AI agents at scale.\n- Operational lessons around debugg ing\, observability\, governance\, and production reliability.\n- Designin g a self-improving agent feedback loop where execution traces are evaluate d\, candidate skills are generated\, validated through A/B testing\, and p romoted only after demonstrating measurable improvements.\n\n---\n\n## Wha t is your experience with this problem?\n\nThis platform powers production AI workflows inside our Risk Platform and has evolved from independent AI agents into a shared execution platform supporting multiple AI-driven use cases.\n\n---\n\n## What approaches failed\, disappointed\, or created un expected problems?\n\nOur initial assumption was that building AI agents w as the primary challenge. In reality\, the operational complexity emerged only after deploying them into production.\n\nSome of the challenges we en countered included:\n\n- Agent implementations diverged across teams\, mak ing them difficult to standardize and maintain.\n- Generic agent framework s provided useful abstractions for building agents but did not fully addre ss our operational requirements around deterministic execution\, resumabil ity\, observability\, governance\, and execution semantics.\n- Long-runnin g workflows required checkpointing\, state persistence\, and resumability to avoid restarting entire workflows after failures.\n- Debugging AI workf lows without structured execution traces and evaluation pipelines became i ncreasingly difficult.\n- Prompt changes and skill updates often required manual experimentation without an objective mechanism to measure whether a gents had actually improved.\n- As adoption increased\, cost management\, retry policies\, and execution guardrails became platform-level concerns r ather than application-level concerns.\n\nThese challenges fundamentally c hanged how we viewed AI systems. Instead of treating agents as isolated ap plications\, we began treating them as workloads running on a production p latform - requiring the same engineering discipline around reliability\, s calability\, observability\, governance\, and continuous improvement that we apply to any other production infrastructure.\n\n## Speaker's Bio\nMana s Chaturvedi is a Director-Senior Technical Architect in IDfy and has abou t 11+ years of experience in building distributed systems\, AI-powered wor kflows and internal developer tools in various startups and product-based companies.\n LAST-MODIFIED:20261104T073900Z LOCATION:TERI\, Bangalore and online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/from-ai-agents-to-ai-platfo rm-lessons-from-building-an-in-house-multi-agent-platform-HUAwqJjuKBZyrnba Ufoq47 BEGIN:VALARM ACTION:display DESCRIPTION:From AI Agents to AI Platform: Lessons From Building an In-Hou se Multi-Agent Platform in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Beyond SSE: Building Resumable Agent Streams DTSTART:20261212T072000Z DTEND:20261212T080500Z DTSTAMP:20261109T024109Z UID:session/9obknJn7Wicb7KigLaTzFh@hasgeek.com SEQUENCE:1 CATEGORIES:To speak,A talk - 30-40 mins CREATED:20261104T073913Z DESCRIPTION:# Rootconf 2026 Proposal Draft\n\n## Session title\n\n**Beyond SSE: Building Resumable Agent Streams**\n\n## One-line summary\n\nWe buil t a resumable streaming architecture for long-running AI agent interaction s at Glance\, where agent execution can continue independently of the clie nt connection and a disconnected client can resume the stream from the las t event it received.\n\n## What problem are you addressing?\n\nSSE makes i t straightforward to stream an LLM response to a client\, but it does not solve resumability when that client disconnects mid-turn. In our architect ure\, that problem became more interesting because the client could reconn ect to a different gateway pod while the original agent execution was stil l running.\n\nWe wanted to separate the gateway which owns authentication\ , conversation/session handling and client-facing concerns from the servic e running the agent. Once we did that\, the lifetime of the client connect ion\, gateway pod and agent execution were no longer the same.\n\nA client could disconnect while an agent was still running and later reconnect to a different gateway pod. We needed to preserve the ordered history of that turn and resume from the last event the client had received\, without req uiring sticky sessions or coupling the agent to a particular gateway insta nce.\n\nThe engineering problem became less about “how do we stream toke ns?” and more about **where the state of an active stream should live an d what exactly the unit of recovery should be.**\n\n## Who is the intended audience?\n\n**Platform Engineering\, Infrastructure\, SRE\, and backend developers building distributed or AI systems.**\n\n## Level\n\n**Intermed iate**\n\n## One or two practical takeaways\n\n1. Define the **recovery un it and recovery contract** before choosing a streaming technology. In our case\, recovery was per agent turn\, from a specific event ID.\n2. Separat ing execution from delivery makes long-running agent systems much easier t o operate across reconnects\, pod changes and asynchronous work.\n\n## Wha t will you share?\n\nA production architecture story covering:\n\n- Why SS E alone was insufficient for resumable streaming.\n- Why separating the ga teway and agent changed the problem.\n- The trade-offs we saw with gRPC st reaming\, Kafka\, and Redis Streams.\n- How the final design enables order ed replay and recovery from a client-provided cursor.\n- What we learned a bout operating this pattern for long-running agent workflows.\n\n## What i s your experience with this problem?\n\n- **Production system**\n- **Inter nal engineering project**\n\n## What approaches failed\, disappointed\, or created unexpected problems?\n\nNone of the alternatives were fundamental ly incapable of solving the problem\, but they did not fit our recovery mo del cleanly.\n\n**Direct SSE** coupled execution and delivery too closely when the system was split across gateway and agent services.\n\n**gRPC str eaming** gave us a good transport between services but did not by itself p rovide the replayable history required when a client reconnected through a nother gateway pod.\n\n**Kafka** provided durability and replay\, but its natural recovery position is partition plus offset\, whereas our applicati on contract was turn ID plus message ID. We could have built additional in dexing and cursor translation around Kafka\, but that introduced considera bly more machinery for the problem we were trying to solve.\n\n## What wil l you do differently today?\n\nWe would start by defining the **recovery c ontract** rather than beginning with a transport choice.\n\nThe key questi ons would be:\n\n- What exactly needs to survive a disconnect?\n- What is the unit the client needs to resume — a session\, turn\, task or workflo w?\n- Who owns the cursor?\n- How long does replay state need to exist?\n- What state belongs to execution versus delivery?\n\nOnce those are explic it\, the choice of streaming and state primitives becomes much easier.\n\n We would also treat stream lifecycle and event granularity as first-class operational concerns from the beginning rather than as implementation deta ils added later.\n\n## What trade-offs did you consider?\n\nOur primary op timization was for **simple per-turn recovery without pod affinity**.\n\nW e considered:\n\n**Direct SSE:** simplest architecture\, but strongly coup les the client connection to execution.\n\n**gRPC:** excellent service-to- service streaming\, but requires another system for replay and reconnectio n state.\n\n**Kafka:** extremely capable for durable event streaming\, but its partition-based recovery model was not naturally aligned with our per -turn addressability requirement.\n\n**Redis Streams:** allowed us to give each turn its own ordered and replayable event history. A reconnecting ga teway only needs the turn and the last received event ID to continue.\n\nT he trade-off is that this introduces Redis as part of the active delivery path and requires us to think about stream lifecycle\, connection scale an d event volume.\n\n## How can this help other practitioners?\n\nThe main p attern is broader than Redis or even LLMs:\n\n> **Choose the state model a round the unit you need to recover.**\n\nTeams building agents\, coding as sistants\, media generation pipelines or other long-running asynchronous s ystems often start with streaming as a transport problem. Once execution c an outlive the original connection\, it becomes a distributed state-manage ment problem.\n\nThe session gives practitioners a way to reason about tha t transition and evaluate technologies based on their actual recovery requ irements rather than simply choosing a familiar streaming system.\n\n## Cu rrent state\n\n**Production experience**\n\n## Tags\n\n`#agents` `#platfor mengineering` `#reliability` `#databases` `#scalability` `#distributed-sys tems` `#casestudy`\n LAST-MODIFIED:20261104T073917Z LOCATION:TERI\, Bangalore and online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/2026/schedule/beyond-sse-building-resumab le-agent-streams-9obknJn7Wicb7KigLaTzFh BEGIN:VALARM ACTION:display DESCRIPTION:Beyond SSE: Building Resumable Agent Streams in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT END:VCALENDAR