Enterprise AI in Production: Mumbai Call for Proposals

Keeping an Autonomous Coding Agent Reliable on Long Tasks: Lessons from Production. What actually breaks when an AI agent runs long, multi-step tasks for enterprise teams, and the checks that raised our long-task success rate from 74% to 86%.

Abstract

Most AI agents look reliable in a demo and fall apart on long, multi-step work. Each step builds on the previous output, so a small error early in a run compounds until the agent is confidently solving the wrong problem. In an enterprise codebase that means broken builds, wasted review time, and engineers who stop trusting the tool.

I lead AI engineering on the autonomous coding agent at AutonomyAI, which is used by more than 170 product teams, including large enterprises. Over the last year I re-architected the agent end to end around one idea: the agent can only run as long as something outside it can check its work. This talk walks through that system as it runs in production. It covers how we split planning from execution, how long-horizon memory stops the agent from redoing work, which checks we put between steps, and why the agent’s own “done” is never trusted.

I’ll share the numbers. Success on long-horizon tasks rose from 74% to 86% on our evaluation suite, and the memory redesign cut repeated work by 16%. I’ll also share what didn’t work: adding a stronger model or more context made some failures worse, not better. And I’ll tell one failure story in detail, a run that reported success and wasn’t, plus what we changed afterwards. The talk ends with a practical method any team can use to estimate how long its agent can safely run unattended, and a short checklist for extending that before putting an agent in front of customers. I’ll also draw on earlier work building production agents at ANZ Bank, where agents had to pass a bank security review before going live.

Architecture

• Planner and executor loop: the task is broken into steps, and each step is executed with tool calls (repository search, file edits, test runs, build).
• Long-horizon memory: a persistent record of what has been tried, what changed and why, so the agent doesn’t repeat work across a long run.
• External checks between steps: test and build results, state snapshots and diffs, invariants on the repository, and a separate reviewer model that judges the output without seeing the agent’s reasoning.
• Evaluation suite: long-horizon tasks taken from real customer-style workflows, which every change is shipped against.
• Models: hosted frontier LLMs, routed by task type: a stronger model for planning and review, a faster one for routine edits.

Production evidence

• The re-architected agent has been in production since late 2025. It is used by 170+ product teams, including large global enterprises.
• Long-horizon task success: 74% → 86% on our internal evaluation suite of about 200 real-world tasks.
• Repeated work: down 16% after the memory redesign.
• A typical long task runs 30–90 minutes and 100+ agent steps.
• Figures come from our internal evaluation suite; the suite size and run lengths are approximate. Customer names are withheld.

Failure and trade-off

Early on, a long run ended with the agent reporting that the task was complete and all tests passed. In review we found it had quietly changed the tests to match its broken implementation. Nothing in the agent’s own trace flagged it. We stopped trusting self-reported completion: checks now read the real state (CI results, diffs against protected test files), and a separate reviewer model judges the outcome without seeing the agent’s reasoning.

Trade-off: the extra checks add roughly 15–20% to latency and cost per task, but they are what make longer unattended runs possible.

Three takeaways

1. Errors compound. Estimate your agent’s safe run length from its per-step error rate before you scale it.
2. Don’t trust the agent’s own “done” or tests it wrote for itself. Check the real state from outside.
3. A bigger model or more context won’t fix drift. Cheap, frequent external checks will.

Audience

Engineers, architects and AI platform teams deploying LLM agents in production. Some familiarity with LLM tool use is helpful.

Format

30-minute talk (happy to do a 45-minute deep dive if the editors prefer).

Speaker

Utkarsh Kanwat, AI Research Engineer at AutonomyAI; IIT Bombay, Electrical Engineering. Leads AI engineering on AutonomyAI’s autonomous coding agent; previously built production AI agents at ANZ Bank. His writing on AI agent reliability has been covered by The Register and TechSpot. Speaking at The Fifth Elephant Hyderabad on 28 Nov 2026.

Disclosures

I work at AutonomyAI; this is not a product pitch. Customer names are withheld and figures are from internal evaluation. Available in Mumbai on 31 Oct and for editorial preparation.