In the Loop - harnesses, loops, graphs, and the evals that keep them honest

Abstract

Teams are putting AI coding agents into production workflows faster than they are putting controls around them. An agent that writes code also runs shell commands, reads secrets, installs packages, and pushes to git, at machine speed. This talk is about the control layer that has held up in production: event-driven hooks that intercept every tool call and allow, deny, or escalate it before execution.

What I’ll cover, all hands-on:

• The interception contract: PreToolUse and PostToolUse events, JSON in / JSON out, deny vs ask, and why a separate-process hook cannot be prompt-injected around
• Real blocking scripts: dangerous commands, .env exfiltration, typosquatted installs, agents skip-marking failing tests to fake a green build
• Keeping checks under 100ms so the agent stays usable: bash vs Node vs Python numbers
• The honest part: while recording demos for CodeSecCon this month, Claude Code bypassed my own secret-protection hook on camera, twice (a grep pattern gap and an uncovered built-in search tool). Both patched upstream with regression tests the same night. What that taught me about testing guardrails is a big part of this talk.
• What hooks cannot catch, and which layers cover the rest

Toolkit: https://github.com/karanb192/claude-code-hooks (MIT, 481 stars, 1,584 tests). Earlier versions of this talk ran at the OWASP GenAI & Agentic Security Virtual Summit (May 2026) and CodeSecCon by SecurityWeek (Aug 2026); both recordings are public. This version leans into the production-honesty angle for The Fifth Elephant.

Format

30-min talk plus 10 min Q&A, per the CFP format. Demos run offline.

Notes for reviewers

• Non-vendor: MIT-licensed personal open source.
• Recordings: https://youtu.be/5LA220Jm7nU (CodeSecCon, Aug 2026) and https://youtu.be/LKZb-qIC2q4 (OWASP Summit, May 2026)
• Write-up: https://karanbansal.in/blog/claude-code-hooks/
• Based in Gurgaon; happy to travel to Hyderabad for the meet-up (Sep 5).

About the speaker

Karan Bansal is Head of AI at ArmorCode, with over 10 years in security. He previously led security and privacy at Urban Company and was a founding engineer at AvidSecure, acquired by Sophos in 2019. He authored “Claude Code’s Most Underrated Feature: Hooks” and maintains the open-source claude-code-hooks toolkit (481 stars), a collection of security hooks for AI coding agents mapped to the OWASP LLM Top 10. He also built reddit-mcp-buddy (799 stars) and curates awesome-claude-skills (492 stars). He spoke on this topic at the OWASP GenAI & Agentic Security Summit and CodeSecCon by SecurityWeek in 2026, and contributes to SGLang and vLLM.