After the Prototype - agents, applied AI, and models put to work

The problem. Before launching a multi-tenant voice agent, we had to decide how many phone calls one GPU could carry for a self-hosted model we are evaluating (Gemma 4 26B-A4B on an RTX PRO 6000). Our first load test said 16 calls at about 0.3 s to the first token. It was wrong: every stream sent the same prompt, so the server cached one copy. With a different prompt per call, as different tenants’ calls would, the same GPU took 2.8 s, and at 64 calls decode fell to 3.5 tokens per second in our first configuration, below the 4 to 5 that speech needs. We found six more wrong answers the same way: fluent nonsense from a serving setting on long prompts; a test that left out tool schemas, so the model booked a pickup nobody had agreed to, 5 of 5 times, where the same test with schemas read the booking back 5 of 5; a model-name setting that our provider path ignored, so a different model answered every turn of a dev call; a claimed 2.36x slowdown that was really a training job holding 90% of the GPU; a speaker-ID threshold copied from another model that accepted 75 of 81 wrong-speaker pairs; and one prompt-test cell that gave 5 of 5 one way, then 38 of 38 the other way, across process boundaries on the same prompt. Each looked sound until we checked it.

Why it is relevant. Most agent talks start above the model. This one starts at the serving layer under the agent, where capacity, model and setting decisions are made from benchmark numbers that rarely get checked, and these errors are silent. They produce clean numbers and a wrong decision, which surfaces only after launch, when changing course is expensive. The capacity result is from one GPU, one model and one date, and the talk says so.

Who it is for. Engineers and tech leads who choose, serve or evaluate LLMs for latency-sensitive products, voice and other real-time agents especially, and anyone who has acted on a benchmark number.

Key takeaways.

• Why a shared-prompt load test can overstate multi-tenant capacity by roughly 10x on first-token time (our estimate), and how to test with realistic per-tenant prompts.
• A preflight checklist to run before trusting any model benchmark: replay the exact request, check answers are correct before timing them (for example with a needle question at your real prompt length), include a baseline arm, repeat across separate processes, confirm which model actually answered.
• A habit that prevents reusing thresholds: record each one with the model it was calibrated on.

Outline (30 min, including questions).

1. Why we benchmarked, and what a wrong answer would have cost (3 min)
2. The capacity test that overstated by roughly 10x (8 min)
3. Four more cases: fluent nonsense, missing schemas, an ignored setting, and a slowdown we blamed on a flag when a training job on the same GPU was the cause (9 min)
4. The threshold and flipping-result cases (4 min)
5. The preflight checklist (4 min)
6. Questions (2 min)

Speakers

• Pramida Tumma, CTO and Co-Founder, ZenMinds.ai (16+ years in engineering leadership and AI/ML platforms; built agentic frameworks and RAG systems; formerly Engineering Director at Thoughtworks).
• Jyothi Pujari, CPO and Co-Founder, ZenMinds.ai (20+ years building AI-powered products; owns AI product vision and roadmaps; formerly General Manager for engineering clusters at ThoughtWorks).