Rootconf 2026 Annual Conference

Session title

We spent a week optimising an LLM pipeline and most of what we measured was noise

Alternative titles, if a plainer one reads better to the editorial team:

• When your variance is bigger than your speedup
• Stop benchmarking, start reading code
• Four optimisations, three null results, and the bugs we found instead

One-line summary

A 12,858-image vision pipeline took 1h51m, so we tested four optimisations
against it. Three showed no effect, one was slower, and our measurements had 43%
run-to-run variance — here’s how we found the real wins anyway, and why they came
from reading code rather than benchmarking it.

What problem are you addressing?

Anyone putting an LLM pipeline into production hits the same wall: it is too
slow, there are obvious things to try, and none of the usual profiling instincts
work. The provider’s latency moves 2x over minutes. Your effect size is 10%.
Your variance is 43%. Every benchmark you run is a coin flip you mistake for a
result.

We ran the experiment properly and it taught us more about measurement than
about latency:

• Sweeping concurrency 100 -> 200 -> 400 -> 800 produced a clean monotonic
  win that was entirely drift. Bracketing every level against repeats of a
  fixed baseline showed no effect at all. The same trap caught a colleague’s
  earlier run, in the opposite direction.
• A Streamlit app left running on the test machine cost 23% on an identical
  config. Every number from the first pass was unusable.
• A prompt-caching change raised cache hit 36.5% -> 42% and made the pipeline
  7% slower. Reproducible in both directions. It is a cost win and a latency
  loss, which is the opposite of what it was built for.
• gs:// URI transport — the change we were most confident about — did
  nothing. It removes 20ms/image of client work, and that is not the
  bottleneck. On the route that does not support gs:// at all, the sanctioned
  alternative measured 5x more expensive than the bytes it saved.

The real wins came from auditing, not benchmarking:

• A proxy-only field in the request body caused 1,402 consecutive HTTP 400s
  and zero labels before anyone noticed. Retries cannot help a malformed
  request, so the retry budget made it slower to fail.
• An uncapped model output let one repetition loop generate 458,752
  characters and stall its entire batch, because every stage gathers
  concurrently and one slow call sets the floor for all of it.
• Synchronous prompt-file reads were 75% of the process’s own CPU and
  blocked the event loop for all 200 in-flight calls — 8,000 disk reads where 4
  would do.

Thesis: when run-to-run variance exceeds your effect size, stop benchmarking
and start reading code. Defects are measurable at n=1; optimisations are not.

Who is this for?

Engineers running inference pipelines in production, and more generally anyone
whose system depends on a service whose latency is noisier than the change they
are trying to make. No ML background needed — the subject is an LLM pipeline,
the content is measurement discipline.

What will attendees take away?

1. Bracket, never sweep. Why interleaving every trial against a fixed
   baseline is the only defence against drift, with the two runs that prove it
   (one drifted up, one drifted down, both looked like clean results).
2. Capture the metric that explains the mechanism, not just the clock. Cache
   hit rate explained a 2.8x gap that wall-clock alone attributed to the wrong
   cause.
3. Where asyncio.gather turns one slow request into a whole-stage stall,
   and why bounding output length fixes it better than bounding time.
4. A short audit checklist that found three defects no benchmark would have
   surfaced.

Rough structure