Open weights, real stakes

Describe your session (2 paragraphs)

This session is a deep dive on building architecture and guardrails around self-hosted open-weight models to deliver quality. We run a production LLM helpline for Gujarati dairy farmers (phone-line and chat) answering questions on animal health, breeding, milk, and schemes, on open-weight models we host ourselves.

We share how far you can get without fine-tuning: an English-reasoning core wrapped in a swappable translation layer, a domain glossary that beats frontier models on terminology, a streaming pipeline that hides translation latency, and the guardrails that catch the failure modes you inherit with open weights. We close on the few things that genuinely need fine-tuning, and how running the product first is what made fine-tuning cheap.

Takeaways (1–2)

1. A reusable, eval-gated escalation ladder for LLM product quality (prompts → glossary → deterministic guards → architecture → fine-tuning) with a measured stopping rule at each gate, so you know how far the cheap, portable interventions get you before you spend on weights. (Most of our quality was bought by orchestration, not training.)
2. Concrete, transferable patterns: the swappable “translation-sandwich” for model optionality; a tiny domain glossary that out-performs frontier models on terminology; a generate∥translate∥speak streaming pipeline to hide translation latency on self-hosted inference; and a clear test for when fine-tuning is actually the right tool (and how production traffic bootstraps the data for it).

Who is this beneficial for?

• Platform / infra engineers running self-hosted LLM inference (vLLM, GPU serving, multi-model orchestration) who need quality and latency without per-token API lock-in.
• Applied-AI / ML engineers building LLM products (especially multilingual / Indic-language and voice) facing domain-terminology, translation, and streaming-latency problems.
• Founders / tech leads deciding between fine-tuning and orchestration, or planning to scale one product across many languages.
• Anyone evaluating open-weight vs managed models for production and wanting an honest account of the failure modes you inherit.

Bio

I’m Kanav Dwevedi, a software engineer at The Flywheel leading infrastructure and application development for AmulAI in partnership with COSS.

Github

Draft Slides

Link to the draft slides