Open weights, real stakes

Running an open-weight model on a laptop is easy; getting it to serve reliably in production - on-prem/on-cloud GPUs - is where most teams stall. This session is a hands-on playbook for that journey: model selection, quantization, choosing a serving stack (vLLM, TensorRT-LLM), batching and KV-cache behaviour, the latency-vs-throughput tradeoff, evals, and routing. I’ll use an open-weight model (Nemotron) as the worked example and walk through the decisions with real numbers, not spec sheets.
The cost and data-control case for open weights only pays off if you can actually operate them - and with local and on-prem inference now realistic even on small footprints, the bottleneck isn’t the model, it’s the operational know-how to serve it well and cheaply. I’ll be honest about what worked, what broke, and what I’d do differently, including the failure modes that never make it into launch posts. The goal is that you leave with a repeatable approach you can apply to your own deployment the next day.

1. A repeatable checklist for taking an open-weight model from prototype to production, with real latency / throughput / cost numbers from actual runs.
2. How to choose between open-source serving stacks for your workload - and the common failure modes (and fixes) to avoid along the way.

Engineers, data scientists, and technical founders who have shipped - or want to ship - something using open-weight models in production. If you’ve deployed or are planning to deploy open models on-prem.

https://drive.google.com/file/d/1etLjfWohPDNY_80P3-yPjZutIgvtCaBm/view?usp=sharing

Nilesh Kumar, Solution Architect at NVIDIA, working with teams on deploying open-weight model inference in production. https://www.linkedin.com/in/nileshkr1203/