Fewer bits, more questions: making quantization work across models, runtimes and hardware
Birds of a Feather with Vivek Kalyanarangan and Alosh Denny
Quantization has gone from a niche optimization to a default step in shipping models. Running weights in 8-bit or 4-bit is routine now. Choosing which format, which method and which toolchain for your workload is not, and most teams make that choice with scattered blog posts and trial and error.
The ecosystem is moving fast. New low-precision data types such as NVFP4 and MXFP4 arrive alongside hardware that supports them natively, while the same checkpoint may need a different format on a different accelerator. A format that looks best on paper only pays off if your runtime has the kernels for it and your hardware has the matching path.
This BOF is built around a few open questions. Which data types are worth adopting today (INT8, FP8, INT4, NVFP4, MXFP4), and for what: weights only, weights and activations, or the KV cache? How do you measure accuracy loss on your task instead of trusting a leaderboard? How do runtimes such as vLLM, TensorRT-LLM, llama.cpp, ONNX Runtime and OpenVINO differ in what they actually support? What happens on older GPUs, CPUs and edge devices without native low-precision paths? And when is quantization the wrong lever?
We will keep this a roundtable, not a talk. We (Vivek Kalyanarangan and Alosh Denny) will open with a short pass over the whole spread, from fundamentals like PTQ, QAT and calibration to methods like GPTQ and AWQ. Most of the time goes to the deployment side: data types, runtimes and hardware compatibility. We will lead the discussion, but the session works best when participants bring their own deployments, failures and surprises.
It is meant for ML engineers, platform and infrastructure engineers, and anyone serving models in production or on constrained hardware. No prior quantization experience is needed, though experience with a real deployment helps the conversation.
Participants should leave with a shared map of data types, runtimes and hardware, a list of pitfalls others have hit, and a clearer sense of which choices are settled and which are still open.
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}