Unavailable

This livestream is restricted

Already a member? Login with your membership email address

Rootconf 2026 Annual Conference

Platforms for AI, and AI for Platforms

Tickets

Loading…

This video is for members only

Alosh Denny

Alosh Denny

@aoxo Facilitator

Vivek Kalyanarangan

@vivekkalyanarangan Facilitator

Birds of Feather (BOF) session: Fewer bits, more questions: making quantization work across models, runtimes and hardware

Submitted Nov 15, 2026

Fewer bits, more questions: making quantization work across models, runtimes and hardware

Birds of a Feather with Vivek Kalyanarangan and Alosh Denny

Quantization has gone from a niche optimization to a default step in shipping models. Running weights in 8-bit or 4-bit is routine now. Choosing which format, which method and which toolchain for your workload is not, and most teams make that choice with scattered blog posts and trial and error.

The ecosystem is moving fast. New low-precision data types such as NVFP4 and MXFP4 arrive alongside hardware that supports them natively, while the same checkpoint may need a different format on a different accelerator. A format that looks best on paper only pays off if your runtime has the kernels for it and your hardware has the matching path.

This BOF is built around a few open questions. Which data types are worth adopting today (INT8, FP8, INT4, NVFP4, MXFP4), and for what: weights only, weights and activations, or the KV cache? How do you measure accuracy loss on your task instead of trusting a leaderboard? How do runtimes such as vLLM, TensorRT-LLM, llama.cpp, ONNX Runtime and OpenVINO differ in what they actually support? What happens on older GPUs, CPUs and edge devices without native low-precision paths? And when is quantization the wrong lever?

We will keep this a roundtable, not a talk. We (Vivek Kalyanarangan and Alosh Denny) will open with a short pass over the whole spread, from fundamentals like PTQ, QAT and calibration to methods like GPTQ and AWQ. Most of the time goes to the deployment side: data types, runtimes and hardware compatibility. We will lead the discussion, but the session works best when participants bring their own deployments, failures and surprises.

It is meant for ML engineers, platform and infrastructure engineers, and anyone serving models in production or on constrained hardware. No prior quantization experience is needed, though experience with a real deployment helps the conversation.

Participants should leave with a shared map of data types, runtimes and hardware, a list of pitfalls others have hit, and a clearer sense of which choices are settled and which are still open.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

One year entry to Rootconf's programmes and activities

Hosted by

We care about site reliability, cloud costs, security and data privacy

Supported by

Platinum Sponsor

Nutanix is a global leader in cloud software, offering organizations a single platform for running apps and data across clouds.

Silver sponsor

Engineers at Roku Bengaluru, building the tech behind the #1 TV streaming platform in the US and Mexico. Come say hello at our meetups.

Bronze Sponsor

Internet infrastructure APIs for IP geolocation and more

Community Sponsor

Real-time Observability & Governance layer for AI agents