A hands-on workshop for platform, infrastructure, MLOps and SRE engineers who have been asked to run an open model on hardware the company already owns, and need to know what they lose when they make it smaller.

Serving an open model is mostly a memory problem. An 8 billion parameter model needs 32 GB at full precision, 16 GB at 16 bits, 8 GB at 8 bits and 4 GB at 4 bits. Quantisation, storing every number in the model with fewer bits, is the lever that decides whether it fits a 24 GB card, a laptop, or nothing you own. Model pages now list Q4, Q8 and fp16 side by side, and serving stacks expose it as a single flag. What nobody ships with it is a way to tell whether the smaller model still works.

This workshop is about finding that out deliberately, before your users do.

Slides: https://docs.google.com/presentation/d/1hcxysI6-TASgSXujkRSNNpz9ki2go1UW1j4XBaa9XYY/edit?usp=sharing

What is this workshop about?

This workshop will help engineers understand what quantisation actually does to a model, where it breaks, and how to measure the damage instead of assuming it.

Participants learn quantisation from first principles as rounding to a grid: q = round(w / s) to store a weight, and ŵ = q × s to get it back. Everything else follows from how the step size s is chosen. The workshop covers how to read a weight distribution and predict whether a precision will hold, how to fix the three most common failures, and how to build an evaluation that catches a model which has quietly stopped working.

What will participants work on?

Participants work in one Google Colab notebook on the free tier, CPU only, with a small open model (Qwen3 0.6B) cached ahead of time. They write the quantiser themselves in about forty lines of PyTorch, then use it on the real model.

Through this setup, participants will examine:

• Memory against accuracy: how much memory each bit saves, and where the accuracy cliff sits for this model.
• Small weights: what happens to a wide layer when the grid is too coarse for its weights.
• Outliers: what one very large weight does to the precision of every other weight, and why clipping it can break a model even at 16 bits.
• Silent failure: whether a benchmark score can tell a cautious model from a dead one.
• Before and after: memory, speed and score for the same model at 16, 8, 6 and 4 bits.

Target audience

• Platform and infrastructure engineers asked to serve an open model on-prem or on a fixed budget, who need it to fit the hardware they already have.
• MLOps, SRE and serving engineers who own the inference stack, the rollout and the rollback. A precision change alters the model without changing its version number.
• Anyone looking at Q4_K_M, Q8_0 and fp16 on the same model page with no way to choose between them.

Basic Python, and having opened a notebook before, is assumed. No machine learning or quantisation background is needed.

Learning outcomes

By the end of the workshop, participants will be able to:

1. Explain quantisation as rounding to a grid, and choose a step size for a given set of weights.
2. Quantise an open model to 8, 6 and 4 bits and measure memory, speed and score at each.
3. Read a weight distribution and predict whether a precision will hold, before paying for an evaluation run.
4. Diagnose and fix the three common failures: weights rounding to zero, clipped outliers, and layers with nowhere to put their scale.
5. Build an evaluation that pairs a capability score with a liveness check, and use it as a go or no-go gate for a precision change.
6. Take a table of memory, speed and score for their own model to whoever owns the GPU budget.

Workshop contents: hands-on experiments

Each experiment starts with a prediction written down before anything runs, and ends with a measured result.

1. Shrink a real model and measure it

We quantise the model to 16, 8, 6 and 4 bits with the quantiser built in the first block. The expectation: 8 bits costs almost nothing, and below that small models fall off first.

We measure: memory, tokens per second and a tool calling score at each precision.

For reference, Qwen3 on the Berkeley Function Calling Leaderboard (840 prompts), at 16 / 8 / 6 bits:

• Qwen3 0.6B: 81.9 / 78.7 / 27.3
• Qwen3 1.7B: 88.7 / 88.6 / 78.9
• Qwen3 4B: 89.6 / 90.4 / 62.9
• Qwen3 8B: 91.3 / 90.2 / 88.0

At 4 bits, every size scored 0.

2. Break one: small weights round to zero

We quantise a wide layer at 3 bits on a fixed grid. Wide layers have small weights, and every one of them lands in the bucket that rounds to zero. It looks like too few bits. The cause is where the grid sits.

We measure: the fraction of weights that became zero, before and after fitting the grid to the weights.

3. Break two: one big weight gets clipped

About one weight in ten thousand sits far outside the usual range, and those are among the most important weights in the layer. Clipping them took a 360 million parameter model’s loss from 2.53 to 7.73 at 16 bits, where precision is plainly not the problem.

We find which layers hold the outliers, see why most layers are protected by the normalisation in front of them, and give the two kinds that are not protected a scale of their own.

We measure: loss before and after, layer by layer.

4. Break three: the score says fine

We compress too far and score the model the obvious way. A model that had stopped calling tools scored 100 on the “do not call a tool” category, and averaged with the others it would have been reported as mediocre rather than dead. It called a tool on 1.2% of prompts.

We measure: score and call rate side by side, and build the check that refuses to ship a model when either one drops.

Session plan (3.5 hours)

• 0:00 Setup check, and why memory decides what you can serve (20 min)
• 0:20 Build the quantiser, about forty lines (35 min)
• 0:55 Shrink a real model and measure it (30 min)
• 1:25 Break (15 min)
• 1:40 The three breaks, on purpose (45 min)
• 2:25 Diagnose from the weights (30 min)
• 2:55 Build the eval, score before and after (35 min)

The last block doubles as overflow. If earlier blocks run long, the eval is demonstrated instead of built together.

Joining late or falling behind: every stage of the notebook is saved with its outputs. Open the stage the room is on and carry on from there. Each block opens with a one minute recap.

Trade-offs we work through

• Memory against accuracy. Each bit removed saves about 12% of weight memory. The first few are free, the last few are not, and where the cliff sits depends on the model.
• Quantise after training, or retrain. Quantising a trained model takes minutes and needs no data. Training with quantisation in the loop buys one or two more bits and costs a training run.
• One scale or many. A scale per tensor is free at inference, a scale per row is nearly free, and a scale per token costs real compute but recovers models the others cannot.
• Cheap checks against full benchmarks. Validation loss costs about a hundredth of a benchmark run and finds the cliff well. It cannot tell you the output has stopped being valid JSON.

What you take back to work

• A cost model for your own stack. Memory and score at each precision for the model you actually serve, which turns a GPU purchase argument into a table.
• A go or no-go rule. Inspect a checkpoint and predict whether a precision will hold, before paying for an evaluation sweep.
• An eval that catches silence. A capability metric with a liveness metric beside it, useful for anything on your platform that can fail by going quiet.
• Fewer vendor numbers. Open weights and open tooling on hardware you own, with figures you measured yourself.

In the final minutes, participants point the notebook at a model their own team serves and leave with its numbers.

Tooling & prerequisites

Bring

• A laptop with a browser and a Google account for Colab

Provisioned in the notebook

• Python 3.10 or newer, PyTorch, Hugging Face Transformers, NumPy, Matplotlib
• The model checkpoint, cached ahead of time
• A pinned requirements file and setup script for anyone who prefers to run locally

No GPU, no paid cloud account and no dataset download.

Before the workshop

A week ahead, participants receive the Colab link and a short primer, about fifteen minutes of reading: what a weight is, what a token is, and why a model that fits in memory is cheaper to serve. No maths beyond division. The primer and notebook stay public afterwards.

Reference material

• A Visual Guide to Quantization, Maarten Grootendorst: https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization
• Hugging Face Transformers, quantization overview: https://huggingface.co/docs/transformers/main/en/quantization/overview
• LLM.int8(), on outlier features in large models: https://arxiv.org/abs/2208.07339
• SmoothQuant: https://arxiv.org/abs/2211.10438
• GPTQ: https://arxiv.org/abs/2210.17323
• AWQ: https://arxiv.org/abs/2306.00978
• Berkeley Function Calling Leaderboard: https://gorilla.cs.berkeley.edu/leaderboard.html
• llama.cpp, for the GGUF quant types you see on model pages: https://github.com/ggml-org/llama.cpp

About the instructor

Alosh Denny is an AI engineer and researcher from Kerala. He heads AI at Aleddo Technologies and runs LLMOps at Intensors, and previously built agentic security systems at Discern Security and Malayalam language models at Eduport. Over the past year he has run close to 900 training and quantisation experiments on how few bits a language model needs, and he designs open source AI accelerator hardware. He has seven peer reviewed publications across IEEE and Springer, and has spoken at IndiaFOSS and GHCI.

How to attend this workshop

This workshop is open to Rootconf members: https://hasgeek.com/rootconf#memberships and ticket holders of Rootconf 2026 Annual Conference: https://hasgeek.com/rootconf/2026/#tickets

The workshop is open to 30 in-person participants, with hybrid access available for remote attendees. Seats for in-person participants will be available on a first-come, first-served basis.

Contact information

For inquiries about the workshop, contact +91-7676332020 or write to info@hasgeek.com