BEGIN:VCALENDAR VERSION:2.0 PRODID:-//HasGeek//NONSGML Funnel//EN DESCRIPTION:A hands-on workshop on shrinking open models to fit the hardwa re you have\, and proving they still work X-WR-CALDESC:A hands-on workshop on shrinking open models to fit the hardw are you have\, and proving they still work NAME:Break a model\, then fix it X-WR-CALNAME:Break a model\, then fix it REFRESH-INTERVAL;VALUE=DURATION:PT12H SUMMARY:Break a model\, then fix it TIMEZONE-ID:Asia/Kolkata X-PUBLISHED-TTL:PT12H X-WR-TIMEZONE:Asia/Kolkata BEGIN:VEVENT SUMMARY:Break a model\, then fix it DTSTART:20261115T050000Z DTEND:20261115T093000Z DTSTAMP:20261025T203739Z UID:session/Um7pqk1CRwrY4pFvErJsMq@hasgeek.com SEQUENCE:10 CREATED:20261025T080609Z DESCRIPTION:A hands-on workshop for platform\, infrastructure\, MLOps and SRE engineers who have been asked to run an open model on hardware the com pany already owns\, and need to know what they lose when they make it smal ler.\n\nServing an open model is mostly a memory problem. An 8 billion par ameter model needs 32 GB at full precision\, 16 GB at 16 bits\, 8 GB at 8 bits and 4 GB at 4 bits. Quantisation\, storing every number in the model with fewer bits\, is the lever that decides whether it fits a 24 GB card\, a laptop\, or nothing you own. Model pages now list Q4\, Q8 and fp16 side by side\, and serving stacks expose it as a single flag. What nobody ship s with it is a way to tell whether the smaller model still works.\n\nThis workshop is about finding that out deliberately\, before your users do.\n\ n**Slides:** https://docs.google.com/presentation/d/1hcxysI6-TASgSXujkRSNN pz9ki2go1UW1j4XBaa9XYY/edit?usp=sharing\n\n# What is this workshop about?\ n\nThis workshop will help engineers understand what quantisation actually does to a model\, where it breaks\, and how to measure the damage instead of assuming it.\n\nParticipants learn quantisation from first principles as rounding to a grid: `q = round(w / s)` to store a weight\, and `ŵ = q × s` to get it back. Everything else follows from how the step size `s` i s chosen. The workshop covers how to read a weight distribution and predic t whether a precision will hold\, how to fix the three most common failure s\, and how to build an evaluation that catches a model which has quietly stopped working.\n\n# What will participants work on?\n\nParticipants work in one Google Colab notebook on the free tier\, CPU only\, with a small o pen model (Qwen3 0.6B) cached ahead of time. They write the quantiser them selves in about forty lines of PyTorch\, then use it on the real model.\n\ nThrough this setup\, participants will examine:\n\n* **Memory against acc uracy:** how much memory each bit saves\, and where the accuracy cliff sit s for this model.\n* **Small weights:** what happens to a wide layer when the grid is too coarse for its weights.\n* **Outliers:** what one very lar ge weight does to the precision of every other weight\, and why clipping i t can break a model even at 16 bits.\n* **Silent failure:** whether a benc hmark score can tell a cautious model from a dead one.\n* **Before and aft er:** memory\, speed and score for the same model at 16\, 8\, 6 and 4 bits .\n\n# Target audience\n\n* Platform and infrastructure engineers asked to serve an open model on-prem or on a fixed budget\, who need it to fit the hardware they already have.\n* MLOps\, SRE and serving engineers who own the inference stack\, the rollout and the rollback. A precision change alt ers the model without changing its version number.\n* Anyone looking at Q4 _K_M\, Q8_0 and fp16 on the same model page with no way to choose between them.\n\nBasic Python\, and having opened a notebook before\, is assumed. No machine learning or quantisation background is needed.\n\n# Learning ou tcomes\n\nBy the end of the workshop\, participants will be able to:\n\n1. Explain quantisation as rounding to a grid\, and choose a step size for a given set of weights.\n2. Quantise an open model to 8\, 6 and 4 bits and measure memory\, speed and score at each.\n3. Read a weight distribution a nd predict whether a precision will hold\, before paying for an evaluation run.\n4. Diagnose and fix the three common failures: weights rounding to zero\, clipped outliers\, and layers with nowhere to put their scale.\n5. Build an evaluation that pairs a capability score with a liveness check\, and use it as a go or no-go gate for a precision change.\n6. Take a table of memory\, speed and score for their own model to whoever owns the GPU bu dget.\n\n# Workshop contents: hands-on experiments\n\nEach experiment star ts with a prediction written down before anything runs\, and ends with a m easured result.\n\n### 1. Shrink a real model and measure it\n\nWe quantis e the model to 16\, 8\, 6 and 4 bits with the quantiser built in the first block. The expectation: 8 bits costs almost nothing\, and below that smal l models fall off first.\n\nWe measure: memory\, tokens per second and a t ool calling score at each precision.\n\nFor reference\, Qwen3 on the Berke ley Function Calling Leaderboard (840 prompts)\, at 16 / 8 / 6 bits:\n\n* Qwen3 0.6B: 81.9 / 78.7 / 27.3\n* Qwen3 1.7B: 88.7 / 88.6 / 78.9\n* Qwen3 4B: 89.6 / 90.4 / 62.9\n* Qwen3 8B: 91.3 / 90.2 / 88.0\n\nAt 4 bits\, ever y size scored 0.\n\n### 2. Break one: small weights round to zero\n\nWe qu antise a wide layer at 3 bits on a fixed grid. Wide layers have small weig hts\, and every one of them lands in the bucket that rounds to zero. It lo oks like too few bits. The cause is where the grid sits.\n\nWe measure: th e fraction of weights that became zero\, before and after fitting the grid to the weights.\n\n### 3. Break two: one big weight gets clipped\n\nAbout one weight in ten thousand sits far outside the usual range\, and those a re among the most important weights in the layer. Clipping them took a 360 million parameter model's loss from 2.53 to 7.73 at 16 bits\, where preci sion is plainly not the problem.\n\nWe find which layers hold the outliers \, see why most layers are protected by the normalisation in front of them \, and give the two kinds that are not protected a scale of their own.\n\n We measure: loss before and after\, layer by layer.\n\n### 4. Break three: the score says fine\n\nWe compress too far and score the model the obviou s way. A model that had stopped calling tools scored 100 on the "do not ca ll a tool" category\, and averaged with the others it would have been repo rted as mediocre rather than dead. It called a tool on 1.2% of prompts.\n\ nWe measure: score and call rate side by side\, and build the check that r efuses to ship a model when either one drops.\n\n# Session plan (3.5 hours )\n\n* **0:00** Setup check\, and why memory decides what you can serve (2 0 min)\n* **0:20** Build the quantiser\, about forty lines (35 min)\n* **0 :55** Shrink a real model and measure it (30 min)\n* **1:25** Break (15 mi n)\n* **1:40** The three breaks\, on purpose (45 min)\n* **2:25** Diagnose from the weights (30 min)\n* **2:55** Build the eval\, score before and a fter (35 min)\n\nThe last block doubles as overflow. If earlier blocks run long\, the eval is demonstrated instead of built together.\n\n**Joining l ate or falling behind:** every stage of the notebook is saved with its out puts. Open the stage the room is on and carry on from there. Each block op ens with a one minute recap.\n\n# Trade-offs we work through\n\n* **Memory against accuracy.** Each bit removed saves about 12% of weight memory. Th e first few are free\, the last few are not\, and where the cliff sits dep ends on the model.\n* **Quantise after training\, or retrain.** Quantising a trained model takes minutes and needs no data. Training with quantisati on in the loop buys one or two more bits and costs a training run.\n* **On e scale or many.** A scale per tensor is free at inference\, a scale per r ow is nearly free\, and a scale per token costs real compute but recovers models the others cannot.\n* **Cheap checks against full benchmarks.** Val idation loss costs about a hundredth of a benchmark run and finds the clif f well. It cannot tell you the output has stopped being valid JSON.\n\n# W hat you take back to work\n\n* **A cost model for your own stack.** Memory and score at each precision for the model you actually serve\, which turn s a GPU purchase argument into a table.\n* **A go or no-go rule.** Inspect a checkpoint and predict whether a precision will hold\, before paying fo r an evaluation sweep.\n* **An eval that catches silence.** A capability m etric with a liveness metric beside it\, useful for anything on your platf orm that can fail by going quiet.\n* **Fewer vendor numbers.** Open weight s and open tooling on hardware you own\, with figures you measured yoursel f.\n\nIn the final minutes\, participants point the notebook at a model th eir own team serves and leave with its numbers.\n\n# Tooling & prerequisit es\n\n**Bring**\n\n* A laptop with a browser and a Google account for Cola b\n\n**Provisioned in the notebook**\n\n* Python 3.10 or newer\, PyTorch\, Hugging Face Transformers\, NumPy\, Matplotlib\n* The model checkpoint\, cached ahead of time\n* A pinned requirements file and setup script for an yone who prefers to run locally\n\nNo GPU\, no paid cloud account and no d ataset download.\n\n# Before the workshop\n\nA week ahead\, participants r eceive the Colab link and a short primer\, about fifteen minutes of readin g: what a weight is\, what a token is\, and why a model that fits in memor y is cheaper to serve. No maths beyond division. The primer and notebook s tay public afterwards.\n\n# Reference material\n\n* A Visual Guide to Quan tization\, Maarten Grootendorst: https://newsletter.maartengrootendorst.co m/p/a-visual-guide-to-quantization\n* Hugging Face Transformers\, quantiza tion overview: https://huggingface.co/docs/transformers/main/en/quantizati on/overview\n* LLM.int8()\, on outlier features in large models: https://a rxiv.org/abs/2208.07339\n* SmoothQuant: https://arxiv.org/abs/2211.10438\n * GPTQ: https://arxiv.org/abs/2210.17323\n* AWQ: https://arxiv.org/abs/230 6.00978\n* Berkeley Function Calling Leaderboard: https://gorilla.cs.berke ley.edu/leaderboard.html\n* llama.cpp\, for the GGUF quant types you see o n model pages: https://github.com/ggml-org/llama.cpp\n\n# About the instru ctor\n\nAlosh Denny is an AI engineer and researcher from Kerala. He heads AI at Aleddo Technologies and runs LLMOps at Intensors\, and previously b uilt agentic security systems at Discern Security and Malayalam language m odels at Eduport. Over the past year he has run close to 900 training and quantisation experiments on how few bits a language model needs\, and he d esigns open source AI accelerator hardware. He has seven peer reviewed pub lications across IEEE and Springer\, and has spoken at IndiaFOSS and GHCI. \n\n## How to attend this workshop\nThis workshop is open to Rootconf memb ers: https://hasgeek.com/rootconf#memberships and ticket holders of Rootco nf 2026 Annual Conference: https://hasgeek.com/rootconf/2026/#tickets\n\nT he workshop is open to 30 in-person participants\, with hybrid access avai lable for remote attendees. Seats for in-person participants will be avail able on a first-come\, first-served basis.\n\n## Contact information\nFor inquiries about the workshop\, contact +91-7676332020 or write to info@has geek.com LAST-MODIFIED:20261025T163629Z LOCATION:Bangalore - https://hasgeek.com/rootconf/quantization-workshop/ ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/quantization-workshop/ BEGIN:VALARM ACTION:display DESCRIPTION:Break a model\, then fix it in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT END:VCALENDAR