The Fifth Elephant 2026 Annual Conference

Inference cost-to-serve is usually treated as a fixed tax: the model needs a GPU, the GPU costs what it costs, and the bill scales with traffic. It isn’t fixed. For a large class of production models — embeddings, CNNs, classic CV and NLP — quantization plus graph fusion turns that GPU tax into a variable you control, cutting cost-to-serve by ~10× at the same latency, throughput, and accuracy envelope.

This is the operational story of doing that across a 40+ model fleet at IDfy, where we did these migration in a repeatable way, enough times to become a playbook: calibrate → quantize → fuse → validate → canary. I’ll work it through one concrete example — a high-volume face-match model moved from GPU to BF16 inference on Intel CPUs via OpenVINO, dropping a 1-RPS pod cost by 10X. I’ll then show two more migrations from the fleet, including one that broke in production and the telemetry that caught it in shadow mode before a single user was affected.

The savings are the headline (~₹50 lakh/year and climbing), but the second-order wins were just as impactful:

GPU capacity reclaimed for the workloads that genuinely need it
Enabling cutting edge models for single tenant customers without GPU access with a lower TCO
A smaller, burstier CPU footprint you can serve from cheaper spot pools with far more elastic autoscaling.

This is not a “CPU beats GPU” pitch. It’s about matching each workload to the cheapest hardware that still meets its accuracy and latency budget — and the engineering discipline that tells you, before production does, which models survive the move and which won’t.

Takeaways

A cost-vs-latency decision matrix for GPU vs. quantized-CPU inference — and the one metric that actually predicts whether a migration survives production (it isn’t average latency).
The three quantization failure modes we see most often — tail blur, silent FP32 fallback, and calibration drift — and the specific observability signal that catches each before users do.
The business case beyond the raw bill: how cheaper cost-to-serve makes single-tenant, per-client deployments economically viable (low TCO, compliance-friendly) and frees GPU capacity for cheaper spot-based autoscaling on the workloads that truly need it.

Audience

Production ML and AI engineers, platform and infra teams, and engineering leaders who own inference cost-to-serve at scale, plus solution architects who deploy models into client environments, where deployment model and TCO are part of the solution

Bio
Vivek Kalyanarangan is Sr. Technical Architect, AI at IDfy, where a 20-person team operates 40+ production ML models across biometric authentication, document recognition and OCR, fraud detection, and large-scale NLP. He has 13+ years across analytics, big data, and deep learning. Author of Quantization and Fast Inference (Manning, MEAP 2026) and the freeCodeCamp course LLMs from Scratch. Contributor to open-source ML and published papers.