Face matching is one of the highest-volume workloads in identity verification. At IDfy, a single GPU pod handling 1 RPS cost us ₹3,500/day. After moving the model to BF16 inference on Intel CPUs via OpenVINO, the same 1 RPS pod cost ₹350/day. Same TAT, same throughput, same accuracy envelope. At our traffic shape (50 RPS sustained for the peak hour, 10 RPS for the remaining 23), that translates t… more

Inference cost-to-serve is usually treated as a fixed tax: the model needs a GPU, the GPU costs what it costs, and the bill scales with traffic. It isn’t fixed. For a large class of production models — embeddings, CNNs, classic CV and NLP — quantization plus graph fusion turns that GPU tax into a variable you control, cutting cost-to-serve by ~10× at the same latency, throughput, and accuracy env… more

Decoding at a million tokens when the KV cache lives in host memory: what actually bounds the step more

Submission type: Talk/session proposal - 30-40 mins. more

Inside a bit-plane KV index: superblock layout, per-query water-filling, and why the kernel is slower in HBM more