Open Models That Keep Getting Better: A Self-Improving Distillation Pipeline on Kubeflow ecosystem

Description

Most open-model demos stop at a single fine-tuning run and a benchmark score. This project goes further: an autonomous pipeline on kubeflow pipeline and the whole kubeflow ecosystem that distills a 32B open-weight (or any LLM) teacher (Qwen2.5-Coder) into a 1.5B SLM student, evaluates it with a structured judge framework, generates targeted training data from its weaknesses, retrains through SFT → DPO → GRPO, and promotes the model to production only when it passes a quality gate , then repeats the cycle. The pipeline is not a notebook or a one-shot script; it is a Kubeflow DAG with orchestrated components, checkpoint chaining across cycles, and eval-gated canary deployments via KServe.

The session focuses on production mechanics and real engineering learnings from running this on a live cluster: preference pairs extracted from real eval failures, GRPO reward functions aligned with evaluation judges, automatic checkpoint selection between training stages, and an outer loop that analyzes weak categories and feeds the next iteration. Everything runs on-cluster with open models, Kubeflow Pipelines, KServe, MLflow, and MinIO — no proprietary APIs, no data leaving your infrastructure. The talk is built around one question: how do you turn open models into production assets that improve over time?

Production challenges & learnings

1. GPU scheduling — Training, Ollama (teacher), and KServe (student) competed for the same GPU pool. We automated KServe scale-to-zero before TrainJobs and reserved node capacity.
2. Multi-stage fine-tuning (SFT → DPO → GRPO) — DPO didn’t always beat SFT (AB_2: 73.3% vs 60%). We added checkpoint routing between stages, aligned train/serve prompts, and built preferences from real eval failures only.
3. Model promotion across cycles — Continual learning needed durable S3 gate markers and version resolution logic, not just MLflow tags, so each cycle starts from the previous best checkpoint.

What the live walkthrough will show

A pipeline cycle starts from the previous cycle’s best checkpoint. The audience will see the full progression in real infrastructure:

• The Kubeflow DAG executing end-to-end on OpenShift AI
• MLflow training curves across SFT, DPO, and GRPO — loss decreasing, rewards climbing
• The eval harness scoring correctness, conciseness, format compliance, and review quality
• Automatic checkpoint routing between DPO and GRPO so the RL stage always starts from the strongest model
• The outer loop identifying weak categories and generating targeted training data for the next cycle
• Three complete cycles (AB_1 → AB_2 → AB_3) with visible knowledge retention — each cycle’s SFT loss starts lower than the last
• How the production fixes above show up in the actual DAG, metrics, and promotion flow

Takeaways from session

1. Open models plus open infrastructure can power a closed-loop improvement system today — Kubeflow, KServe, MLflow, and Hugging Face TRL on OpenShift AI give you a full distillation flywheel with data sovereignty and no vendor lock-in.

2. Eval-driven deployment is the bridge from experimentation to production : the same judges that gate promotion can align with GRPO rewards, turning evaluation from a post-hoc check into the engine that drives continuous improvement.

3. Staged fine-tuning needs production guardrails :GPU orchestration, checkpoint routing between stages, and explicit promotion state across cycles are what make SFT → DPO → GRPO reliable in practice, not just on paper.

Which audiences is this session going to be beneficial for?

This session is especially useful for ML engineers, platform engineers, and technical leads running or planning to run open-weight models in production on Kubernetes or OpenShift.

It will also be relevant for:

• Teams moving from one-shot fine-tuning to iterative model improvement pipelines
• Engineers exploring DPO and GRPO in production settings
• AI infrastructure teams building on Kubeflow, KServe, MLflow
• Organizations that need data sovereignty: everything runs on-cluster
• Architects designing agentic systems where specialized open models replace large general-purpose ones

About me

I am Sridhar Pillai a Software Engineer at Red Hat, working on AI/ML infrastructure for OpenShift AI. I built the agentic continual learning pipeline spanning Kubeflow Pipelines, the Training Operator v2, KServe, and MLflow , focused on making open-model training and serving production-ready on Kubernetes.

• Demo Slides Canva
• LinkedIn: linkedin.com/in/sridharpillai
• GitHub: github.com/Sridhar1030
• Blog on this: Building a Self-Improving Distillation Pipeline on Red Hat OpenShift AI