Open Models That Keep Getting Better: A Self-Improving Distillation Pipeline on OpenShift AI

Description

Most open-model demos stop at a single fine-tuning run and a benchmark score. This project goes further: an autonomous pipeline on kubeflow pipeline and the whole kubeflow ecosystem that distills a 32B open-weight (or any LLM) teacher (Qwen2.5-Coder) into a 1.5B SLM student, evaluates it with a structured judge framework, generates targeted training data from its weaknesses, retrains through SFT → DPO → GRPO, and promotes the model to production only when it passes a quality gate — then repeats the cycle. The pipeline is not a notebook or a one-shot script; it is a Kubeflow DAG with orchestrated components, checkpoint chaining across cycles, and eval-gated canary deployments via KServe.

The session focuses on the production mechanics that make this trustworthy and repeatable: preference pairs extracted from real eval failures, GRPO reward functions aligned with evaluation judges, automatic checkpoint selection between training stages, and an outer loop that analyzes weak categories and feeds the next iteration. Everything runs on-cluster with open models, Kubeflow Pipelines, KServe, MLflow, and MinIO — no proprietary APIs, no data leaving your infrastructure. The talk is built around one question: how do you turn open models into production assets that improve over time?

What the live walkthrough will show

A pipeline cycle starts from the previous cycle’s best checkpoint. The audience will see the full progression in real infrastructure:

• The Kubeflow DAG executing end-to-end on OpenShift AI
• MLflow training curves across SFT, DPO, and GRPO — loss decreasing, rewards climbing
• The eval harness scoring correctness, conciseness, format compliance, and review quality
• Automatic checkpoint routing between DPO and GRPO so the RL stage always starts from the strongest model
• The outer loop identifying weak categories and generating targeted training data for the next cycle
• Three complete cycles (AB_1 → AB_2 → AB_3) with visible knowledge retention , each cycle’s SFT loss starts lower than the last
• Domain portability: swap the config for a new task without changing the core pipeline

Takeaways from session

1. Open models plus open infrastructure can power a closed-loop improvement system today — Kubeflow, KServe, MLflow, and Hugging Face TRL on OpenShift AI give you a full distillation flywheel with data sovereignty and no vendor lock-in.

2. Eval-driven deployment is the bridge from experimentation to production — the same judges that gate promotion can align with GRPO rewards, turning evaluation from a post-hoc check into the engine that drives continuous improvement.

Which audiences is this session going to be beneficial for?

This session is especially useful for ML engineers, platform engineers, and technical leads running or planning to run open-weight models in production on Kubernetes or OpenShift.

It will also be relevant for:

• Teams moving from one-shot fine-tuning to iterative model improvement pipelines
• Engineers exploring DPO and GRPO in production settings
• AI infrastructure teams building on Kubeflow, KServe, MLflow, or OpenShift AI
• Organizations that need data sovereignty — everything runs on-cluster
• Architects designing agentic systems where specialized open models replace large general-purpose ones

About me

I am Sridhar Pillai a Software Engineer at Red Hat, working on AI/ML infrastructure for OpenShift AI. I built the agentic continual learning pipeline spanning Kubeflow Pipelines, the Training Operator v2, KServe, and MLflow , focused on making open-model training and serving production-ready on Kubernetes.

• LinkedIn: linkedin.com/in/sridharpillai
• GitHub: github.com/Sridhar1030
• Blog on this: Building a Self-Improving Distillation Pipeline on Red Hat OpenShift AI