The Fifth Elephant 2025 Annual Conference CfP

Abstract

Large pre-trained models are now the norm, making Parameter-Efficient Fine-Tuning techniques like LoRA essential to reduce computational and storage costs. But why do these methods work so well? This talk explores the theory of Intrinsic Dimension (ID)—the idea that neural networks often need far fewer effective directions to learn a task than their total parameters suggest.

We’ll estimate a task’s ID via random subspace training on an MLP for MNIST, reproducing results from foundational papers. Then, we’ll compare how LoRA approximates subspace training in compute, training time, and accuracy—clarifying key design trade-offs. LoRA succeeds not just from engineering but by exploiting the low-dimensional structure revealed by ID.
We also highlight PyTorch internals that enable flexible subspace training. This talk builds on a four-part blog series bridging theory and engineering.

Motivation & Background

1. The rise of PEFT in the large model era: Why full fine-tuning is wasteful.
2. The role of LoRA as a scalable, resource-friendly alternative.
3. Introducing Intrinsic Dimension (ID): Why models often don’t need all their parameters.

Subspace Training for Measuring ID

1. Intrinsic Dimension of a task and its implications.
2. Hands-on walkthrough: Measuring ID using random subspace training on an MLP for MNIST.
3. Reproducing results from original ID literature.

LoRA as a Subspace Method

1. Matching LoRA’s parameter budget to subspace dimensions.
2. Compare LoRA vs. subspace training in FLOPs, convergence speed, accuracy.

PyTorch Internals

1. How nn.Parameter allows selective fine-tuning.
2. Using register_buffer for fixed subspace bases.
3. Custom forward() logic to implement dynamic re-parameterization.
4. Why such flexibility is hard in static-graph frameworks.

Key Takeaways

1. ID gives a principled lens to understand PEFT.
2. LoRA can be seen as a constrained subspace optimizer.
3. Subspace experiments offer a tool to reason about model capacity and fine-tuning efficiency.

Additional Resources –

This talk builds on the four-part blog series on LoRA and Intrinsic Dimension). These blogs gained good visibility — receiving positive traction on /r/MachineLearning and ranking 7th on Hacker News (front page for a day).

While the first two blogs on LoRA became the basis of the talk at PyCon India 2024 and at the Fifth Elephant Open Source AI meet (April 2025), this submission is based on the later two blogs which dive deeper into intrinsic dimension and measuring model complexity offering new perspectives.