In this talk, we will explore the growing need to fine-tune large pre-trained models for specialized tasks, and the limitations of conventional fine-tuning methods—especially their high computational and storage costs. We begin with Parameter-Efficient Fine-Tuning (PEFT) techniques, focusing on LoRA (Low-Rank Adaptation), an adapter-based approach that enables efficient model adaptation by introd… more

Abstract Large pre-trained models are now the norm, making Parameter-Efficient Fine-Tuning techniques like LoRA essential to reduce computational and storage costs. But why do these methods work so well? This talk explores the theory of Intrinsic Dimension (ID)—the idea that neural networks often need far fewer effective directions to learn a task than their total parameters suggest. more

Abstract Training large deep learning models like LLMs and vision transformers has traditionally required high-end GPUs with large memory, making them inaccessible to many. This talk explores how Fully Sharded Data Parallel (FSDP) in PyTorch can help overcome this barrier by enabling large model training and fine-tuning on smaller GPUs (8–16GB), using commodity hardware or affordable cloud credit… more