Fine-Tuning SLMs with LoRA: Building Specialized On-Device Story Generators

Description

As generative AI moves closer to the edge, developers are looking for ways to build creative, high-quality applications that run privately, efficiently, and without dependence on cloud APIs. This session explores how to shift from server-side story generation using large language models (LLMs) to highly optimized on-device workflows powered by small language models (SLMs). Attendees will learn the end-to-end process of generating content in the cloud, reproducing it locally, and progressively improving output quality using prompt tuning and LoRA-based fine-tuning.

Through a series of practical demonstrations, we will walk through three stages of on-device model refinement: baseline inference, prompt-tuned enhancement, and LoRA-based adapter tuning for personalization. Participants will compare outputs from each stage, understand the trade-offs in quality vs. performance, and learn lightweight evaluation methods for generative storytelling. By the end, they will know how to build efficient, privacy-preserving, specialized story generators that can run directly on mobile or embedded devices.

Key Takeaways

• Learn how to migrate generative workflows from cloud LLMs to optimized, on-device SLMs.
• Understand and apply prompt tuning and LoRA adapter tuning to personalize model behavior.
• Gain practical methods to evaluate improvements in on-device generative quality.

Target Audience

• Level: Intermediate
• Prerequisites:
  • Basic Python knowledge; familiarity with foundational LLM concepts
  • Laptop and internet connectivity!
• Best suited for:
  • AI/ML developers building edge or offline applications
  • Mobile developers exploring on-device inference
  • Researchers working with small, efficient model architectures
  • Engineers evaluating personalization strategies for constrained devices

Mayur is a seasoned engineer specializing in AI, data, and backend systems, with extensive experience building scalable, high-performance platforms at organizations such as JioHotstar, Intuit, Walmart, and SAP. He frequently delivers webinars and technical sessions on AI engineering and distributed systems, and actively shares insights with the developer community. Connect with him at: https://www.linkedin.com/in/mayurmadnani/