Build your own LSM Tree in Rust

🔍 Workshop overview

LSM Trees are behind a lot of database technologies out there when it comes to storing and managing data in database systems. Understanding them makes one better at managing their data and make the right tradeoffs in their software systems. Most modern databases like ClickHouseDB, Cassandra, ScyllaDB, and YugabyteDB use LSM Tree as their underlying storage engine.

In this hands-on workshop, participants will learn how to get started with their own toy implementation of LSM Trees - enough to:

• Reason about how these database systems work under the hood.
• Use this implementation as a baseline to build their own storage engines for niche use cases.

Prerequisite: Cursory familiarity with the Rust programming language will be helpful for participating in this workshop, but we’ll go through a Rust 101 session too.

Note

• This workshop is of ~3 hours duration.
• This is an in-person and hands-on workshop.
• Only 25 seats are open for workshop participation.
• Recording of the workshop will be made available for Rootconf members.
• Workshop level: Intermediate
• Code & materials can be accessed at https://github.com/creativcoder/rootconf-25-lsmtree

🧭 Agenda

1. Introduction & set-up

   • What’s an LSM Tree
   • Why learn about LSM Trees?
   • LSM Tree in the wild
   • Environment setup and finished demo
   • Limitations / assumptions of our toy implementation

2. In-memory LSM Tree

   • Implementing a basic in-memory LSM Tree
   • Add support for basic get, put, and delete

3. Adding data persistence via SSTables

   • Why SSTables
   • Flush criteria
   • Implementing SSTables

4. Adding SSTable to LSM Tree

   • Refactoring LSM Tree API to incorporate SSTables
   • Recovery of data.
   • Testing our implementation

5. Compaction and Garbage Collection

   • Implementing compaction on SSTables
   • Testing our implementation

6. Wrap-up & Q&A

   • Recap key learnings
   • Share next steps and resources
   • Open discussion

💻 Prerequisites

• Your laptop - Windows, MacOS, or Linux
• Rust language compiler toolchain installation: https://rustup.rs

👥 Who should attend

• Anyone curious about how popular databases store and manage data under the hood
• Anyone who wants to explore Rust for building systems software like databases
• Backend developers who want to solidify their understanding of key-value-like database
• Developers exploring how to build low-level database software components

📚 What will participants learn?

By the end of this workshop, participants will:

• Understand what LSM Trees are and why understanding them helps you manage your databases better
• Learn how to build your own database storage engine from the ground up
• Appreciate the elegance and expressiveness of the Rust programming language in building system software components

👩 💻 Instructor bio

Rahul Sharma, currently an indi-hacker, has been tinkering with Rust since pre 1.0 🦀. He writes about his tech experiements at https://www.creativcoder.dev/
Previously, Rahul was a senior software engineer at Ather Energy (data pipelines), and ChainSafe Systems (Forest daemon in Rust). His interests lie at the intersection of time series, concurrent programming, database internals, and GUI libraries.

How to attend this workshop

This workshop is open for Rootconf members and for Rootconf 2025 ticket buyers

This workshop is open to 20 participants only. Seats will be available on first-come-first-serve basis. 🎟️

Contact information ☎️

For inquiries about the workshop, contact +91-7676332020 or write to info@hasgeek.com.