Rootconf 2025 Annual Conference CfP

Abstract

Trading systems need to process hundreds of GBs of data within minutes for backtesting strategies. In this talk, I’ll take you through the journey of Nautilus’s data streaming layer from reading records at 300k/s to 6M/s.

• What? Why? and How? of parquet
• Choosing the right parquet query engine
• Streaming and merging data at scale
• Correctness and performance testing

I will also share two war stories when the system started showing weird memory and performance issues ⚔️☠️ and what tools and techniques I used to debug and then fix it.

Takeaways

• Challenges and design decisions for high performance data streaming for parquet files
• Tools and mental models for debugging and fixing memory and performance issues

Audience

Platform and performance engineers and system architects who build and maintain fast systems that go brrr will enjoy this.

Bio

I am a senior rust engineer with Nautech Systems building their algorithmic trading engine platform. I specialize in building compilers and testing high-performance systems. I consult with companies and help them build fast, safe and easy to maintain systems.