Topical Edition on Databases

Description

There are real advantages to building a specialized database: better performance, less impedance mismatch, and lower operational cost. But the conventional wisdom against rolling your own exists for a reason. ACID is hard, and general-purpose systems are reliable precisely because they’ve been battle-hardened over decades. That calculus has recently changed - cheap object storage and a rich ecosystem of open-source primitives have collapsed the cost of building a new database. Modern testing methods are doing the same for the cost of trusting one.

In this session, I’ll present Pangolin, a production OLAP database used at Antithesis, an autonomous software testing company whose deterministic hypervisor and fuzzer simulate billions of program states. The resulting data, billions of rows with monstrous cardinalities, little schema, and complex query patterns, breaks general-purpose databases. We’ll cover the design decisions that let Pangolin handle this workload, and the testing strategy that made it possible for three engineers to design and deploy it in only 14 months: a PBT-forward approach to development where the same properties catch correctness bugs at the source, watch for violations in production, and guide fuzzing in deterministic simulation. We’ll walk through real bugs from each layer, why we’d never have caught them with conventional testing, and what lessons other teams building reliable software can take away.

Takeaways

• Properties test what your code should do, not what you remember to check. Example-based tests catch bugs you’ve already thought of. PBT lets you bake in invariants and find counterexamples hidden in the seams between assumptions.
• One property, many jobs. The same invariant represents failures in local testing, logs in production, and guides fuzzing in simulation. One specification can replace a stack of test suites and lets a small team ship a reliable database in the time it usually takes to write one.

Target Audience

Engineers interested in specialized data systems, property-based testing, or simulation testing as practical tools for shipping correct and reliable production software fast!

Bio

Shomik Ghose is a database engineer at Antithesis, where he helped design and build Pangolin. Shomik is a fan of distributed systems, software correctness and dogs.

https://drive.google.com/file/d/1hBTAh-84wDqT93WtnFqfBmsVVrid7Anu/view?usp=sharing