Topical Edition on Databases

Description

We sized the storage layer for Nutanix’s Panacea.AI platform — 5 TB and 5 billion log lines a day — three different ways and got three answers an order of magnitude apart. Same workload, same retention, same ingest rate; the engines disagreed on storage by 37× and on CPU by 3×.

This 15-minute talk is the engineering case for why that gap exists, and what it takes to land on the small end of it in production.

We’ll cover why log analytics is unusually well-suited to a columnar layout — access patterns that are almost always per-bundle and time-windowed, compression headroom on raw log messages, and the operational economics that fall out of those two — and the four schema patterns that did the real work in production:

The cluster today holds 154 billion rows across logs, metrics, traces, and AI-generated incident reports, sustains 870k+ inserts/sec on a single node, and runs without a dedicated DBA. We’ll close with the one search-latency trade-off we accepted to land here, what it cost, what it didn’t, and the framework we now use to re-evaluate it quarterly.

Key Takeaways

1. Engine choice is a sizing decision, not a feature decision. The same 5 TB/day workload sized at 750 TB, 500 TB, or 20 TB depending on the storage model — the table above is the artifact you actually defend in a design review.
2. Why log analytics fits a columnar engine — access patterns, compression headroom, operational economics — together with the four production schema patterns (Delta+ZSTD, LowCardinality, tokenbf_v1, partition-aligned TTLs) that delivered the 37× disk reduction.
3. The trade-off we accepted: substring search latency moved from inverted-index speed to bloom-filter speed on a query class used in <15% of sessions. We’ll show the query-mix that made the call defensible.

Target Audience

SREs, platform engineers, and DBAs operating high-volume log telemetry. Engineering managers and architects evaluating storage engines for petabyte-class observability, log search, or AI workloads. Anyone running an inverted-index backend today who suspects the access patterns of their workload have outgrown the model they started with.

Bio

Sohham Seal — SDE-2 at Nutanix on the Panacea AI platform; works on the columnar ingestion and query layer that powers AI-driven incident triage across Nutanix’s customer fleet. His recent AI-related work includes GNN-based recommendation systems, biometric security, and EMG-based gesture prediction (IEEE TIFS, PCEMS Best Paper). Will present in Bangalore.

Mohit Gurnani — SDE-4 at Nutanix; architects and leads Panacea.AI, an agentic AI platform processing 20 PB+ of observability data annually across 29,000+ enterprise clusters — ClickHouse for high-cardinality analytics, Kubernetes-native event-driven pipelines, and LangGraph agents on top. IEEE-published author; previously presented at ICDMAI 2017. Designed the migration described in this talk. Will join Q&A virtually from Columbus, OH.