Rootconf 2026 Annual Conference

Title:

TraceQL Internals: Building a Query Language for Distributed Traces

One-line summary:

How Grafana Tempo stores and queries petabytes of trace data on object storage, and why and how we built TraceQL, a query language designed for traces.

What problem are you addressing? What real engineering challenge, question, or experience is this session about?

Traces are large, deeply nested, and have high-cardinality attributes on span so indexing them in a database gets expensive fast at large scale.

Tempo started out as a daabase that only supported trace lookup by ID, backed by object storage with the goals to making tracing accessable by making it cheaper to store traces.

Our User kept asking to support search and aggregation across traces so we buildt a query layer on top of object storage without incresing the cost.
TraceQL is designed to express trace-specific questions like span relationships, structure that SQL and PromQL don’t support natively.

Who is the intended audience? Platform Engineering/SRE/DevOps/Infrastructure/Developers/Engineering leaders/Other

Platform Engineering, SRE, Infrastructure, and Developers building observability or data systems

Level: beginner/intermediate/advanced

Intermediate to advanced

List one or two practical takeaways.

1. How columnar storage on object stores can be a good way to handle high-volume telemetry data.
2. How TraceQL’s design follows from trace-specific use cases and OpenTelemetry’s data model helped us along the way.

What will you share?

Architecture and design decisions, implementation details, production experience, trade-offs and alternatives, open-source tooling.

Tempo’s storage layout with Apache Parquet, query planning and sharding across object storage, TraceQL language design and TraceQL metrics are all open source at: https://github.com/grafana/tempo

What is your experience with this problem?

Production system, open-source project, internal engineering project, hard-earned engineering lesson

What approaches failed, disappointed, or created unexpected problems?

Many, including how it was not support to support seach on the old block format, and how we started with a Apache Parquet based format, Nested Data Model in Apache Parquet and then how we evolved it across multiple iterations, and we at version 5 of the Apache Parquet based format.

What will you do differently today?

The ecosystem has new file based coloumer storage formats that has showed up in last 2-3 years, so if I was starting today, I would also evalauate those and see if if something like https://vortex.dev/ is a better fit over Apache Parquet.

What trade-offs did you consider?

Object storage and columnar files vs an index database (cost vs query latency).
A purpose-built language vs extending SQL or PromQL.
What we optimized for (storage cost, scale) and what we knowingly gave up (some query latency, indexing, etc).

How can this help other practitioners?

A design approach for storing and querying high-volume telemetry cheaply, and a way to evaluate index vs columnar-on-object-storage trade-offs.
It will also help folks think about designing a purpose built system from the ground up, and what decisions and tradeoffs are made along the way.

Current state

Production experience (and open source)

#observability #databases #scalability