Rootconf Mini 2024

Introduction

The observability stack in modern organizations often consists of multiple vendors handling logs, metrics, and traces. This results in inconsistent data formats and conventions, increasing the operational overhead of maintaining these pipelines.

SREs must navigate various agents, each with its debugging methods. Integrated solutions that combine logs, metrics, and traces tend to use proprietary formats, leading to vendor lock-in. These all-in-one solutions often provide only the most basic features, creating gaps in production observability.

The OpenTelemetry (OTel) project aims to address these issues. As one of the largest CNCF projects in the observability space, it offers a unified agent that uses open standards to collect and send logs, metrics, and traces. OTel supports multiple observability backends, reducing the complexity and avoiding vendor lock-in.

What are the building blocks of OTel?

OTel can be overwhelming for beginners due to its many terminologies and steep learning curve. We will learn the essential basics you need to know to set up your own OpenTelemetry pipeline.

What does it take to run OTel in production?

OTel’s adoption is still evolving, and while there are many resources available, structured guidance on running it in production is sometimes hard to find. We will explore how to architect systems handling terabytes of production data through OTel pipelines. Some examples of these systems include:

• Datadog agents performing dual writes to both Datadog and OTel-supported backends.
• Third-party solutions streaming data to AWS S3, which is then pushed to OTel-supported backends (an approach used by platforms like Cloudflare and Fastly).
• Leveraging AWS S3 as cold storage to rehydrate your observability backend when needed.
• Integrating open-source shippers like Fluentd and Logstash with OTel for seamless data flow.

Key takeaways for the Audience

• Clear understanding of OTel building blocks: A straightforward explanation of the essential components of OpenTelemetry.
• Integrating OTel in production: Practical guidance on how to implement OTel for various production workloads.
• Architectural design patterns: Design patterns for setting up OpenTelemetry in different system architectures.
• Migrating from proprietary solutions to OTel: Overview of the approaches taken to migrate from vendor-specific observability tools to OpenTelemetry.
• Reference setups: I will share a github repository for most of the above OTel configurations.