Rootconf Mini 2024 (on 22nd & 23rd Nov)

Geeking out on systems and security since 2012

This video is for members only

Preeti

Preeti

@preetidewani

Building OpenTelemetry Pipelines

Submitted Oct 14, 2024

Introduction

The observability stack in modern organizations often consists of multiple vendors handling logs, metrics, and traces. This results in inconsistent data formats and conventions, increasing the operational overhead of maintaining these pipelines.

SREs must navigate various agents, each with its debugging methods. Integrated solutions that combine logs, metrics, and traces tend to use proprietary formats, leading to vendor lock-in. These all-in-one solutions often provide only the most basic features, creating gaps in production observability.

The OpenTelemetry (OTel) project aims to address these issues. As one of the largest CNCF projects in the observability space, it offers a unified agent that uses open standards to collect and send logs, metrics, and traces. OTel supports multiple observability backends, reducing the complexity and avoiding vendor lock-in.

What are the building blocks of OTel?

OTel can be overwhelming for beginners due to its many terminologies and steep learning curve. We will learn the essential basics you need to know to set up your own OpenTelemetry pipeline.

What does it take to run OTel in production?

OTel’s adoption is still evolving, and while there are many resources available, structured guidance on running it in production is sometimes hard to find. We will explore how to architect systems handling terabytes of production data through OTel pipelines. Some examples of these systems include:

  • Datadog agents performing dual writes to both Datadog and OTel-supported backends.
  • Third-party solutions streaming data to AWS S3, which is then pushed to OTel-supported backends (an approach used by platforms like Cloudflare and Fastly).
  • Leveraging AWS S3 as cold storage to rehydrate your observability backend when needed.
  • Integrating open-source shippers like Fluentd and Logstash with OTel for seamless data flow.

Key takeaways for the Audience

  • Clear understanding of OTel building blocks: A straightforward explanation of the essential components of OpenTelemetry.
  • Integrating OTel in production: Practical guidance on how to implement OTel for various production workloads.
  • Architectural design patterns: Design patterns for setting up OpenTelemetry in different system architectures.
  • Migrating from proprietary solutions to OTel: Overview of the approaches taken to migrate from vendor-specific observability tools to OpenTelemetry.
  • Reference setups: I will share a github repository for most of the above OTel configurations.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

We care about site reliability, cloud costs, security and data privacy

Supported by

Platinum Sponsor

Nutanix is a global leader in cloud software, offering organizations a single platform for running apps and data across clouds.

Platinum Sponsor

PhonePe was founded in December 2015 and has emerged as India’s largest payments app, enabling digital inclusion for consumers and merchants alike.

Silver Sponsor

The next-gen analytics engine for heavy workloads.

Sponsor

Community sponsor

Peak XV Partners (formerly Sequoia Capital India & SEA) is a leading venture capital firm investing across India, Southeast Asia and beyond.

Venue host - Rootconf workshops

Thoughtworks is a pioneering global technology consultancy, leading the charge in custom software development and technology innovation.

Community Partner

FOSS United is a non-profit foundation that aims at promoting and strengthening the Free and Open Source Software (FOSS) ecosystem in India. more

Community Partner

A community of Rust language contributors and end-users from Bangalore. We have presence on the following telegram channels https://t.me/RustIndia https://t.me/fpncr LinkedIn: https://www.linkedin.com/company/rust-india/ Twitter (not updated frequently): https://twitter.com/rustlangin more