SRE Conf 2023
Rootconf
Rootconf For members

SRE Conf 2023

Availability and reliability 24/7- the SRE life

Tickets

Loading…

Schedule for the conference on 24 November is published.

Why SRE Conf?

When any organization goes from product market fit or beta test phase to production rollout, or from first x customers to 10x or 100x customers and starts scaling, they typically start running into challenges with systems stability and resiliency. These challenges change with every phase of growth. So does the need for having a SRE team and/or a DevOps team, and the role these teams play.
Unfortunately, there is no one-size-fits-all solution when it comes to what roles these teams should play, and which tools various teams should use to track the metrics and processes involved. But there are some common building blocks that apply in similar (and different) ways and forms for most teams. The idea of the SRE Conference is to get together and to know about these building blocks, share and learn about the themes that fall under the SRE umbrella.

SRE Conf tracks

SRE Conf is a two-track conference. The track, “Culture, career and Evolution” is more focused on leadership, team, and organizational topics while the “Stories from the Trenches” track will cover real-world scenarios, and lessons learned which will help engineers and engineering teams to upskill themselves by understanding experiences from their industry peers.

Culture, career, and evolution

  1. SRE v/s DevOps v/s Platform Engineering teams in organizations.
  2. Hiring and building SRE teams.
  3. Blameless postmortems.
  4. Role of AI in SRE/DevOps/Platforms.
  5. FinOps and cost optimization.
  6. SRE Anti-patterns

Stories from the trenches:

  1. Incident management.
  2. Change management.
  3. Scalability and performance.
  4. SLA/SLO and golden signals.
  5. Security and DevSecOps.
  6. Systems and networking.

Key takeaways for participants

  1. Improved understanding of organizational needs and requirements.
  2. Enhanced optimization skills.
  3. Networking opportunities.
  4. Knowledge sharing and community building.

Who should participate

  • Members of SRE, DevOps or platform teams.
  • A software developer or manager who is responsible for services running on any cloud platform or on-prem data center.
  • Technology leader of an engineering team that manages critical systems which should have minimal to zero downtime.

Speaking

If you are interested in speaking at the conference, submit your talk idea here. The editors - Sarika Atri, Safeer CM and Saurabh Hirani - will review your talk description and give feedback.

Speakers will also receive feedback and assistance during rehearsals from past speakers such as Sitaram Shelke.

Guidelines for speaking, speaker honorarium policy, and travel grant policy details are published here.

About the editors

This conference themes were set up by Sarika Atri and Safeer CM. Together with Saurabh Hirani, the three editors have:

  1. Reviewed the talks.
  2. Set up the editorial workflow.
  3. Finalized talk selections.
  4. Curated the schedule.

Sarika Atri is Software Architect with over twenty years experience in the industry. Sarika was reviewer for Rootconf Cloud Costs Optimization conference held in July 2023.
Safeer CM is Senior Staff SRE at Flipkart. He is author of Architecting Cloud-Native Serverless Solutions published by Packt.
Saurabh Hirani is former editor of Rootconf, and a passionate member of the community. Saurabh is SRE at Last9.io,. He has a keen interest in mentoring speakers.

Become a Rootconf Member to join

SRE Conf is a community-funded conference. It will be held in-person. Attendance is open to Rootconf members only. Support this conference with a membership. If you have questions about participation, post a comment here.

Sponsorship

Sponsorship slots are open for:

  1. Tool and solutions providers.
  2. Companies interested in tech branding for hiring.

Email sponsorship queries to sales@hasgeek.com

Contact information

Join the Rootconf Telegram group at https://t.me/rootconf or follow @rootconf on Twitter.
For inquiries, contact Rootconf at +91-7676332020.

Hosted by

Rootconf
Rootconf
We care about site reliability, cloud costs, security and data privacy
All submissions
This submission has been added to the schedule

Preeti

Deep dive into analyzing high cardinality metrics

Submitted Sep 30, 2023

Introduction

Metrics are the fundamental unit of time series databases (TSDB). They consist of labels which denote dimensions e.g. http_status_code, url, etc. Critical insights require metrics to have both breadth (e.g. large number of labels) and depth (e.g. each label having a large number of unique values). This makes up metric cardinality. Higher cardinality == deeper insights. This talk will assume Prometheus-like systems for reference.

Problem

In today’s world of microservices, distributed services having dimensions like tenant, region, and service invariably lead to high cardinality. Unchecked cardinality growth can lead to adverse effects like higher resource consumption, slow load up of dashboards, alerting queries failing, observability systems going blank, reduced retention time, etc.

Avoiding these situations at the enterprise scale requires solutions to understand what is causing high cardinality and how to manage it. This is often an afterthought which ignores the tooling to answer these questions:

  1. How to find your TSDB cardinality limits?
  2. How do you know when your system is approaching these limits?
  3. When it does approach them, how to find out which metrics are contributing to it?
  4. How to dissect these metrics to find the labels which led to cardinality explosion?
  5. What actions need to be taken to fix this?

Solution

The default solutions hover around finding labels with high cardinality and dropping them. But I have seen that in customer production environments, blindly dropping labels gives a false sense of belief of fixing the problem without actually solving anything. I wrote a tool to analyse high cardinality systems, get insights out of them to answer these questions:

  1. What is the overall state of my system - are there any metrics approaching limits?
  2. Which labels of which metrics have the probability of causing cardinality explosion?
  3. If you choose to drop/aggregate these labels - what will the end state look like?
  4. How not to think about dropping labels as the default solution e.g. there are corner cases where dropping the label with highest cardinalty has zero impact on reduction.

Benefits

The audience of this talk will have the following take aways:

  1. Fundamental knowledge to question assumptions on approaching systems with high cardinality metrics.
  2. A handy open source cardinality debugger/explorer well tested in customer production environments to analyze your Prometheus like TSDB systems and have the right numbers upfront which will help you choose where to invest your time - updating your instrumentation code, changing your metric agent configuration, etc.
All submissions

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hybrid access (members only)

Hosted by

Rootconf
Rootconf
We care about site reliability, cloud costs, security and data privacy