The Fifth Elephant 2024 Annual Conference (12th &13th July)

Maximising the Potential of Data — Discussions around data science, machine learning & AI

Swapnik

Pramanik

@pramanik

Harsh Lath

@harshlath

Solving the Data Platform Puzzle: Observability Meets Cost Optimization

Submitted Jun 10, 2024

Outline

This session is aimed at data platform engineers, data architects, and engineering leaders who are looking to significantly reduce costs while maintaining or improving platform performance and reliability. The content will be tailored to those with a strong technical background who are facing challenges around optimizing complex data pipelines and infrastructure.

Problem Statement

Many organizations struggle with rising costs associated with their data platforms due to inefficiencies in data ingestion, transformation, storage, and querying. Without granular observability at each stage, it becomes difficult to identify and address cost drivers, leading to overprovisioning and wasted resources. This session will demonstrate how implementing comprehensive observability allowed us to cut our data platform costs in half.

Scope

The session will cover the end-to-end journey of instrumenting our data pipelines and infrastructure with detailed metrics and dashboards. Key areas of focus will include:

  • Ingestion and transformation layers (e.g. capturing EC2 utilization)
  • Storage layer (e.g. S3 cost optimization through partitioning, object lifecycle management)
  • Query layer (e.g. Trino query metrics to identify expensive queries and tune performance)
  • Infrastructure automation (e.g. Karpenter profiles for auto-scaling, resource consolidation)
  • Cost dashboarding and attribution

Takeaways

  • Attendees will leave with a practical playbook for implementing observability in their own data platforms to identify and eliminate sources of waste.
  • They will gain insights into configuring metrics at each key stage, optimizing compute and storage footprints, and enabling a cost-conscious culture through cost attribution and dashboarding.
  • The 50% cost optimization we achieved at slice DE serves as a compelling case study for the impact of these best practices.

All the takeaways have worked for us at slice and may have impact similar workloads as well.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Jump starting better data engineering and AI futures

Supported by

Gold Sponsor

Atlassian unleashes the potential of every team. Our agile & DevOps, IT service management and work management software helps teams organize, discuss, and compl

Silver Sponsor

Together, we can build for everyone.

Workshop sponsor

Datastax, the real-time AI Company.

Lanyard Sponsor

We reimagine the way the world moves for the better.

Sponsor

MonsterAPI is an easy and cost-effective GenAI computing platform designed for developers to quickly fine-tune, evaluate and deploy LLMs for businesses.

Community Partner

FOSS United is a non-profit foundation that aims at promoting and strengthening the Free and Open Source Software (FOSS) ecosystem in India. more

Beverage Partner

BONOMI is a ready to drink beverage brand based out of Bangalore. Our first segment into the beverage category is ready to drink cold brew coffee.