Tickets

Loading…

Ishika Mittal

@ishikamittal

From Logs to Insights: Building a Scalable Monitoring System with Loki and Mimir

Submitted Oct 24, 2024

Monitoring and collecting metrics and logs is essential for product and service improvement, particularly at scale due to the vast data size and diverse stakeholders. At e6data, we began our journey with our Gen3 Lake House native compute engine in a “Cloudprem” model. Metrics and logs are crucial for understanding customer interactions, such as query success rates and the reasons behind query failures. Currently, we utilize an open-source stack comprising Loki for logs, Mimir for metrics, and Grafana for visualization. This talk will detail our journey in leveraging Loki and Mimir to create an effective monitoring and alerting system.

KEY TAKEAWAYS:

  1. Effective Use of Open-Source Tools: Learn how to leverage open-source technologies like Loki and Mimir to build a robust monitoring and alerting system that scales with your data needs.

  2. Challenges of Scaling: Learn about the key challenges faced at scale regarding heavy data ingestion and reads, and strategies to effectively manage them.

AUDIENCE:

Platform engineering and DevOps engineering developers and enthusiasts working on platforms at scale, or starting their journey towards it.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hybrid Access Ticket

Hosted by

We care about site reliability, cloud costs, security and data privacy

Supported by

Platinum Sponsor

Nutanix is a global leader in cloud software, offering organizations a single platform for running apps and data across clouds.

Platinum Sponsor

PhonePe was founded in December 2015 and has emerged as India’s largest payments app, enabling digital inclusion for consumers and merchants alike.

Silver Sponsor

The next-gen analytics engine for heavy workloads.

Sponsor

Community sponsor

Peak XV Partners (formerly Sequoia Capital India & SEA) is a leading venture capital firm investing across India, Southeast Asia and beyond.

Venue host - Rootconf workshops

Thoughtworks is a pioneering global technology consultancy, leading the charge in custom software development and technology innovation.

Community Partner

FOSS United is a non-profit foundation that aims at promoting and strengthening the Free and Open Source Software (FOSS) ecosystem in India. more

Community Partner

A community of Rust language contributors and end-users from Bangalore. We have presence on the following telegram channels https://t.me/RustIndia https://t.me/fpncr LinkedIn: https://www.linkedin.com/company/rust-india/ Twitter (not updated frequently): https://twitter.com/rustlangin more