Rootconf Mini 2024

Have you ever experienced an abrupt service shutdown in production due to the inability to monitor CPU utilization and memory spikes post-deployment? If so, you understand the critical importance of service metrics monitoring.

At PhonePe, we empower our engineers to continously monitor their systems using Opentsdb. On top of these metrics, we have built in house alerting system Anomaly detection which helps the teams to get real time alert for any anomalies. More than 200 clients push more than 400 billion metrics a day and peak touching 5 millon metrics per sec. We retain these raw metrics for 30 days and rolled up metrics for 365 days. Overall cluster footprint is close 80 Baremetals holding terabytes of data

In this talk, we will talk about -
Systems architecture of our Metrics platform along with Opentsdb.
We will deep dive into system system optimisations we have done over the years to scale our Kafka and HBase which acts as the backbone of our platform.
Production outages and remediations

Key take aways
How we scaled Opentsdb to handle 400 billion metrics a day
Rollup of metrics using Spark
Feedback loop to build intelligence system
Dos/Dont’s for managing larger infrastructure

This session is useful for :
Developers
SRE/Devops
Engineering Managers