Overview

Have you ever had an alarm fail to wake you up, causing a ripple effect of chaos in your morning? At PhonePe, we understand the criticality of such ‘alarms’ in our digital ecosystem.

Take, for instance, our daily Merchant Settlements process. A merchant receives multiple transactions during the day. At the end of the day, we want to ensure the final amount gets credited to their account. A potential delay in this routine job being executed means a merchant not receiving their earnings on time, shaking their trust as a PhonePe Customer.

At PhonePe, we face the colossal task of managing over 2 billion daily callbacks. The ability to handle over 100,000 job schedules per second with single-digit millisecond latency is not just a goal; it’s a necessity. At p99, our system ensures that there’s no lag in job execution, which in the worst case is capped at 1 minute. Follow along to learn how we’ve made this possible.

In this talk, we will take a look at the internals of Clockwork – the system that powers job scheduling across various teams at PhonePe and enables clients to easily onboard and schedule future jobs without the need for heavy lifting on their own.

Agenda

• Why we developed Clockwork
• Deep dive into design and architecture of providing Callbacks at scale (100k callbacks per second)
• Tech stack : Hbase, RMQ, Zookeeper

Takeaways

• How Zookeeper helps in achieving consensus in distributed systems.
• Do’s/Dont’s for managing infrastructure at scale.

Audience

Site Reliability and DevOps Engineers
Engineering leaders
Cloud architects and engineers

Bio

Snehasish is a Software Engineer at PhonePe with around 10 years of experience building distributed systems and core platforms.
Linkedin: https://linkedin.com/in/snehasishroy

Review

This submission went through technical review once https://hasgeek.com/rootconf/clockwork-job-scheduler-technical-review/

• Link to Slide: https://docs.google.com/presentation/d/1Zxwgpcz-dYYbt3sZ44pcEuHk0erkAWqbkrr9E7Gh5xU/
• Link to Blog: https://tech.phonepe.com/clockwork-the-backbone-of-phonepes-2-billion-daily-jobs/