Hotstar Pulse - Processing 25M+ beacons every minute in real-time

Submitted Jul 30, 2019

Section: Full talk Technical level: Intermediate Session type: Lecture

At hotstar, one of the important metrics which we present to end-users and our business partners is the number of people watching live. While this may seem to be a simple problem of rendering a number, but it ain’t. It’s not an approximate number, it’s an actual number and it need to be presented in real-time to users. Each unique user is counted in real-time, every minute we receive millions of beacons and we process it in real time. At peak when we did 25.3M, we were getting about 30M beacons every minute from across the world.

Outline

The goal of the talk is to present the approach of how we created the design for calculating this number using pure open-source tech.
Objective of the Pulse
Be able to calculate 50M users in real time.
Keep the cost low
Avoid Data loss
Assist Scaling of other system by predicting upcoming traffic
Background on Hotstar 25.3M LIVE
What were the solutions evaluated for building this system
Why not a simple Telegraf + InfluxDB worked for this use case.
The framework for calculating any metric in real time
Ensuring that system was horizontally scalable as the hotstar scaled.
Resilient & Recoverable design.

Requirements

None

Speaker bio

I am a architect & an evangelist at hotstar. I have been associated with hotstar for past 1.5 Years, where I have worked on many projects from preparing the platform for 25M to delivery never done before Server-Side ad insertion on Live.
Before hotstar, I had worked with multiple startups. I had head engineering for BabyChakra (A parenting app), and had lead teams for TinyOwl, HackerRank etc.

I am a full-stack engineer and a hard-core design & architecture geek. I love building distributed systems which work at scale!

Slides

https://docs.google.com/presentation/d/1I0zsA3lT2LqQjcbo3lykQuP5yaityql7gkmzNytRA8w/edit?usp=sharing

Call for round the year submissions for Rootconf in 2020