The Fifth Elephant winter edition 2019

Data-driven applications bring immense value to the business. While teams want to build data-driven products, the intricacies of building reliable and scalable ingestion, storage, and computation data platform are challenging.

At Hotstar, we built a unified Data Platform that abstracts all these nuances and provides a seamless experience to the end user.

In this talk, we will discuss the lessons learned building a scalable Real-Time Streaming Data Platform at Hotstar.

Outline

• Ingestion Patterns
  • Unified Ingestion Proxy
  • Schema Definitions
  • In-flight enrichments
  • Highly Available
• Storage Patterns
  • Decouple storage and compute
  • Query Lineage & Optimization
  • Noisy Neighbour
• Consumption Patterns
  • Single GUI and a programmatic interface. All the magic underneath
  • Parity between streaming and stationary data

Speaker bio

I lead the Data Infrastructure team at Hotstar. Over the last 2 years, the company has grown from handling a peek concurrent users of 3 million to 10.3 million.

Along with it has grown the scale at which the Data Platform operates. During the finals of the last IPL, our platform ingested 700K messages per second. In this talk, I will share our story of building the data platform and the challenges we faced during the process.