Patterns for building a scalable Data Platform
Technical level: Intermediate
Data-driven applications bring immense value to the business. While teams want to build data-driven products, the intricacies of building reliable and scalable ingestion, storage, and computation data platform are challenging.
At Hotstar, we built a unified Data Platform that abstracts all these nuances and provides a seamless experience to the end user.
In this talk, we will discuss the lessons learned building a scalable Real-Time Streaming Data Platform at Hotstar.
- Ingestion Patterns
- Unified Ingestion Proxy
- Schema Definitions
- In-flight enrichments
- Highly Available
- Storage Patterns
- Decouple storage and compute
- Query Lineage & Optimization
- Noisy Neighbour
- Consumption Patterns
- Single GUI and a programmatic interface. All the magic underneath
- Parity between streaming and stationary data
I lead the Data Infrastructure team at Hotstar. Over the last 2 years, the company has grown from handling a peek concurrent users of 3 million to 10.3 million.
Along with it has grown the scale at which the Data Platform operates. During the finals of the last IPL, our platform ingested 700K messages per second. In this talk, I will share our story of building the data platform and the challenges we faced during the process.
Product development for fleet management
India’s last mile delivery network is on a rapid rise in the last decade. With the boom in e-commerce being replicated across verticals of food, grocery and pharmacy, last-mile logistics has become a massive scale phenomena. Fast deliveries and high NPS are the keywords that everyone focusses on. In this talk, we will focus on what goes under the hood to enable a pleasant experience to the end cu… more