Problem

Myntra is one of the leading fashion e-commerce companies in India. Myntra delivers best-in-class shopping experience by leveraging many advanced machine learning models, deployed for online or real-time inference. The online inference requires streams of data to be processed, machine learning features to be computed, stored and served in (near) real-time, at Myntra scale.

The features can be hand crafted or generated (e.g. user, product, style, widget, image embeddings). And majority of the features require stateful stream processing with complex computation, in (near) real-time, at very high throughputs (millions of rpm) and low latency. This requires scalable, resilient data engineering systems with stateful stream processing capabilities and feature stores.

Solution

Myntra Data Engineering team designed and built Quicksilver, a real time data ingestion and stateful stream processing platform. It is part of the overall Myntra Data Platform. The Quicksilver platform ingests millions of events every minute, computes the machine learning features in (near) real time and makes them available to machine learning models for online inference.

Outline of the talk

1. Online ML use cases at Myntra
2. Life cycle of an online ML model, including feature engineering
3. Challenges of realtime feature engineering at scale
4. Functional and non-functional requirements
5. Architecture of the QuickSilver platform, design principles and tech choices
6. Integration with Machine learning platform
7. Best practices and learnings

Presenters

Aditya S ( Tech lead, Quicksilver Near Realtime Platform )
Narayana Pattipati ( Senior Architect, Myntra Data & Machine Learning Platforms)