Problem

Myntra is one of the leading fashion e-commerce companies in India. Myntra delivers best-in-class shopping experience by leveraging many advanced machine learning models, deployed for online or real-time inference. The online inference requires streams of data to be processed, machine learning features to be computed, stored and served in (near) real-time, at Myntra scale.

The features can be hand crafted or generated (e.g. user, product, style, widget, image embeddings). And majority of the features require stateful stream processing with complex computation, in (near) real-time, at very high throughputs (millions of rpm) and low latency. This requires scalable, resilient data engineering systems with stateful stream processing capabilities and feature stores.

Solution

Myntra Data Engineering team designed and built Quicksilver, a real time data ingestion and stateful stream processing platform. It is part of the overall Myntra Data Platform. The Quicksilver platform ingests millions of events every minute, computes the machine learning features in (near) real time and makes them available to machine learning models for online inference.

Outline of the talk

Online ML use cases at Myntra
Life cycle of an online ML model, including feature engineering
Challenges of realtime feature engineering at scale
Functional and non-functional requirements
Architecture of the QuickSilver platform, design principles and tech choices
Integration with Machine learning platform
Best practices and learnings