The Fifth Elephant 2023 Monsoon

On AI, industrial applications of ML, and MLOps

Tickets

Loading…

Aditya S

@adityasridhar

Narayana Pattipati

@npattipati Editor

Near Real time feature engineering at scale for machine learning use cases at Myntra

Submitted Jun 30, 2023

Problem

Myntra is one of the leading fashion e-commerce companies in India. Myntra delivers best-in-class shopping experience by leveraging many advanced machine learning models, deployed for online or real-time inference. The online inference requires streams of data to be processed, machine learning features to be computed, stored and served in (near) real-time, at Myntra scale.

The features can be hand crafted or generated (e.g. user, product, style, widget, image embeddings). And majority of the features require stateful stream processing with complex computation, in (near) real-time, at very high throughputs (millions of rpm) and low latency. This requires scalable, resilient data engineering systems with stateful stream processing capabilities and feature stores.

Solution

Myntra Data Engineering team designed and built Quicksilver, a real time data ingestion and stateful stream processing platform. It is part of the overall Myntra Data Platform. The Quicksilver platform ingests millions of events every minute, computes the machine learning features in (near) real time and makes them available to machine learning models for online inference.

Outline of the talk

  1. Online ML use cases at Myntra
  2. Life cycle of an online ML model, including feature engineering
  3. Challenges of realtime feature engineering at scale
  4. Functional and non-functional requirements
  5. Architecture of the QuickSilver platform, design principles and tech choices
  6. Integration with Machine learning platform
  7. Best practices and learnings

Presenters

Aditya S ( Tech lead, Quicksilver Near Realtime Platform )
Narayana Pattipati ( Senior Architect, Myntra Data & Machine Learning Platforms)

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hybrid access (members only)

Hosted by

All about data science and machine learning

Supported by

E2E Cloud is India's first AI hyper scaler, a cloud computing platform providing accelerated cloud-based solutions at maximum optimization and lowest pricing