The Fifth Elephant 2023 Winter

The Fifth Elephant 2023 Winter

On the engineering and business implications of AI & ML

Tickets

Loading…

This video is for members only

Aditya S

@adityasridhar

Narayana Pattipati

@npattipati Author

Near Real time Feature Engineering for Machine Learning use cases at Myntra Scale

Submitted Oct 4, 2023

Problem

Myntra is one of the leading fashion e-commerce companies in India. Myntra delivers best-in-class shopping experience by leveraging many advanced machine learning models, deployed for online or real-time inference. The online inference requires streams of data to be processed, machine learning features to be computed, stored and served in (near) real-time, at Myntra scale.

The features can be hand crafted or generated (e.g. user, product, style, widget, image embeddings). And majority of the features require stateful stream processing with complex computation, in (near) real-time, at very high throughputs (millions of rpm) and low latency. This requires scalable, resilient data engineering systems with stateful stream processing capabilities and feature stores.

Solution

Myntra Data Engineering team designed and built Quicksilver, a real time data ingestion and stateful stream processing platform. It is part of the overall Myntra Data Platform. The Quicksilver platform ingests millions of events every minute, computes the machine learning features in (near) real time and makes them available to machine learning models for online inference.

Outline of the talk

Online ML use cases at Myntra
Life cycle of an online ML model, including feature engineering
Challenges of realtime feature engineering at scale
Functional and non-functional requirements
Architecture of the QuickSilver platform, design principles and tech choices
Integration with Machine learning platform
Best practices and learnings

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hybrid access (members only)

Hosted by

All about data science and machine learning

Supported by

Sponsor

Providing all founders, at any stage, with free resources to build a successful startup.