The Fifth Elephant 2025 Annual Conference CfP

The Fifth Elephant 2025 Annual Conference CfP

Speak at The Fifth Elephant 2025 Annual Conference

Vikas Tomar

@raremajor

Scaling Up With Airflow: From 5K to 180K

Submitted May 20, 2025

Migrating from an in-house MWS system to the Apache Airflow platform was a pivotal step in upscaling the SJIT flow at Myntra. This transition not only enhanced the system’s scalability but also significantly reduced the time required for RP (Request for Proposal) creation. For instance, the migration enabled a remarkable performance improvement—from 45+ minutes to just 4–8 minutes for RP creation involving 10,000 entries. This optimization is influenced by the interplay between the number of vendors and SKU entries, showcasing the adaptability of the new system.

The project also introduced support for Multivendor SJIT, further expanding operational capabilities. Leading this migration, I collaborated with cross-functional teams, ensuring minimal disruption to business operations while delivering results. The feature’s development involved working during weekends and off-hours, reflecting a strong commitment to meeting critical timelines.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Jump starting better data engineering and AI futures