The Fifth Elephant 2019

Gathering of 1000+ practitioners from the data ecosystem

Tickets

Airflow for the Enterprise (Nike's Journey)

Submitted by ChandraSekhar Kandavilli (@chandrakandavilli) on Friday, 14 June 2019


Preview video

Session type: Short talk of 20 mins

Abstract

Nike has a wide variety of systems in the enterprise landscape. All these systems produce data in different shapes and sizes. We are building the Nike data foundation so that we meet the below goals.

  1. Deliver trusted, accurate, timely and consistent information and insights to the business.
  2. Enable governing data as an asset and sharing data at scale.
  3. Enable faster and better informed business decisions resulting in serving our
  4. customers and consumers more efficiently.

One key aspect in the enterprise data foundation is having consolidated data which helps in Decision Support. It is very important that we collect all the transactional data and run transformation heavy and consolidation jobs which provide the data for enabling the reporting dash boards. The Nike’s data landscape is extremely complex with diverse systems, integration points and massive volumes. We need a highly reliable and scalable platform to orchestrate all of these data pipelines.

This talk is about how Nike utilizes the open source technology Apache Airflow to run complex ETL/ELT/Decision Support/Advance Analytics jobs which executes code base developed using various technologies (Spark, Hive, Python, Shell, AWS commands) and handle complex integration with other collaboration tools like E-mail / Slack / Box etc.

Outline

• What makes Airflow Architecture unique?
• How to leverage Custom Operators for repeat operations?
• How to effectively use Netflix Genie with cloud services like AWS.
• How to simplify CI / CD for Airflow using Jenkins and Code Repos like Git/Bitbucket

Speaker bio

Chandra is currently a data engineering manager part of Nike, Enterprise Data & Analytics Team, leading teams in the space of Enterprise Data Foundation and Retail Data Products.

Links

Preview video

https://youtu.be/QY7W3rOG2Hw

Comments

  • Abhishek Balaji (@booleanbalaji) Reviewer 5 months ago

    Hi Chandra,

    Thank you for submitting a proposal. We need to see detailed slides and a preview video to evaluate your proposal. Your slides must cover the following:

    • Problem statement/context, which the audience can relate to and understand. The problem statement has to be a problem (based on this context) that can be generalized for all.
    • What were the tools/frameworks available in the market to solve this problem? How did you evaluate these, and what metrics did you use for the evaluation? Why did you pick the option that you did?
    • Explain how the situation was before the solution you picked/built and how it changed after implementing the solution you picked and built? Show before-after scenario comparisons & metrics.
    • What compromises/trade-offs did you have to make in this process?
    • What is the one takeaway that you want participants to go back with at the end of this talk? What is it that participants should learn/be cautious about when solving similar problems?

    We need your updated slides and preview video by Jun 27, 2019 to evaluate your proposal. If we do not receive an update, we’d be moving your proposal for evaluation under a future event.

    • Abhishek Balaji (@booleanbalaji) Reviewer 4 months ago

      Hi Chandra, since you havent responded with the slides and revised content, we’re moving this proposal for evaluation under a future event. If you choose to update your content, we’ll reconsider it for evaluation.

Login with Twitter or Google to leave a comment