Rootconf Hyderabad edition

On SRE, systems engineering and distributed systems

Tickets

CI/CD in Data Engineering @ Nike Tech

Submitted by Ramanaresh Challa (@ramanareshchalla) on Friday, 14 June 2019

Session type: Short talk of 20 mins Status: Rejected

Abstract

At Nike Tech, we have a diverse technology landscape with teams working on building apps for the end consumer to dashboards for our internal business stakeholders. All this is done at massive scale and a strong focus on CI/CD.
CI/CD in data engineering is a often neglected area and weak practices leads to inefficiencies which hurt in the long run. With a focus on delivering highly trusted and consistent data to business in order to make decisions, the development and testing process needs to be watertight before things move to production.
CI/CD is also usually an afterthought when we are designing and building the data engineering applications. We here at Nike do it differently and ensure CI/CD is a key focus area from the very beginning and design accordingly. This talk throws light into our thinking process when it comes to CI/CD, the tools we use and design examples for a couple of use cases.

Outline

•What is CI/CD?
•Why CI/CD in data engineering?
•How to simplify CI / CD for Airflow using Jenkins and Code Repos like Git/Bitbucket
•How to simplify CI / CD for Nifi using Jenkins/Nifi Registry/Terraform

Speaker bio

Ram is a lead engineer at Nike Enterprise data and analytics organization working in the space of real time master data management, data ingestion and data science engineering.

Links

Slides

https://drive.google.com/file/d/1dI2fsehxJvI3s4IGGnv84ShNrAnCN9KJ/view?usp=sharing

Preview video

https://youtu.be/7ijx8acjGJU

Comments

  •   Abhishek Balaji (@booleanbalaji) Reviewer 9 months ago

    Hi Ram,

    Thank you for submitting a proposal. This looks very interesting! We need to see detailed slides and a preview video to evaluate your proposal. Your slides must cover the following:

    • Problem statement/context, which the audience can relate to and understand. The problem statement has to be a problem (based on this context) that can be generalized for all.
    • What were the tools/frameworks available in the market to solve this problem? How did you evaluate these, and what metrics did you use for the evaluation? Why did you pick the option that you did?
    • Explain how the situation was before the solution you picked/built and how it changed after implementing the solution you picked and built? Show before-after scenario comparisons & metrics.
    • What compromises/trade-offs did you have to make in this process?
    • What is the one takeaway that you want participants to go back with at the end of this talk? What is it that participants should learn/be cautious about when solving similar problems?

    We need your updated slides and preview video by Jun 27, 2019 to evaluate your proposal. If we do not receive an update, we’d be moving your proposal for evaluation under a future event.

    •   Ramanaresh Challa (@ramanareshchalla) Proposer 9 months ago

      Hi Balaji, Thanks for the response. Sure, I will upload the details soon.

      •   Abhishek Balaji (@booleanbalaji) Reviewer 9 months ago

        Hi Ramanaresh, I had a look at the proposal and it would be very relevant to an upcoming edtion of Rootconf - https://hasgeek.com/rootconf in Pune this year. I’m moving this proposal for evaluation under the Pune edition scheduled on 21 September. One of my colleagues working on Rootconf will get back to you on the next steps.

  •   Anwesha Sarkar (@anweshaalt) Reviewer 8 months ago

    Hello Ramanaresh,

    Thank you for your submission. Your proposal is now being considered for Rootconf Pune 2019. Here are the feedback of your proporsal:

    • Include an introduction slide, introducing yourslef.
    • Are you sure that you want to have a seperate slide for Rootconf audience describing CI and CD?
    • Avoid having text heavy slides. (Like: 3rd and 4th slide)
    • Explain situation/circumstances in which you have faced in the problem statement.\
    • Why were you facing such problems that you have mentioned?
    • Explain the problem statement in a lucid manner.
    • How your life has improved by observing CI and CD?
    • The slides seems to have finish abruptly.
    • Include a slide where you are going to tell the situation after.
    • Include your contact credential in the “Thank you” slide.

    You need to subnmit your updated proposal by 5th August latest. Look forward to your answer.

    Cheers
    Anwesha

Login with Twitter or Google to leave a comment