Rootconf Hyderabad edition

On SRE, systems engineering and distributed systems

CI/CD in Data Engineering @ Nike Tech

Submitted by Ramanaresh Challa (@ramanareshchalla) on Jun 14, 2019

Session type: Short talk of 20 mins Status: Rejected

Abstract

At Nike Tech, we have a diverse technology landscape with teams working on building apps for the end consumer to dashboards for our internal business stakeholders. All this is done at massive scale and a strong focus on CI/CD.
CI/CD in data engineering is a often neglected area and weak practices leads to inefficiencies which hurt in the long run. With a focus on delivering highly trusted and consistent data to business in order to make decisions, the development and testing process needs to be watertight before things move to production.
CI/CD is also usually an afterthought when we are designing and building the data engineering applications. We here at Nike do it differently and ensure CI/CD is a key focus area from the very beginning and design accordingly. This talk throws light into our thinking process when it comes to CI/CD, the tools we use and design examples for a couple of use cases.

Outline

•What is CI/CD?
•Why CI/CD in data engineering?
•How to simplify CI / CD for Airflow using Jenkins and Code Repos like Git/Bitbucket
•How to simplify CI / CD for Nifi using Jenkins/Nifi Registry/Terraform

Speaker bio

Ram is a lead engineer at Nike Enterprise data and analytics organization working in the space of real time master data management, data ingestion and data science engineering.

Links

Slides

https://drive.google.com/file/d/1dI2fsehxJvI3s4IGGnv84ShNrAnCN9KJ/view?usp=sharing

Preview video

https://www.youtube.com/watch?v=7ijx8acjGJU

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('You need to be a participant to comment.') }}

{{ formTitle }}
{{ gettext('Post a comment...') }}
{{ gettext('New comment') }}

{{ errorMsg }}