The Fifth Elephant 2019

The adoption of distributed processing infrastructure heralded a new way of building data processing systems. Shifting to a more generic term, Data Pipelines (over legacy ETL), has helped elevate the architecture of data processing systems from being purely batch oriented to a more hybrid one combining batch, live and real-time elements.
With this shift still active, it is imperative that we raise the bar of engineering quality by distilling and adopting different approaches to commonly encountered data engineering problems. This talk will cover ideas that can be adopted by practitioners to develop simpler, more reliable and efficient data pipelines based on Hive, Spark, Flink, Airflow and other open source data engineering technologies.

Outline

Some of the idea/solitions that will be presented -

• Incremental Processing in ETL Pipelines based on a Functional approach
• Aggregate-as-you-join strategies for handling wide middles in data pipelines
• Designing for Backfills and Data Corrections
• Restream/Replay Queues for Managing Data Loss
• Dealing with Stream-Table Duality

Speaker bio

Srihari is a Solutions Architect at Cloudera. Prior to this he was the Technical Principal and a founding member of the Data Analytics practice at ThoughtWorks. He’s played several roles as developer, architect, Head of Technology at ThoughtWorks. He was also part of the Technology Advisory Board and CTO’s office at ThoughtWorks. He is quite passionate about distributed systems, databases & ML/AI Ops and blogs about them on www.systemswemake.com.