The Fifth Elephant 2019

Gathering of 1000+ practitioners from the data ecosystem

Crafting Better Data Pipelines - Some Ideas

Submitted by Srihari Srinivasan (@srihari) on Jun 17, 2019

Session type: Full talk of 40 mins Status: Rejected


The adoption of distributed processing infrastructure heralded a new way of building data processing systems. Shifting to a more generic term, Data Pipelines (over legacy ETL), has helped elevate the architecture of data processing systems from being purely batch oriented to a more hybrid one combining batch, live and real-time elements.
With this shift still active, it is imperative that we raise the bar of engineering quality by distilling and adopting different approaches to commonly encountered data engineering problems. This talk will cover ideas that can be adopted by practitioners to develop simpler, more reliable and efficient data pipelines based on Hive, Spark, Flink, Airflow and other open source data engineering technologies.


Some of the idea/solitions that will be presented -
+ Incremental Processing in ETL Pipelines based on a Functional approach
+ Aggregate-as-you-join strategies for handling wide middles in data pipelines
+ Designing for Backfills and Data Corrections
+ Restream/Replay Queues for Managing Data Loss
+ Dealing with Stream-Table Duality

Speaker bio

Srihari is a Solutions Architect at Cloudera. Prior to this he was the Technical Principal and a founding member of the Data Analytics practice at ThoughtWorks. He’s played several roles as developer, architect, Head of Technology at ThoughtWorks. He was also part of the Technology Advisory Board and CTO’s office at ThoughtWorks. He is quite passionate about distributed systems, databases & ML/AI Ops and blogs about them on


  • Abhishek Balaji (@booleanbalaji) a year ago

    Hi Srihari,

    Thank you for submitting a proposal. We need to see detailed slides and a preview video to evaluate your proposal. Your slides must cover the following:

    • Problem statement/context, which the audience can relate to and understand. The problem statement has to be a problem (based on this context) that can be generalized for all.
    • What were the tools/frameworks available in the market to solve this problem? How did you evaluate these, and what metrics did you use for the evaluation? Why did you pick the option that you did?
    • Explain how the situation was before the solution you picked/built and how it changed after implementing the solution you picked and built? Show before-after scenario comparisons & metrics.
    • What compromises/trade-offs did you have to make in this process?
    • What is the one takeaway that you want participants to go back with at the end of this talk? What is it that participants should learn/be cautious about when solving similar problems?

    We need your updated slides and preview video by Jun 27, 2019 to evaluate your proposal for The Fifth Elephant 2019. If we do not receive an update, we’d be moving your proposal for evaluation under a future event.

Login to leave a comment