Rootconf 2019

Batch processing and Stream processing are two different patterns of solving problems in analytics space. Analytics has transitioned from traditional batch oriented to real time stream oriented with Sub second Latency SLA. Smart end points and dump pipes are the current way of architecting data oriented solutions. This talk is a case stufy of why we(Metapack) transitioned from Traditional Batch oriented ETL to Event streaming architecture. It starts with the problem statement of what we were trying to solve and how streaming architecture helped us to solve that problem. The talk discusses the pros and cons of event streaming and stream processing and what approaches we took to solve those problems. It ends with a positive note of how streaming architecture helped us to create data lake and make the data available further for machine learning, adhoc Reproting and other applications.
Key take aways
1. Event streaming and stream processing concepts
2. When is streaming architecture relevant.
3. Pros and Cons of eventual consistency in Streaming architecture
4. Approaches to solve eventual consistency in streaming scenarios.
Any software engineers, architects and data engineers whishing to hear more about streaming architecture are good to attend the talk.

Outline

The slides would be divived into the following topics
First part - Concepts of Event streaming and streaming architecture in brief.
Second Part - How and why we migrated from Batch oriented ETL’s to Event streaming architure, and how we did stream processing.
Third part - Microservices architecture we built to do stream processing.
Fourth part: Lessons learnt while developing and supporting the streaming solutions in Live, approaches we took to solve the issues faced.
Fifth part: Discussing popular alternatives to Native stream processing/micro batching
Questions?.

Speaker bio

I am a Senior software engineer working at Metapack London with over 14+ years of experience building complex software systems from low end mobile devices to high end application servers.I am part of the team which built this transition architecture from Batch oriented ETL’s to Stream oriented architecture and running it in Live for big customers in e-commerce space.I hold a Bachelor’s degeree in Electronics and Communication from NIT Kurukshetra and Masters in Advanced Computing from University of London.

I have previously given a talk at ODSC india 2018 reading fake news classification with CNN.
https://www.youtube.com/watch?v=soPQupdc-uE&list=PL9Z-JgiTsOYTtwaIoUKbUzWN4uSElmRX5&index=15&t=0s