The Fifth Elephant 2019

We live in an age of ML models, deeply personalised user experiences and quick data driven business decisions. The common denominator enabling all of it is data processing systems, especially real time ones.

We at Flipkart use streaming systems for a variety of real time computations like analytics and reporting in flash sale events, annual Big Billion day sales or personalisation of search and browse experience. These use-cases requires stateful stream processing (like - stream joins and time windowed aggregates) at a very high scale and such systems becomes very complex very fast.

Introducing fStream : A managed Stream Processing Platform
We built fStream to abstract out above complexities and provide a simple declarative interface to define powerful computation graphs (DAG) and execute it without worrying about the underlying setup, infrastructure and scale.

In this presentation we will talk about architecture, interfaces and management layers of fStream which is aimed at simplifying the whole lifecycle of streaming jobs (creation, deployment, monitoring and maintenance).
We will also talk about a few e-commerce domain problems like contextual search, personalisation, analytics and reporting requirements at high scale ‘sale events’ and how we solve them through stateful processing system like fStream.

Outline

Agenda for the talk would be :

• Stream processing use-cases in e-commerce domain - Common patterns and paradigms in stream processing - FStream - Managed stateful stream processing platform - FStream components

Speaker bio

Ayan has been working in Flipkart Data Platform. He holds a masters degree in Computer Science from IIT Kanpur. He is passionate about working on distributed systems, building scalable and robust platforms. He is currently working on fStream platform in FDP.