The Fifth Elephant 2019

Gathering of 1000+ practitioners from the data ecosystem

Participate

fStream - Continuous Intelligence @ scale in Flipkart

Submitted by Ayan Ghatak (@ayanghatak21) on Wednesday, 8 May 2019

Section: Full talk Technical level: Intermediate Session type: Discussion

Abstract

We live in an age of ML models, deeply personalised user experiences and quick data driven business decisions. The common denominator enabling all of it is data processing systems, especially real time ones.

We at Flipkart use streaming systems for a variety of real time computations like analytics and reporting in flash sale events, annual Big Billion day sales or personalisation of search and browse experience. These use-cases requires stateful stream processing (like - stream joins and time windowed aggregates) at a very high scale and such systems becomes very complex very fast.

Introducing fStream : A managed Stream Processing Platform
We built fStream to abstract out above complexities and provide a simple declarative interface to define powerful computation graphs (DAG) and execute it without worrying about the underlying setup, infrastructure and scale.

In this presentation we will talk about architecture, interfaces and management layers of fStream which is aimed at simplifying the whole lifecycle of streaming jobs (creation, deployment, monitoring and maintenance).
We will also talk about a few e-commerce domain problems like contextual search, personalisation, analytics and reporting requirements at high scale ‘sale events’ and how we solve them through stateful processing system like fStream.

Outline

Agenda for the talk would be :
- Stream processing use-cases in e-commerce domain - Common patterns and paradigms in stream processing - FStream - Managed stateful stream processing platform - FStream components

Speaker bio

Ayan has been working in Flipkart Data Platform. He holds a masters degree in Computer Science from IIT Kanpur. He is passionate about working on distributed systems, building scalable and robust platforms. He is currently working on fStream platform in FDP.

Comments

  • saurabh hirani (@saurabh-hirani) a month ago

    Hi Ayan,

    Thanks for submitting the proposal.

    I had the following queries:
    1. Please add draft slides + a short preview video of what you aim to cover and what would be the audience takeaways.
    2. Is this talk going to be more ML heavy or more around the tooling built to solve a specific problem? I ask because you might want to see if this talk is a better fit for https://fifthelephant.in/ depending on the approach you want to take.

  • Abhishek Balaji (@booleanbalaji) Reviewer a month ago

    Hi Ayan,

    We’re evaluating this proposal for The Fifth Elephant. For evaluation, please upload updated slides and a preview video. Your slides must cover the following:

    • Problem statement/context, which the audience can relate to and understand. The problem statement has to be a problem (based on this context) that can be generalized for all.
    • What were the tools/frameworks available in the market to solve this problem? How did you evaluate these, and what metrics did you use for the evaluation? Why did you pick the option that you did?
    • Explain how the situation was before the solution you picked/built and how it changed after implementing the solution you picked and built? Show before-after scenario comparisons & metrics.
    • What compromises/trade-offs did you have to make in this process?
    • What is the one takeaway that you want participants to go back with at the end of this talk? What is it that participants should learn/be cautious about when solving similar problems?

    We need your updated slides and preview video by Jun 27, 2019 to evaluate your proposal. If we do not receive an update, we’d be moving your proposal for evaluation under a future event.

Login with Twitter or Google to leave a comment