The Fifth Elephant 2014

How to build a real-time, analytical dashboads that can enable business take decisions at scale? There are various technologies out there that fill one or the other use case - right from horizontally scalable queues such as kafka, stream processing systems such as storm, data stores such as openTSDB and druid that can provide dimensional lookup on large amount of data and visualisation libraries such as d3, cubism to view them. But there is a lot more that is never discussed if not for the details. In a service-oriented architecture, where dimensions and measures are coming from different sources, where some dimension is larger than the number of seconds in a year (for those who are wondering it is ~32 M), ensuring liveness and correctness at every minute is what Fireball is all about.

Outline

Fireball is a stream processing engine at Flipkart. It powers real time analytical dashboards to enable business take time-sensitive decisions, at scale. Fireball can process millions of events (with flexible, json-like schema) per hour that require:

• executing custom process (usually SQL-like) to derive business metrics from the incoming events
• over large number of dimensions (on an average 10 dimensions for each measure)
• with very low latency and ensuring correctness all the time (enabling time-sensitive decision making)

So how do you build such a system? How do you store such a large amount of time-series data to ensure roll-ups, drill-downs on different dimensions? In this talk we’ll go over the transformation of a standard stream processing platform and a CEP library into Fireball.

Speaker bio

I am Architect @ Flipkart and am part of the Data Platform effort.