The Fifth Elephant 2012

This talk is not about usage of existing tools or frameworks. This is about building a platform for Real Time Analytics, Real time visualization and Real time Complex Event Processing triggered by Events, Event Patterns and Event Aggregates.

Outline

Traditionally people use databases or data-warehouses or even Hadoop and Map Reduce to perform batch analytics. However real time analytics, real time visualization and complex event processing require drastically different architecture, storage and paradigms.

In this talk we will share a model for accomplishing the above using:

Elastic Search: To push large data sets into sharded lucene index. This index can be in memory, partially in memory and partially on disk etc. This is key to real time data.

statsd: Aggregator (provides aggregates on the basic events). This is key to continuous ETL and continuous analytics.

Reverse Pump: The reverse pump, pushes back aggregate information into Elastic Search

Pattern Recognition: This is similar to looking for regular expressions in Perl. We look for patterns in the event stream and match patterns. Note the sliding time window over which is applies is finite and limited by memory as are the number of pattern matchers. This system uses a reverse index and RETE techniques.

Graphite: Basic Visualization (includes simple realtime visualization)

Notification: These will be triggered on matches of static rules or complex pattern match on base event/aggregate events or patterns matched

Extras:
MySQL binlog parser to observe and analyze events (BinLogParser)
Depict the mood of Mysql with music. Make it sing!

Speaker bio

Ramesh Perumalsamy & Vishnu Rao are part of the Supply Chain Technologies(SCT) Platform team at Flipkart. We build SOA systems and event processing and triggers notifications depending on patterns for India’s largest ECommerce entity Flipkart.

The day job involves prototyping to delivering new solutions and technologies to provide productivity multipliers to Flipkart Supply Chain products.

Ramesh loves solving problems, the command line, experimenting with Open source solutions and much more. And secretly saves the world from Super Villains (Like some of his colleagues).

Vishnu is our In-house Mysql Medical Officer. He runs a flipkart internal blog called - MAS*H (Mysql Army surgical hospital) where Mysql queries and Mysql instances alike are treated with love and care :) .A lego freak. He doodles @ http://doodle-vishnu.blogspot.com/