Serving user intent : Facebook style notifications using HBase and Event streams

This submission has been added to the schedule

Serving user intent : Facebook style notifications using HBase and Event streams

Submitted Jan 31, 2014

Section: Full talk Technical level: Intermediate

This talk is about building a low-latency, near real-time Notifications platform for serving user intent using Event based architecture, Complex Event Processing and a data store like HBase. Will also cover how millisecond response times are achieved when accessing data from 100 million rows by interpreting change from immutable events and organizing data as LSM trees.

Outline

Relevant and Personalized notifications in near real-time is a great way of serving user intent. The intent may vary - say liking a Facebook update as compared to a price drop for a browsed product on an e-commerce website. The system characteristics and solution patterns in both these instances may be very similar though.

This talk will cover the design of the Flipkart Notifications platform. The techniques and technologies used to serve product related intent can be easily applied to a different domain. This talk will also introduce projects that were Open Sourced while building the platform.

Architecture, Design patterns and technologies used in this system include:

Pre-creating data that matches user intent - so as to significantly reduce data serving latencies
Storing immutable events and interpreting change
Event driven architectures(EDA) and its variant Staged EDA (SEDA) using technologies like RabbitMQ and Mule.
Complex Event Processing (CEP) using technologies like Esper
Data stores like HBase that organize data between memory and disk as Log Structured Merged (LSM) trees - leveraging Disk transfer better over Disk seek
A data serving API that is resilient to failures and latencies - using Hystrix and Netty

The talk uses a typical e-commerce experience where user intent is either implicit or interpreted from actions - for example a user browsing a product of interest, adding an item to a shopping cart or adding it for future reference via a wish-list. In a dynamic e-commerce marketplace, product data (such as price, stock quantity) is constantly changing across millions of listed products even as user intent is being expressed on the website. User intent may be seen as one Event stream while Product attribute changes is another. An intersection of these two streams is the Notification data. An efficient data store that can store and serve tens of millions of such notifications with very low latencies is the Notification service.

The following projects were open sourced before or when building the Notifications Platform :

The talk will also feature a live view of the data serving metrics with millisecond response times.

Requirements

Just technical curiosity about how those notifications on Facebook or Flipkart are delivered at scale. An appreciation of data stores (SQL and NoSQL) and their characteristics will also help. A big plus if you have spent time trying to solve similar problems.

Speaker bio

Architect and Open source committer. My areas of interest are Distributed Systems, Big Data, Text Mining and Data Stores.

My experience as Architect includes:

Building the World’s largest biometric identity platform in Aadhaar
Customer facing Mobile and Web platforms at India’s leading e-Commerce company - Flipkart

Most of my work in recent years has been around OSS - using it to build large scale systems and in contributing projects back to the community. Some of my OSS work is downloaded and used worldwide:

Active projects on github : https://github.com/regunathb

The Fifth Elephant 2014