The Fifth Elephant 2012

The audience will learn design options for building large scale messaging system and how technical, philosophical and organizational considerations surrounding these played out at Facebook.

Outline

Building a scalable messaging backend for a Billion user is Hard. This talk covers the initial design days when the Facebook messaging architecture was being decided.

Should back-end stores be eventually or strongly consistent? How much availability is highly available? Should we build services or components? How big does a piece of software have to be before it stops being a component. How can a back-end be built for an application that’s yet to be designed? Is it possible to simulate a billion user workload - and does it help?

The issues faced by the design team ranged from the technical to the philosophical - to ultimately the practical. This talk will explore some of these areas.

Requirements

Given the limited time - it would not be possible to introduce background material. A grasp of scalable key value stores, CAP theorem and eventual consistency, reliable storage systems would be useful in getting more out of the talk.

Speaker bio

Joydeep is a co-founder at Qubole and heads their India development team. Prior to starting Qubole - Joydeep worked at Facebook where he boot-strapped the data processing ecosystem based on Hadoop, started the Apache Hive project and led the Data Infrastructure team. Joydeep was a key contributor on the Facebook Messages architecture team that brought Apache HBase to Facebook and to the transactional and reporting backends for Facebook Credits. He has been a driver for other important sub-projects in the Hadoop ecosystem - like the FairScheduler and RCFile. Joydeep studied Computer Science at IIT-Delhi and University of Pittsburgh and started his career working on Oracle’s database kernel and building highly available and scalable file systems at Netapp. In between - he has played founding roles in storage and advertising startups. He cut his teeth building data driven applications as the lead engineer on Yahoo’s in-house Recommendation Platform.

Joydeep holds numerous patents, has many published papers and has been both speaker and panelist at Hadoop summits and at other Silicon Valley conferences.