Messaging architecture at Facebook
Submitted by qubole (@qubole) on Sunday, 10 June 2012
Section: Industry & Business Technical level: Advanced Session type: Lecture
The audience will learn design options for building large scale messaging system and how technical, philosophical and organizational considerations surrounding these played out at Facebook.
Building a scalable messaging backend for a Billion user is Hard. This talk covers the initial design days when the Facebook messaging architecture was being decided.
Should back-end stores be eventually or strongly consistent? How much availability is highly available? Should we build services or components? How big does a piece of software have to be before it stops being a component. How can a back-end be built for an application that's yet to be designed? Is it possible to simulate a billion user workload - and does it help?
The issues faced by the design team ranged from the technical to the philosophical - to ultimately the practical. This talk will explore some of these areas.
Given the limited time - it would not be possible to introduce background material. A grasp of scalable key value stores, CAP theorem and eventual consistency, reliable storage systems would be useful in getting more out of the talk.
Joydeep is a co-founder at Qubole and heads their India development team. Prior to starting Qubole - Joydeep worked at Facebook where he boot-strapped the data processing ecosystem based on Hadoop, started the Apache Hive project and led the Data Infrastructure team. Joydeep was a key contributor on the Facebook Messages architecture team that brought Apache HBase to Facebook and to the transactional and reporting backends for Facebook Credits. He has been a driver for other important sub-projects in the Hadoop ecosystem - like the FairScheduler and RCFile. Joydeep studied Computer Science at IIT-Delhi and University of Pittsburgh and started his career working on Oracle's database kernel and building highly available and scalable file systems at Netapp. In between - he has played founding roles in storage and advertising startups. He cut his teeth building data driven applications as the lead engineer on Yahoo's in-house Recommendation Platform.
Joydeep holds numerous patents, has many published papers and has been both speaker and panelist at Hadoop summits and at other Silicon Valley conferences.