The Fifth Elephant 2012

Finding the elephant in the data.

Messaging architecture at Facebook

Submitted by qubole (@qubole) on Sunday, 10 June 2012

videocam_off

Technical level

Advanced

Section

Industry & Business

Session type

Lecture

Status

Confirmed

Vote on this proposal

Login to vote

Total votes:  +21

Objective

The audience will learn design options for building large scale messaging system and how technical, philosophical and organizational considerations surrounding these played out at Facebook.

Description

Building a scalable messaging backend for a Billion user is Hard. This talk covers the initial design days when the Facebook messaging architecture was being decided.

Should back-end stores be eventually or strongly consistent? How much availability is highly available? Should we build services or components? How big does a piece of software have to be before it stops being a component. How can a back-end be built for an application that's yet to be designed? Is it possible to simulate a billion user workload - and does it help?

The issues faced by the design team ranged from the technical to the philosophical - to ultimately the practical. This talk will explore some of these areas.

Requirements

Given the limited time - it would not be possible to introduce background material. A grasp of scalable key value stores, CAP theorem and eventual consistency, reliable storage systems would be useful in getting more out of the talk.

Speaker bio

Joydeep is a co-founder at Qubole and heads their India development team. Prior to starting Qubole - Joydeep worked at Facebook where he boot-strapped the data processing ecosystem based on Hadoop, started the Apache Hive project and led the Data Infrastructure team. Joydeep was a key contributor on the Facebook Messages architecture team that brought Apache HBase to Facebook and to the transactional and reporting backends for Facebook Credits. He has been a driver for other important sub-projects in the Hadoop ecosystem - like the FairScheduler and RCFile. Joydeep studied Computer Science at IIT-Delhi and University of Pittsburgh and started his career working on Oracle's database kernel and building highly available and scalable file systems at Netapp. In between - he has played founding roles in storage and advertising startups. He cut his teeth building data driven applications as the lead engineer on Yahoo's in-house Recommendation Platform.

Joydeep holds numerous patents, has many published papers and has been both speaker and panelist at Hadoop summits and at other Silicon Valley conferences.

Comments

  • 1
    Prasanna Hebbar (@prasannahebbar) 6 years ago

    Joydeep, Could you please share the material? Thank you, Prasanna

Login with Twitter or Google to leave a comment