Choosing datastores

Abstract
“In its early days, the LinkedIn data ecosystem was quite simple. A single RDBMS contained a handful of tables for user data such as profiles, connections, etc. This RDBMS was augmented with two specialized systems: one provided full text search of the corpus of user profile data, the other provided efficient traversal of the relationship graph. These latter two systems were kept up-to-date by Databus, a change capture stream that propagates writes to the RDBMS
primary data store, in commit order, to the search and graph clusters. Over the years, as LinkedIn evolved, so did its data needs”.

The above is an excerpt from Linkedin’s Espresso paper in 2013. At that time Linkedin had 200 million users worldwide. With a growth phase that followed, the user base today is ~4x that number, add to it the ever increasing user engagement and new feature rollouts. During this growth phase, LinkedIn data systems evolved for each of our use case. In this talk, we will attempt to give a glimpse of our Online Storage ecosystem and its evolution.

Online datasystem like Oracle and MySQL evolved from single datacenter to multi datacenter.
In addition to the above Relational systems, Online storage fleet today houses :

• Custom NoSQL cluster(s)

  • Espresso is Linkedin’s nosql cluster
  • It ‘s sharded and supports secondary index(s)
  • It serves queries in O(M) queries per second.

• Derived Data Store(s)

  • It might be prudent to precompute and transform data from one form to another so that other systems can directly read the transformed data
  • Serving the transformed data for low latency use case

• BLOB Storage

  • Distributed file storage like Azure blob and AWS S3
  • Data is immutable
  • Supports replication for consistency and cross colo reads for Read after Write consistency

• Couchbase

  • Defacto caching solution for our Source of Truth databases

• OLAP system

  • Supports analytics on realtime and offline data stored as segments
  • Segments are time partitioned data

Glue for Data Systems

• Cluster Manager/State Machine

  • Quorum makes sure the monitoring and state map is consistent
  • Helix initiates State Transition to converge to ideal state and also does the job of partition allocator and job scheduler.

• Provisioner

  • Based on user requirements, provisioner allocates resources in a cluster running the required stack for provisioning.
  • Each services have to track cost to serve to upper bound the resource utilization in multi tenant infrastructure

All these components form the online storage stack for Linkedin. Each one has a unique use case and we strongly believe that “one size fits all” isn’t true in the data realm!.