The Fifth Elephant 2017

On data engineering and application of ML in diverse domains

Lessons learned from building a globally distributed database service from the ground up

Submitted by Dharma Shukla (@dharmashukla) on Friday, 26 May 2017

videocam
Preview video

Technical level

Intermediate

Section

Full talk for data engineering track

Status

Confirmed & Scheduled

View proposal in schedule

Vote on this proposal

Login to vote

Total votes:  +25

Abstract

Description:
Dharma and his team has spent past 7 years to build Azure Cosmos DB (http://cosmosdb.com) - a massively scalable, multi-tenant, globally distributed database service from the ground up. The system they have built is currently operating across more than thirty-four geographical regions, managing hundreds of petabytes of indexed data, and serving 100s of trillions of requests every day from thousands of customers worldwide. The database system allows developers to elastically scale both, throughput and storage across any number of geographical regions on a single table. The service offers guaranteed single-digit millisecond low latency at 99the percentile, 99.99% high availability, predictable throughput, and multiple well-defined consistency models. The system is able to offer comprehensive SLAs for latency, availability, throughput and consistency and is used extensively within Microsoft and is available to external Azure customers since 2015. In this session, Dharma will describe the internals of the system design and various design trade-offs they had to make. He will also share his experiences from operating a globally distributed database service worldwide and maintaining comprehensive Service Level Agreements (SLAs).

Takeaways:
The lessons I have learnt from building a globally distributed database can be applied to many distributed systems.

Some of the takeaways are:
1. Well-defined, relaxed consistency models are really powerful in solving real world scenarios
2. A system designed for cloud can be made to run really cheap if it is designed with resource governance in mind
4. What does it mean to build multi-tenant applications? What are the challenges?
5. Applications running on cloud deserve a globally distributed database.
6. A globally distributed database != database with DR
and many more..

Intended audience:
Application developers of all types, distributed systems practitioners, data engineers, system integrators and consultants.

Outline

  1. What does it mean to build a database that leverages the strengths of cloud?
  2. Horizontal partitioning
  3. Elastically scaling throughput (vs. storage) worldwide
  4. Resource governance and fine grained multi-tenancy
  5. Global distribution of data for low latency
  6. Global distribution of data for high availability
  7. Navigating the speed of light
  8. Navigating the CAP theorem
  9. Consistency Models - finding the right shade of grey!
  10. Why hosting on-premises databases (SQL or NoSQL) cannot offer the lowest TCO and best SLAs?
  11. What does it take to offer and maintain comprehensive SLAs for consistency, latency and throughput and availability.
  12. Operating a globally distributed database service, worldwide
  13. Insights from the production workloads
  14. Conclusions

https://speakerdeck.com/dharmashukla/cosmos-db-at-fifth-elephant-2017

Requirements

Familiarity with databases, cloud and challenges to build a scalable applications.

Speaker bio

Dharma Shukla is a Distinguished Engineer at Microsoft. Dharma is also the founder of Azure Cosmos DB (http://cosmosdb.com) - a globally distributed, multi-tenant database service on Azure. Prior to working on the current system, his work spanned a range of distributed systems and databases at Microsoft and other places.

Links

Slides

https://speakerdeck.com/dharmashukla/cosmos-db-at-fifth-elephant-2017

Preview video

https://www.youtube.com/watch?v=L_PPKyAsR3w&t=10s

Comments

  • 4
    madhan gajendran (@madhang) a year ago

    This is gonna be great! Eagerly looking forward for this talk to hear about what it takes to build a fantastic cloud based distributed database from ground up.

  • 3
    Shireesh Thota (@shireesh) a year ago

    Here is what I can guarantee - everyone will walk out with an enlightened understanding of the nuances of building a highly available distributed database for the modern cloud application! This is a talk that you don’t want to miss.

  • 2
    Ashwanth Kumar (@ashwanthkumar) a year ago

    I’m really looking forward to this talk!

  • 2
    Jayanta Mondal (@jonts) a year ago

    Building a distributed database that can scale seamlessly and can maintain comprehensive SLAs is a great feat to achieve! I am looking forward to this.

  • 2
    Kinshuman Patra (@kinshuman) a year ago

    This looks like a fantastic talk! Looking forward to it!

  • 2
    Aditya Kulkarni (@ak22) a year ago

    Really looking forward to this talk! Building large scale distributed systems with guaranteed SLAs and geo distribution is no easy task and we will definitely learn some great lessons from it.

  • 1
    Krishnan Sundaram (@krishnansundaram) a year ago (edited a year ago)

    This is absolutely going to be a very informative talk on how to build a fully managed, elastic, multi-model, geo-distributed database, while still maintaining stringent SLAs, from the ground up!

  • 1
    rimma nehme (@rimmanehme) a year ago

    Agree. This is the talk you don’t want to miss!!

Login with Twitter or Google to leave a comment