The Fifth Elephant 2018

The seventh edition of India's best data conference

Scaling write-heavy OLTP systems with strong data guarantees: learning from Flipkart’s user facing order capture systems

Submitted by Gokulvanan V Velan (Customer Platform - VS) on Saturday, 31 March 2018

videocam_off

Technical level

Intermediate

Section

Full talk

Status

Confirmed & Scheduled

View proposal in schedule

Vote on this proposal

Login to vote

Total votes:  +48

Abstract

Order capture and Order management systems at Flipkart have had to scale by 10X volumes to cater to growth in eCommerce and user base.In addition, these systems need to scale for bursty traffic by 1000x for flash sale business model. These systems are write heavy and need strong data guarantees (Consistency, Data-availability, Durability etc). With scale, the data stores for these systems have outgrown capabilities provided out of the box by databases like MySQL and point solutions for each system in the ecosystem have resulted in fragmentation. This talk focuses on our journey in solving for our datastore needs holistically by customising Hbase at the source code level to support Strong Consistency in Write Heavy workloads, Transactional Change Propagation to enable Lamdba Architecture patterns, Basic index support and provide predictable Scalability using Tenant isolation. This talk will dive into details by introducing concept of regionserver groups (rsgroups) within an hbase cluster, tweaks to balancing algorithms in region rebalancing within rsgroup, ensuring no data loss in change propagation and mvcc style approach to support basic indexes over distributed transaction. We currently are live in production with a single multi-tenant hbase cluster that servers half a million QPS in Order capture and Order management flows.

Outline

Challenges faced with existing order capture systems at Scale
a) Context and landscape of the user-facing order capture systems
b) Scaling problems and gaps in the existing technologies

Consolidation of characteristics
a) Key-value store favouring strong consistency and data guarantees
b) Basic secondary index support
c) Transactional change-propagation

Our choice: HBase
a) Good parts of HBase for us
b) Downsides of HBase: maintenance of multiple components, lack-of transactional change-propagation
c) Overview of HBase

Solving for single multi-tenant cluster
a) Logical components of HBase
b) Custom HBase LoadBalancer with tenant & region-server group awareness
c) Using Hadoop’s favoured node API to bring in isolation at hadoop level replica placements
d) Handling Region Splits and Merges

Solving for Transactional change-capture
a) Using ReplicationEndpoint handlers
b) Solve for no data loss, rsgroup specific balancing

How this helped us
a) Helped reached our scale needs
b) Improved cluster manageability
c) Improved efficiency and reliability

Future work and the way forward
a) Uniform data + replica distribution
b) Memstore flush optimization
c) Compaction optimization

Speaker bio

Gokulvanan is an Architect for Order capture and Order management systems at Flipkart. Prior to Flipkart he worked as Senior Software Engineer for the Mobile team at a media advertising startup, Komli Media. He has close to 10yrs of experience working in Software Industry.

Links

Slides

https://docs.google.com/presentation/d/1sYSn6syDuEJQ9vuHlSHA0Nikp1tvPIq1jV68j2eFPCU/edit?usp=sharing

Comments

  • 1
    Zainab Bawa (@zainabbawa) Reviewer 8 months ago

    Gokulvanan, we need a preview video and draft slides to evaluate this talk.

    • 0
      Gokulvanan V Velan (Customer Platform - VS) Proposer 8 months ago

      updated draft slides, will add preview video by end of the week

      • 1
        Zainab Bawa (@zainabbawa) Reviewer 7 months ago

        Awaiting preview video.

        • 1
          Gokulvanan V Velan (Customer Platform - VS) Proposer 7 months ago

          updated links to youtube video of recent talk on this.

  • 1
    Zainab Bawa (@zainabbawa) Reviewer 7 months ago

    The opening slide contains two persons’ names. Our policy is only one speaker per session. Please confirm that you are the speaker for this session.

    • 1
      Gokulvanan V Velan (Customer Platform - VS) Proposer 7 months ago (edited 7 months ago)

      Yes I would be the speaker, to conform to the policy.

Login with Twitter or Google to leave a comment