Rootconf Hyderabad edition

On SRE, systems engineering and distributed systems

Tickets

Fast object distribution using P2P

Submitted by Ankur Jain (@ankurj11) on Friday, 10 May 2019


Preview video

Section: Full talk Technical level: Beginner Session type: Discussion Section: Full talk (40 mins) Category: Distributed systems

View proposal in schedule

Abstract

In Flipkart, objects like VM/container images, media, ML models, etc are stored in a homegrown object storage service. To improve download latencies and avoid getting rate limited, teams normally introduce nginx or some other cache to access frequently used objects. This seemingly works for small objects, less dynamic content or when only few clients are downloading concurrently. But for large objects which are required to be downloaded concurrently by hundreds of clients, network bandwidth of cache cluster gets choked, thereby increasing download latency. Cache misses further worsen the total download duration.

Shatabdi solves the above problem by transparently converting any given http resource to a torrent and downloading it on all participating clients in peer-to-peer(p2p) fashion. It can also be extended to support non-http sources of truth as well. Shatabdi daemon abstracts out all the noise and provides a REST API hosted locally to manage downloads/uploads.

Presently, Shatabdi has gone live in Flipkart datacenters and used for distributing VM and container images to motherships. We foresee this platform to be used for a myriad of other scenarios, such as:

  • Warming up nodes in caching cluster on startup

  • Distributing pre-computed datasets to multiple instances of a datastore

  • Distributing ML models that are required by multiple nodes

Some numbers for the interested, these involve no cache apart from a single download from a http source of truth(object storage system in Flipkart):

  • Disk-based download
    512 clients downloading 3.2GB concurrently in ~(2-4 mins) with upload speed capped to 24 mbps

  • TmpFS-based download
    50 clients downloading 3.2GB concurrently in ~15-25seconds with warmed up seeder

This talk will focus on the motivation behind building Shatabdi and a technical deep dive in its implementation. Broadly, we will cover performance tuning of open-source torrent clients, functional flows to ensure maximal bandwidth consumption and always healthy magnets and benchmarking large deployments container based clusters in Flipkart.

For any usecase involving object distribution at scale for large object size and a lot of concurrent clients, this talk should give valuable insights around architecting solutions using p2p.

Outline

  • Motivation for fast object distribution
  • Goals for a torrent-based distribution layer
  • Exploring open-source alternatives
  • Design, architecture and functional flows of Shatabdi
    • Indexing store and leader election using etcd
    • Achieving HA for torrent tracker
  • Performance tuning of libtorrent
  • Benchmarking container deployments powered by Shatabdi
  • Adoption @ Flipkart

Speaker bio

Ankur works in Platform-as-a-service team in Flipkart Cloud Platform where he builds dev-centric platforms that work at Flipkart scale.

Links

Slides

https://docs.google.com/presentation/d/144puJStK8ZVvd43XRRP5Jxi9TBGxzdg0xx1CTmmmGyw/edit?usp=sharing

Preview video

https://youtu.be/Xe27VIjfv58

Comments

  • saurabh hirani (@saurabh-hirani) 6 months ago

    Hi Ankur and Yashpal,

    First of all - clever name :)

    Thank you for submitting the proposal. This seems like an interesting approach. I had the following queries:

    1. The distribution overview image has D42 - is that https://www.device42.com/ or something else?
    2. Can you add slides + preview video explaining the flow. An approach that really works well in such a proposal would be -
      2.1 State of the world before Shatabdi (not the train), the problems, some stats
      2.2 What were the design goals for Shatabdi
      2.3 What problems did it solve + a before/after comparison
    3. Is Shatabdi open source? Do you plan to open source it? If it isn’t and there are no plans to do so - the audience would benefit more from the architectural aspects of how it solved a specific problem it because they won’t have the tool to play with after the talk. If it is - then please link the relevant github repos and call it out as it will help your talk gain more traction.
    • Ankur Jain (@ankurj11) Proposer 6 months ago

      d42 is an internal s3-like object storage service.
      Shatabdi is not open source yet. We have built it using open-source components we need to carve out some time and get some things in order before we can open-source it. Since there are no immediate plans around it, adding arch details and numbers would definitely help the audience connect better.

      Your suggestions are strongly welcome. We will definitely look into making the proposal more elaborate by this week.

      • Zainab Bawa (@zainabbawa) Reviewer 5 months ago

        Thanks for the response, Ankur. I am moving your proposal to Rootconf Pune edition. If you want it to be placed in Rootconf Hyd or Delhi editions instead, let us know and we will move it accordingly.

  • Zainab Bawa (@zainabbawa) Reviewer 4 months ago

    Ankur, I have moved your proposal to Rootconf Hyderabad edition. Will schedule a rehearsal in August to carry out peer review.

  • Anwesha Sarkar (@anweshaalt) Reviewer 3 months ago

    Hello Ankur,

    It was a good rehearsal taday. Here are the feedback from the rehearsal:

    1. Why are the target audience of the talk? And why it is so?
    2. Explain the goal of the talk in the start.
    3. Include some references and resources.
    4. Include diagrams.
    5. Include the contact details in the last slides
    6. Include a demo.
    7. Avoid having text heavy slides.
    8. Include comparison of Shatabdi with other open source alternatives.
    9. Include the failures.
    10. Change the title of the talk. Construct it around the problem and not around the tool.
    11. Explain the terminologies in a seperate slide.

    Submit your revised slides by 10th September 2019.

    Regards
    Anwesha

  • Anwesha Sarkar (@anweshaalt) Reviewer 2 months ago

    Hello Ankur,

    The deadline for submitting the revised slides were 10th but I do see nothing till now. Submit your revised slides by 23rd September treat it as a hard deadline.

    Ping me if you have any questions.

    Regards
    Anwesha

    • Ankur Jain (@ankurj11) Proposer 2 months ago

      Our internal metrics system was experiencing issues wrt historical data which was required to complete benchmarking slides. It will take 3 more days to get resolved. 23rd September works for me.

  • Anwesha Sarkar (@anweshaalt) Reviewer a month ago

    Hello Ankur,

    It was good presentation today. Here are the feedback form today’s rehearsal:

    1. Be gender neutral.
    2. Do not mix languages.
    3. The introduction is messy and disconnected. Have a clear cut agenda.The context is missing.
    4. Do not spend so much time on P2P network.
    5. On the “Crash Course on BitTorrent” slide while explaining and follow the order
    6. Please get rid of the music
    7. Text in the demo was not visible, make the font size bigger.

    See you at the conference.

    Regards,
    Anwesha

  • Anwesha Sarkar (@anweshaalt) Reviewer a month ago

    While BitTorrent is the one of the most common protocol for transferring large files. Let us learn how does it work https://skerritt.blog/bit-torrent/. To have detailed discussion on this join us in Rootconf Hyderabad.

  • Anwesha Sarkar (@anweshaalt) Reviewer a month ago

    Content distribution in p2p systems: https://hal.inria.fr/inria-00439171/document to know about this and more join Ankur Jain in his talk on “Fast Object Distribution Using p2p”.

Login with Twitter or Google to leave a comment