PWL Sep 2020: "The Tail at Scale"
Ideas behind building large scale latency tolerant distributed systems.
- MS
Meera Sundar
@mirage
Posted 4 years agoJust came across this interesting post-
https://instagram-engineering.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589
-
Zainab Bawa
@zainabbawa Editor & Promoter
Posted 4 years agoKrace has suggested we consider discussing Chord: https://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf
Chord is a protocol and algorithm for a peer-to-peer distributed hash table - https://en.wikipedia.org/wiki/Chord_(peer-to-peer)Any thoughts, suggestions?
- PG
Piyush Goel
@pigol1 Editor & Promoter
Posted 4 years agoThe slides from the session are available here - https://speakerdeck.com/pigol1/the-tail-at-scale-hasgeek-meetup
-
Swanand Pagnis
@swanand Editor & Promoter
Posted 4 years agoWe already discussed Your Server as a Function: https://www.meetup.com/Papers-we-love-Bangalore/events/268444851/
I guess we can repeat -- but there is a lot of demand for AWS Dynamo paper. Maybe that should be next.
-
Zainab Bawa
@zainabbawa Editor & Promoter
Posted 4 years ago (edited 4 years ago)Anyone among participants who wants to discuss Dynamo paper? r be a respondent?
-
Zainab Bawa
@zainabbawa Editor & Promoter
Posted 4 years agoAs discussed, AWS Dynamo paper next. Kracekumar has volunteered to present the paper. Meera, Navin, are you folks to ok to present Server as a Function paper on a Thursday evening? After the Dynamo paper is presented?
-
-
Zainab Bawa
@zainabbawa Editor & Promoter
Posted 4 years agoNavin and Meera - on Server as a Function - to discuss this at the next meeting. Any other suggestions?
- MS
Meera Sundar
@mirage
Posted 4 years agoLink to "Your server as a function" (PDF) https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.696.9710&rep=rep1&type=pdf
-
- MS
Meera Sundar
@mirage
Posted 4 years agoJaseem also spoke about Facebook's network load balancer Katran, which is detailed in this blogpost- https://engineering.fb.com/open-source/open-sourcing-katran-a-scalable-network-load-balancer/
-
Kracekumar R
@kracekumar
Posted 4 years agoThe paper discusses concepts like enqueuing requests two or more servers and cancel the pending request after the first response. Some of the techniques are available in distributed databases.
- How much of these concepts are pluggable at the application layer to an API server or to a queuing system?
- Are there libraries which perform these operations in an implementation agnostic way?
-
Zainab Bawa
@zainabbawa Editor & Promoter
Posted 4 years agoThanks for the questions, @kracekumar I have asked Piyush to look into these questions and address them here/during the session on Thursday.
- AC
Anand Chitipothu
@anandology
Posted 4 years agoCancelling the pending request after the first response is an optimization and not really required to reduce the tail latency. A naive approach would cause 2x load, assuming the request is sent to two replicas. However, if you pay attention to the idea of "Hedged requests", which sends the second request only after a brief delay, limiting the additional load to 5%.
It seems like a simple implementation of these ideas can be plugged to any API server easily, but I'm not aware of any libraries that perform this out of the box.
Hosted by
What was the last paper within the realm of computing you read and loved? What did it inspire you to build or tinker with? Come share the ideas in an awesome academic/research paper with fellow engineers, programmers, and paper-readers. Lead a session and show off code that you wrote that implement…
more