The Fifth Elephant 2015

A conference on data, machine learning, and distributed and parallel computing

Building a distributed cache system with redis, clojure and math

Submitted by Kapil Reddy (@kapilr) on Monday, 15 June 2015

videocam_off

Technical level

Intermediate

Section

Full Talk

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +23

Objective

Learn how consistent hashing, CRDTs and Clojure protocols can be used to build a distributed cache.

Description

This talk will highlight different things that went into building bunshin (a Distributed Cache system) built on top of Redis.

Key concepts

  • Consistent hashing: This let’s the system distribute load across multiple nodes. Also it reduces remapping load when number of nodes change in cluster.

  • CRDTs: Conflict-free replicated data types help you accept divergence in Distributed systems. CRDTs are designed such that conflicts are mathematically impossible

  • Clojure protocols: Protocols provide powerful mechanism for abstraction. This gives flexibility of changing implementations for future improvements like using a different datastore.

  • Generative Testing: Finding and documenting edge cases in Distributed systems is hard. Also benchmarking things can be tedious. With generative testing and protocols we will see how can solve these problems.

Speaker bio

Kapil Reddy is a Platform and Feature Lead at Helpshift, and is a core developer on many of the scalability projects at Helpshift. He has designed and open sourced Bunshin, which is used to implement a large distributed caching layer capable of serving over 35K requests per second. He believes that skynet is already sentient.

Links

Comments

Login with Twitter or Google to leave a comment