The Fifth Elephant 2015

A conference on data, machine learning, and distributed and parallel computing

Shalin Mangar


Call me maybe: Jepsen and flaky networks

Submitted Jun 15, 2015

  • Tell people that network partitions happen often enough that it is worth caring about how their distributed data stores respond in such situations
  • Introduce people to a new way of testing distributed systems under stress using Jepsen
  • Make people aware of the gap between expected behavior and actual behavior of systems like Cassandra, MongoDB and Elastic
  • Describe the kind of bugs that we have found in Solr using Jepsen, and our plans for the future


In the big data world, our data stores communicate over an asynchronous, unreliable network to provide a facade of consistency. However, to really understand the guarantees of these systems, we must understand the realities of networks and test our data stores against them.

Jepsen is a tool which simulates network partitions in data stores and helps us understand the guarantees of our systems and its failure modes. In this talk, I will help you understand why you should care about network partitions and how can we test datastores against partitions using Jepsen. I will explain what Jepsen is and how it works and the kind of tests it lets you create. We will try to understand the subtleties of distributed consensus, the CAP theorem and demonstrate how different data stores such as MongoDB, Cassandra, Elastic and Solr behave under network partitions. Finally, I will describe the results of the tests I wrote using Jepsen for Apache Solr and discuss the kinds of rare failures which were found by this excellent tool.


  • People using distributed data stores such as Solr, Cassandra, MongoDB, Redis, Elastic
  • Distributed systems aficionados

Knowledge of one or more of these distributed stores is required. A basic familiarity of the CAP theorem will be helpful.

Speaker bio

I am a committer on Apache Lucene/Solr since 2008 as well as a member of the Lucene/Solr project management committee. I’ve worked at AOL for five years on vertical search, content mangement systems, social/community platforms and anti-spam systems as well as AOL WebMail’s Inbox Search system which uses a highly customized version of Apache Solr to service tens of millions of users and more than a billion index/search operations a day. I currently work at Lucidworks Inc. on Apache Solr and Lucidworks Search mostly on the SolrCloud side of things. I also help organize the Bangalore Apache Solr/Lucene Meetup Group which has 450+ members and holds regular meetings of people interested in Lucene, Solr and search in general.



{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

All about data science and machine learning