The Fifth Elephant 2015

• Tell people that network partitions happen often enough that it is worth caring about how their distributed data stores respond in such situations
• Introduce people to a new way of testing distributed systems under stress using Jepsen
• Make people aware of the gap between expected behavior and actual behavior of systems like Cassandra, MongoDB and Elastic
• Describe the kind of bugs that we have found in Solr using Jepsen, and our plans for the future

Outline

In the big data world, our data stores communicate over an asynchronous, unreliable network to provide a facade of consistency. However, to really understand the guarantees of these systems, we must understand the realities of networks and test our data stores against them.

Jepsen is a tool which simulates network partitions in data stores and helps us understand the guarantees of our systems and its failure modes. In this talk, I will help you understand why you should care about network partitions and how can we test datastores against partitions using Jepsen. I will explain what Jepsen is and how it works and the kind of tests it lets you create. We will try to understand the subtleties of distributed consensus, the CAP theorem and demonstrate how different data stores such as MongoDB, Cassandra, Elastic and Solr behave under network partitions. Finally, I will describe the results of the tests I wrote using Jepsen for Apache Solr and discuss the kinds of rare failures which were found by this excellent tool.

Requirements

• People using distributed data stores such as Solr, Cassandra, MongoDB, Redis, Elastic
• Distributed systems aficionados

Knowledge of one or more of these distributed stores is required. A basic familiarity of the CAP theorem will be helpful.

Speaker bio

I am a committer on Apache Lucene/Solr since 2008 as well as a member of the Lucene/Solr project management committee. I’ve worked at AOL for five years on vertical search, content mangement systems, social/community platforms and anti-spam systems as well as AOL WebMail’s Inbox Search system which uses a highly customized version of Apache Solr to service tens of millions of users and more than a billion index/search operations a day. I currently work at Lucidworks Inc. on Apache Solr and Lucidworks Search mostly on the SolrCloud side of things. I also help organize the Bangalore Apache Solr/Lucene Meetup Group which has 450+ members and holds regular meetings of people interested in Lucene, Solr and search in general.

https://twitter.com/shalinmangar

https://in.linkedin.com/in/shalinsmangar

http://shal.in/

Links

• Blog post on testing SolrCloud with Jepsen: http://lucidworks.com/blog/call-maybe-solrcloud-jepsen-flaky-networks/
• Video of my keynote at Lucene Revolution 2014: https://www.youtube.com/watch?v=nxRROble76A
• All of my past presentations at various places: http://www.slideshare.net/shalinmangar
• Jepsen: https://github.com/aphyr/jepsen
• Kyle Kingsbury’s excellent series of blog posts on network partitions: https://aphyr.com/tags/jepsen

Slides

https://coggle.it/diagram/VYgIi_MaH0dZP5JY/dbf5d231a4bce19c5c6996274f040908cc83bf950c80cb80c83392d485db625c