Call me maybe: Jepsen and flaky networks

Jul 2015

13 Mon

14 Tue

15 Wed

16 Thu 08:30 AM – 06:35 PM IST

17 Fri 08:30 AM – 06:30 PM IST

18 Sat 09:00 AM – 06:30 PM IST

19 Sun

Make a submission

NIMHANS Convention center

Machine Learning, Distributed and Parallel Computing, and High-performance Computing are the themes for this year’s edition of Fifth Elephant.

The deadline for submitting a proposal is 15th June 2015

We are looking for talks and workshops from academics and practitioners who are in the business of making sense of data, big and small.

Track 1: Discovering Insights and Driving Decisions #

This track is about general, novel, fundamental, and advanced techniques for making sense of data and driving decisions from data. This could encompass applications of the following ML paradigms:

Statistical Visualizations
Unsupervised Learning
Supervised Learning
Semi-Supervised Learning
Active Learning
Reinforcement Learning
Monte-carlo techniques and probabilistic programming
Deep Learning

Across various data modalities including multi-variate, text, speech, time series, images, video, transactions, etc.

Track 2: Speed at Scale #

This track is about tools and processes for collecting, indexing, and processing vast amounts of data. The theme includes:

Distributed and Parallel Computing
Real Time Analytics and Stream Processing
MapReduce and Graph Computing frameworks
Kafka, Spark, Hadoop, MPI
Stories of parallelizing sequential programs
Cost/Security/Disaster Management of Data

Commitment to Open Source #

HasGeek believes in open source as the binding force of our community. If you are describing a codebase for developers to work with, we’d like it to be available under a permissive open source license. If your software is commercially licensed or available under a combination of commercial and restrictive open source licenses (such as the various forms of the GPL), please consider picking up a sponsorship. We recognize that there are valid reasons for commercial licensing, but ask that you support us in return for giving you an audience. Your session will be marked on the schedule as a sponsored session.

Workshops #

If you are interested in conducting a hands-on session on any of the topics falling under the themes of the two tracks described above, please submit a proposal under the workshops section. We also need you to tell us about your past experience in teaching and/or conducting workshops.

Hosted by

The Fifth Elephant

The Fifth Elephant - known as one of the best data science and Machine Learning conference in Asia - has transitioned into a year-round forum for conversations about data and ML engineering; data science in production; data security and privacy practices. more

All submissions

Previous Next

This submission has been added to the schedule

Call me maybe: Jepsen and flaky networks

Submitted Jun 15, 2015

Section: Full Talk Technical level: Advanced

Tell people that network partitions happen often enough that it is worth caring about how their distributed data stores respond in such situations
Introduce people to a new way of testing distributed systems under stress using Jepsen
Make people aware of the gap between expected behavior and actual behavior of systems like Cassandra, MongoDB and Elastic
Describe the kind of bugs that we have found in Solr using Jepsen, and our plans for the future

Outline #

In the big data world, our data stores communicate over an asynchronous, unreliable network to provide a facade of consistency. However, to really understand the guarantees of these systems, we must understand the realities of networks and test our data stores against them.

Jepsen is a tool which simulates network partitions in data stores and helps us understand the guarantees of our systems and its failure modes. In this talk, I will help you understand why you should care about network partitions and how can we test datastores against partitions using Jepsen. I will explain what Jepsen is and how it works and the kind of tests it lets you create. We will try to understand the subtleties of distributed consensus, the CAP theorem and demonstrate how different data stores such as MongoDB, Cassandra, Elastic and Solr behave under network partitions. Finally, I will describe the results of the tests I wrote using Jepsen for Apache Solr and discuss the kinds of rare failures which were found by this excellent tool.

Requirements #

People using distributed data stores such as Solr, Cassandra, MongoDB, Redis, Elastic
Distributed systems aficionados

Knowledge of one or more of these distributed stores is required. A basic familiarity of the CAP theorem will be helpful.

Speaker bio #

I am a committer on Apache Lucene/Solr since 2008 as well as a member of the Lucene/Solr project management committee. I’ve worked at AOL for five years on vertical search, content mangement systems, social/community platforms and anti-spam systems as well as AOL WebMail’s Inbox Search system which uses a highly customized version of Apache Solr to service tens of millions of users and more than a billion index/search operations a day. I currently work at Lucidworks Inc. on Apache Solr and Lucidworks Search mostly on the SolrCloud side of things. I also help organize the Bangalore Apache Solr/Lucene Meetup Group which has 450+ members and holds regular meetings of people interested in Lucene, Solr and search in general.

https://twitter.com/shalinmangar

https://in.linkedin.com/in/shalinsmangar

http://shal.in/