Dr. Elephant: Achieving Quicker, Easier, and Cost-effective Big Data Analytics

Jul 2017

24 Mon

25 Tue

26 Wed

27 Thu 08:15 AM – 10:00 PM IST

28 Fri 08:15 AM – 06:25 PM IST

29 Sat

30 Sun

MLR Convention Centre, Whitefield, Bengaluru,

Dr. Elephant: Achieving Quicker, Easier, and Cost-effective Big Data Analytics

Submitted Apr 27, 2017

Section: Crisp talk for Data in Government track Technical level: Intermediate

Open Source: https://github.com/linkedin/dr-elephant

Is your job running slower than usual? Do you want to make sense from the thousands of Hadoop & Spark metrics? Do you want to monitor the performance of your flow, get alerts and auto tune them? These are the common questions every Hadoop user asks but there is not a single solution that addresses it. We at Linkedin faced lots of such issues and have built a simple self serve tool for the hadoop users called Dr. Elephant.

Dr. Elephant, which is already open sourced, is a performance monitoring and tuning tool for Hadoop and Spark. It tries to improve the developer productivity and cluster efficiency by making it easier to tune jobs. Since its open source, it has been adopted by multiple organizations and followed with a lot of interest in the Hadoop and Spark community.

In this talk, we will discuss about Dr. Elephant and outline our efforts to expand the scope of Dr. Elephant to be a comprehensive monitoring, debugging and tuning tool for Hadoop and Spark applications. We will talk about how Dr. Elephant performs exception analysis, give clear and specific suggestions on tuning, tracking metrics and monitoring their historical trends.

Outline

Phase 1: Introduction to Dr. Elephant. How and why it all started?

Phase 2: This section will cover how Dr. Elephant helps in quickly and easily optimizing Big data applications, saving millions of dollars for the company. I will also talk about the open source activities and initiatives around Dr. Elephant.

Phase 3: I will talk about our efforts to expand the scope of Dr. Elephant to be a comprehensive monitoring, debugging and tuning tool for Hadoop and Spark applications.

Speaker bio

Akshay Rai is an engineer at Linkedin working with the Grid team. He is also the lead engineer for the open sourced Dr. Elephant project by Linkedin. He has been working on solutions to improve the developer productivity and building systems to monitor Big Data applications in real time.

Slides

https://drive.google.com/file/d/0BzdEJVP7_lZjSDRualFpM19kem8/view?usp=sharing

The Fifth Elephant 2017