Rootconf 2016

Rootconf is India's principal conference where systems and operations engineers share real world knowledge about building resilient and scalable systems.

Participate

DevOps for Big Data infrastructure on the cloud

Submitted by Umang Dhawan (@umangd) on Thursday, 25 February 2016

Section: Crisp talk Technical level: Beginner

Abstract

To discuss the challenges faced in managing and deploying to hadoop clusters on the cloud and how to tackle them

Outline

I will discuss the following unique aspects of our DevOps stack:

  • Requirement of bringing up > 100 node hadoop clusters within 2 mins.
  • Rolling deploys of Hadoop clusters and clients in the presence of long running jobs
  • Autoscaling clusters to ensure optimal efficiency and expenditure
  • Supporting multiple public clouds - AWS, GCE, Azure
  • Allowing users to install their own software.

I’d like to discuss our strategies in handling issues like this and talk about tools that we’ve developed/discovered along the way.

Speaker bio

Currently working as a DevOps engineer at Qubole. Before this was a Platform engineer at BrowserStack, working on AWS and a custom private cloud setup. You can find more about me on LinkedIn - https://in.linkedin.com/in/umangdhawan

Comments

Login with Twitter or Google to leave a comment