Rootconf 2016

Rootconf is India's principal conference where systems and operations engineers share real world knowledge about building resilient and scalable systems.

Umang Dhawan


DevOps for Big Data infrastructure on the cloud

Submitted Feb 25, 2016

To discuss the challenges faced in managing and deploying to hadoop clusters on the cloud and how to tackle them


I will discuss the following unique aspects of our DevOps stack:

  • Requirement of bringing up > 100 node hadoop clusters within 2 mins.
  • Rolling deploys of Hadoop clusters and clients in the presence of long running jobs
  • Autoscaling clusters to ensure optimal efficiency and expenditure
  • Supporting multiple public clouds - AWS, GCE, Azure
  • Allowing users to install their own software.

I’d like to discuss our strategies in handling issues like this and talk about tools that we’ve developed/discovered along the way.

Speaker bio

Currently working as a DevOps engineer at Qubole. Before this was a Platform engineer at BrowserStack, working on AWS and a custom private cloud setup. You can find more about me on LinkedIn -


{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

We care about site reliability, cloud costs, security and data privacy