Miniconf on Cloud Server Management (Chennai)

On costs, scaling and securing cloud servers

PLUTUS : Serverless monitoring and alerting of costs incurred on AWS

Submitted by Aayush Kumar (@akjarvis) on Tuesday, 7 November 2017

videocam
Preview video

Technical level

Intermediate

Section

Crisp Talk

Status

Confirmed & Scheduled

View proposal in schedule

Vote on this proposal

Login to vote

Total votes:  +10

Abstract

We at Indix, a Data as a Service company, are building world’s largest cloud catalog for structured marketplace product information. Our scale of data is in TBs which we deliver to our customers through APIs and Bulk Feeds. We use resources on AWS cloud extensively which comes at a significant cost.

As our infrastructure scaled, we realised cost optimisation on AWS cloud as an important operational challege. We adopted most of the standard best practices for controlling costs which reduced our cloud costs to a large extent. However, one of our biggest challenge was to keep track of costs which were incurred on account of unexpected or unknown events like rapid upscaling/downscaling of Autoscaling Groups, untracked Hadoop jobs with a potential to incur huge data transfer costs, ondemand EC2 instances which are not in use and ofcourse human erros.

This is where we built “Plutus”, a near real time cost monitoring tool, where we could programatically monitor cost on each and every AWS resource and get alerted when a set threshold cost is crossed for a resource on daily/weekly basis. Post deploying this, we are now able to keep track of the cost incurred by all our systems which definitely gives us a tight control over our monthly AWS expenses.

In this talk, I’ll talk about our learnings on cost optimisation, the strategies on AWS we follow at Indix, architecture and implentation of Plutus, it’s direct impact on AWS costs and it’s future scope of being a full fledged open source project.

Outline

  1. Introduction
  2. AWS cost control best practices
  3. Need for building Plutus despite of all the cost control measures
  4. Plutus architecture how it works.
  5. Impact of Plutus and future scope.

Speaker bio

Aayush Kumar works as an Infrastructure and DevOps Engineer at Indix. Focuses on automating the unautomated and is always curious to understand systems if not contribute to it.

Links

Slides

https://speakerdeck.com/aayushkumarjarvis/plutus-aws-cost-management-and-alerting-tool

Preview video

https://www.youtube.com/watch?v=9b_YNM7L6bI

Comments

Login with Twitter or Google to leave a comment