Miniconf on Cloud Server Management (Chennai)

We at Indix, a Data as a Service company, are building world’s largest cloud catalog for structured marketplace product information. Our scale of data is in TBs which we deliver to our customers through APIs and Bulk Feeds. We use resources on AWS cloud extensively which comes at a significant cost.

As our infrastructure scaled, we realised cost optimisation on AWS cloud as an important operational challege. We adopted most of the standard best practices for controlling costs which reduced our cloud costs to a large extent. However, one of our biggest challenge was to keep track of costs which were incurred on account of unexpected or unknown events like rapid upscaling/downscaling of Autoscaling Groups, untracked Hadoop jobs with a potential to incur huge data transfer costs, ondemand EC2 instances which are not in use and ofcourse human erros.

This is where we built “Plutus”, a near real time cost monitoring tool, where we could programatically monitor cost on each and every AWS resource and get alerted when a set threshold cost is crossed for a resource on daily/weekly basis. Post deploying this, we are now able to keep track of the cost incurred by all our systems which definitely gives us a tight control over our monthly AWS expenses.

In this talk, I’ll talk about our learnings on cost optimisation, the strategies on AWS we follow at Indix, architecture and implentation of Plutus, it’s direct impact on AWS costs and it’s future scope of being a full fledged open source project.

Outline

1. Introduction
2. AWS cost control best practices
3. Need for building Plutus despite of all the cost control measures
4. Plutus architecture how it works.
5. Impact of Plutus and future scope.

Speaker bio

Aayush Kumar works as an Infrastructure and DevOps Engineer at Indix. Focuses on automating the unautomated and is always curious to understand systems if not contribute to it.

Links

• Blog article on Medium : https://medium.com/@akJarvis/exploring-cost-optimisation-on-aws-bc7cd64e6594

Slides

https://speakerdeck.com/aayushkumarjarvis/plutus-aws-cost-management-and-alerting-tool