Cloud Costs Optimization

Background

About IDfy

IDfy is a leader in the digital onboarding and verification space.
We enable our customers to seamlessly onboard employees, customers, vendors, users and more while preventing fraud at the same time.
We are a 80 member tech team which contribute to multiple products on the platform.

About our Tech Platform

We

• are multi-cloud capable and manage single and multi-tenant deployments in production.
• run approximately 400 services across our platform primarily hosted on Google Cloud.
• rely heavily on Kubernetes, containers and several other cloud specific services.
• operate multiple kinds of workloads including ML training and inference workloads utilizing GPUs

About this presentation

We will talk about why and how we went on our cost optimization journey and where we are today.

Cost optimization at IDfy

The initial impetus came about when we saw month-on-month increase in infrastrucure costs without a linear increase in volumes. This got us thinking that something had to change.
So in late June 2022, we got around to optimizing our costs which paid off inside of 2 months.

The entire set of activities can be summarized under

• People
• Process
• Tools

People

• Empowering team members to take decisions where and how to optimize and then quickly execute changes in days
• Defined ownership for continued cost monitoring and optimization

Process

• We had a clear set of guiding principles for areas of optimization
  • A baseline target was set
  • Quick improvements and changes with high monetary impact were prioritized
  • Then the more complex changes were picked up
• Knowing when to stop (not over-stretching so as to impact other deliverables)
• Setting up practices for optimal setups
• Continued monitoring

Tools

• Breaking down cost dashboards
• Really understanding where costs come from (Service/SKU level breakdown)
• Profiling

Putting it all into action

• Our costs really came from a few cloud resources

  • Compute
  • Databases
  • Logging
  • Managed Services

• Basic principles

  • Stop what’s not needed, when not needed (Staging environments)
  • Reduce replicas (fewer pods, but nothing is impacted)
  • Cut the fat (overprovisioned cpu and memory from the good days)
  • Consolidate (shared resources)
  • Reduce reliance on costly managed services
  • Optimize (queries)
  • Clean up (unnecessary logs)

• A few steps down the road

  • Get a better deal from your cloud provider (discounts, cheaper performant hardware, etc)
  • Committments (pay upfront for what you know you will need)
  • Deeper view into costs (slice, dice and compare)
  • Basic autoscaling

• Today

  • Nuanced Autoscaling
  • Profiling and optimization before deploying to prod
  • Continuous monitoring (with set ownership of monitoring and reporting)
  • A process of delegated ownership and action with a monthly checkpoint with leads

Outcomes

• A cost reduction of close to 40% inside of 2 months
• Some good practices and principles that have stuck with us
• A sense of ownership around cost and continuous optimization mindset
• Per unit cost that has stayed the same or reduced since Sep 2022

Presentation

Link