Rootconf Hyderabad edition

On SRE, systems engineering and distributed systems

Up next

Dashboards as Code

SM

Sanooj Mananghat

@sanoojm

Understanding the state of your infrastructure and systems is essential for ensuring the reliability and stability of services. Best way to gain this insight is with a robust dashboard that visualizes data and stable alert rules which alerts when things appear to be broken.
Configuring and modifying dashboards and alerts by hand are error-prone. Versioning dashboards and alerts using “Infrastructure as code” are extremely useful in a fast-paced environment.

Outline

How to make use of the world’s top technology to build your Dashboards with more robustness without tech debt.

Why dashboards?

Existing methods of maintaining dashboards.

  • At current scale of using dashbaords and alerts, difficult to scale without proper automation.
  • Consistency issues
    • Dashboards can be easily edited by any layman resulting in consistency issues.
    • Alerts disbaled during maintenance/deployment leading to undetected incidents.
  • Automate challenges
    • 90% of the dashboards are still being created/modified by clicking on the UI.
    • Manual configuration can lead to errors.
  • No history.
    • No rollbacks possible during an unintended modification.

Solution 1: Git

  • Will solve Consistency issues and maintains history.
  • It still cannot validate the huge json files, manual review is required.
  • Automation of stored json files to dashboards are still not solved.

Solution 2: Git + Terraform

  • Terraform providers/plugins. - Automating dashboards using terrafrom providers.
  • Manage state, conflicts, support validation, rollback, everything with zero tech debt.
  • Add value to your skills.

Demo of how to create a dashboard with Terraform and Grafana

Speaker bio

Sanooj Mananghat is working as a DataSRE @ Intuit. With 9 years of experience in Devops world. A blogger, opensource enthusiast with contributions to multiple opensource projects including terraform providers.

Links

Slides

https://docs.google.com/presentation/d/e/2PACX-1vSmx0HtVlyFPO8dfwLDAGTVXq1UzBlu9AK8z0gsWpdBDXroZ-Dk7OQdMZF7BOvMxl_D1bd7duR3GEIz/pub?start=false&loop=false&delayms=3000

Comments