Rootconf Hyderabad edition

On SRE, systems engineering and distributed systems

Dashboards as Code

Submitted by Sanooj Mananghat (@sanoojm) on May 27, 2019

Section: Crisp talk Technical level: Intermediate Session type: Lecture Status: Confirmed & scheduled

Abstract

Understanding the state of your infrastructure and systems is essential for ensuring the reliability and stability of services. Best way to gain this insight is with a robust dashboard that visualizes data and stable alert rules which alerts when things appear to be broken.
Configuring and modifying dashboards and alerts by hand are error-prone. Versioning dashboards and alerts using “Infrastructure as code” are extremely useful in a fast-paced environment.

Outline

How to make use of the world’s top technology to build your Dashboards with more robustness without tech debt.

Why dashboards?

Existing methods of maintaining dashboards.

  • At current scale of using dashbaords and alerts, difficult to scale without proper automation.
  • Consistency issues
    • Dashboards can be easily edited by any layman resulting in consistency issues.
    • Alerts disbaled during maintenance/deployment leading to undetected incidents.
  • Automate challenges
    • 90% of the dashboards are still being created/modified by clicking on the UI.
    • Manual configuration can lead to errors.
  • No history.
    • No rollbacks possible during an unintended modification.

Solution 1: Git

  • Will solve Consistency issues and maintains history.
  • It still cannot validate the huge json files, manual review is required.
  • Automation of stored json files to dashboards are still not solved.

Solution 2: Git + Terraform

  • Terraform providers/plugins. - Automating dashboards using terrafrom providers.
  • Manage state, conflicts, support validation, rollback, everything with zero tech debt.
  • Add value to your skills.

Demo of how to create a dashboard with Terraform and Grafana

Speaker bio

Sanooj Mananghat is working as a DataSRE @ Intuit. With 9 years of experience in Devops world. A blogger, opensource enthusiast with contributions to multiple opensource projects including terraform providers.

Links

Slides

https://docs.google.com/presentation/d/e/2PACX-1vSmx0HtVlyFPO8dfwLDAGTVXq1UzBlu9AK8z0gsWpdBDXroZ-Dk7OQdMZF7BOvMxl_D1bd7duR3GEIz/pub?start=false&loop=false&delayms=3000

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('You need to be a participant to comment.') }}

{{ formTitle }}
{{ gettext('Post a comment...') }}
{{ gettext('New comment') }}

{{ errorMsg }}