Rootconf Hyderabad edition

Rootconf Hyderabad edition

On SRE, systems engineering and distributed systems

Sanooj Mananghat

@sanoojm

Dashboards as Code

Submitted May 27, 2019

Understanding the state of your infrastructure and systems is essential for ensuring the reliability and stability of services. Best way to gain this insight is with a robust dashboard that visualizes data and stable alert rules which alerts when things appear to be broken.
Configuring and modifying dashboards and alerts by hand are error-prone. Versioning dashboards and alerts using “Infrastructure as code” are extremely useful in a fast-paced environment.

Outline

How to make use of the world’s top technology to build your Dashboards with more robustness without tech debt.

Why dashboards?

Existing methods of maintaining dashboards.

  • At current scale of using dashbaords and alerts, difficult to scale without proper automation.
  • Consistency issues
    • Dashboards can be easily edited by any layman resulting in consistency issues.
    • Alerts disbaled during maintenance/deployment leading to undetected incidents.
  • Automate challenges
    • 90% of the dashboards are still being created/modified by clicking on the UI.
    • Manual configuration can lead to errors.
  • No history.
    • No rollbacks possible during an unintended modification.

Solution 1: Git

  • Will solve Consistency issues and maintains history.
  • It still cannot validate the huge json files, manual review is required.
  • Automation of stored json files to dashboards are still not solved.

Solution 2: Git + Terraform

  • Terraform providers/plugins. - Automating dashboards using terrafrom providers.
  • Manage state, conflicts, support validation, rollback, everything with zero tech debt.
  • Add value to your skills.

Demo of how to create a dashboard with Terraform and Grafana

Speaker bio

Sanooj Mananghat is working as a DataSRE @ Intuit. With 9 years of experience in Devops world. A blogger, opensource enthusiast with contributions to multiple opensource projects including terraform providers.

Links

Slides

https://docs.google.com/presentation/d/e/2PACX-1vSmx0HtVlyFPO8dfwLDAGTVXq1UzBlu9AK8z0gsWpdBDXroZ-Dk7OQdMZF7BOvMxl_D1bd7duR3GEIz/pub?start=false&loop=false&delayms=3000

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}