Rootconf 2017

On service reliability

Anatomy of an alert

Submitted by Spencer Krum (@nibalizer) on Jan 27, 2017

Section: Crisp talk of 15 mins duration Technical level: Beginner Status: Cancelled


In this crisp talk, I’ll explain what makes a good alert. I’ll talk about several factors to consider when crafting an alert, such as actionability. I’ll give some examples of great alerts and not so great alerts. Audience members can take this information back to their systems and improve or remove bad alerts, and they can create only good alerts going forward.


I. Introduction to my background (sysadmin and developer and devops)
II. Four Factors of alerts
a) Actionable
b) Containing Links to playbooks/docs
c) Specific
d) Headed to the right person
III. Alerting system behaviours that need to exist (this is very short)
a) I don’t care what system you use but it needs these things:
b) Automatic rotation
c) Automatic escalation
d) Throttling
IV. Examples of good and bad alerts


Just my laptop and a projector

Speaker bio

Spencer (nibalizer) Krum ( has been sysoping Linux since 2010. He works for IBM contributing upstream to OpenStack and Puppet. Spencer is a core contributor to the OpenStack Infrastructure Project. Spencer coordinates the local DevOps user group in Portland and volunteers for an ops-training program at Portland State University called the Braindump. Spencer is a published author and frequent speaker at technical conferences. Spencer is a maintainer for the voxpupuli effort(, which attempts to bring together a network of Puppet developers, modules, and infrastructure.

Spencer lives and works in Portland, Oregon where he enjoys tennis, cheeseburgers and StarCraft II.


Preview video


  • Philip Paeps (@trouble) 3 years ago

    That’s a lot of material for a crisp talk! Could we convince you to turn this into a 40-minute talk?

  • Spencer Krum (@nibalizer) Proposer 3 years ago

    I would be happy to give a 40 minute talk. I do think I can get it into a crisp talk (maybe cut some stuff) if that fits better with your schedule.

    • Zainab Bawa (@zainabbawa) Crew 3 years ago

      Will be better to make a rough slide deck so that we can evaluate how much material there is and how much time the session will take. It’s likely that there is more material but not enough for 40 mins. In which case we can customize the session duration.

Login to leave a comment