FireStorm - An alerting system for your infra
Submitted by Ambarish KC (@ambrishkc) on Wednesday, 1 November 2017
Section: Full talk Technical level: Intermediate
We are living in the era of microservice and serverless architecture. More the moving parts in a system higher the probability of unhandled bugs/errors. We have numerous services like sentry, datadog etc to monitor our infra and log our excpetions, but most of the times developers get 100s of notification per day from these services. Naturally some of the critical notifications might get lost or would be seen at a later point in time. So the best option to react and fix these mission critical issues is to alert the respective team members by making phone calls. In this talk we’re gonna discuss and build a robust alerting system which would give instantaneous alerts.
Key takeaways would be any web or back-end developer can set up an alerting system (make phone calls, push to slack and send sms & emails) at almost negligible COST
- Setting up of AWS alarms and twilio
- Write a Django API server which accepts POST calls for different types of triggers
- How to integrate AWS cloudwatch alarms with the Django API server
- Code to trigger phone calls, push slack message, send sms and email
- Demo of the product
- How this project can be extended for various other functionalities like pager duty, modify auto scaling policies, run play books, predict crashes etc
Note: All the code will be written in Python
Ambarish is a back-end engineer @ MadStreetDen building robust scalable systems serving millions of users. Designed, built and deployed back-end systems from scratch.