SignifAI Blog Stop getting paged for useless alerts

If you've ever been an on-call SRE, you know the struggle of pager fatigue. The everyday balance between maintaining existing infrastructure and developing sustainable solutions is tough enough, but add a constant stream of alerts and it can get exhausting pretty quickly! In an ideal world, on-call engineers would get alerted for only the production issues that are both urgent and important. However, the Eisenhower matrix distribution of many production systems' alerts looks more like this: This means that more than 75% of your time, the notifications stressing you out aren't worth the sweat. So where is all of this noise coming from? Irrelevant alerts Unused services, decommissioned projects, and issues that are actively being handled by other teams are some sources of alert noise that are just prevalent enough to annoy you, but not always enough to go through the legwork of turning them off at their source. These notifications come from all kinds of places in your production system and tend to get quickly "acked" but largely ignored, since there usually isn't an underlying actionable issue. Low-priority alerts Some noisemakers indicate problems that may eventually need to be addressed, but are low on the priority list. Keeping theseCustom Link;The post Stop getting paged for useless alerts appeared first on Thoughts on DevOps and Machine Intelligence by SignifAI.