How to setup a 2-way alerting service over sms/text/pager, in this instance, for Nagios.
We’re really hoping that Part two and three do indeed follow in a timely fashion and deliver the goods, because part one certainly whets the appetite. (Update: Part two and Part three are up)
Good quality alerting is really useful and important, but there are so many issues to deal with:
- Timely. SMS doesn’t promise how quick, or even if, your message will arrive. True it normally happens near instantly, but you can’t bank on it.
- Concise. There is the obvious character limit concern, but more generally you don’t want to be flooded with sms messages. Sure it’s better than none, but it’s better still to not get the same message every 5 minutes for 6 hours.
- Stateful. Even if you fix the problem of not sending multiple duplicate alerts, a machine that flip-flops between good/bad states is even harder to spot.
- Severity. If the load average goes over 10 and it’s 3am in the morning do you want to wake up your sysadmin immediately? What if it happens every day about this time? If it stays that way for 5 minutes? 10 minutes? Is it a different answer at 9am?
- Who. If you have more than one person doing your sysadmin, who gets the 3am call? Does your management team or PR rep want to know about it too?
Not to mention just the basic implementation/setup of it. And then you have one of those Klein Bottle moments and wonder what is monitoring the monitoring?
As ever, your thoughts, experiences, comments or suggestions would be welcomed. Either in the comments below, or drop us an email.
Leave a comment