Hi all,
This seems to be a bug (or at least a flaw) in the alerting system (on hobbit trunk from Dec 2008): a recovery is sent for a MAIL alert that never actually sent an alert in the first place. The last two lines in the notifications.log are recoveries that were sent at the exact same time. But only 1 alert (alarm) was actually sent out.
[root at IVRA1 log]# tail /var/log/hobbit/notifications.log -n 3
Tue Jun 16 14:46:49 2009 Db1.Special2 (192.168.4.xx) meATmyCompanyDOTcom[191] 1245160009 0
Tue Jun 16 14:48:43 2009 Db1.Special2 (192.168.4.xx) meATmyCompanyDOTcom[191] 1245160123 0 492
Tue Jun 16 14:48:43 2009 Db1.Special2 (192.168.4.xx) meATmyCompanyDOTcom[192] 1245160123 0 492
The reason why this happened may be seen from the hobbit-alerts.cfg file. One of the alert lines triggers on persistent alarms (including yellow alarms), while the other is immediate but for red only. Both have RECOVERED alerts. The test was only red for 2 minutes, so only the first MAIL rule fired initially. It seems that the RECOVERED part does not check that this line actually triggered an alert before sending the recovered message!
HOST=Db1 SERVICE=Special2
MAIL meATmyCompanyDOTcom COLOR=red REPEAT=30 RECOVERED
MAIL meATmyCompanyDOTcom COLOR=yellow,red DURATION>10 REPEAT=30RECOVERED
There haven't been any changes to the alerting system since December right? Should I file this as a bug anywhere else?
Kind regards,
SebA