I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this?
Thanks, Larry Barber
What are your alert rules? I believe the same step that "turns it red" also goes through alerting.
On 9/4/09, Larry Barber <lebarber at gmail.com> wrote:
I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this?
Thanks, Larry Barber
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
"When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
The alert rules are very simple:
PAGE=%^APHIS$ SCRIPT /usr/local/hobbit/server/ext/SaveAlert.py DUMMY COLORS=red DURATION>0 REPEAT=30d RECOVERED
... and so on for each page of the Hobbit installation. The script is the start of a process that helps our NOC to manage alerts and notify responsible parties, not being able to automatically remove things that go red->yellow->green will cause problems.
Thanks, Larry Barber
On 9/4/09, Josh Luthman <josh at imaginenetworksllc.com> wrote:
What are your alert rules? I believe the same step that "turns it red" also goes through alerting.
On 9/4/09, Larry Barber <lebarber at gmail.com> wrote:
I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this?
Thanks, Larry Barber
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
"When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Given the state of the alerts module for this type of case (reference my earlier reply), I've had to live with it. Only other solution beside a rewrite of the alert module I've imagined, has been to just use a single rule to send all alerts via script to an external script/process, that does a more complete management of sending alerts. My belief is that the recovery event does happen - there's just no way to write the matching recovery rule. -Alan
Larry Barber wrote:
The alert rules are very simple:
PAGE=%^APHIS$ SCRIPT /usr/local/hobbit/server/ext/SaveAlert.py DUMMY COLORS=red DURATION>0 REPEAT=30d RECOVERED
... and so on for each page of the Hobbit installation. The script is the start of a process that helps our NOC to manage alerts and notify responsible parties, not being able to automatically remove things that go red->yellow->green will cause problems.
Thanks, Larry Barber
On 9/4/09, *Josh Luthman* <josh at imaginenetworksllc.com <mailto:josh at imaginenetworksllc.com>> wrote:
What are your alert rules? I believe the same step that "turns it red" also goes through alerting. On 9/4/09, Larry Barber <lebarber at gmail.com <mailto:lebarber at gmail.com>> wrote: > I've noticed that if I have a situation where I have red alert that turns > yellow before finally returning to green that no recovery messages are being > sent. Alert notifications are only being sent on reds, not yellow. I suspect > that when the red turns to a yellow that Hobbit is "forgetting" that it sent > out the notification when it turned red since it is not sending them out on > yellows. Does anybody out there know how to fix this? > > Thanks, > Larry Barber > -- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk <mailto:hobbit-unsubscribe at hswn.dk>
We had the same problem. We "solved" it by changing yellow to an "OK color" in hobbitserver.cfg, so now it sends out recovered when it goes from red to yellow. But if you have alerts on yellow, you probably don't want to do that.
/Johan
From: Larry Barber [mailto:lebarber at gmail.com] Sent: den 4 september 2009 15:59 To: hobbit Subject: [hobbit] recovery messages
I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this?
Thanks, Larry Barber
This is a known problem with the alert mechanism. I posted an analysis of why (I believe) it happens in this mailing list message: http://www.hswn.dk/hobbiton/2008/07/msg00132.html
Unfortunately, there appears to be no way in current state to work around the problem. -Alan
Johan Sjöberg wrote:
We had the same problem. We “solved” it by changing yellow to an “OK color” in hobbitserver.cfg, so now it sends out recovered when it goes from red to yellow. But if you have alerts on yellow, you probably don’t want to do that.
/Johan
*From:* Larry Barber [mailto:lebarber at gmail.com] *Sent:* den 4 september 2009 15:59 *To:* hobbit *Subject:* [hobbit] recovery messages
I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this?
Thanks, Larry Barber
Actually, that will work for me since I have a separate Hobbit instance handling my alerting from the one that runs the script.
Thanks, Larry Barber
On 9/4/09, Johan Sjöberg <johan.sjoberg at deltamanagement.se> wrote:
We had the same problem. We “solved” it by changing yellow to an “OK color” in hobbitserver.cfg, so now it sends out recovered when it goes from red to yellow. But if you have alerts on yellow, you probably don’t want to do that.
/Johan
*From:* Larry Barber [mailto:lebarber at gmail.com] *Sent:* den 4 september 2009 15:59 *To:* hobbit *Subject:* [hobbit] recovery messages
I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this?
Thanks, Larry Barber
participants (4)
-
asparks@doublesparks.net
-
johan.sjoberg@deltamanagement.se
-
josh@imaginenetworksllc.com
-
lebarber@gmail.com