Alerts on server reboot?
Is there a way to suppress the flood of alerts we get on a server reboot? If the machine is down for 10 minutes and comes back up, we get flooded with purple messages.
-- Stewart
Purple messages on what? Connection alarms should only go red, maybe for a couple polling cycles. Services, I'd imagine, would come back up with the server.
Regardless, you can suppress alarms for a given host and/or service on that host with several ways. The two that come to mind for me are:
- Use the DOWNTIME directive in bb-hosts (see the man-page)
- Use the web interface to disable the host tests for however long you need when you need to restart a server.
The alarms will still be logged, but won't show up or flow up the chain to bb2.html or whatnot.
Hope that helps.
Tod Hansmann Network Engineer
-----Original Message----- From: Stewart [mailto:stl19847 at yahoo.com] Sent: Tuesday, July 10, 2007 8:45 AM To: hobbit at hswn.dk Subject: [hobbit] Alerts on server reboot?
Is there a way to suppress the flood of alerts we get on a server reboot? If the machine is down for 10 minutes and comes back up, we get flooded with purple messages.
-- Stewart
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Re-read my post and I wasn't clear. This is the Hobbit server going down. Not the client machines.
It seems that when the server starts back up after a reboot, it goes, "My god! I have not seen an update for all 4000+ hosts in over an hour! I should send out purple alerts so that the admin can look into this!"
So the issue is the duration of the Hobbit server downtime. If it's down too long, in its mind, it has not received an update in a very long time, so it sends purple messages for everything it owns.
DOWNTIME won't work, because this is the Hobbit server going down. I also can't disable the hobbit host tests because, well, my server is down. :)
Stewart
Tod Hansmann wrote:
Purple messages on what? Connection alarms should only go red, maybe for a couple polling cycles. Services, I'd imagine, would come back up with the server.
Regardless, you can suppress alarms for a given host and/or service on that host with several ways. The two that come to mind for me are:
- Use the DOWNTIME directive in bb-hosts (see the man-page)
- Use the web interface to disable the host tests for however long you need when you need to restart a server.
The alarms will still be logged, but won't show up or flow up the chain to bb2.html or whatnot.
Hope that helps.
Tod Hansmann Network Engineer
-----Original Message----- From: Stewart [mailto:stl19847 at yahoo.com] Sent: Tuesday, July 10, 2007 8:45 AM To: hobbit at hswn.dk Subject: [hobbit] Alerts on server reboot?
Is there a way to suppress the flood of alerts we get on a server reboot? If the machine is down for 10 minutes and comes back up, we get flooded with purple messages.
-- Stewart
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Stewart Larsen
This sig intentionally left blank, other than this text explaining that if not for this text, this sig would be blank.
Last time I faced this scenario, before I started the Hobbit server I edited hobbit-alerts.cfg and temporarily commented out the MAIL lines :) Of course this doesn't help you if you have hobbit set to automatically start when the server boots. :)
-Charles
Stewart wrote:
Re-read my post and I wasn't clear. This is the Hobbit server going down. Not the client machines.
It seems that when the server starts back up after a reboot, it goes, "My god! I have not seen an update for all 4000+ hosts in over an hour! I should send out purple alerts so that the admin can look into this!"
So the issue is the duration of the Hobbit server downtime. If it's down too long, in its mind, it has not received an update in a very long time, so it sends purple messages for everything it owns.
DOWNTIME won't work, because this is the Hobbit server going down. I also can't disable the hobbit host tests because, well, my server is down. :)
Stewart
Tod Hansmann wrote:
Purple messages on what? Connection alarms should only go red, maybe for a couple polling cycles. Services, I'd imagine, would come back up with the server.
Regardless, you can suppress alarms for a given host and/or service on that host with several ways. The two that come to mind for me are:
- Use the DOWNTIME directive in bb-hosts (see the man-page)
- Use the web interface to disable the host tests for however long you need when you need to restart a server.
The alarms will still be logged, but won't show up or flow up the chain to bb2.html or whatnot.
Hope that helps.
Tod Hansmann Network Engineer
-----Original Message----- From: Stewart [mailto:stl19847 at yahoo.com] Sent: Tuesday, July 10, 2007 8:45 AM To: hobbit at hswn.dk Subject: [hobbit] Alerts on server reboot?
Is there a way to suppress the flood of alerts we get on a server reboot? If the machine is down for 10 minutes and comes back up, we get flooded with purple messages.
-- Stewart
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Stewart Larsen
This sig intentionally left blank, other than this text explaining that if not for this text, this sig would be blank.
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
so, is there nothing I can do about this? I thought I read at one point that there was a way to tell the server, "Don't send any alerts within 10 minutes of server startup", or something like that.
Stewart
Charles Jones wrote:
Last time I faced this scenario, before I started the Hobbit server I edited hobbit-alerts.cfg and temporarily commented out the MAIL lines :) Of course this doesn't help you if you have hobbit set to automatically start when the server boots. :)
-Charles
Stewart wrote:
Re-read my post and I wasn't clear. This is the Hobbit server going down. Not the client machines.
It seems that when the server starts back up after a reboot, it goes, "My god! I have not seen an update for all 4000+ hosts in over an hour! I should send out purple alerts so that the admin can look into this!"
So the issue is the duration of the Hobbit server downtime. If it's down too long, in its mind, it has not received an update in a very long time, so it sends purple messages for everything it owns.
DOWNTIME won't work, because this is the Hobbit server going down. I also can't disable the hobbit host tests because, well, my server is down. :)
Stewart
Tod Hansmann wrote:
Purple messages on what? Connection alarms should only go red, maybe for a couple polling cycles. Services, I'd imagine, would come back up with the server.
Regardless, you can suppress alarms for a given host and/or service on that host with several ways. The two that come to mind for me are:
- Use the DOWNTIME directive in bb-hosts (see the man-page)
- Use the web interface to disable the host tests for however long you need when you need to restart a server.
The alarms will still be logged, but won't show up or flow up the chain to bb2.html or whatnot.
Hope that helps.
Tod Hansmann Network Engineer
-----Original Message----- From: Stewart [mailto:stl19847 at yahoo.com] Sent: Tuesday, July 10, 2007 8:45 AM To: hobbit at hswn.dk Subject: [hobbit] Alerts on server reboot?
Is there a way to suppress the flood of alerts we get on a server reboot? If the machine is down for 10 minutes and comes back up, we get flooded with purple messages.
-- Stewart
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Stewart Larsen
This sig intentionally left blank, other than this text explaining that if not for this text, this sig would be blank.
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Stewart Larsen
This sig intentionally left blank, other than this text explaining that if not for this text, this sig would be blank.
On Tue, Jul 10, 2007 at 08:59:08PM -0400, Stewart wrote:
It seems that when the server starts back up after a reboot, it goes, "My god! I have not seen an update for all 4000+ hosts in over an hour! I should send out purple alerts so that the admin can look into this!"
So the issue is the duration of the Hobbit server downtime. If it's down too long, in its mind, it has not received an update in a very long time, so it sends purple messages for everything it owns.
so, is there nothing I can do about this? I thought I read at one point that there was a way to tell the server, "Don't send any alerts within 10 minutes of server startup", or something like that.
It's built into Hobbit that it won't change a status to purple for the first 10 minutes after it starts up. That should give everything time to refresh before the purple storm hits.
So what you're saying is that this doesn't work - I'll have to try and re-create this here.
Regards, Henrik
If it matters, we're running this in redhat Enterprise.
Henrik Stoerner wrote:
On Tue, Jul 10, 2007 at 08:59:08PM -0400, Stewart wrote:
It seems that when the server starts back up after a reboot, it goes, "My god! I have not seen an update for all 4000+ hosts in over an hour! I should send out purple alerts so that the admin can look into this!"
So the issue is the duration of the Hobbit server downtime. If it's down too long, in its mind, it has not received an update in a very long time, so it sends purple messages for everything it owns. so, is there nothing I can do about this? I thought I read at one point that there was a way to tell the server, "Don't send any alerts within 10 minutes of server startup", or something like that.
It's built into Hobbit that it won't change a status to purple for the first 10 minutes after it starts up. That should give everything time to refresh before the purple storm hits.
So what you're saying is that this doesn't work - I'll have to try and re-create this here.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Stewart Larsen
This sig intentionally left blank, other than this text explaining that if not for this text, this sig would be blank.
You can always turn off hobbitd_alert for a few minutes in hobbitlaunch.cfg.
Thanks, Larry Barber
On 7/11/07, Stewart <stl19847 at yahoo.com> wrote:
If it matters, we're running this in redhat Enterprise.
Henrik Stoerner wrote:
On Tue, Jul 10, 2007 at 08:59:08PM -0400, Stewart wrote:
It seems that when the server starts back up after a reboot, it goes, "My god! I have not seen an update for all 4000+ hosts in over an hour! I should send out purple alerts so that the admin can look into this!"
So the issue is the duration of the Hobbit server downtime. If it's down too long, in its mind, it has not received an update in a very long time, so it sends purple messages for everything it owns. so, is there nothing I can do about this? I thought I read at one point that there was a way to tell the server, "Don't send any alerts within 10 minutes of server startup", or something like that.
It's built into Hobbit that it won't change a status to purple for the first 10 minutes after it starts up. That should give everything time to refresh before the purple storm hits.
So what you're saying is that this doesn't work - I'll have to try and re-create this here.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Stewart Larsen
This sig intentionally left blank, other than this text explaining that if not for this text, this sig would be blank.
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
participants (5)
-
henrik@hswn.dk
-
jonescr@cisco.com
-
lebarber@gmail.com
-
stl19847@yahoo.com
-
thansmann@directpointe.com