[hobbit] Is there a way to "quietly" disable hosts that have NOTICE set?
Directly from the man bb command :D
disable HOSTNAME.TESTNAME DURATION <additional text>
Disables a specific test for DURATION minutes. This
will cause the status of this test to be listed as "blue" on the BBDISPLAY server, and no alerts for this host/test will be generated. If DURATION is given as a number followed by s/m/h/d, it is interpreted as being in seconds/minutes/hours/days respectively. To disable all tests for a host, use an asterisk "*" for TESTNAME.
enable HOSTNAME.TESTNAME
Re-enables a test that had been disabled.
So you can execute bb hobbitserver "disable dbservername.* 2h Backup time" before shutting down the database and bb hobbitserver "enable dbserver.*" just at the end.
Francesco
-----Original Message----- From: Charles Jones [mailto:jonescr at cisco.com] Sent: Monday, September 25, 2006 7:56 AM To: hobbit at hswn.dk Subject: [hobbit] Is there a way to "quietly" disable hosts that have NOTICE set?
I have the NOTICE flag set for all of my production hosts - I want pages to go out if someone disables or enables any of them.
However, when backups are done, the oracle databases are brought down, which triggers an alert. If they are manually disabled, the NOTICE message goes out which also wakes people up for no reason.
If I use DOWNTIME in bb-hosts, then I have to specify a window which is guaranteed to be longer than the possible time it could take to backup the databases (which is a dynamic thing which will surely be wrong from time to time). So what ends up happening is for example, I would specify an hour of DOWNTIME, but the backups sometimes only take 30 minutes. That means there is a 30 minute window where a real alert would be masked, which is unacceptable in a production environment.
I guess what I'm looking for, is a way that I can send a commands to Hobbit via a shellscript (called from the db backup script), that would put a host/services in maint mode (disabled - blue dot), and NOT send a NOTICE page.
-Charles
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
That sends out NOTICE alerts though. If you are on call, do you want to get paged at 3am every night when those services are put into MAINT mode? Thats why I asked if there was a way to do a "silent" disable. I basically need something that works exactly like the DOWNTIME option in bb-hosts, except interactively.
-Charles
Francesco Duranti wrote:
Directly from the man bb command :D
disable HOSTNAME.TESTNAME DURATION <additional text> Disables a specific test for DURATION minutes. This will cause the status of this test to be listed as "blue" on the BBDISPLAY server, and no alerts for this host/test will be generated. If DURATION is given as a number followed by s/m/h/d, it is interpreted as being in seconds/minutes/hours/days respectively. To disable all tests for a host, use an asterisk "*" for TESTNAME.
enable HOSTNAME.TESTNAME Re-enables a test that had been disabled.So you can execute bb hobbitserver "disable dbservername.* 2h Backup time" before shutting down the database and bb hobbitserver "enable dbserver.*" just at the end.
Francesco
-----Original Message----- From: Charles Jones [mailto:jonescr at cisco.com] Sent: Monday, September 25, 2006 7:56 AM To: hobbit at hswn.dk Subject: [hobbit] Is there a way to "quietly" disable hosts that have NOTICE set?
I have the NOTICE flag set for all of my production hosts - I want pages to go out if someone disables or enables any of them.
However, when backups are done, the oracle databases are brought down, which triggers an alert. If they are manually disabled, the NOTICE message goes out which also wakes people up for no reason.
If I use DOWNTIME in bb-hosts, then I have to specify a window which is guaranteed to be longer than the possible time it could take to backup the databases (which is a dynamic thing which will surely be wrong from time to time). So what ends up happening is for example, I would specify an hour of DOWNTIME, but the backups sometimes only take 30 minutes. That means there is a 30 minute window where a real alert would be masked, which is unacceptable in a production environment.
I guess what I'm looking for, is a way that I can send a commands to Hobbit via a shellscript (called from the db backup script), that would put a host/services in maint mode (disabled - blue dot), and NOT send a NOTICE page.
-Charles
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
we have put in the following because every night we have batch jobs that run that use almost all the processor power in our server:
SERVICE=cpu MAIL [e-mail address] COLOR=red DURATION=15 TIME=*:0900:1700 HOST=[hostname]
by changing the TIME field to suit you, you will not get paiged, you just have to put an EXHOST=[hostname] in your normal alert rules.
Adam
From: Charles Jones Sent: Monday, September 25, 2006 7:12 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Is there a way to "quietly" disable hosts that have NOTICE set?
That sends out NOTICE alerts though. If you are on call, do you want to get paged at 3am every night when those services are put into MAINT mode? Thats why I asked if there was a way to do a "silent" disable. I basically need something that works exactly like the DOWNTIME option in bb-hosts, except interactively.
-Charles
Francesco Duranti wrote:
Directly from the man bb command :D
disable HOSTNAME.TESTNAME DURATION <additional text> Disables a specific test for DURATION minutes. Thiswill cause the status of this test to be listed as "blue" on the BBDISPLAY server, and no alerts for this host/test will be generated. If DURATION is given as a number followed by s/m/h/d, it is interpreted as being in seconds/minutes/hours/days respectively. To disable all tests for a host, use an asterisk "*" for TESTNAME.
enable HOSTNAME.TESTNAME Re-enables a test that had been disabled.So you can execute bb hobbitserver "disable dbservername.* 2h Backup time" before shutting down the database and bb hobbitserver "enable dbserver.*" just at the end.
Francesco
-----Original Message----- From: Charles Jones [mailto:jonescr at cisco.com] Sent: Monday, September 25, 2006 7:56 AM To: hobbit at hswn.dk Subject: [hobbit] Is there a way to "quietly" disable hosts that have NOTICE set?
I have the NOTICE flag set for all of my production hosts - I want pages to go out if someone disables or enables any of them.
However, when backups are done, the oracle databases are brought down, which triggers an alert. If they are manually disabled, the NOTICE message goes out which also wakes people up for no reason.
If I use DOWNTIME in bb-hosts, then I have to specify a window which is guaranteed to be longer than the possible time it could take to backup the databases (which is a dynamic thing which will surely be wrong from time to time). So what ends up happening is for example, I would specify an hour of DOWNTIME, but the backups sometimes only take 30 minutes. That means there is a 30 minute window where a real alert would be masked, which is unacceptable in a production environment.
I guess what I'm looking for, is a way that I can send a commands to Hobbit via a shellscript (called from the db backup script), that would put a host/services in maint mode (disabled - blue dot), and NOT send a NOTICE page.
-Charles
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Okay, let me explain this again :)
If I ignore pages for a specific time interval, say 2 hours while backups are being done, then every minute that the backups complete before that interval is over, is a minute that the database is essentially not monitored, which is not acceptable.
Example:
via DOWNTIME in bb-hosts, or via TIME specifications in hobbit-alerts.cfg as you mention, [----------window where no pages are sent for red status of "oracle" or "procs"------------] [-----------------database backups--------------][------ something can break here -------]
My problem is that something can break during that post-backup interval, and nobody will be notified. Making the interval shorter is not an option because due to the nature of backups, the amount of time it takes to do the backup fluctuates on a daily basis. It could take 30 minutes monday night, an hour tuesday night, 10 minutes wednesday night, 73 minutes thurs.. Using the bb disable command from a script is not a good solution because that makes NOTICE alerts get sent out, which wakes people up for no reason.
The DOWNTIME option in bb-hosts has the ability to set the status of services blue without sending NOTICE alerts. I would like this same ability on an interactive basis (via a message to hobbit using the bb command).
-Charles
Scheblein, Adam wrote:
we have put in the following because every night we have batch jobs that run that use almost all the processor power in our server:
SERVICE=cpu MAIL [e-mail address] COLOR=red DURATION=15 TIME=*:0900:1700 HOST=[hostname]
by changing the TIME field to suit you, you will not get paiged, you just have to put an EXHOST=[hostname] in your normal alert rules.
Adam
From: Charles Jones Sent: Monday, September 25, 2006 7:12 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Is there a way to "quietly" disable hosts that have NOTICE set?
That sends out NOTICE alerts though. If you are on call, do you want to get paged at 3am every night when those services are put into MAINT mode? Thats why I asked if there was a way to do a "silent" disable. I basically need something that works exactly like the DOWNTIME option in bb-hosts, except interactively.
-Charles
Francesco Duranti wrote:
Directly from the man bb command :D
disable HOSTNAME.TESTNAME DURATION <additional text> Disables a specific test for DURATION minutes. Thiswill cause the status of this test to be listed as "blue" on the BBDISPLAY server, and no alerts for this host/test will be generated. If DURATION is given as a number followed by s/m/h/d, it is interpreted as being in seconds/minutes/hours/days respectively. To disable all tests for a host, use an asterisk "*" for TESTNAME.
enable HOSTNAME.TESTNAME Re-enables a test that had been disabled.So you can execute bb hobbitserver "disable dbservername.* 2h Backup time" before shutting down the database and bb hobbitserver "enable dbserver.*" just at the end.
Francesco
-----Original Message----- From: Charles Jones [mailto:jonescr at cisco.com] Sent: Monday, September 25, 2006 7:56 AM To: hobbit at hswn.dk Subject: [hobbit] Is there a way to "quietly" disable hosts that have NOTICE set?
I have the NOTICE flag set for all of my production hosts - I want pages to go out if someone disables or enables any of them.
However, when backups are done, the oracle databases are brought down, which triggers an alert. If they are manually disabled, the NOTICE message goes out which also wakes people up for no reason.
If I use DOWNTIME in bb-hosts, then I have to specify a window which is guaranteed to be longer than the possible time it could take to backup the databases (which is a dynamic thing which will surely be wrong from time to time). So what ends up happening is for example, I would specify an hour of DOWNTIME, but the backups sometimes only take 30 minutes. That means there is a 30 minute window where a real alert would be masked, which is unacceptable in a production environment.
I guess what I'm looking for, is a way that I can send a commands to Hobbit via a shellscript (called from the db backup script), that would put a host/services in maint mode (disabled - blue dot), and NOT send a NOTICE page.
-Charles
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On 9/25/06, Charles Jones <jonescr at cisco.com> wrote:
The DOWNTIME option in bb-hosts has the ability to set the status of services blue without sending NOTICE alerts. I would like this same ability on an interactive basis (via a message to hobbit using the bb command).
So, essentially you need DOWNTIME to have something like the "until green again" option that was added to the enable/disable page in 4.2, right??
Unless I've completely missed your point, for which I apologise in advance...:)
Ralph Mitchell
Yep...I tried doing a disable for -1 (until OK), but if you do that while the service is still green, it immediately recovers.
Ralph Mitchell wrote:
On 9/25/06, Charles Jones <jonescr at cisco.com> wrote:
The DOWNTIME option in bb-hosts has the ability to set the status of services blue without sending NOTICE alerts. I would like this same ability on an interactive basis (via a message to hobbit using the bb command).
So, essentially you need DOWNTIME to have something like the "until green again" option that was added to the enable/disable page in 4.2, right??
Unless I've completely missed your point, for which I apologise in advance...:)
Ralph Mitchell
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
participants (4)
-
adam.scheblein@marquette.edu
-
fduranti@q8.it
-
jonescr@cisco.com
-
ralphmitchell@gmail.com