Perhaps it's because I'm working on this at 4am, but I'm having a problem with the EXHOST option, that according to hobbitd_alert --test isn't working, I also am not sure how to do a particular host/service exclusion.
Heres basically what my below alert config is meant to accomplish.
- For any alerts on any servers, send alerts to an alert email address.
- For 2 particular web servers (web5.mydomain.com and web6.mydomain.com), send an alert to one person, but *not *the alert alias.
- For a set of oracle servers, send an extra alert message to an alternate email address/cellphone.
- After hours (from 5pm until 8am), only send alerts to an alternate email address (but still need the seperate alert for the web5 and web6 hosts described in #2).
- After hours (from 5pm until 8am), send an alert to my cellphone for any hosts and services being red for more than 30 mins.
- Do not alert for high load average on a particular server from 6-10am.
My first problem is I am not sure how to implement #2 (exclude alerts for the msgs on web5 and web6 from being sent to the main alert email, and instead send them to the alternate address). I'm thinking one solution would be to define it as the very first rule and use the "STOP" option, like: HOST=$WEB_SERVERS SERVICE=msgs COLOR=red MAIL webdev at mydomain.com STOP
Also, according to the tests I did with hobbit_alert --test, the last rule is not working...Is there a more logical way to simply specify that a single host/service combo be ignored during a certain timeframe? My hobbit-alerts.cfg is below, if you see anything wrong or have suggestions on a better way to accomplish my list above I would appreciate it.
hobbitd-alerts.cfg
oradb1-9.mydomain.com
$ORACLE_SERVERS=%oradb(.).mydomain.com
web1-9.mydomain.com
$WEB_SERVERS=%web(.).mydomain.com
All hosts
$ALL_HOSTS=*
Send me an email alert if any service on any host goes red.
Note: This rule will probably be removed once the alert rules are
deemed fully working. HOST=$ALL_HOSTS SERVICE=* COLOR=red MAIL alert at mydomain.com
Notify webdev about (only) web errors on web5 and web6
FIXME: Need to make it so that alert at mydomain.com does NOT get these!
HOST=$WEB_SERVERS SERVICE=msgs COLOR=red MAIL webdev at mydomain.com
Send an alert to dba phone on oracle-specific problems
HOST=$ORACLE_SERVERS SERVICE=msgs,oradb,orasys COLOR=red FORMAT=sms MAIL dbacell at cellphone.com
Send me a page if any hosts go red for more than 30 minutes during
offhours HOST=$ALL_HOSTS SERVICE=* COLOR=red DURATION>30 TIME=1700-0800 FORMAT=sms MAIL mycell at cellphone.com
Ignore high load average warnings for dataproc1 in the mornings
EXHOST=dataproc1.mydomain.com SERVICE=cpu COLOR=red TIME=0600-1000 MAIL me at mydomain.com