The native xymon alert config is easier to read than Big Brother but it doesn't free me up from writing elaborate rules. Right now it takes ****25**** rules to cover our shift changes, and that's just for one set of alerts.

Look at Monday mornings. Nadja in Singapore started at 9 pm EST Sunday night and she's covering until 5:00 am EST. Sam in England starts at 4:00 am EST. The US guys start at 8:00 am EST. So I've got a rule from 9:00 pm to midnight Sundays, then 12:00-am-4:00 am Monday, another rule from 4:00am to 5:00 am Monday, and a third rule from 5:00 am to 8:00 am when the US guys start.... it's just endless. There's another set for Tuesday-Thursday and then more for Friday, Saturday and Sunday.

The operations people aren't going to change everyone's schedule just to make configuring the paging alerts easier! Ideally they should be able to enter shifts and coverage changes into some sort of sane interface and have the paging system know what to do.

We're using email, plus a script that rotates the weekly oncall alias files. But we don't have a similar setup for the shift changes, and I don't want to be manually managing 7x24 alias files. We can start writing a tool to do that but I was really hoping that such a tool already existed! Seems like we can't possibly be the first company to need this.

(Nagios has more advanced capabilities in this regard but I would rather be using xymon...since we're already all set up with BB)

On Sat, Oct 9, 2010 at 3:30 PM, Jerald Sheets <questy at gmail.com> wrote:

Always did it by hand.  3 shifts, 22 people, multiple escalations and calling circles.  Never needed a tool. Just always considered it a part of standard admin work.

I disagree with this, vehemently. That which is complex, repetitive, and error-prone should be *automated*. As senior professionals it is our job to identify these sorts of operational vulnerabilities and remedy them. After all, that's why we're using monitoring software in the first place, instead of just sitting there reading log files all day.

Reminds me of the T-shirt that says "Go Away or I Will Replace You with a Very Small Shell Script". If I have a task that can be replaced by a shell script, it's wasting my time and my employer's money for me to keep doing it. And I have a long list of better things to do. And if *all* of my tasks can be replaced by automation, well, I probably should find a better job :-)

So, question is how best to automate this, not *whether* I should.

Thanks Betsy

In <099D34E1-EA9E-4E4C-9609-B7F139372370 at gmail.com> Jerald Sheets <questy at gmail.com> writes:

[deleted]

Jerald, this was not an appropriate comment for the list. Please behave yourself when posting publicly.

Regards, Henrik

my .04 cents:

1. Henrick: I don't really see any need to change the alerting system. It works fine, it's flexible and highly customizable. The flexibility of the system allows each entity to configure alerts to match their own environment. Complex envirornment or not, it still works well. Some years back when I moved portions of an environment to hobbit from BB I learned to do as Vernon suggests, divide and conquer. The majority of my configurations are in included files. But that was my choice, not some mandated requirement.

2. Vernon: I mentioned a similar solution on Saturday, it's going to take some work to set it up. I divided it further, my alerts cfg does not change, only the names of the poor sap who gets paged are changed, via cron at their scheduled time or via a CGI that allows someone to push the button and flip alerts to someone else.

3. Besty: Maybe I'm over-simplifying things but frankly I don't see a need for 186 alerts as you mentioned. Have a look at my mail from Saturday and reconsider. Also understand that manual work needs to be performed to deploy quality systems, standard sysadmin tasks do need to be performed, this includes scripting things that you'll be doing over and over, in cron or from a prompt on a Monday after coffee. Automation is cool, but it's a choice and I respect those that choose either automation or manually logging in and poking around at their systems.

I've deployed hobbit in an environment with _many_ hosts in several locations and several folks that need to be paged when foo happens to host bar at xyz time. There is a constant which is perhaps overlooked in this discussion. The constant is the service foo on host bar. It is always monitored and always triggers an alert. With one alert for service foo on host bar we can have different people paged at different times, it's really not that complicated. If it's not true that you always watch foo:bar then I'm wrong.

Again, remove the people from the main alerts config and put in roles, page-primary, page-secondary, etc. Call these rolls using the SCRIPT function and use perl to change the values on x time. You mention different time zones, changing shifts and so on, this only one variable: time. Time:person = cron/perl. Nothing else needs to change, script the basic alert rules configured in a 24x7 setup and never change the alert rules directly. Just swap the names at x time.

Thanks Henrick and Group for the excellent software and support.

-t

From: Vernon Everett [everett.vernon at gmail.com] Sent: Monday, October 11, 2010 1:16 AM To: xymon at xymon.com Subject: Re: [xymon] Managing who gets alerts - shifts and rotations

Hmmm. What an interesting thread this is turning out to be. :-)

With regards the problem, how about this one. Divide and conquer. hobbit-alerts.cfg supports "include" statements. The config also doesn't get grumpy if we have 2 instances of the same server with the same tests and different email alerts. You could then have this in the file include /path/to/alerts-teamX.cfg include /path/to/alerts-teamY.cfg include /path/to/alerts-teamZ.cfg Permissions allow each team to edit their own files, and each team is then responsible for ensuring their shifts are updated correctly.

Each team will then have their own version of hobbit-alerts.cfg, which will then be merged by the includes. This still allows you to use the mechanism Henrik proposed earlier within each team config file, but also gives each team a simpler, team-centric config file.

Of course, this doesn't exclude the possibility of somebody writing a nice editor :-)

Cheers V

I'm had running hobbit v4.2.3 with devmon v 0.3.0-rc1 for the last 340 days without any big trouble. Today, I had to restart the server and now, the devmon (dm) icon at hobbit server itself turns purple.

If I check all switches, who are monitored with devmon, I can see, it is still working. also at the logs, it looks like it is working.

I have tried to restart devmon manually, but nothing changed. I also tried to use the latest version of devmon (0.3.1) , but it is still the same.

In several old messages of the list, I found similar problems, but no clear solution.

Would be great, if someone have a hint!

Thank you!

Maik

As for the other comments on this interesting thread, let's play nicely people.

Many years ago, when the internet was not much more than an embryo, even before the age of GUI, we had newsgroups. Some of you may even remember them. No, not the ones starting alt.binaries.pictures, I mean the other ones, the ones with real information. :-) But I digress. Back to the topic. Back then there were two general rules, both of which apply to this list.

• Rule 1. Do not p!ss people off.
• Rule 2. Do not be easily p!ssed off. I leave it to each of you to decide which rule you transgressed. :-)

Please take mine and Henrik's comments contructively. (See Rule 2) This is a really good mailing list, with good people, and while from time to time we will disagree, from what I have seen, most people on this list are all trying to help each other as best we can. There are no factions, there are no hidden agendas, there are no flame wars. Signal to noise ratio is excellent. Let's stick to the two rules, and keep it this way.

Cheers V

P.S. Betsy, we try not to "vehemently disagree", we rather "prefer to do it differently". :-) (See rules above.)

xymon-unsubscribe at xymon.com

We have people in five time zones reporting to three different supervisors/team leads, with shifts staggered for maximal coverage. Today we have 25 permutations of folks on duty, and the company is growing fast, so the complexity is increasing. We need some sort of way to allow the team leads to enter shift descriptions, that we can keep separate from the paging rule sets. We've also got many hundreds of servers, and people are screaming for more sophisticated rule set management, but with BB we've got to duplicate 25 shift rules for each new change - and then modify each set whenever anyone moves around. Xymon makes the rules easier to *read* but doesn't lower the count.

If we have to write this tool, we will. If it lands on my plate, I'll set up a spreadsheet that saves as a CSV and write a perl program to parse the spreadsheet (if I fail to enlist someone who can do a GUI). But I'm still really hoping that someone somewhere has already done this! We can't possibly be the first to need this. And I'm a huge believer in trying to find code before starting from scratch.

My ideal program would have a web interface with 24x7 blocks and the ability to put who's on call into each block, allowing for multiple entries. Then the back end would set up the mail aliases or .forward files for tech on duty, oncall, backup (and be smart enough to know that if there's no tech on duty, go straight to the oncall)

thanks again for any thoughts Betsy

If then else statements? On Oct 9, 2010 9:52 PM, "Elizabeth Schwartz" <betsy.schwartz at gmail.com> wrote:

...

...

$ENGLAND=sam TIME=0400:1200 MAIL $ENGLAND

thanks for the input! Doing it this way makes it a lot easier to read, but it doesn't break into the underlying problem that much, because the shifts are more complicated. Say, John's 9-5 Monday-Friday. Nicks's 7-3 four days and 9-5 Saturday. Larry's 4-12 M-F. So We need a $US Monday 7-9am and Monday 9am-noon and Monday noon-3:00 and so on... I can imagine the day when we have 168 rule sets.

To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com

I forgot to mention that using the SCRIPT directive like I did will cause the 'info' tab to be full of alerting info that is not really helpful, depending on how tou view that sort of thing. A workaround for me was to just take advantage of the 'notes' dir.

From: Tim McCloskey [tm at freedom.com] Sent: Saturday, October 09, 2010 12:28 PM To: xymon at xymon.com Subject: RE: [xymon] Managing who gets alerts - shifts and rotations

Hi,

Might not be what you were hoping to hear but I'm going to share it just the same.

If you think through your rules and come up with a standard format it will help. I know it seems endless but once you've set up a standard it's not as bad as you think. Just tedious in the initial setup, but worth it in the long run.

One of the things that I did was to seperate the email/pager addresses from the actual alert rules. Example below.

On NN schedule cron runs a simple perl pie script to change the values in mail-primary.sh (and page-primary.sh, etc). This is an extra layer that you need to come up with yourself but it's not overly complex. You may end up with 4 or nn more perl scripts but you maintain the variables in those scripts outside of the hobbit system, thus avoiding the typo in your alerts.cfg that breaks things. (The variables will likely be a list or array of people/pagers/email. It's likely that the people will change from time to time but the alert for ICMP on your nameserver will always be required.)

cat hobbit-alerts.cfg ... $alertdir=/usr/local/tolkien/server/alert-scripts/sys_admin $alertdir2=/usr/local/tolkien/server/alert-scripts/dev_app include /usr/local/tolkien/server/etc/inc/alerts/tm-mu ...

cat tm-mu ...

ALL OTHER SERVICES : SERVERS ON WEB SIDE

mail primary on every red level service failure that has been red for over 6 minutes.

send mail once an hour and do not send a recovery email.

this excludes conn, for which they have already been paged.

mail secondary after an hour and once an hour thereafter.

PAGE=%^web/(linux|other|windows|solaris) EXSERVICE=conn COLOR=red,purple DURATION>6m SCRIPT $alertdir/tm-mu/mail-primary.sh mail-web-prim FORMAT=sms REPEAT=1h

PAGE=%^web/(linux|other|windows|solaris) EXSERVICE=conn COLOR=red,purple DURATION>1h SCRIPT $alertdir/tm-mu/mail-secondary.sh mail-web-sec FORMAT=sms REPEAT=1h ...

cat mail-primary.sh:

!/bin/bash /bin/mail -s "$BBHOSTSVC" tm at f...redacted....com < /dev/null

I can't share the complete perl script today but it's failry simple, example stanza.

... elif $GREP $mailb $wd/$mp > /dev/null

    then
            $PERL -p -i -e "s:$mailb:$maila:" $wd/$mp
            $PERL -p -i -e "s:$maila:$mailb:" $wd/$ms

    else `$MAIL -s 'failed on call change' $sysadmin < $wd/$msg`

fi ...

I know it seems like you're banging your head on the wall looking for simplicity, that's the part you may need to create. If something already exists I'm sure someone on the group will let you know.

Good luck.

-t