Managing who gets alerts - shifts and rotations
We're in the process of migratng from Big Brother to xymon.
I'm wondering what folks are doing about handling on-call paging shifts? We have a complex schedule of on-duty techs plus weekly rotations of oncall people for various teams. We've tried to incorporate the schedules into our rule sets and it is getting too complex to manage.
What are people using to manage this? In my ideal world, all my rules would be "page tech; wait 30 minutes, page on-call" but we've got to get some sort of scheduling software or script to manage who is at the other end of the email.
thanks Betsy
Betsy,
Have you looked into the alerting configs in $SERVER_HOME/etc/hobbit-alerts.cfg yet? It's pretty flexible.
--jms
On Oct 9, 2010, at 7:36 AM, Elizabeth Schwartz wrote:
We're in the process of migratng from Big Brother to xymon.
I'm wondering what folks are doing about handling on-call paging shifts? We have a complex schedule of on-duty techs plus weekly rotations of oncall people for various teams. We've tried to incorporate the schedules into our rule sets and it is getting too complex to manage.
What are people using to manage this? In my ideal world, all my rules would be "page tech; wait 30 minutes, page on-call" but we've got to get some sort of scheduling software or script to manage who is at the other end of the email.
thanks Betsy
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
The native xymon alert config is easier to read than Big Brother but it doesn't free me up from writing elaborate rules. Right now it takes ****25**** rules to cover our shift changes, and that's just for one set of alerts.
Look at Monday mornings. Nadja in Singapore started at 9 pm EST Sunday night and she's covering until 5:00 am EST. Sam in England starts at 4:00 am EST. The US guys start at 8:00 am EST. So I've got a rule from 9:00 pm to midnight Sundays, then 12:00-am-4:00 am Monday, another rule from 4:00am to 5:00 am Monday, and a third rule from 5:00 am to 8:00 am when the US guys start.... it's just endless. There's another set for Tuesday-Thursday and then more for Friday, Saturday and Sunday.
The operations people aren't going to change everyone's schedule just to make configuring the paging alerts easier! Ideally they should be able to enter shifts and coverage changes into some sort of sane interface and have the paging system know what to do.
How are the alert messages actually being sent?? If you're using email (email-to-phone or email-to-sms), could you set up Xymon to email a generic mail alias, then create several /etc/aliases file that get swapped around by cron at shift change and weekends??
Ralph Mitchell
On Sat, Oct 9, 2010 at 8:25 AM, Elizabeth Schwartz <betsy.schwartz at gmail.com
wrote:
The native xymon alert config is easier to read than Big Brother but it doesn't free me up from writing elaborate rules. Right now it takes ****25**** rules to cover our shift changes, and that's just for one set of alerts.
Look at Monday mornings. Nadja in Singapore started at 9 pm EST Sunday night and she's covering until 5:00 am EST. Sam in England starts at 4:00 am EST. The US guys start at 8:00 am EST. So I've got a rule from 9:00 pm to midnight Sundays, then 12:00-am-4:00 am Monday, another rule from 4:00am to 5:00 am Monday, and a third rule from 5:00 am to 8:00 am when the US guys start.... it's just endless. There's another set for Tuesday-Thursday and then more for Friday, Saturday and Sunday.
The operations people aren't going to change everyone's schedule just to make configuring the paging alerts easier! Ideally they should be able to enter shifts and coverage changes into some sort of sane interface and have the paging system know what to do.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
We're using email, plus a script that rotates the weekly oncall alias files. But we don't have a similar setup for the shift changes, and I don't want to be manually managing 7x24 alias files. We can start writing a tool to do that but I was really hoping that such a tool already existed! Seems like we can't possibly be the first company to need this.
(Nagios has more advanced capabilities in this regard but I would rather be using xymon...since we're already all set up with BB)
Always did it by hand. 3 shifts, 22 people, multiple escalations and calling circles. Never needed a tool.
Just always considered it a part of standard admin work.
--jms
On Oct 9, 2010, at 12:48 PM, Elizabeth Schwartz wrote:
We're using email, plus a script that rotates the weekly oncall alias files. But we don't have a similar setup for the shift changes, and I don't want to be manually managing 7x24 alias files. We can start writing a tool to do that but I was really hoping that such a tool already existed! Seems like we can't possibly be the first company to need this.
(Nagios has more advanced capabilities in this regard but I would rather be using xymon...since we're already all set up with BB)
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
On Sat, Oct 9, 2010 at 3:30 PM, Jerald Sheets <questy at gmail.com> wrote:
Always did it by hand. 3 shifts, 22 people, multiple escalations and calling circles. Never needed a tool. Just always considered it a part of standard admin work.
I disagree with this, vehemently. That which is complex, repetitive, and error-prone should be *automated*. As senior professionals it is our job to identify these sorts of operational vulnerabilities and remedy them. After all, that's why we're using monitoring software in the first place, instead of just sitting there reading log files all day.
Reminds me of the T-shirt that says "Go Away or I Will Replace You with a Very Small Shell Script". If I have a task that can be replaced by a shell script, it's wasting my time and my employer's money for me to keep doing it. And I have a long list of better things to do. And if *all* of my tasks can be replaced by automation, well, I probably should find a better job :-)
So, question is how best to automate this, not *whether* I should.
Thanks Betsy
Tell ya what. I'll let my mommy take care of dressing me down and you take care of getting your questions answered.
I'll be sure to include your ideas in the next class I teach.
sheesh<
try to help somebody...
On Oct 10, 2010, at 1:28 PM, Elizabeth Schwartz wrote:
On Sat, Oct 9, 2010 at 3:30 PM, Jerald Sheets <questy at gmail.com> wrote:
Always did it by hand. 3 shifts, 22 people, multiple escalations and calling circles. Never needed a tool. Just always considered it a part of standard admin work.
I disagree with this, vehemently. That which is complex, repetitive, and error-prone should be *automated*. As senior professionals it is our job to identify these sorts of operational vulnerabilities and remedy them. After all, that's why we're using monitoring software in the first place, instead of just sitting there reading log files all day.
Reminds me of the T-shirt that says "Go Away or I Will Replace You with a Very Small Shell Script". If I have a task that can be replaced by a shell script, it's wasting my time and my employer's money for me to keep doing it. And I have a long list of better things to do. And if *all* of my tasks can be replaced by automation, well, I probably should find a better job :-)
So, question is how best to automate this, not *whether* I should.
Thanks Betsy
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
In <099D34E1-EA9E-4E4C-9609-B7F139372370 at gmail.com> Jerald Sheets <questy at gmail.com> writes:
[deleted]
Jerald, this was not an appropriate comment for the list. Please behave yourself when posting publicly.
Regards, Henrik
Hmmm. What an interesting thread this is turning out to be. :-)
With regards the problem, how about this one. Divide and conquer. hobbit-alerts.cfg supports "include" statements. The config also doesn't get grumpy if we have 2 instances of the same server with the same tests and different email alerts. You could then have this in the file include /path/to/alerts-teamX.cfg include /path/to/alerts-teamY.cfg include /path/to/alerts-teamZ.cfg Permissions allow each team to edit their own files, and each team is then responsible for ensuring their shifts are updated correctly.
Each team will then have their own version of hobbit-alerts.cfg, which will then be merged by the includes. This still allows you to use the mechanism Henrik proposed earlier within each team config file, but also gives each team a simpler, team-centric config file.
Of course, this doesn't exclude the possibility of somebody writing a nice editor :-)
Cheers V
my .04 cents:
Henrick: I don't really see any need to change the alerting system. It works fine, it's flexible and highly customizable. The flexibility of the system allows each entity to configure alerts to match their own environment. Complex envirornment or not, it still works well. Some years back when I moved portions of an environment to hobbit from BB I learned to do as Vernon suggests, divide and conquer. The majority of my configurations are in included files. But that was my choice, not some mandated requirement.
Vernon: I mentioned a similar solution on Saturday, it's going to take some work to set it up. I divided it further, my alerts cfg does not change, only the names of the poor sap who gets paged are changed, via cron at their scheduled time or via a CGI that allows someone to push the button and flip alerts to someone else.
Besty: Maybe I'm over-simplifying things but frankly I don't see a need for 186 alerts as you mentioned. Have a look at my mail from Saturday and reconsider. Also understand that manual work needs to be performed to deploy quality systems, standard sysadmin tasks do need to be performed, this includes scripting things that you'll be doing over and over, in cron or from a prompt on a Monday after coffee. Automation is cool, but it's a choice and I respect those that choose either automation or manually logging in and poking around at their systems.
I've deployed hobbit in an environment with _many_ hosts in several locations and several folks that need to be paged when foo happens to host bar at xyz time. There is a constant which is perhaps overlooked in this discussion. The constant is the service foo on host bar. It is always monitored and always triggers an alert. With one alert for service foo on host bar we can have different people paged at different times, it's really not that complicated. If it's not true that you always watch foo:bar then I'm wrong.
Again, remove the people from the main alerts config and put in roles, page-primary, page-secondary, etc. Call these rolls using the SCRIPT function and use perl to change the values on x time. You mention different time zones, changing shifts and so on, this only one variable: time. Time:person = cron/perl. Nothing else needs to change, script the basic alert rules configured in a 24x7 setup and never change the alert rules directly. Just swap the names at x time.
Thanks Henrick and Group for the excellent software and support.
-t
From: Vernon Everett [everett.vernon at gmail.com] Sent: Monday, October 11, 2010 1:16 AM To: xymon at xymon.com Subject: Re: [xymon] Managing who gets alerts - shifts and rotations
Hmmm. What an interesting thread this is turning out to be. :-)
With regards the problem, how about this one. Divide and conquer. hobbit-alerts.cfg supports "include" statements. The config also doesn't get grumpy if we have 2 instances of the same server with the same tests and different email alerts. You could then have this in the file include /path/to/alerts-teamX.cfg include /path/to/alerts-teamY.cfg include /path/to/alerts-teamZ.cfg Permissions allow each team to edit their own files, and each team is then responsible for ensuring their shifts are updated correctly.
Each team will then have their own version of hobbit-alerts.cfg, which will then be merged by the includes. This still allows you to use the mechanism Henrik proposed earlier within each team config file, but also gives each team a simpler, team-centric config file.
Of course, this doesn't exclude the possibility of somebody writing a nice editor :-)
Cheers V
On Oct 11, 2010, at 1:07 PM, Tim McCloskey wrote:
my .04 cents: ... Thanks Henrick and Group for the excellent software and support.
-t
I agree completely. I enormously value this list and those on it. I, as well, have set my hobbit-alerts.cfg once and have never had to touch it again. The wonders of using the OS components such as mail, mailman/majordomo, etc. in well laid-out ways! When someone joins or leaves the team, I delete their account in LDAP, and I never have to touch Xymon. removing their mail alias in /etc/aliases and running newaliases on the Xymon server is all that's required if you want to do it by hand. If you want to automate it, you can get pam_mkhomedir to do it, or you can have your sudoers in LDAP and do it there, or you can have a group in LDAP, and removal of the user constitutes removal from an alias. The possibilities are only limited by your personal imagination as a systems engineer.
My <rather emotionally charged ...apologies> response is precisely because of what you enumerate here. I learn a LOT on this list. I try to help where I can, and I try to give good, sound advice from a career in Systems Administration/Engineering and Architecture spanning more than 20 years. The places where I don't understand things, I try to learn from what I'm reading or ask. What I don't want to happen is for support or the question burden to rise too high. Henrik's time is precious, and of all people I value having him around when really important, very internal questions are asked. He answers them clearly, concisely, and even sometimes with some explanation for us non-developers. I don't want the support burden of this list to ever become too great and risk Henrik having to step back because of the load.
Henrik never had to write Xymon (or Big Brother, for that matter). I hold him in the highest regard for doing so, and I told him that earlier this morning. This is by far the most active/engaged list I am a member of, and am proud to be a user of Xymon.
My thing is that after evaluating Hobbit/Xymon, I learned how it was designed, and modified my usage patterns / system setup to align with how it was designed and have been pleased beyond any purchased item I have ever owned. I don't look at a product and say.. that doesn't work like I want it to, it should change. Instead, I ask myself how my usage patterns need to be modified to mold into the way a product works, or look for ways the OS can manage big portions of my needs (whether alerting, or logging, etc).
I'm sorry if I caused anyone grief. I just don't appreciate being talked down to even if it was Aeleen Frisch. I think we all deerve the same mutual respect around here.
Sorry for my outburst.
--jm
I'm had running hobbit v4.2.3 with devmon v 0.3.0-rc1 for the last 340 days without any big trouble. Today, I had to restart the server and now, the devmon (dm) icon at hobbit server itself turns purple.
If I check all switches, who are monitored with devmon, I can see, it is still working. also at the logs, it looks like it is working.
I have tried to restart devmon manually, but nothing changed. I also tried to use the latest version of devmon (0.3.1) , but it is still the same.
In several old messages of the list, I found similar problems, but no clear solution.
Would be great, if someone have a hint!
Thank you!
Maik
On Tuesday, 12 October 2010 05:25:13 Maik Heinelt wrote:
I'm had running hobbit v4.2.3 with devmon v 0.3.0-rc1 for the last 340 days without any big trouble. Today, I had to restart the server and now, the devmon (dm) icon at hobbit server itself turns purple.
Only the dm test on the devmon polling host? Or, all the devmon-related tests?
If I check all switches, who are monitored with devmon, I can see, it is still working. also at the logs, it looks like it is working.
Please provide some output from the logs. E.g. complete log from starting devmon to end of first poll.
I have tried to restart devmon manually, but nothing changed. I also tried to use the latest version of devmon (0.3.1) , but it is still the same.
There are some improvements in devmon svn, which I mentioned on the devmon list months ago, but which no-one has provided any real feedback on ...
Since I hardly see this myself (it has occurred twice in one year here), I haven't been able to reproduce to test much myself, but simulated problems were handled better.
In several old messages of the list, I found similar problems, but no clear solution.
Would be great, if someone have a hint!
If you eliminate Hobbit/Xymon as the problem, please discuss this on the devmon listed instead.
Regards, Buchan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Please do not thread hijack. When you take a message, remove the contents and change the subject line and send it, it still retains header information that ties it to other messages (and is now interleaving it with an involved discussion about managing shifts).
Please start a new thread. Thanks!
On 10/12/2010 12:25 AM, Maik Heinelt wrote:
I'm had running hobbit v4.2.3 with devmon v 0.3.0-rc1 for the last 340 days without any big trouble. Today, I had to restart the server and now, the devmon (dm) icon at hobbit server itself turns purple.
If I check all switches, who are monitored with devmon, I can see, it is still working. also at the logs, it looks like it is working.
I have tried to restart devmon manually, but nothing changed. I also tried to use the latest version of devmon (0.3.1) , but it is still the same.
In several old messages of the list, I found similar problems, but no clear solution.
Would be great, if someone have a hint!
Thank you!
Maik
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzZ7N4ACgkQmb+gadEcsb7T3QCfQhax8MYy2ddiRaF8f+AU56DX PWgAnitNfeiU73HJknwox46KXQoczmWt =K2YM -----END PGP SIGNATURE-----
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
thanks again Betsy
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote:
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzZ7YwACgkQmb+gadEcsb4WvACfQBmgeDiwZn2yYbobfE/iw4C1 aCEAoNx2OVHoZbLGc4cdpsbJnqSAYNEd =lzVg -----END PGP SIGNATURE-----
Betsy, Ryan
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-) Like you said, it is unlikely you are the first to encounter this issue, but others may have solved it in a less elegant manner.
Regards Vernon
On Wed, Nov 10, 2010 at 8:55 AM, Ryan Novosielski <novosirj at umdnj.edu>wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote:
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzZ7YwACgkQmb+gadEcsb4WvACfQBmgeDiwZn2yYbobfE/iw4C1 aCEAoNx2OVHoZbLGc4cdpsbJnqSAYNEd =lzVg -----END PGP SIGNATURE-----
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
I'll definitely talk to the person who is writing it. We have an ancient rotation script for the weekly rotation (nothing interesting, 10 lines of perl that swaps around a .forward file ) but now we need real scheduling.
On Tue, Nov 9, 2010 at 9:23 PM, Vernon Everett <everett.vernon at gmail.com> wrote:
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
What is the best way to share such a script on this list?
On 11/09/2010 09:23 PM, Vernon Everett wrote:
Betsy, Ryan
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-) Like you said, it is unlikely you are the first to encounter this issue, but others may have solved it in a less elegant manner.
Regards Vernon
On Wed, Nov 10, 2010 at 8:55 AM, Ryan Novosielski <novosirj at umdnj.edu <mailto:novosirj at umdnj.edu>> wrote:
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote:
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com <mailto:xymon-unsubscribe at xymon.com>
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkza8sgACgkQmb+gadEcsb7/IQCg1RnUwZwfynwtOgcq5bEaWy4V dTYAniYzpDY7ZsMWRb6JTC5zJ3uqytqb =Ebb2 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Anyone have an answer to this question? Do I just attach the file?
On 11/10/2010 02:30 PM, Ryan Novosielski wrote:
What is the best way to share such a script on this list?
On 11/09/2010 09:23 PM, Vernon Everett wrote:
Betsy, Ryan
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-) Like you said, it is unlikely you are the first to encounter this issue, but others may have solved it in a less elegant manner.
Regards Vernon
On Wed, Nov 10, 2010 at 8:55 AM, Ryan Novosielski <novosirj at umdnj.edu <mailto:novosirj at umdnj.edu>> wrote:
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote:
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com <mailto:xymon-unsubscribe at xymon.com>
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzdg4oACgkQmb+gadEcsb7pTACePDqZFbaiBxysW9wt1y2xdnFN 6/kAoLgexDxu4FtQH4lU+z/3b6sRK3sk =iOvX -----END PGP SIGNATURE-----
Hi, Ryan !
The usual way is to put such scripts, tools, etc ..
to http://xymonton.trantor.org/doku.php
cheers, martin
On Fri, 12 Nov 2010, Ryan Novosielski wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Anyone have an answer to this question? Do I just attach the file?
On 11/10/2010 02:30 PM, Ryan Novosielski wrote:
What is the best way to share such a script on this list?
On 11/09/2010 09:23 PM, Vernon Everett wrote:
Betsy, Ryan
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-) Like you said, it is unlikely you are the first to encounter this issue, but others may have solved it in a less elegant manner.
Regards Vernon
On Wed, Nov 10, 2010 at 8:55 AM, Ryan Novosielski <novosirj at umdnj.edu <mailto:novosirj at umdnj.edu>> wrote:
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote:
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com <mailto:xymon-unsubscribe at xymon.com>
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzdg4oACgkQmb+gadEcsb7pTACePDqZFbaiBxysW9wt1y2xdnFN 6/kAoLgexDxu4FtQH4lU+z/3b6sRK3sk =iOvX -----END PGP SIGNATURE-----
I'm certainly no authority, but try attaching it! ;) Attachments seem to usually get through (though exes, etc. will probably be blocked). The downside is that they are not archived publicly, but someone who find your message in the archive and was not subscribed at the time can probably subscribe and ask someone for it.
SebA
-----Original Message----- From: Ryan Novosielski [mailto:novosirj at umdnj.edu] Sent: 12 November 2010 18:12 To: xymon at xymon.com Subject: Re: [xymon] Managing who gets alerts - shifts and rotations
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Anyone have an answer to this question? Do I just attach the file?
On 11/10/2010 02:30 PM, Ryan Novosielski wrote:
What is the best way to share such a script on this list?
On 11/09/2010 09:23 PM, Vernon Everett wrote:
Betsy, Ryan
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-) Like you said, it is unlikely you are the first to encounter this issue, but others may have solved it in a less elegant manner.
Regards Vernon
On Wed, Nov 10, 2010 at 8:55 AM, Ryan Novosielski <novosirj at umdnj.edu <mailto:novosirj at umdnj.edu>> wrote:
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote:
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com <mailto:xymon-unsubscribe at xymon.com>
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzdg4oACgkQmb+gadEcsb7pTACePDqZFbaiBxysW9wt1y2xdnFN 6/kAoLgexDxu4FtQH4lU+z/3b6sRK3sk =iOvX -----END PGP SIGNATURE-----
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.449 / Virus Database: 271.1.1/3238 - Release Date: 11/07/10 19:34:00
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
OK, well, I hate to post on Xymonton something that is in such an early/unrefined state (it's not terribly configurable all in one place, so I went through and just edited out the UMDNJ specific stuff). So... maybe a first cut like this will help. Note that the script was written to solve a specific problem, though... our on call list was generated as a Word doc in a certain format. My colleague wrote something that would accept a saved TXT with the same formatting and use that to manage who's on call when. It would obviously make more sense to use a CSV or something like that, but that is not how the author of the schedule does what she does.
Initially, it ran Monday mornings, but that didn't take into account times when someone would swap a day or something. So now we run it at 8:00a everyday.
Anyways, without further ado. Hope it helps someone.
On 11/12/2010 03:10 PM, SebA wrote:
I'm certainly no authority, but try attaching it! ;) Attachments seem to usually get through (though exes, etc. will probably be blocked). The downside is that they are not archived publicly, but someone who find your message in the archive and was not subscribed at the time can probably subscribe and ask someone for it.
SebA
-----Original Message----- From: Ryan Novosielski [mailto:novosirj at umdnj.edu] Sent: 12 November 2010 18:12 To: xymon at xymon.com Subject: Re: [xymon] Managing who gets alerts - shifts and rotations
Anyone have an answer to this question? Do I just attach the file?
On 11/10/2010 02:30 PM, Ryan Novosielski wrote:
What is the best way to share such a script on this list?
On 11/09/2010 09:23 PM, Vernon Everett wrote:
Betsy, Ryan
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-) Like you said, it is unlikely you are the first to encounter this issue, but others may have solved it in a less elegant manner.
Regards Vernon
On Wed, Nov 10, 2010 at 8:55 AM, Ryan Novosielski <novosirj at umdnj.edu <mailto:novosirj at umdnj.edu>> wrote:
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote:
Wanted to say a very belated thanks and I am sorry for sounding rant-y. Should not have ridden that hobby horse through this group. (I have my curmudgeon ribbon, I'm afraid).
After a bit of a delay we're going full speed ahead with our xymon conversion. Our tech support manager is writing an external rotation script to handle our peculiar rotation complexities. Still kinda surprised that we haven't been able to find any code for such a tool around - I keep thinking we can't be the first company to need this - but I'm glad we're going with xymon.
Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com <mailto:xymon-unsubscribe at xymon.com>
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.449 / Virus Database: 271.1.1/3238 - Release Date: 11/07/10 19:34:00
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzdoWcACgkQmb+gadEcsb6fIgCfeuZhsV9+RWgAuJdJK+gwhi3D b/UAnieiRAv1ook0+VO95iDmY7fmsUXc =cVE8 -----END PGP SIGNATURE-----
Maybe we get some space
in the svn-repository for such first-shots ... ?
martin
On Fri, 12 Nov 2010, Ryan Novosielski wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
OK, well, I hate to post on Xymonton something that is in such an early/unrefined state (it's not terribly configurable all in one place, so I went through and just edited out the UMDNJ specific stuff). So... maybe a first cut like this will help. Note that the script was written to solve a specific problem, though... our on call list was generated as a Word doc in a certain format. My colleague wrote something that would accept a saved TXT with the same formatting and use that to manage who's on call when. It would obviously make more sense to use a CSV or something like that, but that is not how the author of the schedule does what she does.
Initially, it ran Monday mornings, but that didn't take into account times when someone would swap a day or something. So now we run it at 8:00a everyday.
Anyways, without further ado. Hope it helps someone.
On 11/12/2010 03:10 PM, SebA wrote:
I'm certainly no authority, but try attaching it! ;) Attachments seem to usually get through (though exes, etc. will probably be blocked). The downside is that they are not archived publicly, but someone who find your message in the archive and was not subscribed at the time can probably subscribe and ask someone for it.
SebA
-----Original Message----- From: Ryan Novosielski [mailto:novosirj at umdnj.edu] Sent: 12 November 2010 18:12 To: xymon at xymon.com Subject: Re: [xymon] Managing who gets alerts - shifts and rotations
Anyone have an answer to this question? Do I just attach the file?
On 11/10/2010 02:30 PM, Ryan Novosielski wrote:
What is the best way to share such a script on this list?
On 11/09/2010 09:23 PM, Vernon Everett wrote:
Betsy, Ryan
Why not post what you have/create, so it is available to all for implementation, and, heaven forbid, improvement. :-) Like you said, it is unlikely you are the first to encounter this issue, but others may have solved it in a less elegant manner.
Regards Vernon
On Wed, Nov 10, 2010 at 8:55 AM, Ryan Novosielski <novosirj at umdnj.edu <mailto:novosirj at umdnj.edu>> wrote:
On 11/09/2010 04:00 PM, Elizabeth Schwartz wrote: > Wanted to say a very belated thanks and I am sorry for sounding > rant-y. Should not have ridden that hobby horse through this group. (I > have my curmudgeon ribbon, I'm afraid).
> After a bit of a delay we're going full speed ahead with our xymon > conversion. Our tech support manager is writing an external rotation > script to handle our peculiar rotation complexities. Still kinda > surprised that we haven't been able to find any code for such a tool > around - I keep thinking we can't be the first company to need this - > but I'm glad we're going with xymon.
> Newbie questions to follow in very short order
A coworker of mine wrote a rotation script to rotate one on call e-mail based on a text file provided with the calendar. This does not help with your multiple times a day rotation, but perhaps it would be of interest? I'm sure he does not wish to hold any rights over it.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com <mailto:xymon-unsubscribe at xymon.com>
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.449 / Virus Database: 271.1.1/3238 - Release Date: 11/07/10 19:34:00
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/CST-Academic Svcs. - ADMC 450, Newark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzdoWcACgkQmb+gadEcsb6fIgCfeuZhsV9+RWgAuJdJK+gwhi3D b/UAnieiRAv1ook0+VO95iDmY7fmsUXc =cVE8 -----END PGP SIGNATURE-----
Gruss
Martin FlemmingMartin Flemming DESY / IT office : Building 2b / 008a Notkestr. 85 phone : 040 - 8998 - 4667 22603 Hamburg mail : martin.flemming at desy.de
As for the other comments on this interesting thread, let's play nicely people.
Many years ago, when the internet was not much more than an embryo, even before the age of GUI, we had newsgroups. Some of you may even remember them. No, not the ones starting alt.binaries.pictures, I mean the other ones, the ones with real information. :-) But I digress. Back to the topic. Back then there were two general rules, both of which apply to this list.
- Rule 1. Do not p!ss people off.
- Rule 2. Do not be easily p!ssed off. I leave it to each of you to decide which rule you transgressed. :-)
Please take mine and Henrik's comments contructively. (See Rule 2) This is a really good mailing list, with good people, and while from time to time we will disagree, from what I have seen, most people on this list are all trying to help each other as best we can. There are no factions, there are no hidden agendas, there are no flame wars. Signal to noise ratio is excellent. Let's stick to the two rules, and keep it this way.
Cheers V
P.S. Betsy, we try not to "vehemently disagree", we rather "prefer to do it differently". :-) (See rules above.)
xymon-unsubscribe at xymon.com
Den Sat, 09 Oct 2010 08:25:55 -0400 skrev Elizabeth Schwartz:
The native xymon alert config is easier to read than Big Brother but it doesn't free me up from writing elaborate rules. Right now it takes ****25**** rules to cover our shift changes, and that's just for one set of alerts.
Look at Monday mornings. Nadja in Singapore started at 9 pm EST Sunday night and she's covering until 5:00 am EST. Sam in England starts at 4:00 am EST. The US guys start at 8:00 am EST. So I've got a rule from 9:00 pm to midnight Sundays, then 12:00-am-4:00 am Monday, another rule from 4:00am to 5:00 am Monday, and a third rule from 5:00 am to 8:00 am when the US guys start.... it's just endless. There's another set for Tuesday-Thursday and then more for Friday, Saturday and Sunday.
I can understand that your alert-rules get quite complex, but then it *is* a rather complex environment you have.
Now, I don't know how your time-based rules are intertwined with which systems your people manage. But from the very limited description I have, it sounds like you should structure your alert rules around the people who is manning each time period. Does that change a lot ?
$SINGAPORE=nadja $ENGLAND=sam $US=phil,dan,tom,joe
Something like this is what I'd suggest:
TIME=0:2100:0500 MAIL $SINGAPORE
TIME=0400:1200 MAIL $ENGLAND
TIME=0800:1600 MAIL $US
If you need to distribute alerts further - say, each of the "USguys" have different groups of servers each - you can add this as an extra condition on the MAIL alert, like
TIME=0800:1600 MAIL joe HOST=server1,server2 MAIL tom PAGE=california MAIL $US UNMATCHED
Then Joe would get alerts for those two servers only, Tom would get alerts for the California servers (if they're on one page called "california"), and "usguys" would get all those that didn't go to Joe or Tom.
The conditions can of course also relate to specific services, not just hosts.
Does that help ?
I'd be interested to hear suggestions for a better way of configuring your alerts. It's done the way it is because it seemed flexible enough to handle my needs, and it was much easier to understand than the BB setup (I could never quite figure out how the advanced parts of the BB alert setup worked). But that doesn't mean it is "set in stone" for ever. I am quite open to suggestions on how it can be improved.
Regards, Henrik
We have people in five time zones reporting to three different supervisors/team leads, with shifts staggered for maximal coverage. Today we have 25 permutations of folks on duty, and the company is growing fast, so the complexity is increasing. We need some sort of way to allow the team leads to enter shift descriptions, that we can keep separate from the paging rule sets. We've also got many hundreds of servers, and people are screaming for more sophisticated rule set management, but with BB we've got to duplicate 25 shift rules for each new change - and then modify each set whenever anyone moves around. Xymon makes the rules easier to *read* but doesn't lower the count.
If we have to write this tool, we will. If it lands on my plate, I'll set up a spreadsheet that saves as a CSV and write a perl program to parse the spreadsheet (if I fail to enlist someone who can do a GUI). But I'm still really hoping that someone somewhere has already done this! We can't possibly be the first to need this. And I'm a huge believer in trying to find code before starting from scratch.
My ideal program would have a web interface with 24x7 blocks and the ability to put who's on call into each block, allowing for multiple entries. Then the back end would set up the mail aliases or .forward files for tech on duty, oncall, backup (and be smart enough to know that if there's no tech on duty, go straight to the oncall)
thanks again for any thoughts Betsy
$ENGLAND=sam TIME=0400:1200 MAIL $ENGLAND
thanks for the input! Doing it this way makes it a lot easier to read, but it doesn't break into the underlying problem that much, because the shifts are more complicated. Say, John's 9-5 Monday-Friday. Nicks's 7-3 four days and 9-5 Saturday. Larry's 4-12 M-F. So We need a $US Monday 7-9am and Monday 9am-noon and Monday noon-3:00 and so on... I can imagine the day when we have 168 rule sets.
If then else statements? On Oct 9, 2010 9:52 PM, "Elizabeth Schwartz" <betsy.schwartz at gmail.com> wrote:
$ENGLAND=sam TIME=0400:1200 MAIL $ENGLAND
thanks for the input! Doing it this way makes it a lot easier to read, but it doesn't break into the underlying problem that much, because the shifts are more complicated. Say, John's 9-5 Monday-Friday. Nicks's 7-3 four days and 9-5 Saturday. Larry's 4-12 M-F. So We need a $US Monday 7-9am and Monday 9am-noon and Monday noon-3:00 and so on... I can imagine the day when we have 168 rule sets.
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
Hi,
Might not be what you were hoping to hear but I'm going to share it just the same.
If you think through your rules and come up with a standard format it will help. I know it seems endless but once you've set up a standard it's not as bad as you think. Just tedious in the initial setup, but worth it in the long run.
One of the things that I did was to seperate the email/pager addresses from the actual alert rules. Example below.
On NN schedule cron runs a simple perl pie script to change the values in mail-primary.sh (and page-primary.sh, etc). This is an extra layer that you need to come up with yourself but it's not overly complex. You may end up with 4 or nn more perl scripts but you maintain the variables in those scripts outside of the hobbit system, thus avoiding the typo in your alerts.cfg that breaks things. (The variables will likely be a list or array of people/pagers/email. It's likely that the people will change from time to time but the alert for ICMP on your nameserver will always be required.)
cat hobbit-alerts.cfg ... $alertdir=/usr/local/tolkien/server/alert-scripts/sys_admin $alertdir2=/usr/local/tolkien/server/alert-scripts/dev_app include /usr/local/tolkien/server/etc/inc/alerts/tm-mu ...
cat tm-mu ...
ALL OTHER SERVICES : SERVERS ON WEB SIDE
mail primary on every red level service failure that has been red for over 6 minutes.
send mail once an hour and do not send a recovery email.
this excludes conn, for which they have already been paged.
mail secondary after an hour and once an hour thereafter.
PAGE=%^web/(linux|other|windows|solaris) EXSERVICE=conn COLOR=red,purple DURATION>6m SCRIPT $alertdir/tm-mu/mail-primary.sh mail-web-prim FORMAT=sms REPEAT=1h
PAGE=%^web/(linux|other|windows|solaris) EXSERVICE=conn COLOR=red,purple DURATION>1h SCRIPT $alertdir/tm-mu/mail-secondary.sh mail-web-sec FORMAT=sms REPEAT=1h ...
cat mail-primary.sh:
!/bin/bash /bin/mail -s "$BBHOSTSVC" tm at f...redacted....com < /dev/null
I can't share the complete perl script today but it's failry simple, example stanza.
... elif $GREP $mailb $wd/$mp > /dev/null
then
$PERL -p -i -e "s:$mailb:$maila:" $wd/$mp
$PERL -p -i -e "s:$maila:$mailb:" $wd/$ms
else `$MAIL -s 'failed on call change' $sysadmin < $wd/$msg`fi ...
I know it seems like you're banging your head on the wall looking for simplicity, that's the part you may need to create. If something already exists I'm sure someone on the group will let you know.
Good luck.
-t
I forgot to mention that using the SCRIPT directive like I did will cause the 'info' tab to be full of alerting info that is not really helpful, depending on how tou view that sort of thing. A workaround for me was to just take advantage of the 'notes' dir.
From: Tim McCloskey [tm at freedom.com] Sent: Saturday, October 09, 2010 12:28 PM To: xymon at xymon.com Subject: RE: [xymon] Managing who gets alerts - shifts and rotations
Hi,
Might not be what you were hoping to hear but I'm going to share it just the same.
If you think through your rules and come up with a standard format it will help. I know it seems endless but once you've set up a standard it's not as bad as you think. Just tedious in the initial setup, but worth it in the long run.
One of the things that I did was to seperate the email/pager addresses from the actual alert rules. Example below.
On NN schedule cron runs a simple perl pie script to change the values in mail-primary.sh (and page-primary.sh, etc). This is an extra layer that you need to come up with yourself but it's not overly complex. You may end up with 4 or nn more perl scripts but you maintain the variables in those scripts outside of the hobbit system, thus avoiding the typo in your alerts.cfg that breaks things. (The variables will likely be a list or array of people/pagers/email. It's likely that the people will change from time to time but the alert for ICMP on your nameserver will always be required.)
cat hobbit-alerts.cfg ... $alertdir=/usr/local/tolkien/server/alert-scripts/sys_admin $alertdir2=/usr/local/tolkien/server/alert-scripts/dev_app include /usr/local/tolkien/server/etc/inc/alerts/tm-mu ...
cat tm-mu ...
ALL OTHER SERVICES : SERVERS ON WEB SIDE
mail primary on every red level service failure that has been red for over 6 minutes.
send mail once an hour and do not send a recovery email.
this excludes conn, for which they have already been paged.
mail secondary after an hour and once an hour thereafter.
PAGE=%^web/(linux|other|windows|solaris) EXSERVICE=conn COLOR=red,purple DURATION>6m SCRIPT $alertdir/tm-mu/mail-primary.sh mail-web-prim FORMAT=sms REPEAT=1h
PAGE=%^web/(linux|other|windows|solaris) EXSERVICE=conn COLOR=red,purple DURATION>1h SCRIPT $alertdir/tm-mu/mail-secondary.sh mail-web-sec FORMAT=sms REPEAT=1h ...
cat mail-primary.sh:
!/bin/bash /bin/mail -s "$BBHOSTSVC" tm at f...redacted....com < /dev/null
I can't share the complete perl script today but it's failry simple, example stanza.
... elif $GREP $mailb $wd/$mp > /dev/null
then
$PERL -p -i -e "s:$mailb:$maila:" $wd/$mp
$PERL -p -i -e "s:$maila:$mailb:" $wd/$ms
else `$MAIL -s 'failed on call change' $sysadmin < $wd/$msg`fi ...
I know it seems like you're banging your head on the wall looking for simplicity, that's the part you may need to create. If something already exists I'm sure someone on the group will let you know.
Good luck.
-t
participants (12)
-
betsy.schwartz＠gmail.com
-
bgmilne＠staff.telkomsa.net
-
everett.vernon＠gmail.com
-
henrik＠hswn.dk
-
josh＠imaginenetworksllc.com
-
maik＠vegasystems.com
-
martin.flemming＠desy.de
-
novosirj＠umdnj.edu
-
questy＠gmail.com
-
ralphmitchell＠gmail.com
-
spa＠syntec.co.uk
-
tm＠freedom.com