Double, triple notifications
For some tests and hosts I receive several notification copies (identical)
- sometimes two, sometimes three. I had reviewed the configuration, but can not detect where is the problem.
How can I debug which rule worked out for each copy?
Feature request - I suppose it should be quite easy to avoid redundant notification copies before sending, for example using cat (rows-list of recepients) | sort | uniq
P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.
Best regards,
Andrey Chervonets
SIA CoMinder http://www.cominder.eu/
Two places to look to troubleshoot are the notifications.log, to check to see whether Xymon actually sent multiples or not, (could be your mail server, theoretically) and the "info" test on the server in question to see what the notifications settings parsed to. You can also look at the config report for that kind of info.
-- ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \\UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \\ and Health | novosirj at rutgers.edu - 973/972.0922 (2x0922) || \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `'
From: Xymon [xymon-bounces at xymon.com] On Behalf Of Andrey Chervonets [A.Chervonets at cominder.eu] Sent: Wednesday, October 01, 2014 8:35 AM To: xymon at xymon.com Subject: [Xymon] Double, triple notifications
For some tests and hosts I receive several notification copies (identical) - sometimes two, sometimes three. I had reviewed the configuration, but can not detect where is the problem.
How can I debug which rule worked out for each copy?
Feature request - I suppose it should be quite easy to avoid redundant notification copies before sending, for example using cat (rows-list of recepients) | sort | uniq
P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.
Best regards,
Andrey Chervonets
SIA CoMinder http://www.cominder.eu/
On 2014-10-01 9:02 am, Novosielski, Ryan wrote:
Two places to look to troubleshoot are the notifications.log, to check to see whether Xymon actually sent multiples or not, (could be your mail server, theoretically) and the "info" test on the server in question to see what the notifications settings parsed to. You can also look at the config report for that kind of info.
-- ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \\UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \\ and Health | novosirj at rutgers.edu - 973/972.0922 (2x0922) || \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `'
From: Xymon [xymon-bounces at xymon.com] On Behalf Of Andrey Chervonets [A.Chervonets at cominder.eu] Sent: Wednesday, October 01, 2014 8:35 AM To: xymon at xymon.com Subject: [Xymon] Double, triple notifications
For some tests and hosts I receive several notification copies (identical) - sometimes two, sometimes three. I had reviewed the configuration, but can not detect where is the problem.
How can I debug which rule worked out for each copy?
Feature request - I suppose it should be quite easy to avoid redundant notification copies before sending, for example using cat (rows-list of recepients) | sort | uniq
P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.
Best regards,
Ryan makes some good points, but, as is the case in my setup, it's more likely that you have multiple rules in your alerts.cfg that send emails for the same alarm, covering multiple sets of servers.
-- Mike Burger http://www.bubbanfriends.org
"It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
Turning on the cfid option in xymond_alert might be useful as well.
-jc
--cfid If this option is present, alert messages will include a line with "cfid:N" where N is the linenumber in the alerts.cfg file that caused this message to be sent. This can be useful to track down problems with duplicate alerts.
On Wed, October 1, 2014 6:02 am, Novosielski, Ryan wrote:
Two places to look to troubleshoot are the notifications.log, to check to see whether Xymon actually sent multiples or not, (could be your mail server, theoretically) and the "info" test on the server in question to see what the notifications settings parsed to. You can also look at the config report for that kind of info.
-- ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \\UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \\ and Health | novosirj at rutgers.edu - 973/972.0922 (2x0922) || \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `'
From: Xymon [xymon-bounces at xymon.com] On Behalf Of Andrey Chervonets [A.Chervonets at cominder.eu] Sent: Wednesday, October 01, 2014 8:35 AM To: xymon at xymon.com Subject: [Xymon] Double, triple notifications
For some tests and hosts I receive several notification copies (identical)
- sometimes two, sometimes three. I had reviewed the configuration, but can not detect where is the problem.
How can I debug which rule worked out for each copy?
Feature request - I suppose it should be quite easy to avoid redundant notification copies before sending, for example using cat (rows-list of recepients) | sort | uniq
P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.
Best regards,
Andrey Chervonets
SIA CoMinder http://www.cominder.eu/
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Andrey,
I'm running on RHEL 6.5 xymon version 4.3.17.
I'm having a similar problem but only on recovery notices. Alert picks up the correct rule and stops. Recover hits that rule then continues on down the list and will then duplicate on my catch all default rule. This isn't on all alert recoveries. Appears to only happen on "server" names that have an underscore or dash in it. I'm just starting to dig in and experiment with it.
One thing that is really helpful in diagnosing alerts.cfg is adding the --cfid option to tasks.cfg xymond_channel.
[alert]
ENVFILE /data/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=page
--log=$XYMONSERVERLOGS/alert.log
xymond_alert
--checkpoint-file=$XYMONTMP/alert.chk
--checkpoint-interval=600
--debug
--cfid
That will put the alerts.cfg line number at the end of the alert email subject line. "cat -n alerts.cfg" gives you a nice display of all line numbers.
Single alert notice.
Xymon [6578] PHX_FIKE:trouble CRITICAL (RED) [cfid:243]
Duplicate recovery notices.
Xymon PHX_FIKE:trouble recovered [cfid:243] Xymon PHX_FIKE:trouble recovered [cfid:432]
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Andrey Chervonets Sent: Wednesday, October 01, 2014 7:36 AM To: xymon at xymon.com Subject: [Xymon] Double, triple notifications
For some tests and hosts I receive several notification copies (identical) - sometimes two, sometimes three. I had reviewed the configuration, but can not detect where is the problem.
How can I debug which rule worked out for each copy?
Feature request - I suppose it should be quite easy to avoid redundant notification copies before sending, for example using cat (rows-list of recepients) | sort | uniq
P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.
Best regards,
Andrey Chervonets
SIA CoMinder http://www.cominder.eu/
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
Thanks a lot! Options: --debug --cfid was very useful. I had found at least one place, which most probably caused the problem.
I would like to explain a bit:
In alerts config. I had used some variables like:
$HOSTS_PROD=host1,host2,host3 $HOSTS_TEST=thost1,thost2 $HOSTS_DEV=dhost1
And some rules, including for $HOSTS_DEV (some separate as well as some together with $HOSTS_TEST) and merged variables too: $HOSTS_ALL=$HOSTS_PROD,$HOSTS_TEST,$HOSTS_DEV
This worked fine until we switched of dhost1 host (reason does not matter) and I had commended variabled definition:
$HOSTS_DEV=dhost1
But rules remained. As result there was rules for empty element
development
HOST=$HOSTS_DEV SERVICES=$SVC_DB_BUSN COLOR=red,yellow,purple MAIL=$CM_SUPPORT_DBA DURATION>15m REPEAT=60m RECOVERED DURATION<180m FORMAT=PLAIN
this rule really resulted to: HOST= SERVICES=dbinvobj COLOR=red,yellow,purple MAIL=support at maildomain.eu DURATION>15m REPEAT=60m RECOVERED DURATION<180m FORMAT=PLAIN
This rule most probably worked out in some cases together with correct one.
Best regards,
Andrey Chervonets
SIA CoMinder http://www.cominder.eu/
participants (5)
-
A.Chervonets＠cominder.eu
-
cleaver＠terabithia.org
-
larry＠fni-stl.com
-
mburger＠bubbanfriends.org
-
novosirj＠ca.rutgers.edu