xymon disk not alerting at 100%, need another set of eyes
So I had another thought, I copied the class statement to another file so it's now first in the list and last in the list, and my disk test is still green. Is the class match broken?
I'm on 4.3.27-1 from Terabithia.
Thanks!
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: Scot.Kreienkamp at la-z-boy.com From: Scot Kreienkamp Sent: Thursday, January 5, 2017 1:53 PM To: xymon at xymon.com Subject: RE: xymon disk not alerting at 100%, need another set of eyes
After re-reading I can see how that may not be totally clear. By alerting, I mean that the disk test is still green, even though a partition is at 100%full.
I found two hosts that weren't alerting on disk full condition and started digging into the problem further. As I understand it, xymon matches the first entry from analysis config files. So I dumped the analysis config for disks:
Client line: [collector:] client corpvskreienl,na,lzb,hq.linux linux
[root at monvxymon hosts.d]# /usr/libexec/xymon/xymond_client --dump-config --config=etc/analysis.cfg |grep -i ^disk DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 515) DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 516) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 527) DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 528) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%dayexch.*.na.lzb.hq (line: 539) DISK %^T IGNORE HOST=%dayexch.*.na.lzb.hq (line: 540) DISK %^(1|2|3|4|5|6|7|8|9|0|).* IGNORE HOST=%dayexch.*.na.lzb.hq (line: 541) DISK C 204800U 102400U 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 567) DISK E 101% 101% 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 568) DISK F 99% 100% 0 -1 red HOST=mons6000.na.lzb.hq (line: 576) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red PAGE=infrastructure/fileserv (line: 582) DISK D 99% 100% 0 -1 red HOST=lzbv5223.na.lzb.hq,lzbv6016.na.lzb.hq (line: 746) DISK * 90% 95% 0 -1 red HOST=%dvrvas(0|1)\.mdmza.dmz.hq (line: 762) DISK * 90% 95% 0 -1 red CLASS=powershell (line: 1054) DISK * 90% 95% 0 -1 red CLASS=win32 (line: 1073) DISK * 90% 95% 0 -1 red CLASS=linux (line: 1090) DISK * 90% 95% 0 -1 red (line: 1132)
I can't find any lines above where the hostname matches, it's on page Infrastructure/Miscellaneous so none of the page statements match, so it should match on the class. Or the very last line is the system default which should apply if nothing else. My server is sitting at 100%full on one partition so it SHOULD be alerting.
Thanks for any help.
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
Hi Scott,
What may have happened is that the disk filled up quicker than the client could send the alert. If the client is on the same disk that is full. That's caught me a few times.
HTH Regards Greg Shea
----- Original Message -----
So I had another thought, I copied the class statement to another file so it’s now first in the list and last in the list, and my disk test is still green. Is the class match broken?
I’m on 4.3.27-1 from Terabithia.
Thanks!
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: Scot.Kreienkamp at la-z-boy.com
From: Scot Kreienkamp Sent: Thursday, January 5, 2017 1:53 PM To: xymon at xymon.com Subject: RE: xymon disk not alerting at 100%, need another set of eyes
After re-reading I can see how that may not be totally clear. By alerting, I mean that the disk test is still green, even though a partition is at 100%full.
I found two hosts that weren’t alerting on disk full condition and started digging into the problem further. As I understand it, xymon matches the first entry from analysis config files. So I dumped the analysis config for disks:
Client line:
[collector:]
client corpvskreienl,na,lzb,hq.linux linux
[root at monvxymon hosts.d]# /usr/libexec/xymon/xymond_client --dump-config --config=etc/analysis.cfg |grep -i ^disk
DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 515)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 516)
DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 527)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 528)
DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%dayexch.*.na.lzb.hq (line: 539)
DISK %^T IGNORE HOST=%dayexch.*.na.lzb.hq (line: 540)
DISK %^(1|2|3|4|5|6|7|8|9|0|).* IGNORE HOST=%dayexch.*.na.lzb.hq (line: 541)
DISK C 204800U 102400U 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 567)
DISK E 101% 101% 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 568)
DISK F 99% 100% 0 -1 red HOST=mons6000.na.lzb.hq (line: 576)
DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red PAGE=infrastructure/fileserv (line: 582)
DISK D 99% 100% 0 -1 red HOST=lzbv5223.na.lzb.hq,lzbv6016.na.lzb.hq (line: 746)
DISK * 90% 95% 0 -1 red HOST=%dvrvas(0|1)\.mdmza.dmz.hq (line: 762)
DISK * 90% 95% 0 -1 red CLASS=powershell (line: 1054)
DISK * 90% 95% 0 -1 red CLASS=win32 (line: 1073)
DISK * 90% 95% 0 -1 red CLASS=linux (line: 1090)
DISK * 90% 95% 0 -1 red (line: 1132)
I can’t find any lines above where the hostname matches, it’s on page Infrastructure/Miscellaneous so none of the page statements match, so it should match on the class. Or the very last line is the system default which should apply if nothing else. My server is sitting at 100%full on one partition so it SHOULD be alerting.
Thanks for any help.
This message is intended only for t he individual or entity to which it is addressed. It may contain privileged, confi dential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you .
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Have you tried doing a test on the condition?
xymond_alert --test machine.test --duration=500 |grep -v Failed
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Scot Kreienkamp Sent: Thursday, January 05, 2017 12:59 PM To: xymon at xymon.com Subject: Re: [Xymon] xymon disk not alerting at 100%, need another set of eyes
So I had another thought, I copied the class statement to another file so it's now first in the list and last in the list, and my disk test is still green. Is the class match broken?
I'm on 4.3.27-1 from Terabithia.
Thanks!
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: Scot.Kreienkamp at la-z-boy.com<mailto:Scot.Kreienkamp at la-z-boy.com> From: Scot Kreienkamp Sent: Thursday, January 5, 2017 1:53 PM To: xymon at xymon.com<mailto:xymon at xymon.com> Subject: RE: xymon disk not alerting at 100%, need another set of eyes
After re-reading I can see how that may not be totally clear. By alerting, I mean that the disk test is still green, even though a partition is at 100%full.
I found two hosts that weren't alerting on disk full condition and started digging into the problem further. As I understand it, xymon matches the first entry from analysis config files. So I dumped the analysis config for disks:
Client line: [collector:] client corpvskreienl,na,lzb,hq.linux linux
[root at monvxymon hosts.d]# /usr/libexec/xymon/xymond_client --dump-config --config=etc/analysis.cfg |grep -i ^disk DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 515) DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 516) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 527) DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 528) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%dayexch.*.na.lzb.hq (line: 539) DISK %^T IGNORE HOST=%dayexch.*.na.lzb.hq (line: 540) DISK %^(1|2|3|4|5|6|7|8|9|0|).* IGNORE HOST=%dayexch.*.na.lzb.hq (line: 541) DISK C 204800U 102400U 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 567) DISK E 101% 101% 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 568) DISK F 99% 100% 0 -1 red HOST=mons6000.na.lzb.hq (line: 576) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red PAGE=infrastructure/fileserv (line: 582) DISK D 99% 100% 0 -1 red HOST=lzbv5223.na.lzb.hq,lzbv6016.na.lzb.hq (line: 746) DISK * 90% 95% 0 -1 red HOST=%dvrvas(0|1)\.mdmza.dmz.hq (line: 762) DISK * 90% 95% 0 -1 red CLASS=powershell (line: 1054) DISK * 90% 95% 0 -1 red CLASS=win32 (line: 1073) DISK * 90% 95% 0 -1 red CLASS=linux (line: 1090) DISK * 90% 95% 0 -1 red (line: 1132)
I can't find any lines above where the hostname matches, it's on page Infrastructure/Miscellaneous so none of the page statements match, so it should match on the class. Or the very last line is the system default which should apply if nothing else. My server is sitting at 100%full on one partition so it SHOULD be alerting.
Thanks for any help.
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
Hi Paul,
By alerting, I meant the test is not turning red even though the disk is full. A poor choice of words on my part, sorry.
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: Scot.Kreienkamp at la-z-boy.com From: Root, Paul T [mailto:Paul.Root at CenturyLink.com] Sent: Thursday, January 5, 2017 3:19 PM To: Scot Kreienkamp; xymon at xymon.com Subject: RE: [Xymon] xymon disk not alerting at 100%, need another set of eyes
Have you tried doing a test on the condition?
xymond_alert --test machine.test --duration=500 |grep -v Failed
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Scot Kreienkamp Sent: Thursday, January 05, 2017 12:59 PM To:xymon at xymon.com<mailto:xymon at xymon.com> Subject: Re: [Xymon] xymon disk not alerting at 100%, need another set of eyes
So I had another thought, I copied the class statement to another file so it's now first in the list and last in the list, and my disk test is still green. Is the class match broken?
I'm on 4.3.27-1 from Terabithia.
Thanks! From: Scot Kreienkamp Sent: Thursday, January 5, 2017 1:53 PM To:xymon at xymon.com<mailto:xymon at xymon.com> Subject: RE: xymon disk not alerting at 100%, need another set of eyes
After re-reading I can see how that may not be totally clear. By alerting, I mean that the disk test is still green, even though a partition is at 100%full.
I found two hosts that weren't alerting on disk full condition and started digging into the problem further. As I understand it, xymon matches the first entry from analysis config files. So I dumped the analysis config for disks:
Client line: [collector:] client corpvskreienl,na,lzb,hq.linux linux
[root at monvxymon hosts.d]# /usr/libexec/xymon/xymond_client --dump-config --config=etc/analysis.cfg |grep -i ^disk DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 515) DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mondbexec.*|mondb.*|retmaildb.*).na.lzb.hq (line: 516) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 527) DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=%(mon|new|red|neo|taz|sil|kin|sal|hpt)exch.*.na.lzb.hq (line: 528) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red HOST=%dayexch.*.na.lzb.hq (line: 539) DISK %^T IGNORE HOST=%dayexch.*.na.lzb.hq (line: 540) DISK %^(1|2|3|4|5|6|7|8|9|0|).* IGNORE HOST=%dayexch.*.na.lzb.hq (line: 541) DISK C 204800U 102400U 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 567) DISK E 101% 101% 0 -1 red HOST=mdas4000.mdmza.dmz.hq (line: 568) DISK F 99% 100% 0 -1 red HOST=mons6000.na.lzb.hq (line: 576) DISK %^(D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z) 15728640U 10485760U 0 -1 red PAGE=infrastructure/fileserv (line: 582) DISK D 99% 100% 0 -1 red HOST=lzbv5223.na.lzb.hq,lzbv6016.na.lzb.hq (line: 746) DISK * 90% 95% 0 -1 red HOST=%dvrvas(0|1)\.mdmza.dmz.hq (line: 762) DISK * 90% 95% 0 -1 red CLASS=powershell (line: 1054) DISK * 90% 95% 0 -1 red CLASS=win32 (line: 1073) DISK * 90% 95% 0 -1 red CLASS=linux (line: 1090) DISK * 90% 95% 0 -1 red (line: 1132)
I can't find any lines above where the hostname matches, it's on page Infrastructure/Miscellaneous so none of the page statements match, so it should match on the class. Or the very last line is the system default which should apply if nothing else. My server is sitting at 100%full on one partition so it SHOULD be alerting.
Thanks for any help.
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you. This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
participants (3)
-
Paul.Root＠CenturyLink.com
-
Scot.Kreienkamp＠la-z-boy.com
-
shea4th＠comcast.net