September 2011 - Xymon - Xymon mailman web

Double combo's - returning -1 - xymon 4.3.3
by wrichardson＠llbean.com 15 Sep '11

15 Sep '11

The older version of Xymon that I have running 4.3.0.0-beta2 has the same "&unknown" issue. Will let you know how 4.3.5 goes... -----Original Message----- From: Bill Richardson Sent: Tuesday, September 13, 2011 9:58 AM To: David W Gore; xymon at xymon.com Subject: RE: [Xymon] Double combo's - returning -1 - xymon 4.3.3 I was doing some testing of this feature today and notice the same problem. I am testing with 4.3.3. Will be upgrading to 4.3.5 this week and will let you know what I find. If that doesn't work I have an older 4.3.0 that I can test on just to see if maybe something got broken along the way... This is the first time I have ever tested double level combos. -----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of David W Gore Sent: Friday, July 29, 2011 3:17 PM To: xymon at xymon.com Subject: Re: [Xymon] Double combo's - returning -1 - xymon 4.3.3 Does anyone out there using double leveled combos, combos of combos that work? If so perhaps you could share an example from your config that works? Thanks in advance, Here is the example from the man pages: ---- WebCluster.http = (Web1.http || Web2.http) AppSrvCluster.procs = (AppSrv1.conn && AppSrv1.procs) || (AppSrv2.conn && AppSrv2.procs) Customer.cluster = WebCluster.http && AppSrvCluster.procs The first line defines a new test, with hostname "WebCluster" and the columnname "http". It will be green if the http test on either the "Web1" or the "Web2" server is green. The second line defines a "procs" test for the "AppSrvCluster" host. Each of the AppSrv1 and AppSrv2 hosts is checked for "conn" (ping) and their "procs" test. On each host, both of these must be green, but the combined test is green if that condition is fulfilled on just one of the hosts. The third line uses the two first tests to build a "double combined" test, defining a test that shows the overall health of the system. ---- So in theory it should work? For reference we are using Xymon 4.3.3 on RedHat Linux Enterprise 5.7. ~David From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Rager, Richard R (Rich) Sent: Thursday, June 30, 2011 19:02 To: xymon at xymon.com Subject: [Xymon] Double combo's - returning -1 - xymon 4.3.3 Hi, Anyone using double combo messages? I was following the man pages, which state they are possible, but my tests calculate "-1" and the color comes out $unknown. Thu Jun 30 18:43:24 2011 (iam.db&&gam.db) = (-1&&-1) = 1 &unknown iam.db &unknown gam.db Sample config from combo.cfg. I have changed hostnames to protect the innocent...combos based on the values returned by dbcheck.pl...our DB's are served on 4 node Oracle Grid clusters, which is why we have alias' for the DB, such as servername_dbname. Hostnames are defined in hosts.d as "0.0.0.0 gam # noconn nonongreen noproppurple:*" gam/iam/impact all defined the same in hosts. iam.db = (servername\_iam.audit && servername\_iam.conn && servername\_iam.extent && servername\_iam.info && servername\_iam.invobj && servername\_iam.locks && servername\_iam.rollback && servername\_iam.session && servername\_iam.tblspace && servername\_iam.trends && servername\_iam.uptime) gam.db = (servername\_gam.audit && servername\_gam.conn && servername\_gam.extent && servername\_gam.info && servername\_gam.invobj && servername\_gam.locks && servername\_gam.rollback && servername\_gam.session && servername\_gam.tblspace && servername\_gam.trends && servername\_gam.uptime) These two above work fine. They are single layer combo tests...All my single layer combo tests work. The double combo is this: impact.db = (iam.db && gam.db) - same with our without parenthesys...if I use single ampersand, color propogates up, but staus still shows &unknown. We have tried multiple combinations taking the combo down to barest form, and still get the errors (&unknown). Log doesn't really help... combostatus.log.1:21316 2011-06-30 03:58:03 Symbolic 'event.db&&gam.db&&iam.db&&jbpm.db' converted to '-1&&-1&&-1&&-1' I put the --debug on the cmd in tasks, and not logging much else helpful. Appreciate some thoughs or suggestions. Rich Rager VZB ________________________________________ This message was created by the sender using your @mci.com address. If you wish to continue to receive messages from this sender, please respond informing them to use your @verizonbusiness.com email address. The mci.com email domain will be decommissioned October 31, 2010. Please inform sender to use your @verizonbusiness.com address, if you desire to continue to receive messages from them. Thank You Verizon Business Messaging _______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

2 2

How to get vmstat5 to show on Trends page
by pnixon＠gmail.com 15 Sep '11

15 Sep '11

Hey all, I'm trying to figure out how to get the vmstat5 graph to show up on the trends page for a 4.3.3 installation. I've tried adding vmstat5 to the GRAPHS statement in xymonserver.cfg, as well as vmstat::1, vmstat::5, etc with no success. Suggestions? Thanks! --Patrick

4 4

Combo features was Re: Double combo's - returning -1 - xymon 4.3.3
by betsy.schwartz＠gmail.com 15 Sep '11

15 Sep '11

I have not thought of a scenario for double combos in our enterprise. However I have thought of several places where it would be useful to have combos on *yellow* or combos that could return yellow. ie, go red if more than two servers in a cluster are yellow (solved this one by creatng a second, duplicate local test that just reports red when the original test reports yellow, used only for the combo test, not itself paged on) OR I want to go yellow if two servers out of four in a cluster are yellow or red (haven't got a way to go yellow on a combo test)

2 1

Ignoring file system when two "/" (root partitions) are reported
by waa-hobbitml＠revpol.com 14 Sep '11

14 Sep '11

Hi Larry... Thanks for the reply (moving your reply to the list) Sure I could, but this looks like something where I will not be the only one affected, so I figured I'd report it and see if there was either 1. A correct way in analisys.cfg that I was just missing somehow or 2. Have it 'fixed' in the default (eg: in the release version) rather than me making a one-off fix to some internal variables Having said that, it looks like your solution will probably immediately fix my problem and I will take a look at it as soon as I have a few minutes late today or tomorrow. Thanks -- Bill Arlofski Reverse Polarity, LLC http://www.revpol.com/ On 09/13/11 14:40, Larry Barber wrote: > Could you insert a "grep -v rootfs" in the 'df' section of > xymonclient-linus.sh? > > echo "[df]" > EXCLUDES=`cat /proc/filesystems | grep nodev | awk '{print $2}' | xargs echo > | sed -e 's! ! -x !g'` > df -Pl -x iso9660 -x $EXCLUDES | grep -v rootfs | sed -e '/^[^ ][^ > ]*$/{ > N > s/[ ]*\n[ ]*/ / > }' > > Thanks, > Larry Barber > > On Tue, Sep 13, 2011 at 12:45 PM, Bill Arlofski <waa-hobbitml at revpol.com>wrote: > >> On a Gentoo Linux system, after a recent baselayout update to openrc 2.x >> (at >> least I believe that to be the culprit) xymon clients are now reporting >> back >> TWO "/" (root) filesystems, and xymon v4.3.4 is reporting and graphing >> them >> both like so: >> >> >> Tue Sep 13 13:16:58 EDT 2011 - Filesystems NOT ok >> >> (yellow) / (92% used) has reached the WARNING level (90%) >> (yellow) / (92% used) has reached the WARNING level (90%) >> >> Filesystem 1024-blocks Used Available Capacity Mounted on >> rootfs 96571704 88758608 7813096 92% / >> /dev/root 96571704 88758608 7813096 92% / >> >> In analyze.cfg I have tried to ignore rootfs and then the regular >> expression >> versions just in case xymon matched the whole line reported, and not just >> the >> info in the "Mounted on" column: >> >> HOST=hostname.domain.tld >> DISK rootfs IGNORE >> >> or >> >> HOST=hostname.domain.tld >> DISK %rootfs IGNORE >> >> or >> >> HOST=hostname.domain.tld >> DISK %^rootfs IGNORE >> >> or >> >> HOST=hostname.domain.tld >> DISK /dev/root IGNORE >> >> >> but these of course will not work since the docs for analyze.cfg say that >> the >> DISK test works on the MOUNT POINT, not the device, or partition. >> >> Is there any way to ignore ONE of the two disk reports that are essentially >> the same exact thing, only reported in two different formats? >> >> >> >> Also, as a side note/suggestion, the docs might be a bit >> ambiguous/confusing >> with regards to "filesystem" and "mount point", using them >> semi-interchangeably which can be needlessly confusing: >> >> # DISK filesystem warnlevel paniclevel >> # DISK filesystem IGNORE >> # If the utilization of "filesystem" is reported to exceed "warnlevel" >> >> Then filesystem is described as... >> # "filesystem" is the mount-point where the filesystem is mounted, e.g. >> >> >> Perhaps that could be more clear if the test was defined like so: >> >> # DISK mountpoint warnlevel paniclevel >> # DISK mountpoint IGNORE >> >> Then, there would be no need for the "filesystem is the mount point..." >> line >> since it would be self-evident >> >> Thanks! >> >> -- >> Bill Arlofski >> Reverse Polarity, LLC >> http://www.revpol.com/ >> _______________________________________________ >> Xymon mailing list >> Xymon at xymon.com >> http://lists.xymon.com/mailman/listinfo/xymon >> >

1 1

AIX df command problem
by sholmes42＠mac.com 13 Sep '11

13 Sep '11

I am having a problem similar to one recently posted, but this is on an AIX host and is caused (I think) by a problem in the way xymon does df on AIX. The xymon version is 4.2.3 and the AIX OS version is 3.5 (I think, I'm a Solaris Admin). Anyway, I can find no way to force the df on AIX to use POSIX formatting AND to report only local file systems. Even if I could, I don't think I can override what xymon uses for df since the comments in the hobbitclient.cfg file say that the envariable definition is only there for compatibility and it is not used. The problem is that an NFS mount file system line in the df output is being parsed wrong and the number in the usage column is being used for the percent used (Available) so that is a huge number. Not only that, again similar to the previous problem, the file system name is being picked up as the contents of the last 3 fields in the output line, i.e. it is not /mnt/tsmupdates (which is what it should be) but "3020485 50% /tsm/updates", which, of course, prevents me from effectively ignoring that file system. I've tried using DISK %.*/mnt/.* IGNORE and that doesn't work. Does anyone have any suggestions? We have a big red page where we shouldn't have. Thanks, Steve -- The test of a democracy is not the magnificence of buildings or the speed of automobiles or the efficiency of air transportation, but rather the care given to the welfare of all the people. -Helen Adams Keller, lecturer and author (1880-1968) Truth never damages a cause that is just. -Mohandas Karamchand Gandhi (1869-1948)

3 4

Ignoring file system when two "/" (root partitions) are reported
by waa-hobbitml＠revpol.com 13 Sep '11

13 Sep '11

On a Gentoo Linux system, after a recent baselayout update to openrc 2.x (at least I believe that to be the culprit) xymon clients are now reporting back TWO "/" (root) filesystems, and xymon v4.3.4 is reporting and graphing them both like so: Tue Sep 13 13:16:58 EDT 2011 - Filesystems NOT ok (yellow) / (92% used) has reached the WARNING level (90%) (yellow) / (92% used) has reached the WARNING level (90%) Filesystem 1024-blocks Used Available Capacity Mounted on rootfs 96571704 88758608 7813096 92% / /dev/root 96571704 88758608 7813096 92% / In analyze.cfg I have tried to ignore rootfs and then the regular expression versions just in case xymon matched the whole line reported, and not just the info in the "Mounted on" column: HOST=hostname.domain.tld DISK rootfs IGNORE or HOST=hostname.domain.tld DISK %rootfs IGNORE or HOST=hostname.domain.tld DISK %^rootfs IGNORE or HOST=hostname.domain.tld DISK /dev/root IGNORE but these of course will not work since the docs for analyze.cfg say that the DISK test works on the MOUNT POINT, not the device, or partition. Is there any way to ignore ONE of the two disk reports that are essentially the same exact thing, only reported in two different formats? Also, as a side note/suggestion, the docs might be a bit ambiguous/confusing with regards to "filesystem" and "mount point", using them semi-interchangeably which can be needlessly confusing: # DISK filesystem warnlevel paniclevel # DISK filesystem IGNORE # If the utilization of "filesystem" is reported to exceed "warnlevel" Then filesystem is described as... # "filesystem" is the mount-point where the filesystem is mounted, e.g. Perhaps that could be more clear if the test was defined like so: # DISK mountpoint warnlevel paniclevel # DISK mountpoint IGNORE Then, there would be no need for the "filesystem is the mount point..." line since it would be self-evident Thanks! -- Bill Arlofski Reverse Polarity, LLC http://www.revpol.com/

1 0

xymon iostat causing problems on loaded box, how to mitigate?
by betsy.schwartz＠gmail.com 13 Sep '11

13 Sep '11

We're seeing problems with xymon iostat hanging on heavily loaded, massively large database servers. These boxes are on Oracle RAC clusters with EMC storage devices attached. iostat -x shows 16572 lines. When the box starts to have problems for *other* reasons, all of the iostat commands start hanging or taking forever, further bogging the system down. And making xymon look really bad ... I'm wondering - what does xymon do with all this information? How much of it can I turn off without confusing the server? (part of a PS from 12:13, showing some iostats that have been running for over half an hour) # ps -ef|grep iostat xymon 102 98 2 11:55:47 ? 7:18 iostat -dxsrP 300 2 xymon 16934 16931 0 12:10:00 ? 0:04 iostat -c 300 2 xymon 16911 1 0 11:51:02 ? 0:00 sh -c iostat -dxsrP 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatdisk.db6 xymon 566 1 0 12:05:38 ? 0:00 sh -c iostat -dxsrP 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatdisk.db6 xymon 16931 1 0 12:10:00 ? 0:00 sh -c iostat -c 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatcpu.db6.bo3. xymon 16932 1 0 12:10:00 ? 0:00 sh -c iostat -dxsrP 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatdisk.db6 xymon 15575 15566 2 12:00:47 ? 5:53 iostat -dxsrP 300 2 xymon 15566 1 0 12:00:47 ? 0:00 sh -c iostat -dxsrP 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatdisk.db6 xymon 1349 1344 2 11:45:55 ? 9:34 iostat -dxsrP 300 2 xymon 17276 17271 2 11:40:55 ? 10:55 iostat -dxsrP 300 2 xymon 16935 16932 2 12:10:00 ? 2:57 iostat -dxsrP 300 2 xymon 17271 1 0 11:40:55 ? 0:00 sh -c iostat -dxsrP 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatdisk.db6 xymon 1344 1 0 11:45:55 ? 0:00 sh -c iostat -dxsrP 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatdisk.db6 xymon 572 566 2 12:05:38 ? 5:13 iostat -dxsrP 300 2 xymon 16914 16911 2 11:51:02 ? 8:24 iostat -dxsrP 300 2 xymon 98 1 0 11:55:47 ? 0:00 sh -c iostat -dxsrP 300 2 1>/export/home/xymon/client/tmp/hobbit_iostatdisk.db6 <etc>

2 2

Double combo's - returning -1 - xymon 4.3.3
by david.gore＠one.verizon.com 13 Sep '11

13 Sep '11

Does anyone out there using double leveled combos, combos of combos that work? If so perhaps you could share an example from your config that works? Thanks in advance, Here is the example from the man pages: ---- WebCluster.http = (Web1.http || Web2.http) AppSrvCluster.procs = (AppSrv1.conn && AppSrv1.procs) || (AppSrv2.conn && AppSrv2.procs) Customer.cluster = WebCluster.http && AppSrvCluster.procs The first line defines a new test, with hostname "WebCluster" and the columnname "http". It will be green if the http test on either the "Web1" or the "Web2" server is green. The second line defines a "procs" test for the "AppSrvCluster" host. Each of the AppSrv1 and AppSrv2 hosts is checked for "conn" (ping) and their "procs" test. On each host, both of these must be green, but the combined test is green if that condition is fulfilled on just one of the hosts. The third line uses the two first tests to build a "double combined" test, defining a test that shows the overall health of the system. ---- So in theory it should work? For reference we are using Xymon 4.3.3 on RedHat Linux Enterprise 5.7. ~David From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Rager, Richard R (Rich) Sent: Thursday, June 30, 2011 19:02 To: xymon at xymon.com Subject: [Xymon] Double combo's - returning -1 - xymon 4.3.3 Hi, Anyone using double combo messages? I was following the man pages, which state they are possible, but my tests calculate "-1" and the color comes out $unknown. Thu Jun 30 18:43:24 2011 (iam.db&&gam.db) = (-1&&-1) = 1 &unknown iam.db &unknown gam.db Sample config from combo.cfg. I have changed hostnames to protect the innocent...combos based on the values returned by dbcheck.pl...our DB's are served on 4 node Oracle Grid clusters, which is why we have alias' for the DB, such as servername_dbname. Hostnames are defined in hosts.d as "0.0.0.0 gam # noconn nonongreen noproppurple:*" gam/iam/impact all defined the same in hosts. iam.db = (servername\_iam.audit && servername\_iam.conn && servername\_iam.extent && servername\_iam.info && servername\_iam.invobj && servername\_iam.locks && servername\_iam.rollback && servername\_iam.session && servername\_iam.tblspace && servername\_iam.trends && servername\_iam.uptime) gam.db = (servername\_gam.audit && servername\_gam.conn && servername\_gam.extent && servername\_gam.info && servername\_gam.invobj && servername\_gam.locks && servername\_gam.rollback && servername\_gam.session && servername\_gam.tblspace && servername\_gam.trends && servername\_gam.uptime) These two above work fine. They are single layer combo tests...All my single layer combo tests work. The double combo is this: impact.db = (iam.db && gam.db) - same with our without parenthesys...if I use single ampersand, color propogates up, but staus still shows &unknown. We have tried multiple combinations taking the combo down to barest form, and still get the errors (&unknown). Log doesn't really help... combostatus.log.1:21316 2011-06-30 03:58:03 Symbolic 'event.db&&gam.db&&iam.db&&jbpm.db' converted to '-1&&-1&&-1&&-1' I put the --debug on the cmd in tasks, and not logging much else helpful. Appreciate some thoughs or suggestions. Rich Rager VZB ________________________________________ This message was created by the sender using your @mci.com address. If you wish to continue to receive messages from this sender, please respond informing them to use your @verizonbusiness.com email address. The mci.com email domain will be decommissioned October 31, 2010. Please inform sender to use your @verizonbusiness.com address, if you desire to continue to receive messages from them. Thank You Verizon Business Messaging

2 1

Odd named errors, perhaps related to Xymon or Devmon?
by sberkley＠mcclatchyinteractive.com 13 Sep '11

13 Sep '11

We've been getting an average of 40+/day of these messages from the caching nameserver process on one of our Xymon/Devmon servers: named[N]: sockmgr 0xN: maximum number of FD events (64) received This server is running bind-9.3.6-16.P1 (old, I know, but it's the latest for the distro we are using). I was wondering if anyone else out there was seeing the same errors? I'm making an assumption that Xymon/Devmon (or one of our ext scripts) might be inducing this condition, since this server doesn't have any other processes on it that would be doing a large amount of DNS lookups. This server has over 600 entries in hosts.cfg and 92 entries in the Devmon hosts.db. I have tried reproducing the problem by making a large amount of DNS lookups as simultaneously as possible, but with no luck. The xymonnet status only lists 42 failed DNS lookups out of 797 calls to dnsresolve. Any suggestions on how to find out what specific process is causing named to make periodic high demands for sockets? -- S i m e o n B e r k l e y Systems Engineer McClatchy Interactive phone: 919-861-1244 fax: 919-861-1300 mobile: 919-302-3063 e-mail: sberkley at mcclatchyinteractive.com AIM: sberkleymi www.mcclatchyinteractive.com

2 1

TS - NTP testing
by d.tom.schmitt＠L-3com.com 13 Sep '11

13 Sep '11

Is there a way to select the NTP variance information from each server? I know there is a line at the top of the CPU icon for Linux that says: "System clock is 0 seconds off" I cannot find an equivalent in the Windows client info - we use the BB client currently but I have the BBwin client on my PC and do not see any NTP information. We need a report that will show how our NTP services are doing. We have the BBwin client running on most of our Windows servers. Thanks, Tom Schmitt Senior IT Staff - R&D L-3 Communication Systems West 640 North 2200 West P.O. Box 16850 Salt Lake City, UT 84116 Phone (801) 594-3030 Cell (801) 231-7230 eFax (470) 201-6644 \\\\||//// \ ~ ~ / | @ @ | --oOo---(_)---oOo-- Have A Nice Day !

2 1