Help with disk monitoring
Hello,
I have recently inherited a sizable Hobbit environment so please have patience with my general ignorance of all things Hobbit. Our Hobbit works really well except for one notable exception and I can't figure out why. We have a series of boxes that run an application and for some reason these systems will not monitor the /opt file system and alert us when /opt goes over 90% even though these boxes have the same Hobbit setup like the others on the server side and the client side.
The only difference I can see is that the /opt file system is mapped differently on these systems. Most systems use something like
/dev/sda5 120G 82G 73% /opt
But the problem systems use
/dev/mapper/sysvg-opt 120G 82G 32G 73% /opt
Also the file systems from these problem servers show up differently as well. The standard file system mounts are displayed without an issues but the LVM maps display the mount point but nothing else
/dev/mapper/sysvg-opt /dev/mapper/sysvg-var /dev/cciss/c0d0p1 2016016 216060 1697544 12% / /dev/cciss/c0d0p2 3020172 950024 1916728 34% /usr
The problem servers also have the follow error in the client side log
Mon Dec 11 15:53:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh: line 180: test: : integer expression expected Mon Dec 11 15:58:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh: line 180: test: : integer expression expected Mon Dec 11 15:58:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh: line 180: test: : integer expression expected Mon Dec 11 16:03:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh: line 180: test: : integer expression expected
From bbdef.sh
LOCAL CLIENT MONITORING CONFIGURATION FOR bb-local.sh
WARNING AND PANIC LEVELS FOR LOCAL SYSTEM INFOMRAION
YOU CAN SET VALUES ON A SPECIFIC FILESYSTEM BY USING
THE etc/bb-dftab FILE
DFWARN=90 # (YELLOW) DISK % TO WARN DFPANIC=95 # (RED) DISK % TO PANIC export DFWARN DFPANIC
The lines above and below line 180 of bb-disk.sh
MARK="" if test "$DISKPCT" -ge "$DFWARNVAL" # ABOVE WARNING LEVEL then if test "$DISKPCT" -ge "$DFPANICVAL"
ABOVE PANIC, RED ALERT !!!
then
COLOR="red"
MARK="PANIC"
MARKLEVEL=$DFPANICVAL
HIGHMARK="PANIC"
GIFCOLOR="&red"
REDLINE="${GIFCOLOR} ${DISKNAME}
(${DISKPCT}%) has reached the defined disk space ${MARK} le vel (${MARKLEVEL}%) ${REDLINE}" else
By default all servers are set to alert at 90% and 95% percent.
Thanks in advance for everyone's help
Ed Bailey
CONFIDENTIALITY NOTICE This e-mail and any attachments contain information which may be confidential or privileged and exempt from disclosure under applicable law. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the contents of this information is without authorization and is prohibited. If you have received this email in error, please immediately notify us by returning it to the sender and delete this copy from your computer system. Thank you.
On Mon, Dec 11, 2006 at 04:21:16PM -0500, Bailey, Edward wrote:
Also the file systems from these problem servers show up differently as well. The standard file system mounts are displayed without an issues but the LVM maps display the mount point but nothing else
/dev/mapper/sysvg-opt /dev/mapper/sysvg-var /dev/cciss/c0d0p1 2016016 216060 1697544 12% / /dev/cciss/c0d0p2 3020172 950024 1916728 34% /usr
This is a classical problem with the Big Brother client - you are apparently using the BB client instead of the Hobbit client. What's happening is that the "df" command used on the monitored servers breaks long lines in the output, so your "df" output looks like
Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 579598 192382 356293 36% / /dev/mapper/vg1-lv_usr 2568144 648964 1788724 27% /usr /dev/mapper/vg1-lv_var 1048540 733448 315092 70% /var
and since the BB client does a 'grep "^/dev"' to weed out entries like tmpfs, you only get the /dev/mapper line, not the next line with the actual data in it.
The solution is to change the DFCMD setting on your servers to "df -P", making it use the POSIX format output.
Regards, Henrik
Thanks for your quick response:) I did see that we have the big brother client installed so I need to plan on upgrading the clients first.
Thanks again
Ed
-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Monday, December 11, 2006 4:28 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Help with disk monitoring
On Mon, Dec 11, 2006 at 04:21:16PM -0500, Bailey, Edward wrote:
Also the file systems from these problem servers show up differently as well. The standard file system mounts are displayed without an issues but the LVM maps display the mount point but nothing else
/dev/mapper/sysvg-opt /dev/mapper/sysvg-var /dev/cciss/c0d0p1 2016016 216060 1697544 12% / /dev/cciss/c0d0p2 3020172 950024 1916728 34% /usr
This is a classical problem with the Big Brother client - you are apparently using the BB client instead of the Hobbit client. What's happening is that the "df" command used on the monitored servers breaks long lines in the output, so your "df" output looks like
Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 579598 192382 356293 36% / /dev/mapper/vg1-lv_usr 2568144 648964 1788724 27% /usr /dev/mapper/vg1-lv_var 1048540 733448 315092 70% /var
and since the BB client does a 'grep "^/dev"' to weed out entries like tmpfs, you only get the /dev/mapper line, not the next line with the actual data in it.
The solution is to change the DFCMD setting on your servers to "df -P", making it use the POSIX format output.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
CONFIDENTIALITY NOTICE This e-mail and any attachments contain information which may be confidential or privileged and exempt from disclosure under applicable law. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the contents of this information is without authorization and is prohibited. If you have received this email in error, please immediately notify us by returning it to the sender and delete this copy from your computer system. Thank you.
participants (2)
-
ebailey@transunion.com
-
henrik@hswn.dk