New bit of data on this issue if that helps:
Speaking with my customer, he informs me this particular file system has been monitored / alarmed when full just fine for years(!!). The problems began when they emptied out the file system to (almost) zero content. That's when things went "red".
I'm thinking now that there's some bad data "stuck" in Xymon's state-full data that is yielding the false positives.
Any thoughts on that point?
david
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Mills,David (HHSC Contractor) Sent: Tuesday, January 31, 2017 4:54 PM To: 'xymon at xymon.com' Subject: [Xymon] Absolute free space value for DISK gone rogue?
All -
I have a file system on a host which is alarming "red" even though the file system is brand new / empty. In the notes below, the mis-reporting filesystem is "/ora_data_estdmprd"
The stanza in analysis.cfg that covers it is:
HOST=pwdu008 # Stand-alone prod DB hosts: won't have shared FS's DISK /ora_bin 85 95 # 85%/95% = CRQ000000013038 DISK /ora_data 85 95 # 85%/95% = CRQ000000013038 DISK /ora_admin 85 95 # 85%/95% = CRQ000000013038 DISK /ora_temp 85 95 # 85%/95% = CRQ000000013038 DISK /ora_redoa 85 95 # 85%/95% = CRQ000000013038 DISK /ora_redob 85 95 # 85%/95% = CRQ000000013038 DISK /ora_arch 85 95 # 85%/95% = CRQ000000013038 DISK /ora_data_finprod1 85 95 # 85%/95% = CRQ000000013038 DISK /ora_finbin 85 95 # 85%/95% = CRQ000000013038 DISK %/ora_data_perpr 85 95 # 85%/95% = CRQ000000013038 DISK %/ora_ 262144000U 209715200U # I.e. Yellow: <= 256GB; Red <= 204.8GB
The output from the "Disk" detail page for that host was:
status pwdu008.disk yellow Tue Jan 31 14:06:20 CST 2017 - Filesystems NOT ok &red /ora_data_estdmprd (3045500329 units free) has reached the CRITICAL level (209715200 units)
Filesystem kbytes used avail capacity Mounted on /dev/md/dsk/d1 16525754 7879972 8480525 49% / /dev/md/dsk/d4 20655025 2723628 17724847 14% /var /dev/md/dsk/d7 17197130 2708305 14316854 16% /export /dev/odm 0 0 0 0% /dev/odm sharefs 0 0 0 0% /etc/dfs/sharetab swap 147391344 1936 147389408 1% /etc/svc/volatile swap 147396208 6840 147389368 1% /tmp swap 147389344 0 147389344 0% /dev/vx/dmp swap 147384744 0 147384744 0% /dev/vx/rdmp /dev/vx/dsk/ORA_DATA/ora_data 249925632 5610862 229045230 3% /ora_data /dev/vx/dsk/ORA_ARCH/ora_arch 249958400 685078 233695152 1% /ora_arch /dev/vx/dsk/ORA_BIN/ora_bin 124930048 54292389 66222935 46% /ora_bin /dev/vx/dsk/ORA_ADMIN/ora_admin 62465024 202884 58405611 1% /ora_admin /dev/vx/dsk/ORA_REDOA/ora_redoa 124962816 833493 116371374 1% /ora_redoa /dev/vx/dsk/ORA_ARCH_ESTDMPRD/ora_arch_estdmprd 1749710848 966260000 777330408 56% /ora_arch_estdmprd /dev/vx/dsk/ORA_TEMP/ora_temp 999961600 521070 936975632 1% /ora_temp /dev/vx/dsk/ORA_REDOB/ora_redob 124962816 423857 116755408 1% /ora_redob /dev/vx/dsk/ORA_DATA_DMSTGPRD/ora_data_dmstgprd 4999072768 1751121 4728890415 1% /ora_data_dmstgprd /dev/vx/dsk/ORA_DATA_ESTDMPRD/oradata_estdmprd01 3249462272 929825 3045500329 1% /ora_data_estdmprd
In the report above, we see that the "critical" threshold (in KB on Solaris) is 209,715,200 free space (about 256 GB), but the actual free space for that file system is 3,045,500,329 (around 3 TB). Something is clearly not right!
Also, note the file system second-from-the bottom: /ora_data_dmstgprd. It has _even more_ free space and should be triggered by the same rule in analysis.cfg (DISK %/ora_ ...) yet it is not flagged as having a problem. As a stop-gap to keep the filesystem from alarming, I've had to change the "DISK" rule in analysis.cfg to percentages (85 95) which makes everything go green.
Particulars: Xymon v. 4.3.3 running on Solaris 10 host. (Have also tried restarting the server without impact.)
Any thoughts?
Thanks!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 595-1238 (mobile)