mounted NFS volumes not included in disk utilization graphs
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?
On 31 March 2015 at 09:49, Paul Grondahl <paul at arrowtel.net> wrote:
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?
To monitor NFS-shared disks, you would typically monitor them on the NFS server, rather than one of the clients.
What OS are you using? Can you show your [df] client data?
J
Hi,
Using CentOS 6.4
The last line in the df output shows the NFS mount. [df] Filesystem 1024-blocks Used Available Capacity Mounted on /dev/vda2 19766140 6449876 12312184 35% / /dev/vda1 99150 25418 68612 28% /boot [inode] Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vda2 1256640 192603 1064037 16% / /dev/vda1 25688 38 25650 1% /boot [mount] /dev/vda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) arrow81:/recordings/arrow82 on /mnt/nfs/recordings type nfs (rw,vers=4,clientaddr=172.16.6.167,addr=172.16.6.168)
On Mar 30, 2015, at 10:17 PM, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
On 31 March 2015 at 09:49, Paul Grondahl <paul at arrowtel.net <mailto:paul at arrowtel.net>> wrote: On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?
To monitor NFS-shared disks, you would typically monitor them on the NFS server, rather than one of the clients.
What OS are you using? Can you show your [df] client data?
J
On 1 April 2015 at 06:46, Paul Grondahl <paul at arrowtel.net> wrote:
Using CentOS 6.4
The last line in the df output shows the NFS mount.
[df] Filesystem 1024-blocks Used Available Capacity Mounted on /dev/vda2 19766140 6449876 12312184 35% / /dev/vda1 99150 25418 68612 28% /boot
It's the [df] section of the client data that is used to record disk usage, not the [mount] section. For the Linux client script (xymonclient-linux.sh) the command is "df -Pl" where the "-l" means "local file systems".
The script also looks for "nodev" devices and builds an exclude list of filesystem times to remove from the output of "df". This is intended to remove pseudo filesystems such as proc and tmpfs, but also removes NFS-mounted filesystems.
There are various manipulations that you need to make in order for the "df" command to show the extra filesystems you want. But in my opinion, the easiest way to accomplish this is to simply add an extra "df" command into the [df] part of the script, showing the extra filesystems you want. Such as:
echo "[df]"
EXCLUDES=cat /proc/filesystems | grep nodev | awk '{print $2}' | xargs echo | sed -e 's! ! -x !g'
{ df -Pl -x iso9660 -x $EXCLUDES; df -P -t nfs | tail +2; } | sed -e '/^[^
][^ ]*$/{
N
s/[ ]*\n[ ]*/ /
}'
The bit I added was "df -P -t nfs | tail +2" (along with the {braces} to make it look to awk like a single df command). Give this a go and let me know if it works for you.
HTH
Cheers Jeremy
Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs? There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple.
That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems.
Regards, Henrik
I'll "me, too" Henrik's response (good to see he's still lurking). df on nfs should happen on the server side. Another problem, I've had the same thing happen to SAN disks that were "locally" attached when the server drops away unexpectedly.
=G=
From: Xymon <xymon-bounces at xymon.com> on behalf of Henrik Størner <henrik at hswn.dk> Sent: Wednesday, April 1, 2015 4:13 PM To: xymon at xymon.com Subject: Re: [Xymon] mounted NFS volumes not included in disk utilization graphs
Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs? There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple.
That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems.
Regards, Henrik
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
At my work, my team developed some "safe" coding to list NFS filesystems in df. It uses the showmount command to list exports on the NFS server - if this succeeds, it should be safe to run df on the filesystem.
The comment in the code references the URL http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html but it doesn't seem to exist now.
It's ugly and no doubt could be cleaned up and improved... but it does work, and it's handy to know which clients are affected when a NFS volume becomes full.
Update to xymonclient-linux.sh :
At the end of the code for "df", add these lines :
--- Start - Addition code, to check nfs mounts ---
Added following code, based on http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html
/bin/bash -c "/bin/egrep $'[ \t]nfs[ \t]' /etc/mtab" |
/bin/egrep -v "ignore|indirect" |
/usr/bin/awk '{print $1, $2}' |
while read line
do
set -- $line
NFSHOST=echo $1 | /usr/bin/awk -F: '{print $1}'
SHARE=echo $1 | /usr/bin/awk -F: '{print $2}'
/bin/ping $NFSHOST -c 5 >/dev/null 2>&1
if [ $? -ne 0 ]
then
echo "$1 IS NOT RESPONDING(ping_failed) 999% $2"
else
# Ensure NFS connection is active BEFORE running the "showmount -e" command.
/bin/df -k $2 > /dev/null 2>&1 & /bin/sleep 5 ; kill $! > /dev/null 2>&1
/usr/sbin/showmount -e $NFSHOST >/dev/null 2>&1
if [ $? -ne 0 ]; then
echo "$1 IS NOT AVAILABLE(showmount_failed) 999% $2"
else
/usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE " >/dev/null 2>&1
if [ $? -ne 0 ]; then
# Remove "/share" prefix as required for QNAP NFS servers
SHARE1=${SHARE#/share}
/usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE1 " >/dev/null 2>&1
fi
if [ $? -ne 0 ]; then
echo "$1 IS NOT AVAILABLE(showmount_failed_to_find_share) 999% $2"
else
/bin/df -kP $2 | tail -n +2
fi
fi
fi
done
--- End of - Addition code, to check nfs mounts ---
Regards Mal
On 02/04/15 07:13, Henrik Størner wrote:
Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs? There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple.
That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems.
Regards, Henrik
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Thanks all for your detailed responses. I will just monitor the NFS host.
On Apr 1, 2015, at 6:41 PM, Malcolm <fragbaitmm at gmail.com> wrote:
At my work, my team developed some "safe" coding to list NFS filesystems in df. It uses the showmount command to list exports on the NFS server - if this succeeds, it should be safe to run df on the filesystem.
The comment in the code references the URL http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html but it doesn't seem to exist now.
It's ugly and no doubt could be cleaned up and improved... but it does work, and it's handy to know which clients are affected when a NFS volume becomes full.
Update to xymonclient-linux.sh :
At the end of the code for "df", add these lines :
--- Start - Addition code, to check nfs mounts ---
Added following code, based on http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html
/bin/bash -c "/bin/egrep $'[ \t]nfs[ \t]' /etc/mtab" |
/bin/egrep -v "ignore|indirect" |
/usr/bin/awk '{print $1, $2}' |
while read line do set -- $line NFSHOST=echo $1 | /usr/bin/awk -F: '{print $1}'SHARE=echo $1 | /usr/bin/awk -F: '{print $2}'/bin/ping $NFSHOST -c 5 >/dev/null 2>&1 if [ $? -ne 0 ] then echo "$1 IS NOT RESPONDING(ping_failed) 999% $2" else # Ensure NFS connection is active BEFORE running the "showmount -e" command. /bin/df -k $2 > /dev/null 2>&1 & /bin/sleep 5 ; kill $! > /dev/null 2>&1 /usr/sbin/showmount -e $NFSHOST >/dev/null 2>&1 if [ $? -ne 0 ]; then echo "$1 IS NOT AVAILABLE(showmount_failed) 999% $2" else /usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE " >/dev/null 2>&1 if [ $? -ne 0 ]; then # Remove "/share" prefix as required for QNAP NFS servers SHARE1=${SHARE#/share} /usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE1 " >/dev/null 2>&1 fi if [ $? -ne 0 ]; then echo "$1 IS NOT AVAILABLE(showmount_failed_to_find_share) 999% $2" else /bin/df -kP $2 | tail -n +2 fi fi fi done--- End of - Addition code, to check nfs mounts ---
Regards Mal
On 02/04/15 07:13, Henrik Størner wrote:
Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs? There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple.
That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems.
Regards, Henrik
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
participants (5)
-
fragbaitmm@gmail.com
-
Galen.Johnson@sas.com
-
henrik@hswn.dk
-
jlaidman@rebel-it.com.au
-
paul@arrowtel.net