upgrade to 4.1.2 breaks netapp qtree graphing
Upgraded from hobbit 4.0.4 -> 4.1.2 and hit my first bug: the data collected from my filerstats2bb.pl ext script, which gets data from my NetApp filer and allows me to graph the qtrees, volumes & inodes, no longer works. The data collection still works fine, as well as the display of the results, but only one the first qtree/inode/disk is graphed. The rest are not.
hobbitgraph.cfg definitions are still the same as before, so I poked around the hobbitd/rrd/do_disk.c and compared it to the one used in 4.0.4, but I think that is working well since it populates the rrd files...?
hobbitgraph.c is where I should look, right?
I'm kinda lost here, and of course we need to see our filer graphs to check on a problem, which is how a co-worker disovered the bug in the first place!
Tom
Tom Georgoulias wrote:
Upgraded from hobbit 4.0.4 -> 4.1.2 and hit my first bug: the data collected from my filerstats2bb.pl ext script, which gets data from my NetApp filer and allows me to graph the qtrees, volumes & inodes, no longer works. The data collection still works fine, as well as the display of the results, but only one the first qtree/inode/disk is graphed. The rest are not.
hobbitgraph.cfg definitions are still the same as before, so I poked around the hobbitd/rrd/do_disk.c and compared it to the one used in 4.0.4, but I think that is working well since it populates the rrd files...?
hobbitgraph.c is where I should look, right?
Doh, forgot to add:
the graphs displayed under the "trends" column are fine, just the graphs under the specific parameter are having this problem (i.e. graph in qtree column shows first qtree, qtree graphs in trends shows all qtrees on filer.)
Tom
On Fri, Oct 14, 2005 at 09:17:18AM -0400, Tom Georgoulias wrote:
the graphs displayed under the "trends" column are fine, just the graphs under the specific parameter are having this problem (i.e. graph in qtree column shows first qtree, qtree graphs in trends shows all qtrees on filer.)
BTW, could you check the timestamps on your inode*rrd files ? Hobbit 4.1.2 has a new feature where it ignores stale rrd files. If some of your inode rrd files are not being updated, then you would see only the recently modified ones on the "inode" status page.
Henrik
On Fri, Oct 14, 2005 at 09:04:54AM -0400, Tom Georgoulias wrote:
Upgraded from hobbit 4.0.4 -> 4.1.2 and hit my first bug: the data collected from my filerstats2bb.pl ext script, which gets data from my NetApp filer and allows me to graph the qtrees, volumes & inodes, no longer works. The data collection still works fine, as well as the display of the results, but only one the first qtree/inode/disk is graphed. The rest are not.
Is this on both the individual status display and the one in the "trends" column, or just one of them ?
hobbitgraph.cfg definitions are still the same as before, so I poked around the hobbitd/rrd/do_disk.c and compared it to the one used in 4.0.4, but I think that is working well since it populates the rrd files...?
If the RRD files are being updated, then rrd/do_disk is working OK.
hobbitgraph.c is where I should look, right?
Sounds like it, but there are some complicating factors.
I did some testing here, and copying a set of disk*rrd files to inode-ditto's and bringing up an "inode" status looks OK. So I think it's some upgrade problem you've run into.
Could you try and compare the following files:
bb-hostsvc.sh (from the installed cgi-bin directory) with the hobbitd/bb-hostsvc.sh from the hobbit-4.1.2 build directory ?
hobbitgraph.sh (from the installed cgi-bin directory) with the hobbitd/hobbitgraph.sh from the hobbit-4.1.2 build directory ?
hobbitgraph.cfg (from the installed server/etc/ directory) with the hobbitd/etcfiles/hobbitgraph.cfg from the hobbit-4.1.2 build directory ?
The two cgi-scripts - if there are any differences - can be copied over directly from the build directory. The hobbitgraph.cfg may need some hand-tweaking, if you have defined any graphs yourself.
Regards, Henrik
Henrik Stoerner wrote:
Could you try and compare the following files:
- bb-hostsvc.sh (from the installed cgi-bin directory) with the hobbitd/bb-hostsvc.sh from the hobbit-4.1.2 build directory ?
md5sum bb-hostsvc.sh /home/bb/cgi-bin/bb-hostsvc.sh 5c17106fd83c3c5eda051592b945f497 bb-hostsvc.sh 5c17106fd83c3c5eda051592b945f497 /home/bb/cgi-bin/bb-hostsvc.sh
- hobbitgraph.sh (from the installed cgi-bin directory) with the hobbitd/hobbitgraph.sh from the hobbit-4.1.2 build directory ?
md5sum hobbitgraph.sh /home/bb/cgi-bin/hobbitgraph.sh f778ab8f6fd0e0e3cc5af665628d0889 hobbitgraph.sh f778ab8f6fd0e0e3cc5af665628d0889 /home/bb/cgi-bin/hobbitgraph.sh
- hobbitgraph.cfg (from the installed server/etc/ directory) with the hobbitd/etcfiles/hobbitgraph.cfg from the hobbit-4.1.2 build directory ?
md5sum etcfiles/hobbitgraph.cfg /home/bb/server/etc/hobbitgraph.cfg a65dc5f1a199a684010cc97712277025 etcfiles/hobbitgraph.cfg f70f5a1f180fc59cd9b2f71ffb0fb2be /home/bb/server/etc/hobbitgraph.cfg
I have some custom edits to this file, so that is why they won't match. Since we're using inodes in troubleshooting, here is the def I'm using:
[inode] FNPATTERN inode(.*).rrd TITLE inode Utilization YAXIS % Full DEF:p at RRDIDX@=@RRDFN@:pct:AVERAGE LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@ -u 100 -l 0 GPRINT:p at RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p at RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p at RRDIDX@:MIN: \: %5.1lf (min) GPRINT:p at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
I will check this file further and see if my custom edits are still necessary, but would it make sense to blame the def if the graphs in the trends column are right while the inode column is not? I would think that since they likely use the same def, the graph would be broken in both places.
BTW, could you check the timestamps on your inode*rrd files ? Hobbit 4.1.2 has a new feature where it ignores stale rrd files. If some of your inode rrd files are not being updated, then you would see only the recently modified ones on the "inode" status page.
Sure, they look fine (example below). The graphs in the trends column have all the current data and qtrees/inode/disks, and I can query the rrd files themselves with rrdtool and the data looks fine.
ls -l inode,vol,* -rw-r--r-- 1 bb bb 38532 Oct 14 09:34 inode,vol,prod01,.rrd -rw-r--r-- 1 bb bb 38532 Oct 14 09:34 inode,vol,prod02,.rrd -rw-r--r-- 1 bb bb 38532 Oct 14 09:34 inode,vol,vol0,.rrd
Tom Georgoulias wrote:
I have some custom edits to this file, so that is why they won't match. Since we're using inodes in troubleshooting, here is the def I'm using:
[inode] FNPATTERN inode(.*).rrd TITLE inode Utilization YAXIS % Full DEF:p at RRDIDX@=@RRDFN@:pct:AVERAGE LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@ -u 100 -l 0 GPRINT:p at RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p at RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p at RRDIDX@:MIN: \: %5.1lf (min) GPRINT:p at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
I will check this file further and see if my custom edits are still necessary, but would it make sense to blame the def if the graphs in the trends column are right while the inode column is not? I would think that since they likely use the same def, the graph would be broken in both places.
Poked around some more and noticed that the when I view the graph within the inode column display, the URL referenced by that graph with a single value has the parameter "count=1" in it. If I follow that link, it show the historical graphs with 1 filesystem in each. If I make the count=2+, more filesystems show up. Is the count=1 being passed along somehow (maybe as a default) when I first drill down into the column?
Tom
Henrik,
Just wanted to let you know that this bug (Netapp qtree/vol/inode graphing) was fixed in 4.1.2p1. I guess whatever changes you made to fix the disk graphing issue for normal systems also took care of this.
Thanks for the good work on Hobbit.
Tom
participants (2)
-
henrik＠hswn.dk
-
tomg＠mcclatchyinteractive.com