Hello All,
I've read every post on the list, but still cannot figure out why I can't get rrds to appear.
I've written a custom script that does something like:
/home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
On 1.2.3.4 (server), my hobbitserver.cfg has this:
TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,vmstat,iostat,netstat,temperature,apache,bind,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,Memoryutil=ncv"
The reason the test is memoryutil is because it's memory utilization gotta via snmp for an ironport device.
My interpretation of the hobbitd_rrd.8 page, and this post: http://www.hswn.dk/hobbiton/2005/06/msg00261.html gives me reason to believe that I don't have to do anything but have a script pass bb a "$test (=|:) value" value, and tell TEST2RRD "$test=ncv", and I'd be graphing in no time. However, my ~hobbit/rrd/$hostname/ still doesn't have a $test.rrd.
Am I missing something? Doing something wrong? Assuming some things I shouldn't be? Anyone have the ncv stuff working?
Thanks
Dan
Sorry for the hasty followup, but does anyone have any advice?
Thanks
On 11/10/05, Dan Vande More <bigdan at gmail.com> wrote:
Hello All,
I've read every post on the list, but still cannot figure out why I can't get rrds to appear.
I've written a custom script that does something like:
/home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
On 1.2.3.4 (server), my hobbitserver.cfg has this:
TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,vmstat,iostat,netstat,temperature,apache,bind,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,Memoryutil=ncv"
The reason the test is memoryutil is because it's memory utilization gotta via snmp for an ironport device.
My interpretation of the hobbitd_rrd.8 page, and this post: http://www.hswn.dk/hobbiton/2005/06/msg00261.html gives me reason to believe that I don't have to do anything but have a script pass bb a "$test (=|:) value" value, and tell TEST2RRD "$test=ncv", and I'd be graphing in no time. However, my ~hobbit/rrd/$hostname/ still doesn't have a $test.rrd.
Am I missing something? Doing something wrong? Assuming some things I shouldn't be? Anyone have the ncv stuff working?
Thanks
Dan
Ok, I found the issue.
Though I've written dozens of bb clients, I've never come across exactly how BB expects the information. This might be hobbit specific however. In my original email, I stated I was sending the data from my script like:
home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
This is not the correct way to do this hobbit-wise. You need to put a \n in between the color status and the data you're pushing via ncv. That is:
home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green\nMemoryutil : 11"
Perhaps it's just my interpretation, but I think we should reword the documentation to explicitly say we do not need to do any --extra-script or --extra-tests if one is using ncv.
Finally, I perused a lot of the man pages and code for the past few hours, and only found one error - and that was a spelling error. In ~hobbit/server/etc/hobbitgraph.cfg, temperature is misspelled:
comment out the [tempeature] section above, and
I'm now using ncv for collecting 8 datapoints and it appears to collect and graph data flawlessly. Thanks for the solid implementation Henrik.
-Dan
On 11/11/05, Dan Vande More <bigdan at gmail.com> wrote:
Sorry for the hasty followup, but does anyone have any advice?
Thanks
On 11/10/05, Dan Vande More <bigdan at gmail.com> wrote:
Hello All,
I've read every post on the list, but still cannot figure out why I can't get rrds to appear.
I've written a custom script that does something like:
/home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
On 1.2.3.4 (server), my hobbitserver.cfg has this:
TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,vmstat,iostat,netstat,temperature,apache,bind,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,Memoryutil=ncv"
The reason the test is memoryutil is because it's memory utilization gotta via snmp for an ironport device.
My interpretation of the hobbitd_rrd.8 page, and this post: http://www.hswn.dk/hobbiton/2005/06/msg00261.html gives me reason to believe that I don't have to do anything but have a script pass bb a "$test (=|:) value" value, and tell TEST2RRD "$test=ncv", and I'd be graphing in no time. However, my ~hobbit/rrd/$hostname/ still doesn't have a $test.rrd.
Am I missing something? Doing something wrong? Assuming some things I shouldn't be? Anyone have the ncv stuff working?
Thanks
Dan
On Fri, Nov 11, 2005 at 09:57:31PM -0600, Dan Vande More wrote:
Ok, I found the issue.
Though I've written dozens of bb clients, I've never come across exactly how BB expects the information. This might be hobbit specific
It is a result of how the NCV handler in hobbitd_rrd expects data to be formatted, so yes - this is hobbit specific.
however. In my original email, I stated I was sending the data from my script like:
home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
This is not the correct way to do this hobbit-wise. You need to put a \n in between the color status and the data you're pushing via ncv.
I've updated the hobbitd_rrd man page and added this to the section on how to use the NCV data:
Note that each "NAME : value" must be on a line by itself. If you have a custom script generating the status- or data-message that is fed into the NCV handler, make sure it inserts a newline before each of the data-items you want to track.
That should hopefully make it clearer - I'm sorry it bit you.
Perhaps it's just my interpretation, but I think we should reword the documentation to explicitly say we do not need to do any --extra-script or --extra-tests if one is using ncv.
Agreed - did that as well.
Finally, I perused a lot of the man pages and code for the past few hours, and only found one error - and that was a spelling error. In ~hobbit/server/etc/hobbitgraph.cfg, temperature is misspelled:
comment out the [tempeature] section above, and
Fixed :-)
I'm now using ncv for collecting 8 datapoints and it appears to collect and graph data flawlessly. Thanks for the solid implementation Henrik.
Thank you for reporting what you found - that is the kind of feedback I need to make Hobbit better.
Regards, Henrik
To expand on this some and provide ultimate flexibility how about an optional NCV configuration file? When used, this file could be ordered by TEST, with HOSTs being qualified, and appropriate pattern matching of course (sort of the opposite of hobbit-alerts.cfg). Each entry under the test would contain the text to look for (it could be different for a test/host pair) and the graphing parameters for that specific combination. I would envision:
TEST=pg_util Page_Utilization,GAUGE,0:100 TEST=sp_util Spool_Utilization,GAUGE,0:100 HOST=vm1 Queue_File,GAUGE,0:100 Data_File,GAUGE,0:100 HOST=^vse.* GRAPH=pqueue
Where the pg_util graph will look for the text 'Page_Utilization' and graph the white space delimited value after it (after = or :). And sp_util will look for Spool_Utilization for vm1 and graph that data. The graphing parameters will come from hobbitgraph.cfg (as usual), with the default being the test name. For hosts beginning with 'vse' it will look for two values, Queue_File and Data_File and use the pqueue graphing parameters. Strange, but true, I have a column that has a different number of data points with different names depending upon the host (they are both considered spool utilization).
I certainly don't know all of the variations, but this may eliminate the need for most extra tests.
Henrik Stoerner wrote:
On Fri, Nov 11, 2005 at 09:57:31PM -0600, Dan Vande More wrote:
Ok, I found the issue.
Though I've written dozens of bb clients, I've never come across exactly how BB expects the information. This might be hobbit specific
It is a result of how the NCV handler in hobbitd_rrd expects data to be formatted, so yes - this is hobbit specific.
however. In my original email, I stated I was sending the data from my script like:
home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
This is not the correct way to do this hobbit-wise. You need to put a \n in between the color status and the data you're pushing via ncv.
I've updated the hobbitd_rrd man page and added this to the section on how to use the NCV data:
Note that each "NAME : value" must be on a line by itself. If you have a custom script generating the status- or data-message that is fed into the NCV handler, make sure it inserts a newline before each of the data-items you want to track.
That should hopefully make it clearer - I'm sorry it bit you.
Perhaps it's just my interpretation, but I think we should reword the documentation to explicitly say we do not need to do any --extra-script or --extra-tests if one is using ncv.
Agreed - did that as well.
Finally, I perused a lot of the man pages and code for the past few hours, and only found one error - and that was a spelling error. In ~hobbit/server/etc/hobbitgraph.cfg, temperature is misspelled:
comment out the [tempeature] section above, and
Fixed :-)
I'm now using ncv for collecting 8 datapoints and it appears to collect and graph data flawlessly. Thanks for the solid implementation Henrik.
Thank you for reporting what you found - that is the kind of feedback I need to make Hobbit better.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
Brilliant detective work Dan. I am trying something similar, but haven't had the good luck that you enjoy.
Here is the status message being produced for a column called pg_util (Page Utilization):
status vm1.pg_util green 12 Nov 2005 10:55:57 Page Utilization OK pg_util : 1
z/VM Version 4 Release 4.0, service level 0402 (32-bit) EXTENT EXTENT TOTAL PAGES HIGH % VOLID RDEV START END PAGES IN USE PAGE USED
440RES 0124 257 390 24120 529 559 2% PAGSP1 0102 301 3338 546840 0 0 0% ------ ------ ---- SUMMARY 570960 529 1% USABLE 570960 529 1%
TEST2RRD contains 'pg_util=ncv' (it's not at the end, does it need to be?) and
NCV_pg_util="pg_util:GAUGE"
I've disabled the code in the extra script for this test and removed it from the --extra-tests list. I've verified with ps that hobbitd_channel and hobbitd_larrd aren't using them anymore. I still have a custom graph defined for this test, is that allowed using NCV or doesn't it care?
Dan Vande More wrote:
Ok, I found the issue.
Though I've written dozens of bb clients, I've never come across exactly how BB expects the information. This might be hobbit specific however. In my original email, I stated I was sending the data from my script like:
home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
This is not the correct way to do this hobbit-wise. You need to put a \n in between the color status and the data you're pushing via ncv. That is:
home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green\nMemoryutil : 11"
Perhaps it's just my interpretation, but I think we should reword the documentation to explicitly say we do not need to do any --extra-script or --extra-tests if one is using ncv.
Finally, I perused a lot of the man pages and code for the past few hours, and only found one error - and that was a spelling error. In ~hobbit/server/etc/hobbitgraph.cfg, temperature is misspelled:
comment out the [tempeature] section above, and
I'm now using ncv for collecting 8 datapoints and it appears to collect and graph data flawlessly. Thanks for the solid implementation Henrik.
-Dan
On 11/11/05, Dan Vande More <bigdan at gmail.com> wrote:
Sorry for the hasty followup, but does anyone have any advice?
Thanks
On 11/10/05, Dan Vande More <bigdan at gmail.com> wrote:
Hello All,
I've read every post on the list, but still cannot figure out why I can't get rrds to appear.
I've written a custom script that does something like:
/home/monitor/bb/bin/bb 1.2.3.4 "status iron1-mgmt.Memoryutil green Memoryutil : 11"
On 1.2.3.4 (server), my hobbitserver.cfg has this:
TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,vmstat,iostat,netstat,temperature,apache,bind,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,Memoryutil=ncv"
The reason the test is memoryutil is because it's memory utilization gotta via snmp for an ironport device.
My interpretation of the hobbitd_rrd.8 page, and this post: http://www.hswn.dk/hobbiton/2005/06/msg00261.html gives me reason to believe that I don't have to do anything but have a script pass bb a "$test (=|:) value" value, and tell TEST2RRD "$test=ncv", and I'd be graphing in no time. However, my ~hobbit/rrd/$hostname/ still doesn't have a $test.rrd.
Am I missing something? Doing something wrong? Assuming some things I shouldn't be? Anyone have the ncv stuff working?
Thanks
Dan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
On 11/12/05, Rich Smrcina <rsmrcina at wi.rr.com> wrote:
Brilliant detective work Dan. I am trying something similar, but haven't had the good luck that you enjoy. I am a very bad C coder. I'd call it luck too.:)
Here is the status message being produced for a column called pg_util (Page Utilization):
status vm1.pg_util green 12 Nov 2005 10:55:57 Page Utilization OK pg_util : 1
z/VM Version 4 Release 4.0, service level 0402 (32-bit) EXTENT EXTENT TOTAL PAGES HIGH % VOLID RDEV START END PAGES IN USE PAGE USED
440RES 0124 257 390 24120 529 559 2% PAGSP1 0102 301 3338 546840 0 0 0% ------ ------ ---- SUMMARY 570960 529 1% USABLE 570960 529 1%
Looks good;)
TEST2RRD contains 'pg_util=ncv' (it's not at the end, does it need to be?) and Nope, shouldn't matter where it is.
NCV_pg_util="pg_util:GAUGE"
I've disabled the code in the extra script for this test and removed it from the --extra-tests list. I've verified with ps that hobbitd_channel and hobbitd_larrd aren't using them anymore. I still have a custom graph defined for this test, is that allowed using NCV or doesn't it care?
Custom graphing (with ncv) is really a two step process. The first and most important is collecting the data into an rrd, which is simple and done by adding your COLUMNNAME=ncv to TEST2RRD and correctly passing data back to hobbit:) I used "rrdtool dump foo.rrd > tmp.xml" to verify that it was actually collecting some data.
I would venture to guess that as long as your test/column names stay the same, there will be no problems in switching from --extra-tests to ncv.
The second step, of course, is actually graphing it using data from the rrd. I found this to be the most difficult step, if only because I skim documentation - I don't read unless I'm desperate. Again, I'm going out on a limb to say as long as your datasources/testnames/Y values are all the same, your graph configuration should work post-ncv conversion.
Good luck
Dan
Dan Vande More wrote:
Custom graphing (with ncv) is really a two step process. The first and most important is collecting the data into an rrd, which is simple and done by adding your COLUMNNAME=ncv to TEST2RRD and correctly passing data back to hobbit:) I used "rrdtool dump foo.rrd > tmp.xml" to verify that it was actually collecting some data.
My rrd file isn't even being updated. That leads me to believe that something in the configuration process is missing or wrong.
I would venture to guess that as long as your test/column names stay the same, there will be no problems in switching from --extra-tests to ncv.
The second step, of course, is actually graphing it using data from the rrd. I found this to be the most difficult step, if only because I skim documentation - I don't read unless I'm desperate. Again, I'm going out on a limb to say as long as your datasources/testnames/Y values are all the same, your graph configuration should work post-ncv conversion.
Good luck
Thanks.
Dan
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
On 11/12/05, Rich Smrcina <rsmrcina at wi.rr.com> wrote:
Dan Vande More wrote:
Custom graphing (with ncv) is really a two step process. The first and most important is collecting the data into an rrd, which is simple and done by adding your COLUMNNAME=ncv to TEST2RRD and correctly passing data back to hobbit:) I used "rrdtool dump foo.rrd > tmp.xml" to verify that it was actually collecting some data.
My rrd file isn't even being updated. That leads me to believe that something in the configuration process is missing or wrong.
Ok, I'll share how I "debugged" hobbit, but only if you promise not to laugh. I'm a seek and destroy debugger, so I add printfs inside of loops I think should be executed, and if they're not executed, I'll find out why. Since this isn't perl, it's not as easy as 'print "I'm executing loop foo...\n"'. Fortunately, Henrick has made it almost as easy. Instead of print/printf/sprintf, hobbit has an errprintf function that works quite handily.
My recommendations are these:
#1. Get into your hobbit source directory, I'd assume something like: /home/hobbit/hobbit-4.1.2. #2. Put a line in hobbitd/do_rrd.c on or about line 250 (Which is where you see the if and else ifs, you want to be _above_ these. My line is something like:
if (strcmp(id, "ncv") == 0) errprintf("hostname %s,testname %s qualifies for an ncv with id %s\n", hostname, testname, id);
Note: There's a strcmp(id, ncv) later on, but it won't affect this test.
#3. Now, assuing your TEST2RRD is working, hobbit will log that (more on that later). A bit further down in do_rrd.c, if it matches ncv, it will pass that on to a function appropriately named do_ncv_rrd(). Now, edit hobbitd/rrd/do_ncv.c and add a line on our about line 32 that says something like:
errprintf("hostname %s testname %s msg %s made ncv\n", hostname, testname, msg);
(Just to know you got inside of the do_ncv function.)
#4. On our about line 58, which is clearly after it assigns name & value (From your msg: 'name : value' or 'name = value'), add a line like:
errprintf("hostname %s testname %s msg %s had name/value of %s : %s", hostname, testname, msg, name, value);
#5. Re-compile hobbit so these changes take effect. My line was something like:
sudo make clean && sudo make && sudo make install && sudo rm /var/log/hobbit/rrd-status.log && sudo -u hobbit /home/hobbit/server/hobbit.sh stop && sudo -u hobbit /home/hobbit/server/hobbit.sh start **
Ok, that's it. If any of your steps didn't log all of those messages, it didn't make it that far. Troubleshoot these backwards. Now, errprintf seems smart enought to know which module it's running under, so it will log to the appropriate modules' log. In this case, you'll find these messages under /var/log/hobbit/rrd-status.log. Look for the tests you're running and if these lines show up in that log file.
If I were any smarter, I would have made some kinda "if (defined(debug_ncv)) errprintf(foo)" lines around these three (or more) areas and submitted the diffs to the project for inclusion, but alas, I was too hurried.
If you get stuck or have questions, just ask.
Good luck.
Dan
I would venture to guess that as long as your test/column names stay the same, there will be no problems in switching from --extra-tests to ncv.
The second step, of course, is actually graphing it using data from the rrd. I found this to be the most difficult step, if only because I skim documentation - I don't read unless I'm desperate. Again, I'm going out on a limb to say as long as your datasources/testnames/Y values are all the same, your graph configuration should work post-ncv conversion.
Good luck
Thanks.
Dan
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Great detail Dan, thanks for all of the info. I'll give it a try later this afternoon.
Dan Vande More wrote:
On 11/12/05, Rich Smrcina <rsmrcina at wi.rr.com> wrote:
Dan Vande More wrote:
Custom graphing (with ncv) is really a two step process. The first and most important is collecting the data into an rrd, which is simple and done by adding your COLUMNNAME=ncv to TEST2RRD and correctly passing data back to hobbit:) I used "rrdtool dump foo.rrd > tmp.xml" to verify that it was actually collecting some data.
My rrd file isn't even being updated. That leads me to believe that something in the configuration process is missing or wrong.
Ok, I'll share how I "debugged" hobbit, but only if you promise not to laugh. I'm a seek and destroy debugger, so I add printfs inside of loops I think should be executed, and if they're not executed, I'll find out why. Since this isn't perl, it's not as easy as 'print "I'm executing loop foo...\n"'. Fortunately, Henrick has made it almost as easy. Instead of print/printf/sprintf, hobbit has an errprintf function that works quite handily.
My recommendations are these:
#1. Get into your hobbit source directory, I'd assume something like: /home/hobbit/hobbit-4.1.2. #2. Put a line in hobbitd/do_rrd.c on or about line 250 (Which is where you see the if and else ifs, you want to be _above_ these. My line is something like:
if (strcmp(id, "ncv") == 0) errprintf("hostname %s,testname %s qualifies for an ncv with id %s\n", hostname, testname, id);
Note: There's a strcmp(id, ncv) later on, but it won't affect this test.
#3. Now, assuing your TEST2RRD is working, hobbit will log that (more on that later). A bit further down in do_rrd.c, if it matches ncv, it will pass that on to a function appropriately named do_ncv_rrd(). Now, edit hobbitd/rrd/do_ncv.c and add a line on our about line 32 that says something like:
errprintf("hostname %s testname %s msg %s made ncv\n", hostname, testname, msg);
(Just to know you got inside of the do_ncv function.)
#4. On our about line 58, which is clearly after it assigns name & value (From your msg: 'name : value' or 'name = value'), add a line like:
errprintf("hostname %s testname %s msg %s had name/value of %s : %s", hostname, testname, msg, name, value);
#5. Re-compile hobbit so these changes take effect. My line was something like:
sudo make clean && sudo make && sudo make install && sudo rm /var/log/hobbit/rrd-status.log && sudo -u hobbit /home/hobbit/server/hobbit.sh stop && sudo -u hobbit /home/hobbit/server/hobbit.sh start **
Ok, that's it. If any of your steps didn't log all of those messages, it didn't make it that far. Troubleshoot these backwards. Now, errprintf seems smart enought to know which module it's running under, so it will log to the appropriate modules' log. In this case, you'll find these messages under /var/log/hobbit/rrd-status.log. Look for the tests you're running and if these lines show up in that log file.
If I were any smarter, I would have made some kinda "if (defined(debug_ncv)) errprintf(foo)" lines around these three (or more) areas and submitted the diffs to the project for inclusion, but alas, I was too hurried.
If you get stuck or have questions, just ask.
Good luck.
Dan
I would venture to guess that as long as your test/column names stay the same, there will be no problems in switching from --extra-tests to ncv.
The second step, of course, is actually graphing it using data from the rrd. I found this to be the most difficult step, if only because I skim documentation - I don't read unless I'm desperate. Again, I'm going out on a limb to say as long as your datasources/testnames/Y values are all the same, your graph configuration should work post-ncv conversion.
Good luck
Thanks.
Dan
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
OK, I think I've found the problem. The NCV routine strips out anything that isn't a letter or a number from the column name.
I found this message in the log:
2005-11-13 22:01:21 RRD error updating /home/hobbit/data/rrd/vm1/pg_util.rrd from 192.168.201.1: unknown DS name 'pgutil'
Dan Vande More wrote:
On 11/12/05, Rich Smrcina <rsmrcina at wi.rr.com> wrote:
Dan Vande More wrote:
Custom graphing (with ncv) is really a two step process. The first and most important is collecting the data into an rrd, which is simple and done by adding your COLUMNNAME=ncv to TEST2RRD and correctly passing data back to hobbit:) I used "rrdtool dump foo.rrd > tmp.xml" to verify that it was actually collecting some data.
My rrd file isn't even being updated. That leads me to believe that something in the configuration process is missing or wrong.
Ok, I'll share how I "debugged" hobbit, but only if you promise not to laugh. I'm a seek and destroy debugger, so I add printfs inside of loops I think should be executed, and if they're not executed, I'll find out why. Since this isn't perl, it's not as easy as 'print "I'm executing loop foo...\n"'. Fortunately, Henrick has made it almost as easy. Instead of print/printf/sprintf, hobbit has an errprintf function that works quite handily.
My recommendations are these:
#1. Get into your hobbit source directory, I'd assume something like: /home/hobbit/hobbit-4.1.2. #2. Put a line in hobbitd/do_rrd.c on or about line 250 (Which is where you see the if and else ifs, you want to be _above_ these. My line is something like:
if (strcmp(id, "ncv") == 0) errprintf("hostname %s,testname %s qualifies for an ncv with id %s\n", hostname, testname, id);
Note: There's a strcmp(id, ncv) later on, but it won't affect this test.
#3. Now, assuing your TEST2RRD is working, hobbit will log that (more on that later). A bit further down in do_rrd.c, if it matches ncv, it will pass that on to a function appropriately named do_ncv_rrd(). Now, edit hobbitd/rrd/do_ncv.c and add a line on our about line 32 that says something like:
errprintf("hostname %s testname %s msg %s made ncv\n", hostname, testname, msg);
(Just to know you got inside of the do_ncv function.)
#4. On our about line 58, which is clearly after it assigns name & value (From your msg: 'name : value' or 'name = value'), add a line like:
errprintf("hostname %s testname %s msg %s had name/value of %s : %s", hostname, testname, msg, name, value);
#5. Re-compile hobbit so these changes take effect. My line was something like:
sudo make clean && sudo make && sudo make install && sudo rm /var/log/hobbit/rrd-status.log && sudo -u hobbit /home/hobbit/server/hobbit.sh stop && sudo -u hobbit /home/hobbit/server/hobbit.sh start **
Ok, that's it. If any of your steps didn't log all of those messages, it didn't make it that far. Troubleshoot these backwards. Now, errprintf seems smart enought to know which module it's running under, so it will log to the appropriate modules' log. In this case, you'll find these messages under /var/log/hobbit/rrd-status.log. Look for the tests you're running and if these lines show up in that log file.
If I were any smarter, I would have made some kinda "if (defined(debug_ncv)) errprintf(foo)" lines around these three (or more) areas and submitted the diffs to the project for inclusion, but alas, I was too hurried.
If you get stuck or have questions, just ask.
Good luck.
Dan
I would venture to guess that as long as your test/column names stay the same, there will be no problems in switching from --extra-tests to ncv.
The second step, of course, is actually graphing it using data from the rrd. I found this to be the most difficult step, if only because I skim documentation - I don't read unless I'm desperate. Again, I'm going out on a limb to say as long as your datasources/testnames/Y values are all the same, your graph configuration should work post-ncv conversion.
Good luck
Thanks.
Dan
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
Oh man, I can't believe I missed that when you asked me. I almost suffered from the same problem, too. You are right, those underscores are not allowed and stripped out. You should be able to update your rrd ds to not have underscores by exporting to xml (rrdtool dump foo.rrd > foo.xml), search and replace, import from xml (rrdtool restore foo.xml foo.rrd) to rrd.
On 11/14/05, Rich Smrcina <rsmrcina at wi.rr.com> wrote:
OK, I think I've found the problem. The NCV routine strips out anything that isn't a letter or a number from the column name.
I found this message in the log:
2005-11-13 22:01:21 RRD error updating /home/hobbit/data/rrd/vm1/pg_util.rrd from 192.168.201.1: unknown DS name 'pgutil'
Dan Vande More wrote:
On 11/12/05, Rich Smrcina <rsmrcina at wi.rr.com> wrote:
Dan Vande More wrote:
Custom graphing (with ncv) is really a two step process. The first and most important is collecting the data into an rrd, which is simple and done by adding your COLUMNNAME=ncv to TEST2RRD and correctly passing data back to hobbit:) I used "rrdtool dump foo.rrd > tmp.xml" to verify that it was actually collecting some data.
My rrd file isn't even being updated. That leads me to believe that something in the configuration process is missing or wrong.
Ok, I'll share how I "debugged" hobbit, but only if you promise not to laugh. I'm a seek and destroy debugger, so I add printfs inside of loops I think should be executed, and if they're not executed, I'll find out why. Since this isn't perl, it's not as easy as 'print "I'm executing loop foo...\n"'. Fortunately, Henrick has made it almost as easy. Instead of print/printf/sprintf, hobbit has an errprintf function that works quite handily.
My recommendations are these:
#1. Get into your hobbit source directory, I'd assume something like: /home/hobbit/hobbit-4.1.2. #2. Put a line in hobbitd/do_rrd.c on or about line 250 (Which is where you see the if and else ifs, you want to be _above_ these. My line is something like:
if (strcmp(id, "ncv") == 0) errprintf("hostname %s,testname %s qualifies for an ncv with id %s\n", hostname, testname, id);
Note: There's a strcmp(id, ncv) later on, but it won't affect this test.
#3. Now, assuing your TEST2RRD is working, hobbit will log that (more on that later). A bit further down in do_rrd.c, if it matches ncv, it will pass that on to a function appropriately named do_ncv_rrd(). Now, edit hobbitd/rrd/do_ncv.c and add a line on our about line 32 that says something like:
errprintf("hostname %s testname %s msg %s made ncv\n", hostname, testname, msg);
(Just to know you got inside of the do_ncv function.)
#4. On our about line 58, which is clearly after it assigns name & value (From your msg: 'name : value' or 'name = value'), add a line like:
errprintf("hostname %s testname %s msg %s had name/value of %s : %s", hostname, testname, msg, name, value);
#5. Re-compile hobbit so these changes take effect. My line was something like:
sudo make clean && sudo make && sudo make install && sudo rm /var/log/hobbit/rrd-status.log && sudo -u hobbit /home/hobbit/server/hobbit.sh stop && sudo -u hobbit /home/hobbit/server/hobbit.sh start **
Ok, that's it. If any of your steps didn't log all of those messages, it didn't make it that far. Troubleshoot these backwards. Now, errprintf seems smart enought to know which module it's running under, so it will log to the appropriate modules' log. In this case, you'll find these messages under /var/log/hobbit/rrd-status.log. Look for the tests you're running and if these lines show up in that log file.
If I were any smarter, I would have made some kinda "if (defined(debug_ncv)) errprintf(foo)" lines around these three (or more) areas and submitted the diffs to the project for inclusion, but alas, I was too hurried.
If you get stuck or have questions, just ask.
Good luck.
Dan
I would venture to guess that as long as your test/column names stay the same, there will be no problems in switching from --extra-tests to ncv.
The second step, of course, is actually graphing it using data from the rrd. I found this to be the most difficult step, if only because I skim documentation - I don't read unless I'm desperate. Again, I'm going out on a limb to say as long as your datasources/testnames/Y values are all the same, your graph configuration should work post-ncv conversion.
Good luck
Thanks.
Dan
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On Mon, Nov 14, 2005 at 06:35:22AM -0600, Rich Smrcina wrote:
OK, I think I've found the problem. The NCV routine strips out anything that isn't a letter or a number from the column name.
I found this message in the log:
2005-11-13 22:01:21 RRD error updating /home/hobbit/data/rrd/vm1/pg_util.rrd from 192.168.201.1: unknown DS name 'pgutil'
That is because the RRD library only allows data-set names to contain letters and numbers.
Henrik
Perhaps the underscore is also considered a valid character. I can process dataset names with underscores using the extra tests and script. Also, I changed do_ncv.c to allow underscores in the dataset name.
if ( ((*inp >= 'A') && (*inp <= 'Z')) ||
((*inp >= 'a') && (*inp <= 'z')) ||
((*inp >= '0') && (*inp <= '9')) ||
((*inp == '_') ) ) {Here's a dump from my rrd file:
hobbit at hobbit:~/data/rrd/vm1> rrdtool fetch pg_util.rrd AVERAGE -r 900 -s -60m pg_util
1131975600: 1.0000000000e+00 1131975900: 1.0000000000e+00 1131976200: 1.0000000000e+00
Now that my rrd file is being updated, I have to figure out why the graph isn't appearing.
Henrik Stoerner wrote:
On Mon, Nov 14, 2005 at 06:35:22AM -0600, Rich Smrcina wrote:
OK, I think I've found the problem. The NCV routine strips out anything that isn't a letter or a number from the column name.
I found this message in the log:
2005-11-13 22:01:21 RRD error updating /home/hobbit/data/rrd/vm1/pg_util.rrd from 192.168.201.1: unknown DS name 'pgutil'
That is because the RRD library only allows data-set names to contain letters and numbers.
Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
On Mon, Nov 14, 2005 at 08:54:07AM -0600, Rich Smrcina wrote:
Perhaps the underscore is also considered a valid character. I can process dataset names with underscores using the extra tests and script. Also, I changed do_ncv.c to allow underscores in the dataset name.
if ( ((*inp >= 'A') && (*inp <= 'Z')) || ((*inp >= 'a') && (*inp <= 'z')) || ((*inp >= '0') && (*inp <= '9')) || ((*inp == '_') ) ) {
Ah - you're right, the rrdcreate(1) man-page says as much. Missed that when I first did the NCV module.
I've fixed that in my code.
Regards, Henrik
Thanks, Henrik!
Henrik Stoerner wrote:
On Mon, Nov 14, 2005 at 08:54:07AM -0600, Rich Smrcina wrote:
Perhaps the underscore is also considered a valid character. I can process dataset names with underscores using the extra tests and script. Also, I changed do_ncv.c to allow underscores in the dataset name.
if ( ((*inp >= 'A') && (*inp <= 'Z')) || ((*inp >= 'a') && (*inp <= 'z')) || ((*inp >= '0') && (*inp <= '9')) || ((*inp == '_') ) ) {Ah - you're right, the rrdcreate(1) man-page says as much. Missed that when I first did the NCV module.
I've fixed that in my code.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
participants (3)
-
bigdan＠gmail.com
-
henrik＠hswn.dk
-
rsmrcina＠wi.rr.com