Ooohhh, Charles, don't you think 'ignore' is a little strong? The smileys in your message say one thing, but the wording quite another. Certainly responses to some emails aren't very timely, if they come at all, but I would not fault Henrik for that. I can appreciate that there are formidable demands for his time from many fronts, particularly with a project of this magnitude (not to mention his 'real' job). I think your patch will be very useful and have asked about their inclusion in the next release of Hobbit. I wasn't going to get too worried about it until some alpha or beta type code is available. Charles Goyard wrote:

Stewart wrote :

...

So, I have a bit of a situation.

in that vein, it would be nice if I can get the data stored in a separate RRD file for each interface (the same way that there is a separate RRD file for each partition on a disk test) so that I can use the file @RRDFN@ and @RRDPARAM@ stuff to work, but I'm unsure how to do that.

little help?

Hi,

I posted a patch some time ago that Henrik ignored, but it permits one to do that very thing.

I provide it again in case Henrik notices this time :)

------------ (Date: Thu, 28 Dec 2006 19:07:07 +0100 ) ---- Hi all,

here is a patch that provides two new features to the NCV graph backend. It answers to a request I did on August 2nd (subject: "Dynamic or Split NCV").

- split-ncv : Lets you store an ever-changing set of data on a status column basis. Say, you want to keep track of who is accessing a database. It works by creating a rrd file per ncv line instead of packing a lot of DS in a single rrd file. I use it to keep track of tomcat session number per context, harddrive temperature, database connexions, and a whole bunch of other things, such a the repartition of loaded classes on a jvm (useful to find memory leaks).

- trackmax : it lets you keep track of the maximum value of a dataset, along with the average. It's useful when the maximum is somehow as significant than the average over time. (say, to track a hard-limited ressource, such as telephone lines). I use it to keep track of the maximum number of busy X25 lines on a specific application. It only works for NCV at the time, but I may add it to other tests.

The patch includes the documentation for all this. It applies on today's snapshot and on october' all-in-one. It does not work with the plain release because of a nasty bug in do_ncv.c.

Well, here's the patch. Consider it beta, but yet it works on my production server. I strongly hope it will be included in the mainstream hobbit (I don't like maintaining local patches :)).

------------------------------------------------------------------------

diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.c cvf/hobbitd/do_rrd.c --- upstream-2006-10-03/hobbitd/do_rrd.c 2006-11-24 17:18:09.000000000 +0100 +++ cvf/hobbitd/do_rrd.c 2006-12-15 15:47:58.000000000 +0100 @@ -29,6 +29,7 @@ #include "do_rrd.h"

char *rrddir = NULL; +char *trackmax = NULL; static char *exthandler = NULL; static char **extids = NULL;

@@ -37,11 +38,16 @@ static char rra2[] = "RRA:AVERAGE:0.5:6:576"; static char rra3[] = "RRA:AVERAGE:0.5:24:576"; static char rra4[] = "RRA:AVERAGE:0.5:288:576"; +static char rra5[] = "RRA:MAX:0.5:1:576"; +static char rra6[] = "RRA:MAX:0.5:6:576"; +static char rra7[] = "RRA:MAX:0.5:24:576"; +static char rra8[] = "RRA:MAX:0.5:288:576";

static char *senderip = NULL; static char rrdfn[PATH_MAX]; /* This one used by the modules */ static char filedir[PATH_MAX]; /* This one used here */

+ void setup_exthandler(char *handlerpath, char *ids) { char *p; @@ -100,6 +106,24 @@ rrdfn[sizeof(rrdfn)-1] = '\0'; }

+static int has_trackmax(char *testname) +{ + char testnamecoma[strlen(testname)+3]; + + if(trackmax == NULL) { + return 0; + } + + sprintf(testnamecoma, ",%s,", testname); + if(strstr(trackmax, testnamecoma)) { + return 1; + } + else { + return 0; + } +} + + static int create_and_update_rrd(char *hostname, char *fn, char *creparams[], char *template) { struct stat st; diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.h cvf/hobbitd/do_rrd.h --- upstream-2006-10-03/hobbitd/do_rrd.h 2006-11-24 17:18:09.000000000 +0100 +++ cvf/hobbitd/do_rrd.h 2006-11-24 17:55:53.000000000 +0100 @@ -16,6 +16,7 @@ #include "libbbgen.h"

extern char *rrddir; +extern char *trackmax; extern void setup_exthandler(char *handlerpath, char *ids); extern void update_rrd(char *hostname, char *testname, char *restofmsg, time_t tstamp, char *sender, hobbitrrd_t *ldef);

Only in cvf/hobbitd/: hobbitd_cvfinventaire.c diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.8 cvf/hobbitd/hobbitd_rrd.8 --- upstream-2006-10-03/hobbitd/hobbitd_rrd.8 2006-11-24 17:18:09.000000000 +0100 +++ cvf/hobbitd/hobbitd_rrd.8 2006-11-24 17:55:53.000000000 +0100 @@ -65,6 +65,16 @@ Defines the types of data collected by the "ncv" module in hobbitd_rrd. See below for more information.

+.IP SPLITNCV_testname +The same as NCV_testname, but keeps the data into separate files. That +is, it creates one rrd file per "NAME : value" line found in the +status message. It is useful when the list of NCV lines is varying. + +.IP TRACKMAX +Comma-separated list of columnname for which you want to keep the +maximum values along with the default average values. This only works + for the NCV backend. + .SH COLLECTED DATA The following RRD-file datasets are generated by hobbitd_rrd:

@@ -152,18 +162,22 @@ or "NAME = value". So a generic module in hobbitd_rrd allows for easy tracking of this type of data.

-The "ncv" module will automatically detect all occurrences of -a "NAME : value" or "NAME = value" string in a status message, and -generate an RRD file holding all of the name/value data found in -the message. The colon- or equal-sign must be present - if there is -only whitespace, this module will fail. + The "ncv" module will automatically detect all occurrences of a "NAME + : value" or "NAME = value" string in a status message, and generate an + RRD file holding all of the name/value data found in the message + (unless you use SPLITNCV, see above). The colon- or equal-sign must be + present - if there is only whitespace, this module will fail.

Only the valid letters (A-Z, a-z) and digits (0-9) are used in the dataset names; whitespace and other characters are stripped off -automatically. Only the first 19 characters of a dataset name are -used (this is an RRD limitation). Underscore '_' is not allowed, -even though RRDtool permits this, and will be stripped from the -name. + automatically. Only the first 19 characters of a dataset name are used + (this is an RRD limitation). Underscore '_' is not allowed, even + though RRDtool permits this, and will be stripped from the name. + + When using the alternative SPLITNCV_testname, the dataset name is not + limited in length, and non-valid characters are changed to underscores + instead of being stripped off. The dataset inside the resulting rrd + file is always "lambda.

Note that each "NAME : value" must be on a line by itself. If you have a custom script generating the status- or data-message that is fed @@ -177,12 +191,13 @@ all status-messages for the column COLUMNNAME through the hobbitd_rrd ncv-handler.

-The name of the RRD file will be COLUMNNAME.rrd. +The name of the RRD file will be COLUMNNAME.rrd. When using SPLITNCV, +the name of the RRD file will be COLUMNAME,DATASETNAME.rrd.

By default, all of the datasets are generated as the RRD type "DERIVE" which works for all types of monotonically increasing counters. If you have data that are of the type GAUGE, you can override the default via -an environment variable NCV_COLUMNNAME. +an environment variable NCV_COLUMNNAME (or SPLITNCV_COLUMNAME).

E.g. if you are using the bb-mysqlstatus script from www.deadcat.net to collect data about your MySQL server, it generates a report in the column diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.c cvf/hobbitd/hobbitd_rrd.c --- upstream-2006-10-03/hobbitd/hobbitd_rrd.c 2006-11-24 17:18:09.000000000 +0100 +++ cvf/hobbitd/hobbitd_rrd.c 2006-12-28 18:35:27.000000000 +0100 @@ -74,6 +74,13 @@ rrddir = strdup(xgetenv("BBRRDS")); }

+ /* trackmax initialization */ + if(getenv("TRACKMAX")) { + trackmax = (char*)malloc(strlen(getenv("TRACKMAX"))+3); + sprintf(trackmax, ",%s,", getenv("TRACKMAX")); + dbgprintf("Will track max for: %s\n", trackmax); + } + save_errbuf = 0; setup_signalhandler("hobbitd_rrd"); memset(&sa, 0, sizeof(sa)); @@ -139,7 +146,7 @@ break;

default: - /* Ignore reports with purple, blue or clear - they have no data we want. */ + /* Ignore reports with purple or clear - they have no data we want. */ break; } } diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/rrd/do_ncv.c cvf/hobbitd/rrd/do_ncv.c --- upstream-2006-10-03/hobbitd/rrd/do_ncv.c 2006-11-24 17:18:57.000000000 +0100 +++ cvf/hobbitd/rrd/do_ncv.c 2006-12-28 18:42:58.000000000 +0100 @@ -5,6 +5,7 @@ /* NAME: VALUE */ /* */ /* Copyright (C) 2004-2006 Henrik Storner <henrik at hswn.dk> */ +/* split-ncv added by Charles Goyard November 2006 */ /* */ /* This program is released under the GNU General Public License (GPL), */ /* version 2. See the file "COPYING" for details. */ @@ -17,21 +18,30 @@ { char **params = NULL; int paridx; - char dsdef[1024]; + char dsdef[1024]; /* destination DS syntax for rrd engine */ char *l, *name, *val; char *envnam; - char *dstypes = NULL; - - setupfn("%s.rrd", testname); + char *dstypes = NULL; /* contain NCV_testname value*/ + int split_ncv = 0; + int dslen; sprintf(rrdvalues, "%d", (int)tstamp); - params = (char **)calloc(8, sizeof(char *)); params[0] = "rrdcreate"; - params[1] = rrdfn; paridx = 1;

- envnam = (char *)malloc(4 + strlen(testname) + 1); sprintf(envnam, "NCV_%s", testname); + envnam = (char *)malloc(9 + strlen(testname) + 1); + sprintf(envnam, "SPLITNCV_%s", testname); + l = getenv(envnam); + if (l) { + split_ncv = 1; + dslen = 200; + } + else { + dslen = 19; + setupfn("%s.rrd", testname); + sprintf(envnam, "NCV_%s", testname); l = getenv(envnam); + } if (l) { dstypes = (char *)malloc(strlen(l)+3); sprintf(dstypes, ",%s,", l); @@ -84,62 +94,128 @@

strtod(val, &endptr); if (isspace((int)*endptr) || (*endptr == '\0')) { - char dsname[20]; - char dskey[22]; - char *dstype = NULL; + char dsname[250]; /* name of ncv in status message (with space and al) */ + char dskey[252]; /* name of final DS key (stripped) */ + char *dstype = NULL; /* type of final DS */ char *inp; int outidx; - /* val contains a valid number */ - /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] */ - for (inp=name,outidx=0; (*inp && (outidx < 19)); inp++) { + /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] ... */ + for (inp=name,outidx=0; (*inp && (outidx < dslen)); inp++) { if ( ((*inp >= 'A') && (*inp <= 'Z')) || ((*inp >= 'a') && (*inp <= 'z')) || ((*inp >= '0') && (*inp <= '9')) ) { dsname[outidx++] = *inp; } + /* ... however, for split ncv, we replace anything else */ + /* with an underscore, compacting successive invalid */ + /* characters into a single one */ + else if (split_ncv && (dsname[outidx - 1] != '_')) { + dsname[outidx++] = '_'; + } } + if(dsname[outidx-1] == '_') { + dsname[outidx-1] = '\0'; + } + else { dsname[outidx] = '\0'; + } sprintf(dskey, ",%s:", dsname); + if(split_ncv) { + /* setupfn("%s,%s.rrd", testname, dsname); */ + snprintf(rrdfn, sizeof(rrdfn)-1, "%s,%s.rrd", testname,dsname); + rrdfn[sizeof(rrdfn)-1] = '\0'; + + params[1] = rrdfn; + paridx = 1; + }

if (dstypes) { dstype = strstr(dstypes, dskey); - if (!dstype) { strcpy(dskey, ",*:"); dstype = strstr(dstypes, dskey); } + if (!dstype) { + strcpy(dskey, ",*:"); + dstype = strstr(dstypes, dskey); + } }

- if (dstype) { + if (dstype) { /* if ds type is forced */ char *p;

dstype += strlen(dskey); p = strchr(dstype, ','); if (p) *p = '\0'; + if(split_ncv) { + sprintf(dsdef, "DS:lambda:%s:600:0:U", dstype); + } + else { sprintf(dsdef, "DS:%s:%s:600:0:U", dsname, dstype); + } if (p) *p = ','; } + else { /* nothing specified in the environnement, and no '*:' default */ + if(split_ncv) { + strcpy(dsdef, "DS:lambda:DERIVE:600:0:U"); + } else { sprintf(dsdef, "DS:%s:DERIVE:600:0:U", dsname); } + }

- if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) { + if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) { /* if we have something */ paridx++; params = (char **)realloc(params, (7 + paridx)*sizeof(char *)); params[paridx] = strdup(dsdef); params[paridx+1] = NULL; - sprintf(rrdvalues+strlen(rrdvalues), ":%s", val); } } + + if(split_ncv && (paridx > 1)) { + int i; + params[++paridx] = strdup(rra1); + params[++paridx] = strdup(rra2); + params[++paridx] = strdup(rra3); + params[++paridx] = strdup(rra4); + + if(has_trackmax(testname)) { + params = (char **)realloc(params, (11 + paridx)*sizeof(char *)); + params[++paridx] = strdup(rra5); + params[++paridx] = strdup(rra6); + params[++paridx] = strdup(rra7); + params[++paridx] = strdup(rra8); + } + + params[++paridx] = NULL; + create_and_update_rrd(hostname, rrdfn, params, NULL); + for(i = 2 ; i<paridx ; i++) { + params[i] = NULL; + } + sprintf(rrdvalues, "%d", (int)tstamp); } }

- if (paridx > 1) { + } /* end of while */ + + if (split_ncv) { + for (paridx=2; (params[paridx] != NULL); paridx++) { + xfree(params[paridx]); + } + } + else if(paridx > 1) { params[++paridx] = strdup(rra1); params[++paridx] = strdup(rra2); params[++paridx] = strdup(rra3); params[++paridx] = strdup(rra4); - params[++paridx] = NULL;

- create_and_update_rrd(hostname, rrdfn, params, NULL); + if(has_trackmax(testname)) { + params = (char **)realloc(params, (11 + paridx)*sizeof(char *)); + params[++paridx] = strdup(rra5); + params[++paridx] = strdup(rra6); + params[++paridx] = strdup(rra7); + params[++paridx] = strdup(rra8); + }

+ params[++paridx] = NULL; + create_and_update_rrd(hostname, rrdfn, params, NULL); for (paridx=2; (params[paridx] != NULL); paridx++) xfree(params[paridx]); }

------------------------------------------------------------------------

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007