[hobbit] Representing disk data for graphing
No, the problem is not the root filesystem, the problem is that do_disk.c recognize a disk line if that line contains a "/" inside the line and will skip that line if it start with "&" and it will skip the line if it contains the string "red" or "yellow". This is mostly done for Unix filesystem report that looks something like: yellow Wed Oct 11 21:55:16 CEST 2006 - Filesystems NOT ok &yellow /sapdata/QQ8 (94% used) has reached the WARNING level (90%) Filesystem kbytes used avail capacity Mounted on swap 15566048 160 15565888 1% /var/run swap 15807304 241416 15565888 2% /tmp /dev/md/dsk/d0 20645791 6351115 14088219 32% / /dev/md/dsk/d2 35009161 2241510 32417560 7% /sapexe In this case it need to skip line starting with "&red" or "&yellow" because they are only the alert at the start of the message so that users know with a direct look what's wrong. Those line have to be skipped. So a line containing a "/" and not containing a "&" at the start or the word yellow and red will not be skipped. I think that you've to change the way you send data because probably the line with data you send will be skipped if they're reported to hobbit with "&red" or "&yellow" too. You can use something like the test I pasted above to send data that should be recognized as unix filesystem.
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 9:11 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Oh, I see, it's looking for a root filesystem line? This architecture doesn't implement a root filesystem. In any case I would want to just pass over it, reporting and graphing the root is as important as any of the other (IMHO).
Francesco Duranti wrote:
Not really a problem I think but some misinterpretation by the do_disk.c that is happening if the first line have a "/" your first line is: "color" 11 Oct 2006 07:45:23 Disk/SFS Utilization OK
I added "color" because it's passed also if you don't see it on the first line because it's the page color. The line checking the various line of disk report to get data line are:
/* AS/400 reports must contain the word DASD */ if ((dsystype == DT_AS400) && (strstr(curline, "DASD") == NULL)) goto nextline;
/* All clients except AS/400 report the mount-point with slashes - ALSO Win32 clients. */ if ((dsystype != DT_AS400) && (strchr(curline, '/') == NULL)) goto nextline;
/* red/yellow filesystems show up twice */ --> if ((dsystype != DT_NETAPP) && (dsystype != DT_NETWARE) && (dsystype != DT_AS400)) { --> if (*curline == '&') goto nextline; --> if ((strstr(curline, " red ") || strstr(curline, " yellow "))) goto nextline; }
for (columncount=0; (columncount<20); columncount++) columns[columncount] = ""; fsline = xstrdup(curline); columncount = 0; p = strtok(fsline, " "); while (p && (columncount < 20)) { columns[columncount++] = p; p = strtok(NULL, " "); }
So your dstype is not DT_NETAPP or DT_NETWARE or DT_AS400, the first line have a "/", if the color is not "red" or "yellow" that line will be treated as a data line and it will get some data from it and construct a rrd file.
I had the same problem with netapp filers disk reporting of my script when sending data with the /group option because the /group will be sent to the rrd disk module (you'll not see it in the page but it's there like the color) and in that case I started to have rrd file named with HH:MM:SS ... I got the problem when i started to get 50-60 of those files in each filers rrd file.
So it should be safe to skip the first line of the data reported by the do_disk because it will not contain filesystems data (don't know if it can create problem with netware or as400 but it works with netapp/unix/windows clients at least).
Francesco
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 8:40 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Thanks for the patch. Is there something in the first line itself that might be causing the problem?
Francesco Duranti wrote:
Hi, One of the problem is that the first line is got from the do_disk.c and it have a "/" so it's checked as a data source. I posted a patch some days ago to solve this problem that also happen if you use the /group with the bb command. I don't know if this alo
This will skip the first line of the do_disk message and it should then work as expected (at least it work with /group)
diff -N -r -U5 hobbit-4.2.0/hobbitd/rrd/do_disk.c hobbit-4.2.0new/hobbitd/rrd/do_disk.c --- hobbit-4.2.0/hobbitd/rrd/do_disk.c 2006-08-09 22:10:06.000000000 +0200 +++ hobbit-4.2.0new/hobbitd/rrd/do_disk.c 2006-10-03 19:58:25.000000000 +0200 @@ -50,10 +50,15 @@ else if (strstr(msg, "NetAPP")) dsystype = DT_NETAPP; else if (strstr(msg, "Filesystem")) dsystype = DT_NT; else dsystype = DT_UNIX;
- curline = msg; + /* FD: Skip the first line of message because sometimes it can create problems with + bb /group flag */ + curline = strchr(msg,'\n'); + if (curline) curline++; + /* FD: End of Skip the first line */ while (curline) { char *fsline, *p; char *columns[20]; int columncount; char *diskname = NULL;
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 8:13 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Oops, I spoke too soon. It did not work with the icons as I thought. I'm back to no icons and four data points per graph (which looks good) and the mysterious third token on the first line data point.
Rich Smrcina wrote:
Indeed. I got a suggestion to add the &clear icon to the heading and now it works. Unfortunately I now get one very crowded graph, otherwise good progress.
Thanks for your help!
Hubbard, Greg L wrote: > Yes, the "&color" now counts as a token, so any grep/awk/sed (or > equivalent) is going to be off by one. Is this information included > in the overall data blast from the Hobbit client? If so, you might > consider tacking the colors at the END of each line and see if the > Hobbit parser starts working again. > > For reference, here is something from a Solaris system: > > Wed Oct 11 12:19:50 CDT 2006 - Filesystems OK > Filesystem kbytes used avail capacity Mounted on > swap 18665136 107640 18557496 1% /tmp > /dev/vx/dsk/rootvol 9210413 1561203 7557106 18% / > /dev/vx/dsk/var 8263373 1775692 6405048 22% /var > /dev/vx/dsk/opt2 34291077 2384649 31563518 8% /opt2 > /dev/vx/dsk/home 1021735 614185 346246 64% /export/home > Notice that the filesystem is token 1, the usage is token 2, and > available space is token 3, etc., etc. > > These are just ideas. I've written 7 or 8 custom tests of my own > with my own RRD parsing at the end, so I quickly became acquainted > with the need for agreement between the "pitcher" and the "catcher". > GLH -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Francesco -- I think this explains everything! Good work! GLH -----Original Message----- From: Francesco Duranti [mailto:fduranti at q8.it] Sent: Wednesday, October 11, 2006 2:58 PM To: hobbit at hswn.dk Subject: RE: [hobbit] Representing disk data for graphing No, the problem is not the root filesystem, the problem is that do_disk.c recognize a disk line if that line contains a "/" inside the line and will skip that line if it start with "&" and it will skip the line if it contains the string "red" or "yellow". This is mostly done for Unix filesystem report that looks something like: yellow Wed Oct 11 21:55:16 CEST 2006 - Filesystems NOT ok &yellow /sapdata/QQ8 (94% used) has reached the WARNING level (90%) Filesystem kbytes used avail capacity Mounted on swap 15566048 160 15565888 1% /var/run swap 15807304 241416 15565888 2% /tmp /dev/md/dsk/d0 20645791 6351115 14088219 32% / /dev/md/dsk/d2 35009161 2241510 32417560 7% /sapexe In this case it need to skip line starting with "&red" or "&yellow" because they are only the alert at the start of the message so that users know with a direct look what's wrong. Those line have to be skipped. So a line containing a "/" and not containing a "&" at the start or the word yellow and red will not be skipped. I think that you've to change the way you send data because probably the line with data you send will be skipped if they're reported to hobbit with "&red" or "&yellow" too. You can use something like the test I pasted above to send data that should be recognized as unix filesystem.
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 9:11 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Oh, I see, it's looking for a root filesystem line? This architecture
doesn't implement a root filesystem. In any case I would want to just
pass over it, reporting and graphing the root is as important as any of the other (IMHO).
Francesco Duranti wrote:
Not really a problem I think but some misinterpretation by the do_disk.c that is happening if the first line have a "/" your first line is: "color" 11 Oct 2006 07:45:23 Disk/SFS Utilization OK
I added "color" because it's passed also if you don't see it on the first line because it's the page color. The line checking the various line of disk report to get data line are:
/* AS/400 reports must contain the word DASD */ if ((dsystype == DT_AS400) && (strstr(curline, "DASD") == NULL)) goto nextline;
/* All clients except AS/400 report the mount-point with slashes - ALSO Win32 clients. */ if ((dsystype != DT_AS400) && (strchr(curline, '/') == NULL)) goto nextline;
/* red/yellow filesystems show up twice */ --> if ((dsystype != DT_NETAPP) && (dsystype != DT_NETWARE) && (dsystype != DT_AS400)) { --> if (*curline == '&') goto nextline; --> if ((strstr(curline, " red ") || strstr(curline, " yellow "))) goto nextline; }
for (columncount=0; (columncount<20); columncount++) columns[columncount] = ""; fsline = xstrdup(curline); columncount = 0; p = strtok(fsline, " "); while (p && (columncount < 20)) { columns[columncount++] = p; p = strtok(NULL, " "); }
So your dstype is not DT_NETAPP or DT_NETWARE or DT_AS400, the first line have a "/", if the color is not "red" or "yellow" that line will be treated as a data line and it will get some data from it and construct a rrd file.
I had the same problem with netapp filers disk reporting of my script when sending data with the /group option because the /group will be sent to the rrd disk module (you'll not see it in the page but it's there like the color) and in that case I started to have rrd file named with HH:MM:SS ... I got the problem when i started to get 50-60 of those files in each filers rrd file.
So it should be safe to skip the first line of the data reported by the do_disk because it will not contain filesystems data (don't know if it can create problem with netware or as400 but it works with netapp/unix/windows clients at least).
Francesco
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 8:40 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Thanks for the patch. Is there something in the first line itself that might be causing the problem?
Francesco Duranti wrote:
Hi, One of the problem is that the first line is got from the do_disk.c and it have a "/" so it's checked as a data source. I posted a patch some days ago to solve this problem that also happen if you use the /group with the bb command. I don't know if this alo
This will skip the first line of the do_disk message and it should then work as expected (at least it work with /group)
diff -N -r -U5 hobbit-4.2.0/hobbitd/rrd/do_disk.c hobbit-4.2.0new/hobbitd/rrd/do_disk.c --- hobbit-4.2.0/hobbitd/rrd/do_disk.c 2006-08-09 22:10:06.000000000 +0200 +++ hobbit-4.2.0new/hobbitd/rrd/do_disk.c 2006-10-03 19:58:25.000000000 +0200 @@ -50,10 +50,15 @@ else if (strstr(msg, "NetAPP")) dsystype = DT_NETAPP; else if (strstr(msg, "Filesystem")) dsystype = DT_NT; else dsystype = DT_UNIX;
- curline = msg; + /* FD: Skip the first line of message because sometimes it can create problems with + bb /group flag */ + curline = strchr(msg,'\n'); + if (curline) curline++; + /* FD: End of Skip the first line */ while (curline) { char *fsline, *p; char *columns[20]; int columncount; char *diskname = NULL;
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 8:13 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Oops, I spoke too soon. It did not work with the icons as I thought. I'm back to no icons and four data points per graph (which looks good) and the mysterious third token on the first line data point.
Rich Smrcina wrote:
Indeed. I got a suggestion to add the &clear icon to the heading and now it works. Unfortunately I now get one very crowded graph, otherwise good progress.
Thanks for your help!
Hubbard, Greg L wrote: > Yes, the "&color" now counts as a token, so any grep/awk/sed (or > equivalent) is going to be off by one. Is this information included > in the overall data blast from the Hobbit client? If so, you might > consider tacking the colors at the END of each line and see if the > Hobbit parser starts working again. > > For reference, here is something from a Solaris system: > > Wed Oct 11 12:19:50 CDT 2006 - Filesystems OK > Filesystem kbytes used avail capacity Mounted on > swap 18665136 107640 18557496 1% /tmp > /dev/vx/dsk/rootvol 9210413 1561203 7557106 18% / > /dev/vx/dsk/var 8263373 1775692 6405048 22% /var > /dev/vx/dsk/opt2 34291077 2384649 31563518 8% /opt2 > /dev/vx/dsk/home 1021735 614185 346246 64% /export/home > Notice that the filesystem is token 1, the usage is token 2, and > available space is token 3, etc., etc. > > These are just ideas. I've written 7 or 8 custom tests of my own > with my own RRD parsing at the end, so I quickly became acquainted > with the need for agreement between the "pitcher" and the "catcher". > GLH -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
OK, I understand now. Thanks for all of the information. With that I do not understand why I'm getting a graph with the datapoint with one of the items on my status message: status vm1.disk green Wed Oct 11 22:12:19 CDT 2006 [vm1] Disk/SFS Utilization OK (this is all on one line) At this point '11' is a datapoint on the graphs and has a value of zero/ Francesco Duranti wrote:
No, the problem is not the root filesystem, the problem is that do_disk.c recognize a disk line if that line contains a "/" inside the line and will skip that line if it start with "&" and it will skip the line if it contains the string "red" or "yellow". This is mostly done for Unix filesystem report that looks something like:
yellow Wed Oct 11 21:55:16 CEST 2006 - Filesystems NOT ok
&yellow /sapdata/QQ8 (94% used) has reached the WARNING level (90%)
Filesystem kbytes used avail capacity Mounted on swap 15566048 160 15565888 1% /var/run swap 15807304 241416 15565888 2% /tmp /dev/md/dsk/d0 20645791 6351115 14088219 32% / /dev/md/dsk/d2 35009161 2241510 32417560 7% /sapexe
In this case it need to skip line starting with "&red" or "&yellow" because they are only the alert at the start of the message so that users know with a direct look what's wrong. Those line have to be skipped. So a line containing a "/" and not containing a "&" at the start or the word yellow and red will not be skipped.
I think that you've to change the way you send data because probably the line with data you send will be skipped if they're reported to hobbit with "&red" or "&yellow" too.
You can use something like the test I pasted above to send data that should be recognized as unix filesystem.
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 9:11 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Oh, I see, it's looking for a root filesystem line? This architecture doesn't implement a root filesystem. In any case I would want to just pass over it, reporting and graphing the root is as important as any of the other (IMHO).
Francesco Duranti wrote:
Not really a problem I think but some misinterpretation by the do_disk.c that is happening if the first line have a "/" your first line is: "color" 11 Oct 2006 07:45:23 Disk/SFS Utilization OK
I added "color" because it's passed also if you don't see it on the first line because it's the page color. The line checking the various line of disk report to get data line are:
/* AS/400 reports must contain the word DASD */ if ((dsystype == DT_AS400) && (strstr(curline, "DASD") == NULL)) goto nextline;
/* All clients except AS/400 report the mount-point with slashes - ALSO Win32 clients. */ if ((dsystype != DT_AS400) && (strchr(curline, '/') == NULL)) goto nextline;
/* red/yellow filesystems show up twice */ --> if ((dsystype != DT_NETAPP) && (dsystype != DT_NETWARE) && (dsystype != DT_AS400)) { --> if (*curline == '&') goto nextline; --> if ((strstr(curline, " red ") || strstr(curline, " yellow "))) goto nextline; }
for (columncount=0; (columncount<20); columncount++) columns[columncount] = ""; fsline = xstrdup(curline); columncount = 0; p = strtok(fsline, " "); while (p && (columncount < 20)) { columns[columncount++] = p; p = strtok(NULL, " "); }
So your dstype is not DT_NETAPP or DT_NETWARE or DT_AS400, the first line have a "/", if the color is not "red" or "yellow" that line will be treated as a data line and it will get some data from it and construct a rrd file.
I had the same problem with netapp filers disk reporting of my script when sending data with the /group option because the /group will be sent to the rrd disk module (you'll not see it in the page but it's there like the color) and in that case I started to have rrd file named with HH:MM:SS ... I got the problem when i started to get 50-60 of those files in each filers rrd file.
So it should be safe to skip the first line of the data reported by the do_disk because it will not contain filesystems data (don't know if it can create problem with netware or as400 but it works with netapp/unix/windows clients at least).
Francesco
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 8:40 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Thanks for the patch. Is there something in the first line itself that might be causing the problem?
Francesco Duranti wrote:
Hi, One of the problem is that the first line is got from the do_disk.c and it have a "/" so it's checked as a data source. I posted a patch some days ago to solve this problem that also happen if you use the /group with the bb command. I don't know if this alo This will skip the first line of the do_disk message and it should then work as expected (at least it work with /group)
diff -N -r -U5 hobbit-4.2.0/hobbitd/rrd/do_disk.c hobbit-4.2.0new/hobbitd/rrd/do_disk.c --- hobbit-4.2.0/hobbitd/rrd/do_disk.c 2006-08-09 22:10:06.000000000 +0200 +++ hobbit-4.2.0new/hobbitd/rrd/do_disk.c 2006-10-03 19:58:25.000000000 +0200 @@ -50,10 +50,15 @@ else if (strstr(msg, "NetAPP")) dsystype = DT_NETAPP; else if (strstr(msg, "Filesystem")) dsystype = DT_NT; else dsystype = DT_UNIX;
- curline = msg; + /* FD: Skip the first line of message because sometimes it can create problems with + bb /group flag */ + curline = strchr(msg,'\n'); + if (curline) curline++; + /* FD: End of Skip the first line */ while (curline) { char *fsline, *p; char *columns[20]; int columncount; char *diskname = NULL;
-----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Wednesday, October 11, 2006 8:13 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Representing disk data for graphing
Oops, I spoke too soon. It did not work with the icons as I thought. I'm back to no icons and four data points per graph (which looks good) and the mysterious third token on the first line data point. Rich Smrcina wrote: > Indeed. I got a suggestion to add the &clear icon to the heading and > now it works. Unfortunately I now get one very crowded graph, > otherwise good progress. > > Thanks for your help! > > Hubbard, Greg L wrote: >> Yes, the "&color" now counts as a token, so any grep/awk/sed (or >> equivalent) is going to be off by one. Is this information included >> in the overall data blast from the Hobbit client? If so, you might >> consider tacking the colors at the END of each line and see if the >> Hobbit parser starts working again. >> >> For reference, here is something from a Solaris system: >> >> Wed Oct 11 12:19:50 CDT 2006 - Filesystems OK >> Filesystem kbytes used avail capacity Mounted on >> swap 18665136 107640 18557496 1% /tmp >> /dev/vx/dsk/rootvol 9210413 1561203 7557106 18% / >> /dev/vx/dsk/var 8263373 1775692 6405048 22% /var >> /dev/vx/dsk/opt2 34291077 2384649 31563518 8% /opt2 >> /dev/vx/dsk/home 1021735 614185 346246 64% /export/home >> Notice that the filesystem is token 1, the usage is token 2, and >> available space is token 3, etc., etc. >> >> These are just ideas. I've written 7 or 8 custom tests of my own >> with my own RRD parsing at the end, so I quickly became acquainted >> with the need for agreement between the "pitcher" and the "catcher". >> GLH -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
participants (3)
-
fduranti＠q8.it
-
greg.hubbard＠eds.com
-
rsmrcina＠wi.rr.com