- Xymon - Xymon mailman web

semop failed, Identifier removed
by bigdan＠gmail.com 12 Oct '06

12 Oct '06

Hello, I got these messages today after a week of hobbit 4.2.0 being up and running. I made one change a couple hours before, adding " DURATION>1" for my two notification recipients in hobbit-alert.cfg and restarting. I got all kinds of purple emails, and 2006-10-11 18:50:13 semop failed, Identifier removed 2006-10-11 18:50:13 How did this happen? clients=-1, s.sem_op=0 2006-10-11 18:50:13 semop failed, Identifier removed 2006-10-11 18:50:13 How did this happen? clients=-1, s.sem_op=0 2006-10-11 18:50:13 semop failed, Identifier removed 2006-10-11 18:50:13 How did this happen? clients=-1, s.sem_op=0 2006-10-11 18:50:13 semop failed, Identifier removed 2006-10-11 18:50:13 How did this happen? clients=-1, s.sem_op=0 2006-10-11 18:50:13 semop failed, Identifier removed 2006-10-11 18:50:13 How did this happen? clients=-1, s.sem_op=0 2006-10-11 18:50:13 semop failed, Identifier removed 2006-10-11 18:50:13 How did this happen? clients=-1, s.sem_op=0 2006-10-11 18:50:13 semop failed, Identifier removed 2006-10-11 18:50:13 How did this happen? clients=-1, s.sem_op=0 I assumed it died, so I did: $ ./hobbit.sh start Hobbit appears to be running, doing restart Hobbit stopped Hobbit started Ideas? Thanks Dan

1 0

[hobbit] Representing disk data for graphing
by fduranti＠q8.it 11 Oct '06

11 Oct '06

Not really a problem I think but some misinterpretation by the do_disk.c that is happening if the first line have a "/" your first line is: "color" 11 Oct 2006 07:45:23 Disk/SFS Utilization OK I added "color" because it's passed also if you don't see it on the first line because it's the page color. The line checking the various line of disk report to get data line are: /* AS/400 reports must contain the word DASD */ if ((dsystype == DT_AS400) && (strstr(curline, "DASD") == NULL)) goto nextline; /* All clients except AS/400 report the mount-point with slashes - ALSO Win32 clients. */ if ((dsystype != DT_AS400) && (strchr(curline, '/') == NULL)) goto nextline; /* red/yellow filesystems show up twice */ --> if ((dsystype != DT_NETAPP) && (dsystype != DT_NETWARE) && (dsystype != DT_AS400)) { --> if (*curline == '&') goto nextline; --> if ((strstr(curline, " red ") || strstr(curline, " yellow "))) goto nextline; } for (columncount=0; (columncount<20); columncount++) columns[columncount] = ""; fsline = xstrdup(curline); columncount = 0; p = strtok(fsline, " "); while (p && (columncount < 20)) { columns[columncount++] = p; p = strtok(NULL, " "); } So your dstype is not DT_NETAPP or DT_NETWARE or DT_AS400, the first line have a "/", if the color is not "red" or "yellow" that line will be treated as a data line and it will get some data from it and construct a rrd file. I had the same problem with netapp filers disk reporting of my script when sending data with the /group option because the /group will be sent to the rrd disk module (you'll not see it in the page but it's there like the color) and in that case I started to have rrd file named with HH:MM:SS ... I got the problem when i started to get 50-60 of those files in each filers rrd file. So it should be safe to skip the first line of the data reported by the do_disk because it will not contain filesystems data (don't know if it can create problem with netware or as400 but it works with netapp/unix/windows clients at least). Francesco > -----Original Message----- > From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] > Sent: Wednesday, October 11, 2006 8:40 PM > To: hobbit at hswn.dk > Subject: Re: [hobbit] Representing disk data for graphing > > Thanks for the patch. Is there something in the first line > itself that might be causing the problem? > > Francesco Duranti wrote: > > Hi, > > One of the problem is that the first line is got from the > do_disk.c > > and it have a "/" so it's checked as a data source. > > I posted a patch some days ago to solve this problem that > also happen > > if you use the /group with the bb command. I don't know if this alo > > > > This will skip the first line of the do_disk message and it should > > then work as expected (at least it work with /group) > > > > diff -N -r -U5 hobbit-4.2.0/hobbitd/rrd/do_disk.c > > hobbit-4.2.0new/hobbitd/rrd/do_disk.c > > --- hobbit-4.2.0/hobbitd/rrd/do_disk.c 2006-08-09 > 22:10:06.000000000 > > +0200 > > +++ hobbit-4.2.0new/hobbitd/rrd/do_disk.c 2006-10-03 > > 19:58:25.000000000 +0200 > > @@ -50,10 +50,15 @@ > > else if (strstr(msg, "NetAPP")) dsystype = DT_NETAPP; > > else if (strstr(msg, "Filesystem")) dsystype = DT_NT; > > else dsystype = DT_UNIX; > > > > - curline = msg; > > + /* FD: Skip the first line of message because sometimes it can > > create problems with > > + bb /group flag */ > > + curline = strchr(msg,'\n'); > > + if (curline) curline++; > > + /* FD: End of Skip the first line */ > > while (curline) { > > char *fsline, *p; > > char *columns[20]; > > int columncount; > > char *diskname = NULL; > > > > > > > > > > > >> -----Original Message----- > >> From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] > >> Sent: Wednesday, October 11, 2006 8:13 PM > >> To: hobbit at hswn.dk > >> Subject: Re: [hobbit] Representing disk data for graphing > >> > >> Oops, I spoke too soon. It did not work with the icons as > I thought. > >> I'm back to no icons and four data points per graph (which looks > >> good) and the mysterious third token on the first line data point. > >> > >> Rich Smrcina wrote: > >>> Indeed. I got a suggestion to add the &clear icon to the > >> heading and > >>> now it works. Unfortunately I now get one very crowded graph, > >>> otherwise good progress. > >>> > >>> Thanks for your help! > >>> > >>> Hubbard, Greg L wrote: > >>>> Yes, the "&color" now counts as a token, so any grep/awk/sed (or > >>>> equivalent) is going to be off by one. Is this > >> information included > >>>> in the overall data blast from the Hobbit client? If so, > >> you might > >>>> consider tacking the colors at the END of each line and > see if the > >>>> Hobbit parser starts working again. > >>>> > >>>> For reference, here is something from a Solaris system: > >>>> > >>>> Wed Oct 11 12:19:50 CDT 2006 - Filesystems OK > >>>> Filesystem kbytes used avail capacity Mounted on > >>>> swap 18665136 107640 18557496 1% /tmp > >>>> /dev/vx/dsk/rootvol 9210413 1561203 7557106 18% / > >>>> /dev/vx/dsk/var 8263373 1775692 6405048 22% /var > >>>> /dev/vx/dsk/opt2 34291077 2384649 31563518 8% /opt2 > >>>> /dev/vx/dsk/home 1021735 614185 346246 64% > /export/home > >>>> > >>>> Notice that the filesystem is token 1, the usage is token 2, and > >>>> available space is token 3, etc., etc. > >>>> > >>>> These are just ideas. I've written 7 or 8 custom tests > of my own > >>>> with my own RRD parsing at the end, so I quickly became > acquainted > >>>> with the need for agreement between the "pitcher" and the > >> "catcher". > >>>> GLH > >> -- > >> Rich Smrcina > >> VM Assist, Inc. > >> Phone: 414-491-6001 > >> Ans Service: 360-715-2467 > >> rich.smrcina at vmassist.com > >> > >> Catch the WAVV! http://www.wavv.org > >> WAVV 2007 - Green Bay, WI - May 18-22, 2007 > >> > >> To unsubscribe from the hobbit list, send an e-mail to > >> hobbit-unsubscribe at hswn.dk > >> > >> > >> > > > > To unsubscribe from the hobbit list, send an e-mail to > > hobbit-unsubscribe at hswn.dk > > > > > > -- > Rich Smrcina > VM Assist, Inc. > Phone: 414-491-6001 > Ans Service: 360-715-2467 > rich.smrcina at vmassist.com > > Catch the WAVV! http://www.wavv.org > WAVV 2007 - Green Bay, WI - May 18-22, 2007 > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > > >

2 2

Representing disk data for graphing
by rsmrcina＠wi.rr.com 11 Oct '06

11 Oct '06

I'm interested in changing the way that I report disk utilization for mainframe based operating systems (VM and VSE) so that it can be graphed by RRD. The basic message format seems right, but the graph is not coming out properly. Here is the message text: 11 Oct 2006 07:45:23 Disk/SFS Utilization OK Filesystem kbytes used avail capacity Mounted as green /dev/0124 14400 720 13680 5.0% /440RES/DRCT green /dev/0124 36000 0 36000 0.0% /440RES/TDISK green /dev/0124 128160 32168 95992 25.1% /440RES/SPOOL green /dev/0124 96480 2112 94368 2.2% /440RES/PAGE green /dev/0102 216000 17636 198364 8.2% /PAGSP1/SPOOL green /dev/0102 2187360 0 2187360 0.0% /PAGSP1/PAGE green /VMSYS/group2 120088 5268 114820 4.4% /SFS/VMSYS green /VMSYSU/group2 4280 488 3792 11.4% /SFS/VMSYSU green /VMSYSU/group3 71896 0 71896 0.0% /SFS/VMSYSU The 'green' indicator on the left is a green icon put there to easily spot an offender. Is that messing up the parsing of the message for graphing purposes? Or is it possibly the decimal points in the percentages? The graph is coming up with a single data point called 2006. -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007

5 14

[hobbit] Representing disk data for graphing
by fduranti＠q8.it 11 Oct '06

11 Oct '06

Hi, One of the problem is that the first line is got from the do_disk.c and it have a "/" so it's checked as a data source. I posted a patch some days ago to solve this problem that also happen if you use the /group with the bb command. I don't know if this alo This will skip the first line of the do_disk message and it should then work as expected (at least it work with /group) diff -N -r -U5 hobbit-4.2.0/hobbitd/rrd/do_disk.c hobbit-4.2.0new/hobbitd/rrd/do_disk.c --- hobbit-4.2.0/hobbitd/rrd/do_disk.c 2006-08-09 22:10:06.000000000 +0200 +++ hobbit-4.2.0new/hobbitd/rrd/do_disk.c 2006-10-03 19:58:25.000000000 +0200 @@ -50,10 +50,15 @@ else if (strstr(msg, "NetAPP")) dsystype = DT_NETAPP; else if (strstr(msg, "Filesystem")) dsystype = DT_NT; else dsystype = DT_UNIX; - curline = msg; + /* FD: Skip the first line of message because sometimes it can create problems with + bb /group flag */ + curline = strchr(msg,'\n'); + if (curline) curline++; + /* FD: End of Skip the first line */ while (curline) { char *fsline, *p; char *columns[20]; int columncount; char *diskname = NULL; > -----Original Message----- > From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] > Sent: Wednesday, October 11, 2006 8:13 PM > To: hobbit at hswn.dk > Subject: Re: [hobbit] Representing disk data for graphing > > Oops, I spoke too soon. It did not work with the icons as I thought. > I'm back to no icons and four data points per graph (which > looks good) and the mysterious third token on the first line > data point. > > Rich Smrcina wrote: > > Indeed. I got a suggestion to add the &clear icon to the > heading and > > now it works. Unfortunately I now get one very crowded graph, > > otherwise good progress. > > > > Thanks for your help! > > > > Hubbard, Greg L wrote: > >> Yes, the "&color" now counts as a token, so any grep/awk/sed (or > >> equivalent) is going to be off by one. Is this > information included > >> in the overall data blast from the Hobbit client? If so, > you might > >> consider tacking the colors at the END of each line and see if the > >> Hobbit parser starts working again. > >> > >> For reference, here is something from a Solaris system: > >> > >> Wed Oct 11 12:19:50 CDT 2006 - Filesystems OK > >> Filesystem kbytes used avail capacity Mounted on > >> swap 18665136 107640 18557496 1% /tmp > >> /dev/vx/dsk/rootvol 9210413 1561203 7557106 18% / > >> /dev/vx/dsk/var 8263373 1775692 6405048 22% /var > >> /dev/vx/dsk/opt2 34291077 2384649 31563518 8% /opt2 > >> /dev/vx/dsk/home 1021735 614185 346246 64% /export/home > >> > >> Notice that the filesystem is token 1, the usage is token 2, and > >> available space is token 3, etc., etc. > >> > >> These are just ideas. I've written 7 or 8 custom tests of my own > >> with my own RRD parsing at the end, so I quickly became acquainted > >> with the need for agreement between the "pitcher" and the > "catcher". > >> > >> GLH > > > > -- > Rich Smrcina > VM Assist, Inc. > Phone: 414-491-6001 > Ans Service: 360-715-2467 > rich.smrcina at vmassist.com > > Catch the WAVV! http://www.wavv.org > WAVV 2007 - Green Bay, WI - May 18-22, 2007 > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > > >

2 1

hobbit-clients.cfg file matching algorithm
by JamesHorwath＠glic.com 11 Oct '06

11 Oct '06

How does the matching behave with the hobbit-client.cfg file. For example if I have: HOST=foo DISK /usr 90 95 PORT LOCAL=*.22 DEFAULT PROC cron PROC sched Since there are no PROC directives for the machine "foo", are the default PROC settings applied against the server foo? Is are the DEFAULT rules set always applied to the server being monitored. Is the matching by HOST blocks, or is a list cobbled together with HOST and tests? Thanks in advance. Jim ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

2 1

Hobbit_perl_client v1.14
by fduranti＠q8.it 11 Oct '06

11 Oct '06

Hi all, new version is out... Most of the changes is on the fork functions so please upgrade to this version because it solve some problems with the forking of process. For netapp users the cpu report is now using the average cpu used instead of the total and the script will report as data to hobbit all the stats performance data. The included patch to the hobbit server will allow the user to save all those data to rrd file. Definition of those graphs are on the way, I'm currently working on them and on a better documentation. For people using Perl 5.8.0 (for example red hat 3 users) they have to upgrade the Sys::SigAction module to version 0.07 because the 0.06 version have a bug that will make the script exit without reporting anything. This is the complete changelog: Hobbit_fd_lib.pm v1.14 - Added function for uptime checking - Added an option to change the dbcheck/netapp/beastat reporting column under the hobbit server to report warning or error o nly if there are more than a defined number of errors - Completely changed the way on how child fork and how they'll pass data to the parent process - The program will now exit if the perl version is 5.8.0 and Sys::SigAlert version is less then 0.07 because the module will not work in that configuration - Corrected a bug in a time calculation function not working with some dates and creating a lot of warning Beastat.pl v1.06 - Added an option to have the uptime in a different column (see the beastat.ini file for options) - Corrected a bug with the uptime reporting DbCheck.pl v1.07 - Added an option to have the uptime in a different column (see the dbcheck.ini file for options) - Corrected a bug to get the default value for tblspace check - Corrected a bug in the check for errors in the valtocheck parameter - Corrected some bugs on the locks report queries Netapp.pl v1.10 - Added an option to have the uptime in a different column (see the netapp.ini file for options) - Added an option to check and report for timedifference between the hobbit server and filers - Added the report of extended performance data gathered via stats command (those data will be collected into rrd file after patching the hobbit source and will report about many statistical counters. To know what can be graphed check the stats command and in particular stats list counters with object: aggregate, volume, disk, qtree, iscsi, fcp, cifs, lun, nfsv3, ifnet, processor, system - Corrected cpu statistics gathering via sysstat -m on multiprocessor platform using the average cpu time instead of the total (better reporting of cpu performance). - Full report on cpu information with total, average and single processor use on the cpu column - Some bug fixed to various functions

1 0

udpInOverflows counter in netstat -s on solaris
by Larry.Sherman＠rbsgc.com 11 Oct '06

11 Oct '06

Hi Henrik, We are trying to monitor udp drops in our network, and we use the udpInOverflows counter in netstat -s on Solaris as a way to measure where the data was dropped. We have a measure of who is re-requesting missing data, and we can verify that the data was in fact sent. We use the udpInOverflows to determine if the drop occurred on the host or on the network. Our supposition is that if a machine is generating re-transmission requests, but the udpInOverflows counter is 0 && the data was in fact sent, then the data was dropped on the wire. I see that the clients side code captures the counter in netstat -s, but that it is not included in the static data structure for Solaris in do_netstat.c in the /hobbitd/rrd directory. Is this easy to add? I'm not proficient at C, and not sure of the scope of the work involved. Thanks much, Larry Sherman ----------------------------------------- ******************************************************************* * This e-mail is intended only for the addressee named above. As this e-mail may contain confidential or privileged information, if you are not the named addressee, you are not authorized to retain, read, copy or disseminate this message or any part of it. ******************************************************************* *

1 0

badhttp test problem - alerting on single failure
by jonescr＠cisco.com 11 Oct '06

11 Oct '06

I have a host where I am doing two http tests against. One of the URLs has shown a tendency to false alarm (go red and then a minute later back green), twice a night. In an attempt to reduce these false alarms, I used the "badhttp" tag as so: ---snippet from bb-hosts--- 1.2.3.4 URL-Site_Portal # cont;http://mysite.com/public/index.html;Found http://www.mysite.com/private/index.html badhttp-*-0000-1800:1:2:4 --- The second URL is the one that blips, and adding the badhttp tag has not helped, it still alerts after a single failure, and history shows no status changes to clear or yellow. Here is a cut and paste of the status when it alerted red (The second check was listed below this one, and was green - also interestingly, the second check shown on the status page is the first URL from the bb-hosts line): Wed Oct 11 07:05:56 2006: Server timeout ; OK http://www.mysite.com/private/index.html - Server timeout Seconds: 10.31 Is there any way that I can troubleshoot this? Could it be caused from doing two http tests against the same site (maybe badhttp only takes effect for the first one)? -Charles

4 5

Fresh Hobbit 4.2 installed, no client memory stats available
by hobbit＠visions.se 11 Oct '06

11 Oct '06

Hi, Did a fresh install of the stable 4.2 release of Hobbit and compiled the client on another host. It's showing cpu, processes and so on but no memory stats at all. The hobbit server on the other hand is showing its memory stats. The hobbit-clients-cfg file contains: DEFAULT # These are the built-in defaults. UP 30 LOAD 1.5 3.0 DISK * 90 95 MEMPHYS 100 101 MEMSWAP 50 80 MEMACT 90 97 Any idea why the memory stats don't show up? David

1 0

ifstat and hobbitclient-sunos.sh
by Dominique.Frise＠unil.ch 11 Oct '06

11 Oct '06

Hi, The command "/usr/bin/kstat -p -s '[or]bytes64' | sort" for ifstat in hobbitclient-sunos.sh collects statistics for wrsmd (WCI Remote Shared Memory (WRSM) DLPI driver), which is wrong. Example on a Solaris 9 client: [bb at eliot bin]$ /usr/bin/kstat -p -s '[or]bytes64' | sort eri:0:eri0:obytes64 363076089018 eri:0:eri0:rbytes64 15860668051 ge:0:ge0:obytes64 0 ge:0:ge0:rbytes64 0 wrsmd:0:wrsmd0:obytes64 0 wrsmd:0:wrsmd0:rbytes64 0 wrsmd:10:wrsmd10:obytes64 0 wrsmd:10:wrsmd10:rbytes64 0 wrsmd:11:wrsmd11:obytes64 0 wrsmd:11:wrsmd11:rbytes64 0 wrsmd:12:wrsmd12:obytes64 0 wrsmd:12:wrsmd12:rbytes64 0 wrsmd:13:wrsmd13:obytes64 0 wrsmd:13:wrsmd13:rbytes64 0 wrsmd:14:wrsmd14:obytes64 0 wrsmd:14:wrsmd14:rbytes64 0 wrsmd:15:wrsmd15:obytes64 0 wrsmd:15:wrsmd15:rbytes64 0 wrsmd:1:wrsmd1:obytes64 0 wrsmd:1:wrsmd1:rbytes64 0 wrsmd:2:wrsmd2:obytes64 0 wrsmd:2:wrsmd2:rbytes64 0 wrsmd:3:wrsmd3:obytes64 0 wrsmd:3:wrsmd3:rbytes64 0 wrsmd:4:wrsmd4:obytes64 0 wrsmd:4:wrsmd4:rbytes64 0 wrsmd:5:wrsmd5:obytes64 0 wrsmd:5:wrsmd5:rbytes64 0 wrsmd:6:wrsmd6:obytes64 0 wrsmd:6:wrsmd6:rbytes64 0 wrsmd:7:wrsmd7:obytes64 0 wrsmd:7:wrsmd7:rbytes64 0 wrsmd:8:wrsmd8:obytes64 0 wrsmd:8:wrsmd8:rbytes64 0 wrsmd:9:wrsmd9:obytes64 0 wrsmd:9:wrsmd9:rbytes64 0 A quick fix is to replace it with "/usr/bin/kstat -p -s '[or]bytes64' | grep -v wrsmd | sort" Dominique UNIL - University of Lausanne

4 4