December 2006 - Xymon - Xymon mailman web

hobbitfetch crash
by dan.mcdonald＠austinenergy.com 04 Dec '06

04 Dec '06

I see the following message on the hobbitfetch status page: - Program crashed Fatal signal caught! But it appears to operate properly, I get status information for my two DMZ hosts that I am currently using hobbitfetch for, and there are no messages in hobbitfetch.log hobbitlaunch.log shows intermittent errors: hobbitlaunch.log.1:2006-12-02 20:31:37 Task hobbitfetch terminated by signal 6 hobbitlaunch.log.1:2006-12-02 23:42:09 Task hobbitfetch terminated by signal 6 hobbitlaunch.log.1:2006-12-03 01:07:19 Task hobbitfetch terminated by signal 6 hobbitlaunch.log.1:2006-12-03 01:43:42 Task hobbitfetch terminated by signal 6 hobbitlaunch.log.1:2006-12-03 03:18:57 Task hobbitfetch terminated by signal 6 hobbitlaunch.log.1:2006-12-03 03:44:01 Task hobbitfetch terminated by signal 6 There appears to be a core dump: [root at ldap tmp]# gdb /usr/lib/hobbit/server/bin/hobbitfetch core.7.21 GNU gdb 6.3-5mdk (Mandriva Linux release 2006.0) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i586-mandriva-linux-gnu"... (no debugging symbols found) Using host libthread_db library "/lib/tls/libthread_db.so.1". Reading symbols from shared object read from target memory...(no debugging symbols found)...done. Loaded system supplied DSO at 0xffffe000 Core was generated by `/usr/lib/hobbit/server/bin/hobbitfetch --server=127.0.0.1 --no-daemon --pidfile'. Program terminated with signal 6, Aborted. warning: svr4_current_sos: Can't read pathname for load map: Input/output error Reading symbols from /lib/tls/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...done. Loaded symbols for /lib/libnss_dns.so.2 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 #0 0xffffe410 in __kernel_vsyscall () (gdb) bt #0 0xffffe410 in __kernel_vsyscall () #1 0xb7e6c051 in raise () from /lib/tls/libc.so.6 #2 0xb7e6da3b in abort () from /lib/tls/libc.so.6 #3 0x0804f89f in ?? () #4 0x08056180 in stderr () #5 0x00000000 in ?? () -- Daniel J McDonald, CCIE # 2495, CISSP # 78281, CNX Austin Energy http://www.austinenergy.com

1 0

[patch] "config" download area is now a setting
by cgoyard＠cvf.fr 04 Dec '06

04 Dec '06

Hi Henrik, here is a small but convenient patch, that helps keep the etc/ directory of the hobbit server tidy. Something handy would be to have a automatic suffix-with-hostname before opening the file, and fallback on the straight requested filename. That is, requesting bb-mysql.cfg from "foo.bar.com" would search for "bb-mysql.cfg-foo.bar.com" and then for "bb-mysql.cfg". I may add this early next week if you didn't already :). Regards, -- Charles Goyard - cgoyard at cvf.fr - (+33) 1 45 38 01 31

4 4

DEFAULT client settings applied even when a different rule matches earlier in hobbit-clients.cfg
by tomg＠mcclatchyinteractive.com 04 Dec '06

04 Dec '06

I'm trying to set up my hobbit-clients.cfg file so that specific hosts have certain settings, and the remaining hosts that do not match pick up the default settings. However, it seems that the default settings are picked up and applied to everything, no matter what. If I understand what I've read in the hobbit-clients.cfg manpage, the first rule match should be applied and the remainder of the config file is ignored for the host. That has not been my experience, though. For example, I have these two entries in my hobbit-clients.cfg: HOST=%radm.*t.mcclatchyinteractive.com UP 30m LOAD 3.0 5.0 DISK * 90 95 MEMPHYS 100 101 MEMSWAP 50 80 MEMACT 90 97 PROC adm 1 DEFAULT # These are the built-in defaults. UP 30m LOAD 3.0 5.0 DISK * 90 95 MEMPHYS 100 101 MEMSWAP 50 80 MEMACT 90 97 PROC default 1 Right now I have a red procs alert for server radm004t.mcclatchyinteractive.com, because the process "default" is not running. I feel like I'm just missing something simple, but can't figure out what it is. Any tips? -- Tom Georgoulias Systems Engineer McClatchy Interactive

2 2

NCV graphing & \t\n in NCV module
by dominic.young＠surfcontrol.com 03 Dec '06

03 Dec '06

Dear Henrik, Big fan of hobbit and thanks to yourself and all others involved in the great work, very much appreciated. I have a question re the NCV module: --- release/hobbit-4.2.0/hobbitd/rrd/do_ncv.c 2006-08-09 22:10:06.000000000 +0200 +++ hobbitd/rrd/do_ncv.c 2006-09-14 22:49:06.259129035 +0200 Version 4.2.0 fixed a problem described in the below thread: http://www.hswn.dk/hobbiton/2006/09/msg00277.html However if the line above the NCV line of a custom test does not end in a tab then new line you still encounter the same problem. Is there any reason why a tab is necessary. All the documentation specifies that to get the NCV graphs working all output must be on a new line and formatted as follows: Sun Nov 20 09:03:44 CET 2006 inode_cache : 330624 dentry_cache : 40891068 Thanks Dominic Young -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.430 / Virus Database: 268.15.6/565 - Release Date: 02/12/2006 21:39 This message has been scanned for viruses by SurfControl on-demand services - www.surfcontrol.com

1 0

[hobbit] Custom RRD graphs
by ger＠fwi.ie 03 Dec '06

03 Dec '06

Folks, I'm still having trouble with my graphs. I have turned on the debug mode and I'm getting the following for one of the tests. I've even gone so far to do a complete OS install on a new machine with only hobbit to make there was nothing conflicting with it. The scripts seem to be working but the rrd files are just not updating. Can anyone see anything wrong ? 2006-12-03 12:27:42 Want msg 119403, startpos 128142, fillpos 128142, endpos -1, usedbytes=0, bufleft=400241 2006-12-03 12:27:42 hobbitd_rrd: Got message 119403 @@status#119403|1165148862.825676|10.8.1.10||fwi-lor-02|acpu|1165150662|gree n||green|1165065289|0||0||0 2006-12-03 12:27:42 startpos 128677, fillpos 128677, endpos -1 2006-12-03 12:27:42 -> do_external(fwi-lor-02, acpu) 2006-12-03 12:27:42 <- do_external(fwi-lor-02, acpu) 2006-12-03 12:27:42 Want msg 119404, startpos 128677, fillpos 128677, endpos -1, usedbytes=0, bufleft=399706 2006-12-03 12:27:42 000013787 : Saving msg to file /usr/lib/hobbit/server/tmp/rrd_msg_13787 2006-12-03 12:27:42 000013787 : Calling helper script /usr/lib/hobbit/server/ext/rrd-ext.sh fwi-lor-02 acpu /usr/lib/hobbit/server/tmp/rrd_msg_13787 2006-12-03 12:27:42 000013787 : Unlinking temp file Thanks, Ger. _____ From: Ger Maguire [mailto:ger at fwi.ie] Sent: 30 November 2006 20:17 To: hobbit at hswn.dk Subject: RE: [hobbit] Custom RRD graphs I think I have gone through all the steps and have poured over the docs and manpages So far: 1 Created custom script outputing data to hobbit using status switch $BBHOME/bin/bb $BBDISP "status $HOST.$TEST $COLOUR $OUTPUT" 2 Created rrd-ext script and have it getting data as per below (numeric vale for catching a % value) 3 Modified hobbitserver.cfg Added acpu test to TEST2RRD="cpu=la,disk, ... ,acpu" Added acpu test to GRAPHS="la,disk, ... ,acpu" 4 Modified hobbitlaunch.cfg [rrdstatus] ENVFILE /usr/lib/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=status --log=$BBSERVERLOGS/rrd-status.log hobbitd_rrd --rrddir=$BBVAR/rrd --extra-script=/usr/lib/hobbit/server/ext/rrd-ext.sh --extra-tests=acpu I haven't modified [rrddata] as I'm sending the data in using the status command but I'll stick it in and see if it makes a difference. Added the test itself to hobbitlaunch to run every 5 mins, all works ok pages update in hobbit [acpu] ENVFILE /usr/lib/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD $BBHOME/ext/acpu.sh INTERVAL 5m 5 Modified hobbitgraph.cfg with the following [acpu] TITLE CPU Utilisation YAXIS % Load DEF:acpu=acpu.rrd:acpu:AVERAGE AREA:acpu#00CC00:Utilisation GPRINT:acpu:LAST: \: %5.1lf%s (cur) GPRINT:acpu:MAX: \: %5.1lf%s (max) GPRINT:acpu:MIN: \: %5.1lf%s (min) GPRINT:acpu:AVERAGE: \: %5.1lf%s (avg)\n The rrd files get created in the correct place and the pages in hobbit show the graph but there is just no data being displayed and the timestamp on the acpu.rrd files is showing they are not being updated. This is where I'm stumped, the log files don't show any relevant info or messages to point me to anything thats wrong. I'm running it on debian sarge distro using the .deb package from sourceforge. Ger. _____ From: Hubbard, Greg L [mailto:greg.hubbard at eds.com] Sent: 30 November 2006 19:55 To: hobbit at hswn.dk Subject: RE: [hobbit] Custom RRD graphs Why bother when Hobbit will use your echo output to do that? But there is more to graphing than simply creating the RRD files. Next you have to create graph definitions that will use your new files, and these graphs have to work. Then you can add your new graphs to the Hobbit server so they will show up with the associated test (TEST2RRD variable) and/or in the trends display. You say you see the data being echoed -- are the RRD files not getting created/updated? Did you add your rrd-ext to the right place on the server? Did you add it to hobbit-rrd for both data and status messages? (or whatever, I am going from dim memory) GLH _____ From: Ger Maguire [mailto:ger at fwi.ie] Sent: Thursday, November 30, 2006 1:44 PM To: hobbit at hswn.dk Subject: RE: [hobbit] Custom RRD graphs Guys, When creating custom scripts, has anyone just used rrdcreate and rrdtool update to directly output the data to the rrd files ? If the the tests are all residing on the one machine and not polling data from other clients is it an easier way to update the graphs as I just can't get the status script to update for me ? Thanks, Ger. _____ From: Ger Maguire [mailto:ger at fwi.ie] Sent: 29 November 2006 20:13 To: hobbit at hswn.dk Subject: [hobbit] Custom RRD graphs Hi folks, I have created a custom script for capturing CPU utilisation from a device and updated all the relevant files but I cannot get my rrd files to update correctly. An initial rrd file gets created but the data does not get input into it. I have created a rrd-ext sript which is running and is capturing the incomming status data and by looking at the /tmp files I can see that the echo outputs are being generated correctly every 5 mintes. ----> rrd-ext.sh HOSTNAME="$1" TESTNAME="$2" FNAME="$3" if [ "$TESTNAME" = "acpu" ] then echo "`date`" >> /tmp/hobbit.$HOSTNAME.$TESTNAME echo "$FNAME" >> /tmp/hobbit.$HOSTNAME.$TESTNAME # Analyze the message we got # ACPU=`grep "connected" $FNAME | awk '{print $14}'` ACPU=`grep "The 5" $FNAME | cut -d"(" -f2 | cut -d"." -f1` # The RRD dataset definitions echo "DS:acpu:GAUGE:600:0:100" # The filename echo "acpu.rrd" # The data echo "$ACPU" echo "$ACPU" >> /tmp/hobbit.$HOSTNAME.$TESTNAME fi The value I want to graph is simply a cpu % counter, have I got the above values and setting correct for this ? Any help or pointers appreciated. Output that this produces is below showing that it is getting the data correctly from the status input. ---> /tmp/hobbit.$HOSTNAME.$TESTNAME Wed Nov 29 19:54:07 GMT 2006 /usr/lib/hobbit/server/tmp/rrd_msg_13879 14 Wed Nov 29 19:59:05 GMT 2006 /usr/lib/hobbit/server/tmp/rrd_msg_26569 25 Wed Nov 29 20:04:07 GMT 2006 /usr/lib/hobbit/server/tmp/rrd_msg_2788 17 Wed Nov 29 20:09:12 GMT 2006 /usr/lib/hobbit/server/tmp/rrd_msg_11944 9 Ger.

2 1

bbmap
by ger＠fwi.ie 03 Dec '06

03 Dec '06

For those who haven't seen bbmap has been patched for hobbit, good to have the map status back again. http://bbmap.free.fr/ Thanks to Daniel McDonald. Ger.

1 0

problems with multiple hobbit servers
by khanrahan＠charter.net 02 Dec '06

02 Dec '06

I am having a problem monitoring ports and procs on multiple hobbit servers. I have a main server (server A) watching 2 dozen hosts. I setup a 2nd server (server B) to watch server A. I want each of them to report on themselves as well as the other server. I ser hobbitclient.cfg to look like this : BBDISP="0.0.0.0" # IP address of the Hobbit server BBDISPLAYS="10.164.138.30|10.164.53.55" # IP of multiple Hobbit servers. They both report ALMOST everything to the other. I set up certain procs to be watched on server B on the localclients.cfg: PROC sshd 1 5 PORT "LOCAL=%([.:]22)$" state=LISTEN min=1 max=5 color=red TRACK=SSH "TEXT=SSH" They all show up on both servers but I setup the same things on server A in the same places, and it shows up in NEITHER! What am I doing wrong? Kevin

1 0

printer monitoring v. static bb-hosts
by hobbit＠anastigmatix.net 02 Dec '06

02 Dec '06

Hi, I started writing a server extension for CUPS monitoring (not just testing that the CUPS /server/ is there, but really monitoring the queue states). The script can easily learn all of the existing queues and their states from the CUPS server, and report status with $BB for each queue using the queue name as hostname and a few different columns (running, accepting, jobcount). So, it feels kind of redundant to have to go and add all of the queue names to bb-hosts (or have them all treated as ghosts with no way to specify what page they appear on). How difficult would it be to support a kind of place-holder entry in bb-hosts, maybe as simple as a group-only tag with a title and no explicit hosts beneath it: group-only running|accepting|jobcount Printers and then some modified form of the 'status' command in bb: $BB $BBDISP "gstatus Printers laser1.running green $(date)" I bet such a capability would be useful for monitoring a lot of things besides printers. Does it seem feasible? Chapman Flack

2 4

[hobbit] SMTP and FTP Services Red
by geoff＠hallford.ca 02 Dec '06

02 Dec '06

I tried putting all IP's and hostname's into their respective hosts files and changing the resolution order but it didn't help. It appears the server on the far side is just plain slow to respond with a 220. The only thing that would help is if I can set the time delay (wait 15 seconds before failing). I think this would be a good feature to have if it doesn't exist already. Thanks anyways. On 11/29/06, PNIXON at ci.somerville.ma.us <PNIXON at ci.somerville.ma.us> wrote: > > Did you try putting an entry for Hobbit server into their hosts files? I > think the delay is from the Server trying to do a reverse lookup on the IP > Address. > > > You may also have to change the order in which they resolve DNS.... > > in /etc/nsswitch.conf (or whatever is appropriate for your distribution) > > change from > hosts: db files nisplus nis dns > to > hosts: files nisplus nis dns db > > > ------------------------------ > *From:* Geoff Hallford [mailto:geoff.hallford at gmail.com] > *Sent:* Wednesday, November 29, 2006 3:46 PM > *To:* hobbit at hswn.dk > *Subject:* [hobbit] SMTP and FTP Services Red > > Hello, > > On a few of our servers the SMTP and FTP services are RED but when I > telnet to them, they do show the proper 220 status good. They are quite slow > to respond though, taking sometimes 10-15 seconds. Is there anyway to get > Hobbit to pause and wait for this? If not, how long does it wait before it > shows a red status? > > Thank you very much for your help. > > Geoff Hallford > > > -- 'If my answers frighten you then you should cease asking scary questions.' --Sam Jackson from Pulp Fiction

4 7

Hobbit-alert depends?
by dan.mcdonald＠austinenergy.com 01 Dec '06

01 Dec '06

Currently there is a depends construct in bb-hosts with a fair number of limitations. I would like to set up some dependencies in the alerts section instead. For example, I monitor ups's for estimated runtime and actual time on battery. I currently alert whenever a UPS goes on battery, and when runtime is less than 30 minutes. I would like to alert on runtime only when we are on battery. -- Daniel J McDonald, CCIE # 2495, CISSP # 78281, CNX Austin Energy http://www.austinenergy.com

1 0