- Xymon - Xymon mailman web

hobbitd_alert crashes
by Dominique.Frise＠unil.ch 03 Jun '06

03 Jun '06

Hi, This is snapshot of 01 june running on Solaris 9. [bb at iris tmp]$ gdb ../bin/hobbitd_alert core GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "sparc-sun-solaris2.9"... Core was generated by `hobbitd_alert --checkpoint-file=/soft/pub/BB/hobbit/server/tmp/alert.chk --chec'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libpcre.so.0...done. Loaded symbols for /usr/local/lib/libpcre.so.0 Reading symbols from /usr/lib/libresolv.so.2...done. Loaded symbols for /usr/lib/libresolv.so.2 Reading symbols from /usr/lib/libsocket.so.1...done. Loaded symbols for /usr/lib/libsocket.so.1 Reading symbols from /usr/lib/libnsl.so.1...done. Loaded symbols for /usr/lib/libnsl.so.1 Reading symbols from /usr/lib/libc.so.1...done. Loaded symbols for /usr/lib/libc.so.1 Reading symbols from /usr/lib/libdl.so.1...done. Loaded symbols for /usr/lib/libdl.so.1 Reading symbols from /usr/lib/libmp.so.2...done. Loaded symbols for /usr/lib/libmp.so.2 Reading symbols from /usr/platform/SUNW,Sun-Fire-480R/lib/libc_psr.so.1...done. Loaded symbols for /usr/platform/SUNW,Sun-Fire-480R/lib/libc_psr.so.1 #0 0xff1a05c8 in _libc_kill () from /usr/lib/libc.so.1 (gdb) Dominique UNIL - University of Lausanne

2 7

Generated Reports coredump
by CHRIS.MORRIS＠RWEnpower.com 02 Jun '06

02 Jun '06

Hi, This problem has been outstanding for quite a while now on AIX. When running the pre-generated reports, bbgen coredumps. I have run /usr/local/hobbit/server/bin/bbcmd --env=/usr/local/hobbit/server/etc/hobbitserver.cfg hobbitreports.sh weekly with a --debug on the bbgen command and get output like :- 2006-05-31 16:19:00 find_or_create_column(hobbitd) 2006-05-31 16:19:00 Skipped to entry starting 1133359651 2006-05-31 16:19:00 Got entry starting 1133359351 lasting 300 2006-05-31 16:19:00 Got entry starting 1133359651 lasting 3112 2006-05-31 16:19:00 Got entry starting 1133362763 lasting 301 2006-05-31 16:19:00 Got entry starting 1133363064 lasting 412240 2006-05-31 16:19:00 Got entry starting 1133775304 lasting 301 2006-05-31 16:19:00 Got entry starting 1133775605 lasting 3915090 2006-05-31 16:19:00 Got entry starting 1137690695 lasting 0 2006-05-31 16:19:00 Got entry starting 1137690695 lasting 7770444 2006-05-31 16:19:00 Got entry starting 1145461139 lasting 300 2006-05-31 16:19:00 Got entry starting 1145461439 lasting 493186 2006-05-31 16:19:00 Got entry starting 1145954625 lasting 301 2006-05-31 16:19:00 Got entry starting 1145954926 lasting 8235 2006-05-31 16:19:00 Got entry starting 1145963161 lasting 301 2006-05-31 16:19:00 Got entry starting 1145963462 lasting 1986143 2006-05-31 16:19:00 Got entry starting 1147949605 lasting 301 2006-05-31 16:19:00 Got entry starting 1147949906 lasting 449557 2006-05-31 16:19:00 Got entry starting 1148399463 lasting 300 2006-05-31 16:19:00 Got entry starting 1148399763 lasting 71611 2006-05-31 16:19:00 Reporting starts with this entry: Tue May 23 16:56:03 2006 green 1148399763 71611 2006-05-31 16:19:00 In-range entry starting 1148425200 lasting 46174 color 0: Tue May 23 16:56:03 2006 green 1148399763 71611 2006-05-31 16:19:00 In-range entry starting 1148471374 lasting 301 color 4: Wed May 24 12:49:34 2006 yellow 1148471374 301 2006-05-31 16:19:00 Looking at history logfile /usr/local/hobbit/data/histlogs/tru2407/hobbitd/Wed_May_24_12:49:34_2006 /usr/local/hobbit/server/bin/hobbitreports.sh: line 83: 26868 IOT/Abort trap (core dumped) BBWEB=$REPORTTOPURL/$REPDIR $BBHOME/bin/bbgen --debug --reportopts=$STIME:$ETIME:0:nongr $BBGENREPOPTS $REPORTTOPDIR/$REPDIR If I remove the bbgen & hobbitd history logs, then the report runs through ok. Does this give any more clues? Regards, Chris **************************************************************************** The information contained in this email is intended only for the use of the intended recipient at the email address to which it has been addressed. If the reader of this message is not an intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination or copying of the message or associated attachments is strictly prohibited. If you have received this email in error, please contact the sender by return email or call 01793 877777 and ask for the sender and then delete it immediately from your system.Please note that neither RWE npower nor the sender accepts any responsibility for viruses and it is your responsibility to scan attachments (if any). *****************************************************************************

2 2

Hobbit Ping
by Alun.Watson＠fp.co.nz 02 Jun '06

02 Jun '06

Hello again, Has anyone else noticed the ping times increase by a large amount as soon as you change from fping to hobbit ping? On our site ping times to clients were averaging 1.5m and now they are 29.9m. I have 4.2-alpha-20060502 installed on Solaris 10 and have been updating to the latest snapshot on a fairly regular basis. Cheers, Alun

3 4

Easy question on GRAPHS and trends
by emayoral＠arsys.es 02 Jun '06

02 Jun '06

I have successfully migrated from BB to hobbit (excellent software IMHO), however there is a small thing that I cannot get solved. I collect rrd data for the tests vmstat, vmstat1, vmstat2,vmstat3,vmstat4 and vmstat5 If I build an URL like: http://MYBBDISPLAY/hobbit-cgi/hobbitgraph.sh?host=AHOBBITCLIENT&service=vms… I can see the graph just fine (so the data on the rrd files and the graph definitions - which come standard on hobbitgraph.cfg - are OK), however I cannot get hobbit to display the graph and link in the trends page. It only displays vmstat , but not vmstat1-vmstat5 According to the documentation I should just add the graph names to the GRAPHS variable at hobbitserver.cfg, but I have already done that: GRAPHS="la,disk,inode,qtree,memory,users,vmstat,vmstat3,vmstat4,vmstat5,iostat,tcp.http,tcp,netstat,temperature,ntpstat,apache,bind,sendmail,mailq,socks,bea,iishealth,citrix,bbgen,bbtest,bbproxy,hobbitd,postfix,mrtg::1" The funny thing is that I also made a custom graph 'postfix' for postfix queues and that one shows up on the trends page perfectly, it is only the vmstat family which refuses to appear. My version is hobbit-4.1.2p1-1 on Fedora Core 3. Thanks in advance for your help.

4 9

FreeBSD client where?
by kolbjorn.barmen＠uninett.no 02 Jun '06

02 Jun '06

I'm looking for a hobbit client package for FreeBSD.. is there such a thing? Or perhaps a tarball for ports? I'm not so much into how FreeBSD does these things, from Gentoo I'm used to having an overlay for my own packages, but I've never heard of such functionality in FreeBSD ports. -- Kolbjørn Barmen UNINETT Driftsenter

2 2

msgs clear
by iain＠uk.colt.net 02 Jun '06

02 Jun '06

Morning people, Hopefully, this has not bee discussed on the list before, but here goes anyway... I have replaced my BB server with a hobbit server. Very nice drop in replacement :) I am still getting my head round the differences, but most issues I have had have been solved. Apart from.. My server is called bb02.colt, running on linux. I have added this entry into the ~/server/etc/client-local.cfg [linux] log:/var/log/messages:10240 ignore identification ignore modprobe trigger sudo and this entry into the ~/server/etc/hobbit-clients.cfg LOG /var/log/messages kernel LOG /var/adm/messages ERROR But I get no msgs status on the hobbit servers web pages e.g. System logs at Thu Jun 1 10:07:18 BST 2006 : No log data available The client did not report any logfile data Both the config files are owned by user hobbit. Can a knowledgable person point out where I may have gone wrong please :) Cheers -- Iain Conochie UNIX System Administrator UK COLT Telecom Units 9-13 Powergate Business Centre, Volt Avenue, London NW10 6PW Tel: +44 (0) 20 7947 1286 Tel: +44 (0) 78 1514 5493 Email: iain.conochie at colt.net www.colt.net

3 4

GRAPHS syntax
by Dominique.Frise＠unil.ch 02 Jun '06

02 Jun '06

Hi, I cannot find a complete syntax information for the GRAPHS setting in hobbitserver.cfg. From hobbitserver.cfg: # This defines which RRD files to include on the "trends" column webpage, # and the order in which they appear. From the man page: GRAPHS List of the RRD databases, that should be shown as a graph on the "trends" column. We noticed the use of two separator (:) like "mrtg::1" Can you please describe the use for this. Dominique UNIL - University of Lausanne

2 1

moving hobbit from Ubuntu to Debian
by dan.simoes＠gmail.com 02 Jun '06

02 Jun '06

I got an account on a new virtual host (debian 3.1 on virtuozzo), and dropped in a tar of my hobbit dir. I figured I could just edit bb-hosts and be up and running in minutes. Hours later, I'm still not up. OK, so I was missing fping, and rrdtool. Hobbit did not like the older rrdtool 1.X under debian stable, so I switch to testing, upgraded, and recompiled and reinstalled hobbit. It's still not working, even with a simplified bb-hosts. hobbit runs, but the webpage output is devoid of any host data. I can't figure out what else could be wrong, this should have been a very simple port. Logs show: bb--display: 2006-05-31 17:30:22 Whoops ! bb failed to send message - timeout 2006-05-31 17:30:22 hobbitd status-board not available 2006-05-31 17:30:22 WARNING: Runtime 92 longer than BBSLEEP (60) 2006-05-31 17:31:09 Whoops ! bb failed to send message - timeout bb-network: 2006-05-31 17:30:27 Whoops ! bb failed to send message - timeout 2006-05-31 17:30:27 WARNING: Runtime 140 longer than time limit (120) 2006-05-31 17:31:14 Whoops ! bb failed to send message - timeout hobbitclient.log: 2006-05-31 17:32:47 Whoops ! bb failed to send message - timeout error parsing /proc/net/snmp: Success 2006-05-31 17:34:47 Whoops ! bb failed to send message - timeout error parsing /proc/net/snmp: Success

3 10

[hobbit] moving hobbit from Ubuntu to Debian
by Mike.Rowell＠Rightmove.co.uk 02 Jun '06

02 Jun '06

As another idea, I had a very similar problem over the last few days, same Whoops messages and it turned out to be a problem with the resolv.conf, not sure exactly what the problem with it was as I was in a bit of a rush as I had been trying to figure out the problem all day. I cleared it down to just the nameservers no search or domain parameters and it all started working fine again (This is on Solaris 10) Mike -----Original Message----- From: John Glowacki [mailto:johng at idttechnology.com] Sent: 02 June 2006 05:42 To: hobbit at hswn.dk Subject: Re: [hobbit] moving hobbit from Ubuntu to Debian This probably isn't going to be much help with this problem, but Henrik might find the feed back of some interest. I recently rebooted a Sun server. Later the hobbit client status for the server went purple and I noticed I was getting these errors in the hobbitclient.log on that server: 2006-05-25 06:58:38 Whoops ! bb failed to send message - timeout 2006-05-25 07:03:43 Whoops ! bb failed to send message - timeout 2006-05-25 07:08:47 Whoops ! bb failed to send message - timeout However I was getting one status from the client. It was from an add-on application called SE Tool Kit. It displayed the following status: green Disks 2%busy No activity green Nets No activity green NFS No client NFS/RPC activity green Swap There is a lot of unused swap space green RAM 86%free RAM available green Kmem No worries, mate green CPU CPU idling green Mutex No worries, mate green DNLC No worries, mate green Inode No activity yellow TCP High retransmissions, check TCP patch level From this we were able to determine there was a network problem. Our LAN group got involved and resolved the retransmission errors from their side. Getting back to the original Whoops error. In our case it seems the hobbit client was not getting thru because of the high retransmissions. As for why the addon messages got thru and the rest of the hobbit client messages didn't I will leave that up to Henrik for his thoughts on it. For those who may be interested in setting up SE Tool Kit with the hobbit client for Sun. You can get it from here http://www.sunfreeware.com/setoolkit.html. There is an example file included in the package called bigbrother.se. I renamed it to hobbit.se, updated the path for calling bin/bb and changed the BBHOST variable to BBDISP. Add an entry to clientlaunch.cfg: [se] ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg CMD /opt/RICHPse/bin/se /opt/hobbit/client/hobbit.se LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log We also run Orca(http://www.orcaware.com/orca/). It's a performance and trend analysis package. It also uses the SE Tool kit and creates detailed graphs. Here is an what it looks like. http://www.orcaware.com/orca/docs/orcallator.html#TCP_retransmission_dup licate_received_percentage With how fast Hobbit is growing we may someday be able to drop the Orca package. John Dan Simoes wrote: > I appreciate the help. Running the debug you sent, it seems that the box > cannot connect to itself on 1984 using its public IP. > I will see if I can remove the firewall for now (it's not iptables, it's > something the provider gives you). > Thanks again. > > On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote: > >> >> One more thing before I go to sleep :-) >> >> Check your hobbitserver.cfg - if you have BBDISP set the same as >> BBSERVERIP. That *is* the default, but you might have changed it. >> It should be >> BBDISP="$BBSERVERIP" >> in hobbitserver.cfg. >> >> >> Regards, >> Henrik >> >> On Thu, Jun 01, 2006 at 12:04:55AM +0200, Henrik Stoerner wrote: >> > Login as the hobbit user, then run >> > ./server/bin/bbcmd bbgen --debug | less >> > A bit into that log is where bbgen tries to fetch the current status >> > from the Hobbit daemon. It looks like this: >> > >> > 2006-06-01 00:02:24 load_state() >> > 2006-06-01 00:02:24 Transport setup is: >> > 2006-06-01 00:02:24 bbdportnumber = 1984 >> > 2006-06-01 00:02:24 bbdispproxyhost = NONE >> > 2006-06-01 00:02:24 bbdispproxyport = 0 >> > 2006-06-01 00:02:24 Recipient listed as '127.0.0.1' >> > 2006-06-01 00:02:24 Standard BB protocol on port 1984 >> > 2006-06-01 00:02:24 Will connect to address 127.0.0.1 port 1984 >> > 2006-06-01 00:02:24 Connect status is 0 >> > 2006-06-01 00:02:24 Sent 118 bytes >> > 2006-06-01 00:02:24 Read 3364 bytes >> > >> > Does it report the right IP-address as the 'Recipient' ? >> > >> > Henrik >> > >> > >> > On Wed, May 31, 2006 at 03:00:19PM -0700, Dan Simoes wrote: >> > > Telnet to localhost as well as 127.0.0.1 work, yes. >> > > The hobbitdboard returns a lot of data. >> > > >> > > I've literally been looking at this for hours, I don't know what else >> is >> > > wrong. >> > > I even did a completely fresh install, dropped in my bb-hosts, same >> results. >> > > I know bb-hosts is OK since it's working elsewhere (I did change the >> bbd >> > > line of course). >> > > >> > > On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote: >> > > > >> > > >Can you telnet to 127.0.0.1 port 1984 ? If that times out, then you >> have >> > > >a seriously heavy firewall setup. Something like >> > > > iptables -I INPUT 1 -i lo -j ACCEPT >> > > > iptables -I OUTPUT 1 -i lo -j ACCEPT >> > > >should cancel out any paranoid firewall setups locally on the box, >> for >> > > >the loopback-interface. >> > > > >> > > >Can you do a "~hobbit/server/bin/bb 127.0.0.1 hobbitdboard" and get >> any >> > > >data back ? >> > > > >> > > >If you're running all of the network tests locally, I think you can >> > > >get away with setting >> > > > BBSERVERIP="127.0.0.1" >> > > >in your hobbitserver.cfg, and let all of the communications >> happen on >> > > >the loopback-interface. Of course, for clients to work you'll >> need to >> > > >get port 1984 opened up for access from the clients. But loopback >> should >> > > >be ok for just getting the Hobbit server up and running. >> > > > >> > > > >> > > >Henrik >> > > > >> > > > >> > > >On Wed, May 31, 2006 at 02:48:33PM -0700, Dan Simoes wrote: >> > > >> I thought of that, eventually, and corrected the error. Same >> problem. >> > > >> localhost is responding on 1984, but the site's firewall is >> blocking >> > > >1984. >> > > >> I don't have any clients connecting at the moment, so I don't >> imagine >> > > >this >> > > >> is a problem, but thought I would mention it. >> > > >> >> > > >> On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote: >> > > >> > >> > > >> >Check the IP-adresses for the Hobbit server in the >> hobbitserver.cfg >> > > >> >file. From your logs, it seems that the query to fetch the >> current >> > > >total >> > > >> >status from the Hobbit daemon goes to the wrong server (the old >> IP, >> > > >> >probably), so there is no response and therefore the webpages >> dont >> get >> > > >> >any hosts listed. >> > > >> > >> > > >> > >> > > >> >Henrik >> > > >> > >> > > >> >On Wed, May 31, 2006 at 02:41:25PM -0700, Dan Simoes wrote: >> > > >> >> I got an account on a new virtual host (debian 3.1 on >> virtuozzo), and >> > > >> >> dropped in a tar of my hobbit dir. >> > > >> >> I figured I could just edit bb-hosts and be up and running in >> > > >minutes. >> > > >> >> Hours later, I'm still not up. >> > > >> >> >> > > >> >> OK, so I was missing fping, and rrdtool. Hobbit did not like >> the >> > > >older >> > > >> >> rrdtool 1.X under debian stable, so I switch to testing, >> upgraded, >> > > >and >> > > >> >> recompiled and reinstalled hobbit. >> > > >> >> It's still not working, even with a simplified bb-hosts. >> > > >> >> >> > > >> >> hobbit runs, but the webpage output is devoid of any host data. >> > > >> >> I can't figure out what else could be wrong, this should have >> been a >> > > >> >very >> > > >> >> simple port. >> > > >> >> >> > > >> >> Logs show: >> > > >> >> >> > > >> >> bb--display: >> > > >> >> 2006-05-31 17:30:22 Whoops ! bb failed to send message - >> timeout >> > > >> >> 2006-05-31 17:30:22 hobbitd status-board not available >> > > >> >> 2006-05-31 17:30:22 WARNING: Runtime 92 longer than BBSLEEP >> (60) >> > > >> >> 2006-05-31 17:31:09 Whoops ! bb failed to send message - >> timeout >> > > >> >> >> > > >> >> bb-network: >> > > >> >> 2006-05-31 17:30:27 Whoops ! bb failed to send message - >> timeout >> > > >> >> 2006-05-31 17:30:27 WARNING: Runtime 140 longer than time limit >> (120) >> > > >> >> 2006-05-31 17:31:14 Whoops ! bb failed to send message - >> timeout >> > > >> >> >> > > >> >> >> > > >> >> hobbitclient.log: >> > > >> >> 2006-05-31 17:32:47 Whoops ! bb failed to send message - >> timeout >> > > >> >> error parsing /proc/net/snmp: Success >> > > >> >> 2006-05-31 17:34:47 Whoops ! bb failed to send message - >> timeout >> > > >> >> error parsing /proc/net/snmp: Success >> > > >> > >> > > >> >-- >> > > >> >Henrik Storner >> > > >> > >> > > >> >To unsubscribe from the hobbit list, send an e-mail to >> > > >> >hobbit-unsubscribe at hswn.dk >> > > >> > >> > > >> > >> > > >> > >> > > > >> > > >-- >> > > >Henrik Storner >> > > > >> > > >To unsubscribe from the hobbit list, send an e-mail to >> > > >hobbit-unsubscribe at hswn.dk >> > > > >> > > > >> > > > >> > >> > -- >> > Henrik Storner >> > >> > To unsubscribe from the hobbit list, send an e-mail to >> > hobbit-unsubscribe at hswn.dk >> > >> > >> >> -- >> Henrik Storner >> >> To unsubscribe from the hobbit list, send an e-mail to >> hobbit-unsubscribe at hswn.dk >> >> >> > To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk ________________________________________________________________________ This email has been scanned for all viruses by the MessageLabs service. ________________________________________________________________________ This email has been scanned for all viruses by the MessageLabs service. ________________________________________________________________________

1 0

Multiple Displays
by paul.moore＠verizonbusiness.com 02 Jun '06

02 Jun '06

I'm trying to setup a redundant set of servers to be the BBDISPLAYS and a redundant set of servers to be the BBNET pollers. I've managed to setup everything and get it working the way I like except one problem I am having. I've created several scripts to check various items and send status messages to the BBDISPLAY using the $BB cmd. BB="$BBHOME/bin/bb" BBHOME="/usr/local/hobbit/server" BBDISP="0.0.0.0" BBDISPLAYS="172.16.1.50 172.16.2.50" The cmd I'm running looks like this. $BB $BBDISP "status $MACHINE.$COLUMN $COLOR $DATE\t$MSG\n\n"` The result appears that the /bin/bb command will fire off a status but only to the first BBDISPLAY in the list. Is there some other way to make it fire off status to both servers? Paul -=-=-= Pinky, Are You Pondering What I'm Pondering? Well, I think so Brain but if Jimmy cracks corn and no one cares, why does he keep doing it?

4 3