August 2005 - Xymon - Xymon mailman web

hobbit_rrd stops working after about 1 hour
by Naeem.Maqsud＠sybase.com 23 Aug '05

23 Aug '05

Well, as nobody has suggested anything to my problem I guess that I'm the only one having this issue. I have managed to find the root cause. The hobbitd_rrd process was showing to be in "uninterruptible sleep" state most of the time with high iowait associated with the CPU it was running on. I suspected that the problem may be due to disk IO while updating rrds for the 2000 hosts. I created a tmpfs filesystem and copied the rrd directory into it. Since then (48 hours ago) my rrd graphs have been updating continuously. I do however need to write back to disk periodically to avoid loss of data after a reboot. This is OK as a temporary fix but I would like to have a permanent solution. I would like to hear from other hobbit users who have more than 1000 hosts monitored. What type of servers and disk subsystems are they using? Perhaps my problem is to do with RedHat and Dell server combination. Perhaps I need to stripe over multiple spindles. -Naeem Naeem Maqsud/SYBASE To 08/18/2005 05:02 hobbit at hswn.dk PM cc Subject hobbit_rrd stops working after about 1 hour Hi, I'm testing out hobbit 4.1.1 for possible migration from big brother (with bbgen). I suspected scalability issues with BB as my rrd graphs were updated intermittently. However, hobbit is exhibiting similar problems. After about 1 hr of restarting hobbit, the rrd graphs stop updating except for the cpu utilization for the hobbit server itself. The hobbit server is running RedHat Linux AS 3.0. It has 2 x 2.4 GHz Xeon processors and 1GB of memory. About 800 servers are sending updates to the hobbit server. Another 1200 servers are getting remote tests. Load average has stayed below 1 most of the time. CPU usage has been low with 75% idle. 4 CPUs show up due to hyperthreading and I've noticed that after the restart of hobbit server, hobbitd_rrd process stays on CPU3 with 100% utilization for the one hour that it is busy. I hope someone can shed some light on this. Thanks, Naeem

3 5

issue with vmstat data on RHEL3
by tgeorgoulias＠mcclatchy.com 23 Aug '05

23 Aug '05

I'm still having some problems with Hobbit properly graphing the vmstat data on my RHEL3 systems. It switches the cpu_idl and cpu_wait values, even though I believe I have Hobbit configured correctly. I'm using the 8/22/05 snapshot. I launch the client using: ./runclient.sh --os=rhel3 start The client data file shows that the client data is reporting the vmstat values correctly: [vmstat] procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy wa id 1 0 0 535876 201132 1029684 0 0 0 1 1 0 0 0 0 0 0 0 0 534252 201132 1029716 0 0 0 47 117 42 0 0 1 98 But the values stored in the vmstat.rrd file are reversed, which seems to indicate that hobbitd is still using the "linux" vmstat layout, not the rhel3. timestamp cpu_r cpu_b cpu_w mem_swpd mem_free mem_buff mem_cach mem_si mem_so dsk_bi dsk_bo cpu_int cpu_csw cpu_usr cpu_sys cpu_idl cpu_wait 1124816100: 0.0000000000e+00 0.0000000000e+00 nan 0.0000000000e+00 5.3530218667e+05 2.0113200000e+05 1.0296953067e+06 0.0000000000e+00 0.0000000000e+00 0.0000000000e+00 4.7646666667e+01 1.1894000000e+02 4.4586666667e+01 6.4666666667e-01 0.0000000000e+00 1.6466666667e+00 9.8000000000e+01 Where did I go wrong? Tom

1 0

hobbit_rrd Fatal signal caught! with rrdtools 1.2.11
by lgrilli＠be.tiauto.com 23 Aug '05

23 Aug '05

Hi all, Our hobbit_rrd crash on FreeBSD 5.4 with rrdtools 1.2.11 In rrd-status.log 2005-08-22 23:45:18 RRD error updating /home/hobbit/data/rrd/X.X.X.X/la.rrd from X.X.X.X: reading the cookie off /home/hobbit/data/rrd/X.X.X.X/la.rrd faild 2005-08-22 23:45:18 RRD error updating /home/hobbit/data/rrd/X.X.X.X/procs.rrd from X.X.X.X: reading the cookie off /home/hobbit/data/rrd/X.X.X.X/procs.rrd faild 2005-08-22 23:45:18 RRD error updating /home/hobbit/data/rrd/X.X.X.X/users.rrd from X.X.X.X: '/home/hobbit/data/rrd/X.X.X.X/users.rrd' is not an RRD file 2005-08-22 23:45:18 RRD error updating /home/hobbit/data/rrd/X.X.X.X/disk,C.rrd from X.X.X.X: reading the cookie off /home/hobbit/data/rrd/X.X.X.X/disk,C.rrd faild ..... ..... 2005-08-22 23:49:26 Worker process died with exit code 134, terminating Plenty of these entries, of course X.X.X.X is normal information This is a fresh install of Hobbit 4.1.1. Is it due to the version of rrd ? The same data send by the client to a Hobbit 4.1.1 run fine but with the rrdtools 1.2.10 Here is the changelog of rrd tools 1.2.11 RRDtool 1.2.11 is released ... ------------------------------ You can download the latest release of RRDtool from http://people.ee.ethz.ch/oetiker/webtools/rrdtool/pub New Features ------------ * Graph generation is much faster with this release. An MRTG type graph is about twice as fast, and SmokePing graph about 6 times as fast. * RRDtool in interactive mode does now only quit when it really has to. In all other cases it complains and waits for new input. Bugfixes -------- * When stacking known on unknown data, the graph does not go haywire anymore. * rrdtool graph with --no-gridfit and --only-graph does now produce sensible output * configure works now with gcc-4.0 * changing just the font and not the size of the font as described in the man page now actually works. -- Havard Eidnes * the makefiles now are less GNUish and work with Solaris standard make too. -- Stan Sinyagin * make install locations more 'standard' -- Stan Sinyagin * bottom align the x-axis labels, this helps with languages where not every weekday name has the same 'height' By the way how do you compute the number off rrd that you got on your systems ? Thks for any advice, Laurent Grilli The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. This communication is from TI Automotive.

2 1

badTEST:x:y:z not working
by Heinecke＠hansenet.com 23 Aug '05

23 Aug '05

Dear all, looks to me, as if the "badTEST:x:y:z" tags are not working. I.e. "badssh:1:2:3" turns immediatley red if ssh goes down at the monitored host. I've looked into bbtest-net.c and figured out, that .. - At line 775, the for loop << "for (swalk=svchead; (swalk && strcmp(swalk->testname, testname)); swalk = swalk->next) >> compares swalk->testname and testname ---> which are never equal, because the string testname always includes the :x.y:z settings and swalk->testname (of course) not. Hmmm... your experiences? I've found no posting about this kind of issue on the list, and i think "badTEST" is one of the most important tags. I'am using 4.1.1 (4.1.2 (snapshot 09.08.05 also testet - same effect) @ Linux (Debian sarge and Solaris 8 :-) Thx! Michael --------------- Michael Heinecke CAX / UNIX&VoD HanseNet Telekommunikation GmbH Überseering 33 a, 22297 Hamburg Telefon: +49 (0)40 23726-2768 Telefax: +49 (0)40 23726-3485 http://www.alice-dsl.de <http://www.alice-dsl.de/> , http://www.hansenet.de <http://www.hansenet.de/>

4 3

[hobbit] bug, alarming on memory?
by David.Gore＠mci.com 22 Aug '05

22 Aug '05

I suppose I am the only one, who cannot get the memory settings to fire in hobbit-clients.cfg? Here is a little snippet from bbcmd: hobbit at hobbit:/export/home/hobbit/server> bin/bbcmd hobbitd_client --test 2005-08-22 23:30:36 Using default environment file /export/home/hobbit/server/etc/hobbitserver.cfg Hostname (.=end, ?=dump, !=reload) []: myHost05 Test (cpu, mem, disk, proc): mem Phys: Yellow at 100, red at 101 Swap: Yellow at 90, red at 95 Act.: Yellow at 90, red at 98 Here is the definition: HOST=%myHost05 PROC "%someproc" 1 1 . . . PROC "%another proc" 1 1 PHYS 90 95 MEMSWAP 50 65 DISK / 90 95 DISK "%^/home/.*" 95 98 LOAD 18.0 22.0 # These are the built-in defaults. UP 1h LOAD 5.0 10.0 DISK * 90 95 MEMPHYS 90 98 MEMSWAP 90 95 MEMACT 90 98 There are numerous other missing, non-firing entries like these: MEMSWAP 50 65 DISK / 90 95 DISK %^/home/.* 95 98 LOAD 18.00 22.00 Should these have HOST entries after them when you do a dump with bbcmd? Like the following from the same dump? Perhaps commenting out PROC entries is causing the parsing to lose track of where it is? DISK / 90 95 HOST=%aNewHost05 DISK %^/home/.* 95 98 HOST=%aNewHost05 MEMSWAP 90 95 HOST=%aNewHost05 LOAD 18.00 22.00 HOST=%aNewHost05 I am on the snapshot from the 21st, I have been trying them daily hoping for a fix. May be my .cfg is corrupt some how? -------- Original Message -------- Subject: [hobbit] bug, alarming on memory? Date: Wed, 17 Aug 2005 18:29:26 +0000 From: David Gore <David.Gore at mci.com> To: hobbit at hswn.dk Shouldn't I get a warning alarm. There are no specific MEMxxx settings in hobbit-clients.cfg for this specific host, but it fails to work for the hosts where I do have MEMxxx settings. Physical 5903M 6144M 96% Swap 3679M 17356M 21% # These are the built-in defaults. UP 1h LOAD 5.0 10.0 DISK * 90 95 SWAP 90 95 MEMPHYS 90 98 MEMSWAP 90 95 MEMACT 90 98 I have never received a memory alarm, and I do have several hosts using more than 90% CPU, quite a number more than 95%. Where should I look? -- David Gore (v965-3670) Enhanced Technology Support (ETS) Network Management Systems (NMS) IMPACT Transport Team Lead - SCSA, SCNA Page: 1-800-PAG-eMCI pin 1406090 Vnet: 965-3676 -- David Gore (v965-3670) Enhanced Technology Support (ETS) Network Management Systems (NMS) IMPACT Transport Team Lead - SCSA, SCNA Page: 1-800-PAG-eMCI pin 1406090 Vnet: 965-3676

1 0

Macros in hobbit-clients.cfg
by rgordonjr＠gmail.com 22 Aug '05

22 Aug '05

Hello - Does the hobbit-clients.cfg file support macros? I saw that it supported the HOST directive but I didn't see anything about macro support.... Thanks! -- --==[ Bob Gordon ]==--

1 0

init.d scripts for linux, updated runclient.sh
by smith.js.8＠pg.com 22 Aug '05

22 Aug '05

For general consumption, and to give back a little... I'm running hobbit on several linux servers, most of them Red Hat or variants thereof. Following are the scripts I use to start/stop the hobbit server and client. You'll need to modify the locations for pidfile and config in the script header comments, plus the values for BBUSER and BBINIT in the scripts themselves. These are chkconfig friendly scripts. If you're using chkconfig, after creating the files you should: # chkconfig --add hobbitd # chkconfig hobbitd on # chkconfig --list hobbitd # chkconfig --add hobbitc # chkconfig hobbitc on # chkconfig --list hobbitc Then you can use the 'service' command to control hobbit. See these commands for options: # service hobbitd help # service hobbitc help I've also attached a modified runclient.sh that adds 'restart' and 'status' options. Henrik, please feel free to massage these as necessary (for example, so the configure/make commands fill in the necessary blanks) and include them in the snapshots/packages. In the code below I've use {UPPER_CASE} to indicate areas that need to be changed before use on a user's systems. ---------------------------------------- /etc/init.d/hobbitd ---------------------------------------- #!/bin/sh # # hobbitd - This script starts and stops the Hobbit server daemon # # chkconfig: - 87 13 # # description: Hobbit system monitor server daemon. # processname: hobbitd # pidfile: /{PATH_TO_LOGS}/hobbit/hobbitlaunch.pid # config: /{PATH_TO_HOBBIT}/server/etc/hobbitserver.cfg # Source function library. . /etc/rc.d/init.d/functions # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ ${NETWORKING} = "no" ] && exit 0 BBUSER="{HOBBIT_USER}" BBINIT="/{PATH_TO_HOBBIT}/server/hobbit.sh" # Pass control off to the hobbit init script. Not perfect # for every possible OS, but it does the job well enough. /bin/su - $BBUSER -c "$BBINIT \"$1\"" #EOF(hobbitd) ---------------------------------------- ---------------------------------------- /etc/init.d/hobbitc ---------------------------------------- #!/bin/sh # # hobbitc - This script starts and stops the Hobbit client daemon # # chkconfig: - 88 12 # # description: Hobbit system monitor client daemon. # processname: hobbitc # pidfile: /{PATH_TO_CLIENT_LOGS}/hobbit/clientlaunch.pid # config: /{PATH_TO_HOBBIT}/client/etc/clientlaunch.cfg # Source function library. . /etc/rc.d/init.d/functions # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ ${NETWORKING} = "no" ] && exit 0 BBUSER="{HOBBIT_USER}" BBINIT="/{PATH_TO_HOBBIT}/client/runclient.sh" # Pass control off to the hobbit init script. Not perfect # for every possible OS, but it does the job well enough. /bin/su - $BBUSER -c "$BBINIT \"$1\"" #EOF(hobbitc) ---------------------------------------- ---------------------------------------- {HOBBIT_HOME}/client/runclient.sh ---------------------------------------- #!/bin/sh #----------------------------------------------------------------------------# # Hobbit client bootup script. # # # # This invokes hobbitlaunch, which in turn runs the Hobbit client and any # # extensions configured. # # # # Copyright (C) 2005 Henrik Storner <henrik at hswn.dk> # # # # This program is released under the GNU General Public License (GPL), # # version 2. See the file "COPYING" for details. # # # #----------------------------------------------------------------------------# # # $Id: runclient.sh,v 1.4 2005/08/04 12:16:27 henrik Exp $ # Default settings for this client MACHINEDOTS="`uname -n`" # This systems hostname BBOSTYPE="`uname -s | tr '[A-Z]' '[a-z]'`" # This systems operating system in lowercase ### ### mod(smith.js.8 at pg.com)20050821 ### ### note: this was lifted from the server hobbit.sh script case "`uname -s`" in "SunOS") ID=/usr/xpg4/bin/id ;; *) ID=id ;; esac if test `$ID -un` != {HOBBIT_USER} then echo "Hobbit client must be started as the {HOBBIT_USER} user" exit 1 fi ### ### end(smith.js.8 at pg.com)20050821 ### # Commandline mods for the defaults while test "$1" != "" do case "$1" in --hostname=*) MACHINEDOTS="`echo $1 | sed -e's/--hostname=//'`" ;; --os=*) BBOSTYPE="`echo $1 | sed -e's/--os=//' | tr '[A-Z]' '[a-z]'`" ;; --help) echo "Usage: $0 [--hostname=CLIENTNAME] [--os=rhel3|linux22] start|stop" exit 0 ;; start) CMD=$1 ;; stop) CMD=$1 ;; ### ### mod(smith.js.8 at pg.com)20050821 ### restart) CMD=$1 ;; status) CMD=$1 ;; ### ### end(smith.js.8 at pg.com)20050821 ### esac shift done OLDDIR="`pwd`" cd "`dirname $0`" HOBBITCLIENTHOME="`pwd`" cd "$OLDDIR" MACHINE="`echo $MACHINEDOTS | sed -e's/\./,/g'`" export MACHINE MACHINEDOTS BBOSTYPE HOBBITCLIENTHOME case "$CMD" in "start") if test ! -w $HOBBITCLIENTHOME/logs; then echo "Cannot write to the $HOBBITCLIENTHOME/logs directory" exit 1 fi if test ! -w $HOBBITCLIENTHOME/tmp; then echo "Cannot write to the $HOBBITCLIENTHOME/tmp directory" exit 1 fi $HOBBITCLIENTHOME/bin/hobbitlaunch --config=$HOBBITCLIENTHOME/etc/clientlaunch.cfg --log=$HOBBITCLIENTHOME/logs/clientlaunch.log --pidfile=$HOBBITCLIENTHOME/logs/clientlaunch.pid if test $? -eq 0; then echo "Hobbit client for $BBOSTYPE started on $MACHINEDOTS" else echo "Hobbit client startup failed" fi ;; "stop") ### mod(smith.js.8 at pg.com)20050821-one liner-change test from -f to -s #if test -f $HOBBITCLIENTHOME/logs/clientlaunch.pid; then if test -s $HOBBITCLIENTHOME/logs/clientlaunch.pid; then kill `cat $HOBBITCLIENTHOME/logs/clientlaunch.pid` echo "Hobbit client stopped" else echo "Hobbit client not running" fi ;; ### ### mod(smith.js.8 at pg.com)20050821 ### "restart") if test -s $HOBBITCLIENTHOME/logs/clientlaunch.pid then $0 stop sleep 10 $0 start else echo "Hobbit client does not appear to be running, starting it" $0 start fi ;; "status") if test -s $HOBBITCLIENTHOME/logs/clientlaunch.pid then kill -0 `cat $HOBBITCLIENTHOME/logs/clientlaunch.pid` if test $? -eq 0 then echo "Hobbit client (clientlaunch) running with PID `cat $HOBBITCLIENTHOME/logs/clientlaunch.pid`" else echo "Hobbit client not running, removing stale PID file" rm -f $HOBBITCLIENTHOME/logs/clientlaunch.pid fi else echo "Hobbit client (clientlaunch) does not appear to be running" fi ;; *) echo "Usage: $0 start|stop|restart|status" break; ### ### end(smith.js.8 at pg.com)20050821 ### esac exit 0 #EOF(runclient.sh)

1 0

my turn to thank henrik
by olivier＠qalpit.com 22 Aug '05

22 Aug '05

4 weeks ago, i switched from bbgen3.5 to hobbit 4.1.1 i have a rather large site (over 1500 hosts) hobbit just works; and is much more stable and efficient than bb. i really encouraged those of you who havent switched yet, of course it'll take a couple days to tweak all the little things that differ from bb.. but hobbit has much more potentiel. i am doing more and more; under bb i was kind of stuck by performance and by stability. thank you henrik for this great open-source software. thank you henrik for your precious support and for being so available for us. and there is more great software coming down the line :) Best regards, Olivier Beau

2 1

SMTP service fails since 4.1.1
by matt＠oddprocess.org 20 Aug '05

20 Aug '05

I am monitoring several servers with SMTP and 2 of the servers suddenly started having problems since I upgraded to 4.1.1. I haven't changed anything on the two servers themselves, but they switch from green to yellow, and back again every few minutes. This is the response when it is green: Service smtp on sinclair is OK (up) 554 SMTP synchronization error Seconds: 0.00 And the page when it goes yellow: Service smtp on sinclair is not OK : Unexpected service response 554 SMTP synchronization error Seconds: 0.00 Both servers are running the same version of exim, but haven't been changed and are running the same version that worked in 4.1.0. Any ideas? Thanks in advance. -- Matthew Daubenspeck http://www.oddprocess.org Gentoo Linux 2.6.12-gentoo-r6 x86_64 AMD Athlon(tm) 64 Processor 2800+ 19:38:48 up 28 days, 6:05, 1 user, load average: 0.00, 0.01, 0.00

1 0

[hobbit] graph compare
by henrik＠hswn.dk 20 Aug '05

20 Aug '05

On Sun, Jul 31, 2005 at 02:01:17PM +0200, Olivier Beau wrote: > actually, i think the display should be done a different way than > with larrd-graphs.cgi : > could it be possible (and not to hard to code) to display several DS > from different host on the same graph ? > (so i could really easy compare for example the cpu load of different > host.. using LINE and not AREA fo course..) I've taken a look at your idea, so let me know what you think of this: Load avg. comparison: http://www.hswn.dk/hobbit-cgi/hobbitgraph.sh?host=voodoo.hswn.dk&host=fenri… CPU % utilization comparison (from the vmstat data): http://www.hswn.dk/hobbit-cgi/hobbitgraph.sh?host=voodoo.hswn.dk&host=fenri… I've done a couple of other service definitions - ram, swap, users, procs - but this should give you the general idea. You can of course compare more than one host by adding more "&host=..." items in the URL. Things like zoom should also work (I think it's broken now, but it did work an hour ago...) Regards, Henrik

5 4