March 2006 - Xymon - Xymon mailman web

FW: [hobbit] CUSTOM RRD DATA VIA SCRIPTS
by subhash.gada＠logicacmg.com 14 Mar '06

14 Mar '06

Hi, May be I need to put this more clearly. Few custom test are running on the host side on windows systems which measures print queue length of the print server. Hobbit is receiving data in the following format from the host systems. >From Server 1 3/13/2006 Total queued jobs: 0 PRN324:0 PRN323:0 PRN142:0 PRN141:0 PRN140:0 PRN139:0 PRN138:0 >From server 2 3/13/2006 Total queued jobs: 0 PRN327:0 PRN203:0 PRN202:0 Initially I could thought of the following script to create rrd file but if I use this script the data existing in all the rrd files of different host is similar as am using the NAMES PRN327, PRN203, PRN202 to capture the values (different hosts has different names). The length of the NAMES are also not consistent across the hosts i.e. for server1 7 name:value are sent to hobbit & for server 2 only 3 name:value How can I make the script dynamic ? #!/bin/sh # Input parameters: Hostname, testname (column), and messagefile HOSTNAME="$1" TESTNAME="$2" FNAME="$3" if [ "$TESTNAME" = "printq" ] then p1=`grep "^PRN327:" $FNAME | awk '{print $2}'` p2=`grep "^PRN203:" $FNAME | awk '{print $2}'` p3=`grep "^PRN202:" $FNAME | awk '{print $2}'` # The RRD dataset definitions echo "DS:p1:GAUGE:600:0:U" echo "DS:p2:GAUGE:600:0:U" echo "DS:p3:GAUGE:600:0:U" # The filename echo "printq.rrd" # The data echo "$p1:$p2:$p3" fi exit 0 And how can I make the configuration even dynamic in the hobbitgraph.cfg [printq] TITLE Print Queue Status YAXIS Print Queue Length DEF:p1=printq.rrd:PRN327:AVERAGE DEF:p2=printq.rrd:PRN203:AVERAGE DEF:p3=printq.rrd:PRN202:AVERAGE AREA:p1#00FF00: PRN 327 GPRINT:p1:LAST: \: %5.1lf (cur) GPRINT:p1:MAX: \: %5.1lf (max) GPRINT:p1:MIN: \: %5.1lf (min) GPRINT:p1:AVERAGE: \: %5.1lf (avg)\n AREA:p2#FF0000: PRN 230 GPRINT:p2:LAST: \: %5.1lf (cur) GPRINT:p2:MAX: \: %5.1lf (max) GPRINT:p2:MIN: \: %5.1lf (min) GPRINT:p2:AVERAGE: \: %5.1lf (avg)\n AREA:p3#0000FF: PRN 202 GPRINT:p3:LAST: \: %5.1lf (cur) GPRINT:p3:MAX: \: %5.1lf (max) GPRINT:p3:MIN: \: %5.1lf (min) GPRINT:p3:AVERAGE: \: %5.1lf (avg)\n Please let me know, if more information is required. Regards Subhash ________________________________ From: Gada, Subhash [mailto:subhash.gada at logicacmg.com] Sent: Monday, March 13, 2006 5:23 PM To: hobbit at hswn.dk Subject: [hobbit] CUSTOM RRD DATA VIA SCRIPTS Hi, We have few custom tests that measures printq of the print server (windows). How can I write a script to create a rrd file and generate graphs. For example P Server 1 3/13/2006 Total queued jobs: 0 PRN324:0 PRN323:0 PRN142:0 PRN141:0 PRN140:0 PRN139:0 PRN138:0 P Server 2 3/13/2006 Total queued jobs: 0 PRN314:0 PRN137:0 PRN136:0 How to count the number of columns that vary from host to host. The column names are not common for all the hosts. Can any one help me further ? if [ "$TESTNAME"="printq" ] then p1=`grep "^PRN*:" $FNAME | awk '{print $2}'` p2=`grep "^PRN*:" $FNAME | awk '{print $2}'` p3=`grep "^PRN*:" $FNAME | awk '{print $2}'` # The RRD dataset definitions echo "DS:p1:GAUGE:600:0:U" echo "DS:p2:GAUGE:600:0:U" echo "DS:p3:GAUGE:600:0:U" # The filename echo "printq.rrd" # The data echo "$p1:$p2:$p3" Regards, Subhash Gada This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

1 0

[hobbit] Network test dying
by JamesHorwath＠glic.com 14 Mar '06

14 Mar '06

> On Mon, Mar 13, 2006 at 12:45:27PM -0500, James B Horwath wrote: > > I have been running hobbit for several months now without incident. I am > > running hobbit 4.1.2p1 on Redhat Enterprise 3 on IBM pseries hardware. I > > haven't had any issues until this morning. Now it appears after about one > > hour of running the system flat out dies. I am sent a notification for > > every system connected. Then it appears the network process dies. I was > > running Tcpdump to see what was wrong. I see the completion of a network > > test about 30 minutes ago to a machine on the same subnet. I am not > > running iptables/ipchains. I am not experienced at hard-core hobbit > > debugging. I looked in /var/log/hobbit and don't see anything strange. > > There are no core files on the hobbit directory. > > > > Any advise on where to start? All my network test are now purple. > > Is there a "bbtest-net" and/or "fping" process which hangs ? If there > is, it would be interesting to attach to it with "gdb" and see what > it is doing. Alternatively, kill it with a "kill -6" which will trigger > a core dump in ~hobbit/data/tmp/ - you can run the core dump through > gdb, which might give me an idea what it is doing. > > > You can also try su'ing to the hobbit user and run the command > > bbcmd bbtest-net --debug host1 host2 > > (replace "host1" and "host2" with a couple of the hosts in your > bb-hosts file). > > > Is DNS lookups working on this box ? That is one of the few things that > can cause the network tests to slow down dramatically. But they ought to > time out automatically. Same goes for the other commands that run as > part of the network tests (rpc and ntp queries). Henrik, Thanks for the tips. DNS works fine and the network tests seems to work fine when I try them. My system is pretty much idle and I don't see anything nasty in the system logs. I have included the process table and an lsof bbtest-net process. When I did the kill -6 on the network process it worked once and then failed stopped again. I did a strings on the core and may have found a machine with a slove DNS resolution. I am keeping my fingers crossed. Regards, Jim [root at bigbrother etc]# ps -ef | grep hobbit hobbit 18470 1 0 15:10 ? 00:00:00 /usr/local/hobbit/server/bin/hobbitlaunch --config=/usr/local/hobbit/server/etc/hobbitlaunch.cfg --env=/usr/local/hobbit/server/etc/hobbitserver.cfg --log=/var/log/hobbit/hobbitlaunch.log --pidfile=/var/log/hobbit/hobbitlaunch.pid hobbit 18471 18470 0 15:10 ? 00:00:05 hobbitd --pidfile=/var/log/hobbit/hobbitd.pid --restart=/usr/local/hobbit/server/tmp/hobbitd.chk --checkpoint-file=/usr/local/hobbit/server/tmp/hobbitd.chk --checkpoint-interval=600 --log=/var/log/hobbit/hobbitd.log --admin-senders=127.0.0.1 10.98.200.46 hobbit 18473 18470 0 15:10 ? 00:00:00 hobbitd_channel --channel=stachg --log=/var/log/hobbit/history.log hobbitd_history hobbit 18474 18473 0 15:10 ? 00:00:00 hobbitd_history hobbit 18475 18470 0 15:10 ? 00:00:01 hobbitd_channel --channel=page --log=/var/log/hobbit/page.log hobbitd_alert --checkpoint-file=/usr/local/hobbit/server/tmp/alert.chk --checkpoint-interval=600 hobbit 18476 18475 0 15:10 ? 00:00:00 hobbitd_alert --checkpoint-file=/usr/local/hobbit/server/tmp/alert.chk --checkpoint-interval=600 hobbit 18477 18470 0 15:10 ? 00:00:19 hobbitd_channel --channel=status --log=/var/log/hobbit/rrd-status.log hobbitd_rrd --rrddir=/usr/local/hobbit/rrd hobbit 18478 18470 0 15:10 ? 00:00:00 hobbitd_channel --channel=data --log=/var/log/hobbit/rrd-data.log hobbitd_rrd --rrddir=/usr/local/hobbit/rrd hobbit 18479 18470 0 15:10 ? 00:00:00 hobbitd_channel --channel=client --log=/var/log/hobbit/clientdata.log hobbitd_client hobbit 18480 18478 0 15:10 ? 00:00:00 hobbitd_rrd --rrddir=/usr/local/hobbit/rrd hobbit 18481 18477 0 15:10 ? 00:00:14 hobbitd_rrd --rrddir=/usr/local/hobbit/rrd hobbit 18482 18479 0 15:10 ? 00:00:00 hobbitd_client hobbit 18634 18470 0 15:20 ? 00:00:00 bbtest-net --report --ping --checkresponse --timeout=60 --debug hobbit 21820 1 0 22:02 ? 00:00:00 sh -c vmstat 300 2 1>/usr/local/hobbit/client/tmp/hobbit_vmstat.21809 2>&1; mv /usr/local/hobbit/client/tmp/hobbit_vmstat.21809 /usr/local/hobbit/client/tmp/hobbit_vmstat hobbit 21821 21820 0 22:02 ? 00:00:00 vmstat 300 2 root 21861 21698 0 22:06 pts/0 00:00:00 grep hobbit [root at bigbrother etc]# lsof -p 18634 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME bbtest-ne 18634 hobbit cwd DIR 8,3 4096 376833 /usr/local/hobbit/server bbtest-ne 18634 hobbit rtd DIR 8,11 4096 2 / bbtest-ne 18634 hobbit txt REG 8,3 170076 393236 /usr/local/hobbit/server/bin/bbtest-net bbtest-ne 18634 hobbit mem REG 8,11 61504 80026 /lib/libnss_files-2.3.2.so bbtest-ne 18634 hobbit mem REG 8,11 14592 80056 /lib/liblaus.so.1.0.0 bbtest-ne 18634 hobbit mem REG 8,11 39468 82638 /lib/libpam.so.0.75 bbtest-ne 18634 hobbit mem REG 8,11 29100 80014 /lib/libcrypt-2.3.2.so bbtest-ne 18634 hobbit mem REG 8,7 28672 144356 /usr/lib/libgdbm.so.2.0.0 bbtest-ne 18634 hobbit mem REG 8,7 59608 144390 /usr/lib/libz.so.1.1.4 bbtest-ne 18634 hobbit mem REG 8,11 19992 80016 /lib/libdl-2.3.2.so bbtest-ne 18634 hobbit mem REG 8,11 79916 80036 /lib/libresolv-2.3.2.so bbtest-ne 18634 hobbit mem REG 8,7 78360 272188 /usr/kerberos/lib/libk5crypto.so.3.0 bbtest-ne 18634 hobbit mem REG 8,7 11072 272178 /usr/kerberos/lib/libcom_err.so.3.0 bbtest-ne 18634 hobbit mem REG 8,7 391564 272198 /usr/kerberos/lib/libkrb5.so.3.1 bbtest-ne 18634 hobbit mem REG 8,7 77448 272184 /usr/kerberos/lib/libgssapi_krb5.so.2.2 bbtest-ne 18634 hobbit mem REG 8,7 57768 144429 /usr/lib/libsasl.so.7.1.11 bbtest-ne 18634 hobbit mem REG 8,11 1608896 32013 /lib/tls/libc-2.3.2.so bbtest-ne 18634 hobbit mem REG 8,11 1104580 80070 /lib/libcrypto.so.0.9.7a bbtest-ne 18634 hobbit mem REG 8,11 220772 80071 /lib/libssl.so.0.9.7a bbtest-ne 18634 hobbit mem REG 8,7 49304 144433 /usr/lib/liblber.so.2.0.17 bbtest-ne 18634 hobbit mem REG 8,7 186348 144435 /usr/lib/libldap.so.2.0.17 bbtest-ne 18634 hobbit mem REG 8,11 115228 80005 /lib/ld-2.3.2.so bbtest-ne 18634 hobbit 0r CHR 1,3 65675 /dev/null bbtest-ne 18634 hobbit 1w REG 8,6 5775484 432036 /var/log/hobbit/bb-network.log bbtest-ne 18634 hobbit 2w REG 8,6 5775484 432036 /var/log/hobbit/bb-network.log bbtest-ne 18634 hobbit 3u IPv4 219456 UDP bigbrother:35123->n9000sd1.nro.glic.com:domain ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

1 0

Network test dying
by JamesHorwath＠glic.com 13 Mar '06

13 Mar '06

I have been running hobbit for several months now without incident. I am running hobbit 4.1.2p1 on Redhat Enterprise 3 on IBM pseries hardware. I haven't had any issues until this morning. Now it appears after about one hour of running the system flat out dies. I am sent a notification for every system connected. Then it appears the network process dies. I was running Tcpdump to see what was wrong. I see the completion of a network test about 30 minutes ago to a machine on the same subnet. I am not running iptables/ipchains. I am not experienced at hard-core hobbit debugging. I looked in /var/log/hobbit and don't see anything strange. There are no core files on the hobbit directory. Any advise on where to start? All my network test are now purple. Thanks in advance, Jim Jim Horwath ----------------------------------------- This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

2 1

Latest snapshot : problem with "local" mode client
by frederic.mangeant＠steria.com 13 Mar '06

13 Mar '06

Hi again Henrik I'm trying to use 4.2-pre client in "local" configuration mode, however it doesn't want to start : $ cat /BB/hobbit/client/logs/clientlaunch.log 2006-03-13 14:36:53 hobbitlaunch starting 2006-03-13 14:36:53 Loading tasklist configuration from /BB/hobbit/client/etc/clientlaunch.cfg $ cat /BB/hobbit/client/logs/hobbitclient.log /BB/hobbit/client/bin/hobbitclient.sh: line 54: /BB/hobbit/client/bin/hobbitd_client: No such file or directory I've just added "--local" to the CMD line in /BB/hobbit/client/etc/clientlaunch.cfg The hobbitd_client: file is available in /BB/hobbit/server/bin. Shoud I modify /BB/hobbit/client/bin/hobbitclient.sh to point to it ? It works fine in the usual way (thresholds on the server). -- Frédéric Mangeant Steria EDC Sophia-Antipolis

1 0

CUSTOM RRD DATA VIA SCRIPTS
by subhash.gada＠logicacmg.com 13 Mar '06

13 Mar '06

Hi, We have few custom tests that measures printq of the print server (windows). How can I write a script to create a rrd file and generate graphs. For example P Server 1 3/13/2006 Total queued jobs: 0 PRN324:0 PRN323:0 PRN142:0 PRN141:0 PRN140:0 PRN139:0 PRN138:0 P Server 2 3/13/2006 Total queued jobs: 0 PRN314:0 PRN137:0 PRN136:0 How to count the number of columns that vary from host to host. The column names are not common for all the hosts. Can any one help me further ? if [ "$TESTNAME"="printq" ] then p1=`grep "^PRN*:" $FNAME | awk '{print $2}'` p2=`grep "^PRN*:" $FNAME | awk '{print $2}'` p3=`grep "^PRN*:" $FNAME | awk '{print $2}'` # The RRD dataset definitions echo "DS:p1:GAUGE:600:0:U" echo "DS:p2:GAUGE:600:0:U" echo "DS:p3:GAUGE:600:0:U" # The filename echo "printq.rrd" # The data echo "$p1:$p2:$p3" Regards, Subhash Gada This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

1 0

network testing tool for single host/service
by Dominique.Frise＠unil.ch 13 Mar '06

13 Mar '06

Hi Henrik, Using Big Brother for several years, we wrote quite a few custom server/side scripts where we used bbnet tool from BB. This utility allowed us to test a single host/service pair on the command line. Example: URL=http://test.example.com $BBHOME/bin/bbnet $URL > $BBTMP/HTTP_$PROC$$ Your -definitely better- tool bbtest-net is bounded to $BBHOSTS and it does not seem possible to test a single host/service like above. A hack could be to redefine $BBHOSTS with a single entry before calling bbtest-net. I there another possibility to to this? Thanks for your reply. Dominique

3 5

Snapshot from 12/03
by frederic.mangeant＠steria.com 13 Mar '06

13 Mar '06

Hi Henrik the latest snapshot now builds fine on my test system (Gentoo Linux x86, Hobbit 4.2-pre built with GCC 3.4.5). When starting it, I get this error in hobbitd.log : 2006-03-13 10:24:41 How did this happen? clients=0, s.sem_op=0 2006-03-13 10:28:45 Setup complete However, it runs fine. Time to test the "disable until" and DOWNTIME settings for individual tests... Regards, -- Frédéric Mangeant Steria EDC Sophia-Antipolis

3 3

gmake install and chown/chgrp problem with 4.1.2pre on Solaris
by Dominique.Frise＠unil.ch 13 Mar '06

13 Mar '06

Hi Henrik, I noticed errors while installing stuff in the hobbitd directory. ----extract of gmake install----- ... chown bb /var/log/hobbit /soft/pub/BB/hobbit/server /soft/pub/BB/hobbit/server/* /soft/pub/BB/hobbit/server/bin/* /soft/pub/BB/hobbit/server/etc/* /soft/pub/BB/hobbit/server/ext/* /soft/pub/BB/hobbit/server/web/* /soft/pub/BB/hobbit/server/www/gifs /soft/pub/BB/hobbit/server/www/gifs/* /soft/pub/BB/hobbit/server/www/menu /soft/pub/BB/hobbit/server/www/menu/* /soft/pub/BB/hobbit/server/www/help /soft/pub/BB/hobbit/server/www/notes /soft/pub/BB/hobbit/server/www/html /soft/pub/BB/hobbit/server/www/wml /soft/pub/BB/hobbit/data /soft/pub/BB/hobbit/data/* /soft/pub/BB/hobbit/cgi-bin /soft/pub/BB/hobbit/cgi-bin/* /soft/pub/BB/hobbit/cgi-bin /soft/pub/BB/hobbit/cgi-bin/* chown: getting attributes of `/soft/pub/BB/hobbit/cgi-bin': No such file or directory chown: getting attributes of `/soft/pub/BB/hobbit/cgi-bin/*': No such file or directory chown: getting attributes of `/soft/pub/BB/hobbit/cgi-bin': No such file or directory chown: getting attributes of `/soft/pub/BB/hobbit/cgi-bin/*': No such file or directory gmake[1]: *** [install-cfg] Error 1 gmake[1]: Leaving directory `/soft/pub/hobbit-snapshot-4.1.2pre/hobbitd' gmake: *** [install-hobbitd] Error 2 ... ----------------------------------- At that point, the $(INSTALLROOT)$(CGIDIR) and $(INSTALLROOT)$(SECURECGIDIR) do not exist and the chown or chgrp for them return errors on Solaris. Dominique

2 1

[hobbit] Stop Reporting - All Device Become Purple
by zakki.ashidiqi＠indosat.com 13 Mar '06

13 Mar '06

Hi all, my problem for the purple status is still happening. There is nothing in the log file. Any body has idea? Thanks -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Tuesday, March 07, 2006 1:26 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Stop Reporting - All Device Become Purple On Tue, Mar 07, 2006 at 12:21:33PM +0700, Zakki Ashshidiqi wrote: > > My hobbit reported that all the device stopping the report. And the > status become purple. If I restart the hobbit process, all device will > be in green status. And in the next few hours it will be back to > purple again. Any suggestion? What's in the logfiles ? Especially the bb-network.log, hobbitd.log, hobbitclient.log and the hobbitlaunch.log. Normally, these are mostly empty. Does this happen for only the network tests (conn, ssh, http and so on), or the client-side tests (cpu, disk, memory, procs), or all tests ? Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk ================================== Scanned by Indosat TM-IMSS System ================================= scanned by Indosat TM-IMSS System

1 0

Mail on disabled hosts.
by tlp-hobbit＠holme-pedersen.dk 13 Mar '06

13 Mar '06

How can I make a rule that will send out mails on disabled hosts ? is this possible at all ? /Thomas

1 0