November 2005 - Xymon - Xymon mailman web

Email alert based on specific process
by iqbala-hobbit＠qwestip.net 15 Nov '06

15 Nov '06

Hi All I am using bb-proctab to monitor a list of processes and if any process a whole group get alert. Is there a way I can setup the alert with process specific. So if proc 1 out the list dies only user 1 get alert and so on. I know I could use the SCRIPT feature but like to keep that as the last resort. I am using hobbit 4.1.2 on Solaris 8. The process list I am concern about is running bb1.9c client on Solaris 8 as well. Sorry if it were already discussed in the mailing list. Thanks -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi

6 7

Small enhancement to Hobbit JavaScript menu
by frederic.mangeant＠steria.com 05 Oct '06

05 Oct '06

Hi Henrik I didn't find bb-datepage.sh the first time I set up Hobbit, maybe this small modification to ~/server/www/menu/menu_items.js would be helpful : --- menu_items.js.orig 2005-11-18 15:46:58.000000000 +0100 +++ menu_items.js 2005-11-18 15:53:50.000000000 +0100 @@ -6,7 +6,12 @@ ], ['Reports', null, null, ['Event log Report', '/hobbit-cgi/bb-eventlog.sh'], - ['Availability Report', '/hobbit-cgi/bb-rep.sh'], + ['Availability Reports', null, null, + ['Daily Report', '/hobbit-cgi/bb-datepage.sh?type=day'], + ['Weekly Report', '/hobbit-cgi/bb-datepage.sh?type=week'], + ['Monthly Report', '/hobbit-cgi/bb-datepage.sh?type=month'], + ['Custom Report', '/hobbit-cgi/bb-rep.sh'], + ], ['Snapshot Report', '/hobbit-cgi/bb-snapshot.sh'], ['Config Report', '/hobbit-cgi/hobbit-confreport.sh'], ], Regards, -- Frédéric Mangeant Steria EDC Sophia-Antipolis

3 2

[hobbit] Hobbit, Net-snmp snmptrap,and HPOV
by nfigaro＠effigie.fr 28 Jul '06

28 Jul '06

There is an option to snmptrap (from net snmp) to send full OIDs, otherwise they are cut. Instead of "snmptrap -v 1", the options should be : "snmptrap -v 1 -Of" Otherwise only the end of the OID is sent. Nicolas Figaro > -----Message d'origine----- > De : David Gore [mailto:David.Gore at mci.com] > Envoyé : lundi 19 septembre 2005 17:11 > À : hobbit at hswn.dk > Objet : Re: [hobbit] Hobbit, Net-snmp snmptrap,and HPOV > > > David Gore wrote: > > Coupe, Mark (ITD) wrote: > > > >> All, > >> > >> I'm trying to push alarms from Hobbit to HPOV. We have a custom > >> script that is invoking Netsnmp's snmptrap. The snmp traps are > >> reaching the HP Node Manager Event Browser correctly. > However, the OV > >> operations trap condition is not recognizing the trap. > The trap is > >> being displayed as "unmatched" in the Message Browser even though > >> there is a condition set up for that specific trap. > >> > >> When we look at the unmatched trap, it appears that the OID is not > >> being recognized and an '0' is inserted, for example instead of > >> .1.3.6.1.4.1.7058.0.XX, we are seeing .0.0.XX. > >> > >> Any guidance would be greatly appreaciated. > >> > >> Thanks, > >> > >> -Mark Coupe > > > > > > I, and perhaps others would be interested in what precisely you are > > doing? Are you pushing all hobbit alarms from the hobbit server to > > HPOV? > > > > Sorry, I am HPOV ignorant, but we are interested in > forwarding all the > > hobbit alarms/alerts as SNMP traps to one of Concord's > products. I am > > not to familiar with Concord's product either. I just know > the powers > > that be would like to have snmp traps from all our monitored hosts. > > > > In otherwords, I am interested in any system or methodology that > > someone > > has setup to forward alarms from the hobbit server to an > SNMP server. > > Any setups out there that someone is willing to share the details? > > > > David > > > > > > To unsubscribe from the hobbit list, send an e-mail to > > hobbit-unsubscribe at hswn.dk > > > > > > Here is Mark's script posted with permission: > > SCRIPT > > #!/bin/bash > > #SVCCODES="disk:100,cpu:200,procs:300,svcs:350,msgs:400,conn:5 > 00,http:60 > 0,ftp:721,ssh:722,ssh1:722,ssh2:722,telnet:723,smtp:725,fping: > 727,mq:729 > ,dns:800,pop2:809,pop-2:809,pop:810,pop3:810,pop-3:810,nntp:81 9,imap:843 > ,imap2:843,imap3:843,imap4:843,mrtg:850,swap:860,memory:870,sy > s:880,err: > 890,test:901" > > #Variables > #$OID - OID to be used without trailing value (.1.3.6.1.4.1.7058)(BB > default) > #$OIDTAIL - ending value of OID particluar to alarm type; > determined by "case" statement #$MACHIP - IP address of > device generating alarm; determined by hobbit #$BBHOSTNAME - > hostname of device generating alarm; determined by the hobbit > bb-hosts file #$BBSVCNUM - service code corresponding to the > service being monitored; determined by hobbit configuration > #$BBCOLORLEVEL - alarm level; determined by hobbit > #$RECOVERED - '0' denotes alarm; '1' denotes recovery #$HPOV > - the IP address of the destination HPOV server #TMPDIR - > directory for location of temporary file necessary for sed of $MACHIP > > ###Start script ### > ### > CHRONO=`date` > OID=.1.3.6.1.4.1.7058 > #Put your own OV server IP address in the next line. > HPOV=xxx.xxx.xxx.xxx TMPDIR=/home/hobbit/server/ext > > > case $BBSVCNUM in > 100) #Disk > OIDTAIL=10 > ;; > 200) #CPU > OIDTAIL=20 > ;; > 300) #Procs > OIDTAIL=30 > ;; > 350) #Svcs > OIDTAIL=40 > ;; > 400) #Msgs > OIDTAIL=50 > ;; > 500) #Conn > OIDTAIL=60 > ;; > 600) #Http > OIDTAIL=70 > ;; > #OIDTAIL=80 (not assigned) > 721) #FTP > OIDTAIL=90 > ;; > 722) #SSH > OIDTAIL=100 > ;; > 723) #Telnet > OIDTAIL=110 > ;; > 725) #SMTP > OIDTAIL=120 > ;; > 727) #Fping > OIDTAIL=130 > ;; > 729) #MQ > OIDTAIL=140 > ;; > 800) #DNS > OIDTAIL=150 > ;; > 809) #POP2 > OIDTAIL=160 > ;; > 810) #POP > OIDTAIL=170 > ;; > 819) #NNTP > OIDTAIL=180 > ;; > 843) #IMAP > OIDTAIL=190 > ;; > 850) #MRTG > OIDTAIL=200 > ;; > 860) #Swap > OIDTAIL=210 > ;; > 870) #Memory > OIDTAIL=220 > ;; > 880) #Sys > OIDTAIL=230 > ;; > 890) #ERR > OIDTAIL=240 > ;; > 901) #Test > OIDTAIL=250 > ;; > esac > > if [[ "$RECOVERED" = "0" ]]; then > if [[ "$BBCOLORLEVEL" = "purple" ]]; then > OIDTAIL=$(($OIDTAIL + 1)) > elif [[ "$BBCOLORLEVEL" = "yellow" ]]; then > OIDTAIL=$(($OIDTAIL + 2)) > elif [[ "$BBCOLORLEVEL" = "red" ]]; then > OIDTAIL=$(($OIDTAIL + 3)) > fi > elif [[ "$RECOVERED" = "1" ]]; then > BBCOLORLEVEL=green > fi > > #Following line: reads Hobbit IP into temporary file > echo $MACHIP > $TMPDIR/tmpfile.txt > > #Following lines: puts Hobbit 12 digit IP format in dotted > decimal and removes any leading zeroes. MACHIP=`sed > 's/$...$$...$$...$$...$/\1.\2.\3.\4/' > $TMPDIR/tmpfile.txt` > echo $MACHIP > $TMPDIR/tmpfile.txt > MACHIP=`sed -e 's/^00//' -e 's/^0//' -e 's/\.000/./' -e > 's/\.00/./' -e 's/\.0/./' -e 's/\.000/./' -e 's/\.00/./' -e > 's/\.0/./' -e 's/\.000/./' -e 's/\.00/./' -e 's/\.0/./' -e > 's/\.\.\./.0.0./' -e 's/\.\./.0./' -e 's/\.$/.0/' $TMPDIR/tmpfile.txt` > > #Following line: removes the temporary file > rm -f $TMPDIR/tmpfile.txt > > #Following line: forwards the error message to HPOV as an > SNMP message `snmptrap -v 1 -c mitcovo $HPOV "$OID" "$MACHIP" > 6 $OIDTAIL ""` > > echo "DATE: $CHRONO" >> /home/hobbit/server/ext/hpov-push.log > echo "snmptrap -v 1 -c public $HPOV "$OID" "$MACHIP" 6 > $OIDTAIL "" " >> /home/hobbit/server/ext/hpov-push.log > > > END SCRIPT > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > > >

5 5

Downsecs wrong?
by patrick_a_vaughan＠hotmail.com 01 Jun '06

01 Jun '06

I've just noticed that the $DOWNSECS variable seems to be wrong. I've seen lots of times that it's 0. I found one time today where the history page says the duration was 1:49:57, but $DOWNSECS in the recovery message was set to 48. I don't know when this started, but I've got odd recovery times going back a couple of weeks.

2 1

Disk RRD problem with hobbit-client and OSF1 (Tru64)
by astavitsky＠yahoo.com 04 May '06

04 May '06

Hi! I'm using hobbit 4.1.2 server and clients. Some of my OSF1 (Tru64 4.0&5.1) clients' filesystems are being captured in RRD by mountpoint name and others by the device/domain. These are RRD files for a particular host:: ls -l HOSTNAME/disk* -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,arc.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,nfs,BOSS.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,nfs.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-26 20:13 HOSTNAME/disk,report.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,root_domain#root.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act,CWNY.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act,LongIsland.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act,NewJersey.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,tmp_domain#tmp.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,usr_domain#usr.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,var,ase,mnt,asearc,arc.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,var,ase,mnt,asenfs,nfs.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,var_domain#var.rrd And here is the hobbitlog for this: HOSTNAME|disk|green||1130372298|1130621525|1130623325|0|0|172.16.52.21|-1|||Y green Sat Oct 29 17:31:58 EDT 2005 - Filesystems OK Filesystem 512-blocks Used Available Capacity Mounted on root_domain#root 409600 252902 128032 67% / usr_domain#usr 2516576 1825636 638672 75% /usr var_domain#var 9437184 4114720 5284080 44% /var tmp_domain#tmp 409600 2682 385376 1% /tmp /nfs at asenfs 88821680 37803820 50792192 43% /nfs /arc at asearc 106643104 69304154 37257648 66% /arc nfs_dmn#nfs 88821680 37803820 50792192 43% /var/ase/mnt/asenfs/nfs arc_dmn#arc 106643104 69304154 37257648 66% /var/ase/mnt/asearc/arc /NFS at CEBOSS 377479160 298515328 78963832 80% /nfs/BOSS rpt_dmn#progesa 106643104 61776456 44815856 58% /rpt rptact_dmn#act 106643104 75460796 31101280 71% /rpt/MakPrd/progesaPrd/Act rptcwny_dmn#cwny 106643104 28278562 78320496 27% /rpt/MakPrd/progesaPrd/Act/CWNY rptli_dmn#li 71095392 19095496 51949920 27% /rpt/MakPrd/progesaPrd/Act/LongIsland rptnj_dmn#nj 71095392 3627144 67417760 6% /rpt/MakPrd/progesaPrd/Act/NewJersey I read earlier messages about hobbit_larrd misdetecting NT/UNIX disk output, but in this case it uses different columns for different filesystems on the SAME host. I'm at a loss why this is happening and how to fix it. Thanks,

4 4

[hobbit] Dumb hobbit network test question
by frederic.mangeant＠steria.com 30 Jan '06

30 Jan '06

Hi Tom > -----Original Message----- > From: Kauffman, Tom [mailto:KauffmanT at nibco.com] > Sent: Tuesday, November 29, 2005 11:28 AM > To: hobbit at hswn.dk > Subject: [hobbit] Dumb hobbit network test question [snip] > Here's a sample of the network test error: > Service ftp on hudson is not OK : Unexpected service response > Service smtp on hudson is not OK : Unexpected service response Are you using the "--checkresponse" option ? I had the same "Unexpected service response" warnings until I removed it. http://www.hswn.dk/hobbit/help/manpages/man1/bbtest-net.1.html --checkresponse[=COLOR] When testing well-known services (e.g. FTP, SSH, SMTP, POP-2, POP- 3, IMAP, NNTP and rsync), bbtest-net will look for a valid service- specific "OK" response. If another reponse is seen, this will cause the test to report a warning (yellow) status. Without this option, the response from the service is ignored. The optional color-name is used to select a color other than yellow for the status message when the response is wrong. E.g. "-- checkresponse=red" will cause a "red" status message to be sent when the service does not respond as expected.

2 1

bb-disk_by_fs.sh
by iqbala-hobbit＠qwestip.net 29 Jan '06

29 Jan '06

Hi All Has anyone used bb-disk_by_fs.sh with BB client? I am trying to find out how it will work with HOBBIT? There are few things I am concern about. 1. How will I manage alerts per partition which this script offers? 2. How will the graph work? Thanks for any feedbacks. I guess I could always try it myself on a test system if there is no feedback already. -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change." - Charles Darwin

2 2

hobbitd-client crashes .......
by jens-ole.frimann＠thomson.com 13 Jan '06

13 Jan '06

the hobbitd-client crashes...... hobbit 4.1.2 running on fedora core 4 [hobbit at shire tmp]$ ls -lrt totalt 2440 -rw------- 1 hobbit hobbit 1347584 8 nov 03:17 core.367 -rw------- 1 hobbit hobbit 1347584 9 nov 02:03 core.2159 -rw------- 1 hobbit hobbit 1347584 9 nov 02:19 core.26334 -rw------- 1 hobbit hobbit 1347584 9 nov 02:39 core.26570 -rw------- 1 hobbit hobbit 1347584 9 nov 21:54 core.4670 -rw------- 1 hobbit hobbit 1347584 12 nov 02:29 core.14323 -rw------- 1 hobbit hobbit 1347584 13 nov 15:04 core.3043 <---- used this core file -rw-rw-r-- 1 hobbit hobbit 129 13 nov 23:42 alert.chk.sub -rw-rw-r-- 1 hobbit hobbit 742 13 nov 23:42 alert.chk -rw-rw-r-- 1 hobbit hobbit 276250 13 nov 23:44 hobbitd.chk type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (no debugging symbols found) Using host libthread_db library "/lib/libthread_db.so.1". Reading symbols from shared object read from target memory...(no debugging symbo ls found)...done. Loaded system supplied DSO at 0xfc0000 Core was generated by `hobbitd_client'. Program terminated with signal 6, Aborted. warning: svr4_current_sos: Can't read pathname for load map: Ind/ud-fejl Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2... (no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 #0 0x00fc0402 in __kernel_vsyscall () (gdb) bt #0 0x00fc0402 in __kernel_vsyscall () #1 0x003cd1f8 in raise () from /lib/libc.so.6 #2 0x003ce948 in abort () from /lib/libc.so.6 #3 0x0805515a in dprintf () #4 <signal handler called> #5 0x0804ac55 in ?? () #6 0x08421ed8 in ?? () #7 0x00000004 in ?? () #8 0xbfdc38e0 in ?? () #9 0xbfdc38dc in ?? () #10 0xb7f39f95 in ?? () #11 0xbfdc38f4 in ?? () #12 0xb7f379a8 in ?? () #13 0x00000000 in ?? () (gdb) Jens-Ole Frimann

3 2

[hobbit] Dumb hobbit network test question
by KauffmanT＠nibco.com 30 Nov '05

30 Nov '05

Oh, how it helps to have additional minds on these things. Reverse lookup looks to be the culprit. I cloned all these systems in a bit of a hurry -- and the cloning changed the dns resolution config to point at (in order) my D/R hotsite Win2003 domain controller (active), my D/R hotsite D/R test domain controller (non-existent), my D/R hotsite hobbit system (also not there), and THEN my local DNS server -- so if the D/R DC didn't answer (wonder why IT goes away every two hours?) I would go through multiple retries to non-existent systems. One more item added to the system clone checklist. Thanks! Tom -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Wednesday, November 30, 2005 4:59 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Dumb hobbit network test question On Wed, Nov 30, 2005 at 03:38:56PM -0500, Kauffman, Tom wrote: > Henrik, can you clarify what this really means? > > Address=10.8.224.9:21, open=1, res=0, err=1, connecttime=0.003110, totaltime=10.063026, "open=1" means that the connection to the server succeeded. The interesting thing here is that it took only 0.003 seconds to get a connection, but then Hobbit spent more than 10 seconds waiting for a banner to appear. It never did - at least not within those 10 secs; the "err=1" means it gave up waiting for the data and signals a timeout. > Address=10.8.224.38:21, open=1, res=0, err=0, connecttime=0.003060, totaltime=0.028471, > banner='220 wabash FTP server (Version 4.2 Sat Feb 5 10:12:55 CST 2005) ready. 221 Goodbye.' (86 bytes) This is a different server. Again, connecting takes about 0.003 secs, but the banner appears almost immediately - the entire exchange happens in 28 milliseconds. It might be that the FTP server performs a reverse DNS lookup of the Hobbit servers' IP address when Hobbit connects to check the FTP service. Sometimes DNS lookups take a while - maybe long enough for Hobbit to reach the 10 seconds timeout. Maybe your ftp server has a local DNS cache, and the timeout only happens when the cached DNS entry expires and has to be refreshed. One thing you can try is to add a "--timeout=30" option to the bbtest-net command in hobbitlaunch.cfg; that makes it wait up to 30 seconds before flagging a timeout. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

[hobbit] Dumb hobbit network test question
by KauffmanT＠nibco.com 30 Nov '05

30 Nov '05

OK -- I used the --debug option; it wasn't as bad as I thought it would be, the resulting log was just over 11 MB when my problem occurred and I could turn it off. Henrik, can you clarify what this really means? Address=10.8.224.9:21, open=1, res=0, err=1, connecttime=0.003110, totaltime=10.063026, Address=10.8.224.38:21, open=1, res=0, err=0, connecttime=0.003060, totaltime=0.028471, banner='220 wabash FTP server (Version 4.2 Sat Feb 5 10:12:55 CST 2005) ready. 221 Goodbye. ' (86 bytes) (good response) 2005-11-30 14:03:59 tcp_got_expected: No data in banner 2005-11-30 14:03:59 Adding to combo msg: status volga.ftp yellow  Wed Nov 30 14:03:00 2005 ftp NOT ok This system is showing a load of 0.1 (max 2.0) om a 2-way 1.6 GHz machine; the FTP connect time is 17.4 microseconds (avg) and peaked in the last 48 hours at 5.2 milliseconds TIA Tom Kauffman NIBCO, Inc -----Original Message----- From: Kauffman, Tom [mailto:KauffmanT at nibco.com] Sent: Tuesday, November 29, 2005 11:28 AM To: hobbit at hswn.dk Subject: [hobbit] Dumb hobbit network test question I've just done my tri-annual hardware shuffle, swapping out all my on-lease RS-6000 for brand spanking new systems. Part of this included upgrading from AIX 5.1 to AIX 5.3. Now, on half the systems (the last set replaced) I get errors on the smtp and ftp tests -- typically one test every two hours. Interestingly, all these systems barf on the same test cycle. This is obviously something not quite right in the AIX config, but I'm at a loss on what. I just found out today that we also have production rsh scripts that time out on the same cycle (yeah, I know -- but they've been rsh since year dot, and getting them to ssh is real low on the list . . ) Here's a sample of the network test error: Service ftp on hudson is not OK : Unexpected service response Service smtp on hudson is not OK : Unexpected service response How can I log the actual response? I'm currently running hobbit 4.03rc1; that's scheduled to change sometime next week. Other suggestions? TIA -- Tom Kauffman NIBCO, Inc To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

2 1